Results 1  10
of
40
Sparse modelling using orthogonal forward regression with press statistic and regularization
 IEEE TRANS. SYSTEMS, MAN AND CYBERNETICS, PART B
, 2004
"... The paper introduces an efficient construction algorithm for obtaining sparse linearintheweights regression models based on an approach of directly optimizing model generalization capability. This is achieved by utilizing the delete1 cross validation concept and the associated leaveoneout tes ..."
Abstract

Cited by 49 (23 self)
 Add to MetaCart
The paper introduces an efficient construction algorithm for obtaining sparse linearintheweights regression models based on an approach of directly optimizing model generalization capability. This is achieved by utilizing the delete1 cross validation concept and the associated leaveoneout test error also known as the predicted residual sums of squares (PRESS) statistic, without resorting to any other validation data set for model evaluation in the model construction process. Computational efficiency is ensured using an orthogonal forward regression, but the algorithm incrementally minimizes the PRESS statistic instead of the usual sum of the squared training errors. A local regularization method can naturally be incorporated into the model selection procedure to further enforce model sparsity. The proposed algorithm is fully automatic, and the user is not required to specify any criterion to terminate the model construction procedure. Comparisons with some of the existing stateofart modeling methods are given, and several examples are included to demonstrate the ability of the proposed algorithm to effectively construct sparse models that generalize well.
Probabilistic neuralnetwork structure determination for pattern classi
 IEEE Transactions on Neural Networks
, 2000
"... Abstract—Network structure determination is an important issue in pattern classification based on a probabilistic neural network. In this study, a supervised network structure determination algorithm is proposed. The proposed algorithm consists of two parts and runs in an iterative way. The first p ..."
Abstract

Cited by 26 (0 self)
 Add to MetaCart
(Show Context)
Abstract—Network structure determination is an important issue in pattern classification based on a probabilistic neural network. In this study, a supervised network structure determination algorithm is proposed. The proposed algorithm consists of two parts and runs in an iterative way. The first part identifies an appropriate smoothing parameter using a genetic algorithm, while the second part determines suitable pattern layer neurons using a forward regression orthogonal algorithm. The proposed algorithm is capable of offering a fairly small network structure with satisfactory classification accuracy. Index Terms—Genetic algorithms, orthogonal algorithm, pattern classification, probabilistic neural network (PNN). I.
A kernelbased twoclass classifier for imbalanced data sets
 IEEE Trans Neural Netw. 2007; 18: 28–41. PMID: 17278459
"... ..."
Evolutionary optimization of radial basis function classifiers for data mining applications
 IEEE Transactions on Systems Man and Cybernetics Part BCybernetics
, 2005
"... Abstract—In many data mining applications that address classification problems, feature and model selection are considered as key tasks. That is, appropriate input features of the classifier must be selected from a given (and often large) set of possible features and structure parameters of the clas ..."
Abstract

Cited by 16 (0 self)
 Add to MetaCart
(Show Context)
Abstract—In many data mining applications that address classification problems, feature and model selection are considered as key tasks. That is, appropriate input features of the classifier must be selected from a given (and often large) set of possible features and structure parameters of the classifier must be adapted with respect to these features and a given data set. This paper describes an evolutionary algorithm (EA) that performs feature and model selection simultaneously for radial basis function (RBF) classifiers. In order to reduce the optimization effort, various techniques are integrated that accelerate and improve the EA significantly: hybrid training of RBF networks, lazy evaluation, consideration of soft constraints by means of penalty terms, and temperaturebased adaptive control of the EA. The feasibility and the benefits of the approach are demonstrated by means of four data mining problems: intrusion detection in computer networks, biometric signature verification, customer acquisition with direct marketing methods, and optimization of chemical production processes. It is shown that, compared to earlier EAbased RBF optimization techniques, the runtime is reduced by up to 99% while error rates are lowered by up to 86%, depending on the application. The algorithm is independent of specific applications so that many ideas and solutions can be transferred to other classifier paradigms. Index Terms—Data mining, evolutionary algorithm (EA), feature selection, model selection, radial basis function (RBF) network. I.
Neuroevolutionary reinforcement learning for generalized helicopter control
, 2009
"... Helicopter hovering is an important challenge problem in the field of reinforcement learning. This paper considers several neuroevolutionary approaches to discovering robust controllers for a generalized version of the problem used in the 2008 Reinforcement Learning Competition, in which wind in the ..."
Abstract

Cited by 11 (2 self)
 Add to MetaCart
Helicopter hovering is an important challenge problem in the field of reinforcement learning. This paper considers several neuroevolutionary approaches to discovering robust controllers for a generalized version of the problem used in the 2008 Reinforcement Learning Competition, in which wind in the helicopter’s environment varies from run to run. We present the simple modelfree strategy that won first place in the competition and also describe several more complex modelbased approaches. Our empirical results demonstrate that neuroevolution is effective at optimizing the weights of multilayer perceptrons, that linear regression is faster and more effective than evolution for learning models, and that modelbased approaches can outperform the simple modelfree strategy, especially if prior knowledge is used to aid model learning.
Particle swarm optimization aided orthogonal forward regression for unified data modelling
 IEEE TRANS. EVOLUTION. COMPUT
, 2010
"... We propose a unified data modeling approach that is equally applicable to supervised regression and classification applications, as well as to unsupervised probability density function estimation. A particle swarm optimization (PSO) aided orthogonal forward regression (OFR) algorithm based on leave ..."
Abstract

Cited by 8 (4 self)
 Add to MetaCart
We propose a unified data modeling approach that is equally applicable to supervised regression and classification applications, as well as to unsupervised probability density function estimation. A particle swarm optimization (PSO) aided orthogonal forward regression (OFR) algorithm based on leaveoneout (LOO) criteria is developed to construct parsimonious radial basis function (RBF) networks with tunable nodes. Each stage of the construction process determines the center vector and diagonal covariance matrix of one RBF node by minimizing the LOO statistics. For regression applications, the LOO criterion is chosen to be the LOO mean square error, while the LOO misclassification rate is adopted in twoclass classification applications. By adopting the Parzen window estimate as the desired response, the unsupervised density estimation problem is transformed into a constrained regression problem. This PSO aided OFR algorithm for tunablenode RBF networks is capable of constructing very parsimonious RBF models that generalize well, and our analysis and experimental results demonstrate that the algorithm is computationally even simpler than the efficient regularization assisted orthogonal least square algorithm based on LOO criteria for selecting fixednode RBF models. Another significant advantage of the proposed learning procedure is that it does not have learning hyperparameters that have to be tuned using costly cross validation. The effectiveness of the proposed PSO aided OFR construction procedure is illustrated using several examples taken from regression and classification, as well as density estimation applications.
Construction of Tunable Radial Basis Function Networks Using Orthogonal Forward Selection
"... Abstract—An orthogonal forward selection (OFS) algorithm based on leaveoneout (LOO) criteria is proposed for the construction of radial basis function (RBF) networks with tunable nodes. Each stage of the construction process determines an RBF node, namely, its center vector and diagonal covariance ..."
Abstract

Cited by 6 (4 self)
 Add to MetaCart
Abstract—An orthogonal forward selection (OFS) algorithm based on leaveoneout (LOO) criteria is proposed for the construction of radial basis function (RBF) networks with tunable nodes. Each stage of the construction process determines an RBF node, namely, its center vector and diagonal covariance matrix, by minimizing the LOO statistics. For regression application, the LOO criterion is chosen to be the LOO meansquare error, while the LOO misclassification rate is adopted in twoclass classification application. This OFSLOO algorithm is computationally efficient, and it is capable of constructing parsimonious RBF networks that generalize well. Moreover, the proposed algorithm is fully automatic, and the user does not need to specify a termination criterion for the construction process. The effectiveness of the proposed RBF network construction procedure is demonstrated using examples taken from both regression and classification applications. Index Terms—Classification, leaveoneout (LOO) statistics, orthogonal forward selection (OFS), radial basis function (RBF) network, regression, tunable node. I.
Modeling systems with internal state using Evolino
 In Proc. of the 2005 conference on genetic and evolutionary computation (GECCO
, 2005
"... Existing Recurrent Neural Networks (RNNs) are limited in their ability to model dynamical systems with nonlinearities and hidden internal states. Here we use our general framework for sequence learning, EVOlution of recurrent systems with LINear Outputs (Evolino), to discover good RNN hidden node we ..."
Abstract

Cited by 6 (1 self)
 Add to MetaCart
(Show Context)
Existing Recurrent Neural Networks (RNNs) are limited in their ability to model dynamical systems with nonlinearities and hidden internal states. Here we use our general framework for sequence learning, EVOlution of recurrent systems with LINear Outputs (Evolino), to discover good RNN hidden node weights through evolution, while using linear regression to compute an optimal linear mapping from hidden state to output. Using the Long ShortTerm Memory RNN Architecture, Evolino outperforms previous stateoftheart methods on several tasks: 1) contextsensitive languages, 2) multiple superimposed sine waves. Categories and Subject Descriptors I.2.6 [Artificial Intelligence]: Learning—Connectionism and neural nets
Robust nonlinear model identification method forward regression
 IEEE Transactions on Systems, Man and Cybernetics, Part A
, 2003
"... Abstract—In this correspondence new robust nonlinear model construction algorithms for a large class of linearintheparameters models are introduced to enhance model robustness via combined parameter regularization and new robust structural selective criteria. In parallel to parameter regularizat ..."
Abstract

Cited by 6 (4 self)
 Add to MetaCart
Abstract—In this correspondence new robust nonlinear model construction algorithms for a large class of linearintheparameters models are introduced to enhance model robustness via combined parameter regularization and new robust structural selective criteria. In parallel to parameter regularization, we use two classes of robust model selection criteria based on either experimental design criteria that optimizes model adequacy, or the predicted residual sums of squares (PRESS) statistic that optimizes model generalization capability, respectively. Three robust identification algorithms are introduced, i.e., combined A and Doptimality with regularized orthogonal least squares algorithm, respectively; and combined PRESS statistic with regularized orthogonal least squares algorithm. A common characteristic of these algorithms is that the inherent computation efficiency associated with the orthogonalization scheme in orthogonal least squares or regularized orthogonal least squares has been extended such that the new algorithms are computationally efficient. Numerical examples are included to demonstrate effectiveness of the algorithms. Index Terms—Cross validation, experimental design, forward regression, generalization, structure identification. I.
MEstimator and DOptimality Model Construction Using Orthogonal Forward Regression
"... Abstract—This correspondence introduces a new orthogonal forward regression (OFR) model identification algorithm using Doptimality for model structure selection and is based on an Mestimators of parameter estimates. Mestimator is a classical robust parameter estimation technique to tackle bad dat ..."
Abstract

Cited by 4 (1 self)
 Add to MetaCart
(Show Context)
Abstract—This correspondence introduces a new orthogonal forward regression (OFR) model identification algorithm using Doptimality for model structure selection and is based on an Mestimators of parameter estimates. Mestimator is a classical robust parameter estimation technique to tackle bad data conditions such as outliers. Computationally, The Mestimator can be derived using an iterative reweighted least squares (IRLS) algorithm. Doptimality is a model structure robustness criterion in experimental design to tackle illconditioning in model structure. The orthogonal forward regression (OFR), often based on the modified Gram–Schmidt procedure, is an efficient method incorporating structure selection and parameter estimation simultaneously. The basic idea of the proposed approach is to incorporate an IRLS inner loop into the modified Gram–Schmidt procedure. In this manner, the OFR algorithm for parsimonious model structure determination is extended to bad data conditions with improved performance via the derivation of parameter Mestimators with inherent robustness to outliers. Numerical examples are included to demonstrate the effectiveness of the proposed algorithm. Index Terms—Forward regression, Gram–Schmidt, identification, Mestimator, model structure selection. I.