Results 1  10
of
18
Combining Estimators Using NonConstant Weighting Functions
 Advances in Neural Information Processing Systems 7
, 1995
"... This paper discusses the linearly weighted combination of estimators in which the weighting functions are dependent on the input. We show that the weighting functions can be derived... ..."
Abstract

Cited by 61 (4 self)
 Add to MetaCart
This paper discusses the linearly weighted combination of estimators in which the weighting functions are dependent on the input. We show that the weighting functions can be derived...
Practical Confidence and Prediction Intervals
 Advances in Neural Information Processing Systems 9
, 1997
"... We propose a new method to compute prediction intervals. Especially for small data sets the width of a prediction interval does not only depend on the variance of the target distribution, but also on the accuracy of our estimator of the mean of the target, i.e., on the width of the confidence interv ..."
Abstract

Cited by 23 (2 self)
 Add to MetaCart
We propose a new method to compute prediction intervals. Especially for small data sets the width of a prediction interval does not only depend on the variance of the target distribution, but also on the accuracy of our estimator of the mean of the target, i.e., on the width of the confidence interval. The confidence interval follows from the variation in an ensemble of neural networks, each of them trained and stopped on bootstrap replicates of the original data set. A second improvement is the use of the residuals on validation patterns instead of on training patterns for estimation of the variance of the target distribution. As illustrated on a synthetic example, our method is better than existing methods with regard to extrapolation and interpolation in data regimes with a limited amount of data, and yields prediction intervals which actual confidence levels are closer to the desired confidence levels. 1 STATISTICAL INTERVALS In this paper we will consider feedforward neural netwo...
Predictions with Confidence Intervals (Local Error Bars)
, 1994
"... We present a new method for obtaining local error bars, i.e., estimates of the confidence in the predicted value that depend on the input. We approach this problem of nonlinear regression in a maximum likelihood framework. We demonstrate our technique first on computer generated data with locally ..."
Abstract

Cited by 23 (3 self)
 Add to MetaCart
We present a new method for obtaining local error bars, i.e., estimates of the confidence in the predicted value that depend on the input. We approach this problem of nonlinear regression in a maximum likelihood framework. We demonstrate our technique first on computer generated data with locally varying, normally distributed target noise. We then apply it to the laser data from the Santa Fe Time Series Competition. Finally, we extend the technique to estimate error bars for iterated predictions, and apply it to the exact competition task where it gives the best performance to date. 1 Obtaining Error Bars Using a Maximum Likelihood Framework 1.1 Motivation and Concept Feedforward artificial neural networks are widely used and wellsuited for nonlinear regression. They can be interpreted as predicting the expected value of the conditional target distribution as a function of (or "conditioned on") the input pattern (e.g., Buntine & Weigend, 1991). This target distribution in re...
Averaging Regularized Estimators
 Neural Computation
, 1997
"... We compare the performance of averaged regularized estimators. ..."
Abstract

Cited by 23 (2 self)
 Add to MetaCart
We compare the performance of averaged regularized estimators.
Evaluating neural network predictors by bootstrapping
 Computer Science Department, University of Colorado at Boulder, ftp://ftp.cs.colorado.edu/pub/TimeSeries/MyPapers/bootstrap.ps
, 1994
"... AbstractWe present a new method, inspired by the bootstrap, whose goal it is to determine the quality and reliability of a neural network predictor. Our method leads to more robust forecasting along with a large amount of statistical information on forecast performance that we exploit. We exhibit t ..."
Abstract

Cited by 17 (5 self)
 Add to MetaCart
AbstractWe present a new method, inspired by the bootstrap, whose goal it is to determine the quality and reliability of a neural network predictor. Our method leads to more robust forecasting along with a large amount of statistical information on forecast performance that we exploit. We exhibit the method in the context of multivariate time series prediction on nancial data from the New York Stock Exchange. It turns out that the variation due to di erent resamplings (i.e., splits between training, crossvalidation, and test sets) is signi cantly larger than the variation due to di erent network conditions (such as architecture and initial weights). Furthermore, this method allows us to forecast a probability distribution, as opposed to the traditional case of just a single value at each time step. We demonstrate this on a strictly heldout test set that includes the 1987 stock market crash. We also compare the performance of the class of neural networks to identically bootstrapped linear models. 1
A Bootstrap Evaluation of the Effect of Data Splitting on Financial Time Series
 IEEE Transactions on Neural Networks
, 1998
"... This article exposes problems of the commonly used technique of splitting the available data into training, validation, and test sets that are held fixed, warns about drawing too strong conclusions from such static splits, and shows potential pitfalls of ignoring variability across splits. Using a b ..."
Abstract

Cited by 17 (3 self)
 Add to MetaCart
This article exposes problems of the commonly used technique of splitting the available data into training, validation, and test sets that are held fixed, warns about drawing too strong conclusions from such static splits, and shows potential pitfalls of ignoring variability across splits. Using a bootstrap or resampling method, we compare the uncertainty in the solution stemming from the data splitting with neural network specific uncertainties (parameter initialization, choice of number of hidden units, etc.). We present two results on data from the New York Stock Exchange. First, the variation due to different resamplings is significantly larger than the variation due to different network conditions. This result implies that it is important to not overinterpret a model, or an ensemble of models, estimated on one specific split of the data. Second, on each split, the neural network solution with early stopping is very close to a linear model; no significant nonlinearities are extrac...
Nonlinear Partial Least Squares
, 1995
"... We propose a new nonparametric regression method for highdimensional data, nonlinear partial least squares (NLPLS). NLPLS is motivated by projectionbased regression methods, e.g., partial least squares (PLS), projection pursuit (PPR), and feedforward neural networks. The model takes the form of a ..."
Abstract

Cited by 16 (0 self)
 Add to MetaCart
We propose a new nonparametric regression method for highdimensional data, nonlinear partial least squares (NLPLS). NLPLS is motivated by projectionbased regression methods, e.g., partial least squares (PLS), projection pursuit (PPR), and feedforward neural networks. The model takes the form of a composition of two functions. The first function in the composition projects the predictor variables onto a lowerdimensional curve or surface yielding scores, and the second predicts the response variable from the scores. We implement NLPLS with feedforward neural networks. NLPLS will often produce a more parsimonious model (fewer score vectors) than projectionbased methods, and the model is well suited for detecting outliers and future covariates requiring extrapolation. The scores are also shown to have useful interpretations. We also extend the model for multiple response variables and discuss situations when multiple response variab...
Confidence Intervals and Prediction Intervals for FeedForward Neural Networks
 Clinical Applications of Artificial Neural Networks
, 2001
"... d to feedforward networks. This includes a critique on Bayesian confidence intervals and classification. 1.1 Regression Regression analysis is a common statistical technique for modelling the relationshipb etween a response or dependent) variable y and a set x of regressors x 1 ,... ,x d (also k ..."
Abstract

Cited by 7 (0 self)
 Add to MetaCart
d to feedforward networks. This includes a critique on Bayesian confidence intervals and classification. 1.1 Regression Regression analysis is a common statistical technique for modelling the relationshipb etween a response or dependent) variable y and a set x of regressors x 1 ,... ,x d (also known as independent or explanatory variables). For example, the relationship couldb eb etween whether a patient has a malig1 2 Dybowski & Roberts nantbq ast tumor (the response variab e) and the patient's age and level of serum albq= n (the regressors). When an article includes a discussion of artificial neural networks, it is customary to refer to response variab les as targets and regressors as inputs. Furthermore, the ordered set {x 1 ,... ,x<F7
Construction Of Confidence Intervals In Neural Modeling Using A Linear Taylor Expansion
 Proceedings of the International Workshop on Advanced BlackBox Techniques for Nonlinear Modeling. Katholieke Universiteit
, 1998
"... : We introduce the theoretical results on the construction of confidence intervals for a nonlinear regression, based on the linear Taylor expansion of the corresponding nonlinear model output. The case of neural blackbox modeling is then analyzed, and illustrated on an industrial application. We sh ..."
Abstract

Cited by 5 (3 self)
 Add to MetaCart
: We introduce the theoretical results on the construction of confidence intervals for a nonlinear regression, based on the linear Taylor expansion of the corresponding nonlinear model output. The case of neural blackbox modeling is then analyzed, and illustrated on an industrial application. We show that the linear Taylor expansion not only provides a confidence interval at any point of interest, but also gives a tool to detect overfitting. Keywords: backpropagation algorithm, blackbox modeling, bootstrap methods, confidence intervals, least squares estimator, linear Taylor expansion, neural networks, overfitting detection. I. INTRODUCTION In neural network modeling studies, generally only an average estimate of a neural model reliability is given through its mean square error on a test set. Yet, the problem of the estimation of a given model reliability has been investigated to a great extent in nonlinear regression theory (see for example [Bates & Watts 88] [Seber & Wild 89]). ...
Neural Network Predictions With Error Bars
 Department of Electrical and Electronic Engineering
, 1997
"... When a neural network makes a prediction it will have an error that can be decomposed into the six following sources: (1a) model bias from data, (1b) model bias from training, (2a) model variance from data, (2b) model variance from training, (3) target noise, (4) input noise. We discuss methods for ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
When a neural network makes a prediction it will have an error that can be decomposed into the six following sources: (1a) model bias from data, (1b) model bias from training, (2a) model variance from data, (2b) model variance from training, (3) target noise, (4) input noise. We discuss methods for estimating each of these components of error in linear and nonlinear neural networks. In the nonlinear case, the prediction errors are estimated by using a combination of methods; the 'delta method' to capture model variance from data, a committee of networks to capture the model variance from training and an auxiliary network to capture the target noise. This is motivated by considering each neural net prediction to be a 'mixture' of predictions from different networks where the output of each network is modelled as a Gaussian. 1 Introduction In this paper we investigate methods for estimating the errors associated with predictions from neural networks. Error estimates are useful as they a...