Results 1  10
of
39
Efficient BackProp
, 1998
"... . The convergence of backpropagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers expl ..."
Abstract

Cited by 209 (31 self)
 Add to MetaCart
. The convergence of backpropagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that secondorder optimization methods are advantageous for neural net training. It is shown that most "classical" secondorder methods are impractical for large neural networks. A few methods are proposed that do not have these limitations. 1 Introduction Backpropagation is a very popular neural network learning algorithm because it is conceptually simple, computationally efficient, and because it often works. However, getting it to work well, and sometimes to work at all, can seem more of an art than a science. Designing and training a network using backprop requires making many seemingly arbitrary choices such as the number ...
DENFIS: Dynamic Evolving NeuralFuzzy Inference System and Its Application for TimeSeries Prediction
, 2001
"... This paper introduces a new type of fuzzy inference systems, denoted as DENFIS (dynamic evolving neuralfuzzy inference system), for adaptive online and offline learning, and their application for dynamic time series prediction. DENFIS evolve through incremental, hybrid (supervised/unsupervised), ..."
Abstract

Cited by 114 (19 self)
 Add to MetaCart
(Show Context)
This paper introduces a new type of fuzzy inference systems, denoted as DENFIS (dynamic evolving neuralfuzzy inference system), for adaptive online and offline learning, and their application for dynamic time series prediction. DENFIS evolve through incremental, hybrid (supervised/unsupervised), learning and accommodate new input data, including new features, new classes, etc. through local element tuning. New fuzzy rules are created and updated during the operation of the system. At each time moment the output of DENFIS is calculated through a fuzzy inference system based on mmost activated fuzzy rules which are dynamically chosen from a fuzzy rule set. Two approaches are proposed: (1) dynamic creation of a firstorder TakagiSugeno type fuzzy rule set for a DENFIS online model; (2) creation of a firstorder TakagiSugeno type fuzzy rule set, or an expanded highorder one, for a DENFIS offline model. A set of fuzzy rules can be inserted into DENFIS before, or during its learning process. Fuzzy rules can also be extracted during the learning process or after it. An evolving clustering method (ECM), which is employed in both online and offline DENFIS models, is also introduced. It is demonstrated that DENFIS can effectively learn complex temporal sequences in an adaptive way and outperform some well known, existing models.
Evolving Fuzzy Neural Networks for Supervised/Unsupervised OnLine KnowledgeBased Learning
 IEEE TRANSACTIONS ON SYSTEMS, MAN AND CYBERNETICS
, 2001
"... The paper introduces evolving fuzzy neural networks (EFuNNs) as a means for the implementation of the evolving connectionist systems (ECOS) paradigm that is aimed at building online, adaptive intelligent systems that have both their structure and functionality evolving in time. EFuNNs evolve their ..."
Abstract

Cited by 36 (5 self)
 Add to MetaCart
The paper introduces evolving fuzzy neural networks (EFuNNs) as a means for the implementation of the evolving connectionist systems (ECOS) paradigm that is aimed at building online, adaptive intelligent systems that have both their structure and functionality evolving in time. EFuNNs evolve their structure and parameter values through incremental, hybrid supervised/unsupervised, online learning. They can accommodate new input data, including new features, new classes, etc. through local element tuning. New connections and new neurons are created during the operation of the system. EFuNNs can learn spatialtemporal sequences in an adaptive way through one pass learning, and automatically adapt their parameter values as they operate. Fuzzy or crisp rules can be inserted and extracted at any time of the EFuNN operation. The characteristics of EFuNNs are illustrated on several case study data sets for time series prediction and spoken word classification. Their performance is compared with traditional connectionist methods and systems. The applicability of EFuNNs as general purpose on line learning machines is discussed what concerns systems that learn from large databases, lifelong learning systems, online adaptive systems in different areas of Engineering.
OnLine Learning, Reasoning, Rule Extraction and Aggregation in Locally Optimized...
, 2001
"... A fuzzy neural networks are connectionist systems that facilitate learning from data, reasoning over fuzzy rules, rule insertion, rule extraction, and rule adaptation. The concept of a particular class of fuzzy neural networks, called FuNNs, is further developed in this paper to a new concept of evo ..."
Abstract

Cited by 20 (1 self)
 Add to MetaCart
A fuzzy neural networks are connectionist systems that facilitate learning from data, reasoning over fuzzy rules, rule insertion, rule extraction, and rule adaptation. The concept of a particular class of fuzzy neural networks, called FuNNs, is further developed in this paper to a new concept of evolving neurofuzzy systems (EFuNNs), with respective algorithms for learning, aggregation, rule insertion, rule extraction. EFuNNs operate in an online mode and learn incrementally through locally tuned elements. They grow as data arrive, and regularly shrink through pruning of nodes, or through node aggregation. The aggregation procedure is functionally equivalent to knowledge abstraction. EFuNNs are several orders of magnitude faster than FuNNs and other traditional connectionist models. Their features are illustrated on a benchmark data set. EFuNNs are suitable for fast learning of online incoming data (e.g., "nancial time series, biological process control), adaptive learning of speech and video data, incremental learning and knowledge discovery from large databases (e.g., in Bioinformatics), online tracing processes over time, lifelong learning. The paper includes also a short review of the most common types of rules used in the knowledgebased neural networks. ( 2001 Elsevier Science B.V. All rights reserved.
Dynamic Evolving Fuzzy Neural Networks with 'moutofn' Activation Nodes for Online Adaptive Systems
, 1999
"... The paper introduces a new type of evolving fuzzy neural networks (EFuNNs), denoted as mEFuNNs, for online learning and their applications for dynamic time series analysis and prediction. mEFuNNs evolve through incremental, hybrid (supervised/unsupervised), online learning, like the EFuNNs. They ..."
Abstract

Cited by 14 (4 self)
 Add to MetaCart
The paper introduces a new type of evolving fuzzy neural networks (EFuNNs), denoted as mEFuNNs, for online learning and their applications for dynamic time series analysis and prediction. mEFuNNs evolve through incremental, hybrid (supervised/unsupervised), online learning, like the EFuNNs. They can accommodate new input data, including new features, new classes, etc. through local element tuning. New connections and new neurons are created during the operation of the system. At each time moment the output vector of a mEFuNN is calculated based on the mmost activated rule nodes. Two approaches are proposed: (1) using weighted fuzzy rules of ZadehMamdani type; (2) using TakagiSugeno fuzzy rules that utilise dynamically changing and adapting values for the inference parameters. It is proved that the mEFuNNs can effectively learn complex temporal sequences in an adaptive
Transductive support vector machines and applications in bioinformatics for promoter recognition. Neural Information Processing—Letters and Reviews
"... This paper introduces a novel Transductive Support Vector Machine (TSVM) model and compares it with the traditional inductive SVM on a key problem in Bioinformatics promoter recognition. While inductive reasoning is concerned with the development of a model (a function) to approximate data from the ..."
Abstract

Cited by 12 (2 self)
 Add to MetaCart
This paper introduces a novel Transductive Support Vector Machine (TSVM) model and compares it with the traditional inductive SVM on a key problem in Bioinformatics promoter recognition. While inductive reasoning is concerned with the development of a model (a function) to approximate data from the whole problem space (induction), and consecutively using this model to predict output values for a new input vector (deduction), in the transductive inference systems a model is developed for every new input vector based on some closest to the new vector data from an existing database and this model is used to predict only the output for this vector. The TSVM outperforms by far the inductive SVM models applied on the same problems. Analysis is given on the
NFI: A NeuroFuzzy Inference Method for Transductive Reasoning
 IEEE Transactions on Fuzzy Systems
, 2004
"... Abstract—This paper introduces a novel neural fuzzy inference method—NFI for transductive reasoning systems. NFI develops further some ideas from DENFIS—dynamic neurofuzzy inference systems for both online and offline time series prediction tasks. While inductive reasoning is concerned with the dev ..."
Abstract

Cited by 10 (1 self)
 Add to MetaCart
(Show Context)
Abstract—This paper introduces a novel neural fuzzy inference method—NFI for transductive reasoning systems. NFI develops further some ideas from DENFIS—dynamic neurofuzzy inference systems for both online and offline time series prediction tasks. While inductive reasoning is concerned with the development of a model (a function) to approximate data in the whole problem space (induction), and consecutively—using this model to predict output values for a new input vector (deduction), in transductive reasoning systems a local model is developed for every new input vector, based on some closest to this vector data from an existing database (also generated from an existing model). NFI is compared with both inductive connectionist systems (e.g., MLP, DENFIS) and transductive reasoning systems (e.g., KNN) on three case study prediction/identification problems. The first one is a prediction task on Mackey Glass time series; the second one is a classification on Iris data; and the last one is a real medical decision support problem of estimating the level of renal function of a patient, based on measured clinical parameters for the purpose of their personalised treatment. The case studies have demonstrated better accuracy obtained with the use of the NFI transductive reasoning in comparison with the inductive reasoning systems. Index Terms—Adaptive systems, neuralfuzzy inference (NFI), renal function evaluation, time series prediction, transductive reasoning.
On FokkerPlanck approximations of online learning processes
 Journal of Physics A
, 1994
"... There are several ways to describe online learning in neural networks. The two major ones are a continuoustime master equation and a discretetime randomwalk equation. The randomwalk equation is obtained in case of fixed time intervals between subsequent learning steps, the master equation resul ..."
Abstract

Cited by 8 (6 self)
 Add to MetaCart
(Show Context)
There are several ways to describe online learning in neural networks. The two major ones are a continuoustime master equation and a discretetime randomwalk equation. The randomwalk equation is obtained in case of fixed time intervals between subsequent learning steps, the master equation results when the time intervals are drawn from a Poisson distribution. Following Van Kampen [1], we give a rigorous expansion of both the master and the randomwalk equation in the limit of small learning parameters. The results explain the difference between the FokkerPlanck approaches proposed by Radons et al. [2] and Hansen et al. [3]. Furthermore, we find that the mathematical validity of these approaches is restricted to local properties of the learning process. Yet FokkerPlanck approaches are often suggested as models to study global properties, such as mean first passage times and stationary solutions. To check their accuracy and usefulness in these situations we compare simulations of t...
Stochastic Dynamics of Learning with Momentum in Neural Networks
, 1994
"... We study online learning with momentum term for nonlinear learning rules. Through introduction of auxiliary variables, we show that the learning process can be described by a Markov process. For small learning parameters j and momentum parameters ff close to 1, such that fl = j=(1 \Gamma ff) 2 i ..."
Abstract

Cited by 7 (4 self)
 Add to MetaCart
We study online learning with momentum term for nonlinear learning rules. Through introduction of auxiliary variables, we show that the learning process can be described by a Markov process. For small learning parameters j and momentum parameters ff close to 1, such that fl = j=(1 \Gamma ff) 2 is finite, the time scales for the evolution of the weights and the auxiliary variables are the same. In this case Van Kampen's expansion can be applied in a straightforward manner. We obtain evolution equations for the average network state and the fluctuations around this average. These evolution equations depend (after rescaling of time and fluctuations) only on fl: all combinations (j; ff) with the same value of fl give rise to similar behaviour. The case ff constant and j small requires a completely different analysis. There are two different time scales: a fast time scale on which the auxiliary variables equilibrate and a slow time scale for the change of the weights. By projection on t...