## A general framework for unsupervised processing of structured data (2004)

### Cached

### Download Links

- [www.math.unipd.it]
- [www.informatik.uni-osnabrueck.de]
- [www.informatik.uni-osnabrueck.de]
- [www2.in.tu-clausthal.de]
- DBLP

### Other Repositories/Bibliography

Venue: | NEUROCOMPUTING |

Citations: | 29 - 10 self |

### BibTeX

@MISC{Hammer04ageneral,

author = {Barbara Hammer and Alessio Micheli and Alessandro Sperduti and Marc Strickert},

title = { A general framework for unsupervised processing of structured data},

year = {2004}

}

### OpenURL

### Abstract

### Citations

3494 |
Self-Organizing Maps
- Kohonen
- 2001
(Show Context)
Citation Context ...ric based approach, therefore it can be applied directly to structured data if data comparison is defined and a notion of adaptation within the data space can be found. This has been proposed e.g. in =-=[18,29,48]-=-. Various approaches alternatively extend SOM by recurrent dynamics such as leaky integrators or more general recurrent connections which allow the recursive processing of sequences. Examples are the ... |

274 |
Stochastic Approximation Methods for Constrained and Unconstrained Systems
- Kushner, Clark
- 1978
(Show Context)
Citation Context ...n alternative. Moreover, there exist well-known guarantees for the convergence of a stochastic gradient descent if the learning rate Q (which may vary over time) fulfills certain properties, see e.g. =-=[35,36]-=-. However, a simple computation shows that the learning for structured data as introduced above cannot be interpreted as an exact gradient descent method. The update 22? z z í í ¥ ? ¥ í í í í mv í ... |

270 |
Neural-gas network for vector quantization and its application to time-series prediction
- Martinetz, Berkovich, et al.
- 1993
(Show Context)
Citation Context ...dels are usually trained with Hebbian learning. The general formulation allows to formalize Hebbian learning in a uniform manner and to immediately transfer alternatives like the neural gas algorithm =-=[37]-=- or vector quantization to the existing approaches. For standard vector-based SOM and alternatives like neural gas, Hebbian learning can be (approximately) interpreted as a stochastic gradient descent... |

264 | Learning long-term dependencies with gradient descent is difficult
- Bengio, Simard, et al.
- 1994
(Show Context)
Citation Context ...aining with so-called truncated gradient. Note that the well-known problem of long-term dependencies, i.e. the difficulty to latch information through several recursive layers, arises in this context =-=[4,26]-=-. This problem of long term dependencies is obviously avoided in Hebbian learning because the map is always trained for all substructures of each structure in the given data set. This allows us to dro... |

257 | Long short-term memory
- Hochreiter, Schmidhuber
- 1997
(Show Context)
Citation Context ...aining with so-called truncated gradient. Note that the well-known problem of long-term dependencies, i.e. the difficulty to latch information through several recursive layers, arises in this context =-=[4,26]-=-. This problem of long term dependencies is obviously avoided in Hebbian learning because the map is always trained for all substructures of each structure in the given data set. This allows us to dro... |

186 | Topology representing networks - Martinetz, Schulten - 1994 |

153 |
Neural computation and self-organizing maps: An introduction
- Ritter, Martinez, et al.
- 1991
(Show Context)
Citation Context ...have been transferred to recursive networks [13,15,21]. Unsupervised learning as alternative important paradigm for neural networks has been successfully applied in data mining and visualization (see =-=[31,42]-=-). Since additional structural information is often available in possible applications of selforganizing maps (SOMs), a transfer of standard unsupervised learning methods to sequences and more complex... |

146 | Gradient calculations for dynamic recurrent neural networks: a survey
- Pearlmutter
- 1995
(Show Context)
Citation Context ... discarded. The exact gradient mechanism includes recurrent neural network training as a special case, and explicit formulae comparable to backpropagation through time or real time recurrent learning =-=[40]-=- can be derived for the unsupervised case. This gives some hints to the understanding of the dynamics of unsupervised network training and constitutes a first step towards a general theory of unsuperv... |

121 | Exploiting the past and the future in protein secondary structure prediction
- Baldi, Brunak, et al.
- 1999
(Show Context)
Citation Context ...as been done while the first author was visiting the University of Pisa. She would like to thank the groups of Padua, Pisa, and Siena for their warm hospitality during her stay. 2language processing =-=[1,2,9,13]-=-. The training method for recursive networks is a straightforward generalization of standard backpropagation through time [46,47]. Moreover, important theoretical investigations from the field of feed... |

120 | A general framework for adaptive processing of data structures
- Frasconi, Gori, et al.
- 1998
(Show Context)
Citation Context ...time series prediction [16,17]. They can naturally be generalized to so-called recursive networks such that more complex data structures, tree structures and directed acyclic graphs can be dealt with =-=[14,47]-=-. Since symbolic terms possess a tree-representation, this generalization has successfully been applied in various areas where symbolic or hybrid data structures arise such as theorem proving, chemist... |

93 |
K.: Generalized Learning Vector Quantization
- Sato, Yamada
- 1996
(Show Context)
Citation Context ...ed recurrent networks like BPTT and RTRL. Moreover, this formulation proposes how to transfer different learning paradigms such as generalized vector quantization and variations to the recursive case =-=[23,44]-=-: learning vector quantization (LVQ) [31,32] constitutes a self-organizing supervised training method to learn a prototype based clustering of data with Hebb-style learning rules. The approach [44] pr... |

80 | Clustering based on conditional distributions in an auxiliary space
- Sinkkonen, Kaski
- 2001
(Show Context)
Citation Context ...ta. Since this method often yields too large dimensions, SOM with the standard Euclidian metric suffers from the curse of dimensionality, and methods which adapt the metric as proposed for example in =-=[23,30,45]-=- are advisable. Hierarchical and adaptive preprocessing methods which involve SOMs at various levels can be found e.g. in the WEBSOM approach for document retrieval [34]. Since self-organizing algorit... |

73 | Supervised neural networks for the classification of structures
- Sperduti, Starita
- 1997
(Show Context)
Citation Context ...time series prediction [16,17]. They can naturally be generalized to so-called recursive networks such that more complex data structures, tree structures and directed acyclic graphs can be dealt with =-=[14,47]-=-. Since symbolic terms possess a tree-representation, this generalization has successfully been applied in various areas where symbolic or hybrid data structures arise such as theorem proving, chemist... |

60 |
Learning Vector Quantization
- Kohonen
- 1988
(Show Context)
Citation Context ...reover, this formulation proposes how to transfer different learning paradigms such as generalized vector quantization and variations to the recursive case [23,44]: learning vector quantization (LVQ) =-=[31,32]-=- constitutes a self-organizing supervised training method to learn a prototype based clustering of data with Hebb-style learning rules. The approach [44] proposes a cost function for variants of LVQ a... |

57 |
Generalized relevance learning vector quantization. Neural Networks
- Hammer, Villmann
- 2002
(Show Context)
Citation Context ...ta. Since this method often yields too large dimensions, SOM with the standard Euclidian metric suffers from the curse of dimensionality, and methods which adapt the metric as proposed for example in =-=[23,30,45]-=- are advisable. Hierarchical and adaptive preprocessing methods which involve SOMs at various levels can be found e.g. in the WEBSOM approach for document retrieval [34]. Since self-organizing algorit... |

48 | Bankruptcy analysis with self-organizing maps in learning metrics
- Kaski
- 2001
(Show Context)
Citation Context ...ta. Since this method often yields too large dimensions, SOM with the standard Euclidian metric suffers from the curse of dimensionality, and methods which adapt the metric as proposed for example in =-=[23,30,45]-=- are advisable. Hierarchical and adaptive preprocessing methods which involve SOMs at various levels can be found e.g. in the WEBSOM approach for document retrieval [34]. Since self-organizing algorit... |

47 | Very large two-level SOM for the browsing of newsgroups
- Kohonen, Kaski, et al.
- 1996
(Show Context)
Citation Context ...e possibility of easy visualization #%$&(' if they are directly connected in the lattice. For other + ,-/. space. Given a set of training patterns 01 32 which is used e.g. in data mining applications =-=[34]-=-. Each neuron is equipped 265 0 , with a weight which represents the corresponding region of the data in , the weights of the neurons are adapted by Hebbian learning including neighborhood cooperation... |

43 | SARDNET: A self-organizing feature map for sequences
- James, Miikkulainen
- 1995
(Show Context)
Citation Context ...ore general recurrent connections which allow the recursive processing of sequences. Examples are the temporal Kohonen map (TKM) [5], the recursive SOM (RecSOM) [50–52], or the approaches proposed in =-=[11,27,28,33]-=-. The SOM for structured data (SOMSD) [19,20,46] constitutes a recursive mechanism capable of processing tree structured data, and thus also sequences, in an unsupervised way. Alternative models for u... |

38 | A selforganizing map for adaptive processing of structured data
- Hagenbuchner, Sperduti, et al.
(Show Context)
Citation Context ...ecursive processing of sequences. Examples are the temporal Kohonen map (TKM) [5], the recursive SOM (RecSOM) [50–52], or the approaches proposed in [11,27,28,33]. The SOM for structured data (SOMSD) =-=[19,20,46]-=- constitutes a recursive mechanism capable of processing tree structured data, and thus also sequences, in an unsupervised way. Alternative models for unsupervised time series processing use, for exam... |

38 |
Topology representing networks. Neural Networks 7(3):507–522
- Martinetz, Schulten
- 1994
(Show Context)
Citation Context ...ed prototype. Starting from the winning neuron, map traversal reveals similar known data. Popular alternative self-organizing algorithms are vector quantization (VQ) and the neural gas algorithm (NG) =-=[38]-=-. VQ aims at learning a representation of the data points without topology preservation. Hence no neighborhood structure is given in this case and the learning rule adapts only the winner at each step... |

32 | Self-Organizing Maps on non-euclidean Spaces
- Ritter
- 1999
(Show Context)
Citation Context ...could be the distance of the indices , for example. However, the lattice structure could be more complex such as a hexagonal grid structure or a grid with exponentially increasing number of neighbors =-=[41]-=-. In this case, an algorithm can be written as follows: initialize the weights at random repeat: for u A2*!su $ all ó subtrees in in inverse topological order: y !T2I!"u $ I"$ compute for all neurons ... |

29 | Using the SOM and local models in time-series prediction
- Vesanto
- 1997
(Show Context)
Citation Context ...ric based approach, therefore it can be applied directly to structured data if data comparison is defined and a notion of adaptation within the data space can be found. This has been proposed e.g. in =-=[18,29,48]-=-. Various approaches alternatively extend SOM by recurrent dynamics such as leaky integrators or more general recurrent connections which allow the recursive processing of sequences. Examples are the ... |

28 | Neural-Gas" network for vector quantization and its application to time-series prediction - Martinetz, Berkovich - 1993 |

26 |
Validation Indices for Graph Clustering
- Gunter, Bunke
(Show Context)
Citation Context ...ric based approach, therefore it can be applied directly to structured data if data comparison is defined and a notion of adaptation within the data space can be found. This has been proposed e.g. in =-=[18,29,48]-=-. Various approaches alternatively extend SOM by recurrent dynamics such as leaky integrators or more general recurrent connections which allow the recursive processing of sequences. Examples are the ... |

26 |
Recursive self-organizing maps. Neural Networks, 15:979–992, 2002
- Voegtlin
(Show Context)
Citation Context ...cture is less regular. It need not be in accordance with the lattice, and it proposes an alternative to SOM with data oriented lattice. This method is proposed as an alternative lattice for RecSOM in =-=[51]-=-. Like in the case of vector quantization, the representation in SOMSD requires adaptation. The distance between indices should be dynamically computed during the training algorithm according to the... |

25 | Convergence of Learning Algorithms with Constant Learning Rates - Kuan, Hornik - 1991 |

25 | Self-organizing maps, vector quantization, and mixture modeling
- Heskes
- 2001
(Show Context)
Citation Context ...pproaches. For standard vector-based SOM and alternatives like neural gas, Hebbian learning can be (approximately) interpreted as a stochastic gradient descent method on an appropriate error function =-=[25,37,42]-=-. One can uni3formly formulate analogous cost functions for the general framework for structural self-organizing maps and investigate the connection to Hebbian learning. It turns out that Hebbian lea... |

23 | Recurrent SOM with local linear models in time series prediction
- Koskela, Varsta, et al.
- 1998
(Show Context)
Citation Context ...ore general recurrent connections which allow the recursive processing of sequences. Examples are the temporal Kohonen map (TKM) [5], the recursive SOM (RecSOM) [50–52], or the approaches proposed in =-=[11,27,28,33]-=-. The SOM for structured data (SOMSD) [19,20,46] constitutes a recursive mechanism capable of processing tree structured data, and thus also sequences, in an unsupervised way. Alternative models for u... |

22 | Learning with Recurrent Neural Networks - Hammer - 2000 |

21 |
Time in Self-Organizing Maps: An Overview of Models
- Barreto, Araujo, et al.
(Show Context)
Citation Context ...equences, in an unsupervised way. Alternative models for unsupervised time series processing use, for example, hierarchical network architectures. An overview of important models can be found e.g. in =-=[3]-=-. We will here focus on models based on recursive dynamics for structured data and we will derive a generic formulation of recursive self-organizing maps. We will propose a general framework which tra... |

21 |
Application of cascade correlation networks for structures to chemistry
- Bianucci, Micheli, et al.
(Show Context)
Citation Context ...as been done while the first author was visiting the University of Pisa. She would like to thank the groups of Padua, Pisa, and Siena for their warm hospitality during her stay. 2language processing =-=[1,2,9,13]-=-. The training method for recursive networks is a straightforward generalization of standard backpropagation through time [46,47]. Moreover, important theoretical investigations from the field of feed... |

20 |
Spatio-temporal connectionist networks: A taxonomy and review
- Kremer
- 2001
(Show Context)
Citation Context ...5é Â é Â é Â é Â é Â ¥ ? é h7$ % !sæqç}D#è+!), % é 2Hn#OR, Â 2Hnxrs$& À ? u è+!), % é Œ2Hn OR, Â æqçFD À y é ? ? ? é Â é Â ? ? recurrent neural networks (RNN) have been established in the literature =-=[39]-=-. Most of the models can at least be simulated or approximated within the following simple Elman-dynamic which is proved in the reference [21]. Assume that sequences with entries G in are dealt with. ... |

20 |
Recursive self-organizing maps
- Voegtlin, Dominey
- 2002
(Show Context)
Citation Context ...successfully applied for learning motion-directivity sensitive maps as can be found in the visual cortex [12]. 7y t!ƒKj$& y y Ÿ Recursive SOM The recursive SOM (RecSOM) has been proposed by Voegtlin =-=[50,52]-=- as a mechanism for sequence prediction. The symbols of a given sequence are thereby recursively processed based on the already computed context. Each neuron is equipped with a ,-†. & weight and, addi... |

17 | Towards incremental parsing of natural language using recursive neural networks
- Costa, Frasconi, et al.
- 2003
(Show Context)
Citation Context ...as been done while the first author was visiting the University of Pisa. She would like to thank the groups of Padua, Pisa, and Siena for their warm hospitality during her stay. 2language processing =-=[1,2,9,13]-=-. The training method for recursive networks is a straightforward generalization of standard backpropagation through time [46,47]. Moreover, important theoretical investigations from the field of feed... |

16 |
The temporal Kohonen map,” Neural Networks
- Chappell, Taylor
- 1993
(Show Context)
Citation Context ...atively extend SOM by recurrent dynamics such as leaky integrators or more general recurrent connections which allow the recursive processing of sequences. Examples are the temporal Kohonen map (TKM) =-=[5]-=-, the recursive SOM (RecSOM) [50–52], or the approaches proposed in [11,27,28,33]. The SOM for structured data (SOMSD) [19,20,46] constitutes a recursive mechanism capable of processing tree structure... |

13 |
The Kohohnen algorithm: a powerful tool for analysing and representing multidimensional quantitative and qualitative variables
- Cottrell, Rousset
- 1997
(Show Context)
Citation Context ...structures instead of simple vectors in . An interesting line of research deals with the adaptation of self-organizing maps to qualitative variables where the Euclidian metric cannot be used directly =-=[7,8]-=-. In this article, we are particularly interested in complex discrete structures such as sequences and trees. The article [3] provides an overview of self-organizing networks which have been proposed ... |

13 | Context quantization and contextual self-organizing maps
- Voegtlin
- 2000
(Show Context)
Citation Context ...successfully applied for learning motion-directivity sensitive maps as can be found in the visual cortex [12]. 7y t!ƒKj$& y y Ÿ Recursive SOM The recursive SOM (RecSOM) has been proposed by Voegtlin =-=[50,52]-=- as a mechanism for sequence prediction. The symbols of a given sequence are thereby recursively processed based on the already computed context. Each neuron is equipped with a ,-†. & weight and, addi... |

10 | Modeling the self-organization of directional selectivity in the primary visual cortex
- Farkas, Miikkulainen
- 1999
(Show Context)
Citation Context ...olutions for , optimum weights . Apart from sequence recognition tasks, these models have been successfully applied for learning motion-directivity sensitive maps as can be found in the visual cortex =-=[12]-=-. 7y t!ƒKj$& y y Ÿ Recursive SOM The recursive SOM (RecSOM) has been proposed by Voegtlin [50,52] as a mechanism for sequence prediction. The symbols of a given sequence are thereby recursively proce... |

10 |
Neural networks for adaptive processing of structured data
- Sperduti
- 2001
(Show Context)
Citation Context ...a for their warm hospitality during her stay. 2language processing [1,2,9,13]. The training method for recursive networks is a straightforward generalization of standard backpropagation through time =-=[46,47]-=-. Moreover, important theoretical investigations from the field of feedforward and recurrent neural networks have been transferred to recursive networks [13,15,21]. Unsupervised learning as alternativ... |

10 | A spatiotemporal memory based on SOMs with activity diffusion
- Euliano, Principe
- 1999
(Show Context)
Citation Context ...ore general recurrent connections which allow the recursive processing of sequences. Examples are the temporal Kohonen map (TKM) [5], the recursive SOM (RecSOM) [50–52], or the approaches proposed in =-=[11,27,28,33]-=-. The SOM for structured data (SOMSD) [19,20,46] constitutes a recursive mechanism capable of processing tree structured data, and thus also sequences, in an unsupervised way. Alternative models for u... |

10 | A Supervised Self-Organizing Map for Structured Data - Hagenbuchner, Tsoi, et al. - 2001 |

9 | Theoretical aspects of the SOM algorithm, Neurocomputing 21(1–3 - Cottrell, Fort, et al. - 1998 |

9 |
Special issue on recurrent neural networks for sequence processing. Neurocomputing
- Gori, Mozer, et al.
- 1997
(Show Context)
Citation Context ...cessful approaches have been developed: Supervised recurrent neural networks constitute a well established approach for modeling sequential data e.g. for language processing or time series prediction =-=[16,17]-=-. They can naturally be generalized to so-called recursive networks such that more complex data structures, tree structures and directed acyclic graphs can be dealt with [14,47]. Since symbolic terms ... |

8 | Supervised neural networks for the classi cation of structures - Sperduti, Starita |

7 | The temporal kohonen map. Neural Networks 6 - Chappell, Taylor - 1993 |

7 |
Special issue on dynamic recurrent neural networks
- Giles, Kuhn, et al.
- 1994
(Show Context)
Citation Context ...cessful approaches have been developed: Supervised recurrent neural networks constitute a well established approach for modeling sequential data e.g. for language processing or time series prediction =-=[16,17]-=-. They can naturally be generalized to so-called recursive networks such that more complex data structures, tree structures and directed acyclic graphs can be dealt with [14,47]. Since symbolic terms ... |

6 | Mathematical aspects of neural networks - Hammer, Villmann - 2003 |

6 | An Extended Kohonen Feature Map for Sentence Recognition
- Hoekstra, Drossaers
- 1993
(Show Context)
Citation Context |

6 | Learning Efficiently with Neural Networks: A Theoretical Comparison between Structured and Flat Representations
- Frasconi, Gori, et al.
- 2000
(Show Context)
Citation Context ...f standard backpropagation through time [46,47]. Moreover, important theoretical investigations from the field of feedforward and recurrent neural networks have been transferred to recursive networks =-=[13,15,21]-=-. Unsupervised learning as alternative important paradigm for neural networks has been successfully applied in data mining and visualization (see [31,42]). Since additional structural information is o... |

5 |
Self-organizing maps, convergence properties, and energy functions
- Erwin, Obermayer, et al.
- 1992
(Show Context)
Citation Context ...ce from the given data point. SOM itself does not possess a cost function which could be transferred to the continuous case, i.e. if a data distribution instead of a finite training set is considered =-=[10]-=-. For the discrete case (i.e. a finite data set), an energy function can be constructed [42,43]. The article [25] proposes a cost function for a slightly modified version of SOM: even for the continuo... |