Abstract

We give a tutorial and overview of the field of unsupervised learning from the perspective of statistical modelling. Unsupervised learning can be motivated from information theoretic and Bayesian principles. We briefly review basic models in unsupervised learning, including factor analysis, PCA, mixtures of Gaussians, ICA, hidden Markov models, state-space models, and many variants and extensions. We derive the EM algorithm and give an overview of fundamental concepts in graphical models, and inference algorithms on graphs. This is followed by a quick tour of approximate Bayesian inference, including Markov chain Monte Carlo (MCMC), Laplace approximation, BIC, variational approximations, and expectation propagation (EP). The aim of this chapter is to provide a high-level view of the field. Along the way, many state-of-the-art ideas and future directions are also reviewed. Contents 1

Citations

6232	Maximum likelihood from incomplete data via the EM algorithm - Dempster, Laird, et al. - 1977
5666	Probabilistic reasoning in intelligent systems - Pearl - 1988
1228	A global geometric framework for nonlinear dimensionality reduction - Tenenbaum, Silva, et al. - 2000
1156	Nonlinear dimensionality reduction by locally linear embedding - Roweis, Saul - 2000
1081	Local Computations with Probabilities on graphical structures and their applications to expert systems (with discussion - Lauritzen, Spiegelhalter
979	Near Shannon limit errorcorrecting coding and decoding: Turbo codes - Berrou, Glavieux, et al. - 1993
911	Condensation - conditional density propagation for visual tracking - Isard, Blake - 1998
910	Spatial interaction and the statistical analysis of lattice systems - Besag - 1974
759	Information Theory, Inference and Learning Algorithms - Mackay - 2003
752	Novel approach to nonlinear/non-gaussian bayesian state estimation - Gordon, Salmond, et al. - 1993
710	A tutorial on learning with Bayesian networks - Heckerman - 1995
663	Sequential Monte Carlo methods in practice. Statistics for engineering and information science - Doucet, Freitas, et al. - 2001
656	An introduction to variational methods for graphical models - Jordan, Ghahramani, et al. - 1999
612	A view of the EM algorithm that justifies incremental, sparse, and other variants - Neal, Hinton - 1998
581	Low-Density Parity-Check Codes - Gallager - 1963
496	The computational complexity of probabilistic inference using bayesian belief networks - Cooper - 1990
475	Optimal Filtering - Anderson, Moore - 1979
449	Probabilistic inference using Markov chain Monte Carlo methods - Neal - 1993
393	Dynamic Bayesian Networks: Representation, Inference and Learning - Murphy - 2002
375	Factorial hidden markov models - Ghahramani, Jordan - 1997
364	A learning algorithm for Boltzmann Machines - Ackley, Hinton, et al. - 1985
359	Probabilistic principal component analysis - Tipping, Bishop - 1999
349	Good error correcting codes based on very sparse matrices - MacKay - 1999
334	Mixtures of probabilistic principal component analyzers - Tipping, Bishop - 1999
328	Hierarchical Dirichlet processes - Teh, Jordan, et al. - 2006
325	Semi-supervised learning using gaussian fields and harmonic functions - Zhu, Ghahramani, et al.
308	Y: Generalized belief propagation - Yedidia, Freeman, et al.
291	C.E.: Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems - Antoniak - 1974
291	A new extension of the Kalman filter to nonlinear systems - Julier, Uhlmann - 1997
285	Bayesian density estimation and inference using mixtures - Escobar, West - 1995
276	Monte Carlo filter and smoother for non-Gaussian nonlinear state space models - Kitagawa - 1996
268	Graphical models, exponential families, and variational - Wainwright, Jordan - 2008
267	Complexity of finding embeddings in a k-tree - Arnborg, Corneil, et al. - 1987
257	Regularization theory and neural networks architectures - Girosi, Jones, et al. - 1995
247	Turbo decoding as an instance of Pearl’s belief propagation algorithm - McEliece, McKay, et al. - 1998
245	Markov chain sampling methods for Dirichlet process mixture models - Neal - 2000
238	Multidimensional scaling by optimizing goodness of fit to a non-metric hypothesis. Psychometrika 29 - Kruskal - 1964
234	GTM: The Generative Topographic Mapping - Bishop, Svensen, et al. - 1998
223	Expectation propagation for approximate Bayesian inference - Minka - 2001
208	A unifying review of linear Gaussian models - Roweis - 1999
197	G: The EM algorithm for mixtures of factor analyzers - Ghahramani, Hinton - 1997
190	The wake-sleep algorithm for unsupervised neural networks - Hinton, Dayan, et al. - 1995
189	The Bayesian Structural EM Algorithm - Friedman - 1998
185	A family of algorithms for approximate Bayesian inference - Minka - 2001
178	The analysis of proximities: Multidimensional scaling with an unknown distance function - Shepard - 1962
160	Connectionist learning of belief networks - Neal - 1992
160	An Approach to Time Series Smoothing and Forecasting Using the EM - Shumway, Stoffer - 1982
160	Partially labeled classification with Markov random walks - Szummer, Jaakkola - 2008
156	Being Bayesian about network structure. a Bayesian approach to structure discovery in Bayesian networks - Friedman, Koller
137	Modeling the manifolds of images of handwritten digits - Hinton, Dayan, et al. - 1997