Abstract

I examine two approximate methods for computational implementation of Bayesian hierarchical models, that is, models which include unknown hyperparameters such as regularization constants. In the `evidence framework' the model parameters are integrated over, and the resulting evidence is maximized over the hyperparameters. The optimized hyperparameters are used to define a Gaussian approximation to the posterior distribution. In the alternative `MAP' method, the true posterior probability is found by integrating over the hyperparameters. The true posterior is then maximized over the model parameters, and a Gaussian approximation is made. The similarities of the two approaches, and their relative merits, are discussed, and comparisons are made with the ideal hierarchical Bayesian solution. In moderately ill-posed problems, integration over hyperparameters yields a probability distribution with a skew peak which causes significant biases to arise in the MAP method. In contrast, the evidence framework is shown to introduce negligible predictive error, under straightforward conditions. General lessons are drawn concerning the distinctive properties of inference in many dimensions.

Citations

449	Probabilistic inference using markov chain monte carlo methods - Neal - 1971
417	Bayesian interpolation - MacKay - 1992
346	A practical Bayesian framework for back-propagation networks - MacKay - 1992
325	Bayesian Inference in Statistical Analysis - Box, Tiao - 1973
111	Bayesian back–propagation - Buntine, Weigend - 1991
96	Bayesian spectrum analysis and parameter estimation (Berlin: Springer) BUGS 1996 http://www.mrc-bsu.cam.ac.uk/bugs/welcome.shtml Buck B and Macauly V A eds 1991 Maximum Entropy in action, (Oxford: Oxford University Press) Ciuchini M et al 2001 2000 CKM-Tr - Bretthorst - 1988
71	Bayesian inductive inference and maximum entropy - GULL - 1988
65	Fundamentals of Statistical and Thermal Physics - Reif - 1965
59	A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem - Wahba - 1985
27	Bayesian learning via stochastic dynamics - Neal - 1993
20	Bayesian numerical analysis - Skilling - 1993
16	Bayesian non-linear modelling for the 1993 energy prediction competition - MacKay - 1996
13	Ace of Bayes: application of neural networks with pruning - Thodberg - 1993
7	Alpha, evidence, and the entropic prior - Strauss, Wolpert, et al. - 1993
4	The evidence framework applied to classi cation networks - MacKay - 1992
2	Solving oversampled data problems by Maximum Entropy - Bryan - 1990
2	Applications of maxmimum entropy techniques to HST data - Weir - 1991