Abstract

The Bayesian “evidence ” approximation, which is closely related to generalized maximum likelihood, has recently been employed to determine the noise and weight-penalty terms for training neural nets. This paper shows that it is far simpler to perform the exact calculation than it is to set up the evidence approximation. Moreover, unlike that approximation, the exact result does not have to be re-calculated for every new data set. Nor does it require the running of complex numerical computer code (the exact result is closed form). In addition, it turns out that for neural nets, the evidence procedure’s MAP estimate is in toto approximation error. Another advantage of the exact analysis is that it does not lead to incorrect intuition, like the claim that one can “evaluate different priors in light of the data”. This paper ends by discussing sufficiency conditions for the evidence approximation to hold, along with the implications of those conditions. Although couched in terms of neural nets, the analysis of this paper holds for any Bayesian interpolation problem.

Citations

111	Bayesian back–propagation - Buntine, Weigend - 1991
68	The Axioms of Maximum Entropy - Skilling - 1988
67	Developments in maximum entropy data analysis - Gull - 1988
4	Optimization in the regularization of ill-posed problems - Davies, Anderssen, et al. - 1986
2	Classic maximum entropy. In "Maximum-entropy and Bayesian methods - Skilling - 1989
2	Bayesian Interpolation, and A Practical Framework for Backpropagation Networks - MacKay - 1992