In principle, the Bayesian approach to model selection is straightforward. Prior probability distributions are used to describe the uncertainty surrounding all unknowns. After observing the data, the posterior distribution provides a coherent post data summary of the remaining uncertainty which is relevant for model selection. However, the practical implementation of this approach often requires carefully tailored priors and novel posterior calculation methods. In this article, we illustrate some of the fundamental practical issues that arise for two different model selection problems: the variable selection problem for the linear model and the CART model selection problem.

Bayesian learning in undirected graphical models --- computing posterior distributions over parameters and predictive quantities --- is exceptionally difficult. We conjecture that for general undirected models, there are no tractable MCMC (Markov Chain Monte Carlo) schemes giving the correct equilibrium distribution over parameters. While this intractability, due to the partition function, is familiar to those performing parameter optimisation, Bayesian learning of posterior distributions over undirected model parameters has been unexplored and poses novel challenges. We propose several approximate MCMC schemes and test on fully observed binary models (Boltzmann machines) for a small coronary heart disease data set and larger artificial systems. While approximations must perform well on the model, their interaction with the sampling scheme is also important. Samplers based on variational mean-field approximations generally performed poorly, more advanced methods using loopy propagation, brief sampling and stochastic dynamics lead to acceptable parameter posteriors. Finally, we demonstrate these techniques on a Markov random field with hidden variables.

In a two-way contingency table, one is interested in checking the goodness of fit of simple models such as independence, quasi-independence, symmetry, or constant association, and estimating parameters which describe the association structure of the table. In a large table, one may be interested in detecting a few outlying cells which deviate from the main association pattern in the table. Bayesian tests of the above hypotheses are described using a prior defined on the set of interaction terms of the log-linear model. These tests and associated estimation procedures have several advantages over classical fitting/estimation procedures First, the tests above can give measures of evidence in support of simple hypotheses. Second, the Bayes factors can be used to give estimates of association parameters of the table which allow for uncertainty that the hypothesized model is true. These methods are illustrated for a number of tables. Key words and phrases: Bayes factors, Laplace method, Gib...