This paper is a review of computational strategies for Bayesian shrinkage and variable selection in the linear model. Our focus is less on traditional MCMC methods, which are covered in depth by earlier review papers. Instead, we focus more on recent innovations in stochastic search and adaptive MCMC, along with some comparatively new research on shrinkage priors. One of our conclusions is that true MCMC seems inferior to stochastic search if one’s goal is to discover good models, but that stochastic search can result in biased estimates of variable inclusion probabilities. We also find reasons to question the accuracy of inclusion probabilities generated by traditional MCMC on high-dimensional, nonorthogonal problems, though the matter is far from settled. Some key words: adaptive MCMC; linear models; shrinkage priors; stochastic search; variable selection 1

Choosing the subset of covariates to use in regression or generalized linear models is a ubiquitous problem. The Bayesian paradigm can easily deal with this problem of model uncertainty by considering models corresponding to all possible subsets of the covariates, where the posterior distribution over models is used to select models or combine them in Bayesian model averaging. Although conceptually straightforward, it is often difficult to implement in practice, since either the number of covariates is too large, or calculations cannot be done analytically, or both. For orthogonal designs with the appropriate choice of prior, the posterior probability of any model can be calculated without having to enumerate the entire model space. In this article we propose a novel method, which augments the observed non-orthogonal design by new rows to obtain a design matrix with orthogonal columns. We show that our data augmentation approach keeps the original posterior distribution of interest unaltered, and develop methods to construct Rao-Blackwellized estimates of several quantities of interest, including posterior model probabilities, which may not be available from an ordinary Gibbs sampler. The method can be used for BMA in linear regression with Cauchy or other heavy tailed priors that may be represented as a scale mixture of normals, as well as binary regression. We provide simulated and real examples to illustrate the methodology. Supplemental materials for the manuscript are available online.

Choosing the subset of covariates to use in regression or generalized linear models is a ubiquitous problem. The Bayesian paradigm addresses the problem of model uncertainty by considering models corresponding to all possible subsets of the covariates, where the posterior distribution over models is used to select models or combine them via Bayesian model averaging (BMA). Although conceptually straightforward, BMA is often difficult to implement in practice, since either the number of covariates is too large for enumeration of all subsets, calculations cannot be done analytically, or both. For orthogonal designs with the appropriate choice of prior, the posterior probability of any model can be calculated without having to enumerate the entire model space and scales linearly with the number of predictors, p. In this article we extend this idea to a much broader class of non-orthogonal design matrices. We propose a novel method which augments the observed non-orthogonal design by at most p new rows to obtain a design matrix with orthogonal columns and generate the “missing ” response variables in a data augmentation algorithm. We show that our data augmentation approach

In variable selection problems that preclude enumeration of models, stochastic search algorithms, often based on Markov Chain Monte Carlo, are commonly used to identify a set of models for model selection or model averaging. Because Monte Carlo frequencies of models are often zero or one in high dimensional problems, posterior probabilities calculated from the observed marginal likelihoods, re-normalized over the sampled models are often employed. Such estimates are the only recourse in the newer generation of stochastic search algorithms. In this paper, we show that the approach of estimating model probabilities based on renormalization of posterior probabilities over the set of sampled models leads to bias in many quantities of interest and may not reduce mean squared error.