Elicitation is a key task for subjectivist Bayesians. While skeptics hold that it cannot (or perhaps should not) be done, in practice it brings statisticians closer to their clients and subjectmatter-expert colleagues. This paper reviews the state-of-the-art, reflecting the experience of statisticians informed by the fruits of a long line of psychological research into how people represent uncertain information cognitively, and how they respond to questions about that information. In a discussion of the elicitation process, the first issue to address is what it means for an elicitation to be successful, i.e. what criteria should be employed? Our answer is that a successful elicitation faithfully represents the opinion of the person being elicited. It is not necessarily “true ” in some objectivistic sense, and cannot be judged that way. We see elicitation as simply part of the process of statistical modeling. Indeed in a hierarchical model it is ambiguous at which point the likelihood ends and the prior begins. Thus the same kinds of judgment that inform statistical modeling in general also inform elicitation of prior distributions.

Applications of modern methods for analyzing data with missing values, based primarily on multiple imputation, have in the last half-decade become common in American politics and political behavior. Scholars in this subset of political science have thus increasingly avoided the biases and inefficiencies caused by ad hoc methods like listwise deletion and best guess imputation. However, researchers in much of comparative politics and international relations, and others with similar data, have been unable to do the same because the best available imputation methods work poorly with the time-series cross section data structures common in these fields. Weattempttorectify this situation with three related developments. First, we build a multiple imputation model that allows smooth time trends, shifts across cross-sectional units, and correlations over time and space, resulting in far more accurate imputations. Second, we enable analysts to incorporate knowledge from area studies experts via priors on individual missing cell values, rather than on difficult-to-interpret model parameters. Third, because these tasks could not be accomplished within existing imputation algorithms, in that they cannot handle as many variables as needed even in the simpler cross-sectional data for which they were designed, we also develop a new algorithm that substantially expands the range of computationally feasible data types and sizes for which multiple imputation can be used. These developments also make it possible to implement the methods introduced here in freely available open source software that is considerably more reliable than existing algorithms. We develop an approach to analyzing data with

INTRODUCTION The statistical problems envisaged in our pedagogy are almost always ones in which we acquire new data D that give evidence concerning some hypotheses H; H 0 ; : : : (this includes parameter estimation, since H might be the statement that a parameter lies in a certain interval); and we make inferences about them solely from the data. Indeed, Fisher's maxim, "Let the data speak for themselves" seems to imply that it would be wrong -- a violation of "scientific objectivity" -- to allow ourselves to be influenced by other considerations such as prior knowledge about H . Yet the very act of choosing a model (i.e. a sampling distribution conditional on H) is a means of expressing some kind of prior knowledge about the existence and nature of H , and its observable effects. This was noted by John Tukey (1978), who observed that sampling theory is in the curious

Consider the problem of comparing parametric models M 1 ; : : : ; M k , when at least one of the models has an improper prior ß N i (` i ). Using the Bayes factor for comparing among these is not feasible due to arbitrary multiplicative constants in ß N (` i ). In this work we suggest adjusting the initial priors for each model, ß N i , by ß i (` i ) = Z ß N i (` i jy )m (y )dy where m is a suitable predictive measure on (imaginary) training samples, y . The updated prior, ß , is called the expected posterior prior under m . Some properties of this approach include: (1) The resulting Bayes factors depend only on sufficient statistics. (2) The resulting Bayesian inference is coherent and allows for multiple comparisons. (3) In many cases, it is possible to find m such that, for a sample of minimal size, there is predictive matching for the comparisons of model M i to M j ,i.e., the Bayes factor B ij = 1. (4) In the case of nested models, where M 1 is ...

The traditional approach to building Bayesian networks is to build the graphical structure using a graphical editor and then add probabilities using a separate spreadsheet for each node. This can make it difficult for a design team to get an impression of the total evidence provided by an assessment, especially if the Bayesian network is split into many fragments to make it more manageable. Using the design patterns commonly used to build Bayesian networks for educational assessments, the collection of networks necessary can be specified using two matrixes. An inverse covariance matrix among the proficiency variables (the variables which are the target of interest) specifies the graphical structure and relation strength of the proficiency model. A Q-matrix — an incidence matrix whose rows represent observable outcomes from assessment tasks and whose columns represent proficiency variables — provides the graphical structure of the evidence models (graph fragments linking proficiency variables to observable outcomes). The Q-matrix can be augmented to provide details of relationship strengths and provide a high level overview of the kind of evidence available in the assessment. The representation of the model using matrixes means that the bulk of the specification work can be done using a desktop spreadsheet program and does not require specialized software, facilitating collaboration with external experts. The design idea is illustrated with some examples from prior assessment design projects.

Elicitation is a key task for subjectivist Bayesians. While skeptics hold that it cannot (or perhaps should not) be done, in practice it brings statisticians closer to their clients and subjectmatter-expert colleagues. This paper reviews the state-of-the-art, reflecting both the experience of statisticians and the fruits of a long line of psychological research into how people represent uncertain information cognitively, and how they respond to questions about that information. In a discussion of the elicitation process, the first issue to address is what it means for an elicitation to be successful, i.e. what criteria should be employed? Our answer is that a successful elicitation faithfully represents the opinion of the person being elicited. It is not necessarily “true ” in some objectivistic sense, and cannot be judged that way. We see elicitation as simply part of the process of statistical modeling. Indeed in a hierarchical model it is ambiguous at which point the likelihood ends and the prior begins. Thus the same kinds of judgment that inform statistical modeling in general also inform elicitation of prior distributions.

We propose a benchmark prior for the estimation of vector autoregressions: a prior about initial growth rates of the modelled series. We first show that the Bayesian vs frequentist small sample bias controversy is driven by different default initial conditions. These initial conditions are usually arbitrary and our prior serves to replace them in an intuitive way. To implement this prior we develop a technique for translating priors about observables into priors about parameters. We find that our prior makes a big difference for the estimated persistence of output responses to monetary policy shocks in the United States.