We consider the task of aggregating beliefs of several experts. We assume that these beliefs are represented as probability distributions. We argue that the evaluation of any aggregation technique depends on the semantic context of this task. We propose a framework, in which we assume that nature generates samples from a `true' distribution and different experts form their beliefs based on the subsets of the data they have a chance to observe. Naturally, the optimal aggregate distribution would be the one learned from the combined sample sets. Such a formulation leads to a natural way to measure the accuracy of the aggregation mechanism. We show that the well-known aggregation operator LinOP is ideally suited for that task. We propose a LinOP-based learning algorithm, inspired by the techniques developed for Bayesian learning, which aggregates the experts' distributions represented as Bayesian networks. We show experimentally that this algorithm performs well in practice. 1

In this paper, we examine the relative forecast accuracy of information markets versus expert aggregation. We leverage a unique data source of almost 2000 people’s subjective probability judgments on 2003 US National Football League games and compare with the “market probabilities ” given by two different information markets on exactly the same events. We combine assessments of multiple experts via linear and logarithmic aggregation functions to form pooled predictions. Prices in information markets are used to derive market predictions. Our results show that, at the same time point ahead of the game, information markets provide as accurate predictions as pooled expert assessments. In screening pooled expert predictions, we find that arithmetic average is a robust and efficient pooling function; weighting expert assessments according to their past performance does not improve accuracy of pooled predictions; and logarithmic aggregation functions offer bolder predictions than linear aggregation functions. The results provide insights into the predictive performance of information markets, and the relative merits of selecting among various opinion pooling methods.

We consider Bayesian inference when priors and likelihoods are both available for inputs and outputs of a deterministic simulation model. This problem is fundamentally related to the issue of aggregating (i.e. pooling) expert opinion. We survey alternative strategies for aggregation, then describe computational approaches for implementing pooled inference for simulation models. Our approach (1) numerically transforms all priors to the same space, (2) uses log pooling to combine priors, and (3) then draws standard Bayesian inference. We use importance sampling methods, including an iterative, adaptive approach which is more flexible and has less bias in some instances than a simpler alternative. Our exploratory examples are the first steps toward extension of the approach for highly complex and even noninvertible models. Key Words: Prior Coherization, Adaptive Importance Sampling, Bayesian Statistics, Model Inversion. 1 Introduction Much research of natural processes and systems is bas...

. Methods to aggregate convex sets of probabilities are proposed. Source reliability is taken into account by transforming the given information and making it less precise. An important property of the aggregation will be that the precision of the result will depend on the initial compatibility of sources. Special attention will be paid to the particular case of probability intervals giving adaptations of aggregation procedures. 1 Introduction The problem of aggregating probabilities for the same set of events assigned by different experts has recieved a great deal of attention in the literature. See Genest and Zidek [8] and French [7] for surveys of classical statistical methods. In general, it is assumed that there is a finite set of mutually exclusive and exhaustive hypotheses under consideration. It is also considered that each expert expresses his opinion by means of a probability distribution on the set of hypotheses. In general, the methods for aggregating probabilities calcula...

Expert panels are playing an increasingly important role in U.S. health policy decision making. A fundamental issue in these applications is how to synthesize the judgments of individual experts into a group judgment. In this paper we propose an approach to synthesis based on Bayesian hierarchical models, and apply it to the problem of determining physician staffing at medical centers operated by the U.S. Department of Veteran Affairs (VA). Our starting point is the so-called supra-Bayesian approach to synthesis, whose principal motivation in the present context is to generate an estimate of the uncertainty associated with a panel's evaluation of the number of physicians required under specified conditions. Hierarchical models are particularly natural in this context since variability in the experts' judgments results in part from heterogeneity in their baseline experiences at different VA medical centers. We derive alternative hierarchical Bayes synthesis distributions for the number ...

Consider a group of Bayesians, each with a subjective probability distribution over a set of uncertain events. An opinion pool derives a single consensus distribution over the events, representative of the group as a whole. Several pooling functions have been proposed, each sensible under particular assumptions or measures. Many researchers over many years have failed to form a consensus on which method is best. We propose a market-based pooling procedure, and analyze its properties. Participants bet on securities, each paying off contingent on an uncertain event, so as to maximize their own expected utilities. The consensus probability of each event is defined as the corresponding security's equilibrium price. The market framework provides explicit monetary incentives for participation and honesty, and allows agents to maintain individual rationality and limited privacy. "No arbitrage" arguments ensure that the equilibrium prices form legal probabilities. We show that, when events are...

Information consensus in sensor networks has received much attention due to its numerous applications in distributed decision making. This paper discusses the problem of a distributed group of agents coming to agreement on a probability vector over a network, such as would be required in a decentralized estimation of state transition probabilities or agreement on a probabilistic search map. Unique from other recent consensus literature, however, the agents in this problem must reach agreement while accounting for the uncertainties in their respective probabilities, which are formulated according to generally non-Gaussian distributions. The first part of this paper considers the problem in which the agents seek agreement to the centralized Bayesian estimate of the probabilities, which is accomplished using consensus on hyperparameters. The second part shows that the new hyperparameter consensus methodology can ensure convergence to the centralized estimate even while measurements of a static process are occurring concurrently with the consensus algorithm. A machine repair example is used to illustrate the advantages of hyperparameter consensus over conventional consensus approaches. I.

The problem of modelling expert knowledge about numerical parameters in the field of reliability is reconsidered in the framework of possibility theory. Usually expert opinions about quantities such as failure rates are modelled, assessed and pooled in the setting of probability theory. This approach does not seem to always be natural since probabilistic information looks too rich to be currently supplied by individuals. Indeed, information supplied by individuals is often incomplete, imprecise rather than tainted with randomness. Moreover the probabilistic framework looks somewhat restrictive to express the variety of possible pooling modes. In this paper, we formulate a model of expert opinion by means of possibility distributions that are thought to better reflect the imprecision pervading expert judgements. They are weak substitutes to unreachable subjective probabilities. Assessment evaluation is carried out in terms of calibration and level of precision, respectively measured by ...

The DAKOTA (Design Analysis Kit for Optimization and Terascale Applications) toolkit provides a flexible and extensible interface between simulation codes and iterative analysis methods. DAKOTA contains algorithms for optimization with gradient and nongradient-based methods; uncertainty quantification with sampling, analytic reliability, and stochastic finite element methods; parameter estimation with nonlinear least squares methods; and sensitivity analysis with design of experiments and parameter study methods. These capabilities may be used on their own or as components within advanced strategies such as surrogatebased optimization, mixed integer nonlinear programming, or optimization under uncertainty. By employing object-oriented design to implement abstractions of the key components required for iterative systems analyses, the DAKOTA toolkit provides a flexible and extensible problem-solving environment for design and performance analysis of computational models on high performance computers.

Sigatures are on file in the Graduate School. iii In almost all walks of life, predicting uncertain future events plays an essential role in decision-making processes. However, information related to future events frequently exists only as dispersed opinions, insights, and intuitions of individuals. Each individual only knows a little, but aggregating the dispersed information together may make considerable contribution to decision making. This is typical in many domains including business, politics, and entertainment. Therefore, how to aggregate such dispersed information for useful decision support is a crucial task. Markets have shown great potential as one of the most effective mechanisms for gathering distributed information and generating accurate forecasts, often surpassing many existing methods in practice. This research studies information markets, markets that are specially designed for information aggregation and forecasting, from four different perspectives: theoretical examination, experimental evaluation, empirical analysis, and design.