Results 1  10
of
16
Why we still use our heads instead of formulas: Toward an integrative approach
 Psychological Bulletin
, 1990
"... This review begins with a discussion of Meehl's (1957) query regarding when to use one's head (i.e., intuition) instead of the formula (i.e., statistical or mechanical procedure) for clinical prediction. It then describes the controversy that ensued and analyzes the complexity and contempo ..."
Abstract

Cited by 47 (0 self)
 Add to MetaCart
(Show Context)
This review begins with a discussion of Meehl's (1957) query regarding when to use one's head (i.e., intuition) instead of the formula (i.e., statistical or mechanical procedure) for clinical prediction. It then describes the controversy that ensued and analyzes the complexity and contemporary relevance of the question itself. Going beyond clinical inference, it identifies select cognitive biases and constraints that cause decision errors, and proposes remedial correctives. Given that the evidence shows cognition to be flawed, the article discusses the linear regression, Bayesian, signal detection, and computer approaches as possible decision aids. Their costbenefit tradeoffs, when used either alone or as complements to one another, are examined and evaluated. The critique concludes with a note of cautious optimism regarding the formula's future role as a decision aid and offers several interim solutions. More than three decades ago, an article by Meehl (1957) asked, "When shall we use our heads instead of the formula?" He replied that if people have a formula, then they should use their heads only very, very seldom. Heads in the article's title refers to the processing of data clinically, subjectively, or intu
Evaluating and combining subjective probability estimates
 Journal of Behavioral Decision Making
, 1997
"... This paper concerns the evaluation and combination of subjective probability estimates for categorical events. We argue that the appropriate criterion for evaluating individual and combined estimates depends on the type of uncertainty the decision maker seeks to represent, which in turn depends on h ..."
Abstract

Cited by 30 (5 self)
 Add to MetaCart
(Show Context)
This paper concerns the evaluation and combination of subjective probability estimates for categorical events. We argue that the appropriate criterion for evaluating individual and combined estimates depends on the type of uncertainty the decision maker seeks to represent, which in turn depends on his or her model of the event space. Decision makers require accurate estimates in the presence of aleatory uncertainty about exchangeable events, diagnostic estimates given epistemic uncertainty about unique events, and some combination of the two when the events are not necessarily unique, but the best equivalence class de®nition for exchangeable events is not apparent. Following a brief reveiw of the mathematical and empirical literature on combining judgments, we present an approach to the topic that derives from (1) a weak cognitive model of the individual that assumes subjective estimates are a function of underlying judgment perturbed by random error and (2) a classi®cation of judgment contexts in terms of the underlying information structure. In support of our developments, we present new analyses of two sets of subjective probability estimates, one of exchangeable and the other of unique events. As predicted, mean estimates were more accurate than the individual values in the ®rst case and more diagnostic in
Information markets vs. opinion pools: An empirical comparison
 In Proceedings of the Sixth ACM Conference on Electronic Commerce (EC’05
, 2005
"... In this paper, we examine the relative forecast accuracy of information markets versus expert aggregation. We leverage a unique data source of almost 2000 people’s subjective probability judgments on 2003 US National Football League games and compare with the “market probabilities ” given by two dif ..."
Abstract

Cited by 22 (9 self)
 Add to MetaCart
(Show Context)
In this paper, we examine the relative forecast accuracy of information markets versus expert aggregation. We leverage a unique data source of almost 2000 people’s subjective probability judgments on 2003 US National Football League games and compare with the “market probabilities ” given by two different information markets on exactly the same events. We combine assessments of multiple experts via linear and logarithmic aggregation functions to form pooled predictions. Prices in information markets are used to derive market predictions. Our results show that, at the same time point ahead of the game, information markets provide as accurate predictions as pooled expert assessments. In screening pooled expert predictions, we find that arithmetic average is a robust and efficient pooling function; weighting expert assessments according to their past performance does not improve accuracy of pooled predictions; and logarithmic aggregation functions offer bolder predictions than linear aggregation functions. The results provide insights into the predictive performance of information markets, and the relative merits of selecting among various opinion pooling methods.
Aggregating Learned Probabilistic Beliefs
, 2001
"... We consider the task of aggregating beliefs of several experts. We assume that these beliefs are represented as probability distributions. We argue that the evaluation of any aggregation technique depends on the semantic context of this task. We propose a framework, in which we assume that nature ge ..."
Abstract

Cited by 16 (0 self)
 Add to MetaCart
We consider the task of aggregating beliefs of several experts. We assume that these beliefs are represented as probability distributions. We argue that the evaluation of any aggregation technique depends on the semantic context of this task. We propose a framework, in which we assume that nature generates samples from a `true' distribution and different experts form their beliefs based on the subsets of the data they have a chance to observe. Naturally, the optimal aggregate distribution would be the one learned from the combined sample sets. Such a formulation leads to a natural way to measure the accuracy of the aggregation mechanism. We show that the wellknown aggregation operator LinOP is ideally suited for that task. We propose a LinOPbased learning algorithm, inspired by the techniques developed for Bayesian learning, which aggregates the experts' distributions represented as Bayesian networks. We show experimentally that this algorithm performs well in practice. 1
Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons. Forthcoming in Decision Analysis
, 2005
"... We develop a Bayesian multivariate analysis of expert judgment elicited using an extended form of pairwise comparisons. The method can be used to estimate the effect of multiple factors on the probability of an event and can be applied in risk analysis and other decision problems. The analysis prov ..."
Abstract

Cited by 6 (1 self)
 Add to MetaCart
We develop a Bayesian multivariate analysis of expert judgment elicited using an extended form of pairwise comparisons. The method can be used to estimate the effect of multiple factors on the probability of an event and can be applied in risk analysis and other decision problems. The analysis provides variance predictions of the quantity of interest that incorporate dependencies amongst the various experts. Unlike other combination methods for expert judgment, in this form we may learn about the dependencies between the experts from their responses. The analysis is applied to a data set of expert judgments elicited during the Washington State Ferries Risk Assessment. The effect of the statistical dependence amongst experts is compared to an analysis assuming independence amongst them.
A Market Framework for Pooling Opinions
, 1998
"... Consider a group of Bayesians, each with a subjective probability distribution over a set of uncertain events. An opinion pool derives a single consensus distribution over the events, representative of the group as a whole. Several pooling functions have been proposed, each sensible under particular ..."
Abstract

Cited by 4 (4 self)
 Add to MetaCart
Consider a group of Bayesians, each with a subjective probability distribution over a set of uncertain events. An opinion pool derives a single consensus distribution over the events, representative of the group as a whole. Several pooling functions have been proposed, each sensible under particular assumptions or measures. Many researchers over many years have failed to form a consensus on which method is best. We propose a marketbased pooling procedure, and analyze its properties. Participants bet on securities, each paying off contingent on an uncertain event, so as to maximize their own expected utilities. The consensus probability of each event is defined as the corresponding security's equilibrium price. The market framework provides explicit monetary incentives for participation and honesty, and allows agents to maintain individual rationality and limited privacy. "No arbitrage" arguments ensure that the equilibrium prices form legal probabilities. We show that, when events are...
Learning Performance of Prediction Markets with Kelly Bettors
, 2012
"... In evaluating prediction markets (and other crowdprediction mechanisms), investigators have repeatedly observed a socalled wisdom of crowds effect, which can be roughly summarized as follows: the average of participants performs much better than the average participant. The market price— an average ..."
Abstract

Cited by 3 (0 self)
 Add to MetaCart
(Show Context)
In evaluating prediction markets (and other crowdprediction mechanisms), investigators have repeatedly observed a socalled wisdom of crowds effect, which can be roughly summarized as follows: the average of participants performs much better than the average participant. The market price— an average or at least aggregate of traders ’ beliefs—offers a better estimate than most any individual trader’s opinion. In this paper, we ask a stronger question: how does the market price compare to the best trader’s belief, not just the average trader. We measure the market’s worstcase log regret, a notion common in machine learning theory. To arrive at a meaningful answer, we need to assume something about how traders behave. We suppose that every trader optimizes according to the Kelly criteria, a strategy that
Modeling Protein Secondary Structure by Products of Dependent Experts
 Master of Mathematics in Computer Science
, 2001
"... A phenomenon as complex as protein folding requires a complex model to approximate it. This thesis presents a bottomup approach for building complex probabilistic models of protein secondary structure by incorporating the multiple information sources which we call experts. Expert opinions are repre ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
(Show Context)
A phenomenon as complex as protein folding requires a complex model to approximate it. This thesis presents a bottomup approach for building complex probabilistic models of protein secondary structure by incorporating the multiple information sources which we call experts. Expert opinions are represented by probability distributions over the set of possible structures. Bayesian treatment of a group of experts results in a consensus opinion that combines the experts’ probability distributions using the operators of normalized product, quotient and exponentiation. The expression of this consensus opinion simplifies to a product of the expert opinions with two assumptions: (1) balanced training of experts, i.e., uniform prior probability over all structures, and (2) conditional independence between expert opinions, given the structure. This research also studies how Markov chains and hidden Markov models may be used to
Advances: Aggregate Probabilities Page 1 of 39 Ch 09 060430 V07 9 Aggregation of Expert Probability Judgments
"... probability distributions from experts in risk analysis, ” Risk Analysis, 19, 187203. This chapter is concerned with the aggregation of probability distributions in decision and risk analysis. Experts often provide valuable information regarding important uncertainties in decision and risk analyses ..."
Abstract
 Add to MetaCart
probability distributions from experts in risk analysis, ” Risk Analysis, 19, 187203. This chapter is concerned with the aggregation of probability distributions in decision and risk analysis. Experts often provide valuable information regarding important uncertainties in decision and risk analyses because of the limited availability of “hard data ” to use in those analyses. Multiple experts are often consulted in order to obtain as much information as possible, leading to the problem of how to combine or aggregate their information. Information may also be obtained from other sources such as forecasting techniques or scientific models. Because uncertainties are typically represented in terms of probability distributions, we consider expert and other information in terms of probability distributions. We discuss a variety of models that lead to specific combination methods. The output of these methods is a “combined probability distribution, ” which can be viewed as representing a summary of the current state of information regarding the uncertainty of interest. After presenting the models and methods, we discuss empirical evidence on the performance of the methods. In the conclusion we highlight important
INFORMATION TO USERS
"... This reproduction was made from a copy of a document sent to us for microfilming. While the most advanced technology has been used to photograph and reproduce this document, the quality of the reproduction is heavily dependent upon the quality of the material submitted. The following explanation of ..."
Abstract
 Add to MetaCart
This reproduction was made from a copy of a document sent to us for microfilming. While the most advanced technology has been used to photograph and reproduce this document, the quality of the reproduction is heavily dependent upon the quality of the material submitted. The following explanation of techniques is provided to help clarify markings or notations which may appear on this reproduction. 1.The sign or "target " for pages apparently lacking from the document photographed is "Missing Page(s)". If it was possible to obtain the missing page(s) or section, they are spliced into the film along with adjacent pages. This may have necessitated cutting through an image and duplicating adjacent pages to assure complete continuity. 2. When an image on the film is obliterated with a round black mark, it is an indication of either blurred copy because of movement during exposure, duplicate copy, or copyrighted materials that should not have been filmed. For blurred pages, a good image of the page can be found in the adjacent frame. If copyrighted materials were deleted, a target note will appear listing the pages in the adjacent frame. 3. When a map, drawing or chart, etc., is part of the material being photographed, a definite method of "sectioning " the material has been followed. It is customary to begin filming at the upper left hand corner of a large sheet and to continue from left to right in equal sections with small overlaps. If necessary, sectioning is continued again—beginning below the first row and continuing on until complete. 4. For illustrations that cannot be satisfactorily reproduced by xerographic means, photographic prints can be purchased at additional cost and inserted into your xerographic copy. These prints are available upon request from the Dissertations Customer Services Department. 5. Some pages in any document may have indistinct print. In all cases the best available copy has been filmed.