## Elicitation (2004)

### BibTeX

@MISC{Garthwaite04elicitation,

author = {Paul H. Garthwaite and Joseph B. Kadane},

title = {Elicitation},

year = {2004}

}

### OpenURL

### Abstract

Elicitation is a key task for subjectivist Bayesians. While skeptics hold that it cannot (or perhaps should not) be done, in practice it brings statisticians closer to their clients and subjectmatter-expert colleagues. This paper reviews the state-of-the-art, reflecting both the experience of statisticians and the fruits of a long line of psychological research into how people represent uncertain information cognitively, and how they respond to questions about that information. In a discussion of the elicitation process, the first issue to address is what it means for an elicitation to be successful, i.e. what criteria should be employed? Our answer is that a successful elicitation faithfully represents the opinion of the person being elicited. It is not necessarily “true ” in some objectivistic sense, and cannot be judged that way. We see elicitation as simply part of the process of statistical modeling. Indeed in a hierarchical model it is ambiguous at which point the likelihood ends and the prior begins. Thus the same kinds of judgment that inform statistical modeling in general also inform elicitation of prior distributions.

### Citations

1289 | Judgment under uncertainty: heuristics and biases - Kahneman, Slovic, et al. - 1982 |

420 |
Availability: A Heuristic for Judging Frequency and Probability
- Tversky, Kahneman
- 1973
(Show Context)
Citation Context ...n by their third letter (e.g. park, bird, wire, . . . ). Hence, most people judge that “r” is more likely to be the first letter of a word, rather than the third letter, although the reverse is true (=-=Tversky and Kahneman, 1973-=-). Recall is also affected by factors such as familiarity, salience and recency, and newsworthy events also impact disproportionately on our memory, so you might overestimate the probability of a plan... |

370 |
Uncertainty: A Guide to dealing with uncertainty in quantitative risk and policy analysis
- Morgan, Henrion
- 1990
(Show Context)
Citation Context ...to mark a point on a line whose endpoints are 0 and 1 (for probabilities) or 0% to 100% for proportion. Probability wheels (Spetzler and Stael von Holstein, 1975) have proved particularly successful (=-=Morgan and Henrion, 1990-=-, p126). In its simplest form a probability wheel is a round pie-shaped disc of one colour that is partly covered by a ‘slice’ of a different colour, and a pointer. The size of the slice can be varied... |

351 |
Influence diagrams
- Howard, Matheson
- 1981
(Show Context)
Citation Context ...influence how people consider probabilities have also been explored, such as asking 19them to suggest scenarios that would lead to an unlikely event (Slovic and Fischhoff, 1977), influence diagrams (=-=Howard and Matheson, 1984-=-), getting subjects to think carefully about the substantive details of each judgement (Koriat et al., 1980), and disaggregating an implicit hypothesis into its constituent hypotheses (Johnson et al.,... |

244 |
Decision analysis: Introductory lectures on choices under uncertainty
- Raiffa
- 1968
(Show Context)
Citation Context ...ics have an effect on the way an expert views a problem and the assessments that are elicited. Visual aids to help people quantify their opinions have been tried, such as urns full of coloured balls (=-=Raiffa, 1968-=-), light pens on coloured screens (Barclay and Randall, 1975), or simply asking assessors to mark a point on a line whose endpoints are 0 and 1 (for probabilities) or 0% to 100% for proportion. Probab... |

188 |
The growth of logical thinking from childhood to adolescence
- Inhelder, Piaget
- 1958
(Show Context)
Citation Context ... available data, sometimes basing their judgements on just the proportion of time the positive outcome for one of the binary variables occurred with a positive outcome for the other (Smedsland, 1963; =-=Inhelder and Piaget, 1958-=-; Jenkins and Ward, 1965; Ward and Jenkins, 1965). Statistically, it is important to distinguish between eliciting an expert’s beliefs about a population correlation coefficient and eliciting the valu... |

184 | Combining probability distributions: A critique and an annotated bibliography - Genest, Zidek - 1986 |

153 | Kahneman D. Belief in the law of small numbers
- Tversky
- 1971
(Show Context)
Citation Context ...o tailed). You now have cause to run an additional group of ten subjects. What do you think the probability is that the results will be significant, by a one-tailed test, separately for this group?” (=-=Tversky and Kahneman, 1971-=-, p105) The median answer from the two groups was 0.85. However, if one assumes a non-informative prior distribution for the mean before the first sample was taken, then the true probability is only 0... |

149 | Support theory: a nonextensional representation of subjective probability - Tversky, Koehler - 1994 |

125 |
A progress report on the training of probability assessors
- Alper, Raiffa
- 1982
(Show Context)
Citation Context ... a 1 percent probability the true answer 18would exceed your estimate. (b) Make a low estimate such that you feel there is only a 1 percent probability the true answer would be below this estimate. (=-=Alpert and Raiffa, 1969-=-, pp.16-17) It should have been somewhat of a “surprise” to a subject to find the true value of a quantity falling outside an interval. 43% of all assessments produced such surprises. This information... |

101 | On narrow norms and vague heuristics: A reply to Kahneman and Tversky
- Gigerenzer
- 1996
(Show Context)
Citation Context ...complete it within five years? rather than as single (one-shot) events, like If a new PhD student is picked at random, what is the probability that he or she will complete the PhD within five years? (=-=Gigerenzer, 1996-=-; Koehler, 1996) As noted earlier, a good elicitation method should yield a probability distribution that accurately reflects the expert’s opinion, but this is hard to check and a pragmatic alternativ... |

96 | Comparison of Bayesian and Regression Approaches to the Study - Slovic, Lichtenstein - 1971 |

80 |
Reasons for confidence
- Koriat, Lichtenstein
- 1980
(Show Context)
Citation Context ...that would lead to an unlikely event (Slovic and Fischhoff, 1977), influence diagrams (Howard and Matheson, 1984), getting subjects to think carefully about the substantive details of each judgement (=-=Koriat et al., 1980-=-), and disaggregating an implicit hypothesis into its constituent hypotheses (Johnson et al., 1993). As an example of the effect of disaggregation, Fischhoff, Slovic and Lichtenstein (1978) questioned... |

74 |
Framing, probability distortions, and insurance decisions
- Johnson, Hershey, et al.
- 1993
(Show Context)
Citation Context ... Matheson, 1984), getting subjects to think carefully about the substantive details of each judgement (Koriat et al., 1980), and disaggregating an implicit hypothesis into its constituent hypotheses (=-=Johnson et al., 1993-=-). As an example of the effect of disaggregation, Fischhoff, Slovic and Lichtenstein (1978) questioned experts (car mechanics) about the probable reasons for a car not starting. The experts assessed t... |

61 | Predictive model selection
- Laud, Ibrahim
- 1995
(Show Context)
Citation Context ...pothetical datum and she then states her updated median of their absolute difference. I&L use assessments of 36the mean and variance of the precision (σ−2) to determine ω and δ and, in related work (=-=Laud and Ibrahim, 1995-=-), they use assessments of the median and the 95th percentile of the distribution of the precision. Oman does not elicit ω and δ, and restricts his posterior analysis to inferences that depend only on... |

61 | Kendals Advanced Theory of Statistics, Volume 2b, Bayesian Inference - OHagan - 1994 |

58 | Judgment of contingency between responses and outcomes - Jenkins, Ward - 1965 |

57 |
Use of nonlinear, noncompensatory models as a function of task and amount of information
- Einhorn
- 1971
(Show Context)
Citation Context ...used to forecast a second sample of predictions. The forecasts produced in this way were only slightly less accurate than those produced by a model actually based on the second sample of predictions (=-=Einhorn, 1971-=-; Slovic and Lichtenstein, 1968; Wiggins and Hoffman, 1968). Experiments also show that, provided cues are monotonically related to the predicted variable, a simple linear combination of main effects ... |

55 | Variants of uncertainty - Kahneman, Tversky - 1982 |

50 | Interactive elicitation of opinion for a normal linear model - Kadane, Dickey, et al. - 1980 |

48 | Group consensus probability distributions: a critical survey, Bayesian Stat - French - 1985 |

47 |
Measuring the vague meanings of probability terms
- Wallsten, Budescu, et al.
- 1986
(Show Context)
Citation Context ... significantly improved performance on many assessment tasks”. There has been research aimed at converting probabilistic phrases (such as “quite likely” and “extremely probable”) into numeric values (=-=Wallsten et al., 1986-=-; Mosteller and Yountz, 1990), and at differentiating situations where verbal expressions of probability are preferable to numeric ones, or vice-versa (Winschitl and Wells, 1996). People are generally... |

46 |
Relative importance of probabilities and payoffs in risk taking
- Slovic, Lichtenstein
(Show Context)
Citation Context ...t a second sample of predictions. The forecasts produced in this way were only slightly less accurate than those produced by a model actually based on the second sample of predictions (Einhorn, 1971; =-=Slovic and Lichtenstein, 1968-=-; Wiggins and Hoffman, 1968). Experiments also show that, provided cues are monotonically related to the predicted variable, a simple linear combination of main effects will do a remarkably good job a... |

43 | I knew it would happen: Remembered probabilities of once-future things,” Organizational Behavior and Human Performance - Fischhoff, Beyth - 1975 |

42 | A Probabilistic Analysis of the Sacco and Vanzetti Evidence - Kadane, Schum - 1996 |

40 | Fault trees: Sensitivity of estimated failure probabilities to problem representation - Fischhoff, Slovic, et al. - 1978 |

40 |
Man as an intuitive statistician
- Peterson, Beach
- 1967
(Show Context)
Citation Context ... to perform well; sample sizes, sequence lengths and prior probabilities have been varied. Some of these changes have influenced the degree of conservatism, but they have not eliminated it. (See e.g. =-=Peterson and Beach, 1967-=-, pp 32-33 for a review of such experiments.) In some experiments though, the basic experimental situation has been modified so as to make it more complex and, in these more complex situations, conser... |

37 | Quantifying prior opinion - Diaconis, Ylvisaker - 1985 |

36 | Separating probability elicitation from utilities - Kadane, Winkler - 1988 |

35 | An Introduction to Bayesian Inference and Decision - Winkler - 1972 |

34 | Scoring rules for continuous probability distributions - MATHESON, WINKLER - 1976 |

34 |
Ratio scales and category scales for a dozen perceptual continua
- Stevens, Galanter
- 1957
(Show Context)
Citation Context ... any form of prior distribution. Several experiments investigate subjects’ capability at judging sample proportions (Erlick, 1964; Nash, 1964; Pitz, 1965, 1966; Shuford, 1961; Simpson and Voss, 1961; =-=Stevens and Galanter, 1957-=-). In these experiments, binary data were displayed to subjects for a limited period of time and they were then asked to estimate one of the sample proportions. For example, Shuford (1961) projected 2... |

33 | Social Choice and Individual - Arrow - 1963 |

33 | Eliciting expert beliefs in substantial practical applications. The Statistician 47 - O‘Hagan - 1998 |

31 |
The assessment of prior distributions in Bayesian analysis
- Winkler
- 1967
(Show Context)
Citation Context ...e value of sample information. The quantile method tends to yield distributions that are again too tight, but slightly less tight than the PDF method and much less tight than the HFS and EPS methods (=-=Winkler, 1967-=-). On this basis the quantile method seems preferable, and it also seems the method of choice when judged by scoring rules. (Scoring rules are discussed in Section 4.3). Some experiments have examined... |

30 | Correlations and copulas for decision and risk analysis - Clemen, Reilly - 1999 |

29 | Decision analysis expert use - Morris - 1974 |

29 | Probability encoding in decision analysis - Spetzler, Holstein - 1975 |

28 | Reconciliation of discrete probability distributions - Lindley - 1985 |

27 | Marginalization and Linear Opinion Pools - McConway - 1981 |

24 | Cognitive processes and the assessment of subjective probability distributions - Hogarth - 1975 |

24 |
Empirical scaling of common verbal phrases associated with numerical probabilities
- Lichtenstein, JR
(Show Context)
Citation Context ...ically. Unfortunately, there is considerable variation in the probabilities different people attach to the same phrase, and the context also affects the probability a person associates with a phrase (=-=Lichtenstein and Newman, 1967-=-; Beyth-Marom, 201982; Wallsten et al., 1986). The response mode in which subjects are asked to give assessments also affects judgements. For example, Gigerenzer (1996) found that numeric expressions... |

23 | Encoding subjective probabilities: A psychological and psychometric review - Wallsten, Budescu - 1983 |

21 | How probable is probable? A numerical translation of verbal probability expressions - Beyth-Marom |

21 | Prior elicitation, variable selection, and Bayesian computation for logistic regression models - Chen, Ibrahim, et al. - 1999 |

21 | Approximating priors by mixtures of natural conjugate priors - Dalai, Hall - 1983 |

21 | The concept of correlation in adults - Smedslund - 1963 |

20 | Popular induction: Information is not necessarily informative - Nisbet, Borgida, et al. - 1976 |

19 | A predictive approach to the analysis of designed experiments - Ibrahim, Laud - 1994 |

17 | Procedures guide for structured expert judgment. Nuclear Science and Technology, European Commission - Cooke, Goossens - 2000 |