## Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/bdm.683 A Choice Prediction Competition: Choices from Experience and from Description (2010)

### BibTeX

@MISC{Erev10publishedonline,

author = {Ido Erev and Alvin E. Roth and Ernan Haruvy and Stefan M. Herzog and Robin Hau and Ralph Hertwig and Terrence Stewart and Robert West and Christian Lebiere},

title = {Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/bdm.683 A Choice Prediction Competition: Choices from Experience and from Description},

year = {2010}

}

### OpenURL

### Abstract

Erev, Ert, and Roth organized three choice prediction competitions focused on three related choice tasks: One shot decisions from description (decisions under risk), one shot decisions from experience, and repeated decisions from experience. Each competition was based on two experimental datasets: An estimation dataset, and a competition dataset. The studies that generated the two datasets used the same methods and subject pool, and examined decision problems randomly selected from the same distribution. After collecting the experimental data to be used for estimation, the organizers posted them on the Web, together with their fit with several baseline models, and challenged other researchers to compete to predict the results of the second (competition) set of experimental sessions. Fourteen teams responded to the challenge: The last seven authors of this paper are members of the winning teams. The results highlight the robustness of the difference between decisions from description and decisions from experience. The best predictions of decisions from descriptions were obtained with a stochastic variant of prospect theory assuming that the sensitivity to the

### Citations

3104 | Prospect theory: An analysis of decision under risk
- Kahneman, Tversky
- 1979
(Show Context)
Citation Context ... hence situations in which all outcomes and their probabilities are precisely described, so that there is no room for ambiguity about subjects’ beliefs. Alternative theories, such as prospect theory (=-=Kahneman & Tversky, 1979-=-), have been formulated to explain and generalize the deviations from utility theory observed in this way. The focus on counterexamples and their explanations has many attractive features. It has led ... |

871 | Advances in Prospect Theory: Cumulative Representation of Uncertainty
- Tversky, Kahneman
- 1992
(Show Context)
Citation Context ...r of the strongest baseline. Baseline models for condition description (one-shot decisions under risk) Original (5-parameter) cumulative prospect theory (CPT) According to cumulative prospect theory (=-=Tversky & Kahneman, 1992-=-), decision-makers are assumed to select the prospect with the highest weighted value. The weighted value of Prospect X that pays x1 with Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral... |

336 |
Le Comportement de l’Homme Rationnel Devant le Risque: Critique des Postulats et Axiomes de l’École Américaine
- Allais
- 1953
(Show Context)
Citation Context ..., one of the most important contributions of prospect theory (Kahneman & Tversky, 1979) is the demonstration that two of the best-known counterexamples to expected utility theory, the Allais paradox (=-=Allais, 1953-=-) and the observation that people not only buy lotteries but also insurance (Friedman & Savage, 1948), can be a product of a tendency to overweight rare events. While this tendency is robust, it is no... |

260 |
Verification of forecasts expressed in terms of probability. Monthly Weather Rev
- Brier
- 1950
(Show Context)
Citation Context ...asure has several attractive features. Two of these features are well known: The MSD score underlies traditional statistical methods (like regression and the t-test) and is a proper scoring rule (see =-=Brier, 1950-=-; Selten, 1998; and a discussion of the conditions under which the properness is likely to be important in Yates, Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15... |

229 |
The utility analysis of choices involving risks
- Friedman, Savage
- 1948
(Show Context)
Citation Context ...s the demonstration that two of the best-known counterexamples to expected utility theory, the Allais paradox (Allais, 1953) and the observation that people not only buy lotteries but also insurance (=-=Friedman & Savage, 1948-=-), can be a product of a tendency to overweight rare events. While this tendency is robust, it is not general. The recent studies of decisions from experience demonstrate that in many settings people ... |

203 |
Combining forecasts: A review and annotated bibliography
- Clemen
- 1989
(Show Context)
Citation Context ...erform the latter when the bracketing rate is larger than zero, and therefore, some errors will cancel each other out. 9 Equal weighting is robust and can outperform more elaborate weighting schemes (=-=Clemen, 1989-=-; Einhorn & Hogarth, 1975; Timmermann, 2006. Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI: 10.1002/bdm34 Journal of Behavioral Decision Making ... |

190 |
Reflections of the environment in memory
- Anderson, Schooler
- 1991
(Show Context)
Citation Context ...s) is a random value chosen from a logistic distribution with variance p 2 s 2 /3. Ai ln Xn k1 t d k þ "ðsÞ (9) The learning term of the equation captures the power law of practice and forgetting (=-=Anderson & Schooler, 1991-=-), while the random term implements a stochastic ‘‘softmax’’ (a.k.a. Boltzmann) retrieval process where the probability P i of retrieving i is given by: Pi eAi= t P e Aj=t j where t ffiffi p 2s an... |

110 |
On the psychophysical law
- Stevens
- 1957
(Show Context)
Citation Context ... to the diminishing sensitivity assumption the subjective impact of a change in the absolute payoff decreases with the distance from zero (see Tversky & Kahneman, 1992, and motivating observations in =-=Stevens, 1957-=-). The parameter l > 1 captures the loss aversion assertion suggesting that losses loom larger than equivalent gains. The subjective weights are assumed to depend on the outcomes’ rank and sign, and o... |

85 |
Decisions from experience and the effect of rare events” (unpublished manuscript
- Hertwig, Barron, et al.
- 2002
(Show Context)
Citation Context ...that in many settings people exhibit the opposite bias: They behave as if they underweight rare events (see Barron & Erev, 2003; Erev, Glozman, & Hertwig, 2008; Hau, Pleskac, Kiefer, & Hertwig, 2008; =-=Hertwig, Barron, Weber, & Erev, 2004-=-; Hertwig & Erev, 2009; Rakow, Demes, & Newell, 2008; Ungemach, Chater, & Stewart, 2009; Weber, Shafir, & Blais, 2004). A focus on quantitative predictions may help identify the boundaries of the diff... |

63 | Small feedback-based decisions and their limited correspondence to description-based decisions
- Barron, Erev
- 2003
(Show Context)
Citation Context ...y is robust, it is not general. The recent studies of decisions from experience demonstrate that in many settings people exhibit the opposite bias: They behave as if they underweight rare events (see =-=Barron & Erev, 2003-=-; Erev, Glozman, & Hertwig, 2008; Hau, Pleskac, Kiefer, & Hertwig, 2008; Hertwig, Barron, Weber, & Erev, 2004; Hertwig & Erev, 2009; Rakow, Demes, & Newell, 2008; Ungemach, Chater, & Stewart, 2009; We... |

61 | On adaptation, maximization, and reinforcement learning among cognitive strategies
- Erev, Barron
- 2005
(Show Context)
Citation Context ...alez, and Martin (2007). Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI: 10.1002/bdmI. Erev et al. A Choice Prediction Competition 31 model (see =-=Erev & Barron, 2005-=-; and a similar model in Erev, Bereby-Meyer, & Roth, 1999), over basic reinforcement-learning models. In addition, it shows that it is not easy to find a reinforcement-learning model that outperforms ... |

57 |
Simultaneous over- and underconfidence: The role of error in judgment processes
- Erev, Wallsten, et al.
- 1994
(Show Context)
Citation Context ...5 ( 3.3, 0.97; 10.5) or ( 3.2), the observed R-rate was 0.1, and the mean intuitive prediction was 0.34. This conservatism of the mean judgments can be a product of a stochastic judgment process (see =-=Erev, Wallsten, & Budescu, 1994-=-). Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI: 10.1002/bdmI. Erev et al. A Choice Prediction Competition 33 Condition E-sampling Table 3b pre... |

52 | Forecast Combinations
- Timmermann
- 2006
(Show Context)
Citation Context ... low. Third, research on forecast combination has demonstrated that averaging predictions from different models is a powerful tool for boosting accuracy (e.g. Armstrong, 2001; Hibon & Evgeniou, 2005; =-=Timmermann, 2006-=-). To the extent that individual models predict decisions well above chance, and errors are uncorrelated between models 8 , the average across models may even outperform the best individual model. The... |

50 |
Predicting Risk Sensitivity in Humans and Lower Animals: Risk as Variance or Coefficient of Variation
- Weber, Shafir, et al.
- 2004
(Show Context)
Citation Context ...03; Erev, Glozman, & Hertwig, 2008; Hau, Pleskac, Kiefer, & Hertwig, 2008; Hertwig, Barron, Weber, & Erev, 2004; Hertwig & Erev, 2009; Rakow, Demes, & Newell, 2008; Ungemach, Chater, & Stewart, 2009; =-=Weber, Shafir, & Blais, 2004-=-). A focus on quantitative predictions may help identify the boundaries of the different tendencies. 1 One of the earlier examples is the Pythagorean theorem. Archeological evidence suggests that the ... |

49 |
Axiomatic characterization of the quadratic scoring rule. Experiment. Econom
- Selten
- 1998
(Show Context)
Citation Context ...eral attractive features. Two of these features are well known: The MSD score underlies traditional statistical methods (like regression and the t-test) and is a proper scoring rule (see Brier, 1950; =-=Selten, 1998-=-; and a discussion of the conditions under which the properness is likely to be important in Yates, Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI... |

45 | The Priority Heuristic: Making Choices Without Trade-Offs
- Brandstätter, Gigerenzer, et al.
- 2006
(Show Context)
Citation Context ... this parameter. Other baseline models for condition description The other baseline models considered by EER for condition description include restricted variants of SCPT, and the priority heuristic (=-=Brandstätter, Gigerenzer, & Hertwig, 2006-=-). The analysis of the restricted variants of SCPT highlights the robustness of this model: It provides useful predictions even when it is used with the parameters estimated in previous research. The ... |

43 |
The effect of adding a constant to all payoffs: Experimental Investigation, and a Reinforcement Learning Model with Self-Adjusting Speed of Learning
- Erev, Bereby-Meyer, et al.
- 1999
(Show Context)
Citation Context ...hn Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI: 10.1002/bdmI. Erev et al. A Choice Prediction Competition 31 model (see Erev & Barron, 2005; and a similar model in =-=Erev, Bereby-Meyer, & Roth, 1999-=-), over basic reinforcement-learning models. In addition, it shows that it is not easy to find a reinforcement-learning model that outperforms the explorative sampler model with recency. THE COMPETITI... |

36 |
Unit weighting schemes for decision making
- EINHORN, HOGARTH
- 1975
(Show Context)
Citation Context ...ter when the bracketing rate is larger than zero, and therefore, some errors will cancel each other out. 9 Equal weighting is robust and can outperform more elaborate weighting schemes (Clemen, 1989; =-=Einhorn & Hogarth, 1975-=-; Timmermann, 2006. Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI: 10.1002/bdm34 Journal of Behavioral Decision Making subjective difference, no... |

33 | 2001e), “Combining forecasts
- Armstrong
(Show Context)
Citation Context ...between the models’ errors are relatively low. Third, research on forecast combination has demonstrated that averaging predictions from different models is a powerful tool for boosting accuracy (e.g. =-=Armstrong, 2001-=-; Hibon & Evgeniou, 2005; Timmermann, 2006). To the extent that individual models predict decisions well above chance, and errors are uncorrelated between models 8 , the average across models may even... |

30 |
Adaptation as information restriction: The hot stove effect
- Denrell, March
- 2001
(Show Context)
Citation Context ...t was developed to address this bias. Specifically, the refined model assumes that the most recent outcome with each alternative is always considered. This assumption triggers a hot stove effect (see =-=Denrell & March, 2001-=-): When the recent payoffs are considered, the effect of low outcomes last longer than the effect of high outcomes (because low outcomes reduce the probability of additional exploration and they remai... |

27 |
Model comparisons and model selections based on the generalization criterion methodology
- Busemeyer, Wang
- 2000
(Show Context)
Citation Context ...reads the payoff distributions of the relevant gambles as an input and predicts the proportion of risky choices as an output. Thus, the competitions used the generalization criterion methodology (see =-=Busemeyer & Wang, 2000-=-). 4 The problem selection algorithm Each study focused on 60 problems. The exact problems were determined with a random selection of the parameters (prizes and probabilities) L, M, H, andPh using the... |

26 | Studies of scientific discovery: Complementary approaches and convergent findings - Klahr, Simon - 1999 |

23 |
Intuitions about Combining Opinions: Misappreciation of the Averaging Principle,INSEAD working paper
- Larrick, Soll
- 2003
(Show Context)
Citation Context ... 1 0.38; p order 2 0.62). The difference between the values being compared is transformed into a 8 How strongly the errors of two models are correlated can be summarized by their bracketing rate (=-=Larrick & Soll, 2006-=-), which is the proportion of predictions where the two models err on different sides of the truth (i.e., one model over- and the other underestimates the true value). In the long run, the average pre... |

22 |
Decision making under uncertainty: A comparison of simple scalability, fixed sample, and sequential sampling models
- Busemeyer
- 1985
(Show Context)
Citation Context ...ription. It provided the best fit for the estimation data. This model is a stochastic variant of cumulative prospect theory proposed by Erev, Roth, Slonim, and Barron (2002; and see a similar idea in =-=Busemeyer, 1985-=-). The model assumes that the probability of selecting the risky prospect (R) over the safe prospect (S) increases with the relative advantage of that prospect. Specifically, this probability is: PrðR... |

18 |
To combine or not to combine: selecting among forecasts and their combinations
- Hibon, Evgeniou
- 2005
(Show Context)
Citation Context ...s’ errors are relatively low. Third, research on forecast combination has demonstrated that averaging predictions from different models is a powerful tool for boosting accuracy (e.g. Armstrong, 2001; =-=Hibon & Evgeniou, 2005-=-; Timmermann, 2006). To the extent that individual models predict decisions well above chance, and errors are uncorrelated between models 8 , the average across models may even outperform the best ind... |

17 |
Are probabilities overweighted or underweighted when rare outcomes are experienced (rarely
- Ungemach, Chater, et al.
- 2009
(Show Context)
Citation Context ... rare events (see Barron & Erev, 2003; Erev, Glozman, & Hertwig, 2008; Hau, Pleskac, Kiefer, & Hertwig, 2008; Hertwig, Barron, Weber, & Erev, 2004; Hertwig & Erev, 2009; Rakow, Demes, & Newell, 2008; =-=Ungemach, Chater, & Stewart, 2009-=-; Weber, Shafir, & Blais, 2004). A focus on quantitative predictions may help identify the boundaries of the different tendencies. 1 One of the earlier examples is the Pythagorean theorem. Archeologic... |

15 |
Instancebased learning in real-time dynamic decision making
- Gonzalez, Lerch, et al.
- 2003
(Show Context)
Citation Context ...d such that the mean recall value of each alternative at trial t is the weighted (by Pi) mean over all the recalled experiences. The alternative with the larger mean is selected (see related ideas in =-=Gonzalez et al., 2003-=-). Parameters. The value for parameter d in Equation 1 was set to 0.5, as this is the value used in almost all ACT-R models. The other two parameters were estimated based on the estimation set, using ... |

15 | The probabilistic nature of preferential choice - Rieskamp |

14 | Combining a theoretical prediction with experimental evidence to yield a new prediction: An experimental design with a random sample of tasks. Unpublished manuscript, Columbia University and Faculty of Industrial Engineering and Management - Erev, Roth, et al. - 2002 |

13 |
The description-experience gap in risky choice
- Hertwig, Erev
- 2009
(Show Context)
Citation Context ...he opposite bias: They behave as if they underweight rare events (see Barron & Erev, 2003; Erev, Glozman, & Hertwig, 2008; Hau, Pleskac, Kiefer, & Hertwig, 2008; Hertwig, Barron, Weber, & Erev, 2004; =-=Hertwig & Erev, 2009-=-; Rakow, Demes, & Newell, 2008; Ungemach, Chater, & Stewart, 2009; Weber, Shafir, & Blais, 2004). A focus on quantitative predictions may help identify the boundaries of the different tendencies. 1 On... |

12 |
The description-experience gap in risky choice: The role of sample size and experienced probabilities
- Hau, Plescak, et al.
- 2008
(Show Context)
Citation Context ... decisions from experience demonstrate that in many settings people exhibit the opposite bias: They behave as if they underweight rare events (see Barron & Erev, 2003; Erev, Glozman, & Hertwig, 2008; =-=Hau, Pleskac, Kiefer, & Hertwig, 2008-=-; Hertwig, Barron, Weber, & Erev, 2004; Hertwig & Erev, 2009; Rakow, Demes, & Newell, 2008; Ungemach, Chater, & Stewart, 2009; Weber, Shafir, & Blais, 2004). A focus on quantitative predictions may he... |

12 |
A dynamic ACT-R model of simple games
- Lebiere, West
- 1999
(Show Context)
Citation Context ...pendencies. Each experience is coded into a chunk that includes the context, choice, and obtained outcome. The context is abstracted here by the two previous consecutive choices (see related ideas by =-=Lebiere & West, 1999-=-; West et al., 2005). At each trial, the decision maker 10 The exact means of these subjective distributions depend of the sign of the payoff H and on the maximal absolute payoff (MaxAbs Max[Abs(L),... |

11 |
Biased samples not mode of presentation: Re-examining the apparent underweighting of rare events in experience-based choice. Organizational Behavior and Human Decision Processes
- Rakow, Demes, et al.
- 2008
(Show Context)
Citation Context ... behave as if they underweight rare events (see Barron & Erev, 2003; Erev, Glozman, & Hertwig, 2008; Hau, Pleskac, Kiefer, & Hertwig, 2008; Hertwig, Barron, Weber, & Erev, 2004; Hertwig & Erev, 2009; =-=Rakow, Demes, & Newell, 2008-=-; Ungemach, Chater, & Stewart, 2009; Weber, Shafir, & Blais, 2004). A focus on quantitative predictions may help identify the boundaries of the different tendencies. 1 One of the earlier examples is t... |

11 | Judgment and decision making. Englewood Cliffs, NJ: Prentice-Hall. Authors’ biographies: Peter McGraw is an Assistant Professor at the - Yates - 1990 |

9 | Instance-based decision making model of repeated binary choice - Lebiere, Gonzalez, et al. - 2007 |

7 |
Learning, risk attitude and hot stoves in restless bandit problems
- Biele, Erev, et al.
- 2009
(Show Context)
Citation Context ...rate with experience is clearer for high Ph and low Ph, but not for medium Ph level. This nonlinear relationship explains why previous studies that focus on gambles with equally likely outcomes (like =-=Biele, Erev, & Ert, 2009-=-) found no evidence for the hot stove effect. The learning curves in the medium Ph problem show higher sensitivity to the expected values. This pattern can be a product of the joint effect of underwei... |

7 |
Mathematical cuneiform texts
- Neugebauer, Sachs
- 1945
(Show Context)
Citation Context ... examples is the Pythagorean theorem. Archeological evidence suggests that the underlying regularity (the useful quantitative predictions) were known and used in Babylon 1300 years before Pythagoras (=-=Neugebauer & Sachs, 1945-=-). Pythagoras’ main contribution was the clarification of the theoretical explanation of this rule and its implications. Another important example is provided by Kepler’s laws. As suggested by Klahr a... |

4 |
Stochastic resonance in human cognition: ACT-R vs. game theory, associative neural networks, recursive neural networks, q-learning, and humans
- West, Stewart, et al.
- 2005
(Show Context)
Citation Context ...ience is coded into a chunk that includes the context, choice, and obtained outcome. The context is abstracted here by the two previous consecutive choices (see related ideas by Lebiere & West, 1999; =-=West et al., 2005-=-). At each trial, the decision maker 10 The exact means of these subjective distributions depend of the sign of the payoff H and on the maximal absolute payoff (MaxAbs Max[Abs(L),Abs(M),Abs(H)]). In... |

3 | Competitive Modeling Symposium: PokerBot World Series - Lebiere, Bothell |

2 | People’s participation in and attitudes to gaming, 1985–2000 : Final results of the 2000 survey - Amey - 2001 |

2 |
On the comparative concept of confirmation
- Carnap
- 1953
(Show Context)
Citation Context ...rediction with new data; in this case the ENO is interpreted as the weight of the model’s prediction in a regression that also includes the mean results of an experiment (see a related observation in =-=Carnap, 1953-=-). THE RESULTS OF THE ESTIMATION STUDY The right hand columns in Table 1a present the aggregate results of the estimation study. They show the mean choice proportions of the risky prospect (the R-rate... |

2 |
The Technion 1st prediction tournament. http://tx.technion.ac.il/ erev/Comp
- Erev, Ert, et al.
- 2008
(Show Context)
Citation Context ...and decision makers as described below. The estimation sessions were run in March 2008. After the completion of these experimental sessions EER posted the data (described in Table 1a) on the Web (see =-=Erev, Ert, & Roth, 2008-=-) and challenged researchers to participate in three competitions that focused on the prediction of the data of the second (competition) sessions. 3 The call to participate in the competition was publ... |

2 |
Context, mere presentation and the impact of rare events
- Erev, Glozman, et al.
- 2008
(Show Context)
Citation Context ...t general. The recent studies of decisions from experience demonstrate that in many settings people exhibit the opposite bias: They behave as if they underweight rare events (see Barron & Erev, 2003; =-=Erev, Glozman, & Hertwig, 2008-=-; Hau, Pleskac, Kiefer, & Hertwig, 2008; Hertwig, Barron, Weber, & Erev, 2004; Hertwig & Erev, 2009; Rakow, Demes, & Newell, 2008; Ungemach, Chater, & Stewart, 2009; Weber, Shafir, & Blais, 2004). A f... |

2 |
What have we learned from market design? Hahn lecture
- Roth
- 2008
(Show Context)
Citation Context ... criterion to rank models. Note that to the extent that competitions ameliorate counterincentives to conducting certain kinds of research, they can be viewed as a solution to a market design problem (=-=Roth, 2008-=-). Copyright # 2009 John Wiley & Sons, Ltd. Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI: 10.1002/bdm18 Journal of Behavioral Decision Making Safe: M with certainty Risky: H with proba... |

2 |
Equivalence: A Novel Basis for Model Comparison
- Stewart, West
- 2007
(Show Context)
Citation Context ...in Equation 1 was set to 0.5, as this is the value used in almost all ACT-R models. The other two parameters were estimated based on the estimation set, using the relativized equivalence methodology (=-=Stewart & West, 2007-=-). The estimated values are s 0.35 and t 1.6. It should be noted that these values are very close to the default settings for ACT-R, and there are only minor differences in predictions between thi... |

2 | Proceedings of the 27th annual conference of the cognitive science society (pp. 2074–2079). Mahwah, NJ: Lawrence Erlbaum Associates - Bucciarelli |

1 | Zealand: Department of Internal Affairs. Available at http://dia.govt.nz/Pubforms.nsf/URL/Titlecontentspt1.pdf - Wellington - 1998 |

1 | DOI: 10.1002/bdm Erev et al. A Choice Prediction Competition 47 Stefan M. Herzog is a Research Scientist of Cognitive and Decision Sciences in the Department of Psychology at the University of Basel, Switzerland. His research focuses on bounded rationalit - Wiley, Sons - 2010 |

1 | Learning and the economics of small decisions.In - Erev, Haruvy - 2009 |

1 | Journal of Behavioral Decision Making, 23, 15–47 (2010) DOI: 10.1002/bdm Journal of Behavioral Decision Making Hertwig - Wiley, Sons - 2008 |