Results 1  10
of
25
A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods.
, 2008
"... Abstract We review current methods for evaluating models in the cognitive sciences, including theoreticallybased approaches, such as Bayes Factors and MDL measures, simulation approaches, including model mimicry evaluations, and practical approaches, such as validation and generalization measures. ..."
Abstract

Cited by 55 (18 self)
 Add to MetaCart
Abstract We review current methods for evaluating models in the cognitive sciences, including theoreticallybased approaches, such as Bayes Factors and MDL measures, simulation approaches, including model mimicry evaluations, and practical approaches, such as validation and generalization measures. We argue that, while often useful in specific settings, most of these approaches are limited in their ability to give a general assessment of models. We argue that hierarchical methods generally, and hierarchical Bayesian methods specifically, can provide a more thorough evaluation of models in the cognitive sciences. We present two worked examples of hierarchical Bayesian analyses, to demonstrate how the approach addresses key questions of descriptive adequacy, parameter interference, prediction, and generalization in principled and coherent ways.
Model Selection by Normalized Maximum Likelihood
, 2005
"... The Minimum Description Length (MDL) principle is an information theoretic approach to inductive inference that originated in algorithmic coding theory. In this approach, data are viewed as codes to be compressed by the model. From this perspective, models are compared on their ability to compress a ..."
Abstract

Cited by 26 (9 self)
 Add to MetaCart
The Minimum Description Length (MDL) principle is an information theoretic approach to inductive inference that originated in algorithmic coding theory. In this approach, data are viewed as codes to be compressed by the model. From this perspective, models are compared on their ability to compress a data set by extracting useful information in the data apart from random noise. The goal of model selection is to identify the model, from a set of candidate models, that permits the shortest description length (code) of the data. Since Rissanen originally formalized the problem using the crude ‘twopart code ’ MDL method in the 1970s, many significant strides have been made, especially in the 1990s, with the culmination of the development of the refined ‘universal code’ MDL method, dubbed Normalized Maximum Likelihood (NML). It represents an elegant solution to the model selection problem. The present paper provides a tutorial review on these latest developments with a special focus on NML. An application example of NML in cognitive modeling is also provided.
An empirical study of minimum description length model selection with infinite parametric complexity
 JOURNAL OF MATHEMATICAL PSYCHOLOGY
, 2006
"... Parametric complexity is a central concept in Minimum Description Length (MDL) model selection. In practice it often turns out to be infinite, even for quite simple models such as the Poisson and Geometric families. In such cases, MDL model selection as based on NML and Bayesian inference based on J ..."
Abstract

Cited by 12 (1 self)
 Add to MetaCart
Parametric complexity is a central concept in Minimum Description Length (MDL) model selection. In practice it often turns out to be infinite, even for quite simple models such as the Poisson and Geometric families. In such cases, MDL model selection as based on NML and Bayesian inference based on Jeffreys ’ prior can not be used. Several ways to resolve this problem have been proposed. We conduct experiments to compare and evaluate their behaviour on small sample sizes. We find interestingly poor behaviour for the plugin predictive code; a restricted NML model performs quite well but it is questionable if the results validate its theoretical motivation. A Bayesian marginal distribution with Jeffreys’ prior can still be used if one sacrifices the first observation to make a proper posterior; this approach turns out to be most dependable.
1 Evaluating the reliance on past choices in adaptive learning models
"... This article may not exactly replicate the final version published in Journal of Mathematical Psychology. It is not the copy of record. 23 Adaptive learning models are used to predict behavior in repeated choice tasks. Predictions can be based on previous payoffs or previous choices of the player. T ..."
Abstract

Cited by 11 (4 self)
 Add to MetaCart
(Show Context)
This article may not exactly replicate the final version published in Journal of Mathematical Psychology. It is not the copy of record. 23 Adaptive learning models are used to predict behavior in repeated choice tasks. Predictions can be based on previous payoffs or previous choices of the player. The current paper proposes a new method for evaluating the degree of reliance on past choices, called Equal Payoff Series Extraction (EPSE). Under this method a simulated player has the same exact choices as the player but receives equal constant payoffs from all of the alternatives. Success in predicting the next choice ahead for this simulated player therefore relies strictly on mimicry of previous choices of the actual player. This allows determining the marginal fit of predictions that are not based on the actual task payoffs. To evaluate the reliance on past choices under different models, an experiment was conducted in which 48 participants completed a threealternative choice task in four task conditions. Two different learning rules were evaluated: An interference rule, and a decay rule. The results showed that while the predictions of the decay rule relied more on
A signal detection analysis of fastandfrugal trees
 Psychological Review
, 2011
"... Models of decision making are distinguished by those that aim for an optimal solution in a world that is precisely specified by a set of assumptions (a socalled “small world”) and those that aim for a simple but satisfactory solution in an uncertain world where the assumptions of optimization model ..."
Abstract

Cited by 8 (3 self)
 Add to MetaCart
(Show Context)
Models of decision making are distinguished by those that aim for an optimal solution in a world that is precisely specified by a set of assumptions (a socalled “small world”) and those that aim for a simple but satisfactory solution in an uncertain world where the assumptions of optimization models may not be met (a socalled “large world”). Few connections have been drawn between these 2 families of models. In this study, the authors show how psychological concepts originating in the classic signaldetection theory (SDT), a smallworld approach to decision making, can be used to understand the workings of a class of simple models known as fastandfrugal trees (FFTs). Results indicate that (a) the setting of the subjective decision criterion in SDT corresponds directly to the choice of exit structure in an FFT; (b) the sensitivity of an FFT (measured in d�) is reflected by the order of cues searched and the properties of cues in an FFT, including the mean and variance of cues ’ individual d�s, the intercue correlation, and the number of cues; and (c) compared with the ideal and the optimal sequential sampling models in SDT and a majority model with an information search component, FFTs are extremely frugal (i.e., do not search for much cue information), highly robust, and well adapted to the payoff structure of a task. These findings demonstrate the potential of theory integration in understanding the common underlying psychological structures of apparently disparate theories of cognition.
A Bayesian Analysis of Dynamics in Free Recall
"... We develop a probabilistic model of human memory performance in free recall experiments. In these experiments, a subject first studies a list of words and then tries to recall them. To model these data, we draw on both previous psychological research and statistical topic models of text documents. W ..."
Abstract

Cited by 2 (1 self)
 Add to MetaCart
(Show Context)
We develop a probabilistic model of human memory performance in free recall experiments. In these experiments, a subject first studies a list of words and then tries to recall them. To model these data, we draw on both previous psychological research and statistical topic models of text documents. We assume that memories are formed by assimilating the semantic meaning of studied words (represented as a distribution over topics) into a slowly changing latent context (represented in the same space). During recall, this context is reinstated and used as a cue for retrieving studied words. By conceptualizing memory retrieval as a dynamic latent variable model, we are able to use Bayesian inference to represent uncertainty and reason about the cognitive processes underlying memory. We present a particle filter algorithm for performing approximate posterior inference, and evaluate our model on the prediction of recalled words in experimental data. By specifying the model hierarchically, we are also able to capture intersubject variability. 1
On the Minimum Description Length Complexity of Multinomial Processing Tree Models
"... Multinomial processing tree (MPT) modeling is a statistical methodology that has been widely and successfully applied for measuring hypothesized latent cognitive processes in selected experimental paradigms. This paper concerns model complexity of MPT models. Complexity is a key and necessary conce ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
Multinomial processing tree (MPT) modeling is a statistical methodology that has been widely and successfully applied for measuring hypothesized latent cognitive processes in selected experimental paradigms. This paper concerns model complexity of MPT models. Complexity is a key and necessary concept to consider in the evaluation and selection of quantitative models. A complex model with many parameters often overfits data beyond and above the underlying regularities, and therefore, should be appropriately penalized. It has been well established and demonstrated in multiple studies that in addition to the number of parameters, a model’s functional form, which refers to the way by which parameters are combined in the model equation, can also have significantly effects on complexity. Given that MPT models vary greatly in their functional forms (tree structures and parameter/category assignments), it would be of interest to evaluate their effects on complexity. Addressing this issue from the minimum description length (MDL) viewpoint, we prove a series of propositions concerning various ways in which functional form contributes to the complexity of MPT models. Computational issues of complexity are also discussed. COMPLEXITY OF MULTINOMIAL PROCESSING TREE MODELS 2
Bayesian Versus Frequentist Inference 1 Goals and Outline
"... Throughout this book, the topic of orderrestricted inference is dealt with almost exclusively from a Bayesian perspective. Some readers may wonder why the other main school for statistical inference frequentist inferencehas received so little attention here. Isn't it true that in the field o ..."
Abstract
 Add to MetaCart
(Show Context)
Throughout this book, the topic of orderrestricted inference is dealt with almost exclusively from a Bayesian perspective. Some readers may wonder why the other main school for statistical inference frequentist inferencehas received so little attention here. Isn't it true that in the field of psychology, almost all inference is frequentist inference? The first goal of this chapter is to highlight why frequentist inference is a lessthanideal method for statistical inference. The most fundamental limitation of standard frequentist inference is that it does not condition on the observed data. The resulting paradoxes have sparked a philosophical debate that statistical practitioners have conveniently ignored. What cannot be so easily ignored are the practical limitations of frequentist inference, such as its restriction to nested model comparisons. The second goal of this chapter is to highlight the theoretical and practical advantages of a Bayesian analysis. From a theoretical perspective, Bayesian inference is principled and prescriptive, and in contrast to frequentist inference a method that does condition on the observed data. From a practical perspective, Bayesian inference is becoming more and more attractive, mainly because of recent advances in computational methodology (e.g., Markov chain Monte Carlo and the WinBUGS program
Correspondence concerning this article should be addressed to:
"... The purpose of the recently proposed prep statistic is to estimate the probability of concurrence, that is, the probability that a replicate experiment yields an effect of the same sign (Killeen, 2005a). The influential journal Psychological Science endorses prep and recommends its use over that of ..."
Abstract
 Add to MetaCart
(Show Context)
The purpose of the recently proposed prep statistic is to estimate the probability of concurrence, that is, the probability that a replicate experiment yields an effect of the same sign (Killeen, 2005a). The influential journal Psychological Science endorses prep and recommends its use over that of traditional methods. Here we show that prep overestimates the probability of concurrence. This is because prep was derived under the assumption that all effect sizes in the population are equally likely a priori. In many situations, however, it is advisable to also entertain a null hypothesis of no or approximately no effect. We show how the posterior probability of the null hypothesis is sensitive to a priori considerations and to the evidence provided by the data; and the higher the posterior probability of the null hypothesis, the smaller the probability of concurrence. When the null hypothesis and the alternative hypothesis are equally likely a priori, prep may overestimate the probability of concurrence by 30 % and more. We conclude that prep provides an upper bound on the probability of concurrence, a bound that brings with it the danger of having researchers believe that their experimental effects are much more reliable than they actually are.