This paper is concerned with establishing broadly-based system-theoretic foundations and practical techniques for the problem of system identification that are rigorous, intuitively clear and conceptually powerful. A general formulation is first given in which two order relations are postulated on a class of models: a constant one of complexity; and a variable one of approximation induced by an observed behaviour. An admissible model is such that any less complex model is a worse approximation. The general problem of identification is that of finding the admissible subspace of models induced by a given behaviour. It is proved under very general assumptions that, if deterministic models are required then nearly all behaviours require models of nearly maximum complexity. A general theory of approximation between models and behaviour is then developed based on subjective probability concepts and semantic information theory The role of structural constraints such as causality, locality, finite memory, etc., are then discussed as rules of the game. These concepts and results are applied to the specific problem or stochastic automaton, or grammar, inference. Computational results are given to demonstrate that the theory is complete and fully operational. Finally the formulation of identification proposed in this paper is analysed in terms of Klir’s epistemological hierarchy and both are discussed in terms of the rich philosophical literature on the acquisition of knowledge. 1

vi Abbreviations vii Notation viii 1 Introduction 1 2 Hidden Markov Models 4 2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 HMM Modelling Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 HMM Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.4 Finding the Best Transcription . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.5 Setting the Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3 Objective Functions 19 3.1 Properties of Maximum Likelihood Estimators . . . . . . . . . . . . . . . . . . . 19 3.2 Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.3 Maximum Mutual Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.4 Frame Discrimination . . . . . . . . . . . . . . . . ....

. We present a new way to understand and characterize the choice of scoring, objective, or loss function for a model, machine, or expert estimating (i.e. predicting) the probabilities of events or outcomes (such as class membership). The ultimate value of a probability estimate lies in the actual payoff (utility) accruing to those who use this information to make a decision. We allow that we often cannot specify with certainty that the estimate will be used in a particular decision problem, characterized by a particular decision-outcome payoff matrix (cost schedule), and thus by a particular decision threshold. Instead, we consider the more general case of a distribution over such matrices. The proposed scoring function is the expectation, with respect to this distribution, of the payoff that will actually be received. Square-error loss (or Brier score) and log-likelihood (or various entropic measures) arise from specific examples of such distributions, and even common single-threshold...

If asking subjects their beliefs during repeated game play changes the way those subjects play, using those stated beliefs to evaluate and compare theories of strategic behavior is problematic. We experimentally verify that belief elicitation can alter paths of play in a repeated asymmetric matching pennies game. In this setting, belief elicitation improves the goodness of fit of structural models of belief learning, and the prior beliefs implied by such structural models are both stronger and more realistic when beliefs are elicited than when they are not. These effects are, however, confined to the player type who sees a strong asymmetry between payoff possibilities for her two strategies in the game. We conjecture that this occurs because both automatic, unconscious evaluative processes and more effortful conscious deliberative processes can conflict, and that the latter are enhanced by belief elicitation procedures. We also find that “empirical beliefs ” (beliefs estimated from past observed actions of opponents) can be better predictors of observed actions than the “stated beliefs ” resulting from belief elicitation.

. We present a new way to understand and characterize the choice of scoring rule (probability loss function) for evaluating the performance of a supplier of probabilistic predictions after the outcomes (true classes) are known. The ultimate value of a prediction (estimate) lies in the actual utility (loss reduction) accruing to one who uses this information to make some decision(s). Often we cannot specify with certainty that the prediction will be used in a particular decision problem, characterized by a particular loss matrix (indexed by outcome and decision), and thus having a particular decision threshold. Instead, we consider the more general case of a distribution over such matrices. The proposed scoring rule is the expectation, with respect to this distribution, of the loss that is actually incurred when following the decision recommendation, the latter being the decision that would be considered optimal if we were to assume the predicted probabilities. Logarithmic and quadrati...

Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the predictive distribution and on the event or value that materializes. A scoring rule is proper if it encourages truthful reporting. It is local of order λ if the score depends on the predictive density only through its value and its derivatives of order up to λ at the observation. Previously, only a single local proper scoring rule had been known, namely the logarithmic score, which is local of order λ = 0. Here we introduce the Fisher score, which is a local proper scoring rule of order λ = 2. It relates to the Fisher information in the same way that the logarithmic score relates to the Kullback-Leibler information. The convex cone generated by the logarithmic score and the Fisher score exhausts the class of the local proper scoring rules of order λ ≤ 2, up to equivalence and regularity conditions. In a data example, we use local and non-local proper scoring rules to assess statistically postprocessed ensemble weather forecasts. Finally, we develop a multivariate version of the Fisher score. 1

This paper asks a simple methodological question about belief elicitation: Does asking subjects about their beliefs during a repeated game change the way subjects play the game? In particular, we ask if belief elicitation changes behavior in the direction of belief based learning models, away from reinforcement learning models. Based on a carefully designed laboratory experiment, we show that play is indeed affected by eliciting players beliefs. More importantly, however, the effect is not simply a parametric shift in favor of one or another theory. What we instead observe is that subjects behave as if they construct mental models of their opponents that are both more sophisticated and robust in the presence of belief elicitation. Finally we show that game theoretically motivated learning models fit the behavioral data much better when the action data is generated jointly with belief elicitation than when it is generated without. This supports the conjecture that eliciting beliefs make subjects behave in a more game theoretic manner. Our findings have serious implications both for the choice of econometric models when testing game theoretic and learning models, and for the ability to test these models based on stated, rather than empirical beliefs