Computerized adaptive testing (CAT) based on item response theory (IRT) is viewed from the perspective of graphical modeling (GM). GM provides methods for making inferences about multifaceted skills and knowledge, and for extracting data from complex performances. However, simply incorporating variables for all sources of variation is rarely successful. Thus, researchers must closely analyze the substance and structure of the problem to create more effective models. Researchers regularly employ sophisticated strategies to handle many sources of variability outside the IRT model. Relevant variables can play many roles without appearing in the operational IRT model per se, e.g., in validity studies, assembling tests, and constructing and modeling

In this paper we discuss applications of Bayesian networks to educational testing.

Persistent elements and relationships underlie the design and delivery of educational assessments, despite their widely varying purposes, contexts, and data types. One starting point for analyzing these relationships is the assessment as experienced by the examinee: ‘What kinds of questions are on the test? ’ ‘Can I do them in any order? ’ ‘Which ones did I get wrong?’ ‘What’s my score? ’ These questions, asked by people of all ages and backgrounds, reveal an awareness that an assessment generally entails the selection and presentation of tasks, the scoring of responses, and the accumulation of these response evaluations into some kind of summary score. A four-process architecture is presented for the delivery of assessments: Activity Selection, Presentation, Response Processing, and Summary Scoring. The roles and the interactions among these processes, and how they arise from an assessment design model, are discussed. The ideas are illustrated with hypothetical examples. The complementary modular structures of the delivery processes and the design framework are seen to encourage coherence among assessment purpose, design, and delivery, as well as to promote efficiency through the reuse of design objects and delivery processes.

This paper studies the problem of learning diagnostic policies from training examples. A diagnostic policy is a complete description of the decision-making actions of a diagnostician (i.e., tests followed by a diagnostic decision) for all possible combinations of test results. An optimal diagnostic policy is one that minimizes the expected total cost, which isthe sum of measurement costs and misdiagnosis costs. In most diagnostic settings, there is a tradeo between these two kinds of costs. This paper formalizes diagnostic decision making as a Markov Decision Process (MDP). The paper introduces a new family of systematic search algorithms based on the AO algorithm to solve this MDP.To makeAO e cient, the paper describes an admissible heuristic that enables AO to prune large parts of the search space. The paper also introduces several greedy algorithms including some improvements over previously-published methods. The paper then addresses the question of learning diagnostic policies from examples. When the probabilities of diseases and test results are computed from training data, there is a great danger of over tting. To reduce over tting, regularizers are integrated into the search algorithms. Finally, the paper compares the proposed methods on ve benchmark diagnostic data sets. The studies show that in most cases the systematic search methods produce better diagnostic policies than the greedy methods. In addition, the studies show that for training sets of realistic size, the systematic search algorithms are practical on today's desktop computers. 1.

Medical knowledge about risks consists of a combination of structural information about known biological facts and probabilistic or actuarial information about exposures to hazards and recovery rates. While both types of information present significant practical challenges, probabilistic information is especially difficult to use because (1) it requires constant maintenance as new studies provide new data and (2) it usually comes in the form of study results which are not ideally suited for making individual predictions. The program GRAPHICAL-BELIEF is an environment for building and manipulating complex risk models. Graphical models can store and manipulate both structural and probabilistic knowledge and the model graph—in which nodes represent variables and edges represent relationships—is a natural visual metaphor for the more complex mathematical model. GRAPHICAL-BELIEF provides tools for both model manipulations and maintaining the knowledge bases on which the model is built. This paper introduces GRAPHICAL-BELIEF through an extended example: a model built from a study of patients with coronary artery disease. It shows how the model can provide valuable information about risk to the patient and value of information for medical tests.

In this paper we illustrate a simple scheme for dividing a complex Bayes network into a system model and a collection of smaller evidence models. While the system model maintains a permanent record of the state of the system of interest, the evidence models are only used momentarily to absorb evidence from specific observations or findings and then discarded. This paper describes an implementation of a system model–evidence model complex in which each system and evidence model has a separate Bayes net and Markov tree representation. As necessary, information is propagated between common Markov tree nodes of the evidence and system models. While mathematically equivalent to the full Bayes network, the system model–evidence model complex allows us to (a) separate the seldom used evidence model portions from the core system model thus reducing search and propagation time in the network and (b) easily replace the evidence models (this is particular advantageous in educational examples in which new test items are often introduced to prevent overexposure of assessment tasks). 1 System Models and Evidence Models Putting all possible observable variables of a large, complex model into a computational system is often impractical. One approach to such large problems is to decompose a Bayesian network into many smaller model fragments which would be assembled into the full model as needed. This approach is