Computer algebra systems are frequently used for research. In addition, some instruc-tors have based entire advanced courses around these systems. One benefit is that they allow students to become familiar with the methods of calculus by individual experimentation. How-ever, instructors have generally seen computer algebra systems as unsuitable for introductory algebra since the software is capable of doing all the math for the students. That barrier could be overcome if a suitable user interface were developed. Furthermore, embracing computer algebra could make for a completely different type of introductory course that shifts the focus from calculation to problem-solving. Computer algebra systems could also make life easier for math instructors as problem generators for tests and as automatic on-line graders, even of non-multiple choice problems. Computer algebra systems offer many new opportunities for math education. 1

and the National Science Foundation (PSLC) under contract no. SBE-0354420. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government or any other entity. Keywords: cognitive models, intelligent tutoring systems, machine learning, educational data mining, learning factors, psychometrics, additive factor models, latent variable models, exponential principal component analysis, logistic regression, combinatorial search ii To my parents and to my wife iii CONTENTS Generalized Learning Factors Analysis: Improving Cognitive Models with Machine Learning........................................................................................................................ i

ACT-R cognitive models are examples of complex hidden Markov processes with a large state space and a sparse transition matrix with absorbing states. While these models are designed to be theoretically interpretable as the thought processes generating observable data, mathematical model estimation and evaluation methods are based on simulation and prediction. There is little methodology applied to ACT-R that addresses questions of model complexity, identifiability and generalizability – all topics that can be addressed through study of the likelihood. In this paper I develop likelihood-based estimation using MCMC methods for a class of ACT-R models. I compare selection of ACT-R models for scatterplot generation using current predictive methods and using likelihood-based estimation and Bayes Factors. I propose extending this methodology to include a larger range of ACT-R models and to incorporate other model selection criteria (BIC, MDL, etc.) and decision theoretic approaches, with applications to ACT-R models including individual differences, and general hidden Markov processes. 1 1

In 4 experiments, the authors explored the role of visual layout in rule-based syntactic judgments. Participants judged the validity of a set of algebraic equations that tested their ability to apply the order of operations. In each experiment, a nonmathematical grouping pressure was manipulated to support or interfere with the mathematical convention. Despite the formal irrelevance of these grouping manipulations, accuracy in all experiments was highest when the nonmathematical pressure supported the mathematical grouping. The increase was significantly greater when the correct judgment depended on the order of operator precedence. The result that visual perception impacts rule application in mathematics has broad implications for relational reasoning in general. The authors conclude that formally symbolic reasoning is more visual than is usually proposed.

The paper surveys 15 years of progress in three psychometric research areas: latent dimensionality structure, test fairness, and skills diagnosis of educational tests. It is proposed that one effective model for selecting and carrying out research is to chose one’s research questions from practical challenges facing educational testing, then bring to bear sophisticated probability modeling and statistical analyses to solve these questions, and finally to make effectiveness of the research answers in meeting the educational testing challenges be the ultimate criterion for judging the value of the research. The problem-solving power and the joy of working with a dedicated, focused, and collegial group of colleagues is emphasized. Finally, it is suggested that the summative assessment testing paradigm that has driven test measurement research for over half a century is giving way to a new paradigm that in addition embraces skills level formative assessment, opening up a plethora of challenging, exciting, and societally important research problems for psychometricians.