Abstract

We present a new visual method for assessing the predictive power of models with categorical outcomes. This technique allows the analyst to quickly and easily choose among alternative model specifications based upon the modelsʼ ability to consistently match high-probability predictions to actual occurrences of the event of interest, and low-probability predictions to non-occurrences of the event of interest. Unlike existing methods for assessing predictive power for logit and probit models such as the use of “percent correctly predicted ” statistics, Brier scores and the ROC plot, our “separation plot ” has the advantage of producing a visual display that is more informative and easier to explain to a general audience than a ROC plot, while also remaining insensitive to the user's often arbitrary choice of threshold for distinguishing between events and non-events. We show how to implement this tool as a function in R. 2. How itʼs made Example using a logit model of O-ring failure as a function of launch temperature using data from the Challenger dataset 1. Start with actual and predicted data

Citations