We introduce the DET Curve as a means of representing performance on detection tasks that involve a tradeoff of error types. We discuss why we prefer it to the traditional ROC Curve and offer several examples of its use in speaker recognition and language recognition. We explain why it is likely to produce approximately linear curves. We also note special points that may be included on these curves, how they are used with multiple targets, and possible further applications.

This paper describes and analyzes the results of the Third Message Understanding Conference (MUC-3). It reviews the purpose, history, and methodology of the conference, summarizes the participating systems, discusses issues of measuring system effectiveness, describes the linguistic phenomena tests, and provides a critical look at the evaluation in terms of the lessons learned. One of the common problems with evaluations is that the statistical significance of the results is unknown. In the discussion of system performance, the statistical significance of the evaluation results is reported and the use of approximate randomization to calculate the statistical significance of the results of MUC-3 is described

The current variety of alternative approaches to usability evaluation methods (UEMs) designed to assess and improve usability in software systems is offset by a general lack of understanding of the capabilities and limitations of each. Practitioners need to know which methods are more effective and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard criteria for comparison. In this article, we present a practical discussion of factors, comparison criteria, and UEM performance measures useful in studies comparing UEMs. In demonstrating the importance of developing appropriate UEM evaluation criteria, we offer operational definitions and possible measures of UEM performance. We highlight specific challenges that researchers and practitioners face in comparing UEMs and provide a point of departure for further discussion and refinement of the principles and techniques used to approach UEM evaluation and comparison. 1.

Although Shannon’s information theory is alive and well in a number of fields, after an initial fad in psychology during the 1950s and 1960s it no longer is much of a factor, beyond the word bit, in psychological theory. The author discusses what seems to him (and others) to be the root causes of an actual incompatibility between information theory and the psychological phenomena to which it has been applied. Claude Shannon, the creator of information theory, or communication theory as he preferred to call it, died on February 24, 2001, at age 84. So, I would like to dedicate this brief piece to his memory and in particular to recall his seminal contribution “A Mathematical Theory of Communication, ” which was published in two

Scientifi c tools—measurement and calculation instruments, techniques of inference—straddle the line between the context of discovery and the context of justifi cation. In discovery, new scientifi c tools suggest new theoretical metaphors and concepts; and in justifi cation, these tool-derived theoretical metaphors and concepts are more likely to be accepted by the scientifi c community if the tools are already entrenched in scientifi c practice. Techniques of statistical inference and hypothesis testing entered American psychology fi rst as tools in the 1940s and 1950s and then as cognitive theories in the 1960s and 1970s. Not only did psychologists resist statistical metaphors of mind prior to the institutionalization of inference techniques in their own practice; the cognitive theories they ultimately developed about “the mind as intuitive statistician ” still bear the telltale marks of the practical laboratory context in which the tool was used.

The need for cost-effective usability evaluation has led to the development of methodologies to support the usability practitioner in finding usability problems during formative evaluation. Even though various methods exist for performing usability evaluation, practitioners seldom have the information needed to decide which method is appropriate for their specific purpose. In addition, most methods do not have an integrated relationship with a theoretical foundation for applying the method in a reliable and efficient manner. Practitioners often have to apply their own judgment and techniques, leading to inconsistencies in how the method is applied in the field. Usability practitioners need validated information to determine if a given usability evaluation method is effective and why it should be used instead of some other method. Such a desire motivates the need for formal, empirical comparison studies to evaluate and compare usability evaluation methods. In reality, the current data for comparing usability evaluation methods suffers from a lack of consistent measures, standards, and criteria for identifying effective methods. The work described here addresses three important research activities. First, the User Action Framework was developed to help organize usability concepts and issues into a

This paper presents a new decision tree learning algorithm that takes account of costs of misclassification. The algorithm is based on the hypothesis that non-linear decision nodes provide a better basis for cost-sensitive induction than axis-parallel decision nodes and utilizes discriminant analysis to construct non-linear cost-sensitive decision trees. The performance of the algorithm is evaluated by applying it to seven data sets and the results compared with those obtained by two well known cost-sensitive algorithms, ICET and MetaCost. MetaCost is applied both with a base learner that uses axis-parallel spits and a base learner that uses non-linear splits, thereby enabling an evaluation of the value of non-linear decision nodes. The results show that the new algorithm displays a better profile than ICET as the ratio of costs of misclassification departs from unity and that it provides a better base learner for MetaCost than an axis-parallel learner. 1

Abstract not found

Three studies relating perception of motion to stimulus uncertainty are reported. Generally, detectability declines when the observer is uncertain about the direction in which a target will move, but the visibility loss associated with direction uncertainty can be attenuated if the observer has adequate practice. This attenuation seems to depend upon the observer's ability to switch among directionally selective visual mechanisms in an adaptive fashion. The implications of these findings for models of motion detection are discussed. Although a great deal is known about how visual motion is detected in the laboratory, there are impediments to applying that information outside the laboratory. Inside the laboratory, observers are typically well informed about the stimuli they will have to detect; outside the laboratory, most visual stimuli are not so predictable. This unpredictability, termed stimulus uncertainty, strongly affects visibility of motion. Our concern in this article is with stimulus uncertainty and with the ways in which observers cope with it. More particularly, we are concerned with one source of uncertainty, that associated with the direction in which a target will move. There are two main reasons for examining stimulus uncertainty in the context of motion perception. First, it is already known that stimulus uncertainty produces quite substantial performance changes with moving

NIST has coordinated annual evaluations of textindependent speaker recognition since 1996. During the course of this series of evaluations there have been notable milestones related to the development of the evaluation paradigm and the performance achievements of state-of-the-art systems. We document here the variants of the speaker detection task that have been included in the evaluations and the history of the best performance results for this task.