Abstract

The TREC-5 filtering track, an evaluation of binary text classification systems, was a repeat of the filtering evaluation run in a trial version for TREC-4, with only the data set and participants changing. Seven sites took part, submitting a total of ten runs. We review the nature of the task, the effectiveness measures and evaluation methods used, and briefly discuss the results. Some deficiencies in the evaluation are examined, with an eye toward improving future filtering evaluations. 1 Introduction The goal of the TREC-5 filtering track was to aid research groups in evaluating their approaches to binary text classification. As usual in the TREC context, this was done by making available a large data set and doing blind, impartial evaluation of submitted results. The design used in the TREC-5 filtering track was identical to that tested with four sites in TREC-4, thus much of this paper is identical to the description of the TREC-4 filtering track [10]. We begin by defining binary...

Citations

365	A sequential algorithm for training text classifiers - Lewis, Gale - 1994
359	Sampling Techniques - Cochran - 1977
196	The probability ranking principle in ir - Robertson - 1997
126	Introduction to the practice of statistics - Moore, McCabe - 2006
110	Overview of the fourth text retrieval conference (TREC-4 - Harman - 1995
85	Evaluating and Optimizing Autonomous Text Classification Systems. AT&T Bell Laboratories Murray Hill, NJ 07974 - Lewis - 1995
76	Evaluating text categorization - Lewis - 1991
34	On selecting a measure of retrieval effectiveness - Cooper - 1997
22	TREC-3 ad-hoc, routing retrieval and thresholding experiments using PIRCS - Kwok, Grunfeld, et al.
5	Machine learning for knowledge-based document routing (a report on the TREC-2 experiment - Tong, Appelbaum - 1994
1	in TREC-2: Results of a boolean approximation method for routing and retrieval - GE - 1994