## Where to Stop Reading a Ranked List? ∗

### Cached

### Download Links

Citations: | 3 - 2 self |

### BibTeX

@MISC{Arampatzis_whereto,

author = {Avi Arampatzis and Jaap Kamps},

title = {Where to Stop Reading a Ranked List? ∗},

year = {}

}

### OpenURL

### Abstract

Abstract: We document our participation in the TREC 2008 Legal Track. This year we focused solely on selecting rank cut-offs for optimizing the given evaluation measure per topic. 1

### Citations

8134 | Maximum likelihood from incomplete data via the EM algorithm
- Dempster, Laird, et al.
- 1977
(Show Context)
Citation Context ...), find the bin with the most scores, and if that is not the first bin then and remove all scores in previous bins. 4.2.3 Expectation Maximization EM is an iterative procedure which converges locally =-=[8]-=-. Finding a global fit depends largely on the initial settings of the parameters. Initialization We tried numerous initial settings, but no setting seemed universal. While some settings helped a lot s... |

102 | Evaluating and optimizing autonomous text classification systems
- Lewis
- 1995
(Show Context)
Citation Context ...ty threshold, for others it does not. D. Lewis formulates this in terms of whether or not a measure satisfies the probability thresholding principle, and proves that the F measure does not satisfy it =-=[10]-=- . In other words, how a system should treat documents with, e.g., 50% chance of being relevant depends on how many documents with higher probabilities are available. Consequently, for such measures, ... |

79 | Modeling score distributions for combining the outputs of search engines
- Manmatha, Feng
- 2001
(Show Context)
Citation Context ...ities of relevance as we will see next. 3.2 Probability Thresholds Given the two densities and the generality defined earlier, scores can be normalized to probabilities of relevance straightforwardly =-=[2, 11]-=- by using the Bayes’ rule. Normalizing to probabilities is very important in tasks where several rankings need to be fused or merged such as in meta-search/fusion or distributed retrieval. This may al... |

52 |
e-Handbook of Statistical Methods
- NISTSEMATECH
- 2003
(Show Context)
Citation Context ... Fit To check the quality of the fits, we bin the scores and calculate the χ 2 statistic χ 2 = ∑ |Oi − Ei| 2 i Ei (16) where Oi and Ei are the observed and expected frequencies respectively for bin i =-=[12]-=-. The expected frequency is calculated by Ei = t (F (si,a) − F (si,b)) where si,a and si,b are respectively the lower and upper score limits of bin i, and F (s) = (1 − Gt)F (s|0) + GtF (s|1) is the cu... |

45 | Maximum Likelihood Estimation for Filtering Thresholds
- Zhang, Callan
- 2001
(Show Context)
Citation Context ... distributions have been modeled since the early years of IR with various known distributions [5, 6, 15, 16]. However, the trend during the last few years, which has started in [3] and followed up in =-=[1, 2, 7, 11, 17]-=-, has been to model score distributions by a mixture of normalexponential densities: normal for relevant, exponential for non-relevant. Despite its popularity, it was pointed out recently that, under ... |

40 |
Effectiveness of Information Retrieval Methods
- Swets
- 1969
(Show Context)
Citation Context ... elaborate on the form of the two densities P (s|1) and P (s|0) of Section 3.1 and their estimation. Score distributions have been modeled since the early years of IR with various known distributions =-=[5, 6, 15, 16]-=-. However, the trend during the last few years, which has started in [3] and followed up in [1, 2, 7, 11, 17], has been to model score distributions by a mixture of normalexponential densities: normal... |

38 |
Information retrieval systems
- Swets
- 1963
(Show Context)
Citation Context ... elaborate on the form of the two densities P (s|1) and P (s|0) of Section 3.1 and their estimation. Score distributions have been modeled since the early years of IR with various known distributions =-=[5, 6, 15, 16]-=-. However, the trend during the last few years, which has started in [3] and followed up in [1, 2, 7, 11, 17], has been to model score distributions by a mixture of normalexponential densities: normal... |

34 | The score-distributional threshold optimization for adaptive binary classification tasks - Arampatzis, Hameren - 2001 |

32 | A probabilistic solution to the selection and fusion problem in distributed information retrieval
- Baumgarten
- 1999
(Show Context)
Citation Context ... elaborate on the form of the two densities P (s|1) and P (s|0) of Section 3.1 and their estimation. Score distributions have been modeled since the early years of IR with various known distributions =-=[5, 6, 15, 16]-=-. However, the trend during the last few years, which has started in [3] and followed up in [1, 2, 7, 11, 17], has been to model score distributions by a mixture of normalexponential densities: normal... |

28 | Information filtering, novelty detection and named-page finding
- Ogilvie, Zhang, et al.
- 2002
(Show Context)
Citation Context ... distributions have been modeled since the early years of IR with various known distributions [5, 6, 15, 16]. However, the trend during the last few years, which has started in [3] and followed up in =-=[1, 2, 7, 11, 17]-=-, has been to model score distributions by a mixture of normalexponential densities: normal for relevant, exponential for non-relevant. Despite its popularity, it was pointed out recently that, under ... |

18 |
The parametric description of retrieval tests. part 1: the basic parameters; part 2: overall measures
- Robertson
- 1969
(Show Context)
Citation Context ...ly improved. As an example, let us consider smoothed precision. If it declines as score increases for a part of the score range, that part of the ranking can be improved by a simple random reordering =-=[14]-=-. This is equivalent of “forcing” the two underlying distributions to be uniform (i.e. have linearly increasing cdfs) in that score range. This will replace the offending part of the precision curve w... |

10 | On score distributions and relevance
- Robertson
- 2007
(Show Context)
Citation Context ...spite its popularity, it was pointed out recently that, under a hypothesis of how systems should score and rank documents, this particular mixture of normal-exponential presents a theoretical anomaly =-=[13]-=-. In practice, nevertheless, it has stand the test of time in the light of • its (relative) ease to calculate, • good experimental results, and • lack of a proven alternative. In this paper, we do not... |

8 |
When the most "pertinent" document should not be retrieved - an analysis
- BOOKSTEIN
- 1977
(Show Context)
Citation Context |

7 |
threshold optimization for adaptive document filtering
- Incrementality
- 2001
(Show Context)
Citation Context ...alent to thresholding in binary classification or filtering. Thus, we recruited a method first appeared in the TREC 2000 Filtering Track, namely, the score-distributional threshold optimization (s-d) =-=[2, 3]-=-. The method goes as follows. 3.1 The S-D Threshold Optimization Let us assume an item collection of size n, and a query for which all items are scored and ranked against. Let P (s|1) and P (s|0) be t... |

6 | Unbiased s-d threshold optimization, initial query degradation, decay, and incrementality, for adaptive document filtering
- Arampatzis
- 2002
(Show Context)
Citation Context ... (s|0) be the probability densities of relevant and non-relevant documents as a function of the score s, and F (s|1) and F (s|0) their corresponding cumulative distribution functions (cdfs). Let Gn ∈ =-=[0, 1]-=- be the fraction of relevant documents in the collection, also known as generality. The total number of relevant documents in the collection is given by R = n Gn (1) while the numbers of relevant and ... |

4 | The parametric description of retrieval tests - ROBERTSON - 1969 |

3 | Optimal data-based binning for histograms
- Knuth
(Show Context)
Citation Context ...quencies in Equation 16 before squaring. The χ 2 statistic is sensitive to the choice of bins. Score Binning For binning, we use the optimal number of bins as this is given by the method described in =-=[9]-=-. The method considers the histogram to be a piecewise-constant model of the underlying probability density. Then, it computes the posterior probability of the number of bins for a given data set. Thi... |

2 | Access to legal documents: Exact match, best match, and combinations
- Arampatzis, Kamps, et al.
- 2007
(Show Context)
Citation Context ...cial submissions, results, and additional experiments. Finally, we summarize the findings in Section 6. 2 Experimental Set-up We employed the same experimental set-up as last year, fully described in =-=[4]-=-. Specifically, document pre-processing, indexing, and retrieval model, are the same as for last year’s post-submission run tagged in the last-cited study as textonly, i.e. our best run in terms of me... |

1 | Threshold optimization using truncated score distributions. Unpublished - Arampatzis, Kamps, et al. - 2009 |

1 | Optimal data-based binning for histograms, URL http://arxiv.org/abs/physics/ 0605197v1 - Knuth |