Task-Based Evaluation of Summary Quality: Describing Relationships between Scientific Papers (2001)
| Venue: | In Workshop Automatic Summarization, NAACL |
| Citations: | 7 - 1 self |
BibTeX
@INPROCEEDINGS{Teufel01task-basedevaluation,
author = {Simone Teufel},
title = {Task-Based Evaluation of Summary Quality: Describing Relationships between Scientific Papers},
booktitle = {In Workshop Automatic Summarization, NAACL},
year = {2001},
pages = {12--21}
}
Years of Citing Articles
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
We present a novel method for task-based evalua- tion of summaries of scientific articles. The task we propose is a question-answering task, where the questions are about the relatedness of the current paper to prior research. This evaluation method is time-efficient with respect to material preparation and data collection, so that it is possible to test against many different baselines, something that is not usually feasible in evaluations by relevance decision. We use this methodology to evaluate the quality of summaries our system produces. These summaries are designed to describe the contribution of a scientific article in relation to other work. The re- sults show that this type of summary is indeed more useful than the baselines (random sentences, keyword lists and generic author-written summaries), and nearly as useful as the full texts.
