Abstract

This paper presents a novel automatic method (AutoSummENG) for the evaluation of summarization systems, based on comparing the character n-gram graphs representation of the extracted summaries and a number of model summaries. The presented approach is language neutral, due to its statistical nature, and appears to hold a level of evaluation performance that matches and even exceeds other contemporary evaluation methods. Within this study, we measure the effectiveness of different representation methods, namely word and character n-gram graph and histogram, different n-gram neighbourhood indication methods, as well as different comparison methods between the supplied representations. A theory for the a priori determination of the methods ’ parameters, along with supporting experiments, concludes the study, to provide a complete alternative of existing methods concerning the automatic summary system evaluation process.

Citations