Related materials on Summarization Evaluation

There were so many materials on summarization evaluation. In the recent DUC 2004 conference, there was a summarization evaluation tool named as ROUGE. It's main idea was calculating the n-gram co-occurence rate. Following the successful application of automatic evaluation methods, such as BLEU, in machine learning translation evaluation, Lin and Hovy(2003) showed that methods similiar to BLEU, i.e. n-gram co-occurance statistics, could be applied to evaluate summaries.

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes several automatic evaluation methods that measure the similaity between summaries.


Reference:

[1] Chin-Yew Lin, ROUGE: A Package for Automatic Evaluation of Summaries, ACL2004
[2] Chew-Yew Lin, and E.H.Hovy.2003. Automatic evaluation of summaries using n-gram co-occurance statics. In Proceedings of 2003 Language Technologyu Conference, Edmonton, Canada.