Summarization Evaluation

Evaluation can involve various kinds of comparisons. Three kinds of methods are as following:
1. system summaries compared with human summaries
2. system summaries compared with full-text sources
3. system summaries compared with each other

In March this year I had done something on summarization evaluation. At that time I had used four methods. They were all included in the first type method.

As I must realize some evaluation program to simulating the results of human beings. However, my former four methods were not good enough. Now I could use some linear interpolation methods by the three knids of evaluation methods.

Considering lots of the methods of system summaries comparing with human summaries, I could use one of my original four. That was uniting the human summaries by 5-3 principle and calculating the F score with the system sumaries.

To system summaries comparing with full-text sources, I could borrow the methods of documents classifying. Constructing the vectors of the source text and the system summaries, using the cos-similarity for substitute, I could comparing the trend of cos-similarity to the human socring.

Ok. This idea should be realized tomorrow.