My problem of summarization evaluation

I have studied one of the three papers about summarization evaluation of ACL 2004. The guidline was as follows:

Title: Automatic Evaluation of Summaries Using Document Graphs
Author: Eugene Santos Jr., Ahmed A.Mohamed, and Qunhua Zhao
Organization: Computer Science and Engineering Department University of Connecticut 191 Auditorium Road, U-155, Storrs, CT 0269-3155 {Eugene,amohamed,qzhao}@engr.uconn.edu
Abstract: Summarization evaluation has been always a challenge to researchers in the document summarization field. Usually, human involvement is necessary to evaluate the quality of a summary. Here we present a new method for automatic evaluation of text summaries by using document graphs. Data from Document Understanding Conference 2002 (DUC-2002) has been used in the experiment. We propose measuring the similarity between two summaries or between a summary and a document based on the concept/entities and relation between them in the text.
Essentials: There is a bottleneck in text summarization research field that is how to evaluate the quality of a summary or the performance of a summarization tool.
In this paper, there is a newly view of summaries evaluation based on the document graphs(DG). In the document graphs there are only two kinds of nodes: concept/entity nodes and relation nodes. Currently, only two kinds of relations, "isa" and "related to", are captured for simplicity. Comparing the similarity between two document graphs, we could evaluate two document. There was a trend that the similarity comparison between the summary DG and the source DG is nearly same as that of the evaluation by human. So this tool could be instead of the human involvement of the evaluation.
Use for reference: The concept/entity node and the relation node were generated in the bound of the noun phrase. But in Chinese there is not any good noun phrase identifier. This is a bottleneck of this approach using in Chinese summarization.