chore: import upstream snapshot with attribution

2026-07-13 13:39:52 +08:00
commit e768098d0e
4004 changed files with 2804145 additions and 0 deletions
@@ -0,0 +1,143 @@
+# Summarization Evaluation
+
+This flow implements a reference-free automatic abstractive summarization evaluation across four dimensions: fluency, coherence, consistency, relevance. Each dimension uses a prompt with GPT-4 to score a generated summary against the source document from which it was generated. The implementation is based on the [G-Eval research paper](https://arxiv.org/abs/2303.16634).
+
+## Introduction
+
+### Background
+
+Abstractive summarization evaluation is a hard problem for which many previous automatic methods have performed poorly in-terms of correlation with human judgements. Expert human created ground truths for summarization are hard to obtain and also hard to compare automatically to generated summaries. Prior research defines four dimensions to abstractive summary quality (each scored on a 1-5 Likert scale):
+- Coherence - the collective quality of all sentences in the summary
+- Consistency - factual alignment between the summary and source document
+- Fluency - the quality of individual sentences of the summary
+- Relevance - selection of the most important content from the source document
+
+Thus it is possible to measure summary quality based on the inherent writing quality of the summary alone (in terms of coherence and fluency) and alignment between the summary and the source document (in terms of consistency and relevance). This affords the potential for a reference-free evaluation of abstractive summary generation.
+
+This flow implements G-Eval for summarization evaluation that has been adopted from the [research paper](https://arxiv.org/abs/2303.16634) and associated [Github repository](https://github.com/nlpyang/geval). This method introduces a reference-free prompt based evaluation with GPT-4 for each of the 4 standard dimensions of summarization evaluation and shows state-of-the-art results in terms of correlation to human judgements based on meta-evaluation against the [SummEval benchmark](https://arxiv.org/abs/2007.12626).
+
+## Tools Used in this Flow
+
+Tools used in this flow：
+
+- `python` tool the implements direct calls to GPT-4 (due to the need for using n=20, which is currently unavailable as a parameter in prompt flow LLM nodes) for each dimension's evaluation
+
+## Pre-requisites
+
+Install Prompt Flow SDK and other dependencies in this folder:
+
+```bash
+pip install -r requirements.txt
+```
+
+## Getting Started
+
+### Setup connection
+
+Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base>
+```
+
+Note that this evaluation flow is only validated to work with certain GPT-4 models versions (see meta-evaluation section).
+
+## Usage Examples
+
+This flow will evaluate a generated summary with respect to the source document it was generated from. Thus the flow requires two inputs, a document and a summary, as shown in the examples below. The output of the flow will be a score for each dimension of summarization evaluation (fluency, coherence, consistency, relevance) for each summary and if run in batch mode these dimensions will be averaged over each document and summary being evaluated.
+
+### 1. Test flow with single line data
+
+```bash
+# test with flow inputs
+pf flow test --flow . --inputs document=ABC summary=ABC
+```
+
+### 2. Create flow run with multi-line data
+
+```bash
+pf run create --flow . --data ./data.jsonl --column-mapping document='${data.document}' summary='${data.summary}' --stream
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+## Implementation & Usage Details
+
+The original G-Eval paper and repository has created prompts that refer to inputs as 'news-articles' (inline with the source data in the SummEval benchmark), we have modified the evaluation prompts provided in this flow to be more generic and agnostic to the domain of the source data being evaluated. Our implementation also includes an updated parser (extracting scores from 20 trial outputs) and other code improvements.
+
+This flow scores generated summaries along each summarization evaluation dimension and also aggregates each dimension as an average when run in batch mode.
+
+### Limitations
+
+#### Cost considerations
+An important note regarding the implementation and use of this flow is that for each dimension, the evaluation prompt is run using GPT-4 with temperature=2 and n=20, from which the final score is averaged across each trial. This is done as an approximation for token probabilities which are currently unavailable for GPT-4. This is run for each dimension for each summary being evaluated.
+
+A rough estimate of the output tokens consumed by the flow per summary evaluation would be:
+```
+single_summary_eval_output_tokens = max_output_tokens*number_trials*number_dimensions = 5*20*4 = 400 output tokens
+```
+
+For input token estimation, if we assume a 500 word input doc and 50 word summary to evaluate, then the input tokens would be roughly (assuming 4 tokens per word):
+```
+(dimension input tokens = prompt tokens + source doc tokens + summary tokens)
+
+consistency input tokens = 210 + 125 + 13 = 348 input tokens
+fluency input tokens = 183 + 13 = 196 input tokens
+relevance input tokens = 200 + 125 + 13 = 338 input tokens
+coherence input tokens = 248 + 125 + 13 = 381 input tokens
+
+total input tokens per doc/summary = 1263 input tokens
+```
+For a batch run of 100 documents, the price of using this flow could be estimated as:
+```
+total cost of summary eval flow = ((numb_summaries * input_tokens_per_summary / 1000) * gpt_4_cost_per_1k_output_tokens) + ((numb_summaries * output_tokens_per_summary / 1000) * gpt_4_cost_per_1k_output_tokens)
+
+total cost of summary eval flow = ((100 * 1263 / 1000) * gpt_4_cost_per_1k_input_tokens) + ((100 * 400 / 1000) * gpt_4_cost_per_1k_output_tokens)
+total cost of summary eval flow = (126.3 * gpt_4_cost_per_1k_input_tokens) + (40 * gpt_4_cost_per_1k_output_tokens)
+
+Using GPT-4 8k model in East US region with USD pricing (as of 27/02/2024):
+total cost of summary eval flow = (126.3 * $0.03) + (40 * $0.06) = $6.19 per 100 documents
+```
+
+Please substitute your own values for various variables to estimate the cost of running the evaluation flow on your data.
+
+#### Scoring bias
+
+The G-Eval research paper showed that G-Eval with GPT-4 has a bias to always scoring generated summaries (from GPT-35) higher than human written summaries, even when human reviewers would judge human written summaries to be better. In practice, we have observed this tendency of G-Eval with GPT-4 to produce a higher distribution of scores for each dimension. The mitigation we suggest for this is to sample a wider range at the bottom of the distribution (for each dimension) when conducting evaluation and error analysis.
+
+### Meta-evaluation
+
+The changes introduced in this flow's implementation (compared to the original G-Eval implementation from the research paper) have been meta-evaluated against the SummEval benchmark and show similar performance to the original implementation. As the prompts have been updated to be more generic we expect some change in performance to the original implementation which has tuned prompts to the SummEval benchmark (referring to news articles), but the updated implementation still shows state-of-the-art results compared to other metrics (see G-Eval paper for those results).
+
+Meta-evaluation Spearman correlations between different experiments and human judgements in the SummEval benchmark:
+
+| Dimension/Prompt       | Fluency | Consistency | Coherence | Relevance |
+| ---------------------- | ------- | ----------- | --------- | --------- |
+| GPT-4 0613 8k + original prompts in paper | 0.455  | 0.507      | 0.582    | 0.547    |
+| GPT-4 0613 8k + updated prompts + original parser | 0.5079  | 0.5102      | 0.4998    | 0.4606    |
+| GPT-4 0613 8k + updated prompts + updated parser | 0.5402  | 0.5215      | 0.5137    | 0.4897    |
+| GPT-4 0613 32k + updated prompts + updated parser | 0.4985  | 0.4914      | 0.5038    | 0.4921    |
+
+Note that GPT-4 Turbo has shown poor results when meta-evaluated and is currently not recommended to be used with this flow. It is recommended to use this flow and it's prompts with only the GPT-4 model versions listed above, as the meta-evaluation results have been verified for these model versions. 
+
+There is an assumption that meta-evaluation results against the SummEval benchmark will generalise to the domain of your data, if you have concerns about this you should consider conducting your own meta-evaluation. This would include taking a significant sample of source documents and generated summaries and obtaining expert human judgements for each dimension of summarization evaluation (fluency, coherence, consistency, relevance). Then the prompts for each dimension should be tuned until sufficient correlation with human judgements are obtained.
+
+
+## How to run Unit Tests
+
+1. Make sure you already finished [Prerequisites](#pre-requisites)
+1. Run the following commands
+
+    ```bash
+    pip install pytest
+    python -m pytest tests
+    ```
+
+## Contact
+
+If you have any questions or issues related to this flow, please reach out to either:
+
+- Bhavik Maneck [[Email](mailto:bhavikmaneck@microsoft.com) | [Linked-In](https://www.linkedin.com/in/bhavik-maneck/)]
+- Kosuke Fujimoto [[Email](mailto:kofuji@microsoft.com) | [Linked-In](https://www.linkedin.com/in/kosuke-fuji/)]
@@ -0,0 +1,40 @@
+from typing import Dict, List
+
+from promptflow.core import log_metric, tool
+
+
+@tool
+def aggregate(
+    fluency_list: List[float],
+    consistency_list: List[float],
+    relevance_list: List[float],
+    coherence_list: List[float],
+) -> Dict[str, float]:
+    """
+    Takes list of scores for 4 dims and outputs average for them.
+
+    Args:
+        fluency_list (List(float)): list of fluency scores
+        consistency_list (List(float)): list of consistency scores
+        relevance_list (List(float)): list of relevance scores
+        coherence_list (List(float)): list of coherence scores
+
+    Returns:
+        Dict[str, float]: Returns average scores
+    """
+    average_fluency = sum(fluency_list) / len(fluency_list)
+    average_consistency = sum(consistency_list) / len(consistency_list)
+    average_relevance = sum(relevance_list) / len(relevance_list)
+    average_coherence = sum(coherence_list) / len(coherence_list)
+
+    log_metric("average_fluency", average_fluency)
+    log_metric("average_consistency", average_consistency)
+    log_metric("average_relevance", average_relevance)
+    log_metric("average_coherence", average_coherence)
+
+    return {
+        "average_fluency": average_fluency,
+        "average_consistency": average_consistency,
+        "average_relevance": average_relevance,
+        "average_coherence": average_coherence,
+    }
@@ -0,0 +1,2 @@
+{"document": "this is a test document", "summary": "test document"}
+{"document": "this is a test document2", "summary": "test document2"}
@@ -0,0 +1,104 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+environment:
+  python_requirements_txt: requirements.txt
+inputs:
+  document:
+    type: string
+  summary:
+    type: string
+outputs:
+  coherence:
+    type: double
+    reference: ${score_coherence.output}
+  consistency:
+    type: double
+    reference: ${score_consistency.output}
+  fluency:
+    type: double
+    reference: ${score_fluency.output}
+  relevance:
+    type: double
+    reference: ${score_relevance.output}
+nodes:
+- name: prompt_coherence
+  type: prompt
+  source:
+    type: code
+    path: prompts/coherence.jinja2
+  inputs:
+    Document: ${inputs.document}
+    Summary: ${inputs.summary}
+- name: score_coherence
+  type: python
+  source:
+    type: code
+    path: geval.py
+  inputs:
+    connection: open_ai_connection
+    prompt_with_src_and_gen: ${prompt_coherence.output}
+    max_score: 5
+    deployment_name: gpt-4
+- name: prompt_consistency
+  type: prompt
+  source:
+    type: code
+    path: prompts/consistency.jinja2
+  inputs:
+    Document: ${inputs.document}
+    Summary: ${inputs.summary}
+- name: score_consistency
+  type: python
+  source:
+    type: code
+    path: geval.py
+  inputs:
+    connection: open_ai_connection
+    prompt_with_src_and_gen: ${prompt_consistency.output}
+    max_score: 5
+    deployment_name: gpt-4
+- name: prompt_fluency
+  type: prompt
+  source:
+    type: code
+    path: prompts/fluency.jinja2
+  inputs:
+    Summary: ${inputs.summary}
+- name: score_fluency
+  type: python
+  source:
+    type: code
+    path: geval.py
+  inputs:
+    connection: open_ai_connection
+    prompt_with_src_and_gen: ${prompt_fluency.output}
+    max_score: 3
+    deployment_name: gpt-4
+- name: prompt_relevance
+  type: prompt
+  source:
+    type: code
+    path: prompts/relevance.jinja2
+  inputs:
+    Document: ${inputs.document}
+    Summary: ${inputs.summary}
+- name: score_relevance
+  type: python
+  source:
+    type: code
+    path: geval.py
+  inputs:
+    connection: open_ai_connection
+    prompt_with_src_and_gen: ${prompt_relevance.output}
+    max_score: 5
+    deployment_name: gpt-4
+- name: average_scores
+  type: python
+  source:
+    type: code
+    path: average_scores.py
+  inputs:
+    fluency_list: ${score_fluency.output}
+    consistency_list: ${score_consistency.output}
+    relevance_list: ${score_relevance.output}
+    coherence_list: ${score_coherence.output}
+  aggregation: true
@@ -0,0 +1,184 @@
+import logging
+import logging.config
+import re
+from pathlib import Path
+from typing import List
+
+import promptflow
+import yaml
+from openai import AzureOpenAI
+from promptflow.connections import AzureOpenAIConnection
+from tenacity import (
+    RetryError,
+    Retrying,
+    after_log,
+    before_sleep_log,
+    stop_after_attempt,
+    wait_random_exponential,
+)
+
+
+class Logger:
+    """
+    A class for setting up and getting a logger object.
+
+    Attributes:
+        config_file (str): Path to the YAML file containing the logger configuration.
+        logger (logging.Logger): The logger object.
+    """
+
+    def __init__(self):
+        """
+        Initializes the Logger class.
+
+        Args:
+            config_file (str): Path to YAML file containing the logger configuration.
+        """
+        config_file = Path(__file__).parent.joinpath("log_config.yaml").resolve()
+        with open(config_file, "r") as f:
+            config = yaml.safe_load(f.read())
+            logging.config.dictConfig(config)
+        self.logger = logging.getLogger(__name__)
+
+    def get_logger(self):
+        """
+        Returns the logger object.
+
+        Returns:
+            logging.Logger: The logger object.
+        """
+        return self.logger
+
+
+logger = Logger().get_logger()
+
+
+def parse_output(output: str, max: float) -> float:
+    """
+    Function that extracts numerical score from the beginning of string
+
+    Args:
+        output (str): String to search
+        max (float): Maximum score allowed
+
+    Returns:
+        float: The extracted score
+    """
+    # match with either non-negative float or integer
+    # if number has non-whitespace characture before that, it won't match
+    matched: List[str] = re.findall(r"(?<!\S)\d+(?:\.\d+)?", output)
+    if matched:
+        if len(matched) == 1:
+            score = float(matched[0])
+            if score > max:
+                raise ValueError(
+                    f"Parsed number: {score} was larger than max score: {max}"
+                )
+        else:
+            raise ValueError(
+                f"More than one number detected in input. Input to parser was: {output}"
+            )
+    else:
+        raise ValueError(
+            f'No number detected in input. Input to parser was "{output}". '
+        )
+    return score
+
+
+@promptflow.tool
+def geval_summarization(
+    prompt_with_src_and_gen: str,
+    max_score: float,
+    connection: AzureOpenAIConnection,
+    deployment_name: str = "gpt-4",
+) -> float:
+    """Using GPT, evaluate a generated summary with respect to a source document from
+    which it was generated. This function should be used for four dimensions of
+    summarization evaluation inline with the SummEval benchmark: fluency, coherence,
+    consistency, relevance.
+
+    Args:
+        prompt_with_src_and_gen (str): The prompt containing the source document and generated summary.
+        max_score (float): The maximum score allowed.
+        connection (AzureOpenAIConnection): The connection object for Azure OpenAI.
+        deployment_name (str, optional): The name of the deployment. Defaults to "gpt-4".
+
+    Returns:
+        float: The evaluation score
+    """
+    # make sure you use the same api version/model with the one used for meta evaluation
+    logger.info(
+        f"OpenAI API Base: {connection.api_base} - Version: {connection.api_version}"
+        f" - Deployment: {deployment_name}"
+    )
+    client = AzureOpenAI(
+        azure_endpoint=connection.api_base,
+        api_version=connection.api_version,
+        api_key=connection.api_key,
+    )
+
+    message = {"role": "system", "content": prompt_with_src_and_gen}
+    try:
+        for attempt in Retrying(
+            reraise=True,
+            before_sleep=before_sleep_log(logger, logging.INFO),
+            after=after_log(logger, logging.INFO),
+            wait=wait_random_exponential(multiplier=1, min=1, max=120),
+            stop=stop_after_attempt(10),
+        ):
+            with attempt:
+                response = client.chat.completions.create(
+                    model=deployment_name,
+                    messages=[message],
+                    temperature=2,
+                    max_tokens=5,
+                    top_p=1,
+                    frequency_penalty=0,
+                    presence_penalty=0,
+                    stop=None,
+                    n=20,
+                )
+    except RetryError:
+        logger.exception(f"geval openai call failed\nInput prompt was: {message}")
+        raise
+
+    all_responses = []
+    for i in range(len(response.choices)):
+        try:
+            content = response.choices[i].message.content
+            all_responses.append(content)
+        except KeyError:
+            # `content` won't exist in returned json when openai content_filter is triggered
+            logger.exception(
+                f"""data with key missing was: {response.choices[i]}\nInput prompt was: {message}"""
+            )
+
+    return aggregate_llm_scores(all_responses, max_score=max_score)
+
+
+def aggregate_llm_scores(llm_responses: List[str], max_score: int) -> float:
+    """Parse and average valid scores from the generated responses of
+    the G-Eval LLM call.
+
+    Args:
+        llm_responses (List[str]): List of scores from multiple LLMs
+        max_score (float): The maximum score allowed.
+
+    Returns:
+        float: The average of all the valid scores
+    """
+    all_scores = []
+    error_count = 0
+    for generated in llm_responses:
+        try:
+            parsed = parse_output(generated, max_score)
+            all_scores.append(parsed)
+        except ValueError as e:
+            logger.warning(e)
+            error_count += 1
+    if error_count:
+        logger.warning(
+            f"{error_count} out of 20 scores were discarded due to corrupt g-eval generation"
+        )
+    score = sum(all_scores) / len(all_scores)
+    return score
@@ -0,0 +1,22 @@
+version: 1
+disable_existing_loggers: False
+formatters:
+  simple:
+    format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+handlers:
+  console:
+    class: logging.StreamHandler
+    level: INFO
+    formatter: simple
+    stream: ext://sys.stdout
+  file:
+    class: logging.handlers.RotatingFileHandler
+    level: INFO
+    formatter: simple
+    filename: app.log
+    maxBytes: 10485760
+    backupCount: 20
+    encoding: utf8
+root:
+  level: DEBUG
+  handlers: [console, file]
@@ -0,0 +1,32 @@
+You will be given one summary written for a source document.
+
+Your task is to rate the summary on one metric.
+
+Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
+
+Evaluation Criteria:
+
+Coherence (1-5) - the collective quality of all sentences. We align this dimension with the DUC quality question of structure and coherence whereby "the summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to a coherent body of information about a topic."
+
+Evaluation Steps:
+
+1. Read the source document carefully and identify the main topic and key points.
+2. Read the summary and compare it to the source document. Check if the summary covers the main topic and key points of the source document, and if it presents them in a clear and logical order.
+3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
+
+
+Example:
+
+
+Source Document:
+
+{{Document}}
+
+Summary:
+
+{{Summary}}
+
+
+Evaluation Form (scores ONLY):
+
+- Coherence:
@@ -0,0 +1,33 @@
+You will be given a source document. You will then be given one summary written for this source document.
+
+Your task is to rate the summary on one metric.
+
+Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
+
+
+Evaluation Criteria:
+
+Consistency (1-5) - the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Annotators were also asked to penalize summaries that contained hallucinated facts. 
+
+Evaluation Steps:
+
+1. Read the source document carefully and identify the main facts and details it presents.
+2. Read the summary and compare it to the source document. Check if the summary contains any factual errors that are not supported by the source document.
+3. Assign a score for consistency based on the Evaluation Criteria.
+
+
+Example:
+
+
+Source Document: 
+
+{{Document}}
+
+Summary: 
+
+{{Summary}}
+
+
+Evaluation Form (scores ONLY):
+
+- Consistency:
@@ -0,0 +1,26 @@
+You will be given one summary written for a source document.
+
+Your task is to rate the summary on one metric.
+
+Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
+
+
+Evaluation Criteria:
+
+Fluency (1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
+
+- 1: Poor. The summary has many errors that make it hard to understand or sound unnatural.
+- 2: Fair. The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.
+- 3: Good. The summary has few or no errors and is easy to read and follow.
+
+
+Example:
+
+Summary:
+
+{{Summary}}
+
+
+Evaluation Form (scores ONLY):
+
+- Fluency (1-3):
@@ -0,0 +1,33 @@
+You will be given one summary written for a source document.
+
+Your task is to rate the summary on one metric.
+
+Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
+
+Evaluation Criteria:
+
+Relevance (1-5) - selection of important content from the source. The summary should include only important information from the source document. Annotators were instructed to penalize summaries which contained redundancies and excess information.
+
+Evaluation Steps:
+
+1. Read the summary and the source document carefully.
+2. Compare the summary to the source document and identify the main points of the source document.
+3. Assess how well the summary covers the main points of the source document, and how much irrelevant or redundant information it contains.
+4. Assign a relevance score from 1 to 5.
+
+
+Example:
+
+
+Source Document:
+
+{{Document}}
+
+Summary:
+
+{{Summary}}
+
+
+Evaluation Form (scores ONLY):
+
+- Relevance:
@@ -0,0 +1,4 @@
+promptflow[azure]
+promptflow-tools
+tenacity
+openai
@@ -0,0 +1,50 @@
+import pytest
+
+from geval import parse_output
+
+
+@pytest.mark.parametrize(
+    "input_str, expected_output",
+    [
+        ("4", 4.0),
+        ("4.4", 4.4),
+        ("3 is the score because", 3.0),
+        (" 2 ", 2.0),
+        (" 2a", 2.0),
+        ("0", 0.0),
+        ("0/5", 0.0),
+    ],
+)
+def test_parse_valid_score(input_str, expected_output):
+    assert parse_output(input_str, max=5) == expected_output
+
+
+@pytest.mark.parametrize(
+    "input_str",
+    [
+        "No score at beginning",
+        "",
+        "     ",
+        "aaaa2bbb",
+        "-2-",
+        "-2a",
+        "a-2a",
+        "-2",
+        "-2.0",
+    ],
+)
+def test_parse_no_number_detected(input_str):
+    with pytest.raises(ValueError, match="No number detected in input"):
+        parse_output(input_str, max=5)
+
+
+@pytest.mark.parametrize("input_str", ["6", " 1234 is the score because", "8.8"])
+def test_parse_too_large_score(input_str):
+    with pytest.raises(ValueError, match="larger than max score"):
+        parse_output(input_str, max=5)
+
+
+@pytest.mark.parametrize("input_str", [" 1 to 3 ", "5.5   99 "])
+def test_parse_multiple_number_detected(input_str):
+    with pytest.raises(ValueError, match="More than one number detected in input"):
+        parse_output(input_str, max=5)