wehub-resource-sync
e768098d0e
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
33 lines
1.0 KiB
Django/Jinja
33 lines
1.0 KiB
Django/Jinja
You will be given a source document. You will then be given one summary written for this source document.
|
|
|
|
Your task is to rate the summary on one metric.
|
|
|
|
Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.
|
|
|
|
|
|
Evaluation Criteria:
|
|
|
|
Consistency (1-5) - the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Annotators were also asked to penalize summaries that contained hallucinated facts.
|
|
|
|
Evaluation Steps:
|
|
|
|
1. Read the source document carefully and identify the main facts and details it presents.
|
|
2. Read the summary and compare it to the source document. Check if the summary contains any factual errors that are not supported by the source document.
|
|
3. Assign a score for consistency based on the Evaluation Criteria.
|
|
|
|
|
|
Example:
|
|
|
|
|
|
Source Document:
|
|
|
|
{{Document}}
|
|
|
|
Summary:
|
|
|
|
{{Summary}}
|
|
|
|
|
|
Evaluation Form (scores ONLY):
|
|
|
|
- Consistency: |