wehub-resource-sync
e768098d0e
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
29 lines
2.5 KiB
Django/Jinja
29 lines
2.5 KiB
Django/Jinja
# System:
|
|
You are an AI assistant. You will be given the definition of an evaluation metric for assessing the quality of an answer in a question-answering task. Your job is to compute an accurate evaluation score using the provided evaluation metric.
|
|
# User:
|
|
You will be presented with a CONTEXT and an ANSWER about that CONTEXT. You need to decide whether the ANSWER is entailed by the CONTEXT by choosing one of the following rating:
|
|
1. 5: The ANSWER follows logically from the information contained in the CONTEXT.
|
|
2. 4: Most of the ANSWER follows logically from the information contained in the CONTEXT.
|
|
3. 3: The ANSWER follows logically from the information contained in the CONTEXT.
|
|
4. 2: The ANSWER follows logically from the information contained in the CONTEXT.
|
|
5. 1: The ANSWER is logically false from the information contained in the CONTEXT.
|
|
6. an integer score between 1 and 5 and if such integer score does not exist, use 1: It is not possible to determine whether the ANSWER is true or false without further information. Read the passage of information thoroughly and select the correct answer for it. Read the CONTEXT thoroughly to ensure you know what the CONTEXT entails. Note the ANSWER is generated by a computer system, it can contain certain symbols, which should not be a negative factor in the evaluation.
|
|
|
|
# Examples:
|
|
## Example Task #1 Input:
|
|
{"CONTEXT": "Some are reported as not having been wanted at all.", "QUESTION": "", "ANSWER": "All are reported as being completely and fully wanted."}
|
|
## Example Task #1 Output:
|
|
1
|
|
## Example Task #2 Input:
|
|
{"CONTEXT": "Ten new television shows appeared during the month of September. Five of the shows were sitcoms, three were hourlong dramas, and two were news-magazine shows. By January, only seven of these new shows were still on the air. Five of the shows that remained were sitcoms.", "QUESTION": "", "ANSWER": "At least one of the shows that were cancelled was an hourlong drama."}
|
|
## Example Task #2 Output:
|
|
5
|
|
## Example Task #3 Input:
|
|
{"CONTEXT": "In Quebec, an allophone is a resident, usually an immigrant, whose mother tongue or home language is neither French nor English.", "QUESTION": "", "ANSWER": "In Quebec, an allophone is a resident, usually an immigrant, whose mother tongue or home language is not French."}
|
|
## Example Task #3 Output:
|
|
5
|
|
|
|
## Actual Task Input:
|
|
{"CONTEXT": {{context}}, "QUESTION": "", "ANSWER": {{answer}}}
|
|
Reminder: The return values for each task should be correctly formatted as an integer between 1 and 5. Do not repeat the context and question.
|
|
Actual Task Output: |