chore: import upstream snapshot with attribution
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,25 @@
|
||||
# Basic Eval
|
||||
Basic evaluator prompt for QA scenario
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install `promptflow-devkit`:
|
||||
```bash
|
||||
pip install promptflow-devkit
|
||||
```
|
||||
|
||||
## Run prompty
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
- Setup environment variables
|
||||
|
||||
Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
|
||||
|
||||
```bash
|
||||
cat ../.env
|
||||
```
|
||||
|
||||
- Test flow
|
||||
```bash
|
||||
pf flow test --flow eval.prompty --env --inputs sample.json
|
||||
```
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: basic evaluate
|
||||
description: basic evaluator for QA scenario
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
api_key: ${env:AZURE_OPENAI_API_KEY}
|
||||
azure_endpoint: ${env:AZURE_OPENAI_ENDPOINT}
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
max_tokens: 200
|
||||
top_p: 1.0
|
||||
response_format:
|
||||
type: json_object
|
||||
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
answer:
|
||||
type: string
|
||||
ground_truth:
|
||||
type: string
|
||||
outputs:
|
||||
score:
|
||||
type: string
|
||||
explanation:
|
||||
type: string
|
||||
---
|
||||
system:
|
||||
You are an AI assistant.
|
||||
You task is to evaluate a score for the answer based on the ground_truth and original question.
|
||||
This score value should always be an integer between 1 and 5. So the score produced should be 1 or 2 or 3 or 4 or 5.
|
||||
The output should be valid JSON.
|
||||
|
||||
**Example**
|
||||
question: "What is the capital of France?"
|
||||
answer: "Paris"
|
||||
ground_truth: "Paris"
|
||||
output:
|
||||
{"score": "5", "explanation":"paris is the capital of France"}
|
||||
|
||||
user:
|
||||
question: {{question}}
|
||||
answer: {{answer}}
|
||||
statement: {{statement}}
|
||||
output:
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"question": "what's the capital of China?",
|
||||
"answer": "Shanghai",
|
||||
"ground_truth": "Beijing"
|
||||
}
|
||||
Reference in New Issue
Block a user