chore: import upstream snapshot with attribution
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,2 @@
|
||||
AZURE_OPENAI_API_KEY=<your_AOAI_key>
|
||||
AZURE_OPENAI_ENDPOINT=<your_AOAI_endpoint>
|
||||
@@ -0,0 +1,28 @@
|
||||
# Flex Flow
|
||||
|
||||
You can learn more on flex flow with examples in this folder.
|
||||
|
||||
## SDK examples
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [flex-flow-quickstart.ipynb](./basic/flex-flow-quickstart.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_basic_flexflowquickstart.yml) | A quickstart tutorial to run a flex flow and evaluate it. |
|
||||
| [flex-flow-quickstart-azure.ipynb](./basic/flex-flow-quickstart-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_basic_flexflowquickstartazure.yml) | A quickstart tutorial to run a flex flow and evaluate it in azure. |
|
||||
| [chat-with-class-based-flow-azure.ipynb](./chat-basic/chat-with-class-based-flow-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflowazure.yml) | A quickstart tutorial to run a class based flex flow and evaluate it in azure. |
|
||||
| [chat-with-class-based-flow.ipynb](./chat-basic/chat-with-class-based-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflow.yml) | A quickstart tutorial to run a class based flex flow and evaluate it. |
|
||||
| [chat-stream-with-flex-flow.ipynb](./chat-stream/chat-stream-with-flex-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatstream_chatstreamwithflexflow.yml) | A quickstart tutorial to run a class based flex flow in stream mode and evaluate it. |
|
||||
| [chat-stream-with-async-flex-flow.ipynb](./chat-async-stream/chat-stream-with-async-flex-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatasyncstream_chatstreamwithasyncflexflow.yml) | A quickstart tutorial to run a class based flex flow in stream mode and evaluate it. |
|
||||
| [langchain-eval.ipynb](./eval-criteria-with-langchain/langchain-eval.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_evalcriteriawithlangchain_langchaineval.yml) | A tutorial to converting LangChain criteria evaluator application to flex flow. |
|
||||
|
||||
## CLI examples
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [basic](./basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_basic.yml) | A basic standard flow define using function entry that calls Azure OpenAI with connection info stored in environment variables |
|
||||
| [chat-async-stream](./chat-async-stream/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_async_stream.yml) | A chat flow defined using async class entry that return output in stream mode |
|
||||
| [chat-basic](./chat-basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_basic.yml) | A basic chat flow defined using class entry |
|
||||
| [chat-minimal](./chat-minimal/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_minimal.yml) | A chat flow defined using function with minimal code |
|
||||
| [chat-stream](./chat-stream/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_stream.yml) | A chat flow defined using class entry that return output in stream mode |
|
||||
| [chat-with-functions](./chat-with-functions/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_with_functions.yml) | This flow covers how to use the LLM chat API in combination with external functions to extend the capabilities of GPT models |
|
||||
| [eval-checklist](./eval-checklist/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_checklist.yml) | A example flow defined using class entry which demos how to evaluate the answer pass user specified check list |
|
||||
| [eval-code-quality](./eval-code-quality/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_code_quality.yml) | A example flow defined using class based entry which leverages model config to evaluate the quality of code snippet |
|
||||
| [eval-criteria-with-langchain](./eval-criteria-with-langchain/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_criteria_with_langchain.yml) | A example flow of converting LangChain criteria evaluator application to flex flow |
|
||||
@@ -0,0 +1,105 @@
|
||||
# Basic standard flow
|
||||
A basic standard flow define using function entry that calls Azure OpenAI with connection info stored in environment variables.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup environment variables
|
||||
|
||||
Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
|
||||
|
||||
```bash
|
||||
cat ../.env
|
||||
```
|
||||
|
||||
- Run/Debug as normal Python file
|
||||
```bash
|
||||
python programmer.py
|
||||
```
|
||||
|
||||
- Test with flow entry
|
||||
```bash
|
||||
pf flow test --flow programmer:write_simple_program --inputs text="Java Hello World!"
|
||||
```
|
||||
|
||||
- Test with flow yaml
|
||||
```bash
|
||||
# test with sample input value in flow.flex.yaml
|
||||
pf flow test --flow .
|
||||
```
|
||||
|
||||
```shell
|
||||
# test with UI
|
||||
pf flow test --flow . --ui
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
```bash
|
||||
# using environment from .env file (loaded in user code: hello.py)
|
||||
pf run create --flow . --data ./data.jsonl --column-mapping text='${data.text}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud with connection
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --data ./data.jsonl --column-mapping text='${data.text}' --environment-variables AZURE_OPENAI_API_KEY='${open_ai_connection.api_key}' AZURE_OPENAI_ENDPOINT='${open_ai_connection.api_base}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
```
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pfazure run list -r 3
|
||||
|
||||
# get a sample run name
|
||||
name=$(pfazure run list -r 100 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
|
||||
|
||||
# show specific run detail
|
||||
pfazure run show --name $name
|
||||
|
||||
# show output
|
||||
pfazure run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pfazure run visualize --name $name
|
||||
```
|
||||
@@ -0,0 +1,3 @@
|
||||
{"text": "Python Hello World!"}
|
||||
{"text": "C Hello World!"}
|
||||
{"text": "C# Hello World!"}
|
||||
@@ -0,0 +1,306 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Getting started with flex flow in Azure"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write an LLM application using a notebook and visualize the trace of your application.\n",
|
||||
"- Convert the application into a flow and batch-run it against multiple lines of data.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements-azure.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Connection to workspace"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Configure credential\n",
|
||||
"\n",
|
||||
"We are using `DefaultAzureCredential` to access the workspace. \n",
|
||||
"`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
|
||||
"\n",
|
||||
"Reference for other credentials if this does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" credential = DefaultAzureCredential()\n",
|
||||
" # Check if given credential can get token successfully.\n",
|
||||
" credential.get_token(\"https://management.azure.com/.default\")\n",
|
||||
"except Exception as ex:\n",
|
||||
" # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential does not work\n",
|
||||
" credential = InteractiveBrowserCredential()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Connect to the workspace\n",
|
||||
"\n",
|
||||
"We use a config file to connect to a workspace. The Azure ML workspace should be configured with a computer cluster. [Check this notebook for how to configure a workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.azure import PFClient\n",
|
||||
"\n",
|
||||
"# Connect to the workspace\n",
|
||||
"pf = PFClient.from_config(credential=credential)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create necessary connections\n",
|
||||
"A connection helps securely store and manage secret keys or other sensitive credentials required for interacting with the LLM and other external tools, for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"In this notebook, we will use the `basic` & `eval-code-quality` flex flow, which uses the connection `open_ai_connection`. We need to set up the connection if we haven't added it before.\n",
|
||||
"\n",
|
||||
"To prepare your Azure OpenAI resource, follow these [instructions](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
|
||||
"\n",
|
||||
"Go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
|
||||
"Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as a flow with multi-line data.\n",
|
||||
"\n",
|
||||
"Create a `flow.flex.yaml` file to define a flow whose entry points to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Show the flow.flex.yaml content\n",
|
||||
"with open(\"flow.flex.yaml\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"flow = \".\" # Path to the flow directory\n",
|
||||
"data = \"./data.jsonl\" # Path to the data file\n",
|
||||
"\n",
|
||||
"# Create a run with the flow and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"text\": \"${data.text}\",\n",
|
||||
" },\n",
|
||||
" environment_variables={\n",
|
||||
" \"AZURE_OPENAI_API_KEY\": \"${open_ai_connection.api_key}\",\n",
|
||||
" \"AZURE_OPENAI_ENDPOINT\": \"${open_ai_connection.api_base}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually use an LLM to verify the produced output matches the expected output. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup model configuration with connection\n",
|
||||
"\n",
|
||||
"When using Promptflow in Azure, create a model configuration object with connection name. \n",
|
||||
"The model config will connect to the cloud-hosted Promptflow instance while running the flow."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"model_config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\",\n",
|
||||
" azure_deployment=\"gpt-4o\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Evaluate the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input. The evaluation takes the outputs of that **base_run**, and uses an LLM to compare them to your desired outputs, and then visualizes the results."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_flow = \"../eval-code-quality/flow.flex.yaml\"\n",
|
||||
"\n",
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" init={\"model_config\": model_config},\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify the base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"code\": \"${run.outputs.output}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"You've successfully run your first flex flow and evaluated it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate the next message."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "azure",
|
||||
"section": "Flow",
|
||||
"weight": 10
|
||||
},
|
||||
"description": "A quickstart tutorial to run a flex flow and evaluate it in Azure.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements-azure.txt, examples/flex-flows/basic, examples/flex-flows/eval-code-quality"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,385 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Getting started with flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using notebook and visualize the trace of your application.\n",
|
||||
"- Convert the application into a flow and batch run against multi lines of data.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python function that calls OpenAI API. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"llm.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note: before running below cell, please configure required environment variable `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` by create an `.env` file. Please refer to `../.env.example` as an template."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# control the AOAI deployment (model) used in this example\n",
|
||||
"deployment_name = \"gpt-4o\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llm import my_llm_tool\n",
|
||||
"\n",
|
||||
"# pls configure `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` environment variables first\n",
|
||||
"result = my_llm_tool(\n",
|
||||
" prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
|
||||
" deployment_name=deployment_name,\n",
|
||||
")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"# rerun the function, which will be recorded in the trace\n",
|
||||
"result = my_llm_tool(\n",
|
||||
" prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
|
||||
" deployment_name=deployment_name,\n",
|
||||
")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now, let's add another layer of function call. In `programmer.py` there is a function called `write_simple_program`, which calls a new function called `load_prompt` and previous `my_llm_tool` function."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# show the programmer.py content\n",
|
||||
"with open(\"programmer.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# call the flow entry function\n",
|
||||
"from programmer import write_simple_program\n",
|
||||
"\n",
|
||||
"result = write_simple_program(\"Java Hello, world!\")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup model configuration with environment variables\n",
|
||||
"\n",
|
||||
"When used in local, create a model configuration object with environment variables."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"\n",
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
|
||||
" # load environment variables from .env file\n",
|
||||
" load_dotenv()\n",
|
||||
"\n",
|
||||
"if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
|
||||
" raise Exception(\"Please specify environment variables: AZURE_OPENAI_API_KEY\")\n",
|
||||
"model_config = AzureOpenAIModelConfiguration(\n",
|
||||
" azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
|
||||
" api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
|
||||
" azure_deployment=deployment_name,\n",
|
||||
" api_version=\"2023-07-01-preview\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from code_quality import CodeEvaluator\n",
|
||||
"\n",
|
||||
"evaluator = CodeEvaluator(model_config=model_config)\n",
|
||||
"eval_result = evaluator(result)\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n",
|
||||
"\n",
|
||||
"Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# show the flow.flex.yaml content\n",
|
||||
"with open(\"flow.flex.yaml\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=write_simple_program,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"text\": \"${data.text}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# we can also run flow pointing to yaml file\n",
|
||||
"eval_flow = \"../eval-code-quality/flow.flex.yaml\"\n",
|
||||
"\n",
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" init={\"model_config\": model_config},\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"code\": \"${run.outputs.output}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 10
|
||||
},
|
||||
"description": "A quickstart tutorial to run a flex flow and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/basic"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/flow.schema.json
|
||||
entry: programmer:write_simple_program
|
||||
environment:
|
||||
python_requirements_txt: requirements.txt
|
||||
sample:
|
||||
inputs:
|
||||
text: Java Hello World!
|
||||
@@ -0,0 +1,3 @@
|
||||
system:
|
||||
Write a simple {{text}} program.
|
||||
Output code only.
|
||||
@@ -0,0 +1,64 @@
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from openai.version import VERSION as OPENAI_VERSION
|
||||
|
||||
from promptflow.tracing import trace
|
||||
|
||||
|
||||
def get_client():
|
||||
if OPENAI_VERSION.startswith("0."):
|
||||
raise Exception(
|
||||
"Please upgrade your OpenAI package to version >= 1.0.0 or using the command: pip install --upgrade openai."
|
||||
)
|
||||
api_key = os.environ.get("OPENAI_API_KEY", None)
|
||||
if api_key:
|
||||
from openai import OpenAI
|
||||
|
||||
return OpenAI()
|
||||
else:
|
||||
from openai import AzureOpenAI
|
||||
|
||||
return AzureOpenAI(
|
||||
api_version=os.environ.get("OPENAI_API_VERSION", "2023-07-01-preview")
|
||||
)
|
||||
|
||||
|
||||
@trace
|
||||
def my_llm_tool(
|
||||
prompt: str,
|
||||
# for AOAI, deployment name is customized by user, not model name.
|
||||
deployment_name: str,
|
||||
max_tokens: int = 120,
|
||||
temperature: float = 1.0,
|
||||
top_p: float = 1.0,
|
||||
n: int = 1,
|
||||
) -> str:
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
# load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
raise Exception(
|
||||
"Please specify environment variables: OPENAI_API_KEY or AZURE_OPENAI_API_KEY"
|
||||
)
|
||||
messages = [{"content": prompt, "role": "system"}]
|
||||
response = get_client().chat.completions.create(
|
||||
messages=messages,
|
||||
model=deployment_name,
|
||||
max_tokens=int(max_tokens),
|
||||
temperature=float(temperature),
|
||||
top_p=float(top_p),
|
||||
n=int(n),
|
||||
)
|
||||
|
||||
# get first element because prompt is single.
|
||||
return response.choices[0].message.content
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
result = my_llm_tool(
|
||||
prompt="Write a simple Hello, world! program that displays the greeting message.",
|
||||
deployment_name="gpt-4o",
|
||||
)
|
||||
print(result)
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation code quality module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-code-quality")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1,41 @@
|
||||
from pathlib import Path
|
||||
from typing import TypedDict
|
||||
|
||||
from jinja2 import Template
|
||||
from llm import my_llm_tool
|
||||
|
||||
from promptflow.tracing import trace
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
class Result(TypedDict):
|
||||
output: str
|
||||
|
||||
|
||||
@trace
|
||||
def load_prompt(jinja2_template: str, text: str) -> str:
|
||||
"""Load prompt function."""
|
||||
with open(BASE_DIR / jinja2_template, "r", encoding="utf-8") as f:
|
||||
prompt = Template(
|
||||
f.read(), trim_blocks=True, keep_trailing_newline=True
|
||||
).render(text=text)
|
||||
return prompt
|
||||
|
||||
|
||||
@trace
|
||||
def write_simple_program(
|
||||
text: str = "Hello World!", deployment_name="gpt-4o"
|
||||
) -> Result:
|
||||
"""Ask LLM to write a simple program."""
|
||||
prompt = load_prompt("hello.jinja2", text)
|
||||
output = my_llm_tool(prompt=prompt, deployment_name=deployment_name, max_tokens=120)
|
||||
return Result(output=output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
result = write_simple_program("Hello, world!", "gpt-4o")
|
||||
print(result)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow-azure
|
||||
@@ -0,0 +1,3 @@
|
||||
promptflow[azure]
|
||||
python-dotenv
|
||||
openai>=1.14.0
|
||||
@@ -0,0 +1,10 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
column_mapping:
|
||||
text: ${data.text}
|
||||
environment_variables:
|
||||
# environment variables from connection
|
||||
AZURE_OPENAI_API_KEY: ${open_ai_connection.api_key}
|
||||
AZURE_OPENAI_ENDPOINT: ${open_ai_connection.api_base}
|
||||
AZURE_OPENAI_API_TYPE: azure
|
||||
@@ -0,0 +1,121 @@
|
||||
# Chat stream async
|
||||
A chat flow defined using async class entry that return output in stream mode. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn:
|
||||
- how to compose a chat flow that return output in stream mode.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
- how to consume chat history in prompt.
|
||||
```jinja
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:ChatFlow --init init.json
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
# run chat flow with default question in flow.flex.yaml
|
||||
pf flow test --flow . --init init.json
|
||||
|
||||
# run chat flow with new question
|
||||
pf flow test --flow . --init init.json --inputs question="What's Azure Machine Learning?"
|
||||
|
||||
pf flow test --flow . --init init.json --inputs question="What is ChatGPT? Please explain with consise statement."
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_async_stream_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init ./init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
@@ -0,0 +1,380 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Stream chat with async flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using class based flex flow.\n",
|
||||
"- Use AzureOpenAIModelConfiguration as class init parameter.\n",
|
||||
"- Use prompty to stream completions.\n",
|
||||
"- Convert the application into a async flow and batch run against multi lines of data.\n",
|
||||
"- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python program, which leverage prompty."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"flow.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When `stream=true` is configured in the parameters of a prompt whose output format is text, promptflow sdk will return a generator type, which item is the content of each chunk.\n",
|
||||
"\n",
|
||||
"Reference openai doc on how to do it using plain python code: [how_to_stream_completions](https://cookbook.openai.com/examples/how_to_stream_completions)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"chat.prompty\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
|
||||
"\n",
|
||||
"# client can help manage your runs and connections.\n",
|
||||
"pf = PFClient()\n",
|
||||
"try:\n",
|
||||
" conn_name = \"open_ai_connection\"\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
|
||||
" connection = AzureOpenAIConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" api_key=\"<your_AOAI_key>\",\n",
|
||||
" api_base=\"<your_AOAI_endpoint>\",\n",
|
||||
" api_type=\"azure\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # use this if you have an existing OpenAI account\n",
|
||||
" # connection = OpenAIConnection(\n",
|
||||
" # name=conn_name,\n",
|
||||
" # api_key=\"<user-input>\",\n",
|
||||
" # )\n",
|
||||
"\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"from flow import ChatFlow\n",
|
||||
"\n",
|
||||
"# create a chatFlow obj with connection\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")\n",
|
||||
"chat_flow = ChatFlow(config)\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"\n",
|
||||
"# run the flow as function, which will be recorded in the trace\n",
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with detailed statement\")\n",
|
||||
"# note the type is async generator object as we enabled stream in prompty\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import asyncio\n",
|
||||
"\n",
|
||||
"# print result in stream manner\n",
|
||||
"async for output in result:\n",
|
||||
" print(output, end=\"\")\n",
|
||||
" await asyncio.sleep(0.01)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
|
||||
"\n",
|
||||
"answer = \"\"\n",
|
||||
"async for output in result:\n",
|
||||
" answer += output\n",
|
||||
"answer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from check_list import EvalFlow\n",
|
||||
"\n",
|
||||
"eval_flow = EvalFlow(config)\n",
|
||||
"# evaluate answer agains a set of statement\n",
|
||||
"eval_result = eval_flow(\n",
|
||||
" answer=answer,\n",
|
||||
" statements={\n",
|
||||
" \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
|
||||
" \"consise\": \"It is a consise statement.\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=chat_flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 12
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow in stream mode and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: Stream Chat
|
||||
description: Chat with stream enabled.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
stream: true
|
||||
max_tokens: 1024
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"question": "What is Prompt flow?", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
@@ -0,0 +1 @@
|
||||
entry: flow:ChatFlow
|
||||
@@ -0,0 +1,70 @@
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration, Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
def log(message: str):
|
||||
verbose = os.environ.get("VERBOSE", "false")
|
||||
if verbose.lower() == "true":
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
class ChatFlow:
|
||||
def __init__(
|
||||
self, model_config: AzureOpenAIModelConfiguration, max_total_token=1100
|
||||
):
|
||||
self.model_config = model_config
|
||||
self.max_total_token = max_total_token
|
||||
|
||||
@trace
|
||||
async def __call__(
|
||||
self, question: str = "What is ChatGPT?", chat_history: list = None
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "chat.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > self.max_total_token:
|
||||
chat_history = chat_history[1:]
|
||||
log(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
# output is a generator of string as prompty enabled stream parameter
|
||||
for output in prompty(question=question, chat_history=chat_history):
|
||||
yield output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = ChatFlow(model_config=config)
|
||||
result = flow("What's Azure Machine Learning?", [])
|
||||
|
||||
# print result in stream manner
|
||||
async def consume_result():
|
||||
async for output in result:
|
||||
print(output, end="")
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
asyncio.run(consume_result())
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
connection: open_ai_connection
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1 @@
|
||||
{"question": "What is Prompt flow?"}
|
||||
@@ -0,0 +1,125 @@
|
||||
# Basic chat
|
||||
A basic chat flow defined using class entry. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn
|
||||
- how to compose a chat flow.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
- how to consume chat history in prompt.
|
||||
```jinja
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of prompty supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:ChatFlow --init init.json --inputs question="What's Azure Machine Learning?"
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
# run chat flow with default question in flow.flex.yaml
|
||||
pf flow test --flow .
|
||||
# run chat flow with new question
|
||||
pf flow test --flow . --inputs question="What is ChatGPT? Please explain with consise statement."
|
||||
# run chat flow with specific init and inputs
|
||||
pf flow test --flow . --init init.json --inputs sample.json
|
||||
```
|
||||
|
||||
- Test flow: multi turn
|
||||
```shell
|
||||
# start test in chat UI
|
||||
pf flow test --flow . --ui --init init.json
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_basic_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --init init.json --stream
|
||||
@@ -0,0 +1,295 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Chat with class based flex flow in Azure"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Submit batch run with a flow defined with python class and evaluate it in azure.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements-azure.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Connection to workspace"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Configure credential\n",
|
||||
"\n",
|
||||
"We are using `DefaultAzureCredential` to get access to workspace. \n",
|
||||
"`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
|
||||
"\n",
|
||||
"Reference for more available credentials if it does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" credential = DefaultAzureCredential()\n",
|
||||
" # Check if given credential can get token successfully.\n",
|
||||
" credential.get_token(\"https://management.azure.com/.default\")\n",
|
||||
"except Exception as ex:\n",
|
||||
" # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
|
||||
" credential = InteractiveBrowserCredential()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Get a handle to the workspace\n",
|
||||
"\n",
|
||||
"We use config file to connect to a workspace. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.azure import PFClient\n",
|
||||
"\n",
|
||||
"# Get a handle to workspace\n",
|
||||
"pf = PFClient.from_config(credential=credential)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"In this notebook, we will use flow `basic` flex flow which uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
|
||||
"\n",
|
||||
"Please go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
|
||||
"Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n",
|
||||
"\n",
|
||||
"Create a `flow.flex.yaml` file to define a flow which entry pointing to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# show the flow.flex.yaml content\n",
|
||||
"with open(\"flow.flex.yaml\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"# create the model config to be used in below flow calls\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"flow = \".\" # path to the flow directory\n",
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"\n",
|
||||
"# create run with the flow and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=flow,\n",
|
||||
" init={\n",
|
||||
" \"model_config\": config,\n",
|
||||
" },\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_flow = \"../eval-checklist/flow.flex.yaml\"\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")\n",
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" init={\n",
|
||||
" \"model_config\": config,\n",
|
||||
" },\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "azure",
|
||||
"section": "Flow",
|
||||
"weight": 11
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow and evaluate it in azure.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements-azure.txt, examples/flex-flows/basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,339 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Chat with class based flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using class based flex flow.\n",
|
||||
"- Use AzureOpenAIConnection as class init parameter.\n",
|
||||
"- Convert the application into a flow and batch run against multi lines of data.\n",
|
||||
"- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python program, which leverage promptflow built-in aoai tool. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"flow.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
|
||||
"\n",
|
||||
"# client can help manage your runs and connections.\n",
|
||||
"pf = PFClient()\n",
|
||||
"try:\n",
|
||||
" conn_name = \"open_ai_connection\"\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
|
||||
" connection = AzureOpenAIConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" api_key=\"<your_AOAI_key>\",\n",
|
||||
" api_base=\"<your_AOAI_endpoint>\",\n",
|
||||
" api_type=\"azure\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # use this if you have an existing OpenAI account\n",
|
||||
" # connection = OpenAIConnection(\n",
|
||||
" # name=conn_name,\n",
|
||||
" # api_key=\"<user-input>\",\n",
|
||||
" # )\n",
|
||||
"\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"# create the model config to be used in below flow calls\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from flow import ChatFlow\n",
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"\n",
|
||||
"# create a chatFlow obj with connection\n",
|
||||
"chat_flow = ChatFlow(config)\n",
|
||||
"# run the flow as function, which will be recorded in the trace\n",
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from check_list import EvalFlow\n",
|
||||
"\n",
|
||||
"eval_flow = EvalFlow(config)\n",
|
||||
"# evaluate answer agains a set of statement\n",
|
||||
"eval_result = eval_flow(\n",
|
||||
" answer=result,\n",
|
||||
" statements={\n",
|
||||
" \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
|
||||
" \"consise\": \"It is a consise statement.\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=chat_flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 11
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: Basic Chat
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
max_tokens: 1024
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
chat_history: []
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"question": "What is Prompt flow?", "chat_history":[], "statements": {"correctness": "should explain what's 'Prompt flow'", "consise": "It is a consise statement."}}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement", "chat_history":[], "statements": { "correctness": "should explain what's ChatGPT", "consise": "It is a consise statement."}}
|
||||
{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 2", "consise": "It is a consise statement."}}
|
||||
@@ -0,0 +1,13 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: flow:ChatFlow
|
||||
sample:
|
||||
inputs:
|
||||
question: What's Azure Machine Learning?
|
||||
init:
|
||||
model_config:
|
||||
connection: open_ai_connection
|
||||
azure_deployment: gpt-4o
|
||||
max_total_token: 1024
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,66 @@
|
||||
import os
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration, Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
def log(message: str):
|
||||
verbose = os.environ.get("VERBOSE", "false")
|
||||
if verbose.lower() == "true":
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
class ChatFlow:
|
||||
def __init__(
|
||||
self, model_config: AzureOpenAIModelConfiguration, max_total_token=4096
|
||||
):
|
||||
self.model_config = model_config
|
||||
self.max_total_token = max_total_token
|
||||
|
||||
@trace
|
||||
def __call__(
|
||||
self,
|
||||
question: str = "What's Azure Machine Learning?",
|
||||
chat_history: list = None,
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "chat.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > self.max_total_token:
|
||||
chat_history = chat_history[1:]
|
||||
log(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
# output is a string
|
||||
output = prompty(question=question, chat_history=chat_history)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = ChatFlow(config)
|
||||
result = flow("What's Azure Machine Learning?", [])
|
||||
print(result)
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
},
|
||||
"max_total_token": 2048
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow-azure
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]>=1.11.0
|
||||
@@ -0,0 +1,9 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
model_config:
|
||||
connection: open_ai_connection
|
||||
azure_deployment: gpt-4o
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"question": "How many questions did User ask?",
|
||||
"chat_history": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "where is the nearest coffee shop?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "I'm sorry, I don't know that. Would you like me to look it up for you?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what is the capital of France?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Paris"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
# Minimal chat
|
||||
A chat flow defined using function with minimal code. It demonstrates the minimal code to have a chat flow.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn
|
||||
- how to compose a chat flow.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup environment variables
|
||||
|
||||
Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
|
||||
|
||||
```bash
|
||||
cat ../.env
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:chat --inputs question="What's the capital of France?"
|
||||
```
|
||||
|
||||
- Test flow: multi turn
|
||||
```shell
|
||||
# start test in chat ui
|
||||
pf flow test --flow flow:chat --ui
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow flow:chat --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_minimal_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: Minimal Chat
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
max_tokens: 1024
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"question": "What is Prompt flow?"}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement"}
|
||||
@@ -0,0 +1,8 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: flow:chat
|
||||
sample:
|
||||
inputs:
|
||||
question: What's Azure Machine Learning?
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,30 @@
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from pathlib import Path
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
@trace
|
||||
def chat(question: str = "What's the capital of France?") -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
# load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
prompty = Prompty.load(source=BASE_DIR / "chat.prompty")
|
||||
output = prompty(question=question)
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
|
||||
result = chat("What's the capital of France?")
|
||||
print(result)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]>=1.11.0
|
||||
@@ -0,0 +1,10 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
environment_variables:
|
||||
# environment variables from connection
|
||||
AZURE_OPENAI_API_KEY: ${open_ai_connection.api_key}
|
||||
AZURE_OPENAI_ENDPOINT: ${open_ai_connection.api_base}
|
||||
AZURE_OPENAI_API_TYPE: azure
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1,126 @@
|
||||
# Chat stream
|
||||
A chat flow defined using class entry that return output in stream mode. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn:
|
||||
- how to compose a chat flow that return output in stream mode.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
- how to consume chat history in prompt.
|
||||
```jinja
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:ChatFlow --init init.json
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
# run chat flow with default question in flow.flex.yaml
|
||||
pf flow test --flow . --init init.json
|
||||
|
||||
# run chat flow with new question
|
||||
pf flow test --flow . --init init.json --inputs question="What's Azure Machine Learning?"
|
||||
|
||||
pf flow test --flow . --init init.json --inputs question="What is ChatGPT? Please explain with consise statement."
|
||||
```
|
||||
|
||||
- Test flow with UI
|
||||
```shell
|
||||
pf flow test --flow . --init init.json --ui
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_stream_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init ./init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
@@ -0,0 +1,378 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Stream chat with flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using class based flex flow.\n",
|
||||
"- Use AzureOpenAIModelConfiguration as class init parameter.\n",
|
||||
"- Use prompty to stream completions.\n",
|
||||
"- Convert the application into a flow and batch run against multi lines of data.\n",
|
||||
"- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python program, which leverage prompty."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"flow.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When `stream=true` is configured in the parameters of a prompt whose output format is text, promptflow sdk will return a generator type, which item is the content of each chunk.\n",
|
||||
"\n",
|
||||
"Reference openai doc on how to do it using plain python code: [how_to_stream_completions](https://cookbook.openai.com/examples/how_to_stream_completions)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"chat.prompty\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
|
||||
"\n",
|
||||
"# client can help manage your runs and connections.\n",
|
||||
"pf = PFClient()\n",
|
||||
"try:\n",
|
||||
" conn_name = \"open_ai_connection\"\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
|
||||
" connection = AzureOpenAIConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" api_key=\"<your_AOAI_key>\",\n",
|
||||
" api_base=\"<your_AOAI_endpoint>\",\n",
|
||||
" api_type=\"azure\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # use this if you have an existing OpenAI account\n",
|
||||
" # connection = OpenAIConnection(\n",
|
||||
" # name=conn_name,\n",
|
||||
" # api_key=\"<user-input>\",\n",
|
||||
" # )\n",
|
||||
"\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"from flow import ChatFlow\n",
|
||||
"\n",
|
||||
"# create a chatFlow obj with connection\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")\n",
|
||||
"chat_flow = ChatFlow(config)\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"\n",
|
||||
"# run the flow as function, which will be recorded in the trace\n",
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with detailed statement\")\n",
|
||||
"# note the type is generator object as we enabled stream in prompty\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"\n",
|
||||
"# print result in stream manner\n",
|
||||
"for r in result:\n",
|
||||
" print(r, end=\"\")\n",
|
||||
" # For better animation effects\n",
|
||||
" time.sleep(0.01)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
|
||||
"answer = \"\".join(result)\n",
|
||||
"answer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from check_list import EvalFlow\n",
|
||||
"\n",
|
||||
"eval_flow = EvalFlow(config)\n",
|
||||
"# evaluate answer agains a set of statement\n",
|
||||
"eval_result = eval_flow(\n",
|
||||
" answer=answer,\n",
|
||||
" statements={\n",
|
||||
" \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
|
||||
" \"consise\": \"It is a consise statement.\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=chat_flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 12
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow in stream mode and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: Stream Chat
|
||||
description: Chat with stream enabled.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
stream: true
|
||||
max_tokens: 1024
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"question": "What is Prompt flow?", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
@@ -0,0 +1,5 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: flow:ChatFlow
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,69 @@
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration, Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
def log(message: str):
|
||||
verbose = os.environ.get("VERBOSE", "false")
|
||||
if verbose.lower() == "true":
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
class ChatFlow:
|
||||
def __init__(
|
||||
self, model_config: AzureOpenAIModelConfiguration, max_total_token=1100
|
||||
):
|
||||
self.model_config = model_config
|
||||
self.max_total_token = max_total_token
|
||||
|
||||
@trace
|
||||
def __call__(
|
||||
self, question: str = "What is ChatGPT?", chat_history: list = None
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "chat.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > self.max_total_token:
|
||||
chat_history = chat_history[1:]
|
||||
log(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
# output is a generator of string as prompty enabled stream parameter
|
||||
output = prompty(question=question, chat_history=chat_history)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = ChatFlow(model_config=config)
|
||||
result = flow("What's Azure Machine Learning?", [])
|
||||
|
||||
# print result in stream manner
|
||||
for r in result:
|
||||
print(r, end="")
|
||||
# For better animation effects
|
||||
time.sleep(0.01)
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
connection: open_ai_connection
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1 @@
|
||||
{"question": "What is Prompt flow?"}
|
||||
@@ -0,0 +1,57 @@
|
||||
# Chat with external functions
|
||||
|
||||
This flow covers how to use the LLM chat API in combination with external functions to extend the capabilities of GPT models.
|
||||
|
||||
`tools` is an optional parameter in the Chat Completion API which can be used to provide function specifications. The purpose of this is to enable models to generate function arguments which adhere to the provided specifications. Note that the API will not actually execute any function calls. It is up to developers to execute function calls using model outputs.
|
||||
|
||||
Within the `tools` parameter, if the `functions` parameter is provided then by default the model will decide when it is appropriate to use one of the functions. The API can be forced to use a specific function by setting the `tool_choice` parameter to `{"type": "function", "function": {"name": "my_function"}}`. The API can also be forced to not use any function by setting the `tool_choice` parameter to `"none"`. If a function is used, the output will contain `"finish_reason": "tool_calls"` in the response, as well as a `tool_calls` object that has the name of the function and the generated function arguments.
|
||||
|
||||
You can refer to <a href='https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb' target='_blank'>openai sample</a> for more details.
|
||||
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn how to use external functions with LLM chat models and how to compose function role message in prompt template.
|
||||
|
||||
## Prerequisites
|
||||
Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Getting started
|
||||
|
||||
### 1 prepare api_key env
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup environment variables
|
||||
|
||||
Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
|
||||
|
||||
### 2 understand how chat model choose function
|
||||
```shell
|
||||
# see how llm will chose functions
|
||||
pf flow test --flow chat_with_tools.prompty --env --inputs question="What is the weather like in Boston?"
|
||||
pf flow test --flow chat_with_tools.prompty --env --inputs sample.json
|
||||
```
|
||||
|
||||
### 3 Start chatting with flow
|
||||
|
||||
```bash
|
||||
# run e2e chat flow with question
|
||||
pf flow test --flow flow:chat --env --inputs question="What is the weather like in Boston?"
|
||||
|
||||
# run chat flow with question & chat_history from sample.json
|
||||
pf flow test --flow flow:chat --env --inputs sample.json
|
||||
```
|
||||
|
||||
```shell
|
||||
# chat using ui
|
||||
pf flow test --flow flow:chat --env --ui
|
||||
```
|
||||
|
||||
## References
|
||||
- <a href='https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb' target='_blank'>OpenAI cookbook example</a>
|
||||
- <a href='https://openai.com/blog/function-calling-and-other-api-updates?ref=upstract.com' target='_blank'>OpenAI function calling announcement</a>
|
||||
- <a href='https://platform.openai.com/docs/guides/gpt/function-calling' target='_blank'>OpenAI function calling doc</a>
|
||||
- <a href='https://platform.openai.com/docs/api-reference/chat/create' target='_blank'>OpenAI function calling API</a>
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: Chat with Tools
|
||||
description: A basic prompt with tools
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
max_tokens: 128
|
||||
temperature: 0.2
|
||||
tools: ${file:tools.json}
|
||||
tool_choice: auto
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: What's the weather in Beijing?
|
||||
chat_history: []
|
||||
---
|
||||
# system:
|
||||
Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,103 @@
|
||||
import json
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from pathlib import Path
|
||||
from typing import TypedDict
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
class WeatherInfo(TypedDict):
|
||||
location: str
|
||||
temperature: float
|
||||
format: str
|
||||
forecast: list[str]
|
||||
num_days: int
|
||||
|
||||
|
||||
def get_current_weather(location, format="fahrenheit"):
|
||||
"""Get the current weather in a given location"""
|
||||
return WeatherInfo(
|
||||
location=location, temperature="72", format=format, forecast=["sunny", "windy"]
|
||||
)
|
||||
|
||||
|
||||
def get_n_day_weather_forecast(location, format, num_days):
|
||||
"""Get next num_days weather in a given location"""
|
||||
return WeatherInfo(
|
||||
location=location,
|
||||
temperature="60",
|
||||
format=format,
|
||||
forecast=["rainy"],
|
||||
num_days=num_days,
|
||||
)
|
||||
|
||||
|
||||
@trace
|
||||
def run_function(response_message: dict) -> str:
|
||||
if "tool_calls" in response_message and len(response_message["tool_calls"]) == 1:
|
||||
call = response_message["tool_calls"][0]
|
||||
function = call["function"]
|
||||
function_name = function["name"]
|
||||
function_args = json.loads(function["arguments"])
|
||||
print(function_args)
|
||||
result = globals()[function_name](**function_args)
|
||||
return str(result)
|
||||
|
||||
print("No function call")
|
||||
if isinstance(response_message, dict):
|
||||
result = response_message["content"]
|
||||
else:
|
||||
result = response_message
|
||||
return result
|
||||
|
||||
|
||||
MAX_TOTAL_TOKEN = 2048
|
||||
|
||||
|
||||
@trace
|
||||
def chat(
|
||||
question: str = "What's the weather of Beijing?",
|
||||
chat_history: list = None,
|
||||
max_total_token=2048,
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
# load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
prompty = Prompty.load(source=BASE_DIR / "chat_with_tools.prompty")
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > MAX_TOTAL_TOKEN:
|
||||
chat_history = chat_history[1:]
|
||||
print(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
output = prompty(question=question, chat_history=chat_history)
|
||||
|
||||
function_output = run_function(output)
|
||||
|
||||
return function_output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
|
||||
result = chat("What's the weather of Beijing?")
|
||||
print(result)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"chat_history": [],
|
||||
"question": "How about London next week?"
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
[
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_weather",
|
||||
"description": "Get the current weather",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state, e.g. San Francisco, CA"
|
||||
},
|
||||
"format": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "The temperature unit to use. Infer this from the users location."
|
||||
}
|
||||
},
|
||||
"required": ["location", "format"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_n_day_weather_forecast",
|
||||
"description": "Get an N-day weather forecast",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state, e.g. San Francisco, CA"
|
||||
},
|
||||
"format": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "The temperature unit to use. Infer this from the users location."
|
||||
},
|
||||
"num_days": {
|
||||
"type": "integer",
|
||||
"description": "The number of days to forecast"
|
||||
}
|
||||
},
|
||||
"required": ["location", "format", "num_days"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,89 @@
|
||||
# Eval Check List
|
||||
A example flow defined using class entry which demos how to evaluate the answer pass user specified check list.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python check_list.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
pf flow test --flow . --init init.json --inputs sample.json
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
```
|
||||
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("eval_checklist_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
@@ -0,0 +1,90 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import Prompty, AzureOpenAIModelConfiguration
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
@trace
|
||||
def check(answer: str, statement: str, model_config: AzureOpenAIModelConfiguration):
|
||||
"""Check the answer applies for the check statement."""
|
||||
examples = [
|
||||
{
|
||||
"answer": "ChatGPT is a conversational AI model developed by OpenAI.",
|
||||
"statement": "It contains a brief explanation of ChatGPT.",
|
||||
"score": 5,
|
||||
"explanation": "The statement is correct. The answer contains a brief explanation of ChatGPT.",
|
||||
}
|
||||
]
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "eval.prompty",
|
||||
model={"configuration": model_config},
|
||||
)
|
||||
output = prompty(examples=examples, answer=answer, statement=statement)
|
||||
output = json.loads(output)
|
||||
return output
|
||||
|
||||
|
||||
class EvalFlow:
|
||||
def __init__(self, model_config: AzureOpenAIModelConfiguration):
|
||||
self.model_config = model_config
|
||||
|
||||
def __call__(self, answer: str, statements: dict):
|
||||
"""Check the answer applies for a collection of check statement."""
|
||||
if isinstance(statements, str):
|
||||
statements = json.loads(statements)
|
||||
|
||||
results = {}
|
||||
for key, statement in statements.items():
|
||||
r = check(
|
||||
answer=answer, statement=statement, model_config=self.model_config
|
||||
)
|
||||
results[key] = r
|
||||
return results
|
||||
|
||||
def __aggregate__(self, line_results: list) -> dict:
|
||||
"""Aggregate the results."""
|
||||
total = len(line_results)
|
||||
avg_correctness = (
|
||||
sum(int(r["correctness"]["score"]) for r in line_results) / total
|
||||
)
|
||||
return {
|
||||
"average_correctness": avg_correctness,
|
||||
"total": total,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
|
||||
answer = """ChatGPT is a conversational AI model developed by OpenAI.
|
||||
It is based on the GPT-3 architecture and is designed to generate human-like responses to text inputs.
|
||||
ChatGPT is capable of understanding and responding to a wide range of topics and can be used for tasks such as
|
||||
answering questions, generating creative content, and providing assistance with various tasks.
|
||||
The model has been trained on a diverse range of internet text and is constantly being updated to improve its
|
||||
performance and capabilities. ChatGPT is available through the OpenAI API and can be accessed by developers and
|
||||
researchers to build applications and tools that leverage its capabilities."""
|
||||
statements = {
|
||||
"correctness": "It contains a detailed explanation of ChatGPT.",
|
||||
"consise": "It is a consise statement.",
|
||||
}
|
||||
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = EvalFlow(config)
|
||||
|
||||
result = flow(
|
||||
answer=answer,
|
||||
statements=statements,
|
||||
)
|
||||
print(result)
|
||||
|
||||
# run aggregation
|
||||
aggregation_result = flow.__aggregate__([result])
|
||||
print(aggregation_result)
|
||||
@@ -0,0 +1 @@
|
||||
{"answer": "ChatGPT is a conversational AI model developed by OpenAI. It is based on the GPT-3 architecture and is designed to generate human-like responses to text inputs. ChatGPT is capable of understanding and responding to a wide range of topics and can be used for tasks such as answering questions, generating creative content, and providing assistance with various tasks. The model has been trained on a diverse range of internet text and is constantly being updated to improve its performance and capabilities. ChatGPT is available through the OpenAI API and can be accessed by developers and researchers to build applications and tools that leverage its capabilities.", "statements": { "correctness": "It contains a detailed explanation of ChatGPT." }}
|
||||
@@ -0,0 +1,43 @@
|
||||
|
||||
---
|
||||
name: Evaluate based on a checklist
|
||||
description: Evaluate the quality of code snippet.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
max_tokens: 256
|
||||
temperature: 0.7
|
||||
|
||||
inputs:
|
||||
examples:
|
||||
type: list
|
||||
answer:
|
||||
type: string
|
||||
statement:
|
||||
type: string
|
||||
sample: ${file:sample.json}
|
||||
---
|
||||
|
||||
# system:
|
||||
You are an AI assistant.
|
||||
You task is to evaluate a score based on how the statement applies for the answer.
|
||||
Only accepts valid JSON format response without extra prefix or postfix.
|
||||
|
||||
# user:
|
||||
This score value should always be an integer between 1 and 5. So the score produced should be 1 or 2 or 3 or 4 or 5.
|
||||
|
||||
Here are a few examples:
|
||||
{% for ex in examples %}
|
||||
answer: {{ex.answer}}
|
||||
statement: {{ex.statement}}
|
||||
OUTPUT:
|
||||
{"score": "{{ex.score}}", "explanation":"{{ex.explanation}}"}
|
||||
{% endfor %}
|
||||
|
||||
For a given answer, valuate the answer based on how the statement applies for the answer:
|
||||
answer: {{answer}}
|
||||
statement: {{statement}}
|
||||
OUTPUT:
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
# flow is defined as python function
|
||||
entry: check_list:EvalFlow
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
promptflow>=1.11.0
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
connection: open_ai_connection
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"answer": "ChatGPT is a conversational AI model developed by OpenAI. It is based on the GPT-3 architecture and is designed to generate human-like responses to text inputs. ChatGPT is capable of understanding and responding to a wide range of topics and can be used for tasks such as answering questions, generating creative content, and providing assistance with various tasks. The model has been trained on a diverse range of internet text and is constantly being updated to improve its performance and capabilities. ChatGPT is available through the OpenAI API and can be accessed by developers and researchers to build applications and tools that leverage its capabilities.",
|
||||
"statements": { "correctness": "It contains a detailed explanation of ChatGPT." }
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
# Eval Code Quality
|
||||
A example flow defined using class based entry which leverages model config to evaluate the quality of code snippet.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
```bash
|
||||
python code_quality.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
```bash
|
||||
# correct
|
||||
pf flow test --flow . --inputs code='print(\"Hello, world!\")' --init init.json
|
||||
|
||||
# incorrect
|
||||
pf flow test --flow . --inputs code='printf("Hello, world!")' --init init.json
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
```
|
||||
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("eval_code_quality_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# show metrics
|
||||
pf run show-metrics --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
@@ -0,0 +1,69 @@
|
||||
import json
|
||||
|
||||
from typing import TypedDict
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Template
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration
|
||||
from promptflow.core._flow import Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
@trace
|
||||
def load_prompt(jinja2_template: str, code: str, examples: list) -> str:
|
||||
"""Load prompt function."""
|
||||
with open(BASE_DIR / jinja2_template, "r", encoding="utf-8") as f:
|
||||
tmpl = Template(f.read(), trim_blocks=True, keep_trailing_newline=True)
|
||||
prompt = tmpl.render(code=code, examples=examples)
|
||||
return prompt
|
||||
|
||||
|
||||
class Result(TypedDict):
|
||||
correctness: float
|
||||
readability: float
|
||||
explanation: str
|
||||
|
||||
|
||||
class CodeEvaluator:
|
||||
def __init__(self, model_config: AzureOpenAIModelConfiguration):
|
||||
self.model_config = model_config
|
||||
|
||||
def __call__(self, code: str) -> Result:
|
||||
"""Evaluate the code based on correctness, readability."""
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "eval_code_quality.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
output = prompty(code=code)
|
||||
output = json.loads(output)
|
||||
output = Result(**output)
|
||||
return output
|
||||
|
||||
def __aggregate__(self, line_results: list) -> dict:
|
||||
"""Aggregate the results."""
|
||||
total = len(line_results)
|
||||
avg_correctness = sum(int(r["correctness"]) for r in line_results) / total
|
||||
avg_readability = sum(int(r["readability"]) for r in line_results) / total
|
||||
return {
|
||||
"average_correctness": avg_correctness,
|
||||
"average_readability": avg_readability,
|
||||
"total": total,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
model_config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection",
|
||||
azure_deployment="gpt-4o",
|
||||
)
|
||||
evaluator = CodeEvaluator(model_config)
|
||||
result = evaluator('print("Hello, world!")')
|
||||
print(result)
|
||||
aggregate_result = evaluator.__aggregate__([result])
|
||||
print(aggregate_result)
|
||||
@@ -0,0 +1,2 @@
|
||||
{"code": "print(\"Hello, world!\")"}
|
||||
{"code": "printf(\"Hello, world!\")"}
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: Evaluate code quality
|
||||
description: Evaluate the quality of code snippet.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
inputs:
|
||||
code:
|
||||
type: string
|
||||
sample: ${file:sample.json}
|
||||
---
|
||||
# system:
|
||||
You are an AI assistant.
|
||||
You task is to evaluate the code based on correctness, readability.
|
||||
Only accepts valid JSON format response without extra prefix or postfix.
|
||||
|
||||
# user:
|
||||
This correctness value should always be an integer between 1 and 5. So the correctness produced should be 1 or 2 or 3 or 4 or 5.
|
||||
This readability value should always be an integer between 1 and 5. So the readability produced should be 1 or 2 or 3 or 4 or 5.
|
||||
|
||||
Here are a few examples:
|
||||
|
||||
**Example 1**
|
||||
Code: print(\"Hello, world!\")
|
||||
OUTPUT:
|
||||
{
|
||||
"correctness": 5,
|
||||
"readability": 5,
|
||||
"explanation": "The code is correct as it is a simple question and answer format. The readability is also good as the code is short and easy to understand."
|
||||
}
|
||||
|
||||
For a given code, valuate the code based on correctness, readability:
|
||||
Code: {{code}}
|
||||
OUTPUT:
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
# flow is defined as python function
|
||||
entry: code_quality:CodeEvaluator
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
promptflow
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"code": "print(\"Hello, world!\")"
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
# Eval Conciseness Criteria with LangChain
|
||||
|
||||
A example flow of converting LangChain criteria evaluator application to flex flow.
|
||||
Reference [here](https://python.langchain.com/docs/guides/productionization/evaluation/string/criteria_eval_chain/) for more information.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
- Or prepare your Anthropic resource follow this [instruction](https://python.langchain.com/docs/integrations/platforms/anthropic/) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ./connection.yml --set secrets.openai_api_key=<your_api_key> configs.azure_endpoint=<your_api_base> --name my_llm_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `my_llm_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name my_llm_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
```bash
|
||||
python eval_conciseness.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
```bash
|
||||
pf flow test --flow eval_conciseness:LangChainEvaluator --inputs input="What's 2+2?" prediction="What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four." --init custom_connection=my_llm_connection
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
```bash
|
||||
pf flow test --flow .
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --data ./data.jsonl --init custom_connection=my_llm_connection --stream
|
||||
```
|
||||
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("eval_criteria_with_langchain_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# show metrics
|
||||
pf run show-metrics --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/CustomConnection.schema.json
|
||||
name: my_llm_connection
|
||||
type: custom
|
||||
configs:
|
||||
azure_endpoint: "<to-be-replaced>"
|
||||
secrets: # required
|
||||
openai_api_key: "<to-be-replaced>"
|
||||
@@ -0,0 +1,2 @@
|
||||
{"input": "What's 2+2?", "prediction": "What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four."}
|
||||
{"input": "What's 2x2?", "prediction": "What's 2x2? That's an elementary question. The answer you're looking for is that two and two is four."}
|
||||
@@ -0,0 +1,70 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
from langchain.evaluation import load_evaluator
|
||||
from langchain_community.chat_models import AzureChatOpenAI, ChatAnthropic
|
||||
|
||||
from promptflow.client import PFClient
|
||||
from promptflow.connections import CustomConnection
|
||||
from promptflow.tracing import trace
|
||||
|
||||
|
||||
@dataclass
|
||||
class Result:
|
||||
reasoning: str
|
||||
value: str
|
||||
score: float
|
||||
|
||||
|
||||
class LangChainEvaluator:
|
||||
def __init__(self, custom_connection: CustomConnection):
|
||||
self.custom_connection = custom_connection
|
||||
|
||||
# create llm according to the secrets in custom connection
|
||||
if "anthropic_api_key" in self.custom_connection.secrets:
|
||||
self.llm = ChatAnthropic(
|
||||
temperature=0,
|
||||
anthropic_api_key=self.custom_connection.secrets["anthropic_api_key"],
|
||||
)
|
||||
elif "openai_api_key" in self.custom_connection.secrets:
|
||||
self.llm = AzureChatOpenAI(
|
||||
deployment_name="gpt-4o",
|
||||
openai_api_key=self.custom_connection.secrets["openai_api_key"],
|
||||
azure_endpoint=self.custom_connection.configs["azure_endpoint"],
|
||||
openai_api_type="azure",
|
||||
openai_api_version="2023-07-01-preview",
|
||||
temperature=0,
|
||||
)
|
||||
else:
|
||||
raise ValueError("No valid API key found in the connection.")
|
||||
# evaluate with langchain evaluator for conciseness
|
||||
self.evaluator = load_evaluator(
|
||||
"criteria", llm=self.llm, criteria="conciseness"
|
||||
)
|
||||
|
||||
@trace
|
||||
def __call__(
|
||||
self,
|
||||
input: str,
|
||||
prediction: str,
|
||||
) -> Result:
|
||||
"""Evaluate with langchain evaluator."""
|
||||
|
||||
eval_result = self.evaluator.evaluate_strings(
|
||||
prediction=prediction, input=input
|
||||
)
|
||||
return Result(**eval_result)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
pf = PFClient()
|
||||
connection = pf.connections.get(name="my_llm_connection")
|
||||
evaluator = LangChainEvaluator(custom_connection=connection)
|
||||
result = evaluator(
|
||||
prediction="What's 2+2? That's an elementary question. "
|
||||
"The answer you're looking for is that two and two is four.",
|
||||
input="What's 2+2?",
|
||||
)
|
||||
print(result)
|
||||
@@ -0,0 +1,11 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: eval_conciseness:LangChainEvaluator
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
sample:
|
||||
inputs:
|
||||
input: What's 2+2?
|
||||
prediction: What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.
|
||||
init:
|
||||
custom_connection: my_llm_connection
|
||||
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Evaluate with langchain's evaluator"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Convert LangChain criteria evaluator applications to `flex flow`.\n",
|
||||
"- Use `CustomConnection` to store secrets.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your langchain evaluator with prompt flow\n",
|
||||
"### Initialize a pf client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create a custom connection to protect your API key\n",
|
||||
"\n",
|
||||
"You can protect your API key in custom connection's secrets."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"\n",
|
||||
"from promptflow.entities import CustomConnection\n",
|
||||
"\n",
|
||||
"conn_name = \"my_llm_connection\"\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
|
||||
" # load environment variables from .env file\n",
|
||||
" load_dotenv()\n",
|
||||
"\n",
|
||||
" # put API key in secrets\n",
|
||||
" connection = CustomConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" configs={\n",
|
||||
" \"azure_endpoint\": os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
|
||||
" },\n",
|
||||
" secrets={\n",
|
||||
" # store API key\n",
|
||||
" # \"anthropic_api_key\": \"<your-api-key>\",\n",
|
||||
" \"openai_api_key\": os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
|
||||
" },\n",
|
||||
" )\n",
|
||||
" # Create the connection, note that all secret values will be scrubbed in the returned result\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Test the evaluator with trace"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from eval_conciseness import LangChainEvaluator\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"evaluator = LangChainEvaluator(custom_connection=conn)\n",
|
||||
"result = evaluator(\n",
|
||||
" prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
|
||||
" input=\"What's 2+2?\",\n",
|
||||
")\n",
|
||||
"print(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the evaluator with flow yaml\n",
|
||||
"Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/eval-criteria-with-langchain/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=\"./flow.flex.yaml\",\n",
|
||||
" # reference custom connection by name\n",
|
||||
" init={\n",
|
||||
" \"custom_connection\": \"my_llm_connection\",\n",
|
||||
" },\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"prediction\": \"${data.prediction}\",\n",
|
||||
" \"input\": \"${data.input}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run])"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 60
|
||||
},
|
||||
"description": "A tutorial to converting LangChain criteria evaluator application to flex flow.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/flex-flows/eval-criteria-with-langchain"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
promptflow
|
||||
langchain>=0.1.5
|
||||
python-dotenv
|
||||
langchain-community>=0.2.0
|
||||
Reference in New Issue
Block a user