wehub/microsoft--promptflow
1
0
Fork 0
Code Issues Pull Requests Actions 48 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Flake8 Lint / flake8 (push) Waiting to run
Details
Spell check CI / Spell_Check (push) Waiting to run
Details
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
Details
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Details
Publish Promptflow Doc / Build (push) Has been cancelled
Details
Publish Promptflow Doc / Deploy (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:52 +08:00
commit e768098d0e
4004 changed files with 2804145 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/tutorials/get-started/flow-as-function.ipynb
+225
View File
@@ -0,0 +1,225 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Execute flow as a function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"**Requirements** - In order to benefit from this tutorial, you will need:\n",
"- A python environment\n",
"- Installed prompt flow SDK\n",
"\n",
"**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
"- Execute a flow as a function\n",
"- Execute a flow function with in-memory connection object override\n",
"- Execute a flow function with fields override\n",
"- Execute a flow function with streaming output\n",
"\n",
"**Motivations** - This guide will walk you through the main scenarios of executing flow as a function. You will learn how to consume flow as a function in different scenarios for more pythonnic usage.\n",
"\n",
"\n",
"**Note**: the flow context configs may affect each other in some cases. For example, using `connection` & `overrides` to override same node. \n",
"The behavior is undefined for those scenarios. Pleas avoid such usage."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example1: Load flow as a function with inputs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from promptflow.client import load_flow\n",
"\n",
"\n",
"flow_path = \"../../flows/standard/web-classification\"\n",
"sample_url = \"https://www.youtube.com/watch?v=o5ZQyXaAv1g\"\n",
"\n",
"f = load_flow(source=flow_path)\n",
"result = f(url=sample_url)\n",
"\n",
"print(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example2: Load flow as a function with in-memory connection override"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You will need to have a connection named \"new_ai_connection\" to run flow with new connection."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"# provide parameters to create connection\n",
"\n",
"conn_name = \"new_ai_connection\"\n",
"api_key = \"<user-input>\"\n",
"api_base = \"<user-input>\"\n",
"api_version = \"<user-input>\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# create needed connection\n",
"import promptflow\n",
"from promptflow.entities import AzureOpenAIConnection, OpenAIConnection\n",
"\n",
"\n",
"# Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
"connection = AzureOpenAIConnection(\n",
" name=conn_name,\n",
" api_key=api_key,\n",
" api_base=api_base,\n",
" api_type=\"azure\",\n",
" api_version=api_version,\n",
")\n",
"\n",
"# use this if you have an existing OpenAI account\n",
"# connection = OpenAIConnection(\n",
"# name=conn_name,\n",
"# api_key=api_key,\n",
"# )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f = load_flow(\n",
" source=flow_path,\n",
")\n",
"# directly use connection created above\n",
"f.context.connections = {\"classify_with_llm\": {\"connection\": connection}}\n",
"\n",
"result = f(url=sample_url)\n",
"\n",
"print(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example 3: Local flow as a function with flow inputs override"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from promptflow.entities import FlowContext\n",
"\n",
"f = load_flow(source=flow_path)\n",
"f.context = FlowContext(\n",
" # node \"fetch_text_content_from_url\" will take inputs from the following command instead of from flow input\n",
" overrides={\"nodes.fetch_text_content_from_url.inputs.url\": sample_url},\n",
")\n",
"# the url=\"unknown\" will not take effect\n",
"result = f(url=\"unknown\")\n",
"print(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example 4: Load flow as a function with streaming output"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f = load_flow(source=\"../../flows/chat/chat-basic\")\n",
"f.context.streaming = True\n",
"result = f(\n",
" chat_history=[\n",
" {\n",
" \"inputs\": {\"chat_input\": \"Hi\"},\n",
" \"outputs\": {\"chat_output\": \"Hello! How can I assist you today?\"},\n",
" }\n",
" ],\n",
" question=\"How are you?\",\n",
")\n",
"\n",
"\n",
"answer = \"\"\n",
"# the result will be a generator, iterate it to get the result\n",
"for r in result[\"answer\"]:\n",
" answer += r\n",
"\n",
"print(answer)"
]
}
],
"metadata": {
"build_doc": {
"author": [
"D-W-@github.com",
"wangchao1230@github.com"
],
"category": "local",
"section": "Flow",
"weight": 40
},
"description": "This guide will walk you through the main scenarios of executing flow as a function.",
"kernelspec": {
"display_name": "github_v2",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
},
"resources": "examples/flows/standard/web-classification, examples/flows/chat/chat-basic"
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/tutorials/get-started/quickstart-azure.ipynb
+449
View File
@@ -0,0 +1,449 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Run DAG flow in Azure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"**Requirements** - In order to benefit from this tutorial, you will need:\n",
"- An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
"- An Azure ML workspace - [Configure workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)\n",
"- A python environment\n",
"- Installed prompt flow SDK\n",
"\n",
"**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
"- Connect to your Azure AI workspace from the Python SDK\n",
"- Create and develop a new promptflow run\n",
"- Evaluate the run with a evaluation flow\n",
"\n",
"**Motivations** - This guide will walk you through the main user journey of prompt flow code-first experience. You will learn how to create and develop your first prompt flow, test and evaluate it."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Install dependent packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install -r ../../requirements.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Connect to Azure Machine Learning Workspace\n",
"\n",
"The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
"\n",
"## 1.1 Import the required libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"\n",
"# Import required libraries\n",
"from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
"\n",
"# azure version promptflow apis\n",
"from promptflow.azure import PFClient"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1.2 Configure credential\n",
"\n",
"We are using `DefaultAzureCredential` to get access to workspace. \n",
"`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
"\n",
"Reference for more available credentials if it does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" credential = DefaultAzureCredential()\n",
" # Check if given credential can get token successfully.\n",
" credential.get_token(\"https://management.azure.com/.default\")\n",
"except Exception as ex:\n",
" # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
" credential = InteractiveBrowserCredential()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1.3 Get a handle to the workspace\n",
"\n",
"We use config file to connect to a workspace. The Azure ML workspace should be configured with computer cluster. [Check this notebook for configure a workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Get a handle to workspace\n",
"pf = PFClient.from_config(credential=credential)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1.4 Create necessary connections\n",
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
"\n",
"In this notebook, we will use flow `web-classification` which uses connection `azure_open_ai_connection` inside, we need to set up the connection if we haven't added it before.\n",
"\n",
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
"\n",
"Please go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
"Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Create a new run\n",
"\n",
"`web-classification` is a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Set flow path and input data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# load flow\n",
"flow = \"../../flows/standard/web-classification\"\n",
"data = \"../../flows/standard/web-classification/data.jsonl\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Submit run"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# create run\n",
"base_run = pf.run(\n",
" flow=flow,\n",
" data=data,\n",
" column_mapping={\n",
" \"url\": \"${data.url}\",\n",
" },\n",
")\n",
"print(base_run)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.stream(base_run)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(base_run)\n",
"details.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.visualize(base_run)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Evaluate your flow run result\n",
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score.\n",
"\n",
"In this notebook, we use `eval-classification-accuracy` flow to evaluate. This is a flow illustrating how to evaluate the performance of a classification system. It involves comparing each prediction to the groundtruth and assigns a \"Correct\" or \"Incorrect\" grade, and aggregating the results to produce metrics such as accuracy, which reflects how good the system is at classifying the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"eval_run = pf.run(\n",
" flow=\"../../flows/evaluation/eval-classification-accuracy\",\n",
" data=data,\n",
" run=base_run,\n",
" column_mapping={\n",
" \"groundtruth\": \"${data.answer}\",\n",
" \"prediction\": \"${run.outputs.category}\",\n",
" },\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.stream(eval_run)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(eval_run)\n",
"details.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"metrics = pf.get_metrics(eval_run)\n",
"print(json.dumps(metrics, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.visualize([base_run, eval_run])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create another run with different variant node\n",
"\n",
"In this example, `web-classification`'s node `summarize_text_content` has two variants: `variant_0` and `variant_1`. The difference between them is the inputs parameters:\n",
"\n",
"variant_0:\n",
"\n",
" - inputs:\n",
" - deployment_name: gpt-35-turbo\n",
" - max_tokens: '128'\n",
" - temperature: '0.2'\n",
" - text: ${fetch_text_content_from_url.output}\n",
"\n",
"variant_1:\n",
"\n",
" - inputs:\n",
" - deployment_name: gpt-35-turbo\n",
" - max_tokens: '256'\n",
" - temperature: '0.3'\n",
" - text: ${fetch_text_content_from_url.output}\n",
"\n",
"\n",
"You can check the whole flow definition at [flow.dag.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flows/standard/web-classification/flow.dag.yaml)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# use the variant1 of the summarize_text_content node.\n",
"variant_run = pf.run(\n",
" flow=flow,\n",
" data=data,\n",
" column_mapping={\n",
" \"url\": \"${data.url}\",\n",
" },\n",
" variant=\"${summarize_text_content.variant_1}\", # here we specify node \"summarize_text_content\" to use variant 1 version.\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.stream(variant_run)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(variant_run)\n",
"details.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run evaluation against variant run"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
"\n",
"eval_run_variant = pf.run(\n",
" flow=eval_flow,\n",
" data=\"../../flows/standard/web-classification/data.jsonl\", # path to the data file\n",
" run=variant_run, # use run as the variant\n",
" column_mapping={\n",
" # reference data\n",
" \"groundtruth\": \"${data.answer}\",\n",
" # reference the run's output\n",
" \"prediction\": \"${run.outputs.category}\",\n",
" },\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.stream(eval_run_variant)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(eval_run_variant)\n",
"details.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"metrics = pf.get_metrics(eval_run_variant)\n",
"print(json.dumps(metrics, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.visualize([eval_run, eval_run_variant])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Next Steps\n",
"\n",
"Learn more on how to:\n",
"- run the flow as a component in a azureml pipeline: [flow in pipeline](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/run-flow-with-pipeline/pipeline.ipynb)."
]
}
],
"metadata": {
"build_doc": {
"author": [
"wangchao1230@github.com"
],
"category": "azure",
"section": "Flow",
"weight": 20
},
"description": "A quickstart tutorial to run a flow in Azure AI and evaluate it.",
"kernelspec": {
"display_name": "promptflow",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
},
"resources": "examples/requirements.txt, examples/flows/standard/web-classification, examples/flows/evaluation/eval-classification-accuracy"
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/tutorials/get-started/quickstart.ipynb
+464
View File
@@ -0,0 +1,464 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Getting started with DAG flow"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"**Prerequisite** - To make the most of this tutorial, you'll need:\n",
"- A local clone of the prompt flow repository\n",
"- A Python environment with Jupyter Notebook support (such as Jupyter Lab or the Python extension for Visual Studio Code)\n",
"- Know how to program with Python :)\n",
"\n",
"_A basic understanding of Machine Learning can be beneficial, but it's not mandatory._\n",
"\n",
"\n",
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
"\n",
"- Run your first prompt flow sample\n",
"- Run your first evaluation\n",
"\n",
"\n",
"The sample used in this tutorial is the [web-classification](https://github.com/microsoft/promptflow/tree/main/examples/flows/standard/web-classification) flow, which categorizes URLs into several predefined classes. Classification is a traditional machine learning task, and this sample illustrates how to perform classification using GPT and prompts."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Install dependent packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install -r ../../requirements.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Create necessary connections\n",
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
"\n",
"In this notebook, we will use flow `web-classification` which uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
"\n",
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"from promptflow.client import PFClient\n",
"from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
"\n",
"# client can help manage your runs and connections.\n",
"pf = PFClient()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" conn_name = \"open_ai_connection\"\n",
" conn = pf.connections.get(name=conn_name)\n",
" print(\"using existing connection\")\n",
"except:\n",
" # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
" connection = AzureOpenAIConnection(\n",
" name=conn_name,\n",
" api_key=\"<test_key>\",\n",
" api_base=\"<test_base>\",\n",
" api_type=\"azure\",\n",
" api_version=\"<test_version>\",\n",
" )\n",
"\n",
" # use this if you have an existing OpenAI account\n",
" # connection = OpenAIConnection(\n",
" # name=conn_name,\n",
" # api_key=\"<user-input>\",\n",
" # )\n",
"\n",
" conn = pf.connections.create_or_update(connection)\n",
" print(\"successfully created connection\")\n",
"\n",
"print(conn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Run web-classification flow\n",
"\n",
"`web-classification` is a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Set flow path"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"flow = \"../../flows/standard/web-classification\" # path to the flow directory"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Quick test"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Test flow\n",
"flow_inputs = {\n",
" \"url\": \"https://play.google.com/store/apps/details?id=com.twitter.android\",\n",
"}\n",
"flow_result = pf.test(flow=flow, inputs=flow_inputs)\n",
"print(f\"Flow result: {flow_result}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Test single node in the flow\n",
"node_name = \"fetch_text_content_from_url\"\n",
"node_inputs = {\n",
" \"url\": \"https://play.google.com/store/apps/details?id=com.twitter.android\"\n",
"}\n",
"flow_result = pf.test(flow=flow, inputs=node_inputs, node=node_name)\n",
"print(f\"Node result: {flow_result}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Flow as a function\n",
"\n",
"We have also implemented a syntex sugar where you can consume a flow like a python function, with ability to override connections, inputs and other runtime configs.\n",
"Reference [here](./flow-as-function.ipynb) for more details."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from promptflow.client import load_flow\n",
"\n",
"flow_func = load_flow(flow)\n",
"flow_result = flow_func(**flow_inputs)\n",
"print(f\"Flow function result: {flow_result}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Batch run with a data file (with multiple lines of test data)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = \"../../flows/standard/web-classification/data.jsonl\" # path to the data file\n",
"\n",
"# create run with default variant\n",
"base_run = pf.run(\n",
" flow=flow,\n",
" data=data,\n",
" stream=True,\n",
" column_mapping={\n",
" \"url\": \"${data.url}\",\n",
" },\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(base_run)\n",
"details.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Evaluate your flow\n",
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score.\n",
"\n",
"In this notebook, we use `classification-accuracy-eval` flow to evaluate. This is a flow illustrating how to evaluate the performance of a classification system. It involves comparing each prediction to the groundtruth and assigns a \"Correct\" or \"Incorrect\" grade, and aggregating the results to produce metrics such as accuracy, which reflects how good the system is at classifying the data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run evaluation on the previous batch run\n",
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
"\n",
"eval_run = pf.run(\n",
" flow=eval_flow,\n",
" data=\"../../flows/standard/web-classification/data.jsonl\", # path to the data file\n",
" run=base_run, # specify base_run as the run you want to evaluate\n",
" column_mapping={\n",
" \"groundtruth\": \"${data.answer}\",\n",
" \"prediction\": \"${run.outputs.category}\",\n",
" }, # map the url field from the data to the url input of the flow\n",
" stream=True,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(eval_run)\n",
"details.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"metrics = pf.get_metrics(eval_run)\n",
"print(json.dumps(metrics, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.visualize([base_run, eval_run])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
"\n",
"You can check out the [web-classification](https://github.com/microsoft/promptflow/tree/main/examples/flows/standard/web-classification) flow and the [classification-accuracy](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation/eval-classification-accuracy) flow for more details, and start building your own flow.\n",
"\n",
"Or you can move on for a more advanced topic: experiment with a variant."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Another batch run with a variant\n",
"\n",
"[Variant](https://microsoft.github.io/promptflow/concepts/concept-variants.html) in prompt flow is to allow you do experimentation with LLMs. You can set a variant of Prompt/LLM node pointing to different prompt or use different LLM parameters like temperature.\n",
"\n",
"In this example, `web-classification`'s node `summarize_text_content` has two variants: `variant_0` and `variant_1`. The difference between them is the inputs parameters:\n",
"\n",
"variant_0:\n",
"\n",
" - inputs:\n",
" - deployment_name: gpt-35-turbo\n",
" - max_tokens: '128'\n",
" - temperature: '0.2'\n",
" - text: ${fetch_text_content_from_url.output}\n",
"\n",
"variant_1:\n",
"\n",
" - inputs:\n",
" - deployment_name: gpt-35-turbo\n",
" - max_tokens: '256'\n",
" - temperature: '0.3'\n",
" - text: ${fetch_text_content_from_url.output}\n",
"\n",
"\n",
"You can check the whole flow definition at [flow.dag.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flows/standard/web-classification/flow.dag.yaml)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# use the variant1 of the summarize_text_content node.\n",
"variant_run = pf.run(\n",
" flow=flow,\n",
" data=data,\n",
" variant=\"${summarize_text_content.variant_1}\", # here we specify node \"summarize_text_content\" to use variant 1 version.\n",
" column_mapping={\n",
" \"url\": \"${data.url}\",\n",
" },\n",
" stream=True,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(variant_run)\n",
"details.head(10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run evaluation on the variant run\n",
"So that later we can compare metrics and see which works better."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
"\n",
"eval_run_variant = pf.run(\n",
" flow=eval_flow,\n",
" data=\"../../flows/standard/web-classification/data.jsonl\", # path to the data file\n",
" run=variant_run, # use run as the variant\n",
" column_mapping={\n",
" \"groundtruth\": \"${data.answer}\",\n",
" \"prediction\": \"${run.outputs.category}\",\n",
" }, # map the url field from the data to the url input of the flow\n",
" stream=True,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"details = pf.get_details(eval_run_variant)\n",
"details.head(10)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"metrics = pf.get_metrics(eval_run_variant)\n",
"print(json.dumps(metrics, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pf.visualize([eval_run, eval_run_variant])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Next Steps\n",
"\n",
"Learn more on:\n",
"- [Manage connections](https://github.com/microsoft/promptflow/blob/main/examples/connections/connection.ipynb): how to manage the endpoints/secrets information to access external services including LLMs.\n",
"- [Chat with PDF](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/e2e-development/chat-with-pdf.md): go through an end-to-end tutorial on how to develop a chat application with prompt flow.\n",
"- [Deploy http endpoint](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/flow-deploy): how to deploy the flow as a local http endpoint.\n",
"- [Prompt flow in Azure AI](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/get-started/quickstart-azure.ipynb): run and evaluate flow in Azure AI where you can collaborate with team better."
]
}
],
"metadata": {
"build_doc": {
"author": [
"wangchao1230@github.com"
],
"category": "local",
"section": "Flow",
"weight": 30
},
"description": "A quickstart tutorial to run a flow and evaluate it.",
"kernelspec": {
"display_name": "prompt_flow",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
},
"resources": "examples/requirements.txt, examples/flows/standard/web-classification, examples/flows/evaluation/eval-classification-accuracy"
},
"nbformat": 4,
"nbformat_minor": 2
}