chore: import upstream snapshot with attribution
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
Flake8 Lint / flake8 (push) Has been cancelled
Spell check CI / Spell_Check (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,53 @@
|
||||
# Contributing to examples folder
|
||||
|
||||
Thank you for your interest in contributing to the examples folder. This folder contains a collection of Python notebooks and selected markdown files that demonstrate various usage of this promptflow project. The script will automatically generate a README.md file in the root folder, listing all the notebooks and markdown files with their corresponding workflows.
|
||||
|
||||
## Guidelines for notebooks and markdown files in examples folder
|
||||
|
||||
When creating or modifying a notebook or markdown file, please follow these guidelines:
|
||||
|
||||
- Each notebook or markdown file should have a clear and descriptive title as the first line
|
||||
- Each notebook or markdown file should have a brief introduction that explains the purpose and scope of the example. For details, please refer to the readme workflow generator manual [README.md](../scripts/readme/README.md) file.
|
||||
- The first sentence of first paragraph of the markdown file is important. The introduction should be concise and informative, and end with a period.
|
||||
- Each notebook file should have a metadata area when the file is opened as a big JSON file. The metadata area may contain the following fields:
|
||||
- `.metadata.description`: (Mandatory) A short description of the example that will be displayed in the README.md file. The description should be concise and informative, and end with a period.
|
||||
- `.metadata.stage`: (Optional) A value that indicates whether the script should skip generating a workflow for this notebook or markdown file. If set to `development`, the script will ignore this file. If set to other values or omitted, the script will generate a workflow for this file.
|
||||
- Each notebook or markdown file should have a clear and logical structure, using appropriate headings, subheadings, comments, and code cells. The code cells should be executable and produce meaningful outputs.
|
||||
- Each notebook or markdown file should follow the [PEP 8](https://peps.python.org/pep-0008/) style guide for Python code, and use consistent and readable variable names, indentation, spacing, and punctuation.
|
||||
- Each notebook or markdown file should include relevant references, citations, and acknowledgements.
|
||||
- If you are contributing to [tutorial](./tutorials/), each notebook or markdown file should declare its dependent resources in its metadata, so that the auto generated workflow can listen to the changes of these resources to avoid unexpected breaking. Resources should be declared with relative path to the repo root, and here are examples for [notebook](./tutorials/get-started/quickstart.ipynb) and [markdown](./tutorials/e2e-development/chat-with-pdf.md).
|
||||
|
||||
## Generate workflows, update README.md and submit pull requests
|
||||
|
||||
To run the readme.py script, you need to have Python 3 installed on your system. You also need to install the required packages by running:
|
||||
|
||||
```bash
|
||||
# At this examples folder
|
||||
pip install -r requirements.txt
|
||||
pip install -r dev_requirements.txt
|
||||
```
|
||||
|
||||
Then, you can run the script by:
|
||||
|
||||
```bash
|
||||
# At the root of this repository
|
||||
python scripts/readme/readme.py
|
||||
```
|
||||
|
||||
For detailed usage of readme.py, please refer to the readme workflow generator manual [README.md](../scripts/readme/README.md)
|
||||
|
||||
### Update [README.md](./README.md) in [examples](./) folder
|
||||
|
||||
The readme.py script will scan all the notebooks and markdown files in the examples folder, and generate a README.md file in the root folder. The README.md file will contain a table of contents with links to each notebook and markdown file, as well as their descriptions and workflows.
|
||||
|
||||
### Generations in the [workflows](../.github/workflows/) folder
|
||||
|
||||
This contains two parts:
|
||||
* For notebooks, we'll prepare standard workflow running environment to test the notebook to the end.
|
||||
* For markdowns, The workflows are generated by extracting the `bash` cells from markdown file. The workflows will prepare standard workflow running environment and test these cells to the end.
|
||||
|
||||
The script will also save workflows in the [workflows](../.github/workflows/) folder, where each notebook or markdown file will have a corresponding workflow file with the `.yml` extension. The workflow files can be triggered by creating a new pull request or pushing a new commit to the repository. The workflow will run the notebook or markdown file, and you could check the outputs afterwards.
|
||||
|
||||
## Feedback and Support
|
||||
|
||||
If you have any feedback or need any support regarding this folder, submit an issue on GitHub. We appreciate your contribution and hope you enjoy using our project.
|
||||
@@ -0,0 +1,176 @@
|
||||
# Promptflow examples
|
||||
|
||||
[](https://github.com/psf/black)
|
||||
[](../LICENSE)
|
||||
|
||||
## Get started
|
||||
|
||||
**Install dependencies**
|
||||
|
||||
- Bootstrap your python environment.
|
||||
- e.g: create a new [conda](https://conda.io/projects/conda/en/latest/user-guide/getting-started.html) environment. `conda create -n pf-examples python=3.9`.
|
||||
- install required packages in python environment : `pip install -r requirements.txt`
|
||||
- show installed sdk: `pip show promptflow`
|
||||
|
||||
**Quick start**
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [quickstart.ipynb](tutorials/get-started/quickstart.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstart.yml) | A quickstart tutorial to run a flow and evaluate it. |
|
||||
| [quickstart-azure.ipynb](tutorials/get-started/quickstart-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstartazure.yml) | A quickstart tutorial to run a flow in Azure AI and evaluate it. |
|
||||
|
||||
|
||||
## CLI examples
|
||||
|
||||
### Tutorials ([tutorials](tutorials))
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [chat-with-pdf](tutorials/e2e-development/chat-with-pdf.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_e2e_development_chat_with_pdf.yml) | Retrieval Augmented Generation (or RAG) has become a prevalent pattern to build intelligent application with Large Language Models (or LLMs) since it can infuse external knowledge into the model, which is not trained with those up-to-date or proprietary information |
|
||||
| [azure-app-service](tutorials/flow-deploy/azure-app-service/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_azure_app_service.yml) | This example demos how to deploy a flow using Azure App Service |
|
||||
| [create-service-with-flow](tutorials/flow-deploy/create-service-with-flow/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_create_service_with_flow.yml) | This example shows how to create a simple service with flow |
|
||||
| [distribute-flow-as-executable-app](tutorials/flow-deploy/distribute-flow-as-executable-app/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_distribute_flow_as_executable_app.yml) | This example demos how to package flow as a executable app |
|
||||
| [docker](tutorials/flow-deploy/docker/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_docker.yml) | This example demos how to deploy flow as a docker app |
|
||||
| [kubernetes](tutorials/flow-deploy/kubernetes/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_deploy_kubernetes.yml) | This example demos how to deploy flow as a Kubernetes app |
|
||||
| [promptflow-quality-improvement](tutorials/flow-fine-tuning-evaluation/promptflow-quality-improvement.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_flow_fine_tuning_evaluation_promptflow_quality_improvement.yml) | This tutorial is designed to enhance your understanding of improving flow quality through prompt tuning and evaluation |
|
||||
| [tracing](tutorials/tracing/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tutorials_tracing.yml) | Prompt flow provides the tracing feature to capture and visualize the internal execution details for all flows |
|
||||
|
||||
|
||||
### Prompty ([prompty](prompty))
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [basic](prompty/basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_basic.yml) | A basic prompt that uses the chat API to answer questions, with connection configured using environment variables |
|
||||
| [chat-basic](prompty/chat-basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_chat_basic.yml) | A prompt that uses the chat API to answer questions with chat history, leveraging promptflow connection |
|
||||
| [eval-apology](prompty/eval-apology/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_eval_apology.yml) | A prompt that determines whether a chat conversation contains an apology from the assistant |
|
||||
| [eval-basic](prompty/eval-basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_eval_basic.yml) | Basic evaluator prompt for QA scenario |
|
||||
| [format-output](prompty/format-output/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_format_output.yml) | A few examples that demos different prompty response format like text, json_object, and how to enable stream output |
|
||||
|
||||
|
||||
### Flex Flows ([flex-flows](flex-flows))
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [basic](flex-flows/basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_basic.yml) | A basic standard flow define using function entry that calls Azure OpenAI with connection info stored in environment variables |
|
||||
| [chat-async-stream](flex-flows/chat-async-stream/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_async_stream.yml) | A chat flow defined using async class entry that return output in stream mode |
|
||||
| [chat-basic](flex-flows/chat-basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_basic.yml) | A basic chat flow defined using class entry |
|
||||
| [chat-minimal](flex-flows/chat-minimal/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_minimal.yml) | A chat flow defined using function with minimal code |
|
||||
| [chat-stream](flex-flows/chat-stream/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_stream.yml) | A chat flow defined using class entry that return output in stream mode |
|
||||
| [chat-with-functions](flex-flows/chat-with-functions/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_with_functions.yml) | This flow covers how to use the LLM chat API in combination with external functions to extend the capabilities of GPT models |
|
||||
| [eval-checklist](flex-flows/eval-checklist/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_checklist.yml) | A example flow defined using class entry which demos how to evaluate the answer pass user specified check list |
|
||||
| [eval-code-quality](flex-flows/eval-code-quality/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_code_quality.yml) | A example flow defined using class based entry which leverages model config to evaluate the quality of code snippet |
|
||||
| [eval-criteria-with-langchain](flex-flows/eval-criteria-with-langchain/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_criteria_with_langchain.yml) | A example flow of converting LangChain criteria evaluator application to flex flow |
|
||||
|
||||
|
||||
### Flows ([flows](flows))
|
||||
|
||||
#### [Standard flows](flows/standard/)
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [autonomous-agent](flows/standard/autonomous-agent/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_autonomous_agent.yml) | This is a flow showcasing how to construct a AutoGPT agent with promptflow to autonomously figures out how to apply the given functions to solve the goal, which is film trivia that provides accurate and up-to-date information about movies, directors, actors, and more in this sample |
|
||||
| [basic](flows/standard/basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_basic.yml) | A basic standard flow using custom python tool that calls Azure OpenAI with connection info stored in environment variables |
|
||||
| [basic-with-builtin-llm](flows/standard/basic-with-builtin-llm/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_basic_with_builtin_llm.yml) | A basic standard flow that calls Azure OpenAI with builtin llm tool |
|
||||
| [basic-with-connection](flows/standard/basic-with-connection/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_basic_with_connection.yml) | A basic standard flow that using custom python tool calls Azure OpenAI with connection info stored in custom connection |
|
||||
| [conditional-flow-for-if-else](flows/standard/conditional-flow-for-if-else/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_conditional_flow_for_if_else.yml) | This example is a conditional flow for if-else scenario |
|
||||
| [conditional-flow-for-switch](flows/standard/conditional-flow-for-switch/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_conditional_flow_for_switch.yml) | This example is a conditional flow for switch scenario |
|
||||
| [customer-intent-extraction](flows/standard/customer-intent-extraction/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_customer_intent_extraction.yml) | This sample is using OpenAI chat model(ChatGPT/GPT4) to identify customer intent from customer's question |
|
||||
| [describe-image](flows/standard/describe-image/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_describe_image.yml) | A flow that take image input, flip it horizontally and uses OpenAI GPT-4V tool to describe it |
|
||||
| [flow-with-additional-includes](flows/standard/flow-with-additional-includes/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_flow_with_additional_includes.yml) | User sometimes need to reference some common files or folders, this sample demos how to solve the problem using additional_includes |
|
||||
| [flow-with-symlinks](flows/standard/flow-with-symlinks/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_flow_with_symlinks.yml) | User sometimes need to reference some common files or folders, this sample demos how to solve the problem using symlinks |
|
||||
| [gen-docstring](flows/standard/gen-docstring/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_gen_docstring.yml) | This example can help you automatically generate Python code's docstring and return the modified code |
|
||||
| [maths-to-code](flows/standard/maths-to-code/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_maths_to_code.yml) | Math to Code is a project that utilizes the power of the chatGPT model to generate code that models math questions and then executes the generated code to obtain the final numerical answer |
|
||||
| [named-entity-recognition](flows/standard/named-entity-recognition/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_named_entity_recognition.yml) | A flow that perform named entity recognition task |
|
||||
| [question-simulation](flows/standard/question-simulation/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_question_simulation.yml) | This question simulation flow is used to generate suggestions for the next question based on the previous chat history |
|
||||
| [web-classification](flows/standard/web-classification/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_standard_web_classification.yml) | This is a flow demonstrating multi-class classification with LLM |
|
||||
|
||||
|
||||
#### [Evaluation flows](flows/evaluation/)
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [eval-basic](flows/evaluation/eval-basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_basic.yml) | This example shows how to create a basic evaluation flow |
|
||||
| [eval-chat-math](flows/evaluation/eval-chat-math/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_chat_math.yml) | This example shows how to evaluate the answer of math questions, which can compare the output results with the standard answers numerically |
|
||||
| [eval-classification-accuracy](flows/evaluation/eval-classification-accuracy/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_classification_accuracy.yml) | This is a flow illustrating how to evaluate the performance of a classification system |
|
||||
| [eval-entity-match-rate](flows/evaluation/eval-entity-match-rate/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_entity_match_rate.yml) | This is a flow evaluates: entity match rate |
|
||||
| [eval-groundedness](flows/evaluation/eval-groundedness/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_groundedness.yml) | This is a flow leverage llm to eval groundedness: whether answer is stating facts that are all present in the given context |
|
||||
| [eval-multi-turn-metrics](flows/evaluation/eval-multi-turn-metrics/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_multi_turn_metrics.yml) | This evaluation flow will evaluate a conversation by using Large Language Models (LLM) to measure the quality of the responses |
|
||||
| [eval-perceived-intelligence](flows/evaluation/eval-perceived-intelligence/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_perceived_intelligence.yml) | This is a flow leverage llm to eval perceived intelligence |
|
||||
| [eval-qna-non-rag](flows/evaluation/eval-qna-non-rag/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_qna_non_rag.yml) | This is a flow evaluating the Q&A systems by leveraging Large Language Models (LLM) to measure the quality and safety of responses |
|
||||
| [eval-qna-rag-metrics](flows/evaluation/eval-qna-rag-metrics/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_qna_rag_metrics.yml) | This is a flow evaluating the Q&A RAG (Retrieval Augmented Generation) systems by leveraging the state-of-the-art Large Language Models (LLM) to measure the quality and safety of responses |
|
||||
| [eval-single-turn-metrics](flows/evaluation/eval-single-turn-metrics/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_single_turn_metrics.yml) | This evaluation flow will evaluate a question and answer pair by using Large Language Models (LLM) to measure the quality of the answer |
|
||||
| [eval-summarization](flows/evaluation/eval-summarization/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_evaluation_eval_summarization.yml) | This flow implements a reference-free automatic abstractive summarization evaluation across four dimensions: fluency, coherence, consistency, relevance |
|
||||
|
||||
|
||||
#### [Chat flows](flows/chat/)
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [chat-basic](flows/chat/chat-basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chat_basic.yml) | This example shows how to create a basic chat flow |
|
||||
| [chat-math-variant](flows/chat/chat-math-variant/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chat_math_variant.yml) | This is a prompt tuning case with 3 prompt variants for math question answering |
|
||||
| [chat-with-image](flows/chat/chat-with-image/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chat_with_image.yml) | This flow demonstrates how to create a chatbot that can take image and text as input |
|
||||
| [chat-with-pdf](flows/chat/chat-with-pdf/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chat_with_pdf.yml) | This is a simple flow that allow you to ask questions about the content of a PDF file and get answers |
|
||||
| [chat-with-wikipedia](flows/chat/chat-with-wikipedia/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chat_with_wikipedia.yml) | This flow demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message |
|
||||
| [use_functions_with_chat_models](flows/chat/use_functions_with_chat_models/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_use_functions_with_chat_models.yml) | This flow covers how to use the LLM tool chat API in combination with external functions to extend the capabilities of GPT models |
|
||||
|
||||
|
||||
### Tool Use Cases ([Tool Use Cases](tools/use-cases))
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [cascading-inputs-tool-showcase](tools/use-cases/cascading-inputs-tool-showcase/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tools_use_cases_cascading_inputs_tool_showcase.yml) | This is a flow demonstrating the use of a tool with cascading inputs which frequently used in situations where the selection in one input field determines what subsequent inputs should be shown, and it helps in creating a more efficient, user-friendly, and error-free input process |
|
||||
| [custom-strong-type-connection-package-tool-showcase](tools/use-cases/custom-strong-type-connection-package-tool-showcase/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tools_use_cases_custom_strong_type_connection_package_tool_showcase.yml) | This is a flow demonstrating the use of a package tool with custom string type connection which provides a secure way to manage credentials for external APIs and data sources, and it offers an improved user-friendly and intellisense experience compared to custom connections |
|
||||
| [custom-strong-type-connection-script-tool-showcase](tools/use-cases/custom-strong-type-connection-script-tool-showcase/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tools_use_cases_custom_strong_type_connection_script_tool_showcase.yml) | This is a flow demonstrating the use of a script tool with custom string type connection which provides a secure way to manage credentials for external APIs and data sources, and it offers an improved user-friendly and intellisense experience compared to custom connections |
|
||||
| [custom_llm_tool_showcase](tools/use-cases/custom_llm_tool_showcase/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tools_use_cases_custom_llm_tool_showcase.yml) | This is a flow demonstrating how to use a `custom_llm` tool, which enables users to seamlessly connect to a large language model with prompt tuning experience using a `PromptTemplate` |
|
||||
| [dynamic-list-input-tool-showcase](tools/use-cases/dynamic-list-input-tool-showcase/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tools_use_cases_dynamic_list_input_tool_showcase.yml) | This is a flow demonstrating how to use a tool with a dynamic list input |
|
||||
|
||||
|
||||
### Connections ([connections](connections))
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [connections](connections/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_connections.yml) | This folder contains example `YAML` files for creating `connection` using `pf` cli |
|
||||
|
||||
|
||||
|
||||
## SDK examples
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [quickstart.ipynb](tutorials/get-started/quickstart.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstart.yml) | A quickstart tutorial to run a flow and evaluate it. |
|
||||
| [quickstart-azure.ipynb](tutorials/get-started/quickstart-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_quickstartazure.yml) | A quickstart tutorial to run a flow in Azure AI and evaluate it. |
|
||||
| [flow-as-function.ipynb](tutorials/get-started/flow-as-function.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_getstarted_flowasfunction.yml) | This guide will walk you through the main scenarios of executing flow as a function. |
|
||||
| [pipeline.ipynb](tutorials/run-flow-with-pipeline/pipeline.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_runflowwithpipeline_pipeline.yml) | Create pipeline using components to run a distributed job with tensorflow |
|
||||
| [cloud-run-management.ipynb](tutorials/run-management/cloud-run-management.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_cloudrunmanagement.yml) | Flow run management in Azure AI |
|
||||
| [run-management.ipynb](tutorials/run-management/run-management.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_runmanagement_runmanagement.yml) | Flow run management |
|
||||
| [trace-autogen-groupchat.ipynb](tutorials/tracing/autogen-groupchat/trace-autogen-groupchat.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tracing_autogengroupchat_traceautogengroupchat.yml) | Tracing LLM calls in autogen group chat application |
|
||||
| [otlp-trace-collector.ipynb](tutorials/tracing/custom-otlp-collector/otlp-trace-collector.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tracing_customotlpcollector_otlptracecollector.yml) | A tutorial on how to levarage custom OTLP collector. |
|
||||
| [trace-langchain.ipynb](tutorials/tracing/langchain/trace-langchain.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tracing_langchain_tracelangchain.yml) | Tracing LLM calls in langchain application |
|
||||
| [trace-llm.ipynb](tutorials/tracing/llm/trace-llm.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_tracing_llm_tracellm.yml) | Tracing LLM application |
|
||||
| [connection.ipynb](connections/connection.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_connections_connection.yml) | Manage various types of connections using sdk |
|
||||
| [flex-flow-quickstart-azure.ipynb](flex-flows/basic/flex-flow-quickstart-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_basic_flexflowquickstartazure.yml) | A quickstart tutorial to run a flex flow and evaluate it in Azure. |
|
||||
| [flex-flow-quickstart.ipynb](flex-flows/basic/flex-flow-quickstart.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_basic_flexflowquickstart.yml) | A quickstart tutorial to run a flex flow and evaluate it. |
|
||||
| [chat-stream-with-async-flex-flow.ipynb](flex-flows/chat-async-stream/chat-stream-with-async-flex-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatasyncstream_chatstreamwithasyncflexflow.yml) | A quickstart tutorial to run a class based flex flow in stream mode and evaluate it. |
|
||||
| [chat-with-class-based-flow-azure.ipynb](flex-flows/chat-basic/chat-with-class-based-flow-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflowazure.yml) | A quickstart tutorial to run a class based flex flow and evaluate it in azure. |
|
||||
| [chat-with-class-based-flow.ipynb](flex-flows/chat-basic/chat-with-class-based-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflow.yml) | A quickstart tutorial to run a class based flex flow and evaluate it. |
|
||||
| [chat-stream-with-flex-flow.ipynb](flex-flows/chat-stream/chat-stream-with-flex-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatstream_chatstreamwithflexflow.yml) | A quickstart tutorial to run a class based flex flow in stream mode and evaluate it. |
|
||||
| [langchain-eval.ipynb](flex-flows/eval-criteria-with-langchain/langchain-eval.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_evalcriteriawithlangchain_langchaineval.yml) | A tutorial to converting LangChain criteria evaluator application to flex flow. |
|
||||
| [prompty-quickstart.ipynb](prompty/basic/prompty-quickstart.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_basic_promptyquickstart.yml) | A quickstart tutorial to run a prompty and evaluate it. |
|
||||
| [chat-with-prompty.ipynb](prompty/chat-basic/chat-with-prompty.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_chatbasic_chatwithprompty.yml) | A quickstart tutorial to run a chat prompty and evaluate it. |
|
||||
| [prompty-output-format.ipynb](prompty/format-output/prompty-output-format.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_prompty_formatoutput_promptyoutputformat.yml) | |
|
||||
| [chat-with-pdf-azure.ipynb](flows/chat/chat-with-pdf/chat-with-pdf-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chatwithpdf_chatwithpdfazure.yml) | A tutorial of chat-with-pdf flow that executes in Azure AI |
|
||||
| [chat-with-pdf.ipynb](flows/chat/chat-with-pdf/chat-with-pdf.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flows_chat_chatwithpdf_chatwithpdf.yml) | A tutorial of chat-with-pdf flow that allows user ask questions about the content of a PDF file and get answers |
|
||||
|
||||
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions and suggestions! Please see the [contributing guidelines](../CONTRIBUTING.md) for details.
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). Please see the [code of conduct](../CODE_OF_CONDUCT.md) for details.
|
||||
|
||||
## Reference
|
||||
|
||||
* [Promptflow documentation](https://microsoft.github.io/promptflow/)
|
||||
@@ -0,0 +1,189 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Configuration\n",
|
||||
"\n",
|
||||
"_**Setting up your Azure Machine Learning services workspace and configuring needed resources**_\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"**Requirements** - In order to benefit from this tutorial, you will need:\n",
|
||||
"- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
|
||||
"- An Azure ML workspace\n",
|
||||
"- A python environment\n",
|
||||
"- Install dependent packages for samples via `pip install -r requirements.txt`\n",
|
||||
"\n",
|
||||
"**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
|
||||
"- Connect to your AML workspace from the Python SDK using different auth credentials\n",
|
||||
"- Create workspace config file\n",
|
||||
"\n",
|
||||
"**Motivations** - This notebook covers the scenario that user define components using yaml then use these components to build pipeline."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 1. Import the required libraries & set dependent environment variables"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Import required libraries\n",
|
||||
"from promptflow.azure import PFClient"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 2. Configure credential\n",
|
||||
"\n",
|
||||
"We are using `DefaultAzureCredential` to get access to workspace. When an access token is needed, it requests one using multiple identities(`EnvironmentCredential, ManagedIdentityCredential, SharedTokenCacheCredential, VisualStudioCodeCredential, AzureCliCredential, AzurePowerShellCredential`) in turn, stopping when one provides a token.\n",
|
||||
"Reference [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for more information.\n",
|
||||
"\n",
|
||||
"`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
|
||||
"Reference [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python) for all available credentials if it does not work for you. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from azure.identity import (\n",
|
||||
" InteractiveBrowserCredential,\n",
|
||||
" DefaultAzureCredential,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" credential = DefaultAzureCredential()\n",
|
||||
" # Check if given credential can get token successfully.\n",
|
||||
" credential.get_token(\"https://management.azure.com/.default\")\n",
|
||||
"except Exception as ex:\n",
|
||||
" # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
|
||||
" credential = InteractiveBrowserCredential()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 3. Connect to Azure Machine Learning Workspace\n",
|
||||
"\n",
|
||||
"The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run. \n",
|
||||
"\n",
|
||||
"Check this notebook for creating a [workspace](../resources/workspace/workspace.ipynb).\n",
|
||||
"\n",
|
||||
"To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. \n",
|
||||
"The config details of a workspace can be saved to a file from the Azure Machine Learning [portal](https://ml.azure.com/). Click on the name of the portal on the top right corner to see the link to save the config file.\n",
|
||||
"This config file can be used to load a workspace using `PLClient`. If no path is mentioned, path is defaulted to current folder. If no file name is mentioned, file name will be defaulted to `config.json`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"try:\n",
|
||||
" pf = PFClient.from_config(credential=credential)\n",
|
||||
"except Exception as ex:\n",
|
||||
" # NOTE: Update following workspace information if not correctly configure before\n",
|
||||
" client_config = {\n",
|
||||
" \"subscription_id\": \"<SUBSCRIPTION_ID>\",\n",
|
||||
" \"resource_group\": \"<RESOURCE_GROUP>\",\n",
|
||||
" \"workspace_name\": \"<AML_WORKSPACE_NAME>\",\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" if client_config[\"subscription_id\"].startswith(\"<\"):\n",
|
||||
" print(\n",
|
||||
" \"please update your <SUBSCRIPTION_ID> <RESOURCE_GROUP> <AML_WORKSPACE_NAME> in notebook cell\"\n",
|
||||
" )\n",
|
||||
" raise ex\n",
|
||||
" else: # write and reload from config file\n",
|
||||
" import json, os\n",
|
||||
"\n",
|
||||
" config_path = \"../.azureml/config.json\"\n",
|
||||
" os.makedirs(os.path.dirname(config_path), exist_ok=True)\n",
|
||||
" with open(config_path, \"w\") as fo:\n",
|
||||
" fo.write(json.dumps(client_config))\n",
|
||||
" pf = PFClient.from_config(credential=credential, path=config_path)\n",
|
||||
"print(pf)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Retrieve or create an Azure Machine Learning compute target\n",
|
||||
"\n",
|
||||
"To create a Azure Machine Learning job, you need a compute cluster as prerequisite. Below code ensures computes named `cpu-cluster` exists in your workspace."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from azure.ai.ml import MLClient\n",
|
||||
"from azure.ai.ml.entities import AmlCompute\n",
|
||||
"\n",
|
||||
"# MLClient use the same configuration as PFClient\n",
|
||||
"ml_client = MLClient.from_config(credential=credential)\n",
|
||||
"\n",
|
||||
"# specify aml compute name.\n",
|
||||
"cpu_compute_target = \"cpu-cluster\"\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" ml_client.compute.get(cpu_compute_target)\n",
|
||||
"except Exception:\n",
|
||||
" print(\"Creating a new cpu compute target...\")\n",
|
||||
" compute = AmlCompute(\n",
|
||||
" name=cpu_compute_target, size=\"STANDARD_D2_V2\", min_instances=0, max_instances=4\n",
|
||||
" )\n",
|
||||
" ml_client.compute.begin_create_or_update(compute).result()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO: set up connections"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.17"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
api_key=<your_api_key>
|
||||
@@ -0,0 +1,46 @@
|
||||
# Working with Connection
|
||||
This folder contains example `YAML` files for creating `connection` using `pf` cli. Learn more on all the [connections types](https://microsoft.github.io/promptflow/concepts/concept-connections.html).
|
||||
|
||||
## Prerequisites
|
||||
- Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Get started
|
||||
|
||||
- To create a connection using any of the sample `YAML` files provided in this directory, execute following command:
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create -f custom.yml --set configs.key1='<your_api_key>'
|
||||
pf connection create -f azure_openai.yml --set api_key='<your_api_key>'
|
||||
```
|
||||
|
||||
- To create a custom connection using an `.env` file, execute following command:
|
||||
```bash
|
||||
pf connection create -f .env --name custom_connection
|
||||
```
|
||||
|
||||
- To list the created connection, execute following command:
|
||||
```bash
|
||||
pf connection list
|
||||
```
|
||||
|
||||
- To show one connection details, execute following command:
|
||||
```bash
|
||||
pf connection show --name custom_connection
|
||||
```
|
||||
|
||||
- To update a connection that in workspace, execute following command. Currently only a few fields(description, display_name) support update:
|
||||
```bash
|
||||
# Update an existing connection with --set to override values
|
||||
# Update an azure OpenAI connection with a new api base
|
||||
pf connection update -n open_ai_connection --set api_base='<your_api_base>'
|
||||
# Update a custom connection
|
||||
pf connection update -n custom_connection --set configs.key1='<your_new_key>' secrets.key2='<your_another_key>'
|
||||
```
|
||||
|
||||
- To delete a connection:
|
||||
```bash
|
||||
pf connection delete -n custom_connection
|
||||
```
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/AzureContentSafetyConnection.schema.json
|
||||
name: azure_content_safety_connection
|
||||
type: azure_content_safety
|
||||
api_key: "<to-be-replaced>"
|
||||
endpoint: "endpoint"
|
||||
api_version: "2023-04-30-preview"
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/AzureOpenAIConnection.schema.json
|
||||
name: open_ai_connection
|
||||
type: azure_open_ai
|
||||
api_key: "<user-input>"
|
||||
api_base: "aoai-api-endpoint"
|
||||
api_type: "azure"
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/CognitiveSearchConnection.schema.json
|
||||
name: cognitive_search_connection
|
||||
type: cognitive_search
|
||||
api_key: "<to-be-replaced>"
|
||||
api_base: "endpoint"
|
||||
api_version: "2023-07-01-Preview"
|
||||
@@ -0,0 +1,236 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Connection Management\n",
|
||||
"\n",
|
||||
"Prompt flow provides various prebuilt connections, including Azure OpenAI, OpenAI, Azure Content Safety, etc. Prebuilt connections enable seamless integration with these resources within the built-in tools. \n",
|
||||
"\n",
|
||||
"Additionally, users have the flexibility to create custom connection types using key-value pairs, empowering them to tailor the connections to their specific requirements, particularly in Python tools.\n",
|
||||
"\n",
|
||||
"Reach more details about connection types [here](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2).\n",
|
||||
"## Create different type of connections\n",
|
||||
"We will use Azure OpenAI connection and custom connection as example to show how to create connection with promptflow sdk."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install -r ../requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Initialize a pf client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"# client can help manage your runs and connections.\n",
|
||||
"client = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create an Azure OpenAI connection\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.entities import AzureOpenAIConnection\n",
|
||||
"\n",
|
||||
"# Initialize an AzureOpenAIConnection object\n",
|
||||
"connection = AzureOpenAIConnection(\n",
|
||||
" name=\"my_azure_open_ai_connection\",\n",
|
||||
" api_key=\"<your-api-key>\",\n",
|
||||
" api_base=\"<your-endpoint>\",\n",
|
||||
")\n",
|
||||
"# Create the connection, note that api_key will be scrubbed in the returned result\n",
|
||||
"result = client.connections.create_or_update(connection)\n",
|
||||
"print(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create a custom connection"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.entities import CustomConnection\n",
|
||||
"\n",
|
||||
"# Initialize a custom connection object\n",
|
||||
"connection = CustomConnection(\n",
|
||||
" name=\"my_custom_connection\",\n",
|
||||
" # Secrets is a required field for custom connection\n",
|
||||
" secrets={\"my_key\": \"<your-api-key>\"},\n",
|
||||
" configs={\"endpoint\": \"<your-endpoint>\", \"other_config\": \"other_value\"},\n",
|
||||
")\n",
|
||||
"# Create the connection, note that all secret values will be scrubbed in the returned result\n",
|
||||
"result = client.connections.create_or_update(connection)\n",
|
||||
"print(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## List all connections"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"connections = client.connections.list()\n",
|
||||
"for connection in connections:\n",
|
||||
" print(connection)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get a connection by name"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"connection = client.connections.get(name=\"my_custom_connection\")\n",
|
||||
"print(connection)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Delete a connection by name"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Update a connection\n",
|
||||
"### Update an Azure OpenAI connection"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"connection = client.connections.get(name=\"my_azure_open_ai_connection\")\n",
|
||||
"connection.api_base = \"new_value\"\n",
|
||||
"connection.api_key = (\n",
|
||||
" \"<original-key>\" # secrets are required again when updating connection using sdk\n",
|
||||
")\n",
|
||||
"result = client.connections.create_or_update(connection)\n",
|
||||
"print(connection)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Update a custom connection"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"connection = client.connections.get(name=\"my_custom_connection\")\n",
|
||||
"connection.configs[\"other_config\"] = \"new_value\"\n",
|
||||
"connection.secrets[\n",
|
||||
" \"my_key\"\n",
|
||||
"] = \"new_secret_value\" # ValueError: Connection 'my_custom_connection' secrets ['my_key'] must be filled again when updating it.\n",
|
||||
"result = client.connections.create_or_update(connection)\n",
|
||||
"print(connection)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# client.connections.delete(name=\"my_custom_connection\")"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"description": "Manage various types of connections using sdk",
|
||||
"kernelspec": {
|
||||
"display_name": "promptflow-sdk",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.17"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/CustomConnection.schema.json
|
||||
name: custom_connection
|
||||
type: custom
|
||||
configs:
|
||||
key1: "test1"
|
||||
secrets: # required
|
||||
api-key: "<to-be-replaced>"
|
||||
@@ -0,0 +1,5 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/OpenAIConnection.schema.json
|
||||
name: open_ai_connection
|
||||
type: open_ai
|
||||
api_key: "<user-input>"
|
||||
organization: "" # optional
|
||||
@@ -0,0 +1,3 @@
|
||||
promptflow
|
||||
promptflow-tools
|
||||
python-dotenv
|
||||
@@ -0,0 +1,4 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/SerpConnection.schema.json
|
||||
name: serp_connection
|
||||
type: serp
|
||||
api_key: "<to-be-replaced>"
|
||||
@@ -0,0 +1,14 @@
|
||||
# required for notebook sample ci
|
||||
ipython_genutils
|
||||
ipykernel
|
||||
jinja2 # for readme generations
|
||||
markdown # for readme generator
|
||||
nbformat # for readme generator
|
||||
papermill
|
||||
keyrings.alt
|
||||
black==23.7.0
|
||||
black-nb
|
||||
pypandoc # for markdown reader
|
||||
pypandoc_binary # pypandoc pandoc backend
|
||||
panflute # for pandoc filters
|
||||
numpy<2.0.0 # for compatibility
|
||||
@@ -0,0 +1,2 @@
|
||||
AZURE_OPENAI_API_KEY=<your_AOAI_key>
|
||||
AZURE_OPENAI_ENDPOINT=<your_AOAI_endpoint>
|
||||
@@ -0,0 +1,28 @@
|
||||
# Flex Flow
|
||||
|
||||
You can learn more on flex flow with examples in this folder.
|
||||
|
||||
## SDK examples
|
||||
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [flex-flow-quickstart.ipynb](./basic/flex-flow-quickstart.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_basic_flexflowquickstart.yml) | A quickstart tutorial to run a flex flow and evaluate it. |
|
||||
| [flex-flow-quickstart-azure.ipynb](./basic/flex-flow-quickstart-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_basic_flexflowquickstartazure.yml) | A quickstart tutorial to run a flex flow and evaluate it in azure. |
|
||||
| [chat-with-class-based-flow-azure.ipynb](./chat-basic/chat-with-class-based-flow-azure.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflowazure.yml) | A quickstart tutorial to run a class based flex flow and evaluate it in azure. |
|
||||
| [chat-with-class-based-flow.ipynb](./chat-basic/chat-with-class-based-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatbasic_chatwithclassbasedflow.yml) | A quickstart tutorial to run a class based flex flow and evaluate it. |
|
||||
| [chat-stream-with-flex-flow.ipynb](./chat-stream/chat-stream-with-flex-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatstream_chatstreamwithflexflow.yml) | A quickstart tutorial to run a class based flex flow in stream mode and evaluate it. |
|
||||
| [chat-stream-with-async-flex-flow.ipynb](./chat-async-stream/chat-stream-with-async-flex-flow.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_chatasyncstream_chatstreamwithasyncflexflow.yml) | A quickstart tutorial to run a class based flex flow in stream mode and evaluate it. |
|
||||
| [langchain-eval.ipynb](./eval-criteria-with-langchain/langchain-eval.ipynb) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flexflows_evalcriteriawithlangchain_langchaineval.yml) | A tutorial to converting LangChain criteria evaluator application to flex flow. |
|
||||
|
||||
## CLI examples
|
||||
| path | status | description |
|
||||
------|--------|-------------
|
||||
| [basic](./basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_basic.yml) | A basic standard flow define using function entry that calls Azure OpenAI with connection info stored in environment variables |
|
||||
| [chat-async-stream](./chat-async-stream/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_async_stream.yml) | A chat flow defined using async class entry that return output in stream mode |
|
||||
| [chat-basic](./chat-basic/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_basic.yml) | A basic chat flow defined using class entry |
|
||||
| [chat-minimal](./chat-minimal/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_minimal.yml) | A chat flow defined using function with minimal code |
|
||||
| [chat-stream](./chat-stream/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_stream.yml) | A chat flow defined using class entry that return output in stream mode |
|
||||
| [chat-with-functions](./chat-with-functions/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_chat_with_functions.yml) | This flow covers how to use the LLM chat API in combination with external functions to extend the capabilities of GPT models |
|
||||
| [eval-checklist](./eval-checklist/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_checklist.yml) | A example flow defined using class entry which demos how to evaluate the answer pass user specified check list |
|
||||
| [eval-code-quality](./eval-code-quality/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_code_quality.yml) | A example flow defined using class based entry which leverages model config to evaluate the quality of code snippet |
|
||||
| [eval-criteria-with-langchain](./eval-criteria-with-langchain/README.md) | [](https://github.com/microsoft/promptflow/actions/workflows/samples_flex_flows_eval_criteria_with_langchain.yml) | A example flow of converting LangChain criteria evaluator application to flex flow |
|
||||
@@ -0,0 +1,105 @@
|
||||
# Basic standard flow
|
||||
A basic standard flow define using function entry that calls Azure OpenAI with connection info stored in environment variables.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup environment variables
|
||||
|
||||
Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
|
||||
|
||||
```bash
|
||||
cat ../.env
|
||||
```
|
||||
|
||||
- Run/Debug as normal Python file
|
||||
```bash
|
||||
python programmer.py
|
||||
```
|
||||
|
||||
- Test with flow entry
|
||||
```bash
|
||||
pf flow test --flow programmer:write_simple_program --inputs text="Java Hello World!"
|
||||
```
|
||||
|
||||
- Test with flow yaml
|
||||
```bash
|
||||
# test with sample input value in flow.flex.yaml
|
||||
pf flow test --flow .
|
||||
```
|
||||
|
||||
```shell
|
||||
# test with UI
|
||||
pf flow test --flow . --ui
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
```bash
|
||||
# using environment from .env file (loaded in user code: hello.py)
|
||||
pf run create --flow . --data ./data.jsonl --column-mapping text='${data.text}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud with connection
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --data ./data.jsonl --column-mapping text='${data.text}' --environment-variables AZURE_OPENAI_API_KEY='${open_ai_connection.api_key}' AZURE_OPENAI_ENDPOINT='${open_ai_connection.api_base}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
```
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pfazure run list -r 3
|
||||
|
||||
# get a sample run name
|
||||
name=$(pfazure run list -r 100 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
|
||||
|
||||
# show specific run detail
|
||||
pfazure run show --name $name
|
||||
|
||||
# show output
|
||||
pfazure run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pfazure run visualize --name $name
|
||||
```
|
||||
@@ -0,0 +1,3 @@
|
||||
{"text": "Python Hello World!"}
|
||||
{"text": "C Hello World!"}
|
||||
{"text": "C# Hello World!"}
|
||||
@@ -0,0 +1,306 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Getting started with flex flow in Azure"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write an LLM application using a notebook and visualize the trace of your application.\n",
|
||||
"- Convert the application into a flow and batch-run it against multiple lines of data.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements-azure.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Connection to workspace"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Configure credential\n",
|
||||
"\n",
|
||||
"We are using `DefaultAzureCredential` to access the workspace. \n",
|
||||
"`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
|
||||
"\n",
|
||||
"Reference for other credentials if this does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" credential = DefaultAzureCredential()\n",
|
||||
" # Check if given credential can get token successfully.\n",
|
||||
" credential.get_token(\"https://management.azure.com/.default\")\n",
|
||||
"except Exception as ex:\n",
|
||||
" # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential does not work\n",
|
||||
" credential = InteractiveBrowserCredential()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Connect to the workspace\n",
|
||||
"\n",
|
||||
"We use a config file to connect to a workspace. The Azure ML workspace should be configured with a computer cluster. [Check this notebook for how to configure a workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.azure import PFClient\n",
|
||||
"\n",
|
||||
"# Connect to the workspace\n",
|
||||
"pf = PFClient.from_config(credential=credential)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create necessary connections\n",
|
||||
"A connection helps securely store and manage secret keys or other sensitive credentials required for interacting with the LLM and other external tools, for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"In this notebook, we will use the `basic` & `eval-code-quality` flex flow, which uses the connection `open_ai_connection`. We need to set up the connection if we haven't added it before.\n",
|
||||
"\n",
|
||||
"To prepare your Azure OpenAI resource, follow these [instructions](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
|
||||
"\n",
|
||||
"Go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
|
||||
"Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as a flow with multi-line data.\n",
|
||||
"\n",
|
||||
"Create a `flow.flex.yaml` file to define a flow whose entry points to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Show the flow.flex.yaml content\n",
|
||||
"with open(\"flow.flex.yaml\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"flow = \".\" # Path to the flow directory\n",
|
||||
"data = \"./data.jsonl\" # Path to the data file\n",
|
||||
"\n",
|
||||
"# Create a run with the flow and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"text\": \"${data.text}\",\n",
|
||||
" },\n",
|
||||
" environment_variables={\n",
|
||||
" \"AZURE_OPENAI_API_KEY\": \"${open_ai_connection.api_key}\",\n",
|
||||
" \"AZURE_OPENAI_ENDPOINT\": \"${open_ai_connection.api_base}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually use an LLM to verify the produced output matches the expected output. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup model configuration with connection\n",
|
||||
"\n",
|
||||
"When using Promptflow in Azure, create a model configuration object with connection name. \n",
|
||||
"The model config will connect to the cloud-hosted Promptflow instance while running the flow."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"model_config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\",\n",
|
||||
" azure_deployment=\"gpt-4o\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Evaluate the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input. The evaluation takes the outputs of that **base_run**, and uses an LLM to compare them to your desired outputs, and then visualizes the results."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_flow = \"../eval-code-quality/flow.flex.yaml\"\n",
|
||||
"\n",
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" init={\"model_config\": model_config},\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify the base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"code\": \"${run.outputs.output}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"You've successfully run your first flex flow and evaluated it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate the next message."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "azure",
|
||||
"section": "Flow",
|
||||
"weight": 10
|
||||
},
|
||||
"description": "A quickstart tutorial to run a flex flow and evaluate it in Azure.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements-azure.txt, examples/flex-flows/basic, examples/flex-flows/eval-code-quality"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,385 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Getting started with flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using notebook and visualize the trace of your application.\n",
|
||||
"- Convert the application into a flow and batch run against multi lines of data.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python function that calls OpenAI API. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"llm.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note: before running below cell, please configure required environment variable `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` by create an `.env` file. Please refer to `../.env.example` as an template."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# control the AOAI deployment (model) used in this example\n",
|
||||
"deployment_name = \"gpt-4o\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llm import my_llm_tool\n",
|
||||
"\n",
|
||||
"# pls configure `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` environment variables first\n",
|
||||
"result = my_llm_tool(\n",
|
||||
" prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
|
||||
" deployment_name=deployment_name,\n",
|
||||
")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"# rerun the function, which will be recorded in the trace\n",
|
||||
"result = my_llm_tool(\n",
|
||||
" prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
|
||||
" deployment_name=deployment_name,\n",
|
||||
")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now, let's add another layer of function call. In `programmer.py` there is a function called `write_simple_program`, which calls a new function called `load_prompt` and previous `my_llm_tool` function."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# show the programmer.py content\n",
|
||||
"with open(\"programmer.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# call the flow entry function\n",
|
||||
"from programmer import write_simple_program\n",
|
||||
"\n",
|
||||
"result = write_simple_program(\"Java Hello, world!\")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup model configuration with environment variables\n",
|
||||
"\n",
|
||||
"When used in local, create a model configuration object with environment variables."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"\n",
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
|
||||
" # load environment variables from .env file\n",
|
||||
" load_dotenv()\n",
|
||||
"\n",
|
||||
"if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
|
||||
" raise Exception(\"Please specify environment variables: AZURE_OPENAI_API_KEY\")\n",
|
||||
"model_config = AzureOpenAIModelConfiguration(\n",
|
||||
" azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
|
||||
" api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
|
||||
" azure_deployment=deployment_name,\n",
|
||||
" api_version=\"2023-07-01-preview\",\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from code_quality import CodeEvaluator\n",
|
||||
"\n",
|
||||
"evaluator = CodeEvaluator(model_config=model_config)\n",
|
||||
"eval_result = evaluator(result)\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n",
|
||||
"\n",
|
||||
"Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# show the flow.flex.yaml content\n",
|
||||
"with open(\"flow.flex.yaml\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=write_simple_program,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"text\": \"${data.text}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# we can also run flow pointing to yaml file\n",
|
||||
"eval_flow = \"../eval-code-quality/flow.flex.yaml\"\n",
|
||||
"\n",
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" init={\"model_config\": model_config},\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"code\": \"${run.outputs.output}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 10
|
||||
},
|
||||
"description": "A quickstart tutorial to run a flex flow and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/basic"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/flow.schema.json
|
||||
entry: programmer:write_simple_program
|
||||
environment:
|
||||
python_requirements_txt: requirements.txt
|
||||
sample:
|
||||
inputs:
|
||||
text: Java Hello World!
|
||||
@@ -0,0 +1,3 @@
|
||||
system:
|
||||
Write a simple {{text}} program.
|
||||
Output code only.
|
||||
@@ -0,0 +1,64 @@
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from openai.version import VERSION as OPENAI_VERSION
|
||||
|
||||
from promptflow.tracing import trace
|
||||
|
||||
|
||||
def get_client():
|
||||
if OPENAI_VERSION.startswith("0."):
|
||||
raise Exception(
|
||||
"Please upgrade your OpenAI package to version >= 1.0.0 or using the command: pip install --upgrade openai."
|
||||
)
|
||||
api_key = os.environ.get("OPENAI_API_KEY", None)
|
||||
if api_key:
|
||||
from openai import OpenAI
|
||||
|
||||
return OpenAI()
|
||||
else:
|
||||
from openai import AzureOpenAI
|
||||
|
||||
return AzureOpenAI(
|
||||
api_version=os.environ.get("OPENAI_API_VERSION", "2023-07-01-preview")
|
||||
)
|
||||
|
||||
|
||||
@trace
|
||||
def my_llm_tool(
|
||||
prompt: str,
|
||||
# for AOAI, deployment name is customized by user, not model name.
|
||||
deployment_name: str,
|
||||
max_tokens: int = 120,
|
||||
temperature: float = 1.0,
|
||||
top_p: float = 1.0,
|
||||
n: int = 1,
|
||||
) -> str:
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
# load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
raise Exception(
|
||||
"Please specify environment variables: OPENAI_API_KEY or AZURE_OPENAI_API_KEY"
|
||||
)
|
||||
messages = [{"content": prompt, "role": "system"}]
|
||||
response = get_client().chat.completions.create(
|
||||
messages=messages,
|
||||
model=deployment_name,
|
||||
max_tokens=int(max_tokens),
|
||||
temperature=float(temperature),
|
||||
top_p=float(top_p),
|
||||
n=int(n),
|
||||
)
|
||||
|
||||
# get first element because prompt is single.
|
||||
return response.choices[0].message.content
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
result = my_llm_tool(
|
||||
prompt="Write a simple Hello, world! program that displays the greeting message.",
|
||||
deployment_name="gpt-4o",
|
||||
)
|
||||
print(result)
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation code quality module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-code-quality")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1,41 @@
|
||||
from pathlib import Path
|
||||
from typing import TypedDict
|
||||
|
||||
from jinja2 import Template
|
||||
from llm import my_llm_tool
|
||||
|
||||
from promptflow.tracing import trace
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
class Result(TypedDict):
|
||||
output: str
|
||||
|
||||
|
||||
@trace
|
||||
def load_prompt(jinja2_template: str, text: str) -> str:
|
||||
"""Load prompt function."""
|
||||
with open(BASE_DIR / jinja2_template, "r", encoding="utf-8") as f:
|
||||
prompt = Template(
|
||||
f.read(), trim_blocks=True, keep_trailing_newline=True
|
||||
).render(text=text)
|
||||
return prompt
|
||||
|
||||
|
||||
@trace
|
||||
def write_simple_program(
|
||||
text: str = "Hello World!", deployment_name="gpt-4o"
|
||||
) -> Result:
|
||||
"""Ask LLM to write a simple program."""
|
||||
prompt = load_prompt("hello.jinja2", text)
|
||||
output = my_llm_tool(prompt=prompt, deployment_name=deployment_name, max_tokens=120)
|
||||
return Result(output=output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
result = write_simple_program("Hello, world!", "gpt-4o")
|
||||
print(result)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow-azure
|
||||
@@ -0,0 +1,3 @@
|
||||
promptflow[azure]
|
||||
python-dotenv
|
||||
openai>=1.14.0
|
||||
@@ -0,0 +1,10 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
column_mapping:
|
||||
text: ${data.text}
|
||||
environment_variables:
|
||||
# environment variables from connection
|
||||
AZURE_OPENAI_API_KEY: ${open_ai_connection.api_key}
|
||||
AZURE_OPENAI_ENDPOINT: ${open_ai_connection.api_base}
|
||||
AZURE_OPENAI_API_TYPE: azure
|
||||
@@ -0,0 +1,121 @@
|
||||
# Chat stream async
|
||||
A chat flow defined using async class entry that return output in stream mode. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn:
|
||||
- how to compose a chat flow that return output in stream mode.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
- how to consume chat history in prompt.
|
||||
```jinja
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:ChatFlow --init init.json
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
# run chat flow with default question in flow.flex.yaml
|
||||
pf flow test --flow . --init init.json
|
||||
|
||||
# run chat flow with new question
|
||||
pf flow test --flow . --init init.json --inputs question="What's Azure Machine Learning?"
|
||||
|
||||
pf flow test --flow . --init init.json --inputs question="What is ChatGPT? Please explain with consise statement."
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_async_stream_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init ./init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
@@ -0,0 +1,380 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Stream chat with async flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using class based flex flow.\n",
|
||||
"- Use AzureOpenAIModelConfiguration as class init parameter.\n",
|
||||
"- Use prompty to stream completions.\n",
|
||||
"- Convert the application into a async flow and batch run against multi lines of data.\n",
|
||||
"- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python program, which leverage prompty."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"flow.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When `stream=true` is configured in the parameters of a prompt whose output format is text, promptflow sdk will return a generator type, which item is the content of each chunk.\n",
|
||||
"\n",
|
||||
"Reference openai doc on how to do it using plain python code: [how_to_stream_completions](https://cookbook.openai.com/examples/how_to_stream_completions)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"chat.prompty\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
|
||||
"\n",
|
||||
"# client can help manage your runs and connections.\n",
|
||||
"pf = PFClient()\n",
|
||||
"try:\n",
|
||||
" conn_name = \"open_ai_connection\"\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
|
||||
" connection = AzureOpenAIConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" api_key=\"<your_AOAI_key>\",\n",
|
||||
" api_base=\"<your_AOAI_endpoint>\",\n",
|
||||
" api_type=\"azure\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # use this if you have an existing OpenAI account\n",
|
||||
" # connection = OpenAIConnection(\n",
|
||||
" # name=conn_name,\n",
|
||||
" # api_key=\"<user-input>\",\n",
|
||||
" # )\n",
|
||||
"\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"from flow import ChatFlow\n",
|
||||
"\n",
|
||||
"# create a chatFlow obj with connection\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")\n",
|
||||
"chat_flow = ChatFlow(config)\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"\n",
|
||||
"# run the flow as function, which will be recorded in the trace\n",
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with detailed statement\")\n",
|
||||
"# note the type is async generator object as we enabled stream in prompty\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import asyncio\n",
|
||||
"\n",
|
||||
"# print result in stream manner\n",
|
||||
"async for output in result:\n",
|
||||
" print(output, end=\"\")\n",
|
||||
" await asyncio.sleep(0.01)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
|
||||
"\n",
|
||||
"answer = \"\"\n",
|
||||
"async for output in result:\n",
|
||||
" answer += output\n",
|
||||
"answer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from check_list import EvalFlow\n",
|
||||
"\n",
|
||||
"eval_flow = EvalFlow(config)\n",
|
||||
"# evaluate answer agains a set of statement\n",
|
||||
"eval_result = eval_flow(\n",
|
||||
" answer=answer,\n",
|
||||
" statements={\n",
|
||||
" \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
|
||||
" \"consise\": \"It is a consise statement.\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=chat_flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 12
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow in stream mode and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: Stream Chat
|
||||
description: Chat with stream enabled.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
stream: true
|
||||
max_tokens: 1024
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"question": "What is Prompt flow?", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
@@ -0,0 +1 @@
|
||||
entry: flow:ChatFlow
|
||||
@@ -0,0 +1,70 @@
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration, Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
def log(message: str):
|
||||
verbose = os.environ.get("VERBOSE", "false")
|
||||
if verbose.lower() == "true":
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
class ChatFlow:
|
||||
def __init__(
|
||||
self, model_config: AzureOpenAIModelConfiguration, max_total_token=1100
|
||||
):
|
||||
self.model_config = model_config
|
||||
self.max_total_token = max_total_token
|
||||
|
||||
@trace
|
||||
async def __call__(
|
||||
self, question: str = "What is ChatGPT?", chat_history: list = None
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "chat.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > self.max_total_token:
|
||||
chat_history = chat_history[1:]
|
||||
log(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
# output is a generator of string as prompty enabled stream parameter
|
||||
for output in prompty(question=question, chat_history=chat_history):
|
||||
yield output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = ChatFlow(model_config=config)
|
||||
result = flow("What's Azure Machine Learning?", [])
|
||||
|
||||
# print result in stream manner
|
||||
async def consume_result():
|
||||
async for output in result:
|
||||
print(output, end="")
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
asyncio.run(consume_result())
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
connection: open_ai_connection
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1 @@
|
||||
{"question": "What is Prompt flow?"}
|
||||
@@ -0,0 +1,125 @@
|
||||
# Basic chat
|
||||
A basic chat flow defined using class entry. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn
|
||||
- how to compose a chat flow.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
- how to consume chat history in prompt.
|
||||
```jinja
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of prompty supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:ChatFlow --init init.json --inputs question="What's Azure Machine Learning?"
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
# run chat flow with default question in flow.flex.yaml
|
||||
pf flow test --flow .
|
||||
# run chat flow with new question
|
||||
pf flow test --flow . --inputs question="What is ChatGPT? Please explain with consise statement."
|
||||
# run chat flow with specific init and inputs
|
||||
pf flow test --flow . --init init.json --inputs sample.json
|
||||
```
|
||||
|
||||
- Test flow: multi turn
|
||||
```shell
|
||||
# start test in chat UI
|
||||
pf flow test --flow . --ui --init init.json
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_basic_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --init init.json --stream
|
||||
@@ -0,0 +1,295 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Chat with class based flex flow in Azure"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Submit batch run with a flow defined with python class and evaluate it in azure.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements-azure.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Connection to workspace"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Configure credential\n",
|
||||
"\n",
|
||||
"We are using `DefaultAzureCredential` to get access to workspace. \n",
|
||||
"`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
|
||||
"\n",
|
||||
"Reference for more available credentials if it does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" credential = DefaultAzureCredential()\n",
|
||||
" # Check if given credential can get token successfully.\n",
|
||||
" credential.get_token(\"https://management.azure.com/.default\")\n",
|
||||
"except Exception as ex:\n",
|
||||
" # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
|
||||
" credential = InteractiveBrowserCredential()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Get a handle to the workspace\n",
|
||||
"\n",
|
||||
"We use config file to connect to a workspace. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.azure import PFClient\n",
|
||||
"\n",
|
||||
"# Get a handle to workspace\n",
|
||||
"pf = PFClient.from_config(credential=credential)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"In this notebook, we will use flow `basic` flex flow which uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
|
||||
"\n",
|
||||
"Please go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
|
||||
"Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n",
|
||||
"\n",
|
||||
"Create a `flow.flex.yaml` file to define a flow which entry pointing to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# show the flow.flex.yaml content\n",
|
||||
"with open(\"flow.flex.yaml\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"# create the model config to be used in below flow calls\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"flow = \".\" # path to the flow directory\n",
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"\n",
|
||||
"# create run with the flow and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=flow,\n",
|
||||
" init={\n",
|
||||
" \"model_config\": config,\n",
|
||||
" },\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_flow = \"../eval-checklist/flow.flex.yaml\"\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")\n",
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" init={\n",
|
||||
" \"model_config\": config,\n",
|
||||
" },\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "azure",
|
||||
"section": "Flow",
|
||||
"weight": 11
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow and evaluate it in azure.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements-azure.txt, examples/flex-flows/basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,339 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Chat with class based flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using class based flex flow.\n",
|
||||
"- Use AzureOpenAIConnection as class init parameter.\n",
|
||||
"- Convert the application into a flow and batch run against multi lines of data.\n",
|
||||
"- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python program, which leverage promptflow built-in aoai tool. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"flow.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
|
||||
"\n",
|
||||
"# client can help manage your runs and connections.\n",
|
||||
"pf = PFClient()\n",
|
||||
"try:\n",
|
||||
" conn_name = \"open_ai_connection\"\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
|
||||
" connection = AzureOpenAIConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" api_key=\"<your_AOAI_key>\",\n",
|
||||
" api_base=\"<your_AOAI_endpoint>\",\n",
|
||||
" api_type=\"azure\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # use this if you have an existing OpenAI account\n",
|
||||
" # connection = OpenAIConnection(\n",
|
||||
" # name=conn_name,\n",
|
||||
" # api_key=\"<user-input>\",\n",
|
||||
" # )\n",
|
||||
"\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"# create the model config to be used in below flow calls\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from flow import ChatFlow\n",
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"\n",
|
||||
"# create a chatFlow obj with connection\n",
|
||||
"chat_flow = ChatFlow(config)\n",
|
||||
"# run the flow as function, which will be recorded in the trace\n",
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from check_list import EvalFlow\n",
|
||||
"\n",
|
||||
"eval_flow = EvalFlow(config)\n",
|
||||
"# evaluate answer agains a set of statement\n",
|
||||
"eval_result = eval_flow(\n",
|
||||
" answer=result,\n",
|
||||
" statements={\n",
|
||||
" \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
|
||||
" \"consise\": \"It is a consise statement.\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=chat_flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 11
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
name: Basic Chat
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
max_tokens: 1024
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
chat_history: []
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"question": "What is Prompt flow?", "chat_history":[], "statements": {"correctness": "should explain what's 'Prompt flow'", "consise": "It is a consise statement."}}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement", "chat_history":[], "statements": { "correctness": "should explain what's ChatGPT", "consise": "It is a consise statement."}}
|
||||
{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 2", "consise": "It is a consise statement."}}
|
||||
@@ -0,0 +1,13 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: flow:ChatFlow
|
||||
sample:
|
||||
inputs:
|
||||
question: What's Azure Machine Learning?
|
||||
init:
|
||||
model_config:
|
||||
connection: open_ai_connection
|
||||
azure_deployment: gpt-4o
|
||||
max_total_token: 1024
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,66 @@
|
||||
import os
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration, Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
def log(message: str):
|
||||
verbose = os.environ.get("VERBOSE", "false")
|
||||
if verbose.lower() == "true":
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
class ChatFlow:
|
||||
def __init__(
|
||||
self, model_config: AzureOpenAIModelConfiguration, max_total_token=4096
|
||||
):
|
||||
self.model_config = model_config
|
||||
self.max_total_token = max_total_token
|
||||
|
||||
@trace
|
||||
def __call__(
|
||||
self,
|
||||
question: str = "What's Azure Machine Learning?",
|
||||
chat_history: list = None,
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "chat.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > self.max_total_token:
|
||||
chat_history = chat_history[1:]
|
||||
log(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
# output is a string
|
||||
output = prompty(question=question, chat_history=chat_history)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = ChatFlow(config)
|
||||
result = flow("What's Azure Machine Learning?", [])
|
||||
print(result)
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
},
|
||||
"max_total_token": 2048
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow-azure
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]>=1.11.0
|
||||
@@ -0,0 +1,9 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
model_config:
|
||||
connection: open_ai_connection
|
||||
azure_deployment: gpt-4o
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"question": "How many questions did User ask?",
|
||||
"chat_history": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "where is the nearest coffee shop?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "I'm sorry, I don't know that. Would you like me to look it up for you?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what is the capital of France?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Paris"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
# Minimal chat
|
||||
A chat flow defined using function with minimal code. It demonstrates the minimal code to have a chat flow.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn
|
||||
- how to compose a chat flow.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup environment variables
|
||||
|
||||
Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
|
||||
|
||||
```bash
|
||||
cat ../.env
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:chat --inputs question="What's the capital of France?"
|
||||
```
|
||||
|
||||
- Test flow: multi turn
|
||||
```shell
|
||||
# start test in chat ui
|
||||
pf flow test --flow flow:chat --ui
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow flow:chat --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_minimal_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
name: Minimal Chat
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
max_tokens: 1024
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,2 @@
|
||||
{"question": "What is Prompt flow?"}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement"}
|
||||
@@ -0,0 +1,8 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: flow:chat
|
||||
sample:
|
||||
inputs:
|
||||
question: What's Azure Machine Learning?
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,30 @@
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from pathlib import Path
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
@trace
|
||||
def chat(question: str = "What's the capital of France?") -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
# load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
prompty = Prompty.load(source=BASE_DIR / "chat.prompty")
|
||||
output = prompty(question=question)
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
|
||||
result = chat("What's the capital of France?")
|
||||
print(result)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]>=1.11.0
|
||||
@@ -0,0 +1,10 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
environment_variables:
|
||||
# environment variables from connection
|
||||
AZURE_OPENAI_API_KEY: ${open_ai_connection.api_key}
|
||||
AZURE_OPENAI_ENDPOINT: ${open_ai_connection.api_base}
|
||||
AZURE_OPENAI_API_TYPE: azure
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1,126 @@
|
||||
# Chat stream
|
||||
A chat flow defined using class entry that return output in stream mode. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies in this folder:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn:
|
||||
- how to compose a chat flow that return output in stream mode.
|
||||
- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
|
||||
See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
|
||||
```jinja
|
||||
system:
|
||||
You are a chatbot having a conversation with a human.
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
```
|
||||
- how to consume chat history in prompt.
|
||||
```jinja
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python flow.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
|
||||
```bash
|
||||
pf flow test --flow flow:ChatFlow --init init.json
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
# run chat flow with default question in flow.flex.yaml
|
||||
pf flow test --flow . --init init.json
|
||||
|
||||
# run chat flow with new question
|
||||
pf flow test --flow . --init init.json --inputs question="What's Azure Machine Learning?"
|
||||
|
||||
pf flow test --flow . --init init.json --inputs question="What is ChatGPT? Please explain with consise statement."
|
||||
```
|
||||
|
||||
- Test flow with UI
|
||||
```shell
|
||||
pf flow test --flow . --init init.json --ui
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
```
|
||||
|
||||
You can also skip providing `column-mapping` if provided data has same column name as the flow.
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_stream_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init ./init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
@@ -0,0 +1,378 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Stream chat with flex flow"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Write LLM application using class based flex flow.\n",
|
||||
"- Use AzureOpenAIModelConfiguration as class init parameter.\n",
|
||||
"- Use prompty to stream completions.\n",
|
||||
"- Convert the application into a flow and batch run against multi lines of data.\n",
|
||||
"- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your application with promptflow\n",
|
||||
"\n",
|
||||
"Assume we already have a python program, which leverage prompty."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"flow.py\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When `stream=true` is configured in the parameters of a prompt whose output format is text, promptflow sdk will return a generator type, which item is the content of each chunk.\n",
|
||||
"\n",
|
||||
"Reference openai doc on how to do it using plain python code: [how_to_stream_completions](https://cookbook.openai.com/examples/how_to_stream_completions)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"chat.prompty\") as fin:\n",
|
||||
" print(fin.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"### Create necessary connections\n",
|
||||
"Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
|
||||
"\n",
|
||||
"Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
|
||||
"\n",
|
||||
"Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
|
||||
"\n",
|
||||
"# client can help manage your runs and connections.\n",
|
||||
"pf = PFClient()\n",
|
||||
"try:\n",
|
||||
" conn_name = \"open_ai_connection\"\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
|
||||
" connection = AzureOpenAIConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" api_key=\"<your_AOAI_key>\",\n",
|
||||
" api_base=\"<your_AOAI_endpoint>\",\n",
|
||||
" api_type=\"azure\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # use this if you have an existing OpenAI account\n",
|
||||
" # connection = OpenAIConnection(\n",
|
||||
" # name=conn_name,\n",
|
||||
" # api_key=\"<user-input>\",\n",
|
||||
" # )\n",
|
||||
"\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualize trace by using start_trace\n",
|
||||
"\n",
|
||||
"Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.tracing import start_trace\n",
|
||||
"from promptflow.core import AzureOpenAIModelConfiguration\n",
|
||||
"\n",
|
||||
"from flow import ChatFlow\n",
|
||||
"\n",
|
||||
"# create a chatFlow obj with connection\n",
|
||||
"config = AzureOpenAIModelConfiguration(\n",
|
||||
" connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
|
||||
")\n",
|
||||
"chat_flow = ChatFlow(config)\n",
|
||||
"\n",
|
||||
"# start a trace session, and print a url for user to check trace\n",
|
||||
"start_trace()\n",
|
||||
"\n",
|
||||
"# run the flow as function, which will be recorded in the trace\n",
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with detailed statement\")\n",
|
||||
"# note the type is generator object as we enabled stream in prompty\n",
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import time\n",
|
||||
"\n",
|
||||
"# print result in stream manner\n",
|
||||
"for r in result:\n",
|
||||
" print(r, end=\"\")\n",
|
||||
" # For better animation effects\n",
|
||||
" time.sleep(0.01)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
|
||||
"answer = \"\".join(result)\n",
|
||||
"answer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Eval the result "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%load_ext autoreload\n",
|
||||
"%autoreload 2\n",
|
||||
"\n",
|
||||
"import paths # add the code_quality module to the path\n",
|
||||
"from check_list import EvalFlow\n",
|
||||
"\n",
|
||||
"eval_flow = EvalFlow(config)\n",
|
||||
"# evaluate answer agains a set of statement\n",
|
||||
"eval_result = eval_flow(\n",
|
||||
" answer=answer,\n",
|
||||
" statements={\n",
|
||||
" \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
|
||||
" \"consise\": \"It is a consise statement.\",\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"eval_result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the function as flow with multi-line data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch run with a data file (with multiple lines of test data)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=chat_flow,\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"question\": \"${data.question}\",\n",
|
||||
" \"chat_history\": \"${data.chat_history}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Evaluate your flow\n",
|
||||
"Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run evaluation on the previous batch run\n",
|
||||
"The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"eval_run = pf.run(\n",
|
||||
" flow=eval_flow,\n",
|
||||
" data=\"./data.jsonl\", # path to the data file\n",
|
||||
" run=base_run, # specify base_run as the run you want to evaluate\n",
|
||||
" column_mapping={\n",
|
||||
" \"answer\": \"${run.outputs.output}\",\n",
|
||||
" \"statements\": \"${data.statements}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(eval_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"metrics = pf.get_metrics(eval_run)\n",
|
||||
"print(json.dumps(metrics, indent=4))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run, eval_run])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next steps\n",
|
||||
"\n",
|
||||
"By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
|
||||
"\n",
|
||||
"You can check out more examples:\n",
|
||||
"- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 12
|
||||
},
|
||||
"description": "A quickstart tutorial to run a class based flex flow in stream mode and evaluate it.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: Stream Chat
|
||||
description: Chat with stream enabled.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
stream: true
|
||||
max_tokens: 1024
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: "What is Prompt flow?"
|
||||
---
|
||||
|
||||
system:
|
||||
You are a helpful assistant.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"question": "What is Prompt flow?", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "What is ChatGPT? Please explain with consise statement", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
|
||||
@@ -0,0 +1,5 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: flow:ChatFlow
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,69 @@
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration, Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
def log(message: str):
|
||||
verbose = os.environ.get("VERBOSE", "false")
|
||||
if verbose.lower() == "true":
|
||||
print(message, flush=True)
|
||||
|
||||
|
||||
class ChatFlow:
|
||||
def __init__(
|
||||
self, model_config: AzureOpenAIModelConfiguration, max_total_token=1100
|
||||
):
|
||||
self.model_config = model_config
|
||||
self.max_total_token = max_total_token
|
||||
|
||||
@trace
|
||||
def __call__(
|
||||
self, question: str = "What is ChatGPT?", chat_history: list = None
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "chat.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > self.max_total_token:
|
||||
chat_history = chat_history[1:]
|
||||
log(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
# output is a generator of string as prompty enabled stream parameter
|
||||
output = prompty(question=question, chat_history=chat_history)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = ChatFlow(model_config=config)
|
||||
result = flow("What's Azure Machine Learning?", [])
|
||||
|
||||
# print result in stream manner
|
||||
for r in result:
|
||||
print(r, end="")
|
||||
# For better animation effects
|
||||
time.sleep(0.01)
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import sys
|
||||
import pathlib
|
||||
|
||||
# Add the path to the evaluation module
|
||||
code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
|
||||
sys.path.insert(0, code_path)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow[azure]
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
connection: open_ai_connection
|
||||
column_mapping:
|
||||
question: ${data.question}
|
||||
@@ -0,0 +1 @@
|
||||
{"question": "What is Prompt flow?"}
|
||||
@@ -0,0 +1,57 @@
|
||||
# Chat with external functions
|
||||
|
||||
This flow covers how to use the LLM chat API in combination with external functions to extend the capabilities of GPT models.
|
||||
|
||||
`tools` is an optional parameter in the Chat Completion API which can be used to provide function specifications. The purpose of this is to enable models to generate function arguments which adhere to the provided specifications. Note that the API will not actually execute any function calls. It is up to developers to execute function calls using model outputs.
|
||||
|
||||
Within the `tools` parameter, if the `functions` parameter is provided then by default the model will decide when it is appropriate to use one of the functions. The API can be forced to use a specific function by setting the `tool_choice` parameter to `{"type": "function", "function": {"name": "my_function"}}`. The API can also be forced to not use any function by setting the `tool_choice` parameter to `"none"`. If a function is used, the output will contain `"finish_reason": "tool_calls"` in the response, as well as a `tool_calls` object that has the name of the function and the generated function arguments.
|
||||
|
||||
You can refer to <a href='https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb' target='_blank'>openai sample</a> for more details.
|
||||
|
||||
|
||||
## What you will learn
|
||||
|
||||
In this flow, you will learn how to use external functions with LLM chat models and how to compose function role message in prompt template.
|
||||
|
||||
## Prerequisites
|
||||
Install dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Getting started
|
||||
|
||||
### 1 prepare api_key env
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup environment variables
|
||||
|
||||
Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
|
||||
|
||||
### 2 understand how chat model choose function
|
||||
```shell
|
||||
# see how llm will chose functions
|
||||
pf flow test --flow chat_with_tools.prompty --env --inputs question="What is the weather like in Boston?"
|
||||
pf flow test --flow chat_with_tools.prompty --env --inputs sample.json
|
||||
```
|
||||
|
||||
### 3 Start chatting with flow
|
||||
|
||||
```bash
|
||||
# run e2e chat flow with question
|
||||
pf flow test --flow flow:chat --env --inputs question="What is the weather like in Boston?"
|
||||
|
||||
# run chat flow with question & chat_history from sample.json
|
||||
pf flow test --flow flow:chat --env --inputs sample.json
|
||||
```
|
||||
|
||||
```shell
|
||||
# chat using ui
|
||||
pf flow test --flow flow:chat --env --ui
|
||||
```
|
||||
|
||||
## References
|
||||
- <a href='https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb' target='_blank'>OpenAI cookbook example</a>
|
||||
- <a href='https://openai.com/blog/function-calling-and-other-api-updates?ref=upstract.com' target='_blank'>OpenAI function calling announcement</a>
|
||||
- <a href='https://platform.openai.com/docs/guides/gpt/function-calling' target='_blank'>OpenAI function calling doc</a>
|
||||
- <a href='https://platform.openai.com/docs/api-reference/chat/create' target='_blank'>OpenAI function calling API</a>
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
name: Chat with Tools
|
||||
description: A basic prompt with tools
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
max_tokens: 128
|
||||
temperature: 0.2
|
||||
tools: ${file:tools.json}
|
||||
tool_choice: auto
|
||||
inputs:
|
||||
question:
|
||||
type: string
|
||||
chat_history:
|
||||
type: list
|
||||
sample:
|
||||
question: What's the weather in Beijing?
|
||||
chat_history: []
|
||||
---
|
||||
# system:
|
||||
Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.
|
||||
|
||||
{% for item in chat_history %}
|
||||
{{item.role}}:
|
||||
{{item.content}}
|
||||
{% endfor %}
|
||||
|
||||
user:
|
||||
{{question}}
|
||||
@@ -0,0 +1,103 @@
|
||||
import json
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from pathlib import Path
|
||||
from typing import TypedDict
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
class WeatherInfo(TypedDict):
|
||||
location: str
|
||||
temperature: float
|
||||
format: str
|
||||
forecast: list[str]
|
||||
num_days: int
|
||||
|
||||
|
||||
def get_current_weather(location, format="fahrenheit"):
|
||||
"""Get the current weather in a given location"""
|
||||
return WeatherInfo(
|
||||
location=location, temperature="72", format=format, forecast=["sunny", "windy"]
|
||||
)
|
||||
|
||||
|
||||
def get_n_day_weather_forecast(location, format, num_days):
|
||||
"""Get next num_days weather in a given location"""
|
||||
return WeatherInfo(
|
||||
location=location,
|
||||
temperature="60",
|
||||
format=format,
|
||||
forecast=["rainy"],
|
||||
num_days=num_days,
|
||||
)
|
||||
|
||||
|
||||
@trace
|
||||
def run_function(response_message: dict) -> str:
|
||||
if "tool_calls" in response_message and len(response_message["tool_calls"]) == 1:
|
||||
call = response_message["tool_calls"][0]
|
||||
function = call["function"]
|
||||
function_name = function["name"]
|
||||
function_args = json.loads(function["arguments"])
|
||||
print(function_args)
|
||||
result = globals()[function_name](**function_args)
|
||||
return str(result)
|
||||
|
||||
print("No function call")
|
||||
if isinstance(response_message, dict):
|
||||
result = response_message["content"]
|
||||
else:
|
||||
result = response_message
|
||||
return result
|
||||
|
||||
|
||||
MAX_TOTAL_TOKEN = 2048
|
||||
|
||||
|
||||
@trace
|
||||
def chat(
|
||||
question: str = "What's the weather of Beijing?",
|
||||
chat_history: list = None,
|
||||
max_total_token=2048,
|
||||
) -> str:
|
||||
"""Flow entry function."""
|
||||
|
||||
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
|
||||
# load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
prompty = Prompty.load(source=BASE_DIR / "chat_with_tools.prompty")
|
||||
|
||||
chat_history = chat_history or []
|
||||
# Try to render the prompt with token limit and reduce the history count if it fails
|
||||
while len(chat_history) > 0:
|
||||
token_count = prompty.estimate_token_count(
|
||||
question=question, chat_history=chat_history
|
||||
)
|
||||
if token_count > MAX_TOTAL_TOKEN:
|
||||
chat_history = chat_history[1:]
|
||||
print(
|
||||
f"Reducing chat history count to {len(chat_history)} to fit token limit"
|
||||
)
|
||||
else:
|
||||
break
|
||||
|
||||
output = prompty(question=question, chat_history=chat_history)
|
||||
|
||||
function_output = run_function(output)
|
||||
|
||||
return function_output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
|
||||
result = chat("What's the weather of Beijing?")
|
||||
print(result)
|
||||
@@ -0,0 +1 @@
|
||||
promptflow
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"chat_history": [],
|
||||
"question": "How about London next week?"
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
[
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_current_weather",
|
||||
"description": "Get the current weather",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state, e.g. San Francisco, CA"
|
||||
},
|
||||
"format": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "The temperature unit to use. Infer this from the users location."
|
||||
}
|
||||
},
|
||||
"required": ["location", "format"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_n_day_weather_forecast",
|
||||
"description": "Get an N-day weather forecast",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city and state, e.g. San Francisco, CA"
|
||||
},
|
||||
"format": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "The temperature unit to use. Infer this from the users location."
|
||||
},
|
||||
"num_days": {
|
||||
"type": "integer",
|
||||
"description": "The number of days to forecast"
|
||||
}
|
||||
},
|
||||
"required": ["location", "format", "num_days"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,89 @@
|
||||
# Eval Check List
|
||||
A example flow defined using class entry which demos how to evaluate the answer pass user specified check list.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
|
||||
```bash
|
||||
python check_list.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
|
||||
|
||||
```bash
|
||||
pf flow test --flow . --init init.json --inputs sample.json
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
```
|
||||
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("eval_checklist_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
# run using yaml file
|
||||
pfazure run create --file run.yml --stream
|
||||
@@ -0,0 +1,90 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import Prompty, AzureOpenAIModelConfiguration
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
@trace
|
||||
def check(answer: str, statement: str, model_config: AzureOpenAIModelConfiguration):
|
||||
"""Check the answer applies for the check statement."""
|
||||
examples = [
|
||||
{
|
||||
"answer": "ChatGPT is a conversational AI model developed by OpenAI.",
|
||||
"statement": "It contains a brief explanation of ChatGPT.",
|
||||
"score": 5,
|
||||
"explanation": "The statement is correct. The answer contains a brief explanation of ChatGPT.",
|
||||
}
|
||||
]
|
||||
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "eval.prompty",
|
||||
model={"configuration": model_config},
|
||||
)
|
||||
output = prompty(examples=examples, answer=answer, statement=statement)
|
||||
output = json.loads(output)
|
||||
return output
|
||||
|
||||
|
||||
class EvalFlow:
|
||||
def __init__(self, model_config: AzureOpenAIModelConfiguration):
|
||||
self.model_config = model_config
|
||||
|
||||
def __call__(self, answer: str, statements: dict):
|
||||
"""Check the answer applies for a collection of check statement."""
|
||||
if isinstance(statements, str):
|
||||
statements = json.loads(statements)
|
||||
|
||||
results = {}
|
||||
for key, statement in statements.items():
|
||||
r = check(
|
||||
answer=answer, statement=statement, model_config=self.model_config
|
||||
)
|
||||
results[key] = r
|
||||
return results
|
||||
|
||||
def __aggregate__(self, line_results: list) -> dict:
|
||||
"""Aggregate the results."""
|
||||
total = len(line_results)
|
||||
avg_correctness = (
|
||||
sum(int(r["correctness"]["score"]) for r in line_results) / total
|
||||
)
|
||||
return {
|
||||
"average_correctness": avg_correctness,
|
||||
"total": total,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
|
||||
answer = """ChatGPT is a conversational AI model developed by OpenAI.
|
||||
It is based on the GPT-3 architecture and is designed to generate human-like responses to text inputs.
|
||||
ChatGPT is capable of understanding and responding to a wide range of topics and can be used for tasks such as
|
||||
answering questions, generating creative content, and providing assistance with various tasks.
|
||||
The model has been trained on a diverse range of internet text and is constantly being updated to improve its
|
||||
performance and capabilities. ChatGPT is available through the OpenAI API and can be accessed by developers and
|
||||
researchers to build applications and tools that leverage its capabilities."""
|
||||
statements = {
|
||||
"correctness": "It contains a detailed explanation of ChatGPT.",
|
||||
"consise": "It is a consise statement.",
|
||||
}
|
||||
|
||||
config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection", azure_deployment="gpt-4o"
|
||||
)
|
||||
flow = EvalFlow(config)
|
||||
|
||||
result = flow(
|
||||
answer=answer,
|
||||
statements=statements,
|
||||
)
|
||||
print(result)
|
||||
|
||||
# run aggregation
|
||||
aggregation_result = flow.__aggregate__([result])
|
||||
print(aggregation_result)
|
||||
@@ -0,0 +1 @@
|
||||
{"answer": "ChatGPT is a conversational AI model developed by OpenAI. It is based on the GPT-3 architecture and is designed to generate human-like responses to text inputs. ChatGPT is capable of understanding and responding to a wide range of topics and can be used for tasks such as answering questions, generating creative content, and providing assistance with various tasks. The model has been trained on a diverse range of internet text and is constantly being updated to improve its performance and capabilities. ChatGPT is available through the OpenAI API and can be accessed by developers and researchers to build applications and tools that leverage its capabilities.", "statements": { "correctness": "It contains a detailed explanation of ChatGPT." }}
|
||||
@@ -0,0 +1,43 @@
|
||||
|
||||
---
|
||||
name: Evaluate based on a checklist
|
||||
description: Evaluate the quality of code snippet.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
max_tokens: 256
|
||||
temperature: 0.7
|
||||
|
||||
inputs:
|
||||
examples:
|
||||
type: list
|
||||
answer:
|
||||
type: string
|
||||
statement:
|
||||
type: string
|
||||
sample: ${file:sample.json}
|
||||
---
|
||||
|
||||
# system:
|
||||
You are an AI assistant.
|
||||
You task is to evaluate a score based on how the statement applies for the answer.
|
||||
Only accepts valid JSON format response without extra prefix or postfix.
|
||||
|
||||
# user:
|
||||
This score value should always be an integer between 1 and 5. So the score produced should be 1 or 2 or 3 or 4 or 5.
|
||||
|
||||
Here are a few examples:
|
||||
{% for ex in examples %}
|
||||
answer: {{ex.answer}}
|
||||
statement: {{ex.statement}}
|
||||
OUTPUT:
|
||||
{"score": "{{ex.score}}", "explanation":"{{ex.explanation}}"}
|
||||
{% endfor %}
|
||||
|
||||
For a given answer, valuate the answer based on how the statement applies for the answer:
|
||||
answer: {{answer}}
|
||||
statement: {{statement}}
|
||||
OUTPUT:
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
# flow is defined as python function
|
||||
entry: check_list:EvalFlow
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
promptflow>=1.11.0
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
|
||||
flow: .
|
||||
data: data.jsonl
|
||||
init:
|
||||
connection: open_ai_connection
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"answer": "ChatGPT is a conversational AI model developed by OpenAI. It is based on the GPT-3 architecture and is designed to generate human-like responses to text inputs. ChatGPT is capable of understanding and responding to a wide range of topics and can be used for tasks such as answering questions, generating creative content, and providing assistance with various tasks. The model has been trained on a diverse range of internet text and is constantly being updated to improve its performance and capabilities. ChatGPT is available through the OpenAI API and can be accessed by developers and researchers to build applications and tools that leverage its capabilities.",
|
||||
"statements": { "correctness": "It contains a detailed explanation of ChatGPT." }
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
# Eval Code Quality
|
||||
A example flow defined using class based entry which leverages model config to evaluate the quality of code snippet.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name open_ai_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
```bash
|
||||
python code_quality.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
```bash
|
||||
# correct
|
||||
pf flow test --flow . --inputs code='print(\"Hello, world!\")' --init init.json
|
||||
|
||||
# incorrect
|
||||
pf flow test --flow . --inputs code='printf("Hello, world!")' --init init.json
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
```
|
||||
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("eval_code_quality_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# show metrics
|
||||
pf run show-metrics --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
@@ -0,0 +1,69 @@
|
||||
import json
|
||||
|
||||
from typing import TypedDict
|
||||
from pathlib import Path
|
||||
|
||||
from jinja2 import Template
|
||||
|
||||
from promptflow.tracing import trace
|
||||
from promptflow.core import AzureOpenAIModelConfiguration
|
||||
from promptflow.core._flow import Prompty
|
||||
|
||||
BASE_DIR = Path(__file__).absolute().parent
|
||||
|
||||
|
||||
@trace
|
||||
def load_prompt(jinja2_template: str, code: str, examples: list) -> str:
|
||||
"""Load prompt function."""
|
||||
with open(BASE_DIR / jinja2_template, "r", encoding="utf-8") as f:
|
||||
tmpl = Template(f.read(), trim_blocks=True, keep_trailing_newline=True)
|
||||
prompt = tmpl.render(code=code, examples=examples)
|
||||
return prompt
|
||||
|
||||
|
||||
class Result(TypedDict):
|
||||
correctness: float
|
||||
readability: float
|
||||
explanation: str
|
||||
|
||||
|
||||
class CodeEvaluator:
|
||||
def __init__(self, model_config: AzureOpenAIModelConfiguration):
|
||||
self.model_config = model_config
|
||||
|
||||
def __call__(self, code: str) -> Result:
|
||||
"""Evaluate the code based on correctness, readability."""
|
||||
prompty = Prompty.load(
|
||||
source=BASE_DIR / "eval_code_quality.prompty",
|
||||
model={"configuration": self.model_config},
|
||||
)
|
||||
output = prompty(code=code)
|
||||
output = json.loads(output)
|
||||
output = Result(**output)
|
||||
return output
|
||||
|
||||
def __aggregate__(self, line_results: list) -> dict:
|
||||
"""Aggregate the results."""
|
||||
total = len(line_results)
|
||||
avg_correctness = sum(int(r["correctness"]) for r in line_results) / total
|
||||
avg_readability = sum(int(r["readability"]) for r in line_results) / total
|
||||
return {
|
||||
"average_correctness": avg_correctness,
|
||||
"average_readability": avg_readability,
|
||||
"total": total,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
model_config = AzureOpenAIModelConfiguration(
|
||||
connection="open_ai_connection",
|
||||
azure_deployment="gpt-4o",
|
||||
)
|
||||
evaluator = CodeEvaluator(model_config)
|
||||
result = evaluator('print("Hello, world!")')
|
||||
print(result)
|
||||
aggregate_result = evaluator.__aggregate__([result])
|
||||
print(aggregate_result)
|
||||
@@ -0,0 +1,2 @@
|
||||
{"code": "print(\"Hello, world!\")"}
|
||||
{"code": "printf(\"Hello, world!\")"}
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: Evaluate code quality
|
||||
description: Evaluate the quality of code snippet.
|
||||
model:
|
||||
api: chat
|
||||
configuration:
|
||||
type: azure_openai
|
||||
azure_deployment: gpt-4o
|
||||
parameters:
|
||||
temperature: 0.2
|
||||
inputs:
|
||||
code:
|
||||
type: string
|
||||
sample: ${file:sample.json}
|
||||
---
|
||||
# system:
|
||||
You are an AI assistant.
|
||||
You task is to evaluate the code based on correctness, readability.
|
||||
Only accepts valid JSON format response without extra prefix or postfix.
|
||||
|
||||
# user:
|
||||
This correctness value should always be an integer between 1 and 5. So the correctness produced should be 1 or 2 or 3 or 4 or 5.
|
||||
This readability value should always be an integer between 1 and 5. So the readability produced should be 1 or 2 or 3 or 4 or 5.
|
||||
|
||||
Here are a few examples:
|
||||
|
||||
**Example 1**
|
||||
Code: print(\"Hello, world!\")
|
||||
OUTPUT:
|
||||
{
|
||||
"correctness": 5,
|
||||
"readability": 5,
|
||||
"explanation": "The code is correct as it is a simple question and answer format. The readability is also good as the code is short and easy to understand."
|
||||
}
|
||||
|
||||
For a given code, valuate the code based on correctness, readability:
|
||||
Code: {{code}}
|
||||
OUTPUT:
|
||||
@@ -0,0 +1,6 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
# flow is defined as python function
|
||||
entry: code_quality:CodeEvaluator
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"model_config": {
|
||||
"connection": "open_ai_connection",
|
||||
"azure_deployment": "gpt-4o"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
promptflow
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"code": "print(\"Hello, world!\")"
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
# Eval Conciseness Criteria with LangChain
|
||||
|
||||
A example flow of converting LangChain criteria evaluator application to flex flow.
|
||||
Reference [here](https://python.langchain.com/docs/guides/productionization/evaluation/string/criteria_eval_chain/) for more information.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install promptflow sdk and other dependencies:
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Run flow
|
||||
|
||||
- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
|
||||
- Or prepare your Anthropic resource follow this [instruction](https://python.langchain.com/docs/integrations/platforms/anthropic/) and get your `api_key` if you don't have one.
|
||||
|
||||
- Setup connection
|
||||
|
||||
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
|
||||
|
||||
Or use CLI to create connection:
|
||||
|
||||
```bash
|
||||
# Override keys with --set to avoid yaml file changes
|
||||
pf connection create --file ./connection.yml --set secrets.openai_api_key=<your_api_key> configs.azure_endpoint=<your_api_base> --name my_llm_connection
|
||||
```
|
||||
|
||||
Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `my_llm_connection`.
|
||||
```bash
|
||||
# show registered connection
|
||||
pf connection show --name my_llm_connection
|
||||
```
|
||||
|
||||
- Run as normal Python file
|
||||
```bash
|
||||
python eval_conciseness.py
|
||||
```
|
||||
|
||||
- Test flow
|
||||
```bash
|
||||
pf flow test --flow eval_conciseness:LangChainEvaluator --inputs input="What's 2+2?" prediction="What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four." --init custom_connection=my_llm_connection
|
||||
```
|
||||
|
||||
- Test flow with yaml
|
||||
```bash
|
||||
pf flow test --flow .
|
||||
```
|
||||
|
||||
- Create run with multiple lines data
|
||||
|
||||
```bash
|
||||
pf run create --flow . --data ./data.jsonl --init custom_connection=my_llm_connection --stream
|
||||
```
|
||||
|
||||
Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
|
||||
|
||||
- List and show run meta
|
||||
|
||||
```bash
|
||||
# list created run
|
||||
pf run list
|
||||
|
||||
# get a sample run name
|
||||
|
||||
name=$(pf run list -r 10 | jq '.[] | select(.name | contains("eval_criteria_with_langchain_")) | .name'| head -n 1 | tr -d '"')
|
||||
# show specific run detail
|
||||
pf run show --name $name
|
||||
|
||||
# show output
|
||||
pf run show-details --name $name
|
||||
|
||||
# show metrics
|
||||
pf run show-metrics --name $name
|
||||
|
||||
# visualize run in browser
|
||||
pf run visualize --name $name
|
||||
```
|
||||
|
||||
## Run flow in cloud
|
||||
|
||||
- Assume we already have a connection named `open_ai_connection` in workspace.
|
||||
|
||||
```bash
|
||||
# set default workspace
|
||||
az account set -s <your_subscription_id>
|
||||
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
|
||||
```
|
||||
|
||||
- Create run
|
||||
|
||||
```bash
|
||||
# run with environment variable reference connection in azureml workspace
|
||||
pfazure run create --flow . --init init.json --data ./data.jsonl --stream
|
||||
@@ -0,0 +1,7 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/CustomConnection.schema.json
|
||||
name: my_llm_connection
|
||||
type: custom
|
||||
configs:
|
||||
azure_endpoint: "<to-be-replaced>"
|
||||
secrets: # required
|
||||
openai_api_key: "<to-be-replaced>"
|
||||
@@ -0,0 +1,2 @@
|
||||
{"input": "What's 2+2?", "prediction": "What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four."}
|
||||
{"input": "What's 2x2?", "prediction": "What's 2x2? That's an elementary question. The answer you're looking for is that two and two is four."}
|
||||
@@ -0,0 +1,70 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
from langchain.evaluation import load_evaluator
|
||||
from langchain_community.chat_models import AzureChatOpenAI, ChatAnthropic
|
||||
|
||||
from promptflow.client import PFClient
|
||||
from promptflow.connections import CustomConnection
|
||||
from promptflow.tracing import trace
|
||||
|
||||
|
||||
@dataclass
|
||||
class Result:
|
||||
reasoning: str
|
||||
value: str
|
||||
score: float
|
||||
|
||||
|
||||
class LangChainEvaluator:
|
||||
def __init__(self, custom_connection: CustomConnection):
|
||||
self.custom_connection = custom_connection
|
||||
|
||||
# create llm according to the secrets in custom connection
|
||||
if "anthropic_api_key" in self.custom_connection.secrets:
|
||||
self.llm = ChatAnthropic(
|
||||
temperature=0,
|
||||
anthropic_api_key=self.custom_connection.secrets["anthropic_api_key"],
|
||||
)
|
||||
elif "openai_api_key" in self.custom_connection.secrets:
|
||||
self.llm = AzureChatOpenAI(
|
||||
deployment_name="gpt-4o",
|
||||
openai_api_key=self.custom_connection.secrets["openai_api_key"],
|
||||
azure_endpoint=self.custom_connection.configs["azure_endpoint"],
|
||||
openai_api_type="azure",
|
||||
openai_api_version="2023-07-01-preview",
|
||||
temperature=0,
|
||||
)
|
||||
else:
|
||||
raise ValueError("No valid API key found in the connection.")
|
||||
# evaluate with langchain evaluator for conciseness
|
||||
self.evaluator = load_evaluator(
|
||||
"criteria", llm=self.llm, criteria="conciseness"
|
||||
)
|
||||
|
||||
@trace
|
||||
def __call__(
|
||||
self,
|
||||
input: str,
|
||||
prediction: str,
|
||||
) -> Result:
|
||||
"""Evaluate with langchain evaluator."""
|
||||
|
||||
eval_result = self.evaluator.evaluate_strings(
|
||||
prediction=prediction, input=input
|
||||
)
|
||||
return Result(**eval_result)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from promptflow.tracing import start_trace
|
||||
|
||||
start_trace()
|
||||
pf = PFClient()
|
||||
connection = pf.connections.get(name="my_llm_connection")
|
||||
evaluator = LangChainEvaluator(custom_connection=connection)
|
||||
result = evaluator(
|
||||
prediction="What's 2+2? That's an elementary question. "
|
||||
"The answer you're looking for is that two and two is four.",
|
||||
input="What's 2+2?",
|
||||
)
|
||||
print(result)
|
||||
@@ -0,0 +1,11 @@
|
||||
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
|
||||
entry: eval_conciseness:LangChainEvaluator
|
||||
environment:
|
||||
# image: mcr.microsoft.com/azureml/promptflow/promptflow-python
|
||||
python_requirements_txt: requirements.txt
|
||||
sample:
|
||||
inputs:
|
||||
input: What's 2+2?
|
||||
prediction: What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.
|
||||
init:
|
||||
custom_connection: my_llm_connection
|
||||
@@ -0,0 +1,206 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Evaluate with langchain's evaluator"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
|
||||
"\n",
|
||||
"- Convert LangChain criteria evaluator applications to `flex flow`.\n",
|
||||
"- Use `CustomConnection` to store secrets.\n",
|
||||
"\n",
|
||||
"## 0. Install dependent packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%capture --no-stderr\n",
|
||||
"%pip install -r ./requirements.txt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Trace your langchain evaluator with prompt flow\n",
|
||||
"### Initialize a pf client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from promptflow.client import PFClient\n",
|
||||
"\n",
|
||||
"pf = PFClient()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create a custom connection to protect your API key\n",
|
||||
"\n",
|
||||
"You can protect your API key in custom connection's secrets."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from dotenv import load_dotenv\n",
|
||||
"\n",
|
||||
"from promptflow.entities import CustomConnection\n",
|
||||
"\n",
|
||||
"conn_name = \"my_llm_connection\"\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" conn = pf.connections.get(name=conn_name)\n",
|
||||
" print(\"using existing connection\")\n",
|
||||
"except:\n",
|
||||
" if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
|
||||
" # load environment variables from .env file\n",
|
||||
" load_dotenv()\n",
|
||||
"\n",
|
||||
" # put API key in secrets\n",
|
||||
" connection = CustomConnection(\n",
|
||||
" name=conn_name,\n",
|
||||
" configs={\n",
|
||||
" \"azure_endpoint\": os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
|
||||
" },\n",
|
||||
" secrets={\n",
|
||||
" # store API key\n",
|
||||
" # \"anthropic_api_key\": \"<your-api-key>\",\n",
|
||||
" \"openai_api_key\": os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
|
||||
" },\n",
|
||||
" )\n",
|
||||
" # Create the connection, note that all secret values will be scrubbed in the returned result\n",
|
||||
" conn = pf.connections.create_or_update(connection)\n",
|
||||
" print(\"successfully created connection\")\n",
|
||||
"print(conn)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Test the evaluator with trace"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from eval_conciseness import LangChainEvaluator\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"evaluator = LangChainEvaluator(custom_connection=conn)\n",
|
||||
"result = evaluator(\n",
|
||||
" prediction=\"What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.\",\n",
|
||||
" input=\"What's 2+2?\",\n",
|
||||
")\n",
|
||||
"print(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Batch run the evaluator with flow yaml\n",
|
||||
"Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/eval-criteria-with-langchain/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"data = \"./data.jsonl\" # path to the data file\n",
|
||||
"# create run with the flow function and data\n",
|
||||
"base_run = pf.run(\n",
|
||||
" flow=\"./flow.flex.yaml\",\n",
|
||||
" # reference custom connection by name\n",
|
||||
" init={\n",
|
||||
" \"custom_connection\": \"my_llm_connection\",\n",
|
||||
" },\n",
|
||||
" data=data,\n",
|
||||
" column_mapping={\n",
|
||||
" \"prediction\": \"${data.prediction}\",\n",
|
||||
" \"input\": \"${data.input}\",\n",
|
||||
" },\n",
|
||||
" stream=True,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"details = pf.get_details(base_run)\n",
|
||||
"details.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pf.visualize([base_run])"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"build_doc": {
|
||||
"author": [
|
||||
"D-W-@github.com",
|
||||
"wangchao1230@github.com"
|
||||
],
|
||||
"category": "local",
|
||||
"section": "Flow",
|
||||
"weight": 60
|
||||
},
|
||||
"description": "A tutorial to converting LangChain criteria evaluator application to flex flow.",
|
||||
"kernelspec": {
|
||||
"display_name": "prompt_flow",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.18"
|
||||
},
|
||||
"resources": "examples/flex-flows/eval-criteria-with-langchain"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
promptflow
|
||||
langchain>=0.1.5
|
||||
python-dotenv
|
||||
langchain-community>=0.2.0
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user