wehub/microsoft--promptflow
1
0
Fork 0
Code Issues Pull Requests Actions 11 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
Details
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Details
Publish Promptflow Doc / Build (push) Has been cancelled
Details
Publish Promptflow Doc / Deploy (push) Has been cancelled
Details
Flake8 Lint / flake8 (push) Has been cancelled
Details
Spell check CI / Spell_Check (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:52 +08:00
commit e768098d0e
4004 changed files with 2804145 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/tutorials/tracing/.env.example
+3
View File
@@ -0,0 +1,3 @@
CHAT_DEPLOYMENT_NAME=gpt-35-turbo
AZURE_OPENAI_API_KEY=<your_AOAI_key>
AZURE_OPENAI_ENDPOINT=<your_AOAI_endpoint>
examples/tutorials/tracing/README.md
+114
View File
@@ -0,0 +1,114 @@
---
resources: examples/tutorials/tracing/
cloud: local
category: tracing
---
## Tracing
Prompt flow provides the tracing feature to capture and visualize the internal execution details for all flows.
For `DAG flow`, user can track and visualize node level inputs/outputs of flow execution, it provides critical insights for developer to understand the internal details of execution.
For `Flex flow` developers, who might use different frameworks (langchain, semantic kernel, OpenAI, kinds of agents) to create LLM based applications, prompt flow allow user to instrument their code in a [OpenTelemetry](https://opentelemetry.io/) compatible way, and visualize using UI provided by promptflow devkit.
## Instrumenting user's code
#### Enable trace for LLM calls
Let's start with the simplest example, add single line code `start_trace()` to enable trace for LLM calls in your application.
```python
from openai import OpenAI
from promptflow.tracing import start_trace
# start_trace() will print a url for trace detail visualization
start_trace()
client = OpenAI()
completion = client.chat.completions.create(
model="gpt-4",
messages=[
{"role": "system", "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."},
{"role": "user", "content": "Compose a poem that explains the concept of recursion in programming."}
]
)
print(completion.choices[0].message)
```
With the trace url, user will see a trace list that corresponding to each LLM calls:
![LLM-trace-list](../../../docs/media/trace/LLM-trace-list.png)
Click on line record, the LLM detail will be displayed with chat window experience, together with other LLM call params:
![LLM-trace-detail](../../../docs/media/trace/LLM-trace-detail.png)
More examples of adding trace for [autogen](https://microsoft.github.io/autogen/) and [langchain](https://python.langchain.com/docs/get_started/introduction/):
1. **[Add trace for Autogen](./autogen-groupchat/)**
![autogen-trace-detail](../../../docs/media/trace/autogen-trace-detail.png)
2. **[Add trace for Langchain](./langchain)**
![langchain-trace-detail](../../../docs/media/trace/langchain-trace-detail.png)
#### Trace for any function
More common scenario is the application has complicated code structure, and developer would like to add trace on critical path that they would like to debug and monitor.
See the **[math_to_code](./math_to_code.py)** example on how to use `@trace`.
```python
from promptflow.tracing import trace
# trace your function
@trace
def code_gen(client: AzureOpenAI, question: str) -> str:
sys_prompt = (
"I want you to act as a Math expert specializing in Algebra, Geometry, and Calculus. "
"Given the question, develop python code to model the user's question. "
"Make sure only reply the executable code, no other words."
)
completion = client.chat.completions.create(
model=os.getenv("CHAT_DEPLOYMENT_NAME", "gpt-35-turbo"),
messages=[
{
"role": "system",
"content": sys_prompt,
},
{"role": "user", "content": question},
],
)
raw_code = completion.choices[0].message.content
result = code_refine(raw_code)
return result
```
Execute below command will get an URL to display the trace records and trace details of each test.
```bash
python math_to_code.py
```
## Trace visualization in flow test and batch run
### Flow test
If your application is created with DAG flow, all flow test and batch run will be automatically enable trace function. Take the **[chat_with_pdf](../../flows/chat/chat-with-pdf/)** as example.
Run `pf flow test --flow .`, each flow test will generate single line in the trace UI:
![flow-trace-record](../../../docs/media/trace/flow-trace-records.png)
Click a record, the trace details will be visualized as tree view.
![flow-trace-detail](../../../docs/media/trace/flow-trace-detail.png)
### Evaluate against batch data
Keep using **[chat_with_pdf](../../flows/chat/chat-with-pdf/)** as example, to trigger a batch run, you can use below commands:
```shell
pf run create -f batch_run.yaml
```
Or
```shell
pf run create --flow . --data "./data/bert-paper-qna.jsonl" --column-mapping chat_history='${data.chat_history}' pdf_url='${data.pdf_url}' question='${data.question}'
```
Then you will get a run related trace URL, e.g. http://127.0.0.1:52008/v1.0/ui/traces?run=chat_with_pdf_variant_0_20240226_181222_219335
![batch_run_record](../../../docs/media/trace/batch_run_record.png)
examples/tutorials/tracing/autogen-groupchat/.gitignore
+2
View File
@@ -0,0 +1,2 @@
OAI_CONFIG_LIST.json
groupchat
examples/tutorials/tracing/autogen-groupchat/OAI_CONFIG_LIST.json.example
+16
View File
@@ -0,0 +1,16 @@
[
{
"model": "gpt-4",
"api_key": "<your_AOAI_key>",
"base_url": "<your_AOAI_endpoint>",
"api_type": "azure",
"api_version": "2023-06-01-preview"
},
{
"model": "gpt-35-turbo",
"api_key": "<your_AOAI_key>",
"base_url": "<your_AOAI_endpoint>",
"api_type": "azure",
"api_version": "2023-06-01-preview"
}
]
examples/tutorials/tracing/autogen-groupchat/README.md
+6
View File
@@ -0,0 +1,6 @@
# Tracing existing application using promptflow: Auto Generated Agent Group Chat
AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation.
Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).
Check out this [notebook](./agentchat_groupchat.ipynb) for example.
examples/tutorials/tracing/autogen-groupchat/requirements.txt
+3
View File
@@ -0,0 +1,3 @@
promptflow
pyautogen>=0.2.9
pydantic>=2.6.0
examples/tutorials/tracing/autogen-groupchat/trace-autogen-groupchat.ipynb
+230
View File
@@ -0,0 +1,230 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tracing with AutoGen"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation.\n",
"Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
"\n",
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
"\n",
"- Trace LLM (OpenAI) Calls and visualize the trace of your application.\n",
"\n",
"## Requirements\n",
"\n",
"AutoGen requires `Python>=3.8`. To run this notebook example, please install required dependencies:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"%pip install -r ./requirements.txt"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set your API endpoint\n",
"\n",
"You can create the config file named `OAI_CONFIG_LIST.json` from example file: `OAI_CONFIG_LIST.json.example`.\n",
"\n",
"Below code use the [`config_list_from_json`](https://microsoft.github.io/autogen/0.2/docs/reference/oai/openai_utils/#config_list_from_json) function loads a list of configurations from an environment variable or a json file. \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import autogen\n",
"\n",
"# please ensure you have a json config file\n",
"env_or_file = \"OAI_CONFIG_LIST.json\"\n",
"\n",
"# filters the configs by models (you can filter by other keys as well). Only the gpt-4 models are kept in the list based on the filter condition.\n",
"\n",
"# gpt4\n",
"# config_list = autogen.config_list_from_json(\n",
"# env_or_file,\n",
"# filter_dict={\n",
"# \"model\": [\"gpt-4\", \"gpt-4-0314\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
"# },\n",
"# )\n",
"\n",
"# gpt35\n",
"config_list = autogen.config_list_from_json(\n",
" env_or_file,\n",
" filter_dict={\n",
" \"model\": {\n",
" \"gpt-35-turbo\",\n",
" \"gpt-3.5-turbo\",\n",
" \"gpt-3.5-turbo-16k\",\n",
" \"gpt-3.5-turbo-0301\",\n",
" \"chatgpt-35-turbo-0301\",\n",
" \"gpt-35-turbo-v0301\",\n",
" },\n",
" },\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Construct agents"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"os.environ[\"AUTOGEN_USE_DOCKER\"] = \"False\"\n",
"\n",
"llm_config = {\"config_list\": config_list, \"cache_seed\": 42}\n",
"user_proxy = autogen.UserProxyAgent(\n",
" name=\"User_proxy\",\n",
" system_message=\"A human admin.\",\n",
" code_execution_config={\n",
" \"last_n_messages\": 2,\n",
" \"work_dir\": \"groupchat\",\n",
" \"use_docker\": False,\n",
" }, # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\n",
" human_input_mode=\"TERMINATE\",\n",
")\n",
"coder = autogen.AssistantAgent(\n",
" name=\"Coder\",\n",
" llm_config=llm_config,\n",
")\n",
"pm = autogen.AssistantAgent(\n",
" name=\"Product_manager\",\n",
" system_message=\"Creative in software product ideas.\",\n",
" llm_config=llm_config,\n",
")\n",
"groupchat = autogen.GroupChat(agents=[user_proxy, coder, pm], messages=[], max_round=12)\n",
"manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Start chat with promptflow trace"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from promptflow.tracing import start_trace\n",
"\n",
"# start a trace session, and print a url for user to check trace\n",
"# traces will be collected into below collection name\n",
"start_trace(collection=\"autogen-groupchat\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Open the url you get in start_trace output, when running below code, you will be able to see new traces in the UI. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from opentelemetry import trace\n",
"import json\n",
"\n",
"\n",
"tracer = trace.get_tracer(\"my_tracer\")\n",
"# Create a root span\n",
"with tracer.start_as_current_span(\"autogen\") as span:\n",
" message = \"Find a latest paper about gpt-4 on arxiv and find its potential applications in software.\"\n",
" user_proxy.initiate_chat(\n",
" manager,\n",
" message=message,\n",
" clear_history=True,\n",
" )\n",
" span.set_attribute(\"custom\", \"custom attribute value\")\n",
" # recommend to store inputs and outputs as events\n",
" span.add_event(\n",
" \"promptflow.function.inputs\", {\"payload\": json.dumps(dict(message=message))}\n",
" )\n",
" span.add_event(\n",
" \"promptflow.function.output\", {\"payload\": json.dumps(user_proxy.last_message())}\n",
" )\n",
"# type exit to terminate the chat"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"By now you've successfully tracing LLM calls in your app using prompt flow.\n",
"\n",
"You can check out more examples:\n",
"- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run."
]
}
],
"metadata": {
"build_doc": {
"author": [
"zhengfeiwang@github.com",
"wangchao1230@github.com"
],
"category": "local",
"section": "Tracing",
"weight": 20
},
"description": "Tracing LLM calls in autogen group chat application",
"kernelspec": {
"display_name": "prompt_flow",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
},
"resources": ""
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/tutorials/tracing/custom-otlp-collector/llm.py
+51
View File
@@ -0,0 +1,51 @@
import os
from dotenv import load_dotenv
from openai.version import VERSION as OPENAI_VERSION
from promptflow.tracing import trace
def get_client():
if OPENAI_VERSION.startswith("0."):
raise Exception(
"Please upgrade your OpenAI package to version >= 1.0.0 or using the command: pip install --upgrade openai."
)
api_key = os.environ.get("OPENAI_API_KEY", None)
if api_key:
from openai import OpenAI
return OpenAI()
else:
from openai import AzureOpenAI
return AzureOpenAI(api_version=os.environ.get("OPENAI_API_VERSION", "2023-07-01-preview"))
@trace
def my_llm_tool(prompt: str, deployment_name: str) -> str:
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
# load environment variables from .env file
load_dotenv()
if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
raise Exception("Please specify environment variables: OPENAI_API_KEY or AZURE_OPENAI_API_KEY")
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": prompt},
]
response = get_client().chat.completions.create(
messages=messages,
model=deployment_name,
)
# get first element because prompt is single.
return response.choices[0].message.content
if __name__ == "__main__":
result = my_llm_tool(
prompt="Write a simple Hello, world! python program that displays the greeting message. Output code only.",
deployment_name="gpt-4o",
)
print(result)
examples/tutorials/tracing/custom-otlp-collector/otlp-trace-collector.ipynb
+205
View File
@@ -0,0 +1,205 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tracing with Custom OpenTelemetry Collector"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In certain scenario you might want to user your own [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) and keep your dependency mimimal.\n",
"\n",
"In such case you can avoid the dependency of [promptflow-devkit](https://pypi.org/project/promptflow-devkit/) which provides the default collector from promptflow, and only depdent on [promptflow-tracing](https://pypi.org/project/promptflow-tracing), \n",
"\n",
"\n",
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
"\n",
"- Trace LLM (OpenAI) Calls using Custom OpenTelemetry Collector.\n",
"\n",
"## 0. Install dependent packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"%pip install -r ./requirements.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Set up an OpenTelemetry collector\n",
"\n",
"Implement a simple collector that print the traces to stdout."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import threading\n",
"from http.server import BaseHTTPRequestHandler, HTTPServer\n",
"\n",
"from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import (\n",
" ExportTraceServiceRequest,\n",
")\n",
"\n",
"\n",
"class OTLPCollector(BaseHTTPRequestHandler):\n",
" def do_POST(self):\n",
" content_length = int(self.headers[\"Content-Length\"])\n",
" post_data = self.rfile.read(content_length)\n",
"\n",
" traces_request = ExportTraceServiceRequest()\n",
" traces_request.ParseFromString(post_data)\n",
"\n",
" print(\"Received a POST request with data:\")\n",
" print(traces_request)\n",
"\n",
" self.send_response(200, \"Traces received\")\n",
" self.end_headers()\n",
" self.wfile.write(b\"Data received and printed to stdout.\\n\")\n",
"\n",
"\n",
"def run_server(port: int):\n",
" server_address = (\"\", port)\n",
" httpd = HTTPServer(server_address, OTLPCollector)\n",
" httpd.serve_forever()\n",
"\n",
"\n",
"def start_server(port: int):\n",
" server_thread = threading.Thread(target=run_server, args=(port,))\n",
" server_thread.daemon = True\n",
" server_thread.start()\n",
" print(f\"Server started on port {port}. Access http://localhost:{port}/\")\n",
" return server_thread"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# invoke the collector service, serving on OTLP port\n",
"start_server(port=4318)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2. Trace your application with tracing\n",
"Assume we already have a Python function that calls OpenAI API\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from llm import my_llm_tool\n",
"\n",
"deployment_name = \"gpt-35-turbo-16k\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Call `start_trace()`, and configure the OTLP exporter to above collector."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from promptflow.tracing import start_trace\n",
"\n",
"start_trace()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from opentelemetry import trace\n",
"from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n",
"from opentelemetry.sdk.trace.export import BatchSpanProcessor\n",
"\n",
"tracer_provider = trace.get_tracer_provider()\n",
"otlp_span_exporter = OTLPSpanExporter()\n",
"tracer_provider.add_span_processor(BatchSpanProcessor(otlp_span_exporter))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualize traces in the stdout."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"result = my_llm_tool(\n",
" prompt=\"Write a simple Hello, world! python program that displays the greeting message. Output code only.\",\n",
" deployment_name=deployment_name,\n",
")\n",
"result\n",
"# view the traces under this cell"
]
}
],
"metadata": {
"build_doc": {
"author": [
"zhengfeiwang@github.com",
"wangchao1230@github.com"
],
"category": "local",
"section": "Tracing",
"weight": 40
},
"description": "A tutorial on how to levarage custom OTLP collector.",
"kernelspec": {
"display_name": "tracing-rel",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
},
"resources": ""
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/tutorials/tracing/custom-otlp-collector/requirements.txt
+3
View File
@@ -0,0 +1,3 @@
promptflow-tracing
python-dotenv
opentelemetry-exporter-otlp-proto-http
examples/tutorials/tracing/langchain/requirements.txt
+5
View File
@@ -0,0 +1,5 @@
promptflow
langchain>=0.1.5
langchain_community
opentelemetry-instrumentation-langchain
python-dotenv
examples/tutorials/tracing/langchain/trace-langchain.ipynb
+181
View File
@@ -0,0 +1,181 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tracing with LangChain apps"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The tracing capability provided by Prompt flow is built on top of [OpenTelemetry](https://opentelemetry.io/) that gives you complete observability over your LLM applications. \n",
"And there is already a rich set of OpenTelemetry [instrumentation packages](https://opentelemetry.io/ecosystem/registry/?language=python&component=instrumentation) available in OpenTelemetry Eco System. \n",
"\n",
"In this example we will demo how to use [opentelemetry-instrumentation-langchain](https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-langchain) package provided by [Traceloop](https://www.traceloop.com/) to instrument [LangChain](https://python.langchain.com/docs/tutorials/) apps.\n",
"\n",
"\n",
"**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
"\n",
"- Trace `LangChain` applications and visualize the trace of your application in prompt flow.\n",
"\n",
"## Requirements\n",
"\n",
"To run this notebook example, please install required dependencies:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"%pip install -r ./requirements.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Start tracing LangChain using promptflow\n",
"\n",
"Start trace using `promptflow.start_trace`, click the printed url to view the trace ui."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from promptflow.tracing import start_trace\n",
"\n",
"# start a trace session, and print a url for user to check trace\n",
"start_trace()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By default, `opentelemetry-instrumentation-langchain` instrumentation logs prompts, completions, and embeddings to span attributes. This gives you a clear visibility into how your LLM application is working, and can make it easy to debug and evaluate the quality of the outputs."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# enable langchain instrumentation\n",
"from opentelemetry.instrumentation.langchain import LangchainInstrumentor\n",
"\n",
"instrumentor = LangchainInstrumentor()\n",
"if not instrumentor.is_instrumented_by_opentelemetry:\n",
" instrumentor.instrument()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run a simple LangChain\n",
"\n",
"Below is an example targeting an AzureOpenAI resource. Please configure you `API_KEY` using an `.env` file, see `../.env.example`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from langchain.chat_models import AzureChatOpenAI\n",
"from langchain.prompts.chat import ChatPromptTemplate\n",
"from langchain.chains import LLMChain\n",
"from dotenv import load_dotenv\n",
"\n",
"if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
" # load environment variables from .env file\n",
" load_dotenv()\n",
"\n",
"llm = AzureChatOpenAI(\n",
" deployment_name=os.environ[\"CHAT_DEPLOYMENT_NAME\"],\n",
" openai_api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
" azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
" openai_api_type=\"azure\",\n",
" openai_api_version=\"2023-07-01-preview\",\n",
" temperature=0,\n",
")\n",
"\n",
"prompt = ChatPromptTemplate.from_messages(\n",
" [\n",
" (\"system\", \"You are world class technical documentation writer.\"),\n",
" (\"user\", \"{input}\"),\n",
" ]\n",
")\n",
"\n",
"chain = LLMChain(llm=llm, prompt=prompt, output_key=\"metrics\")\n",
"chain({\"input\": \"What is ChatGPT?\"})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You should be able to see traces of the chain in promptflow UI now. Check the cell with `start_trace` on the trace UI url."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"By now you've successfully tracing LLM calls in your app using prompt flow.\n",
"\n",
"You can check out more examples:\n",
"- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run."
]
}
],
"metadata": {
"build_doc": {
"author": [
"zhengfeiwang@github.com",
"wangchao1230@github.com"
],
"category": "local",
"section": "Tracing",
"weight": 30
},
"description": "Tracing LLM calls in langchain application",
"kernelspec": {
"display_name": "prompt_flow",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.17"
},
"resources": ""
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/tutorials/tracing/llm/.env.example
+2
View File
@@ -0,0 +1,2 @@
AZURE_OPENAI_API_KEY=<your_AOAI_key>
AZURE_OPENAI_ENDPOINT=<your_AOAI_endpoint>
examples/tutorials/tracing/llm/media/trace-llm/trace-detail.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 51 KiB

examples/tutorials/tracing/llm/requirements.txt
+3
View File
@@ -0,0 +1,3 @@
promptflow
openai>=1.0.0
python-dotenv
examples/tutorials/tracing/llm/trace-llm.ipynb
+201
View File
@@ -0,0 +1,201 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Tracing with LLM application"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Tracing is a powerful tool for understanding the behavior of your LLM application, prompt flow tracing capability supports instrumentation for such scenario.\n",
"\n",
"This notebook will demonstrate how to use prompt flow to instrument and understand your LLM application.\n",
"\n",
"**Learning Objective** - Upon completion of this notebook, you will be able to:\n",
"\n",
"- Trace LLM application and visualize with prompt flow."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Requirements\n",
"\n",
"To run this notebook example, please install required dependencies."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"%pip install -r ./requirements.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Please configure your API key using an `.env` file, we have provided an example `.env.example` for reference."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# load api key and endpoint from .env to environ\n",
"from dotenv import load_dotenv\n",
"\n",
"load_dotenv()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create your LLM application\n",
"\n",
"This notebook example will build a LLM application with Azure OpenAI service."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from openai import AzureOpenAI\n",
"\n",
"# in this notebook example, we will use model \"gpt-35-turbo-16k\"\n",
"deployment_name = \"gpt-35-turbo-16k\"\n",
"\n",
"client = AzureOpenAI(\n",
" azure_deployment=deployment_name,\n",
" api_version=\"2024-02-01\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# prepare one classic question for LLM\n",
"conversation = [\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"What is the meaning of life?\"},\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"response = client.chat.completions.create(\n",
" messages=conversation,\n",
" model=deployment_name,\n",
")\n",
"print(response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Start trace using `promptflow.tracing.start_trace` to leverage prompt flow tracing capability; this will print a link to trace UI, where you can visualize the trace."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from promptflow.tracing import start_trace\n",
"\n",
"# start a trace session, and print a url for user to check trace\n",
"start_trace(collection=\"trace-llm\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run the LLM application again, and you should be able to see new trace logged in the trace UI, and it is clickable to see more details.\n",
"\n",
"![trace-detail](./media/trace-llm/trace-detail.png)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"response = client.chat.completions.create(\n",
" messages=conversation,\n",
" model=deployment_name,\n",
")\n",
"print(response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next Steps\n",
"\n",
"By now you have successfully tracing your LLM application with prompt flow.\n",
"\n",
"You can check out more examples:\n",
"\n",
"- [Trace LangChain](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/tracing/langchain/trace-langchain.ipynb): tracing `LangChain` and visualize leveraging prompt flow.\n",
"- [Trace AutoGen](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/tracing/autogen-groupchat/trace-autogen-groupchat.ipynb): tracing `AutoGen` and visualize leveraging prompt flow.\n",
"- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run.\n"
]
}
],
"metadata": {
"build_doc": {
"author": [
"zhengfeiwang@github.com"
],
"category": "local",
"section": "Tracing",
"weight": 10
},
"description": "Tracing LLM application",
"kernelspec": {
"display_name": "pf-dev",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
},
"resources": ""
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/tutorials/tracing/math_to_code.py
+93
View File
@@ -0,0 +1,93 @@
import ast
import os
from dotenv import load_dotenv
from openai import AzureOpenAI
from promptflow.tracing import start_trace, trace
@trace
def infinite_loop_check(code_snippet):
tree = ast.parse(code_snippet)
for node in ast.walk(tree):
if isinstance(node, ast.While):
if not node.orelse:
return True
return False
@trace
def syntax_error_check(code_snippet):
try:
ast.parse(code_snippet)
except SyntaxError:
return True
return False
@trace
def error_fix(code_snippet):
tree = ast.parse(code_snippet)
for node in ast.walk(tree):
if isinstance(node, ast.While):
if not node.orelse:
node.orelse = [ast.Pass()]
return ast.unparse(tree)
@trace
def code_refine(original_code: str) -> str:
original_code = original_code.replace("python", "").replace("`", "").strip()
fixed_code = None
if infinite_loop_check(original_code):
fixed_code = error_fix(original_code)
else:
fixed_code = original_code
if syntax_error_check(fixed_code):
fixed_code = error_fix(fixed_code)
return fixed_code
@trace
def code_gen(client: AzureOpenAI, question: str) -> str:
sys_prompt = (
"I want you to act as a math expert specializing in Algebra, Geometry, and Calculus. "
"Given the question, develop python code to model the user's question. "
"Make sure only reply the executable code, no other words."
)
completion = client.chat.completions.create(
model=os.getenv("CHAT_DEPLOYMENT_NAME", "gpt-35-turbo"),
messages=[
{
"role": "system",
"content": sys_prompt,
},
{"role": "user", "content": question},
],
)
raw_code = completion.choices[0].message.content
result = code_refine(raw_code)
return result
if __name__ == "__main__":
start_trace()
if "AZURE_OPENAI_API_KEY" not in os.environ:
# load environment variables from .env file
load_dotenv()
client = AzureOpenAI(
api_key=os.getenv("AZURE_OPENAI_API_KEY"),
azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
api_version="2023-12-01-preview",
)
question = "What is 37593 * 67?"
code = code_gen(client, question)
print(code)