microsoft--promptflow/examples/tutorials/tracing/llm/trace-llm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tracing with LLM application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tracing is a powerful tool for understanding the behavior of your LLM application, prompt flow tracing capability supports instrumentation for such scenario.\n",
    "\n",
    "This notebook will demonstrate how to use prompt flow to instrument and understand your LLM application.\n",
    "\n",
    "**Learning Objective** - Upon completion of this notebook, you will be able to:\n",
    "\n",
    "- Trace LLM application and visualize with prompt flow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements\n",
    "\n",
    "To run this notebook example, please install required dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -r ./requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please configure your API key using an `.env` file, we have provided an example `.env.example` for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load api key and endpoint from .env to environ\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create your LLM application\n",
    "\n",
    "This notebook example will build a LLM application with Azure OpenAI service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import AzureOpenAI\n",
    "\n",
    "# in this notebook example, we will use model \"gpt-35-turbo-16k\"\n",
    "deployment_name = \"gpt-35-turbo-16k\"\n",
    "\n",
    "client = AzureOpenAI(\n",
    "    azure_deployment=deployment_name,\n",
    "    api_version=\"2024-02-01\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare one classic question for LLM\n",
    "conversation = [\n",
    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
    "    {\"role\": \"user\", \"content\": \"What is the meaning of life?\"},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.chat.completions.create(\n",
    "    messages=conversation,\n",
    "    model=deployment_name,\n",
    ")\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start trace using `promptflow.tracing.start_trace` to leverage prompt flow tracing capability; this will print a link to trace UI, where you can visualize the trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.tracing import start_trace\n",
    "\n",
    "# start a trace session, and print a url for user to check trace\n",
    "start_trace(collection=\"trace-llm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the LLM application again, and you should be able to see new trace logged in the trace UI, and it is clickable to see more details.\n",
    "\n",
    "![trace-detail](./media/trace-llm/trace-detail.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.chat.completions.create(\n",
    "    messages=conversation,\n",
    "    model=deployment_name,\n",
    ")\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "By now you have successfully tracing your LLM application with prompt flow.\n",
    "\n",
    "You can check out more examples:\n",
    "\n",
    "- [Trace LangChain](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/tracing/langchain/trace-langchain.ipynb): tracing `LangChain` and visualize leveraging prompt flow.\n",
    "- [Trace AutoGen](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/tracing/autogen-groupchat/trace-autogen-groupchat.ipynb): tracing `AutoGen` and visualize leveraging prompt flow.\n",
    "- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run.\n"
   ]
  }
 ],
 "metadata": {
  "build_doc": {
   "author": [
    "zhengfeiwang@github.com"
   ],
   "category": "local",
   "section": "Tracing",
   "weight": 10
  },
  "description": "Tracing LLM application",
  "kernelspec": {
   "display_name": "pf-dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  },
  "resources": ""
 },
 "nbformat": 4,
 "nbformat_minor": 2
}