microsoft--promptflow/examples/tutorials/tracing/autogen-groupchat/trace-autogen-groupchat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tracing with AutoGen"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation.\n",
    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
    "\n",
    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
    "\n",
    "- Trace LLM (OpenAI) Calls and visualize the trace of your application.\n",
    "\n",
    "## Requirements\n",
    "\n",
    "AutoGen requires `Python>=3.8`. To run this notebook example, please install required dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -r ./requirements.txt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set your API endpoint\n",
    "\n",
    "You can create the config file named `OAI_CONFIG_LIST.json` from example file: `OAI_CONFIG_LIST.json.example`.\n",
    "\n",
    "Below code use the [`config_list_from_json`](https://microsoft.github.io/autogen/0.2/docs/reference/oai/openai_utils/#config_list_from_json) function loads a list of configurations from an environment variable or a json file. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import autogen\n",
    "\n",
    "# please ensure you have a json config file\n",
    "env_or_file = \"OAI_CONFIG_LIST.json\"\n",
    "\n",
    "# filters the configs by models (you can filter by other keys as well). Only the gpt-4 models are kept in the list based on the filter condition.\n",
    "\n",
    "# gpt4\n",
    "# config_list = autogen.config_list_from_json(\n",
    "#     env_or_file,\n",
    "#     filter_dict={\n",
    "#         \"model\": [\"gpt-4\", \"gpt-4-0314\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
    "#     },\n",
    "# )\n",
    "\n",
    "# gpt35\n",
    "config_list = autogen.config_list_from_json(\n",
    "    env_or_file,\n",
    "    filter_dict={\n",
    "        \"model\": {\n",
    "            \"gpt-35-turbo\",\n",
    "            \"gpt-3.5-turbo\",\n",
    "            \"gpt-3.5-turbo-16k\",\n",
    "            \"gpt-3.5-turbo-0301\",\n",
    "            \"chatgpt-35-turbo-0301\",\n",
    "            \"gpt-35-turbo-v0301\",\n",
    "        },\n",
    "    },\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Construct agents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"AUTOGEN_USE_DOCKER\"] = \"False\"\n",
    "\n",
    "llm_config = {\"config_list\": config_list, \"cache_seed\": 42}\n",
    "user_proxy = autogen.UserProxyAgent(\n",
    "    name=\"User_proxy\",\n",
    "    system_message=\"A human admin.\",\n",
    "    code_execution_config={\n",
    "        \"last_n_messages\": 2,\n",
    "        \"work_dir\": \"groupchat\",\n",
    "        \"use_docker\": False,\n",
    "    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\n",
    "    human_input_mode=\"TERMINATE\",\n",
    ")\n",
    "coder = autogen.AssistantAgent(\n",
    "    name=\"Coder\",\n",
    "    llm_config=llm_config,\n",
    ")\n",
    "pm = autogen.AssistantAgent(\n",
    "    name=\"Product_manager\",\n",
    "    system_message=\"Creative in software product ideas.\",\n",
    "    llm_config=llm_config,\n",
    ")\n",
    "groupchat = autogen.GroupChat(agents=[user_proxy, coder, pm], messages=[], max_round=12)\n",
    "manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start chat with promptflow trace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.tracing import start_trace\n",
    "\n",
    "# start a trace session, and print a url for user to check trace\n",
    "# traces will be collected into below collection name\n",
    "start_trace(collection=\"autogen-groupchat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the url you get in start_trace output, when running below code, you will be able to see new traces in the UI. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from opentelemetry import trace\n",
    "import json\n",
    "\n",
    "\n",
    "tracer = trace.get_tracer(\"my_tracer\")\n",
    "# Create a root span\n",
    "with tracer.start_as_current_span(\"autogen\") as span:\n",
    "    message = \"Find a latest paper about gpt-4 on arxiv and find its potential applications in software.\"\n",
    "    user_proxy.initiate_chat(\n",
    "        manager,\n",
    "        message=message,\n",
    "        clear_history=True,\n",
    "    )\n",
    "    span.set_attribute(\"custom\", \"custom attribute value\")\n",
    "    # recommend to store inputs and outputs as events\n",
    "    span.add_event(\n",
    "        \"promptflow.function.inputs\", {\"payload\": json.dumps(dict(message=message))}\n",
    "    )\n",
    "    span.add_event(\n",
    "        \"promptflow.function.output\", {\"payload\": json.dumps(user_proxy.last_message())}\n",
    "    )\n",
    "# type exit to terminate the chat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "By now you've successfully tracing LLM calls in your app using prompt flow.\n",
    "\n",
    "You can check out more examples:\n",
    "- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run."
   ]
  }
 ],
 "metadata": {
  "build_doc": {
   "author": [
    "zhengfeiwang@github.com",
    "wangchao1230@github.com"
   ],
   "category": "local",
   "section": "Tracing",
   "weight": 20
  },
  "description": "Tracing LLM calls in autogen group chat application",
  "kernelspec": {
   "display_name": "prompt_flow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "resources": ""
 },
 "nbformat": 4,
 "nbformat_minor": 2
}