microsoft--promptflow/examples/prompty/chat-basic/chat-with-prompty.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chat with prompty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
    "\n",
    "- Write LLM application using prompty and visualize the trace of your application.\n",
    "- Understand how to handle chat conversation using prompty\n",
    "- batch run prompty against multi lines of data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Install dependent packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install promptflow-devkit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Prompty\n",
    "\n",
    "Prompty is a file with .prompty extension for developing prompt template. \n",
    "The prompty asset is a markdown file with a modified front matter. \n",
    "The front matter is in yaml format that contains a number of metadata fields which defines model configuration and expected inputs of the prompty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"chat.prompty\") as fin:\n",
    "    print(fin.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create necessary connections\n",
    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
    "\n",
    "Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
    "\n",
    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.client import PFClient\n",
    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
    "\n",
    "# client can help manage your runs and connections.\n",
    "pf = PFClient()\n",
    "try:\n",
    "    conn_name = \"open_ai_connection\"\n",
    "    conn = pf.connections.get(name=conn_name)\n",
    "    print(\"using existing connection\")\n",
    "except:\n",
    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
    "    connection = AzureOpenAIConnection(\n",
    "        name=conn_name,\n",
    "        api_key=\"<your_AOAI_key>\",\n",
    "        api_base=\"<your_AOAI_endpoint>\",\n",
    "        api_type=\"azure\",\n",
    "    )\n",
    "\n",
    "    # use this if you have an existing OpenAI account\n",
    "    # connection = OpenAIConnection(\n",
    "    #     name=conn_name,\n",
    "    #     api_key=\"<user-input>\",\n",
    "    # )\n",
    "\n",
    "    conn = pf.connections.create_or_update(connection)\n",
    "    print(\"successfully created connection\")\n",
    "\n",
    "print(conn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Execute prompty as function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.core import Prompty\n",
    "\n",
    "# load prompty as a flow\n",
    "f = Prompty.load(\"chat.prompty\")\n",
    "# execute the flow as function\n",
    "question = \"What is the capital of France?\"\n",
    "result = f(question=question)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can override connection with `AzureOpenAIModelConfiguration` and `OpenAIModelConfiguration`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.core import AzureOpenAIModelConfiguration, OpenAIModelConfiguration\n",
    "\n",
    "\n",
    "# override configuration with created connection in AzureOpenAIModelConfiguration\n",
    "configuration = AzureOpenAIModelConfiguration(\n",
    "    connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
    ")\n",
    "\n",
    "# override openai connection with OpenAIModelConfiguration\n",
    "# configuration = OpenAIModelConfiguration(\n",
    "#     connection=connection,\n",
    "#     model=\"gpt-3.5-turbo\"\n",
    "# )\n",
    "\n",
    "override_model = {\n",
    "    \"configuration\": configuration,\n",
    "}\n",
    "\n",
    "# load prompty as a flow\n",
    "f = Prompty.load(\"chat.prompty\", model=override_model)\n",
    "# execute the flow as function\n",
    "question = \"What is the capital of France?\"\n",
    "result = f(question=question)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize trace by using start_trace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.tracing import start_trace\n",
    "\n",
    "# start a trace session, and print a url for user to check trace\n",
    "start_trace()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-run below cell will collect a trace in trace UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# rerun the function, which will be recorded in the trace\n",
    "result = f(question=question)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eval the result \n",
    "\n",
    "In this example, we will use a prompt that determines whether a chat conversation contains an apology from the assistant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_prompty = \"../eval-apology/apology.prompty\"\n",
    "\n",
    "with open(eval_prompty) as fin:\n",
    "    print(fin.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: the eval flow returns a `json_object`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load prompty as a flow\n",
    "eval_flow = Prompty.load(eval_prompty)\n",
    "# execute the flow as function\n",
    "result = eval_flow(question=question, answer=result, messages=[])\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Batch run with multi-line data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from promptflow.client import PFClient\n",
    "\n",
    "flow = \"chat.prompty\"  # path to the prompty file\n",
    "data = \"./data.jsonl\"  # path to the data file\n",
    "\n",
    "# create run with the flow and data\n",
    "pf = PFClient()\n",
    "base_run = pf.run(\n",
    "    flow=flow,\n",
    "    data=data,\n",
    "    column_mapping={\n",
    "        \"question\": \"${data.question}\",\n",
    "        \"chat_history\": \"${data.chat_history}\",\n",
    "    },\n",
    "    stream=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "details = pf.get_details(base_run)\n",
    "details.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Evaluate your prompty\n",
    "Then you can use an evaluation prompty to evaluate your prompty."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run evaluation on the previous batch run\n",
    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_run = pf.run(\n",
    "    flow=eval_prompty,\n",
    "    data=\"./data.jsonl\",  # path to the data file\n",
    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
    "    column_mapping={\n",
    "        \"messages\": \"${data.chat_history}\",\n",
    "        \"question\": \"${data.question}\",\n",
    "        \"answer\": \"${run.outputs.output}\",  # TODO refine this mapping\n",
    "    },\n",
    "    stream=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "details = pf.get_details(eval_run)\n",
    "details.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
    "\n",
    "You can check out more [Prompty Examples](https://github.com/microsoft/promptflow/tree/main/examples/prompty)."
   ]
  }
 ],
 "metadata": {
  "build_doc": {
   "author": [
    "lalala123123@github.com",
    "wangchao1230@github.com"
   ],
   "category": "local",
   "section": "Prompty",
   "weight": 20
  },
  "description": "A quickstart tutorial to run a chat prompty and evaluate it.",
  "kernelspec": {
   "display_name": "prompt_flow",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  },
  "resources": "examples/requirements.txt, examples/prompty/chat-basic, examples/prompty/eval-apology"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}