chore: import upstream snapshot with attribution

examples/prompty/chat-basic/README.md

@@ -0,0 +1,94 @@
+# Basic chat
+A prompt that uses the chat API to answer questions with chat history, leveraging promptflow connection.
+
+
+## Prerequisites
+
+Install `promptflow-devkit`:
+```bash
+pip install promptflow-devkit
+```
+
+## What you will learn
+
+In this flow, you will learn
+- how to compose a chat flow.
+- prompt template format of chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
+See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
+    ```jinja
+    system:
+    You are a chatbot having a conversation with a human.
+
+    user:
+    {{question}}
+    ```
+- how to consume chat history in prompt.
+    ```jinja
+    {% for item in chat_history %}
+    {{item.role}}:
+    {{item.content}}
+    {% endfor %}
+    ```
+
+## Getting started
+
+### Create connection for prompty to use
+Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
+
+Currently, there are two connection types supported by LLM tool: "AzureOpenAI" and "OpenAI". If you want to use "AzureOpenAI" connection type, you need to create an Azure OpenAI service first. Please refer to [Azure OpenAI Service](https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/) for more details. If you want to use "OpenAI" connection type, you need to create an OpenAI account first. Please refer to [OpenAI](https://platform.openai.com/) for more details.
+
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base>
+```
+
+Note in [chat.prompty](chat.prompty) we are using connection named `open_ai_connection`.
+```bash
+# show registered connection
+pf connection show --name open_ai_connection
+```
+
+## Run prompty
+
+- Test flow: single turn
+```bash
+# run chat flow with default question in flow.flex.yaml
+pf flow test --flow chat.prompty
+
+# run chat flow with new question
+pf flow test --flow chat.prompty --inputs question="What's Azure Machine Learning?"
+
+# run chat flow with sample.json
+pf flow test --flow chat.prompty --inputs sample.json
+```
+
+- Test flow: multi turn
+```shell
+# start test in chat ui
+pf flow test --flow chat.prompty --ui
+```
+
+- Create run with multiple lines data
+```bash
+# using environment from .env file (loaded in user code: hello.py)
+pf run create --flow chat.prompty --data ./data.jsonl --column-mapping question='${data.question}' --stream
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+- List and show run meta
+```bash
+# list created run
+pf run list
+
+# get a sample run name
+
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_basic_")) | .name'| head -n 1 | tr -d '"')
+# show specific run detail
+pf run show --name $name
+
+# show output
+pf run show-details --name $name
+```

examples/prompty/chat-basic/chat-with-prompty.ipynb

@@ -0,0 +1,378 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Chat with prompty"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Write LLM application using prompty and visualize the trace of your application.\n",
+    "- Understand how to handle chat conversation using prompty\n",
+    "- batch run prompty against multi lines of data.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install promptflow-devkit"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Prompty\n",
+    "\n",
+    "Prompty is a file with .prompty extension for developing prompt template. \n",
+    "The prompty asset is a markdown file with a modified front matter. \n",
+    "The front matter is in yaml format that contains a number of metadata fields which defines model configuration and expected inputs of the prompty."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"chat.prompty\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
+    "\n",
+    "# client can help manage your runs and connections.\n",
+    "pf = PFClient()\n",
+    "try:\n",
+    "    conn_name = \"open_ai_connection\"\n",
+    "    conn = pf.connections.get(name=conn_name)\n",
+    "    print(\"using existing connection\")\n",
+    "except:\n",
+    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
+    "    connection = AzureOpenAIConnection(\n",
+    "        name=conn_name,\n",
+    "        api_key=\"<your_AOAI_key>\",\n",
+    "        api_base=\"<your_AOAI_endpoint>\",\n",
+    "        api_type=\"azure\",\n",
+    "    )\n",
+    "\n",
+    "    # use this if you have an existing OpenAI account\n",
+    "    # connection = OpenAIConnection(\n",
+    "    #     name=conn_name,\n",
+    "    #     api_key=\"<user-input>\",\n",
+    "    # )\n",
+    "\n",
+    "    conn = pf.connections.create_or_update(connection)\n",
+    "    print(\"successfully created connection\")\n",
+    "\n",
+    "print(conn)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Execute prompty as function"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.core import Prompty\n",
+    "\n",
+    "# load prompty as a flow\n",
+    "f = Prompty.load(\"chat.prompty\")\n",
+    "# execute the flow as function\n",
+    "question = \"What is the capital of France?\"\n",
+    "result = f(question=question)\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can override connection with `AzureOpenAIModelConfiguration` and `OpenAIModelConfiguration`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.core import AzureOpenAIModelConfiguration, OpenAIModelConfiguration\n",
+    "\n",
+    "\n",
+    "# override configuration with created connection in AzureOpenAIModelConfiguration\n",
+    "configuration = AzureOpenAIModelConfiguration(\n",
+    "    connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
+    ")\n",
+    "\n",
+    "# override openai connection with OpenAIModelConfiguration\n",
+    "# configuration = OpenAIModelConfiguration(\n",
+    "#     connection=connection,\n",
+    "#     model=\"gpt-3.5-turbo\"\n",
+    "# )\n",
+    "\n",
+    "override_model = {\n",
+    "    \"configuration\": configuration,\n",
+    "}\n",
+    "\n",
+    "# load prompty as a flow\n",
+    "f = Prompty.load(\"chat.prompty\", model=override_model)\n",
+    "# execute the flow as function\n",
+    "question = \"What is the capital of France?\"\n",
+    "result = f(question=question)\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Visualize trace by using start_trace"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "start_trace()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Re-run below cell will collect a trace in trace UI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# rerun the function, which will be recorded in the trace\n",
+    "result = f(question=question)\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Eval the result \n",
+    "\n",
+    "In this example, we will use a prompt that determines whether a chat conversation contains an apology from the assistant."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_prompty = \"../eval-apology/apology.prompty\"\n",
+    "\n",
+    "with open(eval_prompty) as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note: the eval flow returns a `json_object`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load prompty as a flow\n",
+    "eval_flow = Prompty.load(eval_prompty)\n",
+    "# execute the flow as function\n",
+    "result = eval_flow(question=question, answer=result, messages=[])\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run with multi-line data\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "\n",
+    "flow = \"chat.prompty\"  # path to the prompty file\n",
+    "data = \"./data.jsonl\"  # path to the data file\n",
+    "\n",
+    "# create run with the flow and data\n",
+    "pf = PFClient()\n",
+    "base_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"question\": \"${data.question}\",\n",
+    "        \"chat_history\": \"${data.chat_history}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your prompty\n",
+    "Then you can use an evaluation prompty to evaluate your prompty."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_run = pf.run(\n",
+    "    flow=eval_prompty,\n",
+    "    data=\"./data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"messages\": \"${data.chat_history}\",\n",
+    "        \"question\": \"${data.question}\",\n",
+    "        \"answer\": \"${run.outputs.output}\",  # TODO refine this mapping\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out more [Prompty Examples](https://github.com/microsoft/promptflow/tree/main/examples/prompty)."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "lalala123123@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Prompty",
+   "weight": 20
+  },
+  "description": "A quickstart tutorial to run a chat prompty and evaluate it.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": "examples/requirements.txt, examples/prompty/chat-basic, examples/prompty/eval-apology"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/prompty/chat-basic/chat.prompty

@@ -0,0 +1,36 @@
+---
+name: Chat Prompt
+description: A basic prompt that uses the chat API to answer questions with chat_history
+model:
+    api: chat
+    configuration:
+        type: azure_openai
+        connection: open_ai_connection
+        azure_deployment: gpt-4o
+    parameters:
+        max_tokens: 256
+        temperature: 0.2
+
+inputs:
+    question:
+        type: string
+    chat_history:
+        type: list
+        default: []
+sample:
+    question: What is the meaning of life?
+    chat_history: []
+
+---
+system:
+You are an AI assistant who helps people find information.
+As the assistant, you answer questions briefly, succinctly, 
+and in a personable manner using markdown and even add some personal flair with appropriate emojis.
+
+{% for item in chat_history %}
+{{item.role}}:
+{{item.content}}
+{% endfor %}
+
+user:
+{{question}}

examples/prompty/chat-basic/data.jsonl

@@ -0,0 +1,3 @@
+{"question": "What's chat-GPT?", "chat_history": []}
+{"question": "How many questions did John Doe ask?", "chat_history": []}
+{"question": "How many questions did John Doe ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "assistant","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}]}

examples/prompty/chat-basic/sample.json

@@ -0,0 +1,21 @@
+{
+  "question": "How many questions did the user ask?",
+  "chat_history": [
+    {
+      "role": "user",
+      "content": "where is the nearest coffee shop?"
+    },
+    {
+      "role": "assistant",
+      "content": "I'm sorry, I don't know that. Would you like me to look it up for you?"
+    },
+    {
+      "role": "user",
+      "content": "what's the capital of France?"
+    },
+    {
+      "role": "assistant",
+      "content": "Paris"
+    }
+  ]
+}