chore: import upstream snapshot with attribution

examples/flex-flows/chat-async-stream/README.md

@@ -0,0 +1,121 @@
+# Chat stream async
+A chat flow defined using async class entry that return output in stream mode. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
+
+## Prerequisites
+
+Install promptflow sdk and other dependencies in this folder:
+```bash
+pip install -r requirements.txt
+```
+
+## What you will learn
+
+In this flow, you will learn:
+- how to compose a chat flow that return output in stream mode.
+- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
+See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
+    ```jinja
+    system:
+    You are a chatbot having a conversation with a human.
+
+    user:
+    {{question}}
+    ```
+- how to consume chat history in prompt.
+    ```jinja
+    {% for item in chat_history %}
+    {{item.role}}:
+    {{item.content}}
+    {% endfor %}
+    ```
+
+## Run flow
+
+- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+
+- Setup connection
+
+Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
+
+Or use CLI to create connection:
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
+```bash
+# show registered connection
+pf connection show --name open_ai_connection
+```
+
+- Run as normal Python file
+
+```bash
+python flow.py
+```
+
+- Test flow
+
+```bash
+pf flow test --flow flow:ChatFlow --init init.json
+```
+
+- Test flow with yaml
+You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
+
+```bash
+# run chat flow with default question in flow.flex.yaml
+pf flow test --flow . --init init.json
+
+# run chat flow with new question
+pf flow test --flow . --init init.json --inputs question="What's Azure Machine Learning?"
+
+pf flow test --flow . --init init.json --inputs question="What is ChatGPT? Please explain with consise statement."
+```
+
+- Create run with multiple lines data
+
+```bash
+pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+- List and show run meta
+```bash
+# list created run
+pf run list
+
+# get a sample run name
+
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_async_stream_")) | .name'| head -n 1 | tr -d '"')
+# show specific run detail
+pf run show --name $name
+
+# show output
+pf run show-details --name $name
+
+# visualize run in browser
+pf run visualize --name $name
+```
+
+## Run flow in cloud
+
+- Assume we already have a connection named `open_ai_connection` in workspace.
+
+```bash
+# set default workspace
+az account set -s <your_subscription_id>
+az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
+```
+
+- Create run
+
+```bash
+# run with environment variable reference connection in azureml workspace
+pfazure run create --flow . --init ./init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
+# run using yaml file
+pfazure run create --file run.yml --stream

examples/flex-flows/chat-async-stream/chat-stream-with-async-flex-flow.ipynb

@@ -0,0 +1,380 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Stream chat with async flex flow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Write LLM application using class based flex flow.\n",
+    "- Use AzureOpenAIModelConfiguration as class init parameter.\n",
+    "- Use prompty to stream completions.\n",
+    "- Convert the application into a async flow and batch run against multi lines of data.\n",
+    "- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
+    "\n",
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Trace your application with promptflow\n",
+    "\n",
+    "Assume we already have a python program, which leverage prompty."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"flow.py\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When `stream=true` is configured in the parameters of a prompt whose output format is text, promptflow sdk will return a generator type, which item is the content of each chunk.\n",
+    "\n",
+    "Reference openai doc on how to do it using plain python code: [how_to_stream_completions](https://cookbook.openai.com/examples/how_to_stream_completions)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"chat.prompty\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "### Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
+    "\n",
+    "# client can help manage your runs and connections.\n",
+    "pf = PFClient()\n",
+    "try:\n",
+    "    conn_name = \"open_ai_connection\"\n",
+    "    conn = pf.connections.get(name=conn_name)\n",
+    "    print(\"using existing connection\")\n",
+    "except:\n",
+    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
+    "    connection = AzureOpenAIConnection(\n",
+    "        name=conn_name,\n",
+    "        api_key=\"<your_AOAI_key>\",\n",
+    "        api_base=\"<your_AOAI_endpoint>\",\n",
+    "        api_type=\"azure\",\n",
+    "    )\n",
+    "\n",
+    "    # use this if you have an existing OpenAI account\n",
+    "    # connection = OpenAIConnection(\n",
+    "    #     name=conn_name,\n",
+    "    #     api_key=\"<user-input>\",\n",
+    "    # )\n",
+    "\n",
+    "    conn = pf.connections.create_or_update(connection)\n",
+    "    print(\"successfully created connection\")\n",
+    "\n",
+    "print(conn)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Visualize trace by using start_trace\n",
+    "\n",
+    "Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "from promptflow.core import AzureOpenAIModelConfiguration\n",
+    "\n",
+    "from flow import ChatFlow\n",
+    "\n",
+    "# create a chatFlow obj with connection\n",
+    "config = AzureOpenAIModelConfiguration(\n",
+    "    connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
+    ")\n",
+    "chat_flow = ChatFlow(config)\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "start_trace()\n",
+    "\n",
+    "# run the flow as function, which will be recorded in the trace\n",
+    "result = chat_flow(question=\"What is ChatGPT? Please explain with detailed statement\")\n",
+    "# note the type is async generator object as we enabled stream in prompty\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "\n",
+    "# print result in stream manner\n",
+    "async for output in result:\n",
+    "    print(output, end=\"\")\n",
+    "    await asyncio.sleep(0.01)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
+    "\n",
+    "answer = \"\"\n",
+    "async for output in result:\n",
+    "    answer += output\n",
+    "answer"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Eval the result "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload\n",
+    "%autoreload 2\n",
+    "\n",
+    "import paths  # add the code_quality module to the path\n",
+    "from check_list import EvalFlow\n",
+    "\n",
+    "eval_flow = EvalFlow(config)\n",
+    "# evaluate answer agains a set of statement\n",
+    "eval_result = eval_flow(\n",
+    "    answer=answer,\n",
+    "    statements={\n",
+    "        \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
+    "        \"consise\": \"It is a consise statement.\",\n",
+    "    },\n",
+    ")\n",
+    "eval_result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run the function as flow with multi-line data\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch run with a data file (with multiple lines of test data)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = \"./data.jsonl\"  # path to the data file\n",
+    "# create run with the flow function and data\n",
+    "base_run = pf.run(\n",
+    "    flow=chat_flow,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"question\": \"${data.question}\",\n",
+    "        \"chat_history\": \"${data.chat_history}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_run = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    data=\"./data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"answer\": \"${run.outputs.output}\",\n",
+    "        \"statements\": \"${data.statements}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "D-W-@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Flow",
+   "weight": 12
+  },
+  "description": "A quickstart tutorial to run a class based flex flow in stream mode and evaluate it.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  },
+  "resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/flex-flows/chat-async-stream/chat.prompty

@@ -0,0 +1,31 @@
+---
+name: Stream Chat
+description: Chat with stream enabled.
+model:
+  api: chat
+  configuration:
+    type: azure_openai
+    azure_deployment: gpt-4o
+  parameters:
+    temperature: 0.2
+    stream: true
+    max_tokens: 1024
+inputs: 
+  question:
+    type: string
+  chat_history:
+    type: list
+sample:
+  question: "What is Prompt flow?"
+---
+
+system:
+You are a helpful assistant.
+
+{% for item in chat_history %}
+{{item.role}}:
+{{item.content}}
+{% endfor %}
+
+user:
+{{question}}

examples/flex-flows/chat-async-stream/data.jsonl

@@ -0,0 +1,3 @@
+{"question": "What is Prompt flow?", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
+{"question": "What is ChatGPT? Please explain with consise statement", "chat_history": [], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}
+{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 1", "consise": "It is a consise statement."}}

examples/flex-flows/chat-async-stream/flow.flex.yaml

@@ -0,0 +1 @@
+entry: flow:ChatFlow

examples/flex-flows/chat-async-stream/flow.py

@@ -0,0 +1,70 @@
+import asyncio
+import os
+from pathlib import Path
+
+from promptflow.tracing import trace
+from promptflow.core import AzureOpenAIModelConfiguration, Prompty
+
+BASE_DIR = Path(__file__).absolute().parent
+
+
+def log(message: str):
+    verbose = os.environ.get("VERBOSE", "false")
+    if verbose.lower() == "true":
+        print(message, flush=True)
+
+
+class ChatFlow:
+    def __init__(
+        self, model_config: AzureOpenAIModelConfiguration, max_total_token=1100
+    ):
+        self.model_config = model_config
+        self.max_total_token = max_total_token
+
+    @trace
+    async def __call__(
+        self, question: str = "What is ChatGPT?", chat_history: list = None
+    ) -> str:
+        """Flow entry function."""
+
+        prompty = Prompty.load(
+            source=BASE_DIR / "chat.prompty",
+            model={"configuration": self.model_config},
+        )
+
+        chat_history = chat_history or []
+        # Try to render the prompt with token limit and reduce the history count if it fails
+        while len(chat_history) > 0:
+            token_count = prompty.estimate_token_count(
+                question=question, chat_history=chat_history
+            )
+            if token_count > self.max_total_token:
+                chat_history = chat_history[1:]
+                log(
+                    f"Reducing chat history count to {len(chat_history)} to fit token limit"
+                )
+            else:
+                break
+
+        # output is a generator of string as prompty enabled stream parameter
+        for output in prompty(question=question, chat_history=chat_history):
+            yield output
+
+
+if __name__ == "__main__":
+    from promptflow.tracing import start_trace
+
+    start_trace()
+    config = AzureOpenAIModelConfiguration(
+        connection="open_ai_connection", azure_deployment="gpt-4o"
+    )
+    flow = ChatFlow(model_config=config)
+    result = flow("What's Azure Machine Learning?", [])
+
+    # print result in stream manner
+    async def consume_result():
+        async for output in result:
+            print(output, end="")
+            await asyncio.sleep(0.01)
+
+    asyncio.run(consume_result())

examples/flex-flows/chat-async-stream/init.json

@@ -0,0 +1,6 @@
+{
+    "model_config": {
+        "connection": "open_ai_connection",
+        "azure_deployment": "gpt-4o"
+    }
+}

examples/flex-flows/chat-async-stream/paths.py

@@ -0,0 +1,6 @@
+import sys
+import pathlib
+
+# Add the path to the evaluation module
+code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
+sys.path.insert(0, code_path)

examples/flex-flows/chat-async-stream/requirements.txt

@@ -0,0 +1 @@
+promptflow[azure]

examples/flex-flows/chat-async-stream/run.yml

@@ -0,0 +1,7 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
+flow: .
+data: data.jsonl
+init:
+  connection: open_ai_connection
+column_mapping:
+  question: ${data.question}

examples/flex-flows/chat-async-stream/sample.json

@@ -0,0 +1 @@
+{"question": "What is Prompt flow?"}