chore: import upstream snapshot with attribution

examples/flex-flows/chat-basic/README.md

@@ -0,0 +1,125 @@
+# Basic chat
+A basic chat flow defined using class entry. It demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message.
+
+## Prerequisites
+
+Install promptflow sdk and other dependencies in this folder:
+```bash
+pip install -r requirements.txt
+```
+
+## What you will learn
+
+In this flow, you will learn
+- how to compose a chat flow.
+- prompt template format of LLM tool chat api. Message delimiter is a separate line containing role name and colon: "system:", "user:", "assistant:".
+See <a href="https://platform.openai.com/docs/api-reference/chat/create#chat/create-role" target="_blank">OpenAI Chat</a> for more about message role.
+    ```jinja
+    system:
+    You are a chatbot having a conversation with a human.
+
+    user:
+    {{question}}
+    ```
+- how to consume chat history in prompt.
+    ```jinja
+    {% for item in chat_history %}
+    {{item.role}}:
+    {{item.content}}
+    {% endfor %}
+    ```
+
+## Run flow
+
+- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+
+- Setup connection
+
+Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of prompty supported connection types and fill in the configurations.
+
+Or use CLI to create connection:
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
+```bash
+# show registered connection
+pf connection show --name open_ai_connection
+```
+
+- Run as normal Python file
+
+```bash
+python flow.py
+```
+- Test flow
+
+```bash
+pf flow test --flow flow:ChatFlow --init init.json --inputs question="What's Azure Machine Learning?"
+```
+
+- Test flow with yaml
+You'll need to write flow entry `flow.flex.yaml` to test with prompt flow.
+
+```bash
+# run chat flow with default question in flow.flex.yaml
+pf flow test --flow . 
+# run chat flow with new question
+pf flow test --flow . --inputs question="What is ChatGPT? Please explain with consise statement."
+# run chat flow with specific init and inputs
+pf flow test --flow . --init init.json --inputs sample.json
+```
+
+- Test flow: multi turn
+```shell
+# start test in chat UI
+pf flow test --flow . --ui --init init.json
+```
+
+- Create run with multiple lines data
+
+```bash
+pf run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+- List and show run meta
+```bash
+# list created run
+pf run list
+
+# get a sample run name
+
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("chat_basic_")) | .name'| head -n 1 | tr -d '"')
+# show specific run detail
+pf run show --name $name
+
+# show output
+pf run show-details --name $name
+
+# visualize run in browser
+pf run visualize --name $name
+```
+
+## Run flow in cloud
+
+- Assume we already have a connection named `open_ai_connection` in workspace.
+
+```bash
+# set default workspace
+az account set -s <your_subscription_id>
+az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
+```
+
+- Create run
+
+```bash
+# run with environment variable reference connection in azureml workspace
+pfazure run create --flow . --init init.json --data ./data.jsonl --column-mapping question='${data.question}' --stream
+# run using yaml file
+pfazure run create --file run.yml --init init.json --stream

examples/flex-flows/chat-basic/chat-with-class-based-flow-azure.ipynb

@@ -0,0 +1,295 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Chat with class based flex flow in Azure"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Submit batch run with a flow defined with python class and evaluate it in azure.\n",
+    "\n",
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements-azure.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Connection to workspace"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Configure credential\n",
+    "\n",
+    "We are using `DefaultAzureCredential` to get access to workspace. \n",
+    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
+    "\n",
+    "Reference for more available credentials if it does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
+    "\n",
+    "try:\n",
+    "    credential = DefaultAzureCredential()\n",
+    "    # Check if given credential can get token successfully.\n",
+    "    credential.get_token(\"https://management.azure.com/.default\")\n",
+    "except Exception as ex:\n",
+    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
+    "    credential = InteractiveBrowserCredential()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Get a handle to the workspace\n",
+    "\n",
+    "We use config file to connect to a workspace. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.azure import PFClient\n",
+    "\n",
+    "# Get a handle to workspace\n",
+    "pf = PFClient.from_config(credential=credential)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "In this notebook, we will use flow `basic` flex flow which uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
+    "\n",
+    "Please go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
+    "Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run the function as flow with multi-line data\n",
+    "\n",
+    "Create a `flow.flex.yaml` file to define a flow which entry pointing to the python function we defined.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# show the flow.flex.yaml content\n",
+    "with open(\"flow.flex.yaml\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.core import AzureOpenAIModelConfiguration\n",
+    "\n",
+    "# create the model config to be used in below flow calls\n",
+    "config = AzureOpenAIModelConfiguration(\n",
+    "    connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch run with a data file (with multiple lines of test data)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow = \".\"  # path to the flow directory\n",
+    "data = \"./data.jsonl\"  # path to the data file\n",
+    "\n",
+    "# create run with the flow and data\n",
+    "base_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    init={\n",
+    "        \"model_config\": config,\n",
+    "    },\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"question\": \"${data.question}\",\n",
+    "        \"chat_history\": \"${data.chat_history}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_flow = \"../eval-checklist/flow.flex.yaml\"\n",
+    "config = AzureOpenAIModelConfiguration(\n",
+    "    connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
+    ")\n",
+    "eval_run = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    init={\n",
+    "        \"model_config\": config,\n",
+    "    },\n",
+    "    data=\"./data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"answer\": \"${run.outputs.output}\",\n",
+    "        \"statements\": \"${data.statements}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "D-W-@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "azure",
+   "section": "Flow",
+   "weight": 11
+  },
+  "description": "A quickstart tutorial to run a class based flex flow and evaluate it in azure.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  },
+  "resources": "examples/requirements-azure.txt, examples/flex-flows/basic, examples/flex-flows/eval-checklist"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/flex-flows/chat-basic/chat-with-class-based-flow.ipynb

@@ -0,0 +1,339 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Chat with class based flex flow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Write LLM application using class based flex flow.\n",
+    "- Use AzureOpenAIConnection as class init parameter.\n",
+    "- Convert the application into a flow and batch run against multi lines of data.\n",
+    "- Use classed base flow to evaluate the main flow and learn how to do aggregation.\n",
+    "\n",
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Trace your application with promptflow\n",
+    "\n",
+    "Assume we already have a python program, which leverage promptflow built-in aoai tool. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"flow.py\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "### Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "Above prompty uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
+    "\n",
+    "# client can help manage your runs and connections.\n",
+    "pf = PFClient()\n",
+    "try:\n",
+    "    conn_name = \"open_ai_connection\"\n",
+    "    conn = pf.connections.get(name=conn_name)\n",
+    "    print(\"using existing connection\")\n",
+    "except:\n",
+    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
+    "    connection = AzureOpenAIConnection(\n",
+    "        name=conn_name,\n",
+    "        api_key=\"<your_AOAI_key>\",\n",
+    "        api_base=\"<your_AOAI_endpoint>\",\n",
+    "        api_type=\"azure\",\n",
+    "    )\n",
+    "\n",
+    "    # use this if you have an existing OpenAI account\n",
+    "    # connection = OpenAIConnection(\n",
+    "    #     name=conn_name,\n",
+    "    #     api_key=\"<user-input>\",\n",
+    "    # )\n",
+    "\n",
+    "    conn = pf.connections.create_or_update(connection)\n",
+    "    print(\"successfully created connection\")\n",
+    "\n",
+    "print(conn)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.core import AzureOpenAIModelConfiguration\n",
+    "\n",
+    "# create the model config to be used in below flow calls\n",
+    "config = AzureOpenAIModelConfiguration(\n",
+    "    connection=\"open_ai_connection\", azure_deployment=\"gpt-4o\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Visualize trace by using start_trace\n",
+    "\n",
+    "Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from flow import ChatFlow\n",
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "start_trace()\n",
+    "\n",
+    "# create a chatFlow obj with connection\n",
+    "chat_flow = ChatFlow(config)\n",
+    "# run the flow as function, which will be recorded in the trace\n",
+    "result = chat_flow(question=\"What is ChatGPT? Please explain with consise statement\")\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Eval the result "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload\n",
+    "%autoreload 2\n",
+    "\n",
+    "import paths  # add the code_quality module to the path\n",
+    "from check_list import EvalFlow\n",
+    "\n",
+    "eval_flow = EvalFlow(config)\n",
+    "# evaluate answer agains a set of statement\n",
+    "eval_result = eval_flow(\n",
+    "    answer=result,\n",
+    "    statements={\n",
+    "        \"correctness\": \"It contains a detailed explanation of ChatGPT.\",\n",
+    "        \"consise\": \"It is a consise statement.\",\n",
+    "    },\n",
+    ")\n",
+    "eval_result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run the function as flow with multi-line data\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch run with a data file (with multiple lines of test data)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = \"./data.jsonl\"  # path to the data file\n",
+    "# create run with the flow function and data\n",
+    "base_run = pf.run(\n",
+    "    flow=chat_flow,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"question\": \"${data.question}\",\n",
+    "        \"chat_history\": \"${data.chat_history}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_run = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    data=\"./data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"answer\": \"${run.outputs.output}\",\n",
+    "        \"statements\": \"${data.statements}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully run your chat flow and did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Stream Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-stream): demonstrates how to create a chatbot that runs in streaming mode."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "D-W-@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Flow",
+   "weight": 11
+  },
+  "description": "A quickstart tutorial to run a class based flex flow and evaluate it.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  },
+  "resources": "examples/requirements.txt, examples/flex-flows/chat-basic, examples/flex-flows/eval-checklist"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/flex-flows/chat-basic/chat.prompty

@@ -0,0 +1,30 @@
+---
+name: Basic Chat
+model:
+  api: chat
+  configuration:
+    type: azure_openai
+    azure_deployment: gpt-4o
+  parameters:
+    temperature: 0.2
+    max_tokens: 1024
+inputs: 
+  question:
+    type: string
+  chat_history:
+    type: list
+sample:
+  question: "What is Prompt flow?"
+  chat_history: []
+---
+
+system:
+You are a helpful assistant.
+
+{% for item in chat_history %}
+{{item.role}}:
+{{item.content}}
+{% endfor %}
+
+user:
+{{question}}

examples/flex-flows/chat-basic/data.jsonl

@@ -0,0 +1,3 @@
+{"question": "What is Prompt flow?", "chat_history":[], "statements": {"correctness": "should explain what's 'Prompt flow'", "consise": "It is a consise statement."}}
+{"question": "What is ChatGPT? Please explain with consise statement", "chat_history":[],  "statements": { "correctness": "should explain what's ChatGPT", "consise": "It is a consise statement."}}
+{"question": "How many questions did user ask?", "chat_history": [{"role": "user","content": "where is the nearest coffee shop?"},{"role": "system","content": "I'm sorry, I don't know that. Would you like me to look it up for you?"}], "statements": { "correctness": "result should be 2", "consise": "It is a consise statement."}}

examples/flex-flows/chat-basic/flow.flex.yaml

@@ -0,0 +1,13 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+entry: flow:ChatFlow
+sample: 
+  inputs:
+    question: What's Azure Machine Learning?
+  init: 
+    model_config: 
+      connection: open_ai_connection
+      azure_deployment: gpt-4o
+    max_total_token: 1024
+environment:
+  # image: mcr.microsoft.com/azureml/promptflow/promptflow-python
+  python_requirements_txt: requirements.txt

examples/flex-flows/chat-basic/flow.py

@@ -0,0 +1,66 @@
+import os
+
+from pathlib import Path
+
+from promptflow.tracing import trace
+from promptflow.core import AzureOpenAIModelConfiguration, Prompty
+
+BASE_DIR = Path(__file__).absolute().parent
+
+
+def log(message: str):
+    verbose = os.environ.get("VERBOSE", "false")
+    if verbose.lower() == "true":
+        print(message, flush=True)
+
+
+class ChatFlow:
+    def __init__(
+        self, model_config: AzureOpenAIModelConfiguration, max_total_token=4096
+    ):
+        self.model_config = model_config
+        self.max_total_token = max_total_token
+
+    @trace
+    def __call__(
+        self,
+        question: str = "What's Azure Machine Learning?",
+        chat_history: list = None,
+    ) -> str:
+        """Flow entry function."""
+
+        prompty = Prompty.load(
+            source=BASE_DIR / "chat.prompty",
+            model={"configuration": self.model_config},
+        )
+
+        chat_history = chat_history or []
+        # Try to render the prompt with token limit and reduce the history count if it fails
+        while len(chat_history) > 0:
+            token_count = prompty.estimate_token_count(
+                question=question, chat_history=chat_history
+            )
+            if token_count > self.max_total_token:
+                chat_history = chat_history[1:]
+                log(
+                    f"Reducing chat history count to {len(chat_history)} to fit token limit"
+                )
+            else:
+                break
+
+        # output is a string
+        output = prompty(question=question, chat_history=chat_history)
+
+        return output
+
+
+if __name__ == "__main__":
+    from promptflow.tracing import start_trace
+
+    start_trace()
+    config = AzureOpenAIModelConfiguration(
+        connection="open_ai_connection", azure_deployment="gpt-4o"
+    )
+    flow = ChatFlow(config)
+    result = flow("What's Azure Machine Learning?", [])
+    print(result)

examples/flex-flows/chat-basic/init.json

@@ -0,0 +1,7 @@
+{
+    "model_config": {
+        "connection": "open_ai_connection",
+        "azure_deployment": "gpt-4o"
+    },
+    "max_total_token": 2048
+}

examples/flex-flows/chat-basic/paths.py

@@ -0,0 +1,6 @@
+import sys
+import pathlib
+
+# Add the path to the evaluation module
+code_path = str(pathlib.Path(__file__).parent / "../eval-checklist")
+sys.path.insert(0, code_path)

examples/flex-flows/chat-basic/requirements-azure.txt

@@ -0,0 +1 @@
+promptflow-azure

examples/flex-flows/chat-basic/requirements.txt

@@ -0,0 +1 @@
+promptflow[azure]>=1.11.0

examples/flex-flows/chat-basic/run.yml

@@ -0,0 +1,9 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
+flow: .
+data: data.jsonl
+init:
+  model_config:
+    connection: open_ai_connection
+    azure_deployment: gpt-4o
+column_mapping:
+  question: ${data.question}

examples/flex-flows/chat-basic/sample.json

@@ -0,0 +1,21 @@
+{
+    "question": "How many questions did User ask?",
+    "chat_history": [
+        {
+            "role": "user",
+            "content": "where is the nearest coffee shop?"
+        },
+        {
+            "role": "assistant",
+            "content": "I'm sorry, I don't know that. Would you like me to look it up for you?"
+        },
+        {
+            "role": "user",
+            "content": "what is the capital of France?"
+        },
+        {
+            "role": "assistant",
+            "content": "Paris"
+        }
+    ]
+}