chore: import upstream snapshot with attribution

examples/flex-flows/basic/README.md

@@ -0,0 +1,105 @@
+# Basic standard flow
+A basic standard flow define using function entry that calls Azure OpenAI with connection info stored in environment variables.
+
+## Prerequisites
+
+Install promptflow sdk and other dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+## Run flow
+
+- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+
+- Setup environment variables
+
+Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
+
+```bash
+cat ../.env
+```
+
+- Run/Debug as normal Python file
+```bash
+python programmer.py
+```
+
+- Test with flow entry
+```bash
+pf flow test --flow programmer:write_simple_program --inputs text="Java Hello World!"
+```
+
+- Test with flow yaml
+```bash
+# test with sample input value in flow.flex.yaml
+pf flow test --flow .
+```
+
+```shell
+# test with UI
+pf flow test --flow . --ui
+```
+
+- Create run with multiple lines data
+```bash
+# using environment from .env file (loaded in user code: hello.py)
+pf run create --flow . --data ./data.jsonl --column-mapping text='${data.text}' --stream
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+- List and show run meta
+```bash
+# list created run
+pf run list
+
+# get a sample run name
+
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
+# show specific run detail
+pf run show --name $name
+
+# show output
+pf run show-details --name $name
+
+# visualize run in browser
+pf run visualize --name $name
+```
+
+## Run flow in cloud with connection
+
+- Assume we already have a connection named `open_ai_connection` in workspace.
+
+```bash
+# set default workspace
+az account set -s <your_subscription_id>
+az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
+```
+
+- Create run
+```bash
+# run with environment variable reference connection in azureml workspace
+pfazure run create --flow . --data ./data.jsonl --column-mapping text='${data.text}' --environment-variables AZURE_OPENAI_API_KEY='${open_ai_connection.api_key}' AZURE_OPENAI_ENDPOINT='${open_ai_connection.api_base}' --stream
+# run using yaml file
+pfazure run create --file run.yml --stream
+```
+
+- List and show run meta
+```bash
+# list created run
+pfazure run list -r 3
+
+# get a sample run name
+name=$(pfazure run list -r 100 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
+
+# show specific run detail
+pfazure run show --name $name
+
+# show output
+pfazure run show-details --name $name
+
+# visualize run in browser
+pfazure run visualize --name $name
+```

examples/flex-flows/basic/data.jsonl

@@ -0,0 +1,3 @@
+{"text": "Python Hello World!"}
+{"text": "C Hello World!"}
+{"text": "C# Hello World!"}

examples/flex-flows/basic/flex-flow-quickstart-azure.ipynb

@@ -0,0 +1,306 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting started with flex flow in Azure"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Write an LLM application using a notebook and visualize the trace of your application.\n",
+    "- Convert the application into a flow and batch-run it against multiple lines of data.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements-azure.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Connection to workspace"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Configure credential\n",
+    "\n",
+    "We are using `DefaultAzureCredential` to access the workspace. \n",
+    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
+    "\n",
+    "Reference for other credentials if this does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
+    "\n",
+    "try:\n",
+    "    credential = DefaultAzureCredential()\n",
+    "    # Check if given credential can get token successfully.\n",
+    "    credential.get_token(\"https://management.azure.com/.default\")\n",
+    "except Exception as ex:\n",
+    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential does not work\n",
+    "    credential = InteractiveBrowserCredential()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Connect to the workspace\n",
+    "\n",
+    "We use a config file to connect to a workspace. The Azure ML workspace should be configured with a computer cluster. [Check this notebook for how to configure a workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.azure import PFClient\n",
+    "\n",
+    "# Connect to the workspace\n",
+    "pf = PFClient.from_config(credential=credential)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create necessary connections\n",
+    "A connection helps securely store and manage secret keys or other sensitive credentials required for interacting with the LLM and other external tools, for example Azure Content Safety.\n",
+    "\n",
+    "In this notebook, we will use the `basic` & `eval-code-quality` flex flow, which uses the connection `open_ai_connection`.  We need to set up the connection if we haven't added it before.\n",
+    "\n",
+    "To prepare your Azure OpenAI resource, follow these [instructions](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
+    "\n",
+    "Go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
+    "Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run the function as a flow with multi-line data.\n",
+    "\n",
+    "Create a `flow.flex.yaml` file to define a flow whose entry points to the python function we defined.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Show the flow.flex.yaml content\n",
+    "with open(\"flow.flex.yaml\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch run with a data file (with multiple lines of test data)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow = \".\"  # Path to the flow directory\n",
+    "data = \"./data.jsonl\"  # Path to the data file\n",
+    "\n",
+    "# Create a run with the flow and data\n",
+    "base_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"text\": \"${data.text}\",\n",
+    "    },\n",
+    "    environment_variables={\n",
+    "        \"AZURE_OPENAI_API_KEY\": \"${open_ai_connection.api_key}\",\n",
+    "        \"AZURE_OPENAI_ENDPOINT\": \"${open_ai_connection.api_base}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually use an LLM to verify the produced output matches the expected output. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Setup model configuration with connection\n",
+    "\n",
+    "When using Promptflow in Azure, create a model configuration object with connection name. \n",
+    "The model config will connect to the cloud-hosted Promptflow instance while running the flow."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.core import AzureOpenAIModelConfiguration\n",
+    "\n",
+    "model_config = AzureOpenAIModelConfiguration(\n",
+    "    connection=\"open_ai_connection\",\n",
+    "    azure_deployment=\"gpt-4o\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Evaluate the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input.  The evaluation takes the outputs of that **base_run**, and uses an LLM to compare them to your desired outputs, and then visualizes the results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_flow = \"../eval-code-quality/flow.flex.yaml\"\n",
+    "\n",
+    "eval_run = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    init={\"model_config\": model_config},\n",
+    "    data=\"./data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify the base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"code\": \"${run.outputs.output}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "You've successfully run your first flex flow and evaluated it. That's great!\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate the next message."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "D-W-@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "azure",
+   "section": "Flow",
+   "weight": 10
+  },
+  "description": "A quickstart tutorial to run a flex flow and evaluate it in Azure.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  },
+  "resources": "examples/requirements-azure.txt, examples/flex-flows/basic, examples/flex-flows/eval-code-quality"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/flex-flows/basic/flex-flow-quickstart.ipynb

@@ -0,0 +1,385 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting started with flex flow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Write LLM application using notebook and visualize the trace of your application.\n",
+    "- Convert the application into a flow and batch run against multi lines of data.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Trace your application with promptflow\n",
+    "\n",
+    "Assume we already have a python function that calls OpenAI API. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"llm.py\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note: before running below cell, please configure required environment variable `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` by create an `.env` file. Please refer to `../.env.example` as an template."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# control the AOAI deployment (model) used in this example\n",
+    "deployment_name = \"gpt-4o\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llm import my_llm_tool\n",
+    "\n",
+    "# pls configure `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` environment variables first\n",
+    "result = my_llm_tool(\n",
+    "    prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
+    "    deployment_name=deployment_name,\n",
+    ")\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Visualize trace by using start_trace\n",
+    "\n",
+    "Note we add `@trace` in the `my_llm_tool` function, re-run below cell will collect a trace in trace UI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "start_trace()\n",
+    "# rerun the function, which will be recorded in the trace\n",
+    "result = my_llm_tool(\n",
+    "    prompt=\"Write a simple Hello, world! program that displays the greeting message when executed. Output code only.\",\n",
+    "    deployment_name=deployment_name,\n",
+    ")\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, let's add another layer of function call. In `programmer.py` there is a function called `write_simple_program`, which calls a new function called `load_prompt` and previous `my_llm_tool` function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# show the programmer.py content\n",
+    "with open(\"programmer.py\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# call the flow entry function\n",
+    "from programmer import write_simple_program\n",
+    "\n",
+    "result = write_simple_program(\"Java Hello, world!\")\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Setup model configuration with environment variables\n",
+    "\n",
+    "When used in local, create a model configuration object with environment variables."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "from promptflow.core import AzureOpenAIModelConfiguration\n",
+    "\n",
+    "if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
+    "    # load environment variables from .env file\n",
+    "    load_dotenv()\n",
+    "\n",
+    "if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
+    "    raise Exception(\"Please specify environment variables: AZURE_OPENAI_API_KEY\")\n",
+    "model_config = AzureOpenAIModelConfiguration(\n",
+    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
+    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
+    "    azure_deployment=deployment_name,\n",
+    "    api_version=\"2023-07-01-preview\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Eval the result "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%load_ext autoreload\n",
+    "%autoreload 2\n",
+    "\n",
+    "import paths  # add the code_quality module to the path\n",
+    "from code_quality import CodeEvaluator\n",
+    "\n",
+    "evaluator = CodeEvaluator(model_config=model_config)\n",
+    "eval_result = evaluator(result)\n",
+    "eval_result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run the function as flow with multi-line data\n",
+    "\n",
+    "Create a [flow.flex.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flow.flex.yaml) file to define a flow which entry pointing to the python function we defined.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# show the flow.flex.yaml content\n",
+    "with open(\"flow.flex.yaml\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch run with a data file (with multiple lines of test data)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = \"./data.jsonl\"  # path to the data file\n",
+    "# create run with the flow function and data\n",
+    "base_run = pf.run(\n",
+    "    flow=write_simple_program,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"text\": \"${data.text}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# we can also run flow pointing to yaml file\n",
+    "eval_flow = \"../eval-code-quality/flow.flex.yaml\"\n",
+    "\n",
+    "eval_run = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    init={\"model_config\": model_config},\n",
+    "    data=\"./data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"code\": \"${run.outputs.output}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "D-W-@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Flow",
+   "weight": 10
+  },
+  "description": "A quickstart tutorial to run a flex flow and evaluate it.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  },
+  "resources": "examples/requirements.txt, examples/flex-flows/basic"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/flex-flows/basic/flow.flex.yaml

@@ -0,0 +1,7 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/flow.schema.json
+entry: programmer:write_simple_program
+environment:
+  python_requirements_txt: requirements.txt
+sample:
+  inputs:
+    text: Java Hello World!

examples/flex-flows/basic/hello.jinja2

@@ -0,0 +1,3 @@
+system: 
+Write a simple {{text}} program.
+Output code only.

examples/flex-flows/basic/llm.py

@@ -0,0 +1,64 @@
+import os
+
+from dotenv import load_dotenv
+from openai.version import VERSION as OPENAI_VERSION
+
+from promptflow.tracing import trace
+
+
+def get_client():
+    if OPENAI_VERSION.startswith("0."):
+        raise Exception(
+            "Please upgrade your OpenAI package to version >= 1.0.0 or using the command: pip install --upgrade openai."
+        )
+    api_key = os.environ.get("OPENAI_API_KEY", None)
+    if api_key:
+        from openai import OpenAI
+
+        return OpenAI()
+    else:
+        from openai import AzureOpenAI
+
+        return AzureOpenAI(
+            api_version=os.environ.get("OPENAI_API_VERSION", "2023-07-01-preview")
+        )
+
+
+@trace
+def my_llm_tool(
+    prompt: str,
+    # for AOAI, deployment name is customized by user, not model name.
+    deployment_name: str,
+    max_tokens: int = 120,
+    temperature: float = 1.0,
+    top_p: float = 1.0,
+    n: int = 1,
+) -> str:
+    if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
+        # load environment variables from .env file
+        load_dotenv()
+
+    if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
+        raise Exception(
+            "Please specify environment variables: OPENAI_API_KEY or AZURE_OPENAI_API_KEY"
+        )
+    messages = [{"content": prompt, "role": "system"}]
+    response = get_client().chat.completions.create(
+        messages=messages,
+        model=deployment_name,
+        max_tokens=int(max_tokens),
+        temperature=float(temperature),
+        top_p=float(top_p),
+        n=int(n),
+    )
+
+    # get first element because prompt is single.
+    return response.choices[0].message.content
+
+
+if __name__ == "__main__":
+    result = my_llm_tool(
+        prompt="Write a simple Hello, world! program that displays the greeting message.",
+        deployment_name="gpt-4o",
+    )
+    print(result)

examples/flex-flows/basic/paths.py

@@ -0,0 +1,6 @@
+import sys
+import pathlib
+
+# Add the path to the evaluation code quality module
+code_path = str(pathlib.Path(__file__).parent / "../eval-code-quality")
+sys.path.insert(0, code_path)

examples/flex-flows/basic/programmer.py

@@ -0,0 +1,41 @@
+from pathlib import Path
+from typing import TypedDict
+
+from jinja2 import Template
+from llm import my_llm_tool
+
+from promptflow.tracing import trace
+
+BASE_DIR = Path(__file__).absolute().parent
+
+
+class Result(TypedDict):
+    output: str
+
+
+@trace
+def load_prompt(jinja2_template: str, text: str) -> str:
+    """Load prompt function."""
+    with open(BASE_DIR / jinja2_template, "r", encoding="utf-8") as f:
+        prompt = Template(
+            f.read(), trim_blocks=True, keep_trailing_newline=True
+        ).render(text=text)
+        return prompt
+
+
+@trace
+def write_simple_program(
+    text: str = "Hello World!", deployment_name="gpt-4o"
+) -> Result:
+    """Ask LLM to write a simple program."""
+    prompt = load_prompt("hello.jinja2", text)
+    output = my_llm_tool(prompt=prompt, deployment_name=deployment_name, max_tokens=120)
+    return Result(output=output)
+
+
+if __name__ == "__main__":
+    from promptflow.tracing import start_trace
+
+    start_trace()
+    result = write_simple_program("Hello, world!", "gpt-4o")
+    print(result)

examples/flex-flows/basic/requirements-azure.txt

@@ -0,0 +1 @@
+promptflow-azure

examples/flex-flows/basic/requirements.txt

@@ -0,0 +1,3 @@
+promptflow[azure]
+python-dotenv
+openai>=1.14.0

examples/flex-flows/basic/run.yml

@@ -0,0 +1,10 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
+flow: .
+data: data.jsonl
+column_mapping:
+  text: ${data.text}
+environment_variables:
+  # environment variables from connection
+  AZURE_OPENAI_API_KEY: ${open_ai_connection.api_key}
+  AZURE_OPENAI_ENDPOINT: ${open_ai_connection.api_base}
+  AZURE_OPENAI_API_TYPE: azure