chore: import upstream snapshot with attribution

examples/prompty/basic/README.md

@@ -0,0 +1,94 @@
+# Basic prompty
+A basic prompt that uses the chat API to answer questions, with connection configured using environment variables.
+
+## Prerequisites
+
+Install `promptflow-devkit`:
+```bash
+pip install promptflow-devkit
+```
+
+## Run prompty
+
+- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+- Setup environment variables
+
+Ensure you have put your azure OpenAI endpoint key in [.env](../.env) file. You can create one refer to this [example file](../.env.example).
+
+```bash
+cat ../.env
+```
+
+- Test prompty
+```bash
+# test with default sample data 
+# --env to use environment variable from .env 
+pf flow test --flow basic.prompty --env
+
+# test with flow inputs
+pf flow test --flow basic.prompty --env --inputs question="What is the meaning of life?"
+
+# test with another sample data
+pf flow test --flow basic.prompty --env --inputs sample.json
+```
+
+- Create run with multiple lines data
+```bash
+# using environment from .env file
+pf run create --flow basic.prompty --env --data ./data.jsonl --column-mapping question='${data.question}' --stream
+```
+
+You can also skip providing `column-mapping` if provided data has same column name as the flow.
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+- List and show run meta
+```bash
+# list created run
+pf run list
+
+# get a sample run name
+
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
+# show specific run detail
+pf run show --name $name
+
+# show output
+pf run show-details --name $name
+
+# visualize run in browser
+pf run visualize --name $name
+```
+
+## Run prompty with connection
+
+Storing connection info in .env with plaintext is not safe. We recommend to use `pf connection` to guard secrets like `api_key` from leak.
+
+- Show or create `open_ai_connection`
+```bash
+# create connection from `azure_openai.yml` file
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base>
+
+# check if connection exists
+pf connection show -n open_ai_connection
+```
+
+- Test using connection secret specified in environment variables
+**Note**: we used `'` to wrap value since it supports raw value without escape in powershell & bash. For windows command prompt, you may remove the `'` to avoid it become part of the value.
+
+```bash
+# test with default input value in flow.flex.yaml
+pf flow test --flow basic.prompty --inputs sample.json --environment-variables AZURE_OPENAI_API_KEY='${open_ai_connection.api_key}' AZURE_OPENAI_ENDPOINT='${open_ai_connection.api_base}'
+```
+
+- Create run using connection secret binding specified in environment variables, see [run.yml](run.yml)
+```bash
+# create run
+pf run create --flow basic.prompty --data ./data.jsonl --stream --environment-variables AZURE_OPENAI_API_KEY='${open_ai_connection.api_key}' AZURE_OPENAI_ENDPOINT='${open_ai_connection.api_base}' --column-mapping question='${data.question}'
+# create run using yaml file
+pf run create --file run.yml --stream
+
+# show outputs
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("basic_")) | .name'| head -n 1 | tr -d '"')
+pf run show-details --name $name
+```

examples/prompty/basic/basic.prompty

@@ -0,0 +1,23 @@
+---
+name: Basic Prompt
+description: A basic prompt that uses the chat API to answer questions
+model:
+    api: chat
+    configuration:
+        type: azure_openai
+        azure_deployment: gpt-4o
+    parameters:
+        max_tokens: 128
+        temperature: 0.2
+inputs:
+  question:
+    type: string
+sample:
+  "question": "Who is the most famous person in the world?"
+---
+system:
+You are an AI assistant who helps people find information.
+As the assistant, you answer questions briefly, succinctly. 
+
+user:
+{{question}}

examples/prompty/basic/data.jsonl

@@ -0,0 +1,3 @@
+{"question": "What is capital of France?", "ground_truth": "Paris"}
+{"question": "What is the meaning of life?", "ground_truth": "The meaning of life is subjective and can vary greatly depending on one's personal beliefs. Some people may find meaning through personal growth, love, or contribution to others, while others may find it through religious or spiritual beliefs. Ultimately, the meaning of life is a deeply personal and subjective concept."}
+{"question": "What are the planets in Sun system?", "ground_truth":"The planets in the Solar System are Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune."}

examples/prompty/basic/prompty-quickstart.ipynb

@@ -0,0 +1,357 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting started with prompty"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Write LLM application using prompty and visualize the trace of your application.\n",
+    "- batch run prompty against multi lines of data.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install promptflow-core"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Execute a Prompty\n",
+    "\n",
+    "Prompty is a file with .prompty extension for developing prompt template. \n",
+    "The prompty asset is a markdown file with a modified front matter. \n",
+    "The front matter is in yaml format that contains a number of metadata fields which defines model configuration and expected inputs of the prompty."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open(\"basic.prompty\") as fin:\n",
+    "    print(fin.read())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note: before running below cell, please configure required environment variable `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` by create an `.env` file. Please refer to `../.env.example` as an template.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
+    "    # load environment variables from .env file\n",
+    "    load_dotenv()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.core import Prompty\n",
+    "\n",
+    "# load prompty as a flow\n",
+    "f = Prompty.load(source=\"basic.prompty\")\n",
+    "\n",
+    "# execute the flow as function\n",
+    "result = f(question=\"What is the capital of France?\")\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can override configuration with `AzureOpenAIModelConfiguration` and `OpenAIModelConfiguration`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.core import AzureOpenAIModelConfiguration, OpenAIModelConfiguration\n",
+    "\n",
+    "# override configuration with AzureOpenAIModelConfiguration\n",
+    "configuration = AzureOpenAIModelConfiguration(\n",
+    "    # azure_endpoint=\"${env:AZURE_OPENAI_ENDPOINT}\",  # Use ${env:<ENV_NAME>} to surround the environment variable name.\n",
+    "    # api_key=\"${env:AZURE_OPENAI_API_KEY}\",\n",
+    "    azure_deployment=\"gpt-4o\",\n",
+    ")\n",
+    "\n",
+    "# override configuration with OpenAIModelConfiguration\n",
+    "# configuration = OpenAIModelConfiguration(\n",
+    "#     base_url=\"${env:OPENAI_BASE_URL}\",\n",
+    "#     api_key=\"${env:OPENAI_API_KEY}\",\n",
+    "#     model=\"gpt-3.5-turbo\"\n",
+    "# )\n",
+    "\n",
+    "override_model = {\"configuration\": configuration, \"parameters\": {\"max_tokens\": 512}}\n",
+    "\n",
+    "# load prompty as a flow\n",
+    "f = Prompty.load(source=\"basic.prompty\", model=override_model)\n",
+    "\n",
+    "# execute the flow as function\n",
+    "result = f(question=\"What is the capital of France?\")\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Visualize trace by using start_trace"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "start_trace()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Re-run below cell will collect a trace in trace UI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# rerun the function, which will be recorded in the trace\n",
+    "question = \"What is the capital of Japan?\"\n",
+    "ground_truth = \"Tokyo\"\n",
+    "result = f(question=question)\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Eval the result \n",
+    "\n",
+    "Note: the eval flow returns a `json_object`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load prompty as a flow\n",
+    "eval_flow = Prompty.load(\"../eval-basic/eval.prompty\")\n",
+    "# execute the flow as function\n",
+    "result = eval_flow(question=question, ground_truth=ground_truth, answer=result)\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Batch run with multi-line data\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "# batch run requires promptflow-devkit package\n",
+    "%pip install promptflow-devkit"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import PFClient\n",
+    "\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow = \"./basic.prompty\"  # path to the prompty file\n",
+    "data = \"./data.jsonl\"  # path to the data file\n",
+    "\n",
+    "# create run with the flow and data\n",
+    "base_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"question\": \"${data.question}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which usually using LLM assert the produced output matches certain expectation. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_prompty = \"../eval-basic/eval.prompty\"\n",
+    "\n",
+    "eval_run = pf.run(\n",
+    "    flow=eval_prompty,\n",
+    "    data=\"./data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"question\": \"${data.question}\",\n",
+    "        \"answer\": \"${run.outputs.output}\",  # TODO refine this mapping\n",
+    "        \"ground_truth\": \"${data.ground_truth}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# visualize run using ui\n",
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/prompty/chat-basic): demonstrates how to create a chatbot that can remember previous interactions and use the conversation history to generate next message."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "lalala123123@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Prompty",
+   "weight": 10
+  },
+  "description": "A quickstart tutorial to run a prompty and evaluate it.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  },
+  "resources": "examples/requirements.txt, examples/prompty/basic, examples/prompty/eval-basic"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/prompty/basic/run.yml

@@ -0,0 +1,10 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Run.schema.json
+flow: basic.prompty
+data: data.jsonl
+environment_variables:
+  # environment variables from connection
+  AZURE_OPENAI_API_KEY: ${open_ai_connection.api_key}
+  AZURE_OPENAI_ENDPOINT: ${open_ai_connection.api_base}
+  AZURE_OPENAI_API_TYPE: azure
+column_mapping:
+  question: ${data.question}

examples/prompty/basic/sample.json

@@ -0,0 +1,3 @@
+{
+  "question": "Who is the most famous person in the world?"
+}