chore: import upstream snapshot with attribution

examples/tutorials/get-started/quickstart.ipynb

@@ -0,0 +1,464 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting started with DAG flow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Prerequisite** - To make the most of this tutorial, you'll need:\n",
+    "- A local clone of the prompt flow repository\n",
+    "- A Python environment with Jupyter Notebook support (such as Jupyter Lab or the Python extension for Visual Studio Code)\n",
+    "- Know how to program with Python :)\n",
+    "\n",
+    "_A basic understanding of Machine Learning can be beneficial, but it's not mandatory._\n",
+    "\n",
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Run your first prompt flow sample\n",
+    "- Run your first evaluation\n",
+    "\n",
+    "\n",
+    "The sample used in this tutorial is the [web-classification](https://github.com/microsoft/promptflow/tree/main/examples/flows/standard/web-classification) flow, which categorizes URLs into several predefined classes. Classification is a traditional machine learning task, and this sample illustrates how to perform classification using GPT and prompts."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -r ../../requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "In this notebook, we will use flow `web-classification` which uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from promptflow.client import PFClient\n",
+    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
+    "\n",
+    "# client can help manage your runs and connections.\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    conn_name = \"open_ai_connection\"\n",
+    "    conn = pf.connections.get(name=conn_name)\n",
+    "    print(\"using existing connection\")\n",
+    "except:\n",
+    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
+    "    connection = AzureOpenAIConnection(\n",
+    "        name=conn_name,\n",
+    "        api_key=\"<test_key>\",\n",
+    "        api_base=\"<test_base>\",\n",
+    "        api_type=\"azure\",\n",
+    "        api_version=\"<test_version>\",\n",
+    "    )\n",
+    "\n",
+    "    # use this if you have an existing OpenAI account\n",
+    "    # connection = OpenAIConnection(\n",
+    "    #     name=conn_name,\n",
+    "    #     api_key=\"<user-input>\",\n",
+    "    # )\n",
+    "\n",
+    "    conn = pf.connections.create_or_update(connection)\n",
+    "    print(\"successfully created connection\")\n",
+    "\n",
+    "print(conn)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Run web-classification flow\n",
+    "\n",
+    "`web-classification` is a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set flow path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow = \"../../flows/standard/web-classification\"  # path to the flow directory"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Quick test"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test flow\n",
+    "flow_inputs = {\n",
+    "    \"url\": \"https://play.google.com/store/apps/details?id=com.twitter.android\",\n",
+    "}\n",
+    "flow_result = pf.test(flow=flow, inputs=flow_inputs)\n",
+    "print(f\"Flow result: {flow_result}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test single node in the flow\n",
+    "node_name = \"fetch_text_content_from_url\"\n",
+    "node_inputs = {\n",
+    "    \"url\": \"https://play.google.com/store/apps/details?id=com.twitter.android\"\n",
+    "}\n",
+    "flow_result = pf.test(flow=flow, inputs=node_inputs, node=node_name)\n",
+    "print(f\"Node result: {flow_result}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Flow as a function\n",
+    "\n",
+    "We have also implemented a syntex sugar where you can consume a flow like a python function, with ability to override connections, inputs and other runtime configs.\n",
+    "Reference [here](./flow-as-function.ipynb) for more details."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import load_flow\n",
+    "\n",
+    "flow_func = load_flow(flow)\n",
+    "flow_result = flow_func(**flow_inputs)\n",
+    "print(f\"Flow function result: {flow_result}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch run with a data file (with multiple lines of test data)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = \"../../flows/standard/web-classification/data.jsonl\"  # path to the data file\n",
+    "\n",
+    "# create run with default variant\n",
+    "base_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    stream=True,\n",
+    "    column_mapping={\n",
+    "        \"url\": \"${data.url}\",\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score.\n",
+    "\n",
+    "In this notebook, we use `classification-accuracy-eval` flow to evaluate. This is a flow illustrating how to evaluate the performance of a classification system. It involves comparing each prediction to the groundtruth and assigns a \"Correct\" or \"Incorrect\" grade, and aggregating the results to produce metrics such as accuracy, which reflects how good the system is at classifying the data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
+    "\n",
+    "eval_run = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    data=\"../../flows/standard/web-classification/data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"groundtruth\": \"${data.answer}\",\n",
+    "        \"prediction\": \"${run.outputs.category}\",\n",
+    "    },  # map the url field from the data to the url input of the flow\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out the [web-classification](https://github.com/microsoft/promptflow/tree/main/examples/flows/standard/web-classification) flow and the [classification-accuracy](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation/eval-classification-accuracy) flow for more details, and start building your own flow.\n",
+    "\n",
+    "Or you can move on for a more advanced topic: experiment with a variant."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Another batch run with a variant\n",
+    "\n",
+    "[Variant](https://microsoft.github.io/promptflow/concepts/concept-variants.html) in prompt flow is to allow you do experimentation with LLMs. You can set a variant of Prompt/LLM node pointing to different prompt or use different LLM parameters like temperature.\n",
+    "\n",
+    "In this example, `web-classification`'s node `summarize_text_content` has two variants: `variant_0` and `variant_1`. The difference between them is the inputs parameters:\n",
+    "\n",
+    "variant_0:\n",
+    "\n",
+    "    - inputs:\n",
+    "        - deployment_name: gpt-35-turbo\n",
+    "        - max_tokens: '128'\n",
+    "        - temperature: '0.2'\n",
+    "        - text: ${fetch_text_content_from_url.output}\n",
+    "\n",
+    "variant_1:\n",
+    "\n",
+    "    - inputs:\n",
+    "        - deployment_name: gpt-35-turbo\n",
+    "        - max_tokens: '256'\n",
+    "        - temperature: '0.3'\n",
+    "        - text: ${fetch_text_content_from_url.output}\n",
+    "\n",
+    "\n",
+    "You can check the whole flow definition at [flow.dag.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flows/standard/web-classification/flow.dag.yaml)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# use the variant1 of the summarize_text_content node.\n",
+    "variant_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    variant=\"${summarize_text_content.variant_1}\",  # here we specify node \"summarize_text_content\" to use variant 1 version.\n",
+    "    column_mapping={\n",
+    "        \"url\": \"${data.url}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(variant_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the variant run\n",
+    "So that later we can compare metrics and see which works better."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
+    "\n",
+    "eval_run_variant = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    data=\"../../flows/standard/web-classification/data.jsonl\",  # path to the data file\n",
+    "    run=variant_run,  # use run as the variant\n",
+    "    column_mapping={\n",
+    "        \"groundtruth\": \"${data.answer}\",\n",
+    "        \"prediction\": \"${run.outputs.category}\",\n",
+    "    },  # map the url field from the data to the url input of the flow\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run_variant)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = pf.get_metrics(eval_run_variant)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([eval_run, eval_run_variant])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Next Steps\n",
+    "\n",
+    "Learn more on:\n",
+    "- [Manage connections](https://github.com/microsoft/promptflow/blob/main/examples/connections/connection.ipynb): how to manage the endpoints/secrets information to access external services including LLMs.\n",
+    "- [Chat with PDF](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/e2e-development/chat-with-pdf.md): go through an end-to-end tutorial on how to develop a chat application with prompt flow.\n",
+    "- [Deploy http endpoint](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/flow-deploy): how to deploy the flow as a local http endpoint.\n",
+    "- [Prompt flow in Azure AI](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/get-started/quickstart-azure.ipynb): run and evaluate flow in Azure AI where you can collaborate with team better."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Flow",
+   "weight": 30
+  },
+  "description": "A quickstart tutorial to run a flow and evaluate it.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": "examples/requirements.txt, examples/flows/standard/web-classification, examples/flows/evaluation/eval-classification-accuracy"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}