chore: import upstream snapshot with attribution

2026-07-13 13:39:52 +08:00
commit e768098d0e
4004 changed files with 2804145 additions and 0 deletions
@@ -0,0 +1,149 @@
+---
+category: rag
+weight: 30
+---
+
+# Develop copilot with promptflow
+
+In this tutorial, we will provide a detailed walkthrough on creating a RAG-based copilot using the Azure Machine Learning promptflow toolkit. Our tutorial will cover a range of essential topics, including:
+
+- Initiating a RAG-based copilot flow through the AzureML Workspace Portal.
+- Generating synthetic test data for the copilot.
+- Evaluating the copilot's performance using test data.
+- Enhancing the functionality and efficiency of your copilot flow.
+- Deploying your copilot for customer use.
+
+While we will focus on constructing a copilot for promptflow as a case study, the methodologies and steps outlined can be adapted to develop your customized copilot solutions.
+
+## Prerequisites
+
+- An Azure subscription. If you don't have an Azure subscription, create a free account before you begin
+- An Azure Machine Learning workspace. If you don't have a workspace, create one before you begin
+- Azure OpenAI connection or OpenAI connection
+- Azure AI Search connection
+
+## Step 1: Initialize a RAG based copilot flow
+
+First, begin by cloning the promptFlow repository to your local machine. Subsequently, within your Azure Machine Learning workspace, proceed to create a vector index utilizing the document files located in the `./docs` folder. For comprehensive guidance on creating a vector index, kindly consult the documentation available at [here](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-vector-index?view=azureml-api-2#create-a-vector-index-by-using-machine-learning-studio).
+
+Upon successful creation of the vector index, an example flow will be automatically generated within your workspace. This example flow, which is a standard Retrieval-Augmented Generation (RAG) based copilot flow, serves as an excellent starting point for developing your own copilot. You can locate the link to this example flow on the vector index's detail page.
+
+This is how the example flow looks like:
+
+![example-flow](example-flow.png)
+
+With some minor configuration, you can open the chat panel and directly chat with your copilot to see how good it works.
+
+![chat-panel](chat-panel.png)
+
+### Tips
+
+```
+Currently, the volume of test data generated cannot be directly manipulated by the user. Instead, it is contingent upon the number of segments your documents are divided into. This segmentation can be adjusted by modifying the 'document_chunk_size' and 'document_chunk_overlap' parameters in your config.yml file. Additionally, you have the option to alter the 'temperature' parameter of the LLM tool within the 'gen_test_data' example flow. By executing the 'gen_test_data' script multiple times, you can indirectly increase the quantity of test data produced.
+```
+
+## Step 2: Generate synthetic test data
+
+To ensure the quality of the promptFlow copilot, it's imperative to conduct extensive testing using a broad dataset. Ideally, this dataset would consist of real user inquiries, such as those found on platforms like StackOverflow. However, real-world cases often fall short in both quantity and diversity. To address this gap, the creation of synthetic test data is necessary to encompass a wider array of scenarios.
+
+Promptflow has provided comprehensive guidelines for generating synthetic test data using Large Language Models (LLMs). For step-by-step instructions, please refer to the document available at [here](../generate-test-data/README.md).
+
+To facilitate evaluation of your copilot in Azure, consider creating a new Data Asset in your workspace specifically for this purpose.
+
+### Tips
+
+```
+Currently, you cannot directly control how much test data you want to generate. The amount of test data is determined by how many trunks your documents are split into, you can configure that by changing the 'document_chunk_size' and 'document_chunk_overlap' parameters in your config.yml file. Meanwhile, you can also change the temperature parameter of the LLM tool in the gen_test_data example flow and run the 'gen_test_data' script for multiple times to generate more test data.
+```
+
+## Step 3: Evaluate your copilot with test data
+After preparing the test data, we can utilize the evaluation flow to assess the performance of our copilot against the test data. Promptflow has developed various evaluation flows tailored for different scenarios. For our RAG-based copilot, we can leverage the evaluation flow in [this folder](../../../examples/flows/evaluation/eval-single-turn-metrics/) to ensure comprehensive and accurate performance analysis.
+
+Clone this evaluation flow folder to your local machine or upload it to your workspace.
+
+**Remember to update the connections used in the flow before you start evaluation.**
+![trigger-eval](trigger-eval.png)
+![select-eval](select-eval.png)
+
+### Tips
+
+```
+- The evaluation process is designed to compute multiple metrics, each accompanied by comprehensive explanations in the readme file. It is imperative to understand these metrics thoroughly and select those most applicable to your project.
+
+- The answer produced by the initial copilot flow will have a "(Source: citation)" part at the end. This is because we told the model to do that in the prompt. You can modify the default prompt to remove this part in case it affects the evaluation results as we did not append this part when generating the test data.
+
+- Furthermore, the evaluation process will present aggregated metrics. It is essential to closely examine the results for each line, especially for the line with lower metric.
+Typically, suboptimal results stem from one of two issues: either the process is underperforming, possibly due to inadequate context retrieval or prompt formulation, or the quality of the test data is insufficient.
+
+To address the first issue, consider debugging or refining the process either locally or within the workspace. For the latter, you might either revise the problematic test cases or exclude them from your test dataset altogether.
+```
+
+## Step 4: Improve your copilot flow
+
+After evaluation, you will find that the initial copilot flow works well and can achieve relatively good metrics. We can continue to improve the copilot in various ways.
+
+### Improve context retrieval
+The context retrieval is the most important part of RAG based approach, the quality of the retrieved context will directly affect the performance of the copilot. Take a close look at the initial copilot flow, you will find that the context retrieval is achieved by 'lookup_question_from_indexed_docs' node which is using 'Index Lookup' tool.
+
+![index-lookup-tool](index-lookup-tool.png)
+
+There are two paramters can be adjusted in the 'Index Lookup' tool: 'query_type' and 'top_k'. The initial copilot flow used 'query_type' as 'vector' and 'top_k' as 2. Try to adjust these two parameters and then run the evaluation to see which one can achieve the best metrics.
+
+### Tune the prompt
+There are two LLM nodes in the initial copilot flow: the 'modify_query_with_history' node is used for rewriting the question with the history to make it more clear; the 'answer_the_question_with_context' node is used for generating the answer based on the question and context.
+
+You can tune the prompt of these two nodes by leveraging the variants feature of promptflow. For detailed introduction of variants, you can reference [this doc](../../../docs/how-to-guides/tune-prompts-with-variants.md).
+
+### Add doc link to the answer
+It's important to add the link of the document which is used as the context to generate the answer. This will help the user to understand where the answer comes from and also help the user to find more information if needed.
+
+The answer generated by the initial flow will include a citation in the format "(Source: citation)." However, this citation format does not present a clickable link, making it inconvenient for end-users to directly access the source.
+To address this, we propose modifications to the code within the 'generate_prompt_context' node. These adjustments aim to transform the citation into an accessible hyperlink. Furthermore, alterations to the prompt in the 'answer_the_question_with_context' node are suggested to ensure the document link is seamlessly integrated into the response. By implementing these changes, the final response will effectively incorporate the document link in a user-friendly format. The final answer will look like this:
+
+![doc-link](doc-link.png)
+
+You can find the specific code changes in the source of the promptflow copilot flow in [this folder](../../../examples/flows/chat/promptflow-copilot/).
+
+
+### Avoid abuse of the copilot
+Avoid abuse is a critical topic when you want to deploy your copilot to production, due to the cost consideration of LLM model. It is recommended that you add an additional authentication layer above your copilot to prevent abuse from anonymous users.
+
+But what if we cannot add the authentication layer or we want to save the login effort for the users ? How do we avoid the abuse of the copilot in this case?
+
+One common approach is to refine the prompts used in the 'answer_the_question_with_context' function to instruct the model to only respond if the answer can be sourced from the provided context. Despite this, test results indicate that the model may still respond to queries unrelated to the context, particularly with general inquiries such as "What is the capital of China?" or when chat histories extend over multiple interactions.
+
+A more effective strategy involves integrating an additional LLM node tasked with evaluating the relevance of a query to the copilot's capabilities (in this scenario, referred to as 'promptflow'). This node assigns a relevance score to each query. Queries with a relevance score below a predetermined threshold would bypass the context retrieval phase, and the system would instead inform the user that their question is not pertinent to the copilot's functionality. Users would be encouraged to rephrase their queries for better alignment with the copilot's capabilities.
+
+You can find the specific code changes in the source of the promptflow copilot flow in [this folder](../../../examples/flows/chat/promptflow-copilot/).
+
+
+## Step 5: Bring your copilot to customers
+
+The final step is to bring our intelligent copilot to customers. Obviously, we cannot ask customers to run the flow directly from their side, so we must firstly deploy the flow. Promptflow has provided plenty of ways to deploy the flow, you can find the details [here](../../../docs/how-to-guides/deploy-a-flow/). The best way to do it depends on the specific scenario of your copilot.
+
+We want our customers to access promptflow copilot through a web page with chat UI experience, so we will deploy the flow as a managed online endpoint. You can find the detailed instructions [here](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-deploy-for-real-time-inference?view=azureml-api-2).
+
+### Host web app with Azure App Service
+Currently, the managed online endpoint does not support Cross-Origin Resource Sharing (CORS), preventing direct access from a webpage. To facilitate interaction with the endpoint, it is necessary to host a web application. Azure App Service offers a comprehensive solution for this requirement, providing a fully managed platform designed for building, deploying, and scaling web applications. By utilizing Azure App Service, you can host your web application efficiently and establish interaction with the promptflow copilot endpoint.
+
+### Chat UI experience
+The chat interface significantly impacts the overall user experience with the copilot, directly influencing how users interact with the system. While constructing a ChatGPT-style interface from the ground up is feasible, utilizing established open-source projects can greatly streamline and expedite the process. One of the projects we have tried is `chatgpt-lite`, we had built our promptflow copilot's UI based on it. You can find the source code of the chat UI [here](https://github.com/melionel/chatgpt-lite/tree/talk_to_endpoint_appservice).
+
+![chat-ui](chat-ui.png)
+
+### Provide suggested follow-up questions
+
+Incorporating follow-up question suggestions is an effective strategy to enhance user experience and communication efficiency. One approach is to instruct the model to include follow-up questions in its responses. However, this method may not always be reliable and could complicate response processing. An alternative strategy involves utilizing a separate flow dedicated to generating follow-up question suggestions. For this purpose, you can employ the 'question_simulation' flow found in [this folder](../../../examples/flows/standard/question-simulation/).
+
+Deploying the `question_simulation` flow as a managed online endpoint and integrating it into your web application allows for dynamic generation of pertinent follow-up questions based on previous chat interactions.
+
+### Collect user feedback
+
+User feedback is invaluable for enhancing the copilot's performance and user experience. By incorporating a feedback mechanism into the chat interface, users can provide real-time feedback on the copilot's responses. This feedback can be used to refine the copilot's performance and prompt formulation, ensuring continuous improvement.
+
+When you deploy the flow as a managed online endpoint, you will find the endpoint has provided two POST APIs if you check the endpoint's swagger: `/score` for trigger your copilot flow with inputs and `/feedback` for collecting user feedback into the trace data.
+
+```
+- You are recommended to turn on this options when you deploy the MIR endpoint, so that you can collect the related data and calculate online metrics to your workspace's application insights.
+```
+![mir-options](mir-options.png)
@@ -0,0 +1,391 @@
+---
+resources: examples/connections/azure_openai.yml, examples/flows/chat/chat-with-pdf
+cloud: local
+category: rag
+---
+
+# Tutorial: Chat with PDF
+
+## Overview
+Retrieval Augmented Generation (or RAG) has become a prevalent pattern to build intelligent application with Large Language Models (or LLMs) since it can infuse external knowledge into the model, which is not trained with those up-to-date or proprietary information. The screenshot below shows how new Bing in Edge sidebar can answer questions based on the page content on the left - in this case, a PDF file.
+![edge-chat-pdf](../../flows/chat/chat-with-pdf/assets/edge-chat-pdf.png)
+Note that new Bing will also search web for more information to generate the answer, let's ignore that part for now.
+
+In this tutorial we will try to mimic the functionality of retrieval of relevant information from the PDF to generate an answer with GPT.
+
+**We will guide you through the following steps:**
+
+Creating a console chatbot "chat_with_pdf" that takes a URL to a PDF file as an argument and answers questions based on the PDF's content.
+Constructing a prompt flow for the chatbot, primarily reusing the code from the first step.
+Creating a dataset with multiple questions to swiftly test the flow.
+Evaluating the quality of the answers generated by the chat_with_pdf flow.
+Incorporating these tests and evaluations into your development cycle, including unit tests and CI/CD.
+Deploying the flow to Azure App Service and Streamlit to handle real user traffic.
+
+## Prerequisite
+To go through this tutorial you should:
+1. Install dependencies
+```bash
+   cd ../../flows/chat/chat-with-pdf/
+   pip install -r requirements.txt
+```
+
+1. Install and configure [Prompt flow for VS Code extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) follow [Quick Start Guide](https://microsoft.github.io/promptflow/how-to-guides/quick-start.html). (_This extension is optional but highly recommended for flow development and debugging._)
+
+2. Deploy an OpenAI or Azure OpenAI chat model (e.g. gpt4 or gpt-35-turbo-16k), and an Embedding model (text-embedding-ada-002).  Follow the [how-to](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal) for an Azure OpenAI example.
+
+## Console chatbot chat_with_pdf
+A typical RAG process consists of two steps:
+- **Retrieval**: Retrieve contextual information from external systems (database, search engine, files, etc.)
+- **Generation**: Construct the prompt with the retrieved context and get response from LLMs.
+
+The retrieval step, being more of a search problem, can be quite complex. A widely used, simple yet effective approach is vector search, which requires an index building process. Suppose you have one or more documents containing the contextual information, the index building process would look something like this:
+1. **Chunk**: Break down the documents into multiple chunks of text.
+2. **Embedding**: Each text chunk is then processed by an embedding model to convert it into an array of floating-point numbers, also known as embedding or vector.
+3. **Indexing**: These vectors are then stored in an index or a database that supports vector search. This allows for the retrieval of the top K relevant or similar vectors from the index or database.
+
+Once the index is built, the **Retrieval** step simply involves converting the question into an embedding/vector and performing a vector search on the index to obtain the most relevant context for the question.
+
+OK now back to the chatbot we want to build, a simplified design could be:
+
+<img src="../../flows/chat/chat-with-pdf/assets/chat_with_pdf_simple.png" width="300" alt="chat with pdf simple design"/>
+
+A more robust or practical application might consider using an external vector database to store the vectors. For this simple example we're using a [FAISS](https://github.com/facebookresearch/faiss) index, which can be saved as a file. However, a more robust or practical application should consider using an external vector database with advanced management capabilities to store the vectors.  With this sample's FAISS index, to prevent repetitive downloading and index building for same PDF file, we will add a check that if the PDF file already exists then we won't download, same for index building.
+
+This design is quite effective for question and answering, but it may fall short when it comes to multi-turn conversations with the chatbot. Consider a scenario like this:
+
+> $User: what is BERT?
+>
+> $Bot: BERT stands for Bidirectional Encoder Representations from Transformers.
+>
+> $User: is it better than GPT?
+>
+> $Bot: ...
+
+You would typically expect the chatbot to be intelligent enough to decipher that the "it" in your second question refers to BERT, and your actual question is "is BERT better than GPT". However, if you present the question "is it better than GPT" to the embedding model and then to the vector index/database, they won't recognize that "it" represents BERT. Consequently, you won't receive the most relevant context from the index. To address this issue, we will enlist the assistance of a Large Language Model (LLM), such as GPT, to "rewrite" the question based on the previous question. The updated design is as follows:
+
+<img src="../../flows/chat/chat-with-pdf/assets/chat_with_pdf_with_rewrite.png" width="400" alt="chat with pdf better design"/>
+
+A "rewrite_question" step is performed before feeding the question to "find_context" step.
+
+### Configurations
+Despite being a minimalistic LLM application, there are several aspects we may want to adjust or experiment with in the future. We'll store these in environment variables for ease of access and modification. In the subsequent sections, we'll guide you on how to experiment with these configurations to enhance your chat application's quality.
+
+Create a .env file in the second chat_with_pdf directory (same directory with the main.py) and populate it with the following content. We can use the load_dotenv() function (from the python-dotenv package) to import these into our environment variables later on. We'll delve into what these variables represent when discussing how each step of the process is implemented.
+
+Rename the .env.example file in chat_with_pdf directory and modify per your need.
+
+> If you're using OpenAI, your .env should look like:
+```ini
+OPENAI_API_KEY=<open_ai_key>
+EMBEDDING_MODEL_DEPLOYMENT_NAME=<text-embedding-ada-002>
+CHAT_MODEL_DEPLOYMENT_NAME=<gpt-4>
+PROMPT_TOKEN_LIMIT=3000
+MAX_COMPLETION_TOKENS=1024
+CHUNK_SIZE=256
+CHUNK_OVERLAP=64
+VERBOSE=False
+```
+Note: if you have an org id, it can be set via OPENAI_ORG_ID=<your_org_id>
+
+> If you're using Azure OpenAI, you .env should look like:
+```ini
+OPENAI_API_TYPE=azure
+OPENAI_API_BASE=<AOAI_endpoint>
+OPENAI_API_KEY=<AOAI_key>
+OPENAI_API_VERSION=2023-05-15
+EMBEDDING_MODEL_DEPLOYMENT_NAME=<text-embedding-ada-002>
+CHAT_MODEL_DEPLOYMENT_NAME=<gpt-4>
+PROMPT_TOKEN_LIMIT=3000
+MAX_COMPLETION_TOKENS=1024
+CHUNK_SIZE=256
+CHUNK_OVERLAP=64
+VERBOSE=False
+```
+Note: CHAT_MODEL_DEPLOYMENT_NAME should point to a chat model like gpt-3.5-turbo or gpt-4, OPENAI_API_KEY should use the deployment key, and EMBEDDING_MODEL_DEPLOYMENT_NAME should point to a text embedding model like text-embedding-ada-002.
+
+### Take a look at the chatbot in action!
+You should be able to run the console app by:
+```shell
+python chat_with_pdf/main.py https://arxiv.org/pdf/1810.04805.pdf
+```
+> Note: https://arxiv.org/pdf/1810.04805.pdf is the paper about one of the most famous earlier LLMs: BERT.
+
+It looks like below if everything goes fine:
+![chatbot console](../../flows/chat/chat-with-pdf/assets/chatbot_console.gif)
+
+Now, let's delve into the actual code that implements the chatbot.
+
+### Implementation of each steps
+#### Download pdf: [download.py](../../flows/chat/chat-with-pdf/chat_with_pdf/download.py)
+The downloaded PDF file will be stored into a temp folder.
+
+#### Build index: [build_index.py](../../flows/chat/chat-with-pdf/chat_with_pdf/build_index.py)
+Several libraries are used in this step to build index:
+1. PyPDF2 for extraction of text from the PDF file.
+2. OpenAI python library for generating embeddings.
+3. The FAISS library is utilized to build a vector index and save it to a file. It's important to note that an additional dictionary is used to maintain the mapping from the vector index to the actual text snippet. This is because when we later attempt to query for the most relevant context, we need to locate the text snippets, not just the embedding or vector.
+The environment variables used in this step:
+- OPENAI_API_* and EMBEDDING_MODEL_DEPLOYMENT_NAME: to access the Azure OpenAI embedding model
+- CHUNK_SIZE and CHUNK_OVERLAP: controls how to split the PDF file into chunks for embedding
+
+#### Rewrite question: [rewrite_question.py](../../flows/chat/chat-with-pdf/chat_with_pdf/rewrite_question.py)
+This step is to use ChatGPT/GPT4 to rewrite the question to be better fit for finding relevant context from the vector index. The prompt file [rewrite_question.md](../../flows/chat/chat-with-pdf/chat_with_pdf/rewrite_question_prompt.md) should give you a better idea how it works.
+
+#### Find context: [find_context.py](../../flows/chat/chat-with-pdf/chat_with_pdf/find_context.py)
+In this step we load the FAISS index and the dict that were built in the "build index" step. We then turn the question into a vector using the same embedding function in the build index step. There is a small trick in this step to make sure the context will not exceed the token limit of model input prompt ([aoai model max request tokens](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models), OpenAI has similar limit). The output of this step is the final prompt that QnA step will send to the chat model. The PROMPT_TOKEN_LIMIT environment variable decides how big the context is.
+
+#### QnA: [qna.py](../../flows/chat/chat-with-pdf/chat_with_pdf/qna.py)
+Use OpenAI's ChatGPT or GPT4 model and ChatCompletion API to get an answer with the previous conversation history and context from PDF.
+
+#### The main loop: [main.py](../../flows/chat/chat-with-pdf/chat_with_pdf/main.py)
+This is the main entry of the chatbot, which includes a loop that reads questions from user input and subsequently calls the steps mentioned above to provide an answer.
+
+To simplify this example, we store the downloaded file and the constructed index as local files. Although there is a mechanism in place to utilize cached files/indices, loading the index still takes a certain amount of time and contributes to a latency that users may notice. Moreover, if the chatbot is hosted on a server, it requires requests for the same PDF file to hit the same server node in order to effectively use the cache. In a real-world scenario, it's likely preferable to store the index in a centralized service or database. There're many such database available, such as [Azure Cognitive Search](https://learn.microsoft.com/en-us/azure/search/vector-search-overview), [Pinecone](https://www.pinecone.io/), [Qdrant](https://qdrant.tech/), ...
+
+## Prompt flow: when you start considering the quality of your LLM app
+Having a functioning chatbot is a great start, but it's only the beginning of the journey. Much like any application based on machine learning, the development of a high-quality LLM app usually involves a substantial amount of tuning. This could include experimenting with different prompts such as rewriting questions or QnAs, adjusting various parameters like chunk size, overlap size, or context limit, or even redesigning the workflow (for instance, deciding whether to include the rewrite_question step in our example).
+
+Appropriate tooling is essential for facilitating this experimentation and fine-tuning process with LLM apps. This is where the concept of prompt flow comes into play. It enables you to test your LLM apps by:
+
+- Running a few examples and manually verifying the results.
+- Running larger scale tests with a formal approach (using metrics) to assess your app's quality.
+
+You may have already learned how to create a prompt flow from scratch. Building a prompt flow from existing code is also straightforward. You can construct a chat flow either by composing the YAML file or using the visual editor of [Visual Studio Code extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) and create a few wrappers for existing code.
+
+Check out below:
+- [flow.dag.yaml](../../flows/chat/chat-with-pdf/flow.dag.yaml)
+- [setup_env.py](../../flows/chat/chat-with-pdf/setup_env.py)
+- [download_tool.py](../../flows/chat/chat-with-pdf/download_tool.py)
+- [build_index_tool.py](../../flows/chat/chat-with-pdf/build_index_tool.py)
+- [rewrite_question_tool.py](../../flows/chat/chat-with-pdf/rewrite_question_tool.py)
+- [find_context_tool.py](../../flows/chat/chat-with-pdf/find_context_tool.py)
+- [qna_tool.py](../../flows/chat/chat-with-pdf/qna_tool.py)
+
+E.g. build_index_tool wrapper:
+```python
+from promptflow.core import tool
+from chat_with_pdf.build_index import create_faiss_index
+
+
+@tool
+def build_index_tool(pdf_path: str) -> str:
+    return create_faiss_index(pdf_path)
+```
+
+The setup_env node requires some explanation: you might recall that we use environment variables to manage different configurations, including OpenAI API key in the console chatbot, in prompt flow we use [Connection](https://microsoft.github.io/promptflow/concepts/concept-connections.html) to manage access to external services like OpenAI and support passing configuration object into flow so that you can do experimentation easier. The setup_env node is to write the properties from connection and configuration object into environment variables. This allows the core code of the chatbot remain unchanged.
+
+We're using Azure OpenAI in this example, below is the shell command to do so:
+
+**CLI**
+```bash
+# create connection needed by flow
+if pf connection list | grep open_ai_connection; then
+    echo "open_ai_connection already exists"
+else
+    pf connection create --file ../../../connections/azure_openai.yml --name open_ai_connection --set api_key=<your_api_key> api_base=<your_api_base>
+fi
+```
+
+If you plan to use OpenAI instead you can use below instead:
+```shell
+# create connection needed by flow
+if pf connection list | grep open_ai_connection; then
+    echo "open_ai_connection already exists"
+else
+    pf connection create --file ../../../connections/openai.yml --name open_ai_connection --set api_key=<your_api_key>
+fi
+```
+
+The flow looks like:
+
+<img src="../../flows/chat/chat-with-pdf/assets/multi-node-flow-chat-with-pdf.png" width="500" alt="chat with pdf flow, multi-node"/>
+
+## Prompt flow evaluations
+Now the prompt flow for chat_with_pdf is created, you might have already run/debug flow through the Visual Studio Code extension. It's time to do some testing and evaluation, which starts with:
+1. Create a test dataset which contains a few question and pdf_url pairs.
+2. Use existing [evaluation flows](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation) or develop new evaluation flows to generate metrics.
+
+A small dataset can be found here: [bert-paper-qna.jsonl](../../flows/chat/chat-with-pdf/data/bert-paper-qna.jsonl) which contains around 10 questions for the BERT paper.
+
+Evaluations are executed through 'batch runs'.  Conceptually, they are a batch run of an evaluation flow which uses the previous run as input.
+
+Here is an example of how to create a batch run for the chat_with_pdf flow using the test dataset and manually reviewing the output. This can be done through the Visual Studio Code extension, or CLI or Python SDK.
+
+**batch_run.yaml**
+```yaml
+name: chat_with_pdf_default_20230820_162219_559000
+flow: .
+data: ./data/bert-paper-qna.jsonl
+#run: <Uncomment to select a run input>
+column_mapping:
+  chat_history: ${data.chat_history}
+  pdf_url: ${data.pdf_url}
+  question: ${data.question}
+  config:
+    EMBEDDING_MODEL_DEPLOYMENT_NAME: text-embedding-ada-002
+    CHAT_MODEL_DEPLOYMENT_NAME: gpt-35-turbo
+    PROMPT_TOKEN_LIMIT: 3000
+    MAX_COMPLETION_TOKENS: 1024
+    VERBOSE: true
+    CHUNK_SIZE: 256
+    CHUNK_OVERLAP: 64
+```
+**CLI**
+```bash
+run_name="chat_with_pdf_"$(openssl rand -hex 12)
+pf run create --file batch_run.yaml --stream --name $run_name
+```
+
+The output will include something like below:
+```json
+{
+    "name": "chat_with_pdf_default_20230820_162219_559000",
+    "created_on": "2023-08-20T16:23:39.608101",
+    "status": "Completed",
+    "display_name": "chat_with_pdf_default_20230820_162219_559000",
+    "description": null,
+    "tags": null,
+    "properties": {
+        "flow_path": "/Users/<user>/Work/azure-promptflow/scratchpad/chat_with_pdf",
+        "output_path": "/Users/<user>/.promptflow/.runs/chat_with_pdf_default_20230820_162219_559000"
+    },
+    "flow_name": "chat_with_pdf",
+    "data": "/Users/<user>/Work/azure-promptflow/scratchpad/chat_with_pdf/data/bert-paper-qna.jsonl",
+    "output": "/Users/<user>/.promptflow/.runs/chat_with_pdf_default_20230820_162219_559000/    flow_outputs/output.jsonl"
+}
+```
+
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+And we developed two evaluation flows one for "[groundedness](../../flows/evaluation/eval-groundedness/)" and one for "[perceived intelligence](../../flows/evaluation/eval-perceived-intelligence/)". These two flows are using GPT models (ChatGPT or GPT4) to "grade" the answers. Reading the prompts will give you better idea what are these two metrics:
+- [groundedness prompt](../../flows/evaluation/eval-groundedness/gpt_groundedness.md)
+- [perceived intelligence prompt](../../flows/evaluation/eval-perceived-intelligence/gpt_perceived_intelligence.md)
+
+The following example creates an evaluation flow.
+
+**eval_run.yaml:**
+```yaml
+flow: ../../evaluation/eval-groundedness
+run: chat_with_pdf_default_20230820_162219_559000
+column_mapping:
+  question: ${run.inputs.question}
+  answer: ${run.outputs.answer}
+  context: ${run.outputs.context}
+```
+> NOTE: the run property in eval_run.yaml is the run name of batch_run.yaml
+
+**CLI:**
+```bash
+eval_run_name="eval_groundedness_"$(openssl rand -hex 12)
+pf run create --file eval_run.yaml --run $run_name --name $eval_run_name
+```
+> Note: this assumes that you have followed previous steps to create OpenAI/Azure OpenAI connection with name "open_ai_connection".
+
+After the run completes you can use below commands to get detail of the runs:
+```bash
+pf run show-details --name $eval_run_name
+pf run show-metrics --name $eval_run_name
+pf run visualize --name $eval_run_name
+```
+
+## Experimentation!!
+We have now explored how to conduct tests and evaluations for prompt flow. Additionally, we have defined two metrics to gauge the performance of our chat_with_pdf flow. By trying out various settings and configurations, running evaluations, and then comparing the metrics, we can determine the optimal configuration for production deployment.
+
+There are several aspects we can experiment with, including but not limited to:
+
+* Varying prompts for the rewrite_question and/or QnA steps.
+* Adjusting the chunk size or chunk overlap during index building.
+* Modifying the context limit.
+
+These elements can be managed through the "config" object in the flow inputs. If you wish to experiment with the first point (varying prompts), you can add properties to the config object to control this behavior - simply by directing it to different prompt files.
+
+Take a look at how we experiment with #3 in below test: [test_eval in tests/chat_with_pdf_test.py](../../flows/chat/chat-with-pdf/tests/azure_chat_with_pdf_test.py). This test will create 6 runs in total:
+
+1. chat_with_pdf_2k_context
+2. chat_with_pdf_3k_context
+3. eval_groundedness_chat_with_pdf_2k_context
+4. eval_perceived_intelligence_chat_with_pdf_2k_context
+5. eval_groundedness_chat_with_pdf_3k_context
+6. eval_perceived_intelligence_chat_with_pdf_3k_context
+
+As you can probably tell through the names: run #3 and #4 generate metrics for run #1, run #5 and #6 generate metrics for run #2. You can compare these metrics to decide which performs better - 2K context or 3K context.
+
+NOTE: [azure_chat_with_pdf_test](../../flows/chat/chat-with-pdf/tests/azure_chat_with_pdf_test.py) does the same tests but using Azure AI as backend, so you can see all the runs in a nice web portal with all the logs and metrics comparison etc.
+
+
+Further reading:
+- Learn [how to experiment with the chat-with-pdf flow](../../flows/chat/chat-with-pdf/chat-with-pdf.ipynb)
+- Learn [how to experiment with the chat-with-pdf flow on Azure](../../flows/chat/chat-with-pdf/chat-with-pdf-azure.ipynb) so that you can collaborate with your team.
+
+## Integrate prompt flow into your CI/CD workflow
+It's also straightforward to integrate these into your CI/CD workflow using either CLI or SDK. In this example we have various unit tests to run tests/evaluations for chat_with_pdf flow.
+
+Check the [test](../../flows/chat/chat-with-pdf/tests/) folder.
+
+```bash
+# run all the tests
+python -m unittest discover -s tests -p '*_test.py'
+```
+
+## Deployment
+The flow can be deployed across multiple platforms, such as a local development service, within a Docker container, onto a Kubernetes cluster, etc.
+
+The following sections will guide you through the process of deploying the flow to a Docker container, for more details about
+the other choices, please refer to [flow deploy docs](https://microsoft.github.io/promptflow/how-to-guides/deploy-a-flow/index.html).
+
+
+### Build a flow as docker format app
+
+Use the command below to build a flow as docker format app:
+
+```bash
+pf flow build --source . --output dist --format docker
+```
+
+### Deploy with Docker
+#### Build Docker image
+
+Like other Dockerfile, you need to build the image first. You can tag the image with any name you want. In this example, we use `promptflow-serve`.
+
+Run the command below to build image:
+
+```shell
+docker build dist -t chat-with-pdf-serve
+```
+
+#### Run Docker image
+
+Run the docker image will start a service to serve the flow inside the container.
+
+##### Connections
+If the service involves connections, all related connections will be exported as yaml files and recreated in containers.
+Secrets in connections won't be exported directly. Instead, we will export them as a reference to environment variables:
+```yaml
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/OpenAIConnection.schema.json
+type: open_ai
+name: open_ai_connection
+module: promptflow.connections
+api_key: ${env:OPEN_AI_CONNECTION_API_KEY} # env reference
+```
+You'll need to set up the environment variables in the container to make the connections work.
+
+#### Run with `docker run`
+
+
+You can run the docker image directly set via below commands:
+```shell
+# The started service will listen on port 8080.You can map the port to any port on the host machine as you want.
+docker run -p 8080:8080 -e OPEN_AI_CONNECTION_API_KEY=<secret-value> chat-with-pdf-serve
+```
+
+#### Test the endpoint
+After start the service, you can open the test page at `http://localhost:8080/` and test it:
+
+![test-page](../../flows/chat/chat-with-pdf/assets/chat_with_pdf_test_page.png)
+
+or use curl to test it from cli:
+
+```shell
+curl http://localhost:8080/score --data '{"question":"what is BERT?", "chat_history": [], "pdf_url": "https://arxiv.org/pdf/1810.04805.pdf", "config": {"EMBEDDING_MODEL_DEPLOYMENT_NAME": "text-embedding-ada-002", "CHAT_MODEL_DEPLOYMENT_NAME": "gpt-35-turbo", "PROMPT_TOKEN_LIMIT": 3000, "MAX_COMPLETION_TOKENS": 256, "VERBOSE": true, "CHUNK_SIZE": 1024, "CHUNK_OVERLAP": 64}}' -X POST  -H "Content-Type: application/json"
+```
+![test-endpoint](../../flows/chat/chat-with-pdf/assets/chat_with_pdf_test_endpoint.png)
@@ -0,0 +1,4 @@
+*.sqlite
+**/build/flow
+**/build/connections
+**/build/settings.json
@@ -0,0 +1,3 @@
+# Deploy flow as applications
+
+This folder contains examples of how to build & deploy flow as applications like Web Application packaged in Docker format.
@@ -0,0 +1,75 @@
+---
+resources: examples/connections/azure_openai.yml, examples/flows/standard/web-classification
+category: deployment
+weight: 60
+---
+
+# Deploy flow using Azure App Service
+
+This example demos how to deploy a flow using Azure App Service.
+
+[Azure App Service](https://learn.microsoft.com/azure/app-service/) is an HTTP-based service for hosting web applications, REST APIs, and mobile back ends.
+The scripts (`deploy.sh` for bash and `deploy.ps1` for powershell) under this folder are here to help deploy the docker image to Azure App Service.
+
+We will use [web-classification](../../../flows/standard/web-classification/README.md) as example in this tutorial.
+
+## Build a flow as docker format app
+
+Note that all dependent connections must be created before building as docker.
+```bash
+# create connection if not created before
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Use the command below to build a flow as docker format app:
+
+```bash
+pf flow build --source ../../../flows/standard/web-classification --output dist --format docker
+```
+
+
+## Deploy with Azure App Service
+The two scripts will do the following things:
+1. Create a resource group if not exists.
+2. Build and push the image to docker registry.
+3. Create an app service plan with the give sku.
+4. Create an app with specified name, set the deployment container image to the pushed docker image.
+5. Set up the environment variables for the app.
+
+Example command to use bash script:
+```shell
+bash deploy.sh --path dist -i <image_tag> --name my-app-23d8m -r <docker registry> -g <resource_group>
+```
+
+Example command to use powershell script:
+```powershell
+.\deploy.ps1 -Path dist -i <image_tag> -n my-app-23d8m -r <docker registry> -g <resource_group>
+```
+Note that the `name` will produce a unique FQDN as AppName.azurewebsites.net.
+
+See the full parameters by `bash deploy.sh -h` or `.\deploy.ps1 -h`.
+
+## View and test the web app
+The web app can be found via [azure portal](https://portal.azure.com/)
+
+![img](assets/azure_portal_img.png)
+
+After the app created, you will need to go to https://portal.azure.com/ find the app and set up the environment variables
+at (Settings>Configuration) or (Settings>Environment variables), then restart the app.
+
+![img](assets/set_env_var.png)
+
+Browse the app at Overview and see the test page:
+
+![img](assets/test_page.png)
+
+You can also test the app by sending a POST request to the app like:
+```shell
+curl http://<Default-domain-of-app-service>/score --data '{"url":"https://play.google.com/store/apps/details?id=com.twitter.android"}' -X POST  -H "Content-Type: application/json"
+```
+
+
+Tips:
+- Reach deployment logs at (Deployment>Deployment Central) and app logs at (Monitoring>Log stream).
+- Reach advanced deployment tools at  (Development Tools>Advanced Tools).
+- Reach more details about app service at [Azure App Service](https://learn.microsoft.com/azure/app-service/).
@@ -0,0 +1,171 @@
+<#
+.DESCRIPTION
+Script to deploy promptflow to Azure App Service.
+
+.PARAMETER path
+The folder path to be deployed
+.PARAMETER image_tag
+The container image tag.
+.PARAMETER registry
+The container registry name, for example 'xx.azurecr.io'.
+.PARAMETER name
+The app name to produce a unique FQDN as AppName.azurewebsites.net.
+.PARAMETER location
+The app location, default to 'centralus'.
+.PARAMETER sku
+The app sku, default to 'F1'(free).
+.PARAMETER resource_group
+The app resource group.
+.PARAMETER subscription
+The app subscription, default using az account subscription.
+.PARAMETER verbose
+verbose mode.
+
+.EXAMPLE
+PS> .\deploy.ps1 -Path <folder-path> -i <image_tag> -r <registry> -n <app_name> -g <resource_group>
+.EXAMPLE
+PS> .\deploy.ps1 -Path <folder-path> -i <image_tag> -r <registry> -n <app_name> -g <resource_group> -Subscription "xxxx-xxxx-xxxx-xxxx-xxxx" -Verbose
+#>
+[CmdletBinding()]
+param(
+    [string]$Path,
+    [Alias("i", "image_tag")][string]$ImageTag,
+    [Alias("r")][string]$Registry,
+    [Alias("n")][string]$Name,
+    [Alias("l")][string]$Location = "eastus",
+    [string]$Sku = "F1",
+    [Alias("g", "resource_group")][string]$ResourceGroup,
+    [string]$Subscription
+)
+
+####################### Validate args ############################
+$ErrorActionPreference = "Stop"
+
+# fail if image_tag not provided
+if (!$ImageTag) {
+    Write-Host "***************************"
+    Write-Host "* Error: image_tag is required.*"
+    Write-Host "***************************"
+    exit 1
+}
+
+# check if : in image_tag
+if (!$ImageTag.Contains(":")) {
+    $version="v$(Get-Date -Format 'yyyyMMdd-HHmmss')"
+    $image_tag="${ImageTag}:${version}"
+}
+
+Write-Host "image_tag: $ImageTag"
+
+# fail if Registry not provided
+if (!$Registry) {
+    Write-Host "***************************"
+    Write-Host "* Error: registry is required.*"
+    Write-Host "***************************"
+    exit
+}
+
+# fail if name not provided
+if (!$Name) {
+    Write-Host "***************************"
+    Write-Host "* Error: name is required.*"
+    Write-Host "***************************"
+    exit
+}
+
+# fail if resource_group not provided
+if (!$ResourceGroup) {
+    Write-Host "***************************"
+    Write-Host "* Error: resource_group is required.*"
+    Write-Host "***************************"
+    exit
+}
+
+# fail if image_tag not provided
+if (!$Path) {
+    Write-Host "***************************"
+    Write-Host "* Error: Path is required.*"
+    Write-Host "***************************"
+    exit 1
+}
+
+####################### Build and push image ############################
+Write-Host "Change working directory to $Path"
+cd $Path
+docker build -t "$ImageTag" .
+
+if ($Registry.Contains("azurecr.io")) {
+    Write-Host "Trying to login to $Registry..."
+    az acr login -n "$Registry"
+
+    $AcrImageTag = $Registry + "/" + $ImageTag
+    Write-Host "ACR image tag: $AcrImageTag"
+    docker tag "$ImageTag" "$AcrImageTag"
+    $ImageTag = $AcrImageTag
+}
+else {
+    Write-Host "***************************************************\n"
+    Write-Host "* WARN: Make sure you have docker account login!!!*\n"
+    Write-Host "***************************************************\n"
+
+    $DockerImageTag = $Registry + "/" + $ImageTag
+
+    Write-Host "Docker image tag: $DockerImageTag"
+    docker tag "$ImageTag" "$DockerImageTag"
+    $ImageTag = $DockerImageTag
+}
+
+Write-Host "Start pushing image...$ImageTag"
+docker push "$ImageTag"
+
+####################### Create and config app ############################
+
+function Append-To-Command {
+    param (
+        [string] $Command
+    )
+  if ($Subscription) {
+        $Command = "$Command --subscription $Subscription"
+  }
+  if ($VerbosePreference -eq "Continue") {
+        $Command="$Command --debug"
+  }
+  Write-Host "$Command"
+    return $Command
+}
+
+function Invoke-Expression-And-Check{
+    param (
+        [string]$Command
+    )
+    $Command=$(Append-To-Command "$Command")
+    Invoke-Expression $Command
+    if ($LASTEXITCODE -gt 0) {
+        exit $LASTEXITCODE
+    }
+}
+# Check and create resource group if not exist
+$Result = (az group exists --name $ResourceGroup)
+if ($Result -eq "false") {
+    Write-Host "Creating resource group...$ResourceGroup"
+    $Command="az group create --name $ResourceGroup -l $Location"
+    Invoke-Expression-And-Check "$Command"
+}
+# Create service plan
+$ServicePlanName = $Name + "_service_plan"
+Write-Host "Creating service plan...$ServicePlanName"
+$Command="az appservice plan create --name $ServicePlanName --sku $Sku --location $location --is-linux -g $ResourceGroup"
+Invoke-Expression-And-Check "$Command"
+# Create app
+Write-Host "Creating app...$Name"
+$Command="az webapp create --name $Name -p $ServicePlanName --deployment-container-image-name $ImageTag --startup-file 'bash start.sh' -g $ResourceGroup"
+Invoke-Expression-And-Check "$Command"
+# Config environment variable
+Write-Host "Config app...$Name"
+# Port default to 8080 corresponding to the DockerFile
+$Command="az webapp config appsettings set -g $ResourceGroup --name $Name --settings USER_AGENT=promptflow-appservice WEBSITES_PORT=8080 ('@settings.json')"
+Invoke-Expression-And-Check "$Command"
+Write-Host "Please go to https://portal.azure.com/ to config environment variables and restart the app: $Name at (Settings>Configuration) or (Settings>Environment variables)"
+Write-Host "Reach deployment logs at (Deployment>Deployment Central) and app logs at (Monitoring>Log stream)"
+Write-Host "Reach advanced deployment tools at https://$Name.scm.azurewebsites.net/"
+Write-Host "Reach more details about app service at https://learn.microsoft.com/en-us/azure/app-service/"
@@ -0,0 +1,189 @@
+#! /bin/bash
+
+set -e
+
+program_name=$0
+
+function usage {
+    echo "usage: $program_name [-i|-image_tag|--image_tag]"
+    echo "  -i|-image_tag|--image_tag                     specify container image tag"
+    echo "  -r|-registry|--registry                       specify container registry name, for example 'xx.azurecr.io'"
+    echo "  -n|-name|--name                               specify app name to produce a unique FQDN as AppName.azurewebsites.net."
+    echo "  -l|-location|--location                       specify app location, default to 'centralus'"
+    echo "  -sku|--sku                                    specify app sku, default to 'F1'(free)"
+    echo "  -g|-resource_group|--resource_group           specify app resource group"
+    echo "  -subscription|--subscription                  specify app subscription, default using az account subscription"
+    echo "  -v|-verbose|--verbose                         specify verbose mode"
+    echo "  -p|-path|--path                               specify folder path to be deployed"
+    exit 1
+}
+if [ "$1" == "-help" ] || [ "$1" == "-h" ]; then
+  usage
+  exit 0
+fi
+
+location="eastus"
+sku="F1"
+verbose=false
+
+####################### Parse and validate args ############################
+while [ $# -gt 0 ]; do
+  case "$1" in
+    -i|-image_tag|--image_tag)
+      image_tag="$2"
+      ;;
+    -r|-registry|--registry)
+      registry_name="$2"
+      ;;
+    -n|-name|--name)
+      name="$2"
+      ;;
+    -l|-location|--location)
+      location="$2"
+      ;;
+    -sku|--sku)
+      sku="$2"
+      ;;
+    -g|-resource_group|--resource_group)
+      resource_group="$2"
+      ;;
+    -subscription|--subscription)
+      subscription="$2"
+      ;;
+    -v|-verbose|--verbose)
+      verbose=true
+      ;;
+    -p|-path|--path)
+      path="$2"
+      ;;
+    *)
+    printf "***************************\n"
+    printf "* Error: Invalid argument.*\n"
+    printf "***************************\n"
+    exit 1
+  esac
+  shift
+  shift
+done
+
+# fail if image_tag not provided
+if [ -z "$image_tag" ]; then
+    printf "***************************\n"
+    printf "* Error: image_tag is required.*\n"
+    printf "***************************\n"
+    exit 1
+fi
+
+# check if : in image_tag
+if [[ $image_tag == *":"* ]]; then
+    echo "image_tag: $image_tag"
+else
+    version="v$(date '+%Y%m%d-%H%M%S')"
+
+    image_tag="$image_tag:$version"
+    echo "image_tag: $image_tag"
+fi
+
+# fail if registry_name not provided
+if [ -z "$registry_name" ]; then
+    printf "***************************\n"
+    printf "* Error: registry is required.*\n"
+    printf "***************************\n"
+fi
+
+# fail if name not provided
+if [ -z "$name" ]; then
+    printf "***************************\n"
+    printf "* Error: name is required.*\n"
+    printf "***************************\n"
+fi
+
+# fail if resource_group not provided
+if [ -z "$resource_group" ]; then
+    printf "***************************\n"
+    printf "* Error: resource_group is required.*\n"
+    printf "***************************\n"
+fi
+
+# fail if path not provided
+if [ -z "$path" ]; then
+    printf "***************************\n"
+    printf "* Error: path is required.*\n"
+    printf "***************************\n"
+    exit 1
+fi
+
+####################### Build and push image ############################
+echo "Change working directory to $path"
+cd "$path"
+docker build -t "$image_tag" .
+
+if [[ $registry_name == *"azurecr.io" ]]; then
+    echo "Trying to login to $registry_name..."
+    az acr login -n "$registry_name"
+
+    acr_image_tag=$registry_name/$image_tag
+    echo "ACR image tag: $acr_image_tag"
+    docker tag "$image_tag" "$acr_image_tag"
+    image_tag=$acr_image_tag
+else
+    echo "Make sure you have docker account login!!!"
+    printf "***************************************************\n"
+    printf "* WARN: Make sure you have docker account login!!!*\n"
+    printf "***************************************************\n"
+
+    docker_image_tag=$registry_name/$image_tag
+
+    echo "Docker image tag: $docker_image_tag"
+    docker tag "$image_tag" "$docker_image_tag"
+    image_tag=$docker_image_tag
+fi
+
+echo "Start pushing image...$image_tag"
+docker push "$image_tag"
+
+####################### Create and config app ############################
+
+function append_to_command {
+  command=$1
+  if [ -n "$subscription" ]; then
+    command="$command --subscription $subscription"
+  fi
+  if $verbose; then
+    command="$command --debug"
+  fi
+  echo "$command"
+}
+
+# Check and create resource group if not exist
+result=$(az group exists --name "$resource_group")
+if [ "$result" = "false" ]; then
+    echo  "Creating resource group...$resource_group"
+    command="az group create --name $resource_group -l $location"
+    command=$(append_to_command "$command")
+    eval "$command"
+fi
+# Create service plan
+service_plan_name=$name"_service_plan"
+echo "Creating service plan...$service_plan_name"
+command="az appservice plan create --name $service_plan_name --sku $sku --location $location --is-linux -g $resource_group"
+command=$(append_to_command "$command")
+echo "$command"
+eval "$command"
+# Create app
+echo "Creating app...$name"
+command="az webapp create --name $name -p $service_plan_name --deployment-container-image-name $image_tag --startup-file 'bash start.sh' -g $resource_group"
+command=$(append_to_command "$command")
+echo "$command"
+eval "$command"
+# Config environment variable
+echo "Config app...$name"
+# Port default to 8080 corresponding to the DockerFile
+command="az webapp config appsettings set -g $resource_group --name $name --settings USER_AGENT=promptflow-appservice WEBSITES_PORT=8080 @settings.json "
+command=$(append_to_command "$command")
+echo "$command"
+eval "$command"
+echo "Please go to https://portal.azure.com/ to config environment variables and restart the app: $name at (Settings>Configuration) or (Settings>Environment variables)"
+echo "Reach deployment logs at (Deployment>Deployment Central) and app logs at (Monitoring>Log stream)"
+echo "Reach advanced deployment tools at https://$name.scm.azurewebsites.net/"
+echo "Reach more details about app service at https://learn.microsoft.com/en-us/azure/app-service/"
@@ -0,0 +1,44 @@
+---
+resources: examples/tutorials/flow-deploy/create-service-with-flow
+category: deployment
+weight: 10
+---
+
+# Create service with flow
+
+This example shows how to create a simple service with flow.
+
+You can create your own service by utilize `flow-as-function`.
+
+This folder contains a example on how to build a service with a flow.
+Reference [here](./simple_score.py) for a minimal service example.
+The output of score.py will be a json serialized dictionary.
+You can use json parser to parse the output.
+
+## 1. Start the service and put in background
+
+```bash
+nohup python simple_score.py &
+# Note: added this to run in our CI pipeline, not needed for user.
+sleep 10
+```
+
+## 2. Test the service with request
+
+Executing the following command to send a request to execute a flow.
+
+```bash
+curl -X POST http://127.0.0.1:5000/score --header "Content-Type: application/json" --data '{"flow_input": "some_flow_input", "node_input": "some_node_input"}'
+```
+
+Sample output of the request:
+
+```json
+{
+  "output": {
+    "value": "some_flow_input"
+  }
+}
+```
+
+Reference [here](./simple_score.py) for more.
@@ -0,0 +1,11 @@
+from promptflow.core import tool
+from promptflow.connections import AzureOpenAIConnection
+
+
+@tool
+def echo_connection(flow_input: str, node_input: str, connection: AzureOpenAIConnection):
+    print(f"Flow input: {flow_input}")
+    print(f"Node input: {node_input}")
+    print(f"Flow connection: {connection._to_dict()}")
+    # get from env var
+    return {"value": flow_input}
@@ -0,0 +1,18 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+inputs:
+  flow_input:
+    type: string
+outputs:
+  output:
+    type: object
+    reference: ${echo_connection.output}
+nodes:
+- name: echo_connection
+  type: python
+  source:
+    type: code
+    path: echo_connection.py
+  inputs:
+    flow_input: ${inputs.flow_input}
+    node_input: "dummy_node_input"
+    connection: open_ai_connection
@@ -0,0 +1,85 @@
+import json
+import logging
+
+from flask import Flask, jsonify, request
+
+from promptflow.client import load_flow
+from promptflow.connections import AzureOpenAIConnection
+from promptflow.entities import FlowContext
+from promptflow.exceptions import SystemErrorException, UserErrorException
+
+
+class SimpleScoreApp(Flask):
+    pass
+
+
+app = SimpleScoreApp(__name__)
+logging.basicConfig(format="%(threadName)s:%(message)s")
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+# load flow as a function, the function object can be shared accross threads.
+f = load_flow("./echo_connection_flow/")
+
+
+@app.errorhandler(Exception)
+def handle_error(e):
+    if isinstance(e, UserErrorException):
+        return jsonify({"message": e.message, "additional_info": e.additional_info}), 400
+    elif isinstance(e, SystemErrorException):
+        return jsonify({"message": e.message, "additional_info": e.additional_info}), 500
+    else:
+        from promptflow._internal import ErrorResponse, ExceptionPresenter
+
+        # handle other unexpected errors, can use internal class to format them
+        # but interface may change in the future
+        presenter = ExceptionPresenter.create(e)
+        trace_back = presenter.formatted_traceback
+
+        resp = ErrorResponse(presenter.to_dict(include_debug_info=False))
+        response_code = resp.response_code
+        result = resp.to_simplified_dict()
+        result.update({"trace_back": trace_back})
+        return jsonify(result), response_code
+
+
+@app.route("/health", methods=["GET"])
+def health():
+    """Check if the runtime is alive."""
+    return {"status": "Healthy"}
+
+
+@app.route("/score", methods=["POST"])
+def score():
+    """process a flow request in the runtime."""
+    raw_data = request.get_data()
+    logger.info(f"Start loading request data '{raw_data}'.")
+    data = json.loads(raw_data)
+
+    # create a dummy connection object
+    # the connection object will only exist in memory and won't store in local db.
+    llm_connection = AzureOpenAIConnection(
+        name="llm_connection", api_key="[determined by request]", api_base="[determined by request]"
+    )
+
+    # configure flow contexts, create a new context object for each request to make sure they are thread safe.
+    f.context = FlowContext(
+        # override flow connections with connection object created above
+        connections={"echo_connection": {"connection": llm_connection}},
+        # override the flow nodes' inputs or other flow configs, the overrides may come from the request
+        # **Note**: after this change, node "echo_connection" will take input node_input from request
+        overrides={"nodes.echo_connection.inputs.node_input": data["node_input"]} if "node_input" in data else {},
+    )
+    # data in request will be passed to flow as kwargs
+    result_dict = f(**data)
+    # Note: if specified streaming=True in the flow context, the result will be a generator
+    # reference promptflow.core._serving.response_creator.ResponseCreator on how to handle it in app.
+    return jsonify(result_dict)
+
+
+def create_app(**kwargs):
+    return app
+
+
+if __name__ == "__main__":
+    # test this with curl -X POST http://127.0.0.1:5000/score --header "Content-Type: application/json" --data '{"flow_input": "some_flow_input", "node_input": "some_node_input"}'  # noqa: E501
+    create_app().run(debug=True)
@@ -0,0 +1,80 @@
+---
+resources: examples/connections/azure_openai.yml, examples/flows/standard/web-classification
+category: deployment
+weight: 50
+---
+
+# Distribute flow as executable app
+This example demos how to package flow as a executable app.
+We will use [web-classification](../../../flows/standard/web-classification/README.md) as example in this tutorial.
+
+Please ensure that you have installed all the required dependencies. You can refer to the "Prerequisites" section in the README of the [web-classification](https://github.com/microsoft/promptflow/tree/main/examples/flows/standard/web-classification/) for a comprehensive list of prerequisites and installation instructions. And we recommend you to add a `requirements.txt` to indicate all the required dependencies for each flow.
+
+[Pyinstaller](https://pyinstaller.org/en/stable/installation.html) is a popular tool used for converting Python applications into standalone executables. It allows you to package your Python scripts into a single executable file, which can be run on a target machine without requiring the Python interpreter to be installed.
+[Streamlit](https://docs.streamlit.io/library/get-started) is an open-source Python library used for creating web applications quickly and easily. It's designed for data scientists and engineers who want to turn data scripts into shareable web apps with minimal effort.
+We use Pyinstaller to package the flow and Streamlit to create custom web apps. Prior to distributing the workflow, kindly ensure that you have installed them.
+In this example, we use PyInstaller version 5.13.2 and Streamlit version 1.26.0 within a Python 3.10.8 environment.
+
+## Build a flow as executable format
+Note that all dependent connections must be created before building as executable.
+```bash
+# create connection if not created before
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+Use the command below to build a flow as executable format app:
+```shell
+pf flow build --source ../../../flows/standard/web-classification --output target --format executable
+```
+
+## Executable format folder structure
+Exported files & its dependencies are located in the same folder. The structure is as below:
+- flow: the folder contains all the flow files.
+- connections: the folder contains yaml files to create all related connections.
+- app.py: the entry file is included as the entry point for the bundled application.
+- app.spec: the spec file tells PyInstaller how to process your script.
+- main.py: it will start Streamlit service and be called by the entry file.
+- settings.json: a json file to store the settings of the executable application.
+- build: a folder contains various log and working files.
+- dist: a folder contains the executable application.
+- README.md: Simple introduction of the files.
+
+### A template script of the entry file
+PyInstaller reads a spec file or Python script written by you. It analyzes your code to discover every other module and library your script needs in order to execute. Then it collects copies of all those files, including the active Python interpreter, and puts them with your script in a single folder, or optionally in a single executable file.
+
+We provide a Python entry script named [app.py](https://github.com/microsoft/promptflow/blob/main/src/promptflow-devkit/promptflow/_sdk/data/executable/app.py) as the entry point for the bundled app, which enables you to serve a flow folder as an endpoint.
+
+
+
+### A template script of the spec file
+The spec file tells PyInstaller how to process your script. It encodes the script names and most of the options you give to the pyinstaller command. The spec file is actually executable Python code. PyInstaller builds the app by executing the contents of the spec file.
+
+To streamline this process, we offer a [app.spec.jinja2](https://github.com/microsoft/promptflow/blob/main/src/promptflow-devkit/promptflow/_sdk/data/executable/app.spec.jinja2) spec template file that bundles the application into a single file. For additional information on spec files, you can refer to the [Using Spec Files](https://pyinstaller.org/en/stable/spec-files.html).
+Please replace {{streamlit_runtime_interpreter_path}} with the path of streamlit runtime interpreter in your environment.
+
+
+### The bundled application using Pyinstaller
+Once you've build a flow as executable format following [Build a flow as executable format](#build-a-flow-as-executable-format).
+It will create two folders named `build` and `dist` within your specified output directory, denoted as <your-output-dir>. The `build` folder houses various log and working files, while the `dist` folder contains the `app` executable application.
+
+#### Connections
+If the service involves connections, all related connections will be exported as yaml files and recreated in the executable package.
+Secrets in connections won't be exported directly. Instead, we will export them as a reference to environment variables:
+```yaml
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/OpenAIConnection.schema.json
+type: open_ai
+name: open_ai_connection
+module: promptflow.connections
+api_key: ${env:OPEN_AI_CONNECTION_API_KEY} # env reference
+```
+
+## Test the endpoint
+Finally, You can distribute the bundled application `app` to other people. They can execute your program by double clicking the executable file, e.g. `app.exe` in Windows system or running the binary file, e.g. `app` in Linux system.
+
+The development server has a built-in web page they can use to test the flow by opening 'http://localhost:8501' in the browser. The expected result is as follows: if the flow served successfully, the process will keep alive until it is killed manually.
+
+To your users, the app is self-contained. They do not need to install any particular version of Python or any modules. They do not need to have Python installed at all.
+
+**Note**: The executable generated is not cross-platform. One platform (e.g. Windows) packaged executable can't run on others (Mac, Linux).
+
+## Known issues
+1. Note that Python 3.10.0 contains a bug making it unsupportable by PyInstaller. PyInstaller will also not work with beta releases of Python 3.13.
@@ -0,0 +1,67 @@
+---
+resources: examples/connections/azure_openai.yml, examples/flows/standard/web-classification
+category: deployment
+weight: 40
+---
+
+# Deploy a flow using Docker
+
+This example demos how to deploy flow as a docker app.
+We will use [web-classification](../../../flows/standard/web-classification/README.md) as example in this tutorial.
+
+## Build a flow as docker format app
+
+Note that all dependent connections must be created before building as docker.
+```bash
+# create connection if not created before
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Use the command below to build a flow as docker format app:
+
+```bash
+pf flow build --source ../../../flows/standard/web-classification --output dist --format docker
+```
+
+## Deploy with Docker
+### Build Docker image
+
+Like other Dockerfile, you need to build the image first. You can tag the image with any name you want. In this example, we use `promptflow-serve`.
+
+Run the command below to build image:
+
+```shell
+docker build dist -t web-classification-serve
+```
+
+### Run Docker image
+
+Run the docker image will start a service to serve the flow inside the container.
+
+#### Connections
+If the service involves connections, all related connections will be exported as yaml files and recreated in containers.
+Secrets in connections won't be exported directly. Instead, we will export them as a reference to environment variables:
+```yaml
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/OpenAIConnection.schema.json
+type: open_ai
+name: open_ai_connection
+module: promptflow.connections
+api_key: ${env:OPEN_AI_CONNECTION_API_KEY} # env reference
+```
+You'll need to set up the environment variables in the container to make the connections work.
+
+### Run with `docker run`
+
+
+You can run the docker image directly set via below commands:
+```shell
+# The started service will listen on port 8080.You can map the port to any port on the host machine as you want.
+docker run -p 8080:8080 -e OPEN_AI_CONNECTION_API_KEY=<secret-value> web-classification-serve
+```
+
+### Test the endpoint
+After start the service, you can use curl to test it:
+
+```shell
+curl http://localhost:8080/score --data '{"url":"https://play.google.com/store/apps/details?id=com.twitter.android"}' -X POST  -H "Content-Type: application/json"
+```
@@ -0,0 +1,162 @@
+---
+resources: examples/connections/azure_openai.yml, examples/flows/standard/web-classification
+category: deployment
+weight: 70
+---
+
+# Deploy flow using Kubernetes
+This example demos how to deploy flow as a Kubernetes app.
+We will use [web-classification](../../../flows/standard/web-classification/README.md) as example in this tutorial.
+
+Please ensure that you have installed all the required dependencies. You can refer to the "Prerequisites" section in the README of the [web-classification](../../../flows/standard/web-classification/README.md#Prerequisites) for a comprehensive list of prerequisites and installation instructions.
+
+## Build a flow as docker format
+
+Note that all dependent connections must be created before building as docker.
+```bash
+# create connection if not created before
+pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Use the command below to build a flow as docker format app:
+
+```bash
+pf flow build --source ../../../flows/standard/web-classification --output dist --format docker
+```
+
+## Deploy with Kubernetes
+### Build Docker image
+
+Like other Dockerfile, you need to build the image first. You can tag the image with any name you want. In this example, we use `web-classification-serve`.
+
+Then run the command below:
+
+```shell
+cd dist
+docker build . -t web-classification-serve
+```
+
+### Create Kubernetes deployment yaml.
+The Kubernetes deployment yaml file acts as a guide for managing your docker container in a Kubernetes pod. It clearly specifies important information like the container image, port configurations, environment variables, and various settings. Below, you'll find a simple deployment template that you can easily customize to meet your needs.
+
+**Note**: You need encode the secret using base64 firstly and input the <encoded_secret> as 'open-ai-connection-api-key' in the deployment configuration. For example, you can run below commands in linux:
+```shell
+encoded_secret=$(echo -n <your_api_key> | base64)
+```
+
+```yaml
+---
+kind: Namespace
+apiVersion: v1
+metadata:
+  name: web-classification
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: open-ai-connection-api-key
+  namespace: web-classification
+type: Opaque
+data:
+  open-ai-connection-api-key: <encoded_secret>
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: web-classification-service
+  namespace: web-classification
+spec:
+  type: NodePort
+  ports:
+  - name: http
+    port: 8080
+    targetPort: 8080
+    nodePort: 30123
+  selector:
+    app: web-classification-serve-app
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: web-classification-serve-app
+  namespace: web-classification
+spec:
+  selector:
+    matchLabels:
+      app: web-classification-serve-app
+  template:
+    metadata:
+      labels:
+        app: web-classification-serve-app
+    spec:
+      containers:
+      - name: web-classification-serve-container
+        image: web-classification-serve
+        imagePullPolicy: Never
+        ports:
+        - containerPort: 8080
+        env:
+        - name: OPEN_AI_CONNECTION_API_KEY
+          valueFrom:
+            secretKeyRef:
+              name: open-ai-connection-api-key
+              key: open-ai-connection-api-key
+```
+
+### Apply the deployment.
+Before you can deploy your application, ensure that you have set up a Kubernetes cluster and installed [kubectl](https://kubernetes.io/docs/reference/kubectl/) if it's not already installed. In this documentation, we will use [Minikube](https://minikube.sigs.k8s.io/docs/) as an example. To start the cluster, execute the following command:
+```shell
+minikube start
+```
+Once your Kubernetes cluster is up and running, you can proceed to deploy your application by using the following command:
+```shell
+kubectl apply -f deployment.yaml
+```
+This command will create the necessary pods to run your application within the cluster.
+
+**Note**: You need replace <pod_name> below with your specific pod_name. You can retrieve it by running `kubectl get pods -n web-classification`.
+
+### Retrieve flow service logs of the container
+The kubectl logs command is used to retrieve the logs of a container running within a pod, which can be useful for debugging, monitoring, and troubleshooting applications deployed in a Kubernetes cluster.
+
+```shell
+kubectl -n web-classification logs <pod-name>
+```
+
+#### Connections
+If the service involves connections, all related connections will be exported as yaml files and recreated in containers.
+Secrets in connections won't be exported directly. Instead, we will export them as a reference to environment variables:
+```yaml
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/OpenAIConnection.schema.json
+type: open_ai
+name: open_ai_connection
+module: promptflow.connections
+api_key: ${env:OPEN_AI_CONNECTION_API_KEY} # env reference
+```
+You'll need to set up the environment variables in the container to make the connections work.
+
+### Test the endpoint
+- Option1:
+
+  Once you've started the service, you can establish a connection between a local port and a port on the pod. This allows you to conveniently test the endpoint from your local terminal.
+  To achieve this, execute the following command:
+
+  ```shell
+  kubectl port-forward <pod_name> 8080:8080 -n web-classification
+  ```
+  With the port forwarding in place, you can use the curl command to initiate the endpoint test:
+
+  ```shell
+  curl http://localhost:8080/score --data '{"url":"https://play.google.com/store/apps/details?id=com.twitter.android"}' -X POST  -H "Content-Type: application/json"
+  ```
+
+- Option2:
+
+  `minikube service web-classification-service --url -n web-classification` runs as a process, creating a tunnel to the cluster. The command exposes the service directly to any program running on the host operating system.
+
+  The command above will retrieve the URL of a service running within a Minikube Kubernetes cluster (e.g. http://<ip>:<assigned_port>), which you can click to interact with the flow service in your web browser. Alternatively, you can use the following command to test the endpoint:
+
+  **Note**: Minikube will use its own external port instead of nodePort to listen to the service. So please substitute <assigned_port> with the port obtained above.
+    ```shell
+  curl http://localhost:<assigned_port>/score --data '{"url":"https://play.google.com/store/apps/details?id=com.twitter.android"}' -X POST  -H "Content-Type: application/json"
+  ```
@@ -0,0 +1,56 @@
+---
+kind: Namespace
+apiVersion: v1
+metadata:
+  name: web-classification
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: open-ai-connection-api-key
+  namespace: web-classification
+type: Opaque
+data:
+  open-ai-connection-api-key: <encoded_secret>
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: web-classification-service
+  namespace: web-classification
+spec:
+  type: NodePort
+  ports:
+  - name: http
+    port: 8080
+    targetPort: 8080
+    nodePort: 30123
+  selector:
+    app: web-classification-serve-app
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: web-classification-serve-app
+  namespace: web-classification
+spec:
+  selector:
+    matchLabels:
+      app: web-classification-serve-app
+  template:
+    metadata:
+      labels:
+        app: web-classification-serve-app
+    spec:
+      containers:
+      - name: web-classification-serve-container
+        image: web-classification-serve
+        imagePullPolicy: Never
+        ports:
+        - containerPort: 8080
+        env:
+        - name: OPEN_AI_CONNECTION_API_KEY
+          valueFrom:
+            secretKeyRef:
+              name: open-ai-connection-api-key
+              key: open-ai-connection-api-key
@@ -0,0 +1,501 @@
+---
+resources: examples/connections/azure_openai.yml, examples/flows/chat/chat-basic, examples/flows/chat/chat-math-variant, examples/flows/evaluation/eval-chat-math
+category: rag
+---
+
+# Tutorial: How prompt flow helps on quality improvement
+
+This tutorial is designed to enhance your understanding of improving flow quality through prompt tuning and evaluation.
+
+Embark on a journey to overcome the inherent randomness of Language Models (LLMs) and enhance output reliability through **prompt fine-tuning** with this comprehensive tutorial. Explore how prompt flow can simplify this process, enabling you to swiftly build high-quality, LLM-native apps.
+
+Prompt fine-tuning involves optimizing the input prompts given to an LLM. This strategic adjustment helps the model to focus on specific information needed for a task, thereby improving the accuracy and reliability of the LLM's responses.
+
+When we talk about "high quality", it's not just about accuracy. It's equally important to strike a balance between the accuracy and the token cost of the LLM. Spend just 15 minutes with us to discover how prompt flow expedites the process of prompt tuning, testing, and evaluation, guiding you towards finding the ideal prompt **(accuracy ↑,token ↓)**
+
+<img src="./media/realcase.png" alt="comparison result" width=60%>
+
+## Video tutorial
+
+Before practicing, you can watch the video for a quick understand. This video shows how to use the **prompt flow VS code extension** to develop your chat flow, fine tune the prompt, batch test the flow, and evaluate the quality.
+
+<a href="http://www.youtube.com/watch?feature=player_embedded&v=gcIe6nk2gA4
+" target="_blank"><img src="./media/Screenshot-video.png"
+alt="video demo" border="5" /></a>
+
+## Hands-on practice
+
+* Option 1 - VS Code Extension: [Install the prompt flow extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) in VS Code and follow the [video tutorial](https://youtu.be/gcIe6nk2gA4) above for a guided practice.
+* Option 2 - CLI：Follow the steps below to gain hands-on experience with the prompt flow CLI.
+
+It's time to put theory into practice! Execute our sample and witness the effects.
+
+### Prerequisite
+
+Before moving ahead, ensure you've completed the [Quick Start](../../../README.md#get-started-with-prompt-flow-⚡) guidance. Ensure you have the following setup:
+* [Install prompt flow](../../../README.md#installation)
+* [Setup a connection for your API key](../../../README.md#quick-start-⚡)
+
+> ℹ️ For testing quickly, this tutorial uses CLI command.
+
+Clone the promptflow repository to your local machine:
+
+```shell
+git clone https://github.com/microsoft/promptflow.git
+```
+
+Setup sample `open_ai_connection` connection
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Next, let's get started with customizing the flow for a specific task.
+
+### Customize the flow for a specific task
+
+In the `promptflow/examples/flows/chat` folder, you can see a `chat-basic` folder, which represents a chat template flow as same as the one you created in the [Quick Start](../../../README.md#get-started-with-prompt-flow-⚡) guidance. We'll use this flow as a starting point to build a math problem solver.
+
+```bash
+cd ../../flows/chat/chat-basic/
+```
+
+To enable your chatbot flow to solve math problems, you need to instruct the LLM about the task and target in the prompt. Open `chat.jinja2`, update the prompt as below:
+
+```jinja
+# system:
+You are an assistant to calculate the answer to the provided math problems.
+Please return the final numerical answer only, without any accompanying reasoning or explanation.
+
+{% for item in chat_history %}
+# user:
+{{item.inputs.question}}
+# assistant:
+{{item.outputs.answer}}
+{% endfor %}
+
+# user:
+{{question}}
+```
+
+Before run, check your connection settings in `flow.dag.yaml` file. The default connection name is `open_ai_connection`, and the default model is `gpt-3.5-turbo`. If you have a different connection name or model, please modify the `flow.dag.yaml` file accordingly.
+
+><details>
+><summary>(click to toggle details) For example, if you use Azure OpenAI, please modify the `flow.dag.yaml` file to specify your connection and deployment</summary>
+>
+> Replace the 'node:' section with following content, specify the 'connection_name' to your Azure OpenAI connection, and specify the 'deployment_name' to the model deployment you'd like to use.
+> ```yaml
+>nodes:
+>- name: chat
+>  type: llm
+>  source:
+>    type: code
+>    path: chat.jinja2
+>  inputs:
+>    deployment_name: <your_azure_open_ai_deployment_name> #specify your deployment name
+>    max_tokens: '256'
+>    temperature: '0'
+>    chat_history: ${inputs.chat_history}
+>    question: ${inputs.question}
+>  api: chat
+>  connection: <your_azure_open_ai_connection_name> #specify your azure openai connection name
+> ```
+</details>
+
+Go back to the `promptflow/examples/flows/chat` path, run the following command to test the flow with a simple math problem:
+
+```bash
+cd ..
+pf flow test --flow ./chat-basic --inputs question="1+1=?"
+```
+
+This will yield the following output:
+```json
+{
+    "answer": "2"
+}
+```
+
+Sometime, the question may be challenging. Now, let's test it with a complex math problem, such as:
+
+```bash
+pf flow test --flow ./chat-basic --inputs question="We are allowed to remove exactly one integer from the list $$-1,0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,11,$$and then we choose two distinct integers at random from the remaining list. What number should we remove if we wish to maximize the probability that the sum of the two chosen numbers is 10?"
+```
+
+The output is:
+```json
+{
+    "answer": "-1"
+}
+```
+However, the correct answer is 5, so the output answer is incorrect! (Don't be surprised if you got the correct answer, as the randiness of LLM. You can try multiple times for different answers.) It indicates that we need to further evaluate the performance. Therefore, in the next step, we will test the flow with more math problems to better evaluate the quality.
+
+### Evaluate the quality of your prompt
+
+With prompt flow, you can quickly trigger a batch-run to test your prompt with a larger dataset, and evaluate the quality of the answers.
+
+There is a `data.jsonl` file in the `promptflow/examples/flows/chat/chat-math-variant` folder, which is a dataset containing 20 test data entries (a subset of [the Math Dataset](https://github.com/hendrycks/math/)). It includes the input question, the ground truth for numerical answer, and the reasoning (raw_answer). Here's one example:
+
+```json
+{
+    "question": "Determine the number of ways to arrange the letters of the word PROOF.",
+    "answer": "60",
+    "raw_answer": "There are two O's and five total letters, so the answer is $\\dfrac{5!}{2!} = \\boxed{60}$."
+}
+
+```
+
+Run the following command to test your prompt with this dataset:
+
+First, set the environment variable `base_run_name` to specify the run name.
+
+```bash
+base_run_name="base_run"
+```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle </summary>
+
+```shell
+set base_run_name=base_run
+```
+
+</details>
+
+
+>ℹ️ The default model is `gpt-turbo-3.5`, let's try `gpt-4` to see if it's smarter to get better results. Use `--connections <node_name>.connection=<connection_name>...`to specify.
+
+```bash
+pf run create --flow ./chat-basic --data ./chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --connections chat.connection=open_ai_connection chat.model=gpt-4 --stream --name $base_run_name
+```
+
+>ℹ️ For Azure OpenAI, run the following command instead:
+> ```shell
+> pf run create --flow ./chat_math_variant --data test_data.jsonl --column-mapping question='${data.question}' chat_history=[] --connections chat.connection=azure_open_ai_connection chat.deployment_name=gpt-4 --stream --name $base_run_name
+> ```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+pf run create --flow ./chat-basic --data ./chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --connections chat.connection=open_ai_connection chat.model=gpt-4 --stream --name %base_run_name%
+```
+
+</details>
+
+> ℹ️ The run name must be unique. Please specify a new name in `--name`.
+> If you see "Run 'base_run' already exists.", you can specify another name. But please remember the name you specified, because you'll need it in the next step.
+
+
+When it completes, you can run the following command to see the details of results:
+> Specify the run name of your completed run in `--name` argument:
+
+```bash
+pf run show-details --name $base_run_name
+```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+pf run show-details --name %base_run_name%
+```
+
+</details>
+
+This can show the line by line input and output of the run:
+```
+----+---------------+-----------------+---------------+---------------+
+|    | inputs.chat   | inputs.question |   inputs.line | outputs.ans   |
+|    | _history      |                 |       _number | wer           |
+====+===============+=================+===============+===============+
+|  0 | []            | Compute $\dbi   |             0 | 4368          |
+|    |               | nom{16}{5}$.    |               |               |
+----+---------------+-----------------+---------------+---------------+
+|  1 | []            | Determine the   |             1 | 60            |
+|    |               | number of       |               |               |
+|    |               | ways to         |               |               |
+|    |               | arrange the     |               |               |
+|    |               | letters of      |               |               |
+|    |               | the word        |               |               |
+|    |               | PROOF.          |               |               |
+----+---------------+-----------------+---------------+---------------+
+| .. | ...           | ...             |...            | ...           |
+```
+
+Next, create an **evaluation run** to calculate the accuracy of the answers based on the previous run.
+
+In the `promptflow/examples/flows/evaluation` folder, you can see a `eval-chat-math` folder, which represents an evaluation flow. We'll use this flow to evaluate the accuracy of the answers.
+
+```bash
+cd ../evaluation
+```
+
+Run the following command to create an evaluation run:
+
+```bash
+eval_run_name="eval_run"
+pf run create --flow ./eval-chat-math --data ../chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --stream --run $base_run_name --name $eval_run_name
+```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+set eval_run_name=eval_run
+pf run create --flow ./eval-chat-math --data ../chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --run %base_run_name% --name %eval_run_name% --stream
+```
+
+</details>
+
+> If needed, specify the run name which you want to evaluate in `--run` argument, and specify this evaluation run name in `--name` argument.
+
+Then get metrics of the `eval_run`:
+
+```bash
+pf run show-metrics --name $eval_run_name
+```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+pf run show-details --name %eval_run_name%
+```
+
+</details>
+
+You can visualize and compare the output line by line of `base_run` and `eval_run` in a web browser:
+
+```bash
+pf run visualize --name "$base_run_name,$eval_run_name"
+```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+pf run visualize --name "%base_run_name%,%eval_run_name%"
+```
+
+</details>
+
+Because of the randomness of the LLM, the accuracy may vary. For example, in my run, the metrics are as follows:
+
+```json
+{
+    "accuracy": 0.35,
+    "error_rate": 0.65
+}
+```
+
+Oops! The accuracy isn't satisfactory. It's time to fine-tune your prompt for higher quality!
+
+### Fine-tuning your prompt and evaluate the improvement
+
+In the `/chat` folder, you can see a `chat-math-variant` folder, which represents a flow with two additional prompt variants compared to the original one you customized based on the `chat-basic`.
+
+In this sample flow, you'll find three Jinja files:
+* `chat.jinja2` is the original prompt as same as the one you customized in `chat-basic`.
+* `chat_variant_1.jinja2` and `chat_variant_2.jinja2` are the 2 additional prompt variants.
+
+We leverage the Chain of Thought (CoT) prompt engineering method to adjust the prompt. The goal is to activate the Language Model's reasoning capability of the questions, by providing a few CoT examples.
+
+<details>
+<summary>Variant_1: 2 CoT examples</summary>
+
+```jinja
+# system:
+You are an assistant to calculate the answer to the provided math problems.
+Please think step by step.
+Return the final numerical answer only and any accompanying reasoning or explanation seperately as json format. <br>
+# user:
+A jar contains two red marbles, three green marbles, ten white marbles and no other marbles. Two marbles are randomly drawn from this jar without replacement. What is the probability that these two marbles drawn will both be red? Express your answer as a common fraction.
+# assistant:
+{Chain of thought: "The total number of marbles is $2+3+10=15$.  The probability that the first marble drawn will be red is $2/15$.  Then, there will be one red left, out of 14.  Therefore, the probability of drawing out two red marbles will be: $$\\frac{2}{15}\\cdot\\frac{1}{14}=\\boxed{\\frac{1}{105}}$$.", "answer": "1/105"}
+# user:
+Find the greatest common divisor of $7!$ and $(5!)^2.$
+# assistant:
+{"Chain of thought": "$$ \\begin{array} 7! &=& 7 \\cdot 6 \\cdot 5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1 &=& 2^4 \\cdot 3^2 \\cdot 5^1 \\cdot 7^1 \\\\ (5!)^2 &=& (5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1)^2 &=& 2^6 \\cdot 3^2 \\cdot 5^2 \\\\ \\text{gcd}(7!, (5!)^2) &=& 2^4 \\cdot 3^2 \\cdot 5^1 &=& \\boxed{720} \\end{array} $$.", "answer": "720"}
+```
+</details>
+
+<details>
+<summary>Variant_2 : 6 CoT examples.</summary>
+
+```jinja
+# system:
+You are an assistant to calculate the answer to the provided math problems.
+Please think step by step.
+Return the final numerical answer only and any accompanying reasoning or explanation seperately as json format.
+
+# user:
+A jar contains two red marbles, three green marbles, ten white marbles and no other marbles. Two marbles are randomly drawn from this jar without replacement. What is the probability that these two marbles drawn will both be red? Express your answer as a common fraction.
+# assistant:
+{Chain of thought: "The total number of marbles is $2+3+10=15$.  The probability that the first marble drawn will be red is $2/15$.  Then, there will be one red left, out of 14.  Therefore, the probability of drawing out two red marbles will be: $$\\frac{2}{15}\\cdot\\frac{1}{14}=\\boxed{\\frac{1}{105}}$$.", "answer": "1/105"}
+# user:
+Find the greatest common divisor of $7!$ and $(5!)^2.$
+# assistant:
+{"Chain of thought": "$$ \\begin{array} 7! &=& 7 \\cdot 6 \\cdot 5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1 &=& 2^4 \\cdot 3^2 \\cdot 5^1 \\cdot 7^1 \\\\ (5!)^2 &=& (5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1)^2 &=& 2^6 \\cdot 3^2 \\cdot 5^2 \\\\ \\text{gcd}(7!, (5!)^2) &=& 2^4 \\cdot 3^2 \\cdot 5^1 &=& \\boxed{720} \\end{array} $$.", "answer": "720"}
+# user:
+A club has 10 members, 5 boys and 5 girls.  Two of the members are chosen at random.  What is the probability that they are both girls?
+# assistant:
+{"Chain of thought": "There are $\\binomial{10}{2} = 45$ ways to choose two members of the group, and there are $\\binomial{5}{2} = 10$ ways to choose two girls.  Therefore, the probability that two members chosen at random are girls is $\\dfrac{10}{45} = \\boxed{\\dfrac{2}{9}}$.", "answer": "2/9"}
+# user:
+Allison, Brian and Noah each have a 6-sided cube. All of the faces on Allison's cube have a 5. The faces on Brian's cube are numbered 1, 2, 3, 4, 5 and 6. Three of the faces on Noah's cube have a 2 and three of the faces have a 6. All three cubes are rolled. What is the probability that Allison's roll is greater than each of Brian's and Noah's? Express your answer as a common fraction.
+# assistant:
+{"Chain of thought": "Since Allison will always roll a 5, we must calculate the probability that both Brian and Noah roll a 4 or lower. The probability of Brian rolling a 4 or lower is $\\frac{4}{6} = \\frac{2}{3}$ since Brian has a standard die. Noah, however, has a $\\frac{3}{6} = \\frac{1}{2}$ probability of rolling a 4 or lower, since the only way he can do so is by rolling one of his 3 sides that have a 2. So, the probability of both of these independent events occurring is $\\frac{2}{3} \\cdot \\frac{1}{2} = \\boxed{\\frac{1}{3}}$.", "answer": "1/3"}
+# user:
+Compute $\\density binomial{50}{2}$.
+# assistant:
+{"Chain of thought": "$\\density binomial{50}{2} = \\dfrac{50!}{2!48!}=\\dfrac{50\\times 49}{2\\times 1}=\\boxed{1225}.$", "answer": "1225"}
+# user:
+The set $S = \\{1, 2, 3, \\ldots , 49, 50\\}$ contains the first $50$ positive integers.  After the multiples of 2 and the multiples of 3 are removed, how many integers remain in the set $S$?
+# assistant:
+{"Chain of thought": "The set $S$ contains $25$ multiples of 2 (that is, even numbers).  When these are removed, the set $S$ is left with only the odd integers from 1 to 49. At this point, there are $50-25=25$ integers in $S$. We still need to remove the multiples of 3 from $S$.\n\nSince $S$ only contains odd integers after the multiples of 2 are removed,  we must remove the odd multiples of 3 between 1 and 49.  These are 3, 9, 15, 21, 27, 33, 39, 45, of which there are 8.  Therefore, the number of integers remaining in the set $S$ is $25 - 8 = \\boxed{17}$.", "answer": "17"}
+```
+</details>
+
+These two jinja files are specified in the `flow.dag.yaml` file, which defines the flow structure. You can see that the `chat` node has 3 variants, which point to these 3 Jinja files.
+
+### Test and evaluate your prompt variants
+
+First, you need to modify your flow to add two more prompt variants into the chat node, in addition to the existed default one. In the flow.dag.yaml file, you can see 3 variants definition of the `chat` node, which point to these 3 Jinja files.
+
+Run the CLI command below to start the experiment: test all variants, evaluate them, get the visualized comparison results of the experiment.
+
+> ℹ️ By default, the connection is set to `open_ai_connection` and and the model is set to `gpt-4` for each variant, as specified in the `flow.dag.yaml` file. However, you have the flexibility to specify a different connection and model by adding `--connections chat.connection=<your_connection_name> chat.deployment_name=<model_name>` in the test run command.
+
+Navigate to the `promptflow/examples/flows` folder
+
+```bash
+cd ..
+```
+
+Set the environment variable `base_run_name` and `eval_run_name` to specify the run name.
+
+```bash
+base_run_name="base_run_variant_"
+eval_run_name="eval_run_variant_"
+```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+set base_run_name=base_run_variant_
+set eval_run_name=eval_run_variant_
+```
+
+</details>
+
+Run the following command to test and evaluate the variants:
+
+```bash
+# Test and evaluate variant_0:
+# Test-run
+pf run create --flow ./chat/chat-math-variant --data ./chat/chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --variant '${chat.variant_0}' --stream  --name "${base_run_name}0"
+# Evaluate-run
+pf run create --flow ./evaluation/eval-chat-math --data ./chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --stream --run "${base_run_name}0" --name "${eval_run_name}0"
+
+# Test and evaluate variant_1:
+# Test-run
+pf run create --flow ./chat/chat-math-variant --data ./chat/chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --variant '${chat.variant_1}' --stream --name "${base_run_name}1"
+# Evaluate-run
+pf run create --flow ./evaluation/eval-chat-math --data ./chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --stream --run "${base_run_name}1" --name "${eval_run_name}1"
+
+# Test and evaluate variant_2:
+# Test-run
+pf run create --flow ./chat/chat-math-variant --data ./chat/chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --variant '${chat.variant_2}' --stream --name "${base_run_name}2"
+# Evaluate-run
+pf run create --flow ./evaluation/eval-chat-math --data ./chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --stream --run "${base_run_name}2" --name "${eval_run_name}2"
+```
+
+<!-- > If encounter the 'execution timeout' error, just try again. It might be caused by the LLM service congestion. -->
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+# Test and evaluate variant_0:
+# Test-run
+pf run create --flow ./chat/chat-math-variant --data ./chat/chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --variant '${chat.variant_0}' --stream --name %base_run_name%0
+# Evaluate-run
+pf run create --flow ./evaluation/eval-chat-math --data ./chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --stream --run %base_run_name%0 --name %eval_run_name%0
+
+# Test and evaluate variant_1:
+# Test-run
+pf run create --flow ./chat/chat-math-variant --data ./chat/chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --variant '${chat.variant_1}' --stream --name %base_run_name%1
+# Evaluate-run
+pf run create --flow ./evaluation/eval-chat-math --data ./chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --stream --run %base_run_name%1 --name %eval_run_name%1
+
+# Test and evaluate variant_2:
+# Test-run
+pf run create --flow ./chat/chat-math-variant --data ./chat/chat-math-variant/data.jsonl --column-mapping question='${data.question}' chat_history=[] --variant '${chat.variant_2}' --stream --name %base_run_name%2
+# Evaluate-run
+pf run create --flow ./evaluation/eval-chat-math --data ./chat/chat-math-variant/data.jsonl --column-mapping groundtruth='${data.answer}' prediction='${run.outputs.answer}' --stream --run %base_run_name%2 --name %eval_run_name%2
+```
+
+</details>
+
+Get metrics of the all evaluations:
+
+```bash
+pf run show-metrics --name "${eval_run_name}0"
+pf run show-metrics --name "${eval_run_name}1"
+pf run show-metrics --name "${eval_run_name}2"
+```
+
+You may get the familiar output like this:
+```
+# eval_variant_0_run
+{
+    "accuracy": 0.3,
+    "error_rate": 0.7
+}
+# eval_variant_1_run
+{
+    "accuracy": 0.9,
+    "error_rate": 0.1
+}
+# eval_variant_2_run
+{
+    "accuracy": 0.9,
+    "error_rate": 0.1
+}
+```
+
+Visualize the results:
+
+```bash
+pf run visualize --name "${base_run_name}0,${eval_run_name}0,${base_run_name}1,${eval_run_name}1,${base_run_name}2,${eval_run_name}2"
+```
+
+<details>
+<summary>For Windows CMD users, run commnad in toggle</summary>
+
+```shell
+pf run visualize --name "%base_run_name%0,%eval_run_name%0,base_run_name%1,%eval_run_name%1,base_run_name%2,%eval_run_name%2"
+```
+
+</details>
+
+Click the HTML link, to get the experiment results. Click on column in the **Output** table will allow you to view the snapshot of each line.
+
+The snapshot of chat flow:
+![Visualization chat flow](media/visualization_chat_flow.png)
+
+The snapshot of evaluation flow:
+![Visualization eval flow](media/visualization_eval_flow.png)
+
+Excellent! Now you can compare their performances and token costs, and choose the prompt that best suits your needs. We can see that variant_1 and variant_2 have the same accuracy, but variant_1 has a lower token cost (only 2 few shots rather than the 6 in in variant_2). So variant_1 is the best choice for the quality and cost balance.
+
+### Conclusion
+
+Great! Now you can compare their performances and token costs to choose the prompt that best suits your needs. Upon comparison, we can observe that variant_1 and variant_2 have the similar accuracy. However, variant_1 stands out as the better choice due to its lower token cost (2 few-shots vs. 6 few-shots).
+
+<img src="media/realcase.png" alt="comparison result" width=65%/>
+
+It is evident that adding more CoT examples in the prompt does not necessarily improve the accuracy further. Instead, we should identify the optimal point where the number of shots maximizes accuracy while minimizing cost.
+
+Just in a few steps, we identified that variant_1 strikes the ideal balance between quality and cost! This is where the value of prompt tuning and evaluation using prompt flow becomes apparent. With prompt flow, you can easily test and evaluate different prompt variants, enabling you to facilitate high quality LLM-native apps to production.
@@ -0,0 +1,88 @@
+---
+category: rag
+weight: 40
+---
+
+# How to generate test data based on documents
+In this doc, you will learn how to generate test data based on your documents for RAG app.
+This approach helps relieve the efforts of manual data creation, which is typically time-consuming and labor-intensive, or the expensive option of purchasing pre-packaged test data.
+By leveraging the capabilities of llm, this guide streamlines the test data generation process, making it more efficient and cost-effective.
+
+
+## Prerequisites
+
+1. Prepare documents. The test data generator supports the following file types:
+    - .md - Markdown
+    - .docx - Microsoft Word
+    - .pdf - Portable Document Format
+    - .ipynb - Jupyter Notebook
+    - .txt - Text
+
+    **Limitations:**
+
+    - The test data generator may not function effectively for non-Latin characters, such as Chinese, in certain document types. The limitation is caused by dependent text loader capabilities, such as `pypdf`.
+    - The test data generator may not generate meaningful questions if the document is not well-organized or contains massive code snippets/links, such as API introduction documents or reference documents.
+
+2. Prepare local environment. Go to [example_gen_test_data](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) folder and install required packages.
+
+    ```
+    pip install -r requirements.txt
+    ```
+
+    For specific document file types, you may need to install extra packages:
+      - .docx - `pip install docx2txt`
+      - .pdf - `pip install pypdf`
+      - .ipynb - `pip install nbconvert`
+      > !Note: the example uses llama index `SimpleDirectoryReader` to load documents. For the latest information of different file type required packages, please check [here](https://docs.llamaindex.ai/en/stable/examples/data_connectors/simple_directory_reader.html).
+
+3. Install VSCode extension `Prompt flow`.
+
+4. Create your AzureOpenAI or OpenAI connection by following [this doc](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/tune-prompts-with-variants.md).
+
+5. Prepare test data generation setting.
+    - Navigate to [example_gen_test_data](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) folder.
+    - Prepare `config.yml` by copying [`config.yml.example`](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/config.yml.example).
+    - Fill in configurations in the `config.yml` by following inline comment instructions. The config is made up of 3 sections:
+      - Common section: this section provides common values for all other sections. Required.
+      - Local section: this section is for local test data generation related configuration. Can skip if not run in local.
+      - Cloud section: this section is for cloud test data generation related configuration. Can skip if not run in cloud.
+
+      > !Note: Recommend to use `gpt-4` series models than the `gpt-3.5` for better performance.
+
+      > !Note: Recommend to use `gpt-4` model (Azure OpenAI `gpt-4` model with version `0613`) than `gpt-4-turbo` model (Azure OpenAI `gpt-4` model with version `1106`) for better performance. Due to inferior performance of `gpt-4-turbo` model, when you use it, sometimes you might need to open [example test data generation flow](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/flow.dag.yaml) in visual editor and set `response_format` input of nodes `validate_text_chunk`, `validate_question`, and `validate_suggested_answer` to `json`, in order to make sure the llm can generate valid json response.
+
+
+## Generate test data
+- Navigate to [example_gen_test_data](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) folder.
+
+- After configuration, run the following command to generate the test data set:
+  ```
+  python -m gen_test_data.run
+  ```
+
+- The generated test data will be a data jsonl file. See detailed log print in console "Saved ... valid test data to ..." to find it.
+
+If you expect to generate a large amount of test data beyond your local compute capability, you may try generating test data in cloud, please see this [guide](https://github.com/microsoft/promptflow/blob/main/docs/cloud/azureai/generate-test-data-cloud.md) for more detailed steps.
+
+## [*Optional*] Customize test data generation flow
+
+- Open the [example test data generation flow](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) in "Prompt flow" VSCode Extension. This flow is designed to generate a pair of question and suggested answer based on the given text chunk. The flow also includes validation prompts to ensure the quality of the generated test data.
+
+- Customize your test data generation logic refering to [tune-prompts-with-variants](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/tune-prompts-with-variants.md).
+
+  **Understanding the prompts**
+
+  The test data generation flow contains 5 prompts, classified into two categories based on their roles: generation prompts and validation prompts. Generation prompts are used to create questions, suggested answers, etc., while validation prompts are used to verify the validity of the text chunk, generated question or answer.
+  - Generation prompts
+    - [*generate question prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/generate_question_prompt.jinja2): frame a question based on the given text chunk.
+    - [*generate suggested answer prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/generate_suggested_answer_prompt.jinja2): generate suggested answer for the question based on the given text chunk.
+  - Validation prompts
+    - [*score text chunk prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/score_text_chunk_prompt.jinja2): score 0-10 to validate if the given text chunk is worthy of framing a question. If the score is lower than `score_threshold` (a node input that is adjustable), validation fails.
+    - [*validate question prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/validate_question_prompt.jinja2): validate if the generated question is good.
+    - [*validate suggested answer*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/validate_suggested_answer_prompt.jinja2): validate if the generated suggested answer is good.
+
+    If the validation fails, would lead to empty string `question`/`suggested_answer` which are removed from final output test data set.
+
+- Fill in node inputs including `connection`, `model` or `deployment_name`, `response_format`, `score_threshold` or other parameters. Click run button to test the flow in VSCode Extension by referring to [Test flow with VS Code Extension](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/develop-a-dag-flow/init-and-test-a-flow.md#visual-editor-on-the-vs-code-for-prompt-flow).
+
+Once the customized flow has been verified, you can proceed to batch generate test data by following the steps outlined in ["Prerequisites"](#prerequisites) and ["Generate test data"](#generate-test-data).
@@ -0,0 +1,11 @@
+name: test_data_gen_conda_env
+channels:
+  - defaults
+dependencies:
+  - python=3.10.12
+  - pip=23.2.1
+  - pip:
+    - mldesigner==0.1.0b18
+    - llama_index==0.9.48
+    - docx2txt==0.8
+    - promptflow>=1.7.0
@@ -0,0 +1,57 @@
+# Common section: this section provides common values for all other sections. Required.
+# Configure 'document_folder', 'document_chunk_size' and 'document_chunk_overlap' if you require document splitting.
+documents_folder: <your-document-folder-path>
+document_chunk_size: 512 # The token chunk size for each chunk.
+document_chunk_overlap: 100 # The token overlap of each chunk when splitting.
+
+# However, if you wish to bypass the document split process, simply provide the 'document_nodes_file', which is a JSONL file.
+# When both 'documents_folder' and 'document_nodes_file' are configured, will use 'document_nodes_file' and ignore 'documents_folder'.
+# For cloud mode, both local files and data assets can be used.
+# document_nodes_file: <your-node-file-path>
+
+# Test data gen flow configs
+# You can utilize our provided example test data generation flow directly. Alternatively, you can create your own flow and set up corresponding node inputs override.
+# The example flow folder path is <promptflow github repo>\examples\gen_test_data\example_flow
+flow_folder: <your-test-data-gen-flow-folder-path>
+node_inputs_override: # Override some node inputs, if not fill in 'node_inputs_override', will use the values in flow.dag.yaml
+  validate_text_chunk: # node name in flow.dag.yaml
+    connection: <your-connection-name> # connection name of node 'validate_text_chunk'
+    # Use 'deployment_name' for Azure OpenAI connection, 'model' for OpenAI
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  generate_question:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  validate_question:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  generate_suggested_answer:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  validate_suggested_answer:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+
+
+# Local section: this section is for local test data generation related configuration. Can skip if not run in local.
+output_folder: <your-output-folder-path>
+flow_batch_run_size: 4 # Higher values may speed up flow runs but risk hitting OpenAI's rate limit.
+
+
+# Cloud section: this section is for cloud test data generation related configuration. Can skip if not run in cloud.
+subscription_id: <your-sub-id>
+resource_group: <your-resource-group>
+workspace_name: <your-workspace-name>
+aml_cluster: <your-compute-name>
+
+# Parallel run step configs
+prs_instance_count: 2
+prs_mini_batch_size: 1
+prs_max_concurrency_per_instance: 4
+prs_max_retry_count: 3
+prs_run_invocation_time: 800
+prs_allowed_failed_count: -1
@@ -0,0 +1,129 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+environment:
+  python_requirements_txt: requirements.txt
+inputs:
+  text_chunk:
+    type: string
+    is_chat_input: false
+    default: Prompt flow is a suite of development tools designed to streamline the
+      end-to-end development cycle of LLM-based AI applications, from ideation,
+      prototyping, testing, evaluation to production deployment and monitoring.
+      It makes prompt engineering much easier and enables you to build LLM apps
+      with production quality.
+outputs:
+  question:
+    type: string
+    reference: ${validate_question.output.question}
+  suggested_answer:
+    type: string
+    reference: ${validate_suggested_answer.output.suggested_answer}
+  debug_info:
+    type: string
+    reference: ${generate_debug_info.output}
+nodes:
+- name: score_text_chunk_prompt
+  type: prompt
+  source:
+    type: code
+    path: score_text_chunk_prompt.jinja2
+  inputs:
+    context: ${inputs.text_chunk}
+  use_variants: false
+- name: validate_question_prompt
+  type: prompt
+  source:
+    type: code
+    path: validate_question_prompt.jinja2
+  inputs:
+    question: ${generate_question.output}
+    context: ${inputs.text_chunk}
+  use_variants: false
+- name: generate_question_prompt
+  type: prompt
+  source:
+    type: code
+    path: generate_question_prompt.jinja2
+  inputs:
+    context: ${inputs.text_chunk}
+  use_variants: false
+- name: generate_suggested_answer_prompt
+  type: prompt
+  source:
+    type: code
+    path: generate_suggested_answer_prompt.jinja2
+  inputs:
+    context: ${inputs.text_chunk}
+    question: ${validate_question.output.question}
+  use_variants: false
+- name: generate_question
+  type: python
+  source:
+    type: code
+    path: generate_question.py
+  inputs:
+    connection: ""
+    context: ${validate_text_chunk.output.context}
+    temperature: 0.2
+    generate_question_prompt: ${generate_question_prompt.output}
+  use_variants: false
+- name: validate_question
+  type: python
+  source:
+    type: code
+    path: validate_question.py
+  inputs:
+    connection: ""
+    temperature: 0.2
+    generated_question: ${generate_question.output}
+    validate_question_prompt: ${validate_question_prompt.output}
+  use_variants: false
+- name: generate_suggested_answer
+  type: python
+  source:
+    type: code
+    path: generate_suggested_answer.py
+  inputs:
+    connection: ""
+    context: ${inputs.text_chunk}
+    generate_suggested_answer_prompt: ${generate_suggested_answer_prompt.output}
+    question: ${validate_question.output.question}
+    temperature: 0.2
+  use_variants: false
+- name: generate_debug_info
+  type: python
+  source:
+    type: code
+    path: generate_debug_info.py
+  inputs:
+    text_chunk: ${inputs.text_chunk}
+    validate_suggested_answer_output: ${validate_suggested_answer.output}
+    text_chunk_validation_res: ${validate_text_chunk.output.validation_res}
+    validate_question_output: ${validate_question.output}
+- name: validate_suggested_answer_prompt
+  type: prompt
+  source:
+    type: code
+    path: validate_suggested_answer_prompt.jinja2
+  inputs:
+    answer: ${generate_suggested_answer.output}
+- name: validate_suggested_answer
+  type: python
+  source:
+    type: code
+    path: validate_suggested_answer.py
+  inputs:
+    connection: ""
+    suggested_answer: ${generate_suggested_answer.output}
+    validate_suggested_answer_prompt: ${validate_suggested_answer_prompt.output}
+    temperature: 0.2
+- name: validate_text_chunk
+  type: python
+  source:
+    type: code
+    path: validate_text_chunk.py
+  inputs:
+    connection: ""
+    score_text_chunk_prompt: ${score_text_chunk_prompt.output}
+    context: ${inputs.text_chunk}
+    score_threshold: 4
+    temperature: 0.2
@@ -0,0 +1,47 @@
+from utils import ValidateObj, ValidationResult
+
+from promptflow.core import tool
+
+
+# The inputs section will change based on the arguments of the tool function, after you save the code
+# Adding type to arguments and return value will help the system show the types properly
+# Please update the function name/signature per need
+@tool
+def my_python_tool(
+    text_chunk: str,
+    text_chunk_validation_res: ValidationResult = None,
+    validate_question_output: dict = None,
+    validate_suggested_answer_output: dict = None,
+) -> dict:
+    question_validation_res = validate_question_output["validation_res"]
+
+    generated_suggested_answer = validate_suggested_answer_output["suggested_answer"]
+    suggested_answer_validation_res = validate_suggested_answer_output["validation_res"]
+
+    is_generation_success = generated_suggested_answer != ""
+    is_text_chunk_valid = text_chunk_validation_res["pass_validation"] if text_chunk_validation_res else None
+    is_seed_question_valid = question_validation_res["pass_validation"] if question_validation_res else None
+    is_suggested_answer_valid = (
+        suggested_answer_validation_res["pass_validation"] if suggested_answer_validation_res else None
+    )
+
+    failed_step = ""
+    if not is_generation_success:
+        if is_text_chunk_valid is False:
+            failed_step = ValidateObj.TEXT_CHUNK
+        elif is_seed_question_valid is False:
+            failed_step = ValidateObj.QUESTION
+        elif is_suggested_answer_valid is False:
+            failed_step = ValidateObj.SUGGESTED_ANSWER
+
+    return {
+        # TODO: support more question types like multi-context etc.
+        # "question_type": question_type,
+        "text_chunk": text_chunk,
+        "validation_summary": {"success": is_generation_success, "failed_step": failed_step},
+        "validation_details": {
+            ValidateObj.TEXT_CHUNK: text_chunk_validation_res,
+            ValidateObj.QUESTION: question_validation_res,
+            ValidateObj.SUGGESTED_ANSWER: suggested_answer_validation_res,
+        },
+    }
@@ -0,0 +1,39 @@
+from typing import Union
+
+from utils import llm_call
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def generate_question(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    generate_question_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    context: str = None,
+    temperature: float = 0.2,
+):
+    """
+    Generates a question based on the given context.
+
+    Returns:
+        str: The generated seed question.
+    """
+    # text chunk is not valid, just skip test data gen.
+    if not context:
+        return ""
+
+    seed_question = llm_call(connection, model, deployment_name, generate_question_prompt, temperature=temperature)
+    return seed_question
@@ -0,0 +1,17 @@
+# system:
+
+Your task is to formulate a question from given context satisfying the rules given below:
+
+1.The question should better be framed from the overall context, serving as a general question, rather than just framed from some details.
+2.The question should be specific and answerable from the given context.
+3.The question must be reasonable and must be understood and responded by humans.
+4.The question should not contain phrases like 'provided' or 'given' in the question.
+5.The question should be a question asked by the hypothetical user without any given context.
+6.The question should not contain any links.
+7.The question should not contain more than 20 words, use abbreviation wherever possible.
+
+# user:
+context:
+{{context}}
+
+question:
@@ -0,0 +1,44 @@
+from typing import Union
+
+from utils import llm_call
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def generate_suggested_answer(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    question: str,
+    context: str,
+    generate_suggested_answer_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    temperature: float = 0.2,
+):
+    """
+    Generates a suggested answer based on the given prompts and context information.
+
+    Returns:
+        str: The generated suggested answer.
+    """
+    if question and context:
+        return llm_call(
+            connection,
+            model,
+            deployment_name,
+            generate_suggested_answer_prompt,
+            temperature=temperature,
+        )
+    else:
+        return ""
@@ -0,0 +1,12 @@
+system:
+Provide the answer for the question using the information from the given context based on the following criteria:
+1. The answer is correct and complete.
+2. The answer is derived from the given context.
+3. The answer can totally answer the question.
+4. The answer should not use the words like "in the context". The answer should be enough to answer the question without the context.
+5. If the answer for the question cannot be generated from the given context, just return empty string.
+
+user:
+question:{{question}}
+context:{{context}}
+answer:
@@ -0,0 +1 @@
+promptflow[azure]>=1.7.0
@@ -0,0 +1,49 @@
+# system:
+
+Given a text chunk from a document as context, perform the following tasks：
+
+1. Exclude any references, acknowledgments, personal information, code snippets, or other non-essential elements from the original context.
+
+2. Evaluate the cleaned context against specific criteria for content quality and depth.
+
+3. Assign a numerical score between 0 and 10 based on the following criteria:
+    - Award a high score (closer to 10) if:
+        a) cleaned context delves into and explains concepts.
+        b) cleaned context contains substantial information that could lead to meaningful questions.
+
+    - Award a lower score (closer to 0) if:
+        a) cleaned context is very brief, containing fewer than five words.
+        b) cleaned context is not meaningful.
+
+
+4. Output a valid JSON containing the score and a reason. The reason must directly relate to the criteria outlined above, explaining the basis for the given score.
+
+Here are some examples:
+
+example 1:
+context:
+Albert Einstein (14 March 1879 - 18 April 1955) was a German-born theoretical physicist who is widely held to be one of the greatest and most influential scientists of all time.
+
+output:
+{
+  "score": "8.0",
+  "reason": "The context provides substantial information that could lead to meaningful questions, hence the high score."
+}
+
+example 2:
+context:
+Next step\n- Open the provided examples.
+
+output:
+{
+  "score": "0.0",
+  "reason": "The context lacks detailed information about the provided example and previous steps, resulting in a low score."
+}
+
+
+# user:
+
+context:
+{{context}}
+
+output:
@@ -0,0 +1,170 @@
+import json
+import re
+from collections import namedtuple
+
+from numpy.random import default_rng
+
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.tools.aoai import chat as aoai_chat
+from promptflow.tools.openai import chat as openai_chat
+
+
+class QuestionType:
+    SIMPLE = "simple"
+    # MULTI_CONTEXT = "multi_context"
+
+
+class ValidateObj:
+    QUESTION = "validate_question"
+    TEXT_CHUNK = "validate_text_chunk"
+    SUGGESTED_ANSWER = "validate_suggested_answer"
+
+
+class ResponseFormat:
+    TEXT = "text"
+    JSON = "json_object"
+
+
+class ErrorMsg:
+    INVALID_JSON_FORMAT = "Invalid json format. Response: {0}"
+    INVALID_TEXT_CHUNK = "Skipping generating seed question due to invalid text chunk: {0}"
+    INVALID_QUESTION = "Invalid seed question: {0}"
+    INVALID_ANSWER = "Invalid answer: {0}"
+
+
+ValidationResult = namedtuple("ValidationResult", ["pass_validation", "reason"])
+ScoreResult = namedtuple("ScoreResult", ["score", "reason", "pass_validation"])
+
+
+def llm_call(
+    connection, model, deployment_name, prompt, response_format=ResponseFormat.TEXT, temperature=1.0, max_tokens=None
+):
+    response_format = "json_object" if response_format.lower() == "json" else response_format
+    # avoid unnecessary jinja2 template re-rendering and potential error.
+    prompt = f"{{% raw %}}{prompt}{{% endraw %}}"
+    if isinstance(connection, AzureOpenAIConnection):
+        return aoai_chat(
+            connection=connection,
+            prompt=prompt,
+            deployment_name=deployment_name,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            response_format={"type": response_format},
+        )
+    elif isinstance(connection, OpenAIConnection):
+        return openai_chat(
+            connection=connection,
+            prompt=prompt,
+            model=model,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            response_format={"type": response_format},
+        )
+
+
+def get_question_type(testset_distribution) -> str:
+    """
+    Decides question evolution type based on probability
+    """
+    rng = default_rng()
+    prob = rng.uniform(0, 1)
+    return next((key for key in testset_distribution.keys() if prob <= testset_distribution[key]), QuestionType.SIMPLE)
+
+
+def get_suggested_answer_validation_res(
+    connection,
+    model,
+    deployment_name,
+    prompt,
+    suggested_answer: str,
+    temperature: float,
+    max_tokens: int = None,
+    response_format: ResponseFormat = ResponseFormat.TEXT,
+):
+    rsp = llm_call(
+        connection,
+        model,
+        deployment_name,
+        prompt,
+        temperature=temperature,
+        max_tokens=max_tokens,
+        response_format=response_format,
+    )
+    return retrieve_verdict_and_print_reason(
+        rsp=rsp, validate_obj_name=ValidateObj.SUGGESTED_ANSWER, validate_obj=suggested_answer
+    )
+
+
+def get_question_validation_res(
+    connection,
+    model,
+    deployment_name,
+    prompt,
+    question: str,
+    response_format: ResponseFormat,
+    temperature: float,
+    max_tokens: int = None,
+):
+    rsp = llm_call(connection, model, deployment_name, prompt, response_format, temperature, max_tokens)
+    return retrieve_verdict_and_print_reason(rsp=rsp, validate_obj_name=ValidateObj.QUESTION, validate_obj=question)
+
+
+def get_text_chunk_score(
+    connection,
+    model,
+    deployment_name,
+    prompt,
+    response_format: ResponseFormat,
+    score_threshold: float,
+    temperature: float,
+    max_tokens: int = None,
+):
+    rsp = llm_call(connection, model, deployment_name, prompt, response_format, temperature, max_tokens)
+    data = _load_json_rsp(rsp)
+    score_float = 0
+    reason = ""
+
+    if data and isinstance(data, dict) and "score" in data and "reason" in data:
+        # Extract the verdict and reason
+        score = data["score"].lower()
+        reason = data["reason"]
+        print(f"Score {ValidateObj.TEXT_CHUNK}: {score}\nReason: {reason}")
+        try:
+            score_float = float(score)
+        except ValueError:
+            reason = ErrorMsg.INVALID_JSON_FORMAT.format(rsp)
+    else:
+        reason = ErrorMsg.INVALID_JSON_FORMAT.format(rsp)
+    pass_validation = score_float >= score_threshold
+
+    return ScoreResult(score_float, reason, pass_validation)
+
+
+def retrieve_verdict_and_print_reason(rsp: str, validate_obj_name: str, validate_obj: str) -> ValidationResult:
+    data = _load_json_rsp(rsp)
+
+    if data and isinstance(data, dict) and "verdict" in data and "reason" in data:
+        # Extract the verdict and reason
+        verdict = data["verdict"].lower()
+        reason = data["reason"]
+        print(f"Is valid {validate_obj_name}: {verdict}\nReason: {reason}")
+        if verdict == "yes":
+            return ValidationResult(True, reason)
+        elif verdict == "no":
+            return ValidationResult(False, reason)
+        else:
+            print(f"Unexpected llm response to validate {validate_obj_name}: {validate_obj}")
+
+    return ValidationResult(False, ErrorMsg.INVALID_JSON_FORMAT.format(rsp))
+
+
+def _load_json_rsp(rsp: str):
+    try:
+        # It is possible that even the response format is required as json, the response still contains ```json\n
+        rsp = re.sub(r"```json\n?|```", "", rsp)
+        data = json.loads(rsp)
+    except json.decoder.JSONDecodeError:
+        print(ErrorMsg.INVALID_JSON_FORMAT.format(rsp))
+        data = None
+
+    return data
@@ -0,0 +1,60 @@
+from typing import Union
+
+from utils import ErrorMsg, QuestionType, ResponseFormat, get_question_validation_res
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def validate_question(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    generated_question: str,
+    validate_question_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    response_format: str = ResponseFormat.TEXT,
+    temperature: float = 0.2,
+):
+    """
+    1. Validates the given seed question.
+    2. Generates a test question based on the given prompts and distribution ratios.
+
+    Returns:
+        dict: The generated test question and its type.
+    """
+    # text chunk is not valid, seed question not generated.
+    if not generated_question:
+        return {"question": "", "question_type": "", "validation_res": None}
+
+    validation_res = get_question_validation_res(
+        connection,
+        model,
+        deployment_name,
+        validate_question_prompt,
+        generated_question,
+        response_format,
+        temperature,
+    )
+    is_valid_seed_question = validation_res.pass_validation
+    question = ""
+    question_type = ""
+    failed_reason = ""
+    if not is_valid_seed_question:
+        failed_reason = ErrorMsg.INVALID_QUESTION.format(generated_question)
+        print(failed_reason)
+    else:
+        question = generated_question
+        question_type = QuestionType.SIMPLE
+
+    return {"question": question, "question_type": question_type, "validation_res": validation_res._asdict()}
@@ -0,0 +1,64 @@
+# system:
+
+Verdict a question based on following rules:
+
+1. If there are acronyms or terms in the question, then please check if they exist in the given context. If no, verdict no. If yes, check if other rules are satisfied.
+2. Determine if the given question can be clearly understood and give the reason.
+
+Output a valid json with reason and verdict.
+
+Here are some examples:
+question: What is the discovery about space?
+answer:
+{
+    "reason":"The question is too vague and does not specify which discovery about space it is referring to."
+    "verdict":"no"
+}
+
+question: What caused the Great Depression?
+answer:
+{
+    "reason":"The question is specific and refers to a well-known historical economic event, making it clear and answerable.",
+    "verdict":"yes"
+}
+
+question: What is the keyword that best describes the paper's focus in natural language understanding tasks?
+answer:
+{
+    "reason": "The question mentions a 'paper' in it without referring it's name which makes it unclear without it",
+    "verdict": "no"
+}
+
+question: Who wrote 'Romeo and Juliet'?
+answer:
+{
+    "reason": "The question is clear and refers to a specific work by name therefore it is clear",
+    "verdict": "yes"
+}
+
+question: What did the study mention?
+answer:
+{
+    "reason": "The question is vague and does not specify which study it is referring to",
+    "verdict": "no"
+}
+
+question: What is the focus of the REPLUG paper?
+answer:
+{
+    "reason": "The question refers to a specific work by it's name hence can be understood",
+    "verdict": "yes"
+}
+
+question: What is the purpose of the reward-driven stage in the training process?
+answer:
+{
+    "reason": "The question lacks specific context regarding the type of training process, making it potentially ambiguous and open to multiple interpretations.",
+    "verdict": "no"
+}
+
+# user:
+context: {{context}}
+
+question: {{question}}
+answer:
@@ -0,0 +1,55 @@
+from typing import Union
+
+from utils import ErrorMsg, get_suggested_answer_validation_res
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+@tool
+def validate_suggested_answer(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    suggested_answer: str,
+    validate_suggested_answer_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    temperature: float = 0.2,
+    response_format: str = "text",
+):
+    """
+    1. Validates the given suggested answer.
+
+    Returns:
+        dict: The generated suggested answer and its validation result.
+    """
+    if not suggested_answer:
+        return {"suggested_answer": "", "validation_res": None}
+
+    validation_res = get_suggested_answer_validation_res(
+        connection,
+        model,
+        deployment_name,
+        validate_suggested_answer_prompt,
+        suggested_answer,
+        temperature,
+        response_format=response_format,
+    )
+    is_valid_gt = validation_res.pass_validation
+    failed_reason = ""
+    if not is_valid_gt:
+        failed_reason = ErrorMsg.INVALID_ANSWER.format(suggested_answer)
+        print(failed_reason)
+        suggested_answer = ""
+
+    return {"suggested_answer": suggested_answer, "validation_res": validation_res._asdict()}
@@ -0,0 +1,43 @@
+# system:
+
+Given an answer, verdict if the provided answer is valid and provide the reason in valid json format.
+The answer is not valid if the answer suggests that the context does not provide information or indicates uncertainty (such as 'I don't know'), it is deemed invalid. For any other case, the answer is considered valid.
+
+# user:
+
+Output a json format with the reason and verdict.
+Here are some examples:
+answer:
+The steps to build and install your tool package for use in VS Code extension are not provided in the context.
+output:
+{
+    "reason":"The answer is invalid because it states that the context does not provide the necessary steps.",
+    "verdict":"no"
+}
+
+answer:
+The context does not provide specific information on what the possible provider values are in supported configs for a connection provider.
+output:
+{
+    "reason":"The answer is invalid as it indicates that the context lacks specific information.",
+    "verdict":"no"
+}
+
+answer:
+I don't know.
+output:
+{
+    "reason":"The answer is invalid because it conveys don't know.",
+    "verdict":"no"
+}
+
+answer:
+The two essential components of an activate config in a node flow are `activate.when` and `activate.is`.
+output:
+{
+    "reason":"The answer is valid because it is clear and true.",
+    "verdict":"yes"
+}
+
+answer:{{answer}}
+output:
@@ -0,0 +1,49 @@
+from typing import Union
+
+from utils import ErrorMsg, ResponseFormat, get_text_chunk_score
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def validate_text_chunk(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    score_text_chunk_prompt: str,
+    score_threshold: float,
+    deployment_name: str = "",
+    model: str = "",
+    context: str = None,
+    response_format: str = ResponseFormat.TEXT,
+    temperature: float = 0.2,
+):
+    """
+    Validates the given text chunk. If the validation fails, return an empty context and the validation result.
+
+    Returns:
+        dict: Text chunk context and its validation result.
+    """
+    text_chunk_score_res = get_text_chunk_score(
+        connection,
+        model,
+        deployment_name,
+        score_text_chunk_prompt,
+        response_format,
+        score_threshold,
+        temperature,
+    )
+    if not text_chunk_score_res.pass_validation:
+        print(ErrorMsg.INVALID_TEXT_CHUNK.format(context))
+        return {"context": "", "validation_res": text_chunk_score_res._asdict()}
+
+    return {"context": context, "validation_res": text_chunk_score_res._asdict()}
@@ -0,0 +1,249 @@
+import json
+import re
+import sys
+import time
+import typing as t
+from pathlib import Path
+
+from constants import DOCUMENT_NODE, NODES_FILE_NAME, SUPPORT_FILE_TYPE, TEXT_CHUNK
+
+from promptflow._utils.logger_utils import get_logger
+
+
+def split_document(chunk_size, chunk_overlap, documents_folder, document_node_output):
+    try:
+        from llama_index import SimpleDirectoryReader
+        from llama_index.node_parser import SentenceSplitter
+        from llama_index.readers.schema import Document as LlamaindexDocument
+        from llama_index.schema import BaseNode
+    except ImportError as e:
+        raise ImportError(
+            f"{str(e)}. It appears that `llama_index` may not be installed, or the installed version may be incorrect."
+            "Please check `requirements.txt` file and install all the dependencies."
+        )
+
+    logger = get_logger("doc.split")
+    logger.info("Step 1: Start to split documents to document nodes...")
+    # count the number of files in documents_folder, including subfolders.
+    all_files = [f for f in Path(documents_folder).rglob("*") if f.is_file()]
+    filtered_num_files = sum(1 for _ in all_files if _.suffix.lower() in SUPPORT_FILE_TYPE)
+    logger.info(
+        f"Found {len(all_files)} files in the documents folder '{documents_folder}'. "
+        f"After filtering out unsupported file types, {filtered_num_files} files remain."
+        f"Using chunk size: {chunk_size} to split."
+    )
+    # `SimpleDirectoryReader` by default chunk the documents based on heading tags and paragraphs, which may lead to small chunks.  # noqa: E501
+    reader = SimpleDirectoryReader(documents_folder, required_exts=SUPPORT_FILE_TYPE, recursive=True, encoding="utf-8")
+    # Disable the default suffixes to avoid splitting the documents into small chunks.
+    # TODO: find a better way to disable the default suffixes.
+    SimpleDirectoryReader.supported_suffix = []
+    chunks = reader.load_data()
+    # Convert documents into nodes
+    node_parser = SentenceSplitter.from_defaults(
+        chunk_size=chunk_size, chunk_overlap=chunk_overlap, include_metadata=True
+    )
+    chunks = t.cast(t.List[LlamaindexDocument], chunks)
+    document_nodes: t.List[BaseNode] = node_parser.get_nodes_from_documents(documents=chunks)
+    logger.info(f"Split the documents and created {len(document_nodes)} document nodes.")
+    document_nodes_output_path = document_node_output / Path(NODES_FILE_NAME)
+    with open(document_nodes_output_path, "wt") as text_file:
+        for doc in document_nodes:
+            print(json.dumps({TEXT_CHUNK: doc.text, DOCUMENT_NODE: doc.to_json()}), file=text_file)
+
+    logger.info(f"Saved document nodes to '{document_nodes_output_path}'.")
+    return str(Path(document_node_output) / NODES_FILE_NAME)
+
+
+def clean_data(test_data_set: list, test_data_output_path: str):
+    logger = get_logger("data.clean")
+    logger.info("Step 3: Start to clean invalid test data...")
+    logger.info(f"Collected {len(test_data_set)} test data after the batch run.")
+    cleaned_data = []
+
+    for test_data in test_data_set:
+        if test_data and all(
+            val and val != "(Failed)" for key, val in test_data.items() if key.lower() != "line_number"
+        ):
+            data_line = {"question": test_data["question"], "suggested_answer": test_data["suggested_answer"]}
+            cleaned_data.append(data_line)
+
+    jsonl_str = "\n".join(map(json.dumps, cleaned_data))
+    with open(test_data_output_path, "wt") as text_file:
+        print(f"{jsonl_str}", file=text_file)
+
+    # TODO: aggregate invalid data root cause and count, and log it.
+    # log debug info path.
+    logger.info(
+        f"Removed {len(test_data_set) - len(cleaned_data)} invalid test data. "
+        f"Saved {len(cleaned_data)} valid test data to '{test_data_output_path}'."
+    )
+
+
+def count_non_blank_lines(file_path):
+    with open(file_path, "r") as file:
+        lines = file.readlines()
+
+    non_blank_lines = len([line for line in lines if line.strip()])
+    return non_blank_lines
+
+
+def print_progress(log_file_path: str, process):
+    from tqdm import tqdm
+
+    logger = get_logger("data.gen")
+    finished_log_pattern = re.compile(r".*execution.bulk\s+INFO\s+Finished (\d+) / (\d+) lines\.")
+    progress_log_pattern = re.compile(
+        r".*execution.bulk\s+INFO.*\[Finished: (\d+)\] \[Processing: (\d+)\] \[Pending: (\d+)\]"
+    )
+    # wait for the log file to be created
+    start_time = time.time()
+    while not Path(log_file_path).is_file():
+        time.sleep(1)
+        # if the log file is not created within 5 minutes, raise an error
+        if time.time() - start_time > 300:
+            raise Exception(f"Log file '{log_file_path}' is not created within 5 minutes.")
+
+    logger.info(f"Click '{log_file_path}' to see detailed batch run log. Showing the progress here...")
+    progress_bar = None
+    try:
+        last_data_time = time.time()
+        with open(log_file_path, "r") as f:
+            while True:
+                status = process.poll()
+                # status is None if not finished, 0 if finished successfully, and non-zero if failed
+                if status:
+                    stdout, _ = process.communicate()
+                    raise Exception(f"Batch run failed due to {stdout.decode('utf-8')}")
+
+                line = f.readline().strip()
+                if line:
+                    last_data_time = time.time()  # Update the time when the last data was received
+                    progress_match = progress_log_pattern.match(line)
+                    finished_match = finished_log_pattern.match(line)
+                    if not progress_match and not finished_match:
+                        continue
+
+                    if progress_match:
+                        finished, processing, pending = map(int, progress_match.groups())
+                        total = finished + processing + pending
+                        if progress_bar is None:
+                            # Set mininterval=0 to refresh the progress bar when it calls progress_bar.update
+                            # after initialization.
+                            progress_bar = tqdm(total=total, desc="Processing", mininterval=0, file=sys.stdout)
+                        progress_bar.update(finished - progress_bar.n)
+
+                    if finished_match:
+                        finished, total = map(int, finished_match.groups())
+                        if progress_bar is None:
+                            progress_bar = tqdm(total=total, desc="Processing", mininterval=0, file=sys.stdout)
+                        progress_bar.update(finished - progress_bar.n)
+
+                        if finished == total:
+                            progress_bar.close()
+                            logger.info("Batch run is completed.")
+
+                            break
+                elif time.time() - last_data_time > 300:
+                    logger.info(
+                        "No new log line received for 5 minutes. Stop reading. "
+                        f"See the log file '{log_file_path}' for more details."
+                    )
+                    break
+                else:
+                    time.sleep(1)  # wait for 1 second if no new line is available
+    except Exception as e:
+        raise Exception(f"Error occurred while printing batch run progress: {e}.")
+    finally:
+        if progress_bar:
+            progress_bar.close()
+
+
+def convert_to_abs_path(file_path: str) -> str:
+    if not file_path:
+        return file_path
+
+    path = Path(file_path)
+    if path.is_absolute():
+        return str(path)
+    elif path.exists():
+        abs = str(path.resolve())
+        return abs
+    else:
+        return file_path
+
+
+def local_path_exists(path):
+    return Path(path).exists()
+
+
+def non_padding_path(path):
+    return not (path.startswith("<") and path.endswith(">"))
+
+
+def _retrieve_file_names_from_document_nodes_file(document_nodes_file_path) -> t.List[str]:
+    text_info = {}
+    with open(document_nodes_file_path, "r") as file:
+        for line in file:
+            # Should skip empty new lines, otherwise, json.loads would throw error.
+            if not line.strip():
+                continue
+            line_json = json.loads(line)
+            text_chunk = line_json[TEXT_CHUNK]
+            document_node = json.loads(line_json["document_node"])
+            file_path = document_node["metadata"]["file_path"]
+            text_info[text_chunk] = file_path
+    return text_info
+
+
+def _count_lines(file_path) -> int:
+    with open(file_path, "r") as f:
+        return sum(1 for line in f if line.strip())
+
+
+def summarize_batch_run_res(gen_details_file_path, document_nodes_file_path, output_file_path):
+    success_count = 0
+    validate_failed_count = 0
+    validate_failed_steps = {}
+    validate_failed_distribution = {}
+
+    nodes_file_lines_count = _count_lines(document_nodes_file_path)
+    document_nodes_info = _retrieve_file_names_from_document_nodes_file(document_nodes_file_path)
+
+    with open(gen_details_file_path, "r") as details_f:
+        for details_line in details_f:
+            # Should skip empty new lines, otherwise, json.loads would throw error.
+            if not details_line.strip():
+                continue
+            data = json.loads(details_line)
+            if data["debug_info"] == "(Failed)":
+                continue
+
+            if data["debug_info"]["validation_summary"]["success"]:
+                success_count += 1
+            else:
+                validate_failed_count += 1
+                failed_step = data["debug_info"]["validation_summary"]["failed_step"]
+
+                if failed_step in validate_failed_steps:
+                    validate_failed_steps[failed_step] += 1
+                else:
+                    validate_failed_steps[failed_step] = 1
+                    validate_failed_distribution[failed_step] = {}
+
+                document_name = document_nodes_info[data["debug_info"]["text_chunk"]]
+                if document_name in validate_failed_distribution[failed_step]:
+                    validate_failed_distribution[failed_step][document_name] += 1
+                else:
+                    validate_failed_distribution[failed_step][document_name] = 1
+
+    data = {
+        "total_count": nodes_file_lines_count,
+        "success_count": success_count,
+        "run_failed_count": nodes_file_lines_count - success_count - validate_failed_count,
+        "validate_failed_count": validate_failed_count,
+        "validate_failed_steps": validate_failed_steps,
+        "validate_failed_distribution": validate_failed_distribution,
+    }
+
+    with open(output_file_path, "w") as file:
+        json.dump(data, file, indent=4)
@@ -0,0 +1,90 @@
+import json
+from pathlib import Path
+
+from common import clean_data, split_document, summarize_batch_run_res
+from constants import NODES_FILE_NAME, PARALLEL_RUN_STEP_FILE_NAME, SUMMARY_FILE_NAME, TEST_DATA_FILE_NAME
+from mldesigner import Input, Output, command_component
+
+conda_file = Path(__file__).parent.parent / "conda.yml"
+env_image = "mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04"
+
+
+@command_component(
+    name="split_document_component",
+    display_name="split documents",
+    description="Split documents into document nodes.",
+    environment=dict(
+        conda_file=conda_file,
+        image=env_image,
+    ),
+)
+def split_document_component(
+    documents_folder: Input(type="uri_folder"),
+    chunk_size: int,
+    chunk_overlap: int,
+    document_node_output: Output(type="uri_folder"),
+) -> str:
+    """Split documents into document nodes.
+
+    Args:
+        documents_folder: The folder containing documents to be split.
+        chunk_size: The size of each chunk.
+        document_node_output: The output folder
+        chunk_overlap: The size of chunk overlap
+
+    Returns:
+        The folder containing the split documents.
+    """
+    return split_document(chunk_size, chunk_overlap, documents_folder, document_node_output)
+
+
+@command_component(
+    name="clean_data_component",
+    display_name="clean dataset",
+    description="Clean test data set to remove empty lines.",
+    environment=dict(
+        conda_file=conda_file,
+        image=env_image,
+    ),
+)
+def clean_data_component(
+    test_data_set_folder: Input(type="uri_folder"), test_data_output: Output(type="uri_folder")
+) -> str:
+    test_data_set_path = Path(test_data_set_folder) / PARALLEL_RUN_STEP_FILE_NAME
+
+    with open(test_data_set_path, "r") as f:
+        data = [json.loads(line) for line in f]
+
+    test_data_output_path = test_data_output / Path(TEST_DATA_FILE_NAME)
+    clean_data(data, test_data_output_path)
+
+    return str(test_data_output_path)
+
+
+@command_component(
+    name="summarize_generation_details_component",
+    display_name="summarize generation details",
+    description="Summarize generation details.",
+    environment=dict(
+        conda_file=conda_file,
+        image=env_image,
+    ),
+)
+def summarize_generation_details_component(
+    document_node_output: Input(type="uri_folder"),
+    test_data_set_folder: Input(type="uri_folder"),
+    summary_output: Output(type="uri_folder"),
+) -> str:
+    test_data_set_path = Path(test_data_set_folder) / PARALLEL_RUN_STEP_FILE_NAME
+    document_node_output_path = Path(document_node_output)
+
+    summary_output_path = summary_output / Path(SUMMARY_FILE_NAME)
+    if document_node_output_path.is_dir():
+        document_node_output_path = document_node_output_path / NODES_FILE_NAME
+    summarize_batch_run_res(
+        gen_details_file_path=test_data_set_path,
+        document_nodes_file_path=document_node_output_path,
+        output_file_path=summary_output_path,
+    )
+
+    return str(summary_output_path)
@@ -0,0 +1,8 @@
+DOCUMENT_NODE = "document_node"
+TEXT_CHUNK = "text_chunk"
+NODES_FILE_NAME = "document_nodes.jsonl"
+DETAILS_FILE_NAME = "test-data-gen-details.jsonl"
+PARALLEL_RUN_STEP_FILE_NAME = "parallel_run_step.jsonl"
+SUMMARY_FILE_NAME = "test-data-gen-summary.json"
+TEST_DATA_FILE_NAME = "test-data.jsonl"
+SUPPORT_FILE_TYPE = [".docx", ".pdf", ".ipynb", ".md", ".txt"]
@@ -0,0 +1,324 @@
+import argparse
+import json
+import os
+import time
+from datetime import datetime
+from pathlib import Path
+
+from promptflow._utils.logger_utils import get_logger
+from promptflow._utils.yaml_utils import load_yaml
+
+CONFIG_FILE = (Path(__file__).parents[1] / "config.yml").resolve()
+
+# in order to import from absolute path, which is required by mldesigner
+os.sys.path.insert(0, os.path.abspath(Path(__file__).parent))
+
+from common import (  # noqa: E402
+    clean_data,
+    convert_to_abs_path,
+    count_non_blank_lines,
+    local_path_exists,
+    non_padding_path,
+    print_progress,
+    split_document,
+    summarize_batch_run_res,
+)
+from constants import DETAILS_FILE_NAME, SUMMARY_FILE_NAME, TEST_DATA_FILE_NAME, TEXT_CHUNK  # noqa: E402
+
+logger = get_logger("data.gen")
+
+
+def batch_run_flow(flow_folder: str, flow_input_data: str, flow_batch_run_size: int, node_inputs_override: dict):
+    logger.info(f"Step 2: Start to batch run '{flow_folder}'...")
+    import subprocess
+
+    run_name = f"test_data_gen_{datetime.now().strftime('%b-%d-%Y-%H-%M-%S')}"
+    # TODO: replace the separate process to submit batch run with batch run async method when it's available.
+    connections_str = ""
+    for node_name, node_val in node_inputs_override.items():
+        for k, v in node_val.items():
+            # need to double quote the value to make sure the value can be passed correctly
+            # when the value contains special characters like "<".
+            connections_str += f'{node_name}.{k}="{v}" '
+    connections_str = connections_str.rstrip()
+
+    cmd = (
+        f'pf run create --flow "{flow_folder}" --data "{flow_input_data}" --name {run_name} '
+        f"--environment-variables PF_WORKER_COUNT='{flow_batch_run_size}' PF_BATCH_METHOD='spawn' "
+        f"--column-mapping {TEXT_CHUNK}='${{data.text_chunk}}' --connections {connections_str} --debug"
+    )
+    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    logger.info(
+        f"Submit batch run successfully. process id {process.pid}. Please wait for the batch run to complete..."
+    )
+    return run_name, process
+
+
+def get_batch_run_output(output_path: Path):
+    logger.info(f"Reading batch run output from '{output_path}'.")
+    # wait for the output file to be created
+    start_time = time.time()
+    while not Path(output_path).is_file():
+        time.sleep(1)
+        # if the log file is not created within 5 minutes, raise an error
+        if time.time() - start_time > 300:
+            raise Exception(f"Output jsonl file '{output_path}' is not created within 5 minutes.")
+
+    output_lines = []
+    try:
+        with open(output_path, "r", encoding="utf-8") as f:
+            output_lines = list(map(json.loads, f))
+    except json.decoder.JSONDecodeError as e:
+        logger.warning(
+            f"Error reading the output file: {e}. It could be that the batch run output is empty. "
+            "Please check your flow and ensure it can run successfully."
+        )
+
+    return [
+        {"question": line["question"], "suggested_answer": line["suggested_answer"], "debug_info": line["debug_info"]}
+        for line in output_lines
+    ]
+
+
+def run_local(
+    documents_folder: str,
+    document_chunk_size: int,
+    document_chunk_overlap: int,
+    document_nodes_file: str,
+    flow_folder: str,
+    flow_batch_run_size: int,
+    output_folder: str,
+    should_skip_split: bool,
+    node_inputs_override: dict,
+):
+    text_chunks_path = document_nodes_file
+    output_folder = Path(output_folder) / datetime.now().strftime("%b-%d-%Y-%H-%M-%S")
+    if not Path(output_folder).is_dir():
+        Path(output_folder).mkdir(parents=True, exist_ok=True)
+
+    if not should_skip_split:
+        text_chunks_path = split_document(document_chunk_size, document_chunk_overlap, documents_folder, output_folder)
+
+    run_name, process = batch_run_flow(flow_folder, text_chunks_path, flow_batch_run_size, node_inputs_override)
+
+    run_folder_path = Path.home() / f".promptflow/.runs/{run_name}"
+    print_progress(run_folder_path / "logs.txt", process)
+    test_data_set = get_batch_run_output(run_folder_path / "outputs.jsonl")
+    # Store intermedian batch run output results
+    jsonl_str = "\n".join(map(json.dumps, test_data_set))
+    batch_run_details_file = Path(output_folder) / DETAILS_FILE_NAME
+    with open(batch_run_details_file, "wt") as text_file:
+        print(f"{jsonl_str}", file=text_file)
+
+    clean_data_output = Path(output_folder) / TEST_DATA_FILE_NAME
+    clean_data(test_data_set, clean_data_output)
+    logger.info(f"More debug info of test data generation can be found in '{batch_run_details_file}'.")
+
+    try:
+        summary_output_file = Path(output_folder) / SUMMARY_FILE_NAME
+        summarize_batch_run_res(
+            gen_details_file_path=batch_run_details_file,
+            document_nodes_file_path=text_chunks_path,
+            output_file_path=summary_output_file,
+        )
+        logger.info(f"Check test data generation summary in '{summary_output_file}'.")
+    except Exception as e:
+        logger.warning(f"Error to analyze batch run results: {e}")
+
+
+def run_cloud(
+    documents_folder: str,
+    document_chunk_size: int,
+    document_chunk_overlap: int,
+    document_nodes_file: str,
+    flow_folder: str,
+    subscription_id: str,
+    resource_group: str,
+    workspace_name: str,
+    aml_cluster: str,
+    prs_instance_count: int,
+    prs_mini_batch_size: int,
+    prs_max_concurrency_per_instance: int,
+    prs_max_retry_count: int,
+    prs_run_invocation_time: int,
+    prs_allowed_failed_count: int,
+    should_skip_split: bool,
+    node_inputs_override: dict,
+):
+    # lazy import azure dependencies
+    try:
+        from azure.ai.ml import Input as V2Input
+        from azure.ai.ml import MLClient, dsl, load_component
+        from azure.ai.ml.entities import RetrySettings
+        from azure.identity import DefaultAzureCredential
+    except ImportError:
+        raise ImportError(
+            "Please install azure dependencies using the following command: "
+            + "`pip install -r requirements_cloud.txt`"
+        )
+
+    @dsl.pipeline(
+        non_pipeline_inputs=[
+            "flow_yml_path",
+            "should_skip_doc_split",
+            "instance_count",
+            "mini_batch_size",
+            "max_concurrency_per_instance",
+            "max_retry_count",
+            "run_invocation_time",
+            "allowed_failed_count",
+        ]
+    )
+    def gen_test_data_pipeline(
+        data_input: V2Input,
+        flow_yml_path: str,
+        should_skip_doc_split: bool,
+        chunk_size=1024,
+        chunk_overlap=200,
+        instance_count=1,
+        mini_batch_size=1,
+        max_concurrency_per_instance=2,
+        max_retry_count=3,
+        run_invocation_time=600,
+        allowed_failed_count=-1,
+    ):
+        from components import clean_data_component, split_document_component, summarize_generation_details_component
+
+        data = (
+            data_input
+            if should_skip_doc_split
+            else split_document_component(
+                documents_folder=data_input, chunk_size=chunk_size, chunk_overlap=chunk_overlap
+            ).outputs.document_node_output
+        )
+        flow_node = load_component(flow_yml_path, params_override=[{"name": "gen_test_data_example_flow"}])(
+            data=data, text_chunk="${data.text_chunk}", connections=node_inputs_override
+        )
+        flow_node.mini_batch_size = mini_batch_size
+        flow_node.max_concurrency_per_instance = max_concurrency_per_instance
+        flow_node.set_resources(instance_count=instance_count)
+        flow_node.retry_settings = RetrySettings(max_retry_count=max_retry_count, timeout=run_invocation_time)
+        flow_node.mini_batch_error_threshold = allowed_failed_count
+        # Should use `mount` mode to ensure PRS complete merge output lines.
+        flow_node.outputs.flow_outputs.mode = "mount"
+        clean_data_component(test_data_set_folder=flow_node.outputs.flow_outputs).outputs.test_data_output
+        summarize_generation_details_component(
+            document_node_output=data, test_data_set_folder=flow_node.outputs.flow_outputs
+        ).outputs.summary_output
+
+    def get_ml_client(subscription_id: str, resource_group: str, workspace_name: str):
+        credential = DefaultAzureCredential(exclude_shared_token_cache_credential=True)
+        return MLClient(
+            credential=credential,
+            subscription_id=subscription_id,
+            resource_group_name=resource_group,
+            workspace_name=workspace_name,
+        )
+
+    ml_client = get_ml_client(subscription_id, resource_group, workspace_name)
+
+    if should_skip_split:
+        data_input = V2Input(path=document_nodes_file, type="uri_file")
+    else:
+        data_input = V2Input(path=documents_folder, type="uri_folder")
+
+    prs_configs = {
+        "instance_count": prs_instance_count,
+        "mini_batch_size": prs_mini_batch_size,
+        "max_concurrency_per_instance": prs_max_concurrency_per_instance,
+        "max_retry_count": prs_max_retry_count,
+        "run_invocation_time": prs_run_invocation_time,
+        "allowed_failed_count": prs_allowed_failed_count,
+    }
+
+    pipeline_with_flow = gen_test_data_pipeline(
+        data_input=data_input,
+        flow_yml_path=os.path.join(flow_folder, "flow.dag.yaml"),
+        should_skip_doc_split=should_skip_split,
+        chunk_size=document_chunk_size,
+        chunk_overlap=document_chunk_overlap,
+        **prs_configs,
+    )
+    pipeline_with_flow.compute = aml_cluster
+    studio_url = ml_client.jobs.create_or_update(pipeline_with_flow).studio_url
+    logger.info(f"Completed to submit pipeline. Experiment Link: {studio_url}")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--cloud", action="store_true", help="Run test data generation at cloud.")
+    args = parser.parse_args()
+
+    if Path(CONFIG_FILE).is_file():
+        with open(CONFIG_FILE, "r") as stream:
+            config = load_yaml(stream)
+    else:
+        raise Exception(
+            f"'{CONFIG_FILE}' does not exist. "
+            + "Please check if you are under the wrong directory or the file is missing."
+        )
+
+    should_skip_split_documents = False
+    document_nodes_file = convert_to_abs_path(config.get("document_nodes_file", None))
+    documents_folder = convert_to_abs_path(config.get("documents_folder", None))
+    flow_folder = convert_to_abs_path(config.get("flow_folder", None))
+    output_folder = convert_to_abs_path(config.get("output_folder", None))
+    validate_path_func = non_padding_path if args.cloud else local_path_exists
+    node_inputs_override = config.get("node_inputs_override", None)
+
+    if document_nodes_file and validate_path_func(document_nodes_file):
+        should_skip_split_documents = True
+    elif not documents_folder or not validate_path_func(documents_folder):
+        raise Exception(
+            "Neither 'documents_folder' nor 'document_nodes_file' is valid.\n"
+            f"documents_folder: '{documents_folder}'\ndocument_nodes_file: '{document_nodes_file}'"
+        )
+
+    if not validate_path_func(flow_folder):
+        raise Exception(f"Invalid flow folder: '{flow_folder}'")
+
+    if args.cloud:
+        logger.info("Start to generate test data at cloud...")
+    else:
+        logger.info("Start to generate test data at local...")
+
+    if should_skip_split_documents:
+        logger.info(
+            "Skip step 1 'Split documents to document nodes' as received document nodes from "
+            f"input file path '{document_nodes_file}'."
+        )
+        if Path(document_nodes_file).is_file():
+            logger.info(f"Collected {count_non_blank_lines(document_nodes_file)} document nodes.")
+
+    if args.cloud:
+        run_cloud(
+            documents_folder,
+            config.get("document_chunk_size", 512),
+            config.get("document_chunk_overlap", 100),
+            document_nodes_file,
+            flow_folder,
+            config["subscription_id"],
+            config["resource_group"],
+            config["workspace_name"],
+            config["aml_cluster"],
+            config.get("prs_instance_count", 2),
+            config.get("prs_mini_batch_size", 1),
+            config.get("prs_max_concurrency_per_instance", 4),
+            config.get("prs_max_retry_count", 3),
+            config.get("prs_run_invocation_time", 800),
+            config.get("prs_allowed_failed_count", -1),
+            should_skip_split_documents,
+            node_inputs_override,
+        )
+    else:
+        run_local(
+            documents_folder,
+            config.get("document_chunk_size", 512),
+            config.get("document_chunk_overlap", 100),
+            document_nodes_file,
+            flow_folder,
+            config.get("flow_batch_run_size", 16),
+            output_folder,
+            should_skip_split_documents,
+            node_inputs_override,
+        )
@@ -0,0 +1,3 @@
+promptflow>=1.7.0
+promptflow-tools
+llama_index==0.9.48
@@ -0,0 +1,4 @@
+promptflow>=1.7.0
+promptflow-tools
+azure-ai-ml==1.15.0
+mldesigner==0.1.0b18
@@ -0,0 +1,225 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Execute flow as a function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
+    "- A python environment\n",
+    "- Installed prompt flow SDK\n",
+    "\n",
+    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
+    "- Execute a flow as a function\n",
+    "- Execute a flow function with in-memory connection object override\n",
+    "- Execute a flow function with fields override\n",
+    "- Execute a flow function with streaming output\n",
+    "\n",
+    "**Motivations** - This guide will walk you through the main scenarios of executing flow as a function. You will learn how to consume flow as a function in different scenarios for more pythonnic usage.\n",
+    "\n",
+    "\n",
+    "**Note**: the flow context configs may affect each other in some cases. For example, using `connection` & `overrides` to override same node. \n",
+    "The behavior is undefined for those scenarios. Pleas avoid such usage."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example1: Load flow as a function with inputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import load_flow\n",
+    "\n",
+    "\n",
+    "flow_path = \"../../flows/standard/web-classification\"\n",
+    "sample_url = \"https://www.youtube.com/watch?v=o5ZQyXaAv1g\"\n",
+    "\n",
+    "f = load_flow(source=flow_path)\n",
+    "result = f(url=sample_url)\n",
+    "\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example2: Load flow as a function with in-memory connection override"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You will need to have a connection named \"new_ai_connection\" to run flow with new connection."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "parameters"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# provide parameters to create connection\n",
+    "\n",
+    "conn_name = \"new_ai_connection\"\n",
+    "api_key = \"<user-input>\"\n",
+    "api_base = \"<user-input>\"\n",
+    "api_version = \"<user-input>\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create needed connection\n",
+    "import promptflow\n",
+    "from promptflow.entities import AzureOpenAIConnection, OpenAIConnection\n",
+    "\n",
+    "\n",
+    "# Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
+    "connection = AzureOpenAIConnection(\n",
+    "    name=conn_name,\n",
+    "    api_key=api_key,\n",
+    "    api_base=api_base,\n",
+    "    api_type=\"azure\",\n",
+    "    api_version=api_version,\n",
+    ")\n",
+    "\n",
+    "# use this if you have an existing OpenAI account\n",
+    "# connection = OpenAIConnection(\n",
+    "#     name=conn_name,\n",
+    "#     api_key=api_key,\n",
+    "# )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "f = load_flow(\n",
+    "    source=flow_path,\n",
+    ")\n",
+    "# directly use connection created above\n",
+    "f.context.connections = {\"classify_with_llm\": {\"connection\": connection}}\n",
+    "\n",
+    "result = f(url=sample_url)\n",
+    "\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example 3: Local flow as a function with flow inputs override"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.entities import FlowContext\n",
+    "\n",
+    "f = load_flow(source=flow_path)\n",
+    "f.context = FlowContext(\n",
+    "    # node \"fetch_text_content_from_url\" will take inputs from the following command instead of from flow input\n",
+    "    overrides={\"nodes.fetch_text_content_from_url.inputs.url\": sample_url},\n",
+    ")\n",
+    "# the url=\"unknown\" will not take effect\n",
+    "result = f(url=\"unknown\")\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example 4: Load flow as a function with streaming output"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "f = load_flow(source=\"../../flows/chat/chat-basic\")\n",
+    "f.context.streaming = True\n",
+    "result = f(\n",
+    "    chat_history=[\n",
+    "        {\n",
+    "            \"inputs\": {\"chat_input\": \"Hi\"},\n",
+    "            \"outputs\": {\"chat_output\": \"Hello! How can I assist you today?\"},\n",
+    "        }\n",
+    "    ],\n",
+    "    question=\"How are you?\",\n",
+    ")\n",
+    "\n",
+    "\n",
+    "answer = \"\"\n",
+    "# the result will be a generator, iterate it to get the result\n",
+    "for r in result[\"answer\"]:\n",
+    "    answer += r\n",
+    "\n",
+    "print(answer)"
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "D-W-@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Flow",
+   "weight": 40
+  },
+  "description": "This guide will walk you through the main scenarios of executing flow as a function.",
+  "kernelspec": {
+   "display_name": "github_v2",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": "examples/flows/standard/web-classification, examples/flows/chat/chat-basic"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,449 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Run DAG flow in Azure"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
+    "- An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
+    "- An Azure ML workspace - [Configure workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)\n",
+    "- A python environment\n",
+    "- Installed prompt flow SDK\n",
+    "\n",
+    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
+    "- Connect to your Azure AI workspace from the Python SDK\n",
+    "- Create and develop a new promptflow run\n",
+    "- Evaluate the run with a evaluation flow\n",
+    "\n",
+    "**Motivations** - This guide will walk you through the main user journey of prompt flow code-first experience. You will learn how to create and develop your first prompt flow, test and evaluate it."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -r ../../requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 1. Connect to Azure Machine Learning Workspace\n",
+    "\n",
+    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
+    "\n",
+    "## 1.1 Import the required libraries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "# Import required libraries\n",
+    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
+    "\n",
+    "# azure version promptflow apis\n",
+    "from promptflow.azure import PFClient"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1.2 Configure credential\n",
+    "\n",
+    "We are using `DefaultAzureCredential` to get access to workspace. \n",
+    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
+    "\n",
+    "Reference for more available credentials if it does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    credential = DefaultAzureCredential()\n",
+    "    # Check if given credential can get token successfully.\n",
+    "    credential.get_token(\"https://management.azure.com/.default\")\n",
+    "except Exception as ex:\n",
+    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
+    "    credential = InteractiveBrowserCredential()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1.3 Get a handle to the workspace\n",
+    "\n",
+    "We use config file to connect to a workspace. The Azure ML workspace should be configured with computer cluster. [Check this notebook for configure a workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get a handle to workspace\n",
+    "pf = PFClient.from_config(credential=credential)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1.4 Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "In this notebook, we will use flow `web-classification` which uses connection `azure_open_ai_connection` inside, we need to set up the connection if we haven't added it before.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
+    "\n",
+    "Please go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
+    "Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Create a new run\n",
+    "\n",
+    "`web-classification` is a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set flow path and input data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load flow\n",
+    "flow = \"../../flows/standard/web-classification\"\n",
+    "data = \"../../flows/standard/web-classification/data.jsonl\""
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Submit run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create run\n",
+    "base_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"url\": \"${data.url}\",\n",
+    "    },\n",
+    ")\n",
+    "print(base_run)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.stream(base_run)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize(base_run)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow run result\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score.\n",
+    "\n",
+    "In this notebook, we use `eval-classification-accuracy` flow to evaluate. This is a flow illustrating how to evaluate the performance of a classification system. It involves comparing each prediction to the groundtruth and assigns a \"Correct\" or \"Incorrect\" grade, and aggregating the results to produce metrics such as accuracy, which reflects how good the system is at classifying the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_run = pf.run(\n",
+    "    flow=\"../../flows/evaluation/eval-classification-accuracy\",\n",
+    "    data=data,\n",
+    "    run=base_run,\n",
+    "    column_mapping={\n",
+    "        \"groundtruth\": \"${data.answer}\",\n",
+    "        \"prediction\": \"${run.outputs.category}\",\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.stream(eval_run)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create another run with different variant node\n",
+    "\n",
+    "In this example, `web-classification`'s node `summarize_text_content` has two variants: `variant_0` and `variant_1`. The difference between them is the inputs parameters:\n",
+    "\n",
+    "variant_0:\n",
+    "\n",
+    "    - inputs:\n",
+    "        - deployment_name: gpt-35-turbo\n",
+    "        - max_tokens: '128'\n",
+    "        - temperature: '0.2'\n",
+    "        - text: ${fetch_text_content_from_url.output}\n",
+    "\n",
+    "variant_1:\n",
+    "\n",
+    "    - inputs:\n",
+    "        - deployment_name: gpt-35-turbo\n",
+    "        - max_tokens: '256'\n",
+    "        - temperature: '0.3'\n",
+    "        - text: ${fetch_text_content_from_url.output}\n",
+    "\n",
+    "\n",
+    "You can check the whole flow definition at [flow.dag.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flows/standard/web-classification/flow.dag.yaml)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# use the variant1 of the summarize_text_content node.\n",
+    "variant_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    column_mapping={\n",
+    "        \"url\": \"${data.url}\",\n",
+    "    },\n",
+    "    variant=\"${summarize_text_content.variant_1}\",  # here we specify node \"summarize_text_content\" to use variant 1 version.\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.stream(variant_run)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(variant_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation against variant run"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
+    "\n",
+    "eval_run_variant = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    data=\"../../flows/standard/web-classification/data.jsonl\",  # path to the data file\n",
+    "    run=variant_run,  # use run as the variant\n",
+    "    column_mapping={\n",
+    "        # reference data\n",
+    "        \"groundtruth\": \"${data.answer}\",\n",
+    "        # reference the run's output\n",
+    "        \"prediction\": \"${run.outputs.category}\",\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.stream(eval_run_variant)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run_variant)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = pf.get_metrics(eval_run_variant)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([eval_run, eval_run_variant])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Next Steps\n",
+    "\n",
+    "Learn more on how to:\n",
+    "- run the flow as a component in a azureml pipeline: [flow in pipeline](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/run-flow-with-pipeline/pipeline.ipynb)."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "wangchao1230@github.com"
+   ],
+   "category": "azure",
+   "section": "Flow",
+   "weight": 20
+  },
+  "description": "A quickstart tutorial to run a flow in Azure AI and evaluate it.",
+  "kernelspec": {
+   "display_name": "promptflow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": "examples/requirements.txt, examples/flows/standard/web-classification, examples/flows/evaluation/eval-classification-accuracy"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,464 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting started with DAG flow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Prerequisite** - To make the most of this tutorial, you'll need:\n",
+    "- A local clone of the prompt flow repository\n",
+    "- A Python environment with Jupyter Notebook support (such as Jupyter Lab or the Python extension for Visual Studio Code)\n",
+    "- Know how to program with Python :)\n",
+    "\n",
+    "_A basic understanding of Machine Learning can be beneficial, but it's not mandatory._\n",
+    "\n",
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Run your first prompt flow sample\n",
+    "- Run your first evaluation\n",
+    "\n",
+    "\n",
+    "The sample used in this tutorial is the [web-classification](https://github.com/microsoft/promptflow/tree/main/examples/flows/standard/web-classification) flow, which categorizes URLs into several predefined classes. Classification is a traditional machine learning task, and this sample illustrates how to perform classification using GPT and prompts."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -r ../../requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "In this notebook, we will use flow `web-classification` which uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from promptflow.client import PFClient\n",
+    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
+    "\n",
+    "# client can help manage your runs and connections.\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    conn_name = \"open_ai_connection\"\n",
+    "    conn = pf.connections.get(name=conn_name)\n",
+    "    print(\"using existing connection\")\n",
+    "except:\n",
+    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
+    "    connection = AzureOpenAIConnection(\n",
+    "        name=conn_name,\n",
+    "        api_key=\"<test_key>\",\n",
+    "        api_base=\"<test_base>\",\n",
+    "        api_type=\"azure\",\n",
+    "        api_version=\"<test_version>\",\n",
+    "    )\n",
+    "\n",
+    "    # use this if you have an existing OpenAI account\n",
+    "    # connection = OpenAIConnection(\n",
+    "    #     name=conn_name,\n",
+    "    #     api_key=\"<user-input>\",\n",
+    "    # )\n",
+    "\n",
+    "    conn = pf.connections.create_or_update(connection)\n",
+    "    print(\"successfully created connection\")\n",
+    "\n",
+    "print(conn)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Run web-classification flow\n",
+    "\n",
+    "`web-classification` is a flow demonstrating multi-class classification with LLM. Given an url, it will classify the url into one web category with just a few shots, simple summarization and classification prompts."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set flow path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow = \"../../flows/standard/web-classification\"  # path to the flow directory"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Quick test"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test flow\n",
+    "flow_inputs = {\n",
+    "    \"url\": \"https://play.google.com/store/apps/details?id=com.twitter.android\",\n",
+    "}\n",
+    "flow_result = pf.test(flow=flow, inputs=flow_inputs)\n",
+    "print(f\"Flow result: {flow_result}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Test single node in the flow\n",
+    "node_name = \"fetch_text_content_from_url\"\n",
+    "node_inputs = {\n",
+    "    \"url\": \"https://play.google.com/store/apps/details?id=com.twitter.android\"\n",
+    "}\n",
+    "flow_result = pf.test(flow=flow, inputs=node_inputs, node=node_name)\n",
+    "print(f\"Node result: {flow_result}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Flow as a function\n",
+    "\n",
+    "We have also implemented a syntex sugar where you can consume a flow like a python function, with ability to override connections, inputs and other runtime configs.\n",
+    "Reference [here](./flow-as-function.ipynb) for more details."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import load_flow\n",
+    "\n",
+    "flow_func = load_flow(flow)\n",
+    "flow_result = flow_func(**flow_inputs)\n",
+    "print(f\"Flow function result: {flow_result}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Batch run with a data file (with multiple lines of test data)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data = \"../../flows/standard/web-classification/data.jsonl\"  # path to the data file\n",
+    "\n",
+    "# create run with default variant\n",
+    "base_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    stream=True,\n",
+    "    column_mapping={\n",
+    "        \"url\": \"${data.url}\",\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Evaluate your flow\n",
+    "Then you can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score.\n",
+    "\n",
+    "In this notebook, we use `classification-accuracy-eval` flow to evaluate. This is a flow illustrating how to evaluate the performance of a classification system. It involves comparing each prediction to the groundtruth and assigns a \"Correct\" or \"Incorrect\" grade, and aggregating the results to produce metrics such as accuracy, which reflects how good the system is at classifying the data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the previous batch run\n",
+    "The **base_run** is the batch run we completed in step 2 above, for web-classification flow with \"data.jsonl\" as input."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
+    "\n",
+    "eval_run = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    data=\"../../flows/standard/web-classification/data.jsonl\",  # path to the data file\n",
+    "    run=base_run,  # specify base_run as the run you want to evaluate\n",
+    "    column_mapping={\n",
+    "        \"groundtruth\": \"${data.answer}\",\n",
+    "        \"prediction\": \"${run.outputs.category}\",\n",
+    "    },  # map the url field from the data to the url input of the flow\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = pf.get_metrics(eval_run)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([base_run, eval_run])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By now you've successfully run your first prompt flow and even did evaluation on it. That's great!\n",
+    "\n",
+    "You can check out the [web-classification](https://github.com/microsoft/promptflow/tree/main/examples/flows/standard/web-classification) flow and the [classification-accuracy](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation/eval-classification-accuracy) flow for more details, and start building your own flow.\n",
+    "\n",
+    "Or you can move on for a more advanced topic: experiment with a variant."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Another batch run with a variant\n",
+    "\n",
+    "[Variant](https://microsoft.github.io/promptflow/concepts/concept-variants.html) in prompt flow is to allow you do experimentation with LLMs. You can set a variant of Prompt/LLM node pointing to different prompt or use different LLM parameters like temperature.\n",
+    "\n",
+    "In this example, `web-classification`'s node `summarize_text_content` has two variants: `variant_0` and `variant_1`. The difference between them is the inputs parameters:\n",
+    "\n",
+    "variant_0:\n",
+    "\n",
+    "    - inputs:\n",
+    "        - deployment_name: gpt-35-turbo\n",
+    "        - max_tokens: '128'\n",
+    "        - temperature: '0.2'\n",
+    "        - text: ${fetch_text_content_from_url.output}\n",
+    "\n",
+    "variant_1:\n",
+    "\n",
+    "    - inputs:\n",
+    "        - deployment_name: gpt-35-turbo\n",
+    "        - max_tokens: '256'\n",
+    "        - temperature: '0.3'\n",
+    "        - text: ${fetch_text_content_from_url.output}\n",
+    "\n",
+    "\n",
+    "You can check the whole flow definition at [flow.dag.yaml](https://github.com/microsoft/promptflow/blob/main/examples/flows/standard/web-classification/flow.dag.yaml)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# use the variant1 of the summarize_text_content node.\n",
+    "variant_run = pf.run(\n",
+    "    flow=flow,\n",
+    "    data=data,\n",
+    "    variant=\"${summarize_text_content.variant_1}\",  # here we specify node \"summarize_text_content\" to use variant 1 version.\n",
+    "    column_mapping={\n",
+    "        \"url\": \"${data.url}\",\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(variant_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Run evaluation on the variant run\n",
+    "So that later we can compare metrics and see which works better."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_flow = \"../../flows/evaluation/eval-classification-accuracy\"\n",
+    "\n",
+    "eval_run_variant = pf.run(\n",
+    "    flow=eval_flow,\n",
+    "    data=\"../../flows/standard/web-classification/data.jsonl\",  # path to the data file\n",
+    "    run=variant_run,  # use run as the variant\n",
+    "    column_mapping={\n",
+    "        \"groundtruth\": \"${data.answer}\",\n",
+    "        \"prediction\": \"${run.outputs.category}\",\n",
+    "    },  # map the url field from the data to the url input of the flow\n",
+    "    stream=True,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(eval_run_variant)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics = pf.get_metrics(eval_run_variant)\n",
+    "print(json.dumps(metrics, indent=4))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.visualize([eval_run, eval_run_variant])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Next Steps\n",
+    "\n",
+    "Learn more on:\n",
+    "- [Manage connections](https://github.com/microsoft/promptflow/blob/main/examples/connections/connection.ipynb): how to manage the endpoints/secrets information to access external services including LLMs.\n",
+    "- [Chat with PDF](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/e2e-development/chat-with-pdf.md): go through an end-to-end tutorial on how to develop a chat application with prompt flow.\n",
+    "- [Deploy http endpoint](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/flow-deploy): how to deploy the flow as a local http endpoint.\n",
+    "- [Prompt flow in Azure AI](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/get-started/quickstart-azure.ipynb): run and evaluate flow in Azure AI where you can collaborate with team better."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Flow",
+   "weight": 30
+  },
+  "description": "A quickstart tutorial to run a flow and evaluate it.",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": "examples/requirements.txt, examples/flows/standard/web-classification, examples/flows/evaluation/eval-classification-accuracy"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,15 @@
+name: my-env
+channels:
+  - conda-forge
+dependencies:
+  - python=3.9
+  - pip
+  - pip:
+      - mlflow
+      - azureml-core
+      - azure-ai-ml>=1.12.0
+      - azureml-dataset-runtime[pandas,fuse]
+      - azureml-telemetry
+      - mltable>=1.2.0
+      - pandas
+      - pillow
@@ -0,0 +1,21 @@
+# Copyright (c) Microsoft. All rights reserved.
+# Licensed under the MIT license.
+
+import os
+import pandas as pd
+import argparse
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--input_data_file", type=str)
+parser.add_argument("--output_data_folder", type=str)
+
+args, _ = parser.parse_known_args()
+
+input_df = pd.read_json(args.input_data_file, lines=True)
+
+# data preparation, e.g. data sampling, data cleaning, etc.
+processed_data = input_df.sample(n=20, replace=True, random_state=1)
+
+# export data into output folder
+output_file_path = os.path.join(args.output_data_folder, "processed_data.csv")
+processed_data.to_csv(output_file_path, index=False, header=True)
@@ -0,0 +1,26 @@
+$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
+
+type: command
+
+name: data_prep
+display_name: data preparation
+version: 0.0.1
+
+inputs:
+  input_data_file:
+    type: uri_file
+
+outputs:
+  output_data_folder:
+    type: uri_folder
+
+code: ./
+
+environment: 
+  image: mcr.microsoft.com/azureml/inference-base-2004:latest
+  conda_file: ./conda.yaml
+
+command: >-
+  python data-prep.py
+  --input_data_file ${{inputs.input_data_file}}
+  --output_data_folder ${{outputs.output_data_folder}}
@@ -0,0 +1,15 @@
+name: my-env
+channels:
+  - conda-forge
+dependencies:
+  - python=3.9
+  - pip
+  - pip:
+      - mlflow
+      - azureml-core
+      - azure-ai-ml>=1.12.0
+      - azureml-dataset-runtime[pandas,fuse]
+      - azureml-telemetry
+      - mltable>=1.2.0
+      - pandas
+      - pillow
@@ -0,0 +1,51 @@
+# Copyright (c) Microsoft. All rights reserved.
+# Licensed under the MIT license.
+
+import os
+import pandas as pd
+import argparse
+import glob
+import numpy as np
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--source_data", type=str)
+parser.add_argument("--pf_output_data", type=str)
+parser.add_argument("--pf_debug_data", type=str)
+parser.add_argument("--merged_data", type=str)
+
+args, _ = parser.parse_known_args()
+
+source_data_path = os.path.join(args.source_data, "processed_data.csv")
+pf_output_path = os.path.join(args.pf_output_data, "parallel_run_step.jsonl")
+merged_data_path = os.path.join(args.merged_data, "merged_data.jsonl")
+
+if args.pf_debug_data is not None:
+    pf_debug_files = glob.glob(os.path.join(args.pf_debug_data, "flow_artifacts/*.jsonl"))
+
+source_data_df = pd.read_csv(source_data_path)
+pf_output_df = pd.read_json(pf_output_path, lines=True)
+pf_output_df.sort_values(by="line_number", inplace=True, ignore_index=True)
+
+if len(source_data_df) != len(pf_output_df):
+    raise Exception("Index mismatch between data source and pf result")
+
+source_data_df.loc[:, "line_number"] = pf_output_df.loc[:, "line_number"]
+source_data_df.loc[:, "pred_category"] = pf_output_df.loc[:, "category"]
+source_data_df.loc[:, "pred_evidence"] = pf_output_df.loc[:, "evidence"]
+
+if pf_debug_files is not None and len(pf_debug_files) > 0:
+    debug_df = pd.concat([pd.read_json(file, lines=True) for file in pf_debug_files])
+    debug_df.sort_values(by="line_number", inplace=True, ignore_index=True)
+    for i in range(len(debug_df)):
+        source_data_df.loc[i, "prompt_tokens"] = debug_df.loc[i, "run_info"]["system_metrics"]["prompt_tokens"]
+        source_data_df.loc[i, "duration"] = debug_df.loc[i, "run_info"]["system_metrics"]["duration"]
+        source_data_df.loc[i, "completion_tokens"] = debug_df.loc[i, "run_info"]["system_metrics"]["completion_tokens"]
+        source_data_df.loc[i, "total_tokens"] = debug_df.loc[i, "run_info"]["system_metrics"]["total_tokens"]
+else:
+    source_data_df.loc[:, "prompt_tokens"] = np.nan
+    source_data_df.loc[:, "duration"] = np.nan
+    source_data_df.loc[:, "completion_tokens"] = np.nan
+    source_data_df.loc[:, "total_tokens"] = np.nan
+
+with open(merged_data_path, "w") as file:
+    file.write(source_data_df.to_json(orient="records", lines=True))
@@ -0,0 +1,34 @@
+$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
+
+type: command
+
+name: result_parser
+description: Aggregate pf source data with output and debug data.
+display_name: result parser.
+version: 0.0.1
+
+inputs:
+  source_data:
+    type: uri_folder
+  pf_output_data:
+    type: uri_folder
+  pf_debug_data:
+    optional: true
+    type: uri_folder
+
+outputs:
+  merged_data :
+    type: uri_folder
+
+code: ./
+
+environment: 
+  image: mcr.microsoft.com/azureml/inference-base-2004:latest
+  conda_file: ./conda.yaml
+
+command: >-
+  python result-parser.py
+  --source_data ${{inputs.source_data}}
+  --pf_output_data ${{inputs.pf_output_data}}
+  $[[--pf_debug_data ${{inputs.pf_debug_data}}]]
+  --merged_data ${{outputs.merged_data}}
@@ -0,0 +1,656 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Run flows in Azure ML pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Why use Azure machine learning(ML) pipelines to run your flows on the cloud?\n",
+    "In real-world scenarios, flows serve various purposes. For example, consider a flow designed to evaluate the relevance score for a communication session between humans and agents. Suppose you want to trigger this flow every night to assess today’s performance and avoid peak hours for LLM (Language Model) endpoints. In this common scenario, people often encounter the following needs:\n",
+    "- Handling Large Data Inputs: Running flows with thousands or millions of data inputs at once.\n",
+    "- Scalability and Efficiency: Requiring a scalable, efficient, and resilient platform to ensure success.\n",
+    "- Automations: Automatically triggering batch flows when upstream data is ready or at fixed intervals.\n",
+    "\n",
+    "__Azure ML pipelines__ address all these offline requirements effectively. With the integration of prompt flows and Azure ML pipeline, flow users could very easily achieve above goals and in this tutorial, you can learn:\n",
+    "- How to use python SDK to automatically convert your flow into a 'step' in Azure ML pipeline.\n",
+    "- How to feed your data into pipeline to trigger the batch flow runs.\n",
+    "- How to build other pipeline steps ahead or behind your prompt flow step. e.g. data preprocessing or result aggregation.\n",
+    "- How to setup a simple scheduler on my pipeline.\n",
+    "- How to deploy pipeline to an Azure ML batch endpoint. Then I can invoke it with new data when needed.\n",
+    "\n",
+    "Before you begin, consider the following prerequisites:\n",
+    "- Introduction to Azure ML Platform:\n",
+    "    - [Core site of Azure ML platform](https://learn.microsoft.com/en-us/azure/machine-learning/overview-what-is-azure-machine-learning?view=azureml-api-2).\n",
+    "    - Understand what [Azure ML pipelines](https://learn.microsoft.com/en-us/azure/machine-learning/concept-ml-pipelines?view=azureml-api-2) and [component](https://learn.microsoft.com/en-us/azure/machine-learning/concept-component?view=azureml-api-2) are.\n",
+    "- Azure cloud setup:\n",
+    "    - An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
+    "    - Create an Azure ML resource from Azure portal - [Create a Azure ML workspace](https://ms.portal.azure.com/#view/Microsoft_Azure_Marketplace/MarketplaceOffersBlade/searchQuery/machine%20learning)\n",
+    "    - Connect to your workspace then setup a basic computer cluster - [Configure workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)\n",
+    "- Local environment setup:\n",
+    "    - A python environment\n",
+    "    - Installed Azure Machine Learning Python SDK v2 - [install instructions](https://github.com/microsoft/promptflow/blob/main/examples/README.md) - check the getting started section and make sure version of 'azure-ai-ml' is higher than `1.12.0`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 1. Connect to Azure Machine Learning Workspace\n",
+    "\n",
+    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
+    "\n",
+    "## 1.1 Import the required libraries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# import required libraries\n",
+    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
+    "from azure.ai.ml import MLClient, load_component, Input, Output\n",
+    "from azure.ai.ml.constants import AssetTypes\n",
+    "from azure.ai.ml.dsl import pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1.2 Configure credential\n",
+    "\n",
+    "We are using `DefaultAzureCredential` to get access to workspace. \n",
+    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
+    "\n",
+    "Reference for more available credentials if it does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    credential = DefaultAzureCredential()\n",
+    "    # Check if given credential can get token successfully.\n",
+    "    credential.get_token(\"https://management.azure.com/.default\")\n",
+    "except Exception as ex:\n",
+    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
+    "    credential = InteractiveBrowserCredential()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1.3 Get a handle to the workspace\n",
+    "\n",
+    "We use 'config file' to connect to your workspace. Check [this notebook](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb) to get your config file from Azure ML workspace portal and paste it into this folder. Then if you pass the next code block, you've all set for the environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get a handle to workspace\n",
+    "ml_client = MLClient.from_config(credential=credential)\n",
+    "\n",
+    "# Retrieve an already attached Azure Machine Learning Compute.\n",
+    "cluster_name = \"cpu-cluster\"\n",
+    "print(ml_client.compute.get(cluster_name))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 2. Load flow as component\n",
+    "If you’ve already authored a flow using the Promptflow SDK or portal, you can locate the flow.dag.yaml file within the flow folder. This YAML specification is essential for loading your flow into an Azure ML component.\n",
+    "\n",
+    "> __REMARK:__ To use `load_component` function with flow.dag.yaml, please ensure the following:</br>\n",
+    "> - The `$schema` should be defined in target DAG yaml file. For example: `$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json`.</br>\n",
+    "> - Flow metadata must be generated and kept up-to-date by verifying the file '<my-flow-directory>/.promptflow/flow.tools.json'. If it doesn't exist, run the following command to generate and update it: `pf flow validate --source <my-flow-directory>`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "flow_component = load_component(\"../../flows/standard/web-classification/flow.dag.yaml\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When using the `load_component` function and the flow YAML specification, your flow is automatically transformed into a __[parallel component](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-parallel-job-in-pipeline?view=azureml-api-2&tabs=cliv2)__. This parallel component is designed for large-scale, offline, parallelized processing with efficiency and resilience. Here are some key features of this auto-converted component:\n",
+    "\n",
+    " - Pre-defined input and output ports:\n",
+    "\n",
+    "| port name | type | description |\n",
+    "| --------- | ---- | ----------- |\n",
+    "| data | uri_folder or uri_file | Accepts batch data input to your flow. You can use either the `uri_file` data type if your data is a single file or the `uri_folder` data type if your folder contains multiple files with the same schema. The default data type is jsonl, but you can customize this setting after declaring an instance of this flow component in your pipeline. Note that your data will be converted into a dataframe, so ensure that your CSV or TSV data includes a header line for proper mapping. |\n",
+    "| flow_outputs | uri_file | Generates a single output file named parallel_run_step.jsonl. Each line in this data file corresponds to a JSON object representing the flow returns, along with an additional column called line_number indicating its position from the original file. |\n",
+    "| debug_info | uri_folder | If you run your flow component in __debug mode__, this port provides debugging information for each run of your lines. E.g. intermediate outputs between steps, or LLM response and token usage. |\n",
+    "\n",
+    "![prompt flow base component image](./media/cloud/flow-in-pipeline/pf-base-component.png)\n",
+    "\n",
+    "> __NOTE:__</br>\n",
+    "> \n",
+    "> `flow_outputs` and `debug_info` outputs are required to be set `mount` mode as the output mode when you run pf component with multiple nodes. \n",
+    "\n",
+    " - Auto-generated parameters \n",
+    " \n",
+    "   These parameters represent all your flow inputs and connections associated with your flow steps. You can set default values in the flow/run definition, and they can be further customized during job submission. Use '[web-classification](https://github.com/microsoft/promptflow/blob/main/examples/flows/standard/web-classification/flow.dag.yaml)' sample flow for example, this flow has only one input named 'url' and 2 LLM steps 'summarize_text_content' and 'classify_with_llm'. The input parameters of this flow component are:\n",
+    " \n",
+    "   ![prompt flow base component image](./media/cloud/flow-in-pipeline/pf-component-parameters.png)\n",
+    "\n",
+    " - Auto-generated environment\n",
+    "\n",
+    "   The environment of the created component will be inherited by latest promptflow runtime image. User can include custom packages in the environment by specifying the `environment` attribute in `flow.dag.yaml`, along with a 'requirements.txt' file located under the same flow folder:\n",
+    "   ```yaml\n",
+    "      ...\n",
+    "      environment:\n",
+    "         python_requirements_txt: requirements.txt\n",
+    "   ```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 3. Build your pipeline\n",
+    "## 3.1 Declare input and output\n",
+    "To supply your pipeline with data, you need to declare an input using the `path`, `type`, and `mode` properties. Please note: `mount` is the default and suggested mode for your file or folder data input.\n",
+    "\n",
+    "Declaring the pipeline output is optional. However, if you require a customized output path in the cloud, you can follow the example below to set the path on the datastore. For more detailed information on valid path values, refer to this documentation - [manage pipeline inputs outputs](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-inputs-outputs-pipeline?view=azureml-api-2&tabs=cli)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data_input = Input(\n",
+    "    path=\"../../flows/standard/web-classification/data.jsonl\",\n",
+    "    type=AssetTypes.URI_FILE,\n",
+    "    mode=\"mount\",\n",
+    ")\n",
+    "\n",
+    "pipeline_output = Output(\n",
+    "    # Provide custom flow output file path if needed\n",
+    "    # path=\"azureml://datastores/<data_store_name>/paths/<path>\",\n",
+    "    type=AssetTypes.URI_FOLDER,\n",
+    "    # rw_mount is suggested for flow output\n",
+    "    mode=\"rw_mount\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.2.1 Run pipeline with single flow component\n",
+    "Since all Promptflow components are based on Azure ML parallel components, users can leverage specific __run settings__  to control the parallelization of flow runs. Below are some useful settings:\n",
+    "\n",
+    "| run settings | description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | allowed values | default value |\n",
+    "| ------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -------------- | ------------- |\n",
+    "| PF_INPUT_FORMAT | When utilizing `uri_folder` as the input data, this setting allows you to specify which file extensions should be treated as data files for initializing flow runs.                                                                                                                                                                                                                                                                                                                                                                                  | json, jsonl, csv, tsv | jsonl |\n",
+    "| compute | Defines which compute cluster from your Azure ML workspace will be used for this job.                                                                                                                                                                                                                                                                                                                                                                                                                                                                | | |\n",
+    "| instance_count | Define how many nodes from your compute cluster will be assigned to this job.                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | from 1 to node count of compute cluster. | 1 |\n",
+    "| max_concurrency_per_instance | Defines how many dedicated processors will run the flow in parallel on 1 node. When combined with the 'instance_count' setting, the total parallelization of your flow will be instance_count*max_concurrency_per_instance.                                                                                                                                                                                                                                                                                                                          | >1 | 1 |\n",
+    "| mini_batch_size | Define the number of lines for each mini-batches. A __mini-batch__ is the basic granularity for processing full data with parallelization. Each worker processor handles one mini-batch at a time, and all workers work in parallel across different nodes.                                                                                                                                                                                                                                                                                          | > 0 | 1 |\n",
+    "| max_retries | Defines the retry count if any mini-batch encounters an inner exception. </br></br> Remark: The retry granularity is based on mini-batches. For instance, with the previous setting, you can set 100 lines per mini-batch. When one line execution encounters a transient issue or an unhandled exception, these 100 lines will be retried together, even if the remaining 99 lines are successful. Additionally, LLM responses with status code 429 will be handled internally for flow runs in most cases and will not trigger mini-batch failure. | >= 0 | 3 |\n",
+    "| error_threshold | Defines how many failed lines are acceptable. If the count of failed lines exceeds this threshold, the job will be stopped and marked as failed. Set '-1' to disable this failure check.                                                                                                                                                                                                                                                                                                                                                             | -1 or >=0 | -1 |\n",
+    "| mini_batch_error_threshold | Defines the maximum number of failed mini-batches that can be tolerated after all retries. Set '-1' to disable this failure check.                                                                                                                                                                                                                                                                                                                                                                                                                   | -1 or >=0 | -1 |\n",
+    "| logging_level | Determines how parallel jobs save logs to disk. Setting to 'DEBUG' for the flow component allows the component to output intermediate flow logs into the 'debug_info' port.                                                                                                                                                                                                                                                                                                                                                                          | INFO, WARNING, DEBUG | INFO |\n",
+    "| timeout | Sets the timeout checker for each mini-batch execution in milliseconds. If a mini-batch runs longer than this threshold, it will be marked as failed and trigger the next retry. Consider setting a higher value based on your mini-batch size and total traffic throughput for your LLM endpoints.                                                                                                                                                                                                                                                  | > 0 | 600 |\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define the pipeline as a function\n",
+    "@pipeline()\n",
+    "def pipeline_func_with_flow(\n",
+    "    # Function inputs will be treated as pipeline input data or parameters.\n",
+    "    # Pipeline input could be linked to step inputs to pass data between steps.\n",
+    "    # Users are not required to define pipeline inputs.\n",
+    "    # With pipeline inputs, user can provide the different data or values when they trigger different pipeline runs.\n",
+    "    pipeline_input_data: Input,\n",
+    "    parallel_node_count: int = 1,\n",
+    "):\n",
+    "    # Declare pipeline step 'flow_node' by using flow component\n",
+    "    flow_node = flow_component(\n",
+    "        # Bind the pipeline intput data to the port 'data' of the flow component\n",
+    "        # If you don't have pipeline input, you can directly pass the 'data_input' object to the 'data'\n",
+    "        # But with this approach, you can't provide different data when you trigger different pipeline runs.\n",
+    "        # data=data_input,\n",
+    "        data=pipeline_input_data,\n",
+    "        # Declare which column of input data should be mapped to flow input\n",
+    "        # the value pattern follows ${data.<column_name_from_data_input>}\n",
+    "        url=\"${data.url}\",\n",
+    "        # Provide the connection values of the flow component\n",
+    "        # The value of connection and deployment_name should align with your workspace connection settings.\n",
+    "        connections={\n",
+    "            \"summarize_text_content\": {\n",
+    "                \"connection\": \"azure_open_ai_connection\",\n",
+    "                \"deployment_name\": \"gpt-35-turbo\",\n",
+    "            },\n",
+    "            \"classify_with_llm\": {\n",
+    "                \"connection\": \"azure_open_ai_connection\",\n",
+    "                \"deployment_name\": \"gpt-35-turbo\",\n",
+    "            },\n",
+    "        },\n",
+    "    )\n",
+    "\n",
+    "    # Provide run settings of your flow component\n",
+    "    # Only 'compute' is required and other setting will keep default value if not provided.\n",
+    "    flow_node.environment_variables = {\n",
+    "        \"PF_INPUT_FORMAT\": \"jsonl\",\n",
+    "    }\n",
+    "    flow_node.compute = \"cpu-cluster\"\n",
+    "    flow_node.resources = {\"instance_count\": parallel_node_count}\n",
+    "    flow_node.mini_batch_size = 5\n",
+    "    flow_node.max_concurrency_per_instance = 2\n",
+    "    flow_node.retry_settings = {\n",
+    "        \"max_retries\": 1,\n",
+    "        \"timeout\": 1200,\n",
+    "    }\n",
+    "    flow_node.error_threshold = -1\n",
+    "    flow_node.mini_batch_error_threshold = -1\n",
+    "    flow_node.logging_level = \"DEBUG\"\n",
+    "\n",
+    "    # Function return will be treated as pipeline output. This is not required.\n",
+    "    return {\"flow_result_folder\": flow_node.outputs.flow_outputs}\n",
+    "\n",
+    "\n",
+    "# create pipeline instance\n",
+    "pipeline_job_def = pipeline_func_with_flow(pipeline_input_data=data_input)\n",
+    "pipeline_job_def.outputs.flow_result_folder = pipeline_output"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Submit the pipeline job to your workspace then check the status of your job on UI through the link in the output."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Submit the pipeline job to your workspace\n",
+    "pipeline_job_run = ml_client.jobs.create_or_update(\n",
+    "    pipeline_job_def, experiment_name=\"Single_flow_component_pipeline_job\"\n",
+    ")\n",
+    "pipeline_job_run\n",
+    "\n",
+    "ml_client.jobs.stream(pipeline_job_run.name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> __NOTE:__</br>\n",
+    "> \n",
+    "> - The choice of `mini_batch_size` significantly affects the efficiency of the flow job. Since the lines within each mini-batch run sequentially, setting a higher value for this parameter increases the chunk size, which reduces parallelization. On the other hand, larger batch sizes also raise the cost of retries, as retries are based on the entire mini-batch. Conversely, opting for the lowest value (e.g., mini_batch_size=1) may introduce additional overhead, affecting efficiency across multiple mini-batches during orchestration or result summarization. So it is recommended to start with a value between 10 and 100 and fine-tune it later based on your specific requirements.</br>\n",
+    "> - The `max_concurrency_per_instance` setting can significantly enhance parallel efficiency within a single compute node. However, it also introduces several potential issues: 1) increase the risk of running out of memory, 2) LLM endpoint may experience throttling when too many requests arrive simultaneously. In general, it is advisable to set the max_concurrency_per_instance number equal to the core count of your compute to strike a balance between parallelism and resource constraints. </br>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3.2.2 Run complex pipeline with multiple component\n",
+    "In a typical pipeline, you’ll find multiple steps that encompass all your offline business requirements. If you’re aiming to construct a more intricate pipeline for production, explore the following resources:\n",
+    " - [how to create component with SDK v2](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-component-pipeline-python?view=azureml-api-2)\n",
+    " - Various component types:\n",
+    "    - [Command](https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-component-command?view=azureml-api-2)\n",
+    "    - [Spark](https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-component-spark?view=azureml-api-2)\n",
+    "    - [Pipeline](https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-component-pipeline?view=azureml-api-2)\n",
+    "\n",
+    "\n",
+    "Additionally, consider the following sample code that loads two extra command components from a repository to construct a single offline pipeline:\n",
+    " - __data_prep_component__ : This dummy data preprocessing step performs simple data sampling.\n",
+    " - __result_parser_component__: Combining source data, flow results, and debugging output, it generates a single file containing origin queries, LLM predictions, and LLM token usages."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load Azure ML components\n",
+    "data_prep_component = load_component(\"./components/data-prep/data-prep.yaml\")\n",
+    "result_parser_component = load_component(\n",
+    "    \"./components/result-parser/result-parser.yaml\"\n",
+    ")\n",
+    "\n",
+    "# load flow as component\n",
+    "flow_component = load_component(\"../../flows/standard/web-classification/flow.dag.yaml\")\n",
+    "\n",
+    "\n",
+    "@pipeline()\n",
+    "def pipeline_func_with_flow(pipeline_input_data):\n",
+    "    data_prep_node = data_prep_component(\n",
+    "        input_data_file=pipeline_input_data,\n",
+    "    )\n",
+    "    data_prep_node.compute = \"cpu-cluster\"\n",
+    "\n",
+    "    flow_node = flow_component(\n",
+    "        # Feed the output of data_prep_node to the flow component\n",
+    "        data=data_prep_node.outputs.output_data_folder,\n",
+    "        url=\"${data.url}\",\n",
+    "        connections={\n",
+    "            \"summarize_text_content\": {\n",
+    "                \"connection\": \"azure_open_ai_connection\",\n",
+    "                \"deployment_name\": \"gpt-35-turbo\",\n",
+    "            },\n",
+    "            \"classify_with_llm\": {\n",
+    "                \"connection\": \"azure_open_ai_connection\",\n",
+    "                \"deployment_name\": \"gpt-35-turbo\",\n",
+    "            },\n",
+    "        },\n",
+    "    )\n",
+    "\n",
+    "    flow_node.environment_variables = {\"PF_INPUT_FORMAT\": \"csv\"}\n",
+    "    flow_node.compute = \"cpu-cluster\"\n",
+    "    flow_node.mini_batch_size = 5\n",
+    "    flow_node.max_concurrency_per_instance = 2\n",
+    "    flow_node.resources = {\"instance_count\": 1}\n",
+    "    flow_node.logging_level = \"DEBUG\"\n",
+    "\n",
+    "    # set output mode to 'mount'\n",
+    "    # This is required for the flow component when the 'instance_count' is set higher than 1\n",
+    "    flow_node.outputs.flow_outputs.mode = \"mount\"\n",
+    "    flow_node.outputs.debug_info.mode = \"mount\"\n",
+    "\n",
+    "    result_parser_node = result_parser_component(\n",
+    "        source_data=data_prep_node.outputs.output_data_folder,\n",
+    "        pf_output_data=flow_node.outputs.flow_outputs,\n",
+    "        pf_debug_data=flow_node.outputs.debug_info,\n",
+    "    )\n",
+    "\n",
+    "    flow_node.retry_settings = {\n",
+    "        \"max_retries\": 1,\n",
+    "        \"timeout\": 6000,\n",
+    "    }\n",
+    "\n",
+    "    result_parser_node.compute = \"cpu-cluster\"\n",
+    "\n",
+    "    return {\"flow_result_folder\": result_parser_node.outputs.merged_data}\n",
+    "\n",
+    "\n",
+    "# create pipeline instance\n",
+    "pipeline_job_def = pipeline_func_with_flow(pipeline_input_data=data_input)\n",
+    "pipeline_job_def.outputs.flow_result_folder = pipeline_output"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Submit the pipeline job to your workspace then check the status of your job on UI through the link in the output."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# submit job to workspace\n",
+    "pipeline_job_run = ml_client.jobs.create_or_update(\n",
+    "    pipeline_job_def, experiment_name=\"Complex_flow_component_pipeline_job\"\n",
+    ")\n",
+    "pipeline_job_run\n",
+    "\n",
+    "ml_client.jobs.stream(pipeline_job_run.name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 4 Next steps\n",
+    "## 4.1 Next step - Setup scheduler for your pipeline\n",
+    "\n",
+    "Azure Machine Learning pipelines support native __scheduler__ to help users regularly run their pipeline jobs with predefined time triggers. Here’s a code example for setting up a scheduler on a newly created pipeline using the flow component.\n",
+    "\n",
+    "Let’s begin by declaring a scheduler with a customized recurrence pattern."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datetime import datetime\n",
+    "from azure.ai.ml.entities import JobSchedule, RecurrenceTrigger, RecurrencePattern\n",
+    "from azure.ai.ml.constants import TimeZone\n",
+    "\n",
+    "schedule_name = \"simple_sdk_create_schedule_recurrence\"\n",
+    "schedule_start_time = datetime.utcnow()\n",
+    "\n",
+    "recurrence_trigger = RecurrenceTrigger(\n",
+    "    frequency=\"day\",  # could accept \"hour\", \"minute\", \"day\", \"week\", \"month\"\n",
+    "    interval=1,\n",
+    "    schedule=RecurrencePattern(hours=10, minutes=[0, 1]),\n",
+    "    start_time=schedule_start_time,\n",
+    "    time_zone=TimeZone.UTC,\n",
+    ")\n",
+    "\n",
+    "job_schedule = JobSchedule(\n",
+    "    name=schedule_name,\n",
+    "    trigger=recurrence_trigger,\n",
+    "    # Declare the pipeline job to be scheduled. Here we uses the pipeline job created in previous example.\n",
+    "    create_job=pipeline_job_def,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To initiate the scheduler, follow this example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "job_schedule = ml_client.schedules.begin_create_or_update(\n",
+    "    schedule=job_schedule\n",
+    ").result()\n",
+    "print(job_schedule)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To review all your scheduled jobs, navigate to the __Job List__ page within the Azure Machine Learning workspace UI. Any job triggered by the scheduler will have a display name following this format: `<schedule_name>-<trigger_time>`. For instance, if you have a schedule named \"named-schedule\", a job triggered on January 1, 2021, at 06:00:00 UTC will have the display name \"named-schedule-20210101T060000Z\".\n",
+    "\n",
+    "To disable or shut down a running scheduler, follow this example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "job_schedule = ml_client.schedules.begin_disable(name=schedule_name).result()\n",
+    "job_schedule.is_enabled"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To explore further details about scheduling Azure Machine Learning pipeline jobs, visit this article on [how to schedule pipeline job](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-schedule-pipeline-job?view=azureml-api-2&tabs=python)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4.2 Next step - Deploy pipeline to an endpoint\n",
+    "Azure Machine Learning also offers __batch endpoints__, which enable you to deploy pipelines to an endpoint for efficient operationalization. If you require scheduling for your flow pipeline using an external orchestrator, such as Azure Data Factory or Microsoft Fabric, utilizing batch endpoints is the optimal recommendation for your flow pipeline.\n",
+    "\n",
+    "Let’s start by creating a new batch endpoint in your workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azure.ai.ml.entities import BatchEndpoint, PipelineComponentBatchDeployment\n",
+    "\n",
+    "# from azure.ai.ml.entities import ModelBatchDeployment, ModelBatchDeploymentSettings, Model, AmlCompute, Data, BatchRetrySettings, CodeConfiguration, Environment, Data\n",
+    "# from azure.ai.ml.constants import BatchDeploymentOutputAction\n",
+    "\n",
+    "\n",
+    "endpoint_name = \"hello-my-pipeline-endpoint\"\n",
+    "endpoint = BatchEndpoint(\n",
+    "    name=endpoint_name,\n",
+    "    description=\"A hello world endpoint for pipeline\",\n",
+    ")\n",
+    "\n",
+    "ml_client.batch_endpoints.begin_create_or_update(endpoint).result()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Each endpoint can support multiple deployments, each associated with distinct pipelines. In this context, we initiate a new deployment using our flow pipeline job, targeting the recently established endpoint."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "deployment = PipelineComponentBatchDeployment(\n",
+    "    name=\"my-pipeline-deployment\",\n",
+    "    description=\"A hello world deployment with a pipeline job.\",\n",
+    "    endpoint_name=endpoint.name,\n",
+    "    # Make sure 'pipeline_job_run' run successfully before deploying the endpoint\n",
+    "    job_definition=pipeline_job_run,\n",
+    "    settings={\"default_compute\": \"cpu-cluster\", \"continue_on_step_failure\": False},\n",
+    ")\n",
+    "\n",
+    "ml_client.batch_deployments.begin_create_or_update(deployment).result()\n",
+    "\n",
+    "# Refresh the default deployment to the latest one at our endpoint.\n",
+    "endpoint = ml_client.batch_endpoints.get(endpoint.name)\n",
+    "endpoint.defaults.deployment_name = deployment.name\n",
+    "ml_client.batch_endpoints.begin_create_or_update(endpoint).result()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Invoke the default deployment to target endpoint with proper data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "batch_endpoint_job = ml_client.batch_endpoints.invoke(\n",
+    "    endpoint_name=endpoint.name,\n",
+    "    inputs={\"pipeline_input_data\": data_input},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, verify the invocation on the workspace UI using the following link:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ml_client.jobs.get(batch_endpoint_job.name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To explore further details about Azure Machine Learning batch endpoint, visit this article on [how-to-use-batch-pipeline-deployments](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-batch-pipeline-deployments?view=azureml-api-2&tabs=python)"
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "elliotzh@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "azure",
+   "section": "Flow",
+   "weight": 40
+  },
+  "description": "Create pipeline using components to run a distributed job with tensorflow",
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  },
+  "resources": "examples/flows/standard/web-classification"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,338 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Flow run management in Azure"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
+    "- An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
+    "- An Azure ML workspace - [Configure workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)\n",
+    "- A python environment\n",
+    "- Installed prompt flow SDK\n",
+    "\n",
+    "\n",
+    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
+    "- create run with remote data\n",
+    "- create run which references another runs inputs\n",
+    "- manage runs via run.yaml\n",
+    "- create run with connection override\n",
+    "\n",
+    "**Motivations** - This guide will walk you through cloud run management abilities."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -r ../../requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Connect to Azure Machine Learning Workspace\n",
+    "\n",
+    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
+    "\n",
+    "## 1.1 Import the required libraries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
+    "from azure.ai.ml.entities import Data\n",
+    "from azure.core.exceptions import ResourceNotFoundError\n",
+    "\n",
+    "from promptflow.azure import PFClient\n",
+    "from promptflow.entities import Run"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1.2 Configure credential\n",
+    "\n",
+    "We are using `DefaultAzureCredential` to get access to workspace. \n",
+    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
+    "\n",
+    "Reference for more available credentials if it does not work for you: [configure credential example](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    credential = DefaultAzureCredential()\n",
+    "    # Check if given credential can get token successfully.\n",
+    "    credential.get_token(\"https://management.azure.com/.default\")\n",
+    "except Exception as ex:\n",
+    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
+    "    credential = InteractiveBrowserCredential()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1.3 Get a handle to the workspace\n",
+    "\n",
+    "We use config file to connect to a workspace. The Azure ML workspace should be configured with computer cluster. [Check this notebook for configure a workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get a handle to workspace\n",
+    "pf = PFClient.from_config(credential=credential)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1.4 Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "In this notebook, we will use flow `web-classification` which uses connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.\n",
+    "\n",
+    "Please go to [workspace portal](https://ml.azure.com/), click `Prompt flow` -> `Connections` -> `Create`, then follow the instruction to create your own connections. \n",
+    "Learn more on [connections](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-connections?view=azureml-api-2)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Create run with remote data\n",
+    "\n",
+    "Instead of relying on local files, there may be situations where you want to reuse data that's already available in your workspace when submitting a flow.\n",
+    "The following code cells show how to create flow run with remote data.\n",
+    "\n",
+    "### 2.1 Create or update remote data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data_name, data_version = \"flow_run_test_data\", \"1\"\n",
+    "\n",
+    "try:\n",
+    "    data = pf.ml_client.data.get(name=data_name, version=data_version)\n",
+    "except ResourceNotFoundError:\n",
+    "    data = Data(\n",
+    "        name=data_name,\n",
+    "        version=data_version,\n",
+    "        path=f\"../../flows/standard/web-classification/data.jsonl\",\n",
+    "        type=\"uri_file\",\n",
+    "    )\n",
+    "    data = pf.ml_client.data.create_or_update(data)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2.2 Prepare remote data id"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data_id = f\"azureml:{data.name}:{data.version}\"\n",
+    "print(data_id)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2.3 Create a flow run with remote data\n",
+    "\n",
+    "You can change instance type or idle time of the runtime or reset it to clean state. The following code cell shows how to do so."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create run\n",
+    "run = Run(\n",
+    "    # local flow file\n",
+    "    flow=\"../../flows/standard/web-classification\",\n",
+    "    # remote data\n",
+    "    data=data_id,\n",
+    "    # to customize runtime instance type and compute instance, you can provide them in resources\n",
+    "    # resources={\n",
+    "    #     \"instance_type\": \"STANDARD_DS11_V2\",\n",
+    "    #     \"compute\": \"my_compute_instance\"\n",
+    "    # }\n",
+    "    # to customize identity, you can provide them in identity\n",
+    "    # identity={\n",
+    "    #     \"type\": \"managed\",\n",
+    "    # }\n",
+    ")\n",
+    "\n",
+    "base_run = pf.runs.create_or_update(run=run)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2.4 Stream the flow run to make sure it runs successfully"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pf.runs.stream(base_run)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3 Create a flow run which uses an existing run's inputs\n",
+    "\n",
+    "When running a flow with an existing run, you can reference either it's inputs or outputs in column mapping.\n",
+    "The following code cell show how to reference a run's inputs in column mapping."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run = Run(\n",
+    "    # local flow file\n",
+    "    flow=\"../../flows/standard/web-classification\",\n",
+    "    # run name\n",
+    "    run=run,\n",
+    "    column_mapping={\n",
+    "        # reference another run's input data columns\n",
+    "        \"url\": \"${run.inputs.url}\",\n",
+    "        \"answer\": \"${run.inputs.answer}\",\n",
+    "        \"evidence\": \"${run.inputs.evidence}\",\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "base_run = pf.runs.create_or_update(\n",
+    "    run=run,\n",
+    ")\n",
+    "\n",
+    "pf.runs.stream(base_run)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Create a flow run with connection override\n",
+    "\n",
+    "Sometime you want to switch connection or deployment name inside a flow when submitting it.\n",
+    "Connection override provided an easy way to do it without changing original `flow.dag.yaml`.\n",
+    "In the following code cell, we will submit flow `web-classification` and override it's connection `open_ai_connection` to `azure_open_ai_connection`. \n",
+    "Please make sure the connection `azure_open_ai_connection` exists in your workspace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run = Run(\n",
+    "    # local flow file\n",
+    "    flow=\"../../flows/standard/web-classification\",\n",
+    "    data=\"../../flows/standard/web-classification/data.jsonl\",\n",
+    "    # override connection for node classify_with_llm & summarize_text_content\n",
+    "    connections={\n",
+    "        \"classify_with_llm\": {\"connection\": \"azure_open_ai_connection\"},\n",
+    "        \"summarize_text_content\": {\"connection\": \"azure_open_ai_connection\"},\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "base_run = pf.runs.create_or_update(\n",
+    "    run=run,\n",
+    ")\n",
+    "\n",
+    "pf.runs.stream(base_run)"
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "0mza987@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "azure",
+   "section": "Flow",
+   "weight": 50
+  },
+  "description": "Flow run management in Azure AI",
+  "kernelspec": {
+   "display_name": "github_v2",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": "examples/requirements.txt, examples/flows/standard/web-classification"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,252 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Flow run management"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "**Prerequisite** - To make the most of this tutorial, you'll need:\n",
+    "- A local clone of the prompt flow repository\n",
+    "- A Python environment with Jupyter Notebook support (such as Jupyter Lab or the Python extension for Visual Studio Code)\n",
+    "- Know how to program with Python :)\n",
+    "\n",
+    "_A basic understanding of Machine Learning can be beneficial, but it's not mandatory._\n",
+    "\n",
+    "\n",
+    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
+    "- manage runs via run.yaml\n",
+    "- create run which references another runs inputs\n",
+    "- create run with connection override\n",
+    "\n",
+    "\n",
+    "**Motivations** - This guide will walk you through local run management abilities."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -r ../../requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Create necessary connections\n",
+    "Connection helps securely store and manage secret keys or other sensitive credentials required for interacting with LLM and other external tools for example Azure Content Safety.\n",
+    "\n",
+    "This notebook's will use connection `open_ai_connection` inside, we need to set up the connection if we haven't added it before. After created, it's stored in local db and can be used in any flow.\n",
+    "\n",
+    "Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from promptflow.client import PFClient\n",
+    "from promptflow.connections import AzureOpenAIConnection, OpenAIConnection\n",
+    "\n",
+    "# client can help manage your runs and connections.\n",
+    "pf = PFClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "try:\n",
+    "    conn_name = \"open_ai_connection\"\n",
+    "    conn = pf.connections.get(name=conn_name)\n",
+    "    print(\"using existing connection\")\n",
+    "except:\n",
+    "    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.\n",
+    "    connection = AzureOpenAIConnection(\n",
+    "        name=conn_name,\n",
+    "        api_key=\"<test_key>\",\n",
+    "        api_base=\"<test_base>\",\n",
+    "        api_type=\"azure\",\n",
+    "        api_version=\"<test_version>\",\n",
+    "    )\n",
+    "\n",
+    "    # use this if you have an existing OpenAI account\n",
+    "    # connection = OpenAIConnection(\n",
+    "    #     name=conn_name,\n",
+    "    #     api_key=\"<user-input>\",\n",
+    "    # )\n",
+    "\n",
+    "    conn = pf.connections.create_or_update(connection)\n",
+    "    print(\"successfully created connection\")\n",
+    "\n",
+    "print(conn)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Create run with YAML file\n",
+    "\n",
+    "You can save configurations for a run in a YAML file to save the effort to repeately provide them in SDK/CLI.\n",
+    "In this step, we will create a sample run with a YAML file. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.client import load_run\n",
+    "\n",
+    "# load a run from YAML file\n",
+    "base_run = load_run(\n",
+    "    source=\"../../flows/standard/web-classification/run.yml\",\n",
+    "    # override the default params in the YAML file\n",
+    "    params_override=[{\"column_mapping\": {\"url\": \"${data.url}\"}}],\n",
+    ")\n",
+    "\n",
+    "# create the run\n",
+    "base_run = pf.runs.create_or_update(run=base_run)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "details = pf.get_details(base_run)\n",
+    "details.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3 Create a flow run which uses an existing run's inputs\n",
+    "\n",
+    "When running a flow with an existing run, you can reference either it's inputs or outputs in column mapping.\n",
+    "The following code cell show how to reference a run's inputs in column mapping."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.entities import Run\n",
+    "\n",
+    "# directly create the run object\n",
+    "run = Run(\n",
+    "    # local flow file\n",
+    "    flow=\"../../flows/standard/web-classification\",\n",
+    "    # run name\n",
+    "    run=base_run,\n",
+    "    column_mapping={\n",
+    "        # reference another run's inputs data column\n",
+    "        \"url\": \"${run.inputs.url}\",\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "base_run = pf.runs.create_or_update(\n",
+    "    run=run,\n",
+    ")\n",
+    "\n",
+    "pf.runs.stream(base_run)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Create a flow run with connection override\n",
+    "\n",
+    "Sometime you want to switch connection or deployment name inside a flow when submitting it.\n",
+    "Connection override provided an easy way to do it without changing original `flow.dag.yaml`.\n",
+    "In the following code cell, we will submit flow `web-classification` and override it's connection to `open_ai_connection`. \n",
+    "Please make sure the connection `open_ai_connection` exists in your local environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run = Run(\n",
+    "    # local flow file\n",
+    "    flow=\"../../flows/standard/web-classification\",\n",
+    "    data=\"../../flows/standard/web-classification/data.jsonl\",\n",
+    "    # override connection for node classify_with_llm & summarize_text_content\n",
+    "    # you can replace connection to your local connections\n",
+    "    connections={\n",
+    "        \"classify_with_llm\": {\"connection\": \"open_ai_connection\"},\n",
+    "        \"summarize_text_content\": {\"connection\": \"open_ai_connection\"},\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "base_run = pf.runs.create_or_update(\n",
+    "    run=run,\n",
+    ")\n",
+    "\n",
+    "pf.runs.stream(base_run)"
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "D-W-@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Flow",
+   "weight": 50
+  },
+  "description": "Flow run management",
+  "kernelspec": {
+   "display_name": "github_v2",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.13"
+  },
+  "resources": "examples/requirements.txt, examples/flows/standard/web-classification"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,3 @@
+CHAT_DEPLOYMENT_NAME=gpt-35-turbo
+AZURE_OPENAI_API_KEY=<your_AOAI_key>
+AZURE_OPENAI_ENDPOINT=<your_AOAI_endpoint>
@@ -0,0 +1,114 @@
+---
+resources: examples/tutorials/tracing/
+cloud: local
+category: tracing
+---
+
+## Tracing
+
+Prompt flow provides the tracing feature to capture and visualize the internal execution details for all flows.
+
+For `DAG flow`, user can track and visualize node level inputs/outputs of flow execution, it provides critical insights for developer to understand the internal details of execution.
+
+For `Flex flow` developers, who might use different frameworks (langchain, semantic kernel, OpenAI, kinds of agents) to create LLM based applications, prompt flow allow user to instrument their code in a [OpenTelemetry](https://opentelemetry.io/) compatible way, and visualize using UI provided by promptflow devkit.
+
+## Instrumenting user's code
+#### Enable trace for LLM calls
+Let's start with the simplest example, add single line code `start_trace()` to enable trace for LLM calls in your application.
+```python
+from openai import OpenAI
+from promptflow.tracing import start_trace
+
+# start_trace() will print a url for trace detail visualization
+start_trace()
+
+client = OpenAI()
+
+completion = client.chat.completions.create(
+  model="gpt-4",
+  messages=[
+    {"role": "system", "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."},
+    {"role": "user", "content": "Compose a poem that explains the concept of recursion in programming."}
+  ]
+)
+
+print(completion.choices[0].message)
+```
+
+With the trace url, user will see a trace list that corresponding to each LLM calls:
+![LLM-trace-list](../../../docs/media/trace/LLM-trace-list.png)
+
+Click on line record, the LLM detail will be displayed with chat window experience, together with other LLM call params:
+![LLM-trace-detail](../../../docs/media/trace/LLM-trace-detail.png)
+
+More examples of adding trace for [autogen](https://microsoft.github.io/autogen/) and [langchain](https://python.langchain.com/docs/get_started/introduction/):
+
+1. **[Add trace for Autogen](./autogen-groupchat/)**
+
+![autogen-trace-detail](../../../docs/media/trace/autogen-trace-detail.png)
+
+2. **[Add trace for Langchain](./langchain)**
+
+![langchain-trace-detail](../../../docs/media/trace/langchain-trace-detail.png)
+
+#### Trace for any function
+More common scenario is the application has complicated code structure, and developer would like to add trace on critical path that they would like to debug and monitor.
+
+See the **[math_to_code](./math_to_code.py)** example on how to use `@trace`.
+
+```python
+from promptflow.tracing import trace
+# trace your function
+@trace
+def code_gen(client: AzureOpenAI, question: str) -> str:
+    sys_prompt = (
+        "I want you to act as a Math expert specializing in Algebra, Geometry, and Calculus. "
+        "Given the question, develop python code to model the user's question. "
+        "Make sure only reply the executable code, no other words."
+    )
+    completion = client.chat.completions.create(
+        model=os.getenv("CHAT_DEPLOYMENT_NAME", "gpt-35-turbo"),
+        messages=[
+            {
+                "role": "system",
+                "content": sys_prompt,
+            },
+            {"role": "user", "content": question},
+        ],
+    )
+    raw_code = completion.choices[0].message.content
+    result = code_refine(raw_code)
+    return result
+```
+
+Execute below command will get an URL to display the trace records and trace details of each test.
+
+```bash
+python math_to_code.py
+```
+
+## Trace visualization in flow test and batch run
+### Flow test
+
+If your application is created with DAG flow, all flow test and batch run will be automatically enable trace function. Take the **[chat_with_pdf](../../flows/chat/chat-with-pdf/)** as example.
+
+Run `pf flow test --flow .`, each flow test will generate single line in the trace UI:
+![flow-trace-record](../../../docs/media/trace/flow-trace-records.png)
+
+Click a record, the trace details will be visualized as tree view.
+
+![flow-trace-detail](../../../docs/media/trace/flow-trace-detail.png)
+
+### Evaluate against batch data
+Keep using **[chat_with_pdf](../../flows/chat/chat-with-pdf/)** as example, to trigger a batch run, you can use below commands:
+
+```shell
+pf run create -f batch_run.yaml
+```
+Or
+```shell
+pf run create --flow . --data "./data/bert-paper-qna.jsonl" --column-mapping chat_history='${data.chat_history}' pdf_url='${data.pdf_url}' question='${data.question}'
+```
+Then you will get a run related trace URL, e.g. http://127.0.0.1:52008/v1.0/ui/traces?run=chat_with_pdf_variant_0_20240226_181222_219335
+
+![batch_run_record](../../../docs/media/trace/batch_run_record.png)
@@ -0,0 +1,2 @@
+OAI_CONFIG_LIST.json
+groupchat
@@ -0,0 +1,16 @@
+[
+    {
+        "model": "gpt-4",
+        "api_key": "<your_AOAI_key>",
+        "base_url": "<your_AOAI_endpoint>",
+        "api_type": "azure",
+        "api_version": "2023-06-01-preview"
+    },
+    {
+        "model": "gpt-35-turbo",
+        "api_key": "<your_AOAI_key>",        
+        "base_url": "<your_AOAI_endpoint>",
+        "api_type": "azure",
+        "api_version": "2023-06-01-preview"
+    }
+]
@@ -0,0 +1,6 @@
+# Tracing existing application using promptflow: Auto Generated Agent Group Chat
+
+AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation.
+Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).
+
+Check out this [notebook](./agentchat_groupchat.ipynb) for example.
@@ -0,0 +1,3 @@
+promptflow
+pyautogen>=0.2.9
+pydantic>=2.6.0
@@ -0,0 +1,230 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tracing with AutoGen"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation.\n",
+    "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Trace LLM (OpenAI) Calls and visualize the trace of your application.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "AutoGen requires `Python>=3.8`. To run this notebook example, please install required dependencies:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set your API endpoint\n",
+    "\n",
+    "You can create the config file named `OAI_CONFIG_LIST.json` from example file: `OAI_CONFIG_LIST.json.example`.\n",
+    "\n",
+    "Below code use the [`config_list_from_json`](https://microsoft.github.io/autogen/0.2/docs/reference/oai/openai_utils/#config_list_from_json) function loads a list of configurations from an environment variable or a json file. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import autogen\n",
+    "\n",
+    "# please ensure you have a json config file\n",
+    "env_or_file = \"OAI_CONFIG_LIST.json\"\n",
+    "\n",
+    "# filters the configs by models (you can filter by other keys as well). Only the gpt-4 models are kept in the list based on the filter condition.\n",
+    "\n",
+    "# gpt4\n",
+    "# config_list = autogen.config_list_from_json(\n",
+    "#     env_or_file,\n",
+    "#     filter_dict={\n",
+    "#         \"model\": [\"gpt-4\", \"gpt-4-0314\", \"gpt4\", \"gpt-4-32k\", \"gpt-4-32k-0314\", \"gpt-4-32k-v0314\"],\n",
+    "#     },\n",
+    "# )\n",
+    "\n",
+    "# gpt35\n",
+    "config_list = autogen.config_list_from_json(\n",
+    "    env_or_file,\n",
+    "    filter_dict={\n",
+    "        \"model\": {\n",
+    "            \"gpt-35-turbo\",\n",
+    "            \"gpt-3.5-turbo\",\n",
+    "            \"gpt-3.5-turbo-16k\",\n",
+    "            \"gpt-3.5-turbo-0301\",\n",
+    "            \"chatgpt-35-turbo-0301\",\n",
+    "            \"gpt-35-turbo-v0301\",\n",
+    "        },\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Construct agents"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"AUTOGEN_USE_DOCKER\"] = \"False\"\n",
+    "\n",
+    "llm_config = {\"config_list\": config_list, \"cache_seed\": 42}\n",
+    "user_proxy = autogen.UserProxyAgent(\n",
+    "    name=\"User_proxy\",\n",
+    "    system_message=\"A human admin.\",\n",
+    "    code_execution_config={\n",
+    "        \"last_n_messages\": 2,\n",
+    "        \"work_dir\": \"groupchat\",\n",
+    "        \"use_docker\": False,\n",
+    "    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.\n",
+    "    human_input_mode=\"TERMINATE\",\n",
+    ")\n",
+    "coder = autogen.AssistantAgent(\n",
+    "    name=\"Coder\",\n",
+    "    llm_config=llm_config,\n",
+    ")\n",
+    "pm = autogen.AssistantAgent(\n",
+    "    name=\"Product_manager\",\n",
+    "    system_message=\"Creative in software product ideas.\",\n",
+    "    llm_config=llm_config,\n",
+    ")\n",
+    "groupchat = autogen.GroupChat(agents=[user_proxy, coder, pm], messages=[], max_round=12)\n",
+    "manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Start chat with promptflow trace"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "# traces will be collected into below collection name\n",
+    "start_trace(collection=\"autogen-groupchat\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Open the url you get in start_trace output, when running below code, you will be able to see new traces in the UI. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from opentelemetry import trace\n",
+    "import json\n",
+    "\n",
+    "\n",
+    "tracer = trace.get_tracer(\"my_tracer\")\n",
+    "# Create a root span\n",
+    "with tracer.start_as_current_span(\"autogen\") as span:\n",
+    "    message = \"Find a latest paper about gpt-4 on arxiv and find its potential applications in software.\"\n",
+    "    user_proxy.initiate_chat(\n",
+    "        manager,\n",
+    "        message=message,\n",
+    "        clear_history=True,\n",
+    "    )\n",
+    "    span.set_attribute(\"custom\", \"custom attribute value\")\n",
+    "    # recommend to store inputs and outputs as events\n",
+    "    span.add_event(\n",
+    "        \"promptflow.function.inputs\", {\"payload\": json.dumps(dict(message=message))}\n",
+    "    )\n",
+    "    span.add_event(\n",
+    "        \"promptflow.function.output\", {\"payload\": json.dumps(user_proxy.last_message())}\n",
+    "    )\n",
+    "# type exit to terminate the chat"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully tracing LLM calls in your app using prompt flow.\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "zhengfeiwang@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Tracing",
+   "weight": 20
+  },
+  "description": "Tracing LLM calls in autogen group chat application",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  },
+  "resources": ""
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,51 @@
+import os
+
+from dotenv import load_dotenv
+from openai.version import VERSION as OPENAI_VERSION
+
+from promptflow.tracing import trace
+
+
+def get_client():
+    if OPENAI_VERSION.startswith("0."):
+        raise Exception(
+            "Please upgrade your OpenAI package to version >= 1.0.0 or using the command: pip install --upgrade openai."
+        )
+    api_key = os.environ.get("OPENAI_API_KEY", None)
+    if api_key:
+        from openai import OpenAI
+
+        return OpenAI()
+    else:
+        from openai import AzureOpenAI
+
+        return AzureOpenAI(api_version=os.environ.get("OPENAI_API_VERSION", "2023-07-01-preview"))
+
+
+@trace
+def my_llm_tool(prompt: str, deployment_name: str) -> str:
+    if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
+        # load environment variables from .env file
+        load_dotenv()
+
+    if "OPENAI_API_KEY" not in os.environ and "AZURE_OPENAI_API_KEY" not in os.environ:
+        raise Exception("Please specify environment variables: OPENAI_API_KEY or AZURE_OPENAI_API_KEY")
+    messages = [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": prompt},
+    ]
+    response = get_client().chat.completions.create(
+        messages=messages,
+        model=deployment_name,
+    )
+
+    # get first element because prompt is single.
+    return response.choices[0].message.content
+
+
+if __name__ == "__main__":
+    result = my_llm_tool(
+        prompt="Write a simple Hello, world! python program that displays the greeting message. Output code only.",
+        deployment_name="gpt-4o",
+    )
+    print(result)
@@ -0,0 +1,205 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tracing with Custom OpenTelemetry Collector"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In certain scenario you might want to user your own [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) and keep your dependency mimimal.\n",
+    "\n",
+    "In such case you can avoid the dependency of [promptflow-devkit](https://pypi.org/project/promptflow-devkit/) which provides the default collector from promptflow, and only depdent on [promptflow-tracing](https://pypi.org/project/promptflow-tracing), \n",
+    "\n",
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Trace LLM (OpenAI) Calls using Custom OpenTelemetry Collector.\n",
+    "\n",
+    "## 0. Install dependent packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 1. Set up an OpenTelemetry collector\n",
+    "\n",
+    "Implement a simple collector that print the traces to stdout."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import threading\n",
+    "from http.server import BaseHTTPRequestHandler, HTTPServer\n",
+    "\n",
+    "from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import (\n",
+    "    ExportTraceServiceRequest,\n",
+    ")\n",
+    "\n",
+    "\n",
+    "class OTLPCollector(BaseHTTPRequestHandler):\n",
+    "    def do_POST(self):\n",
+    "        content_length = int(self.headers[\"Content-Length\"])\n",
+    "        post_data = self.rfile.read(content_length)\n",
+    "\n",
+    "        traces_request = ExportTraceServiceRequest()\n",
+    "        traces_request.ParseFromString(post_data)\n",
+    "\n",
+    "        print(\"Received a POST request with data:\")\n",
+    "        print(traces_request)\n",
+    "\n",
+    "        self.send_response(200, \"Traces received\")\n",
+    "        self.end_headers()\n",
+    "        self.wfile.write(b\"Data received and printed to stdout.\\n\")\n",
+    "\n",
+    "\n",
+    "def run_server(port: int):\n",
+    "    server_address = (\"\", port)\n",
+    "    httpd = HTTPServer(server_address, OTLPCollector)\n",
+    "    httpd.serve_forever()\n",
+    "\n",
+    "\n",
+    "def start_server(port: int):\n",
+    "    server_thread = threading.Thread(target=run_server, args=(port,))\n",
+    "    server_thread.daemon = True\n",
+    "    server_thread.start()\n",
+    "    print(f\"Server started on port {port}. Access http://localhost:{port}/\")\n",
+    "    return server_thread"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# invoke the collector service, serving on OTLP port\n",
+    "start_server(port=4318)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 2. Trace your application with tracing\n",
+    "Assume we already have a Python function that calls OpenAI API\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llm import my_llm_tool\n",
+    "\n",
+    "deployment_name = \"gpt-35-turbo-16k\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Call `start_trace()`, and configure the OTLP exporter to above collector."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "start_trace()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from opentelemetry import trace\n",
+    "from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n",
+    "from opentelemetry.sdk.trace.export import BatchSpanProcessor\n",
+    "\n",
+    "tracer_provider = trace.get_tracer_provider()\n",
+    "otlp_span_exporter = OTLPSpanExporter()\n",
+    "tracer_provider.add_span_processor(BatchSpanProcessor(otlp_span_exporter))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Visualize traces in the stdout."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "result = my_llm_tool(\n",
+    "    prompt=\"Write a simple Hello, world! python program that displays the greeting message. Output code only.\",\n",
+    "    deployment_name=deployment_name,\n",
+    ")\n",
+    "result\n",
+    "# view the traces under this cell"
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "zhengfeiwang@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Tracing",
+   "weight": 40
+  },
+  "description": "A tutorial on how to levarage custom OTLP collector.",
+  "kernelspec": {
+   "display_name": "tracing-rel",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": ""
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,3 @@
+promptflow-tracing
+python-dotenv
+opentelemetry-exporter-otlp-proto-http
@@ -0,0 +1,5 @@
+promptflow
+langchain>=0.1.5
+langchain_community
+opentelemetry-instrumentation-langchain
+python-dotenv
@@ -0,0 +1,181 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tracing with LangChain apps"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The tracing capability provided by Prompt flow is built on top of [OpenTelemetry](https://opentelemetry.io/) that gives you complete observability over your LLM applications. \n",
+    "And there is already a rich set of OpenTelemetry [instrumentation packages](https://opentelemetry.io/ecosystem/registry/?language=python&component=instrumentation) available in OpenTelemetry Eco System. \n",
+    "\n",
+    "In this example we will demo how to use [opentelemetry-instrumentation-langchain](https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-langchain) package provided by [Traceloop](https://www.traceloop.com/) to instrument [LangChain](https://python.langchain.com/docs/tutorials/) apps.\n",
+    "\n",
+    "\n",
+    "**Learning Objectives** - Upon completing this tutorial, you should be able to:\n",
+    "\n",
+    "- Trace `LangChain` applications and visualize the trace of your application in prompt flow.\n",
+    "\n",
+    "## Requirements\n",
+    "\n",
+    "To run this notebook example, please install required dependencies:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Start tracing LangChain using promptflow\n",
+    "\n",
+    "Start trace using `promptflow.start_trace`, click the printed url to view the trace ui."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "start_trace()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By default, `opentelemetry-instrumentation-langchain` instrumentation logs prompts, completions, and embeddings to span attributes. This gives you a clear visibility into how your LLM application is working, and can make it easy to debug and evaluate the quality of the outputs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# enable langchain instrumentation\n",
+    "from opentelemetry.instrumentation.langchain import LangchainInstrumentor\n",
+    "\n",
+    "instrumentor = LangchainInstrumentor()\n",
+    "if not instrumentor.is_instrumented_by_opentelemetry:\n",
+    "    instrumentor.instrument()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run a simple LangChain\n",
+    "\n",
+    "Below is an example targeting an AzureOpenAI resource. Please configure you `API_KEY` using an `.env` file, see `../.env.example`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "from langchain.chat_models import AzureChatOpenAI\n",
+    "from langchain.prompts.chat import ChatPromptTemplate\n",
+    "from langchain.chains import LLMChain\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "if \"AZURE_OPENAI_API_KEY\" not in os.environ:\n",
+    "    # load environment variables from .env file\n",
+    "    load_dotenv()\n",
+    "\n",
+    "llm = AzureChatOpenAI(\n",
+    "    deployment_name=os.environ[\"CHAT_DEPLOYMENT_NAME\"],\n",
+    "    openai_api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
+    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
+    "    openai_api_type=\"azure\",\n",
+    "    openai_api_version=\"2023-07-01-preview\",\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "prompt = ChatPromptTemplate.from_messages(\n",
+    "    [\n",
+    "        (\"system\", \"You are world class technical documentation writer.\"),\n",
+    "        (\"user\", \"{input}\"),\n",
+    "    ]\n",
+    ")\n",
+    "\n",
+    "chain = LLMChain(llm=llm, prompt=prompt, output_key=\"metrics\")\n",
+    "chain({\"input\": \"What is ChatGPT?\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You should be able to see traces of the chain in promptflow UI now. Check the cell with `start_trace` on the trace UI url."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "By now you've successfully tracing LLM calls in your app using prompt flow.\n",
+    "\n",
+    "You can check out more examples:\n",
+    "- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run."
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "zhengfeiwang@github.com",
+    "wangchao1230@github.com"
+   ],
+   "category": "local",
+   "section": "Tracing",
+   "weight": 30
+  },
+  "description": "Tracing LLM calls in langchain application",
+  "kernelspec": {
+   "display_name": "prompt_flow",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.17"
+  },
+  "resources": ""
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,2 @@
+AZURE_OPENAI_API_KEY=<your_AOAI_key>
+AZURE_OPENAI_ENDPOINT=<your_AOAI_endpoint>
@@ -0,0 +1,3 @@
+promptflow
+openai>=1.0.0
+python-dotenv
@@ -0,0 +1,201 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tracing with LLM application"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Tracing is a powerful tool for understanding the behavior of your LLM application, prompt flow tracing capability supports instrumentation for such scenario.\n",
+    "\n",
+    "This notebook will demonstrate how to use prompt flow to instrument and understand your LLM application.\n",
+    "\n",
+    "**Learning Objective** - Upon completion of this notebook, you will be able to:\n",
+    "\n",
+    "- Trace LLM application and visualize with prompt flow."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Requirements\n",
+    "\n",
+    "To run this notebook example, please install required dependencies."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture --no-stderr\n",
+    "%pip install -r ./requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Please configure your API key using an `.env` file, we have provided an example `.env.example` for reference."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load api key and endpoint from .env to environ\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "load_dotenv()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create your LLM application\n",
+    "\n",
+    "This notebook example will build a LLM application with Azure OpenAI service."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import AzureOpenAI\n",
+    "\n",
+    "# in this notebook example, we will use model \"gpt-35-turbo-16k\"\n",
+    "deployment_name = \"gpt-35-turbo-16k\"\n",
+    "\n",
+    "client = AzureOpenAI(\n",
+    "    azure_deployment=deployment_name,\n",
+    "    api_version=\"2024-02-01\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# prepare one classic question for LLM\n",
+    "conversation = [\n",
+    "    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+    "    {\"role\": \"user\", \"content\": \"What is the meaning of life?\"},\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.chat.completions.create(\n",
+    "    messages=conversation,\n",
+    "    model=deployment_name,\n",
+    ")\n",
+    "print(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Start trace using `promptflow.tracing.start_trace` to leverage prompt flow tracing capability; this will print a link to trace UI, where you can visualize the trace."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from promptflow.tracing import start_trace\n",
+    "\n",
+    "# start a trace session, and print a url for user to check trace\n",
+    "start_trace(collection=\"trace-llm\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Run the LLM application again, and you should be able to see new trace logged in the trace UI, and it is clickable to see more details.\n",
+    "\n",
+    "![trace-detail](./media/trace-llm/trace-detail.png)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.chat.completions.create(\n",
+    "    messages=conversation,\n",
+    "    model=deployment_name,\n",
+    ")\n",
+    "print(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next Steps\n",
+    "\n",
+    "By now you have successfully tracing your LLM application with prompt flow.\n",
+    "\n",
+    "You can check out more examples:\n",
+    "\n",
+    "- [Trace LangChain](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/tracing/langchain/trace-langchain.ipynb): tracing `LangChain` and visualize leveraging prompt flow.\n",
+    "- [Trace AutoGen](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/tracing/autogen-groupchat/trace-autogen-groupchat.ipynb): tracing `AutoGen` and visualize leveraging prompt flow.\n",
+    "- [Trace your flow](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/flex-flow-quickstart.ipynb): using promptflow @trace to structurally tracing your app and do evaluation on it with batch run.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "build_doc": {
+   "author": [
+    "zhengfeiwang@github.com"
+   ],
+   "category": "local",
+   "section": "Tracing",
+   "weight": 10
+  },
+  "description": "Tracing LLM application",
+  "kernelspec": {
+   "display_name": "pf-dev",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  },
+  "resources": ""
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
@@ -0,0 +1,93 @@
+import ast
+import os
+
+from dotenv import load_dotenv
+from openai import AzureOpenAI
+
+from promptflow.tracing import start_trace, trace
+
+
+@trace
+def infinite_loop_check(code_snippet):
+    tree = ast.parse(code_snippet)
+    for node in ast.walk(tree):
+        if isinstance(node, ast.While):
+            if not node.orelse:
+                return True
+    return False
+
+
+@trace
+def syntax_error_check(code_snippet):
+    try:
+        ast.parse(code_snippet)
+    except SyntaxError:
+        return True
+    return False
+
+
+@trace
+def error_fix(code_snippet):
+    tree = ast.parse(code_snippet)
+    for node in ast.walk(tree):
+        if isinstance(node, ast.While):
+            if not node.orelse:
+                node.orelse = [ast.Pass()]
+    return ast.unparse(tree)
+
+
+@trace
+def code_refine(original_code: str) -> str:
+    original_code = original_code.replace("python", "").replace("`", "").strip()
+    fixed_code = None
+
+    if infinite_loop_check(original_code):
+        fixed_code = error_fix(original_code)
+    else:
+        fixed_code = original_code
+
+    if syntax_error_check(fixed_code):
+        fixed_code = error_fix(fixed_code)
+
+    return fixed_code
+
+
+@trace
+def code_gen(client: AzureOpenAI, question: str) -> str:
+    sys_prompt = (
+        "I want you to act as a math expert specializing in Algebra, Geometry, and Calculus. "
+        "Given the question, develop python code to model the user's question. "
+        "Make sure only reply the executable code, no other words."
+    )
+    completion = client.chat.completions.create(
+        model=os.getenv("CHAT_DEPLOYMENT_NAME", "gpt-35-turbo"),
+        messages=[
+            {
+                "role": "system",
+                "content": sys_prompt,
+            },
+            {"role": "user", "content": question},
+        ],
+    )
+    raw_code = completion.choices[0].message.content
+    result = code_refine(raw_code)
+    return result
+
+
+if __name__ == "__main__":
+    start_trace()
+
+    if "AZURE_OPENAI_API_KEY" not in os.environ:
+        # load environment variables from .env file
+        load_dotenv()
+
+    client = AzureOpenAI(
+        api_key=os.getenv("AZURE_OPENAI_API_KEY"),
+        azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
+        api_version="2023-12-01-preview",
+    )
+
+    question = "What is 37593 * 67?"
+
+    code = code_gen(client, question)
+    print(code)