chore: import upstream snapshot with attribution

2026-07-13 13:39:52 +08:00
commit e768098d0e
4004 changed files with 2804145 additions and 0 deletions
@@ -0,0 +1,88 @@
+---
+category: rag
+weight: 40
+---
+
+# How to generate test data based on documents
+In this doc, you will learn how to generate test data based on your documents for RAG app.
+This approach helps relieve the efforts of manual data creation, which is typically time-consuming and labor-intensive, or the expensive option of purchasing pre-packaged test data.
+By leveraging the capabilities of llm, this guide streamlines the test data generation process, making it more efficient and cost-effective.
+
+
+## Prerequisites
+
+1. Prepare documents. The test data generator supports the following file types:
+    - .md - Markdown
+    - .docx - Microsoft Word
+    - .pdf - Portable Document Format
+    - .ipynb - Jupyter Notebook
+    - .txt - Text
+
+    **Limitations:**
+
+    - The test data generator may not function effectively for non-Latin characters, such as Chinese, in certain document types. The limitation is caused by dependent text loader capabilities, such as `pypdf`.
+    - The test data generator may not generate meaningful questions if the document is not well-organized or contains massive code snippets/links, such as API introduction documents or reference documents.
+
+2. Prepare local environment. Go to [example_gen_test_data](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) folder and install required packages.
+
+    ```
+    pip install -r requirements.txt
+    ```
+
+    For specific document file types, you may need to install extra packages:
+      - .docx - `pip install docx2txt`
+      - .pdf - `pip install pypdf`
+      - .ipynb - `pip install nbconvert`
+      > !Note: the example uses llama index `SimpleDirectoryReader` to load documents. For the latest information of different file type required packages, please check [here](https://docs.llamaindex.ai/en/stable/examples/data_connectors/simple_directory_reader.html).
+
+3. Install VSCode extension `Prompt flow`.
+
+4. Create your AzureOpenAI or OpenAI connection by following [this doc](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/tune-prompts-with-variants.md).
+
+5. Prepare test data generation setting.
+    - Navigate to [example_gen_test_data](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) folder.
+    - Prepare `config.yml` by copying [`config.yml.example`](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/config.yml.example).
+    - Fill in configurations in the `config.yml` by following inline comment instructions. The config is made up of 3 sections:
+      - Common section: this section provides common values for all other sections. Required.
+      - Local section: this section is for local test data generation related configuration. Can skip if not run in local.
+      - Cloud section: this section is for cloud test data generation related configuration. Can skip if not run in cloud.
+
+      > !Note: Recommend to use `gpt-4` series models than the `gpt-3.5` for better performance.
+
+      > !Note: Recommend to use `gpt-4` model (Azure OpenAI `gpt-4` model with version `0613`) than `gpt-4-turbo` model (Azure OpenAI `gpt-4` model with version `1106`) for better performance. Due to inferior performance of `gpt-4-turbo` model, when you use it, sometimes you might need to open [example test data generation flow](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/flow.dag.yaml) in visual editor and set `response_format` input of nodes `validate_text_chunk`, `validate_question`, and `validate_suggested_answer` to `json`, in order to make sure the llm can generate valid json response.
+
+
+## Generate test data
+- Navigate to [example_gen_test_data](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) folder.
+
+- After configuration, run the following command to generate the test data set:
+  ```
+  python -m gen_test_data.run
+  ```
+
+- The generated test data will be a data jsonl file. See detailed log print in console "Saved ... valid test data to ..." to find it.
+
+If you expect to generate a large amount of test data beyond your local compute capability, you may try generating test data in cloud, please see this [guide](https://github.com/microsoft/promptflow/blob/main/docs/cloud/azureai/generate-test-data-cloud.md) for more detailed steps.
+
+## [*Optional*] Customize test data generation flow
+
+- Open the [example test data generation flow](https://github.com/microsoft/promptflow/tree/main/examples/tutorials/generate-test-data/) in "Prompt flow" VSCode Extension. This flow is designed to generate a pair of question and suggested answer based on the given text chunk. The flow also includes validation prompts to ensure the quality of the generated test data.
+
+- Customize your test data generation logic refering to [tune-prompts-with-variants](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/tune-prompts-with-variants.md).
+
+  **Understanding the prompts**
+
+  The test data generation flow contains 5 prompts, classified into two categories based on their roles: generation prompts and validation prompts. Generation prompts are used to create questions, suggested answers, etc., while validation prompts are used to verify the validity of the text chunk, generated question or answer.
+  - Generation prompts
+    - [*generate question prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/generate_question_prompt.jinja2): frame a question based on the given text chunk.
+    - [*generate suggested answer prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/generate_suggested_answer_prompt.jinja2): generate suggested answer for the question based on the given text chunk.
+  - Validation prompts
+    - [*score text chunk prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/score_text_chunk_prompt.jinja2): score 0-10 to validate if the given text chunk is worthy of framing a question. If the score is lower than `score_threshold` (a node input that is adjustable), validation fails.
+    - [*validate question prompt*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/validate_question_prompt.jinja2): validate if the generated question is good.
+    - [*validate suggested answer*](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/generate-test-data/example_flow/validate_suggested_answer_prompt.jinja2): validate if the generated suggested answer is good.
+
+    If the validation fails, would lead to empty string `question`/`suggested_answer` which are removed from final output test data set.
+
+- Fill in node inputs including `connection`, `model` or `deployment_name`, `response_format`, `score_threshold` or other parameters. Click run button to test the flow in VSCode Extension by referring to [Test flow with VS Code Extension](https://github.com/microsoft/promptflow/blob/main/docs/how-to-guides/develop-a-dag-flow/init-and-test-a-flow.md#visual-editor-on-the-vs-code-for-prompt-flow).
+
+Once the customized flow has been verified, you can proceed to batch generate test data by following the steps outlined in ["Prerequisites"](#prerequisites) and ["Generate test data"](#generate-test-data).
@@ -0,0 +1,11 @@
+name: test_data_gen_conda_env
+channels:
+  - defaults
+dependencies:
+  - python=3.10.12
+  - pip=23.2.1
+  - pip:
+    - mldesigner==0.1.0b18
+    - llama_index==0.9.48
+    - docx2txt==0.8
+    - promptflow>=1.7.0
@@ -0,0 +1,57 @@
+# Common section: this section provides common values for all other sections. Required.
+# Configure 'document_folder', 'document_chunk_size' and 'document_chunk_overlap' if you require document splitting.
+documents_folder: <your-document-folder-path>
+document_chunk_size: 512 # The token chunk size for each chunk.
+document_chunk_overlap: 100 # The token overlap of each chunk when splitting.
+
+# However, if you wish to bypass the document split process, simply provide the 'document_nodes_file', which is a JSONL file.
+# When both 'documents_folder' and 'document_nodes_file' are configured, will use 'document_nodes_file' and ignore 'documents_folder'.
+# For cloud mode, both local files and data assets can be used.
+# document_nodes_file: <your-node-file-path>
+
+# Test data gen flow configs
+# You can utilize our provided example test data generation flow directly. Alternatively, you can create your own flow and set up corresponding node inputs override.
+# The example flow folder path is <promptflow github repo>\examples\gen_test_data\example_flow
+flow_folder: <your-test-data-gen-flow-folder-path>
+node_inputs_override: # Override some node inputs, if not fill in 'node_inputs_override', will use the values in flow.dag.yaml
+  validate_text_chunk: # node name in flow.dag.yaml
+    connection: <your-connection-name> # connection name of node 'validate_text_chunk'
+    # Use 'deployment_name' for Azure OpenAI connection, 'model' for OpenAI
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  generate_question:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  validate_question:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  generate_suggested_answer:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+  validate_suggested_answer:
+    connection: <your-connection-name>
+    deployment_name: <your-deployment-name>
+    # model: <your-model>
+
+
+# Local section: this section is for local test data generation related configuration. Can skip if not run in local.
+output_folder: <your-output-folder-path>
+flow_batch_run_size: 4 # Higher values may speed up flow runs but risk hitting OpenAI's rate limit.
+
+
+# Cloud section: this section is for cloud test data generation related configuration. Can skip if not run in cloud.
+subscription_id: <your-sub-id>
+resource_group: <your-resource-group>
+workspace_name: <your-workspace-name>
+aml_cluster: <your-compute-name>
+
+# Parallel run step configs
+prs_instance_count: 2
+prs_mini_batch_size: 1
+prs_max_concurrency_per_instance: 4
+prs_max_retry_count: 3
+prs_run_invocation_time: 800
+prs_allowed_failed_count: -1
@@ -0,0 +1,129 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+environment:
+  python_requirements_txt: requirements.txt
+inputs:
+  text_chunk:
+    type: string
+    is_chat_input: false
+    default: Prompt flow is a suite of development tools designed to streamline the
+      end-to-end development cycle of LLM-based AI applications, from ideation,
+      prototyping, testing, evaluation to production deployment and monitoring.
+      It makes prompt engineering much easier and enables you to build LLM apps
+      with production quality.
+outputs:
+  question:
+    type: string
+    reference: ${validate_question.output.question}
+  suggested_answer:
+    type: string
+    reference: ${validate_suggested_answer.output.suggested_answer}
+  debug_info:
+    type: string
+    reference: ${generate_debug_info.output}
+nodes:
+- name: score_text_chunk_prompt
+  type: prompt
+  source:
+    type: code
+    path: score_text_chunk_prompt.jinja2
+  inputs:
+    context: ${inputs.text_chunk}
+  use_variants: false
+- name: validate_question_prompt
+  type: prompt
+  source:
+    type: code
+    path: validate_question_prompt.jinja2
+  inputs:
+    question: ${generate_question.output}
+    context: ${inputs.text_chunk}
+  use_variants: false
+- name: generate_question_prompt
+  type: prompt
+  source:
+    type: code
+    path: generate_question_prompt.jinja2
+  inputs:
+    context: ${inputs.text_chunk}
+  use_variants: false
+- name: generate_suggested_answer_prompt
+  type: prompt
+  source:
+    type: code
+    path: generate_suggested_answer_prompt.jinja2
+  inputs:
+    context: ${inputs.text_chunk}
+    question: ${validate_question.output.question}
+  use_variants: false
+- name: generate_question
+  type: python
+  source:
+    type: code
+    path: generate_question.py
+  inputs:
+    connection: ""
+    context: ${validate_text_chunk.output.context}
+    temperature: 0.2
+    generate_question_prompt: ${generate_question_prompt.output}
+  use_variants: false
+- name: validate_question
+  type: python
+  source:
+    type: code
+    path: validate_question.py
+  inputs:
+    connection: ""
+    temperature: 0.2
+    generated_question: ${generate_question.output}
+    validate_question_prompt: ${validate_question_prompt.output}
+  use_variants: false
+- name: generate_suggested_answer
+  type: python
+  source:
+    type: code
+    path: generate_suggested_answer.py
+  inputs:
+    connection: ""
+    context: ${inputs.text_chunk}
+    generate_suggested_answer_prompt: ${generate_suggested_answer_prompt.output}
+    question: ${validate_question.output.question}
+    temperature: 0.2
+  use_variants: false
+- name: generate_debug_info
+  type: python
+  source:
+    type: code
+    path: generate_debug_info.py
+  inputs:
+    text_chunk: ${inputs.text_chunk}
+    validate_suggested_answer_output: ${validate_suggested_answer.output}
+    text_chunk_validation_res: ${validate_text_chunk.output.validation_res}
+    validate_question_output: ${validate_question.output}
+- name: validate_suggested_answer_prompt
+  type: prompt
+  source:
+    type: code
+    path: validate_suggested_answer_prompt.jinja2
+  inputs:
+    answer: ${generate_suggested_answer.output}
+- name: validate_suggested_answer
+  type: python
+  source:
+    type: code
+    path: validate_suggested_answer.py
+  inputs:
+    connection: ""
+    suggested_answer: ${generate_suggested_answer.output}
+    validate_suggested_answer_prompt: ${validate_suggested_answer_prompt.output}
+    temperature: 0.2
+- name: validate_text_chunk
+  type: python
+  source:
+    type: code
+    path: validate_text_chunk.py
+  inputs:
+    connection: ""
+    score_text_chunk_prompt: ${score_text_chunk_prompt.output}
+    context: ${inputs.text_chunk}
+    score_threshold: 4
+    temperature: 0.2
@@ -0,0 +1,47 @@
+from utils import ValidateObj, ValidationResult
+
+from promptflow.core import tool
+
+
+# The inputs section will change based on the arguments of the tool function, after you save the code
+# Adding type to arguments and return value will help the system show the types properly
+# Please update the function name/signature per need
+@tool
+def my_python_tool(
+    text_chunk: str,
+    text_chunk_validation_res: ValidationResult = None,
+    validate_question_output: dict = None,
+    validate_suggested_answer_output: dict = None,
+) -> dict:
+    question_validation_res = validate_question_output["validation_res"]
+
+    generated_suggested_answer = validate_suggested_answer_output["suggested_answer"]
+    suggested_answer_validation_res = validate_suggested_answer_output["validation_res"]
+
+    is_generation_success = generated_suggested_answer != ""
+    is_text_chunk_valid = text_chunk_validation_res["pass_validation"] if text_chunk_validation_res else None
+    is_seed_question_valid = question_validation_res["pass_validation"] if question_validation_res else None
+    is_suggested_answer_valid = (
+        suggested_answer_validation_res["pass_validation"] if suggested_answer_validation_res else None
+    )
+
+    failed_step = ""
+    if not is_generation_success:
+        if is_text_chunk_valid is False:
+            failed_step = ValidateObj.TEXT_CHUNK
+        elif is_seed_question_valid is False:
+            failed_step = ValidateObj.QUESTION
+        elif is_suggested_answer_valid is False:
+            failed_step = ValidateObj.SUGGESTED_ANSWER
+
+    return {
+        # TODO: support more question types like multi-context etc.
+        # "question_type": question_type,
+        "text_chunk": text_chunk,
+        "validation_summary": {"success": is_generation_success, "failed_step": failed_step},
+        "validation_details": {
+            ValidateObj.TEXT_CHUNK: text_chunk_validation_res,
+            ValidateObj.QUESTION: question_validation_res,
+            ValidateObj.SUGGESTED_ANSWER: suggested_answer_validation_res,
+        },
+    }
@@ -0,0 +1,39 @@
+from typing import Union
+
+from utils import llm_call
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def generate_question(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    generate_question_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    context: str = None,
+    temperature: float = 0.2,
+):
+    """
+    Generates a question based on the given context.
+
+    Returns:
+        str: The generated seed question.
+    """
+    # text chunk is not valid, just skip test data gen.
+    if not context:
+        return ""
+
+    seed_question = llm_call(connection, model, deployment_name, generate_question_prompt, temperature=temperature)
+    return seed_question
@@ -0,0 +1,17 @@
+# system:
+
+Your task is to formulate a question from given context satisfying the rules given below:
+
+1.The question should better be framed from the overall context, serving as a general question, rather than just framed from some details.
+2.The question should be specific and answerable from the given context.
+3.The question must be reasonable and must be understood and responded by humans.
+4.The question should not contain phrases like 'provided' or 'given' in the question.
+5.The question should be a question asked by the hypothetical user without any given context.
+6.The question should not contain any links.
+7.The question should not contain more than 20 words, use abbreviation wherever possible.
+
+# user:
+context:
+{{context}}
+
+question:
@@ -0,0 +1,44 @@
+from typing import Union
+
+from utils import llm_call
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def generate_suggested_answer(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    question: str,
+    context: str,
+    generate_suggested_answer_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    temperature: float = 0.2,
+):
+    """
+    Generates a suggested answer based on the given prompts and context information.
+
+    Returns:
+        str: The generated suggested answer.
+    """
+    if question and context:
+        return llm_call(
+            connection,
+            model,
+            deployment_name,
+            generate_suggested_answer_prompt,
+            temperature=temperature,
+        )
+    else:
+        return ""
@@ -0,0 +1,12 @@
+system:
+Provide the answer for the question using the information from the given context based on the following criteria:
+1. The answer is correct and complete.
+2. The answer is derived from the given context.
+3. The answer can totally answer the question.
+4. The answer should not use the words like "in the context". The answer should be enough to answer the question without the context.
+5. If the answer for the question cannot be generated from the given context, just return empty string.
+
+user:
+question:{{question}}
+context:{{context}}
+answer:
@@ -0,0 +1 @@
+promptflow[azure]>=1.7.0
@@ -0,0 +1,49 @@
+# system:
+
+Given a text chunk from a document as context, perform the following tasks：
+
+1. Exclude any references, acknowledgments, personal information, code snippets, or other non-essential elements from the original context.
+
+2. Evaluate the cleaned context against specific criteria for content quality and depth.
+
+3. Assign a numerical score between 0 and 10 based on the following criteria:
+    - Award a high score (closer to 10) if:
+        a) cleaned context delves into and explains concepts.
+        b) cleaned context contains substantial information that could lead to meaningful questions.
+
+    - Award a lower score (closer to 0) if:
+        a) cleaned context is very brief, containing fewer than five words.
+        b) cleaned context is not meaningful.
+
+
+4. Output a valid JSON containing the score and a reason. The reason must directly relate to the criteria outlined above, explaining the basis for the given score.
+
+Here are some examples:
+
+example 1:
+context:
+Albert Einstein (14 March 1879 - 18 April 1955) was a German-born theoretical physicist who is widely held to be one of the greatest and most influential scientists of all time.
+
+output:
+{
+  "score": "8.0",
+  "reason": "The context provides substantial information that could lead to meaningful questions, hence the high score."
+}
+
+example 2:
+context:
+Next step\n- Open the provided examples.
+
+output:
+{
+  "score": "0.0",
+  "reason": "The context lacks detailed information about the provided example and previous steps, resulting in a low score."
+}
+
+
+# user:
+
+context:
+{{context}}
+
+output:
@@ -0,0 +1,170 @@
+import json
+import re
+from collections import namedtuple
+
+from numpy.random import default_rng
+
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.tools.aoai import chat as aoai_chat
+from promptflow.tools.openai import chat as openai_chat
+
+
+class QuestionType:
+    SIMPLE = "simple"
+    # MULTI_CONTEXT = "multi_context"
+
+
+class ValidateObj:
+    QUESTION = "validate_question"
+    TEXT_CHUNK = "validate_text_chunk"
+    SUGGESTED_ANSWER = "validate_suggested_answer"
+
+
+class ResponseFormat:
+    TEXT = "text"
+    JSON = "json_object"
+
+
+class ErrorMsg:
+    INVALID_JSON_FORMAT = "Invalid json format. Response: {0}"
+    INVALID_TEXT_CHUNK = "Skipping generating seed question due to invalid text chunk: {0}"
+    INVALID_QUESTION = "Invalid seed question: {0}"
+    INVALID_ANSWER = "Invalid answer: {0}"
+
+
+ValidationResult = namedtuple("ValidationResult", ["pass_validation", "reason"])
+ScoreResult = namedtuple("ScoreResult", ["score", "reason", "pass_validation"])
+
+
+def llm_call(
+    connection, model, deployment_name, prompt, response_format=ResponseFormat.TEXT, temperature=1.0, max_tokens=None
+):
+    response_format = "json_object" if response_format.lower() == "json" else response_format
+    # avoid unnecessary jinja2 template re-rendering and potential error.
+    prompt = f"{{% raw %}}{prompt}{{% endraw %}}"
+    if isinstance(connection, AzureOpenAIConnection):
+        return aoai_chat(
+            connection=connection,
+            prompt=prompt,
+            deployment_name=deployment_name,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            response_format={"type": response_format},
+        )
+    elif isinstance(connection, OpenAIConnection):
+        return openai_chat(
+            connection=connection,
+            prompt=prompt,
+            model=model,
+            temperature=temperature,
+            max_tokens=max_tokens,
+            response_format={"type": response_format},
+        )
+
+
+def get_question_type(testset_distribution) -> str:
+    """
+    Decides question evolution type based on probability
+    """
+    rng = default_rng()
+    prob = rng.uniform(0, 1)
+    return next((key for key in testset_distribution.keys() if prob <= testset_distribution[key]), QuestionType.SIMPLE)
+
+
+def get_suggested_answer_validation_res(
+    connection,
+    model,
+    deployment_name,
+    prompt,
+    suggested_answer: str,
+    temperature: float,
+    max_tokens: int = None,
+    response_format: ResponseFormat = ResponseFormat.TEXT,
+):
+    rsp = llm_call(
+        connection,
+        model,
+        deployment_name,
+        prompt,
+        temperature=temperature,
+        max_tokens=max_tokens,
+        response_format=response_format,
+    )
+    return retrieve_verdict_and_print_reason(
+        rsp=rsp, validate_obj_name=ValidateObj.SUGGESTED_ANSWER, validate_obj=suggested_answer
+    )
+
+
+def get_question_validation_res(
+    connection,
+    model,
+    deployment_name,
+    prompt,
+    question: str,
+    response_format: ResponseFormat,
+    temperature: float,
+    max_tokens: int = None,
+):
+    rsp = llm_call(connection, model, deployment_name, prompt, response_format, temperature, max_tokens)
+    return retrieve_verdict_and_print_reason(rsp=rsp, validate_obj_name=ValidateObj.QUESTION, validate_obj=question)
+
+
+def get_text_chunk_score(
+    connection,
+    model,
+    deployment_name,
+    prompt,
+    response_format: ResponseFormat,
+    score_threshold: float,
+    temperature: float,
+    max_tokens: int = None,
+):
+    rsp = llm_call(connection, model, deployment_name, prompt, response_format, temperature, max_tokens)
+    data = _load_json_rsp(rsp)
+    score_float = 0
+    reason = ""
+
+    if data and isinstance(data, dict) and "score" in data and "reason" in data:
+        # Extract the verdict and reason
+        score = data["score"].lower()
+        reason = data["reason"]
+        print(f"Score {ValidateObj.TEXT_CHUNK}: {score}\nReason: {reason}")
+        try:
+            score_float = float(score)
+        except ValueError:
+            reason = ErrorMsg.INVALID_JSON_FORMAT.format(rsp)
+    else:
+        reason = ErrorMsg.INVALID_JSON_FORMAT.format(rsp)
+    pass_validation = score_float >= score_threshold
+
+    return ScoreResult(score_float, reason, pass_validation)
+
+
+def retrieve_verdict_and_print_reason(rsp: str, validate_obj_name: str, validate_obj: str) -> ValidationResult:
+    data = _load_json_rsp(rsp)
+
+    if data and isinstance(data, dict) and "verdict" in data and "reason" in data:
+        # Extract the verdict and reason
+        verdict = data["verdict"].lower()
+        reason = data["reason"]
+        print(f"Is valid {validate_obj_name}: {verdict}\nReason: {reason}")
+        if verdict == "yes":
+            return ValidationResult(True, reason)
+        elif verdict == "no":
+            return ValidationResult(False, reason)
+        else:
+            print(f"Unexpected llm response to validate {validate_obj_name}: {validate_obj}")
+
+    return ValidationResult(False, ErrorMsg.INVALID_JSON_FORMAT.format(rsp))
+
+
+def _load_json_rsp(rsp: str):
+    try:
+        # It is possible that even the response format is required as json, the response still contains ```json\n
+        rsp = re.sub(r"```json\n?|```", "", rsp)
+        data = json.loads(rsp)
+    except json.decoder.JSONDecodeError:
+        print(ErrorMsg.INVALID_JSON_FORMAT.format(rsp))
+        data = None
+
+    return data
@@ -0,0 +1,60 @@
+from typing import Union
+
+from utils import ErrorMsg, QuestionType, ResponseFormat, get_question_validation_res
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def validate_question(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    generated_question: str,
+    validate_question_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    response_format: str = ResponseFormat.TEXT,
+    temperature: float = 0.2,
+):
+    """
+    1. Validates the given seed question.
+    2. Generates a test question based on the given prompts and distribution ratios.
+
+    Returns:
+        dict: The generated test question and its type.
+    """
+    # text chunk is not valid, seed question not generated.
+    if not generated_question:
+        return {"question": "", "question_type": "", "validation_res": None}
+
+    validation_res = get_question_validation_res(
+        connection,
+        model,
+        deployment_name,
+        validate_question_prompt,
+        generated_question,
+        response_format,
+        temperature,
+    )
+    is_valid_seed_question = validation_res.pass_validation
+    question = ""
+    question_type = ""
+    failed_reason = ""
+    if not is_valid_seed_question:
+        failed_reason = ErrorMsg.INVALID_QUESTION.format(generated_question)
+        print(failed_reason)
+    else:
+        question = generated_question
+        question_type = QuestionType.SIMPLE
+
+    return {"question": question, "question_type": question_type, "validation_res": validation_res._asdict()}
@@ -0,0 +1,64 @@
+# system:
+
+Verdict a question based on following rules:
+
+1. If there are acronyms or terms in the question, then please check if they exist in the given context. If no, verdict no. If yes, check if other rules are satisfied.
+2. Determine if the given question can be clearly understood and give the reason.
+
+Output a valid json with reason and verdict.
+
+Here are some examples:
+question: What is the discovery about space?
+answer:
+{
+    "reason":"The question is too vague and does not specify which discovery about space it is referring to."
+    "verdict":"no"
+}
+
+question: What caused the Great Depression?
+answer:
+{
+    "reason":"The question is specific and refers to a well-known historical economic event, making it clear and answerable.",
+    "verdict":"yes"
+}
+
+question: What is the keyword that best describes the paper's focus in natural language understanding tasks?
+answer:
+{
+    "reason": "The question mentions a 'paper' in it without referring it's name which makes it unclear without it",
+    "verdict": "no"
+}
+
+question: Who wrote 'Romeo and Juliet'?
+answer:
+{
+    "reason": "The question is clear and refers to a specific work by name therefore it is clear",
+    "verdict": "yes"
+}
+
+question: What did the study mention?
+answer:
+{
+    "reason": "The question is vague and does not specify which study it is referring to",
+    "verdict": "no"
+}
+
+question: What is the focus of the REPLUG paper?
+answer:
+{
+    "reason": "The question refers to a specific work by it's name hence can be understood",
+    "verdict": "yes"
+}
+
+question: What is the purpose of the reward-driven stage in the training process?
+answer:
+{
+    "reason": "The question lacks specific context regarding the type of training process, making it potentially ambiguous and open to multiple interpretations.",
+    "verdict": "no"
+}
+
+# user:
+context: {{context}}
+
+question: {{question}}
+answer:
@@ -0,0 +1,55 @@
+from typing import Union
+
+from utils import ErrorMsg, get_suggested_answer_validation_res
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+@tool
+def validate_suggested_answer(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    suggested_answer: str,
+    validate_suggested_answer_prompt: str,
+    deployment_name: str = "",
+    model: str = "",
+    temperature: float = 0.2,
+    response_format: str = "text",
+):
+    """
+    1. Validates the given suggested answer.
+
+    Returns:
+        dict: The generated suggested answer and its validation result.
+    """
+    if not suggested_answer:
+        return {"suggested_answer": "", "validation_res": None}
+
+    validation_res = get_suggested_answer_validation_res(
+        connection,
+        model,
+        deployment_name,
+        validate_suggested_answer_prompt,
+        suggested_answer,
+        temperature,
+        response_format=response_format,
+    )
+    is_valid_gt = validation_res.pass_validation
+    failed_reason = ""
+    if not is_valid_gt:
+        failed_reason = ErrorMsg.INVALID_ANSWER.format(suggested_answer)
+        print(failed_reason)
+        suggested_answer = ""
+
+    return {"suggested_answer": suggested_answer, "validation_res": validation_res._asdict()}
@@ -0,0 +1,43 @@
+# system:
+
+Given an answer, verdict if the provided answer is valid and provide the reason in valid json format.
+The answer is not valid if the answer suggests that the context does not provide information or indicates uncertainty (such as 'I don't know'), it is deemed invalid. For any other case, the answer is considered valid.
+
+# user:
+
+Output a json format with the reason and verdict.
+Here are some examples:
+answer:
+The steps to build and install your tool package for use in VS Code extension are not provided in the context.
+output:
+{
+    "reason":"The answer is invalid because it states that the context does not provide the necessary steps.",
+    "verdict":"no"
+}
+
+answer:
+The context does not provide specific information on what the possible provider values are in supported configs for a connection provider.
+output:
+{
+    "reason":"The answer is invalid as it indicates that the context lacks specific information.",
+    "verdict":"no"
+}
+
+answer:
+I don't know.
+output:
+{
+    "reason":"The answer is invalid because it conveys don't know.",
+    "verdict":"no"
+}
+
+answer:
+The two essential components of an activate config in a node flow are `activate.when` and `activate.is`.
+output:
+{
+    "reason":"The answer is valid because it is clear and true.",
+    "verdict":"yes"
+}
+
+answer:{{answer}}
+output:
@@ -0,0 +1,49 @@
+from typing import Union
+
+from utils import ErrorMsg, ResponseFormat, get_text_chunk_score
+
+from promptflow._core.tool import InputSetting
+from promptflow.connections import AzureOpenAIConnection, OpenAIConnection
+from promptflow.core import tool
+
+
+@tool(
+    input_settings={
+        "deployment_name": InputSetting(
+            enabled_by="connection",
+            enabled_by_type=["AzureOpenAIConnection"],
+            capabilities={"completion": False, "chat_completion": True, "embeddings": False},
+        ),
+        "model": InputSetting(enabled_by="connection", enabled_by_type=["OpenAIConnection"]),
+    }
+)
+def validate_text_chunk(
+    connection: Union[OpenAIConnection, AzureOpenAIConnection],
+    score_text_chunk_prompt: str,
+    score_threshold: float,
+    deployment_name: str = "",
+    model: str = "",
+    context: str = None,
+    response_format: str = ResponseFormat.TEXT,
+    temperature: float = 0.2,
+):
+    """
+    Validates the given text chunk. If the validation fails, return an empty context and the validation result.
+
+    Returns:
+        dict: Text chunk context and its validation result.
+    """
+    text_chunk_score_res = get_text_chunk_score(
+        connection,
+        model,
+        deployment_name,
+        score_text_chunk_prompt,
+        response_format,
+        score_threshold,
+        temperature,
+    )
+    if not text_chunk_score_res.pass_validation:
+        print(ErrorMsg.INVALID_TEXT_CHUNK.format(context))
+        return {"context": "", "validation_res": text_chunk_score_res._asdict()}
+
+    return {"context": context, "validation_res": text_chunk_score_res._asdict()}
@@ -0,0 +1,249 @@
+import json
+import re
+import sys
+import time
+import typing as t
+from pathlib import Path
+
+from constants import DOCUMENT_NODE, NODES_FILE_NAME, SUPPORT_FILE_TYPE, TEXT_CHUNK
+
+from promptflow._utils.logger_utils import get_logger
+
+
+def split_document(chunk_size, chunk_overlap, documents_folder, document_node_output):
+    try:
+        from llama_index import SimpleDirectoryReader
+        from llama_index.node_parser import SentenceSplitter
+        from llama_index.readers.schema import Document as LlamaindexDocument
+        from llama_index.schema import BaseNode
+    except ImportError as e:
+        raise ImportError(
+            f"{str(e)}. It appears that `llama_index` may not be installed, or the installed version may be incorrect."
+            "Please check `requirements.txt` file and install all the dependencies."
+        )
+
+    logger = get_logger("doc.split")
+    logger.info("Step 1: Start to split documents to document nodes...")
+    # count the number of files in documents_folder, including subfolders.
+    all_files = [f for f in Path(documents_folder).rglob("*") if f.is_file()]
+    filtered_num_files = sum(1 for _ in all_files if _.suffix.lower() in SUPPORT_FILE_TYPE)
+    logger.info(
+        f"Found {len(all_files)} files in the documents folder '{documents_folder}'. "
+        f"After filtering out unsupported file types, {filtered_num_files} files remain."
+        f"Using chunk size: {chunk_size} to split."
+    )
+    # `SimpleDirectoryReader` by default chunk the documents based on heading tags and paragraphs, which may lead to small chunks.  # noqa: E501
+    reader = SimpleDirectoryReader(documents_folder, required_exts=SUPPORT_FILE_TYPE, recursive=True, encoding="utf-8")
+    # Disable the default suffixes to avoid splitting the documents into small chunks.
+    # TODO: find a better way to disable the default suffixes.
+    SimpleDirectoryReader.supported_suffix = []
+    chunks = reader.load_data()
+    # Convert documents into nodes
+    node_parser = SentenceSplitter.from_defaults(
+        chunk_size=chunk_size, chunk_overlap=chunk_overlap, include_metadata=True
+    )
+    chunks = t.cast(t.List[LlamaindexDocument], chunks)
+    document_nodes: t.List[BaseNode] = node_parser.get_nodes_from_documents(documents=chunks)
+    logger.info(f"Split the documents and created {len(document_nodes)} document nodes.")
+    document_nodes_output_path = document_node_output / Path(NODES_FILE_NAME)
+    with open(document_nodes_output_path, "wt") as text_file:
+        for doc in document_nodes:
+            print(json.dumps({TEXT_CHUNK: doc.text, DOCUMENT_NODE: doc.to_json()}), file=text_file)
+
+    logger.info(f"Saved document nodes to '{document_nodes_output_path}'.")
+    return str(Path(document_node_output) / NODES_FILE_NAME)
+
+
+def clean_data(test_data_set: list, test_data_output_path: str):
+    logger = get_logger("data.clean")
+    logger.info("Step 3: Start to clean invalid test data...")
+    logger.info(f"Collected {len(test_data_set)} test data after the batch run.")
+    cleaned_data = []
+
+    for test_data in test_data_set:
+        if test_data and all(
+            val and val != "(Failed)" for key, val in test_data.items() if key.lower() != "line_number"
+        ):
+            data_line = {"question": test_data["question"], "suggested_answer": test_data["suggested_answer"]}
+            cleaned_data.append(data_line)
+
+    jsonl_str = "\n".join(map(json.dumps, cleaned_data))
+    with open(test_data_output_path, "wt") as text_file:
+        print(f"{jsonl_str}", file=text_file)
+
+    # TODO: aggregate invalid data root cause and count, and log it.
+    # log debug info path.
+    logger.info(
+        f"Removed {len(test_data_set) - len(cleaned_data)} invalid test data. "
+        f"Saved {len(cleaned_data)} valid test data to '{test_data_output_path}'."
+    )
+
+
+def count_non_blank_lines(file_path):
+    with open(file_path, "r") as file:
+        lines = file.readlines()
+
+    non_blank_lines = len([line for line in lines if line.strip()])
+    return non_blank_lines
+
+
+def print_progress(log_file_path: str, process):
+    from tqdm import tqdm
+
+    logger = get_logger("data.gen")
+    finished_log_pattern = re.compile(r".*execution.bulk\s+INFO\s+Finished (\d+) / (\d+) lines\.")
+    progress_log_pattern = re.compile(
+        r".*execution.bulk\s+INFO.*\[Finished: (\d+)\] \[Processing: (\d+)\] \[Pending: (\d+)\]"
+    )
+    # wait for the log file to be created
+    start_time = time.time()
+    while not Path(log_file_path).is_file():
+        time.sleep(1)
+        # if the log file is not created within 5 minutes, raise an error
+        if time.time() - start_time > 300:
+            raise Exception(f"Log file '{log_file_path}' is not created within 5 minutes.")
+
+    logger.info(f"Click '{log_file_path}' to see detailed batch run log. Showing the progress here...")
+    progress_bar = None
+    try:
+        last_data_time = time.time()
+        with open(log_file_path, "r") as f:
+            while True:
+                status = process.poll()
+                # status is None if not finished, 0 if finished successfully, and non-zero if failed
+                if status:
+                    stdout, _ = process.communicate()
+                    raise Exception(f"Batch run failed due to {stdout.decode('utf-8')}")
+
+                line = f.readline().strip()
+                if line:
+                    last_data_time = time.time()  # Update the time when the last data was received
+                    progress_match = progress_log_pattern.match(line)
+                    finished_match = finished_log_pattern.match(line)
+                    if not progress_match and not finished_match:
+                        continue
+
+                    if progress_match:
+                        finished, processing, pending = map(int, progress_match.groups())
+                        total = finished + processing + pending
+                        if progress_bar is None:
+                            # Set mininterval=0 to refresh the progress bar when it calls progress_bar.update
+                            # after initialization.
+                            progress_bar = tqdm(total=total, desc="Processing", mininterval=0, file=sys.stdout)
+                        progress_bar.update(finished - progress_bar.n)
+
+                    if finished_match:
+                        finished, total = map(int, finished_match.groups())
+                        if progress_bar is None:
+                            progress_bar = tqdm(total=total, desc="Processing", mininterval=0, file=sys.stdout)
+                        progress_bar.update(finished - progress_bar.n)
+
+                        if finished == total:
+                            progress_bar.close()
+                            logger.info("Batch run is completed.")
+
+                            break
+                elif time.time() - last_data_time > 300:
+                    logger.info(
+                        "No new log line received for 5 minutes. Stop reading. "
+                        f"See the log file '{log_file_path}' for more details."
+                    )
+                    break
+                else:
+                    time.sleep(1)  # wait for 1 second if no new line is available
+    except Exception as e:
+        raise Exception(f"Error occurred while printing batch run progress: {e}.")
+    finally:
+        if progress_bar:
+            progress_bar.close()
+
+
+def convert_to_abs_path(file_path: str) -> str:
+    if not file_path:
+        return file_path
+
+    path = Path(file_path)
+    if path.is_absolute():
+        return str(path)
+    elif path.exists():
+        abs = str(path.resolve())
+        return abs
+    else:
+        return file_path
+
+
+def local_path_exists(path):
+    return Path(path).exists()
+
+
+def non_padding_path(path):
+    return not (path.startswith("<") and path.endswith(">"))
+
+
+def _retrieve_file_names_from_document_nodes_file(document_nodes_file_path) -> t.List[str]:
+    text_info = {}
+    with open(document_nodes_file_path, "r") as file:
+        for line in file:
+            # Should skip empty new lines, otherwise, json.loads would throw error.
+            if not line.strip():
+                continue
+            line_json = json.loads(line)
+            text_chunk = line_json[TEXT_CHUNK]
+            document_node = json.loads(line_json["document_node"])
+            file_path = document_node["metadata"]["file_path"]
+            text_info[text_chunk] = file_path
+    return text_info
+
+
+def _count_lines(file_path) -> int:
+    with open(file_path, "r") as f:
+        return sum(1 for line in f if line.strip())
+
+
+def summarize_batch_run_res(gen_details_file_path, document_nodes_file_path, output_file_path):
+    success_count = 0
+    validate_failed_count = 0
+    validate_failed_steps = {}
+    validate_failed_distribution = {}
+
+    nodes_file_lines_count = _count_lines(document_nodes_file_path)
+    document_nodes_info = _retrieve_file_names_from_document_nodes_file(document_nodes_file_path)
+
+    with open(gen_details_file_path, "r") as details_f:
+        for details_line in details_f:
+            # Should skip empty new lines, otherwise, json.loads would throw error.
+            if not details_line.strip():
+                continue
+            data = json.loads(details_line)
+            if data["debug_info"] == "(Failed)":
+                continue
+
+            if data["debug_info"]["validation_summary"]["success"]:
+                success_count += 1
+            else:
+                validate_failed_count += 1
+                failed_step = data["debug_info"]["validation_summary"]["failed_step"]
+
+                if failed_step in validate_failed_steps:
+                    validate_failed_steps[failed_step] += 1
+                else:
+                    validate_failed_steps[failed_step] = 1
+                    validate_failed_distribution[failed_step] = {}
+
+                document_name = document_nodes_info[data["debug_info"]["text_chunk"]]
+                if document_name in validate_failed_distribution[failed_step]:
+                    validate_failed_distribution[failed_step][document_name] += 1
+                else:
+                    validate_failed_distribution[failed_step][document_name] = 1
+
+    data = {
+        "total_count": nodes_file_lines_count,
+        "success_count": success_count,
+        "run_failed_count": nodes_file_lines_count - success_count - validate_failed_count,
+        "validate_failed_count": validate_failed_count,
+        "validate_failed_steps": validate_failed_steps,
+        "validate_failed_distribution": validate_failed_distribution,
+    }
+
+    with open(output_file_path, "w") as file:
+        json.dump(data, file, indent=4)
@@ -0,0 +1,90 @@
+import json
+from pathlib import Path
+
+from common import clean_data, split_document, summarize_batch_run_res
+from constants import NODES_FILE_NAME, PARALLEL_RUN_STEP_FILE_NAME, SUMMARY_FILE_NAME, TEST_DATA_FILE_NAME
+from mldesigner import Input, Output, command_component
+
+conda_file = Path(__file__).parent.parent / "conda.yml"
+env_image = "mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04"
+
+
+@command_component(
+    name="split_document_component",
+    display_name="split documents",
+    description="Split documents into document nodes.",
+    environment=dict(
+        conda_file=conda_file,
+        image=env_image,
+    ),
+)
+def split_document_component(
+    documents_folder: Input(type="uri_folder"),
+    chunk_size: int,
+    chunk_overlap: int,
+    document_node_output: Output(type="uri_folder"),
+) -> str:
+    """Split documents into document nodes.
+
+    Args:
+        documents_folder: The folder containing documents to be split.
+        chunk_size: The size of each chunk.
+        document_node_output: The output folder
+        chunk_overlap: The size of chunk overlap
+
+    Returns:
+        The folder containing the split documents.
+    """
+    return split_document(chunk_size, chunk_overlap, documents_folder, document_node_output)
+
+
+@command_component(
+    name="clean_data_component",
+    display_name="clean dataset",
+    description="Clean test data set to remove empty lines.",
+    environment=dict(
+        conda_file=conda_file,
+        image=env_image,
+    ),
+)
+def clean_data_component(
+    test_data_set_folder: Input(type="uri_folder"), test_data_output: Output(type="uri_folder")
+) -> str:
+    test_data_set_path = Path(test_data_set_folder) / PARALLEL_RUN_STEP_FILE_NAME
+
+    with open(test_data_set_path, "r") as f:
+        data = [json.loads(line) for line in f]
+
+    test_data_output_path = test_data_output / Path(TEST_DATA_FILE_NAME)
+    clean_data(data, test_data_output_path)
+
+    return str(test_data_output_path)
+
+
+@command_component(
+    name="summarize_generation_details_component",
+    display_name="summarize generation details",
+    description="Summarize generation details.",
+    environment=dict(
+        conda_file=conda_file,
+        image=env_image,
+    ),
+)
+def summarize_generation_details_component(
+    document_node_output: Input(type="uri_folder"),
+    test_data_set_folder: Input(type="uri_folder"),
+    summary_output: Output(type="uri_folder"),
+) -> str:
+    test_data_set_path = Path(test_data_set_folder) / PARALLEL_RUN_STEP_FILE_NAME
+    document_node_output_path = Path(document_node_output)
+
+    summary_output_path = summary_output / Path(SUMMARY_FILE_NAME)
+    if document_node_output_path.is_dir():
+        document_node_output_path = document_node_output_path / NODES_FILE_NAME
+    summarize_batch_run_res(
+        gen_details_file_path=test_data_set_path,
+        document_nodes_file_path=document_node_output_path,
+        output_file_path=summary_output_path,
+    )
+
+    return str(summary_output_path)
@@ -0,0 +1,8 @@
+DOCUMENT_NODE = "document_node"
+TEXT_CHUNK = "text_chunk"
+NODES_FILE_NAME = "document_nodes.jsonl"
+DETAILS_FILE_NAME = "test-data-gen-details.jsonl"
+PARALLEL_RUN_STEP_FILE_NAME = "parallel_run_step.jsonl"
+SUMMARY_FILE_NAME = "test-data-gen-summary.json"
+TEST_DATA_FILE_NAME = "test-data.jsonl"
+SUPPORT_FILE_TYPE = [".docx", ".pdf", ".ipynb", ".md", ".txt"]
@@ -0,0 +1,324 @@
+import argparse
+import json
+import os
+import time
+from datetime import datetime
+from pathlib import Path
+
+from promptflow._utils.logger_utils import get_logger
+from promptflow._utils.yaml_utils import load_yaml
+
+CONFIG_FILE = (Path(__file__).parents[1] / "config.yml").resolve()
+
+# in order to import from absolute path, which is required by mldesigner
+os.sys.path.insert(0, os.path.abspath(Path(__file__).parent))
+
+from common import (  # noqa: E402
+    clean_data,
+    convert_to_abs_path,
+    count_non_blank_lines,
+    local_path_exists,
+    non_padding_path,
+    print_progress,
+    split_document,
+    summarize_batch_run_res,
+)
+from constants import DETAILS_FILE_NAME, SUMMARY_FILE_NAME, TEST_DATA_FILE_NAME, TEXT_CHUNK  # noqa: E402
+
+logger = get_logger("data.gen")
+
+
+def batch_run_flow(flow_folder: str, flow_input_data: str, flow_batch_run_size: int, node_inputs_override: dict):
+    logger.info(f"Step 2: Start to batch run '{flow_folder}'...")
+    import subprocess
+
+    run_name = f"test_data_gen_{datetime.now().strftime('%b-%d-%Y-%H-%M-%S')}"
+    # TODO: replace the separate process to submit batch run with batch run async method when it's available.
+    connections_str = ""
+    for node_name, node_val in node_inputs_override.items():
+        for k, v in node_val.items():
+            # need to double quote the value to make sure the value can be passed correctly
+            # when the value contains special characters like "<".
+            connections_str += f'{node_name}.{k}="{v}" '
+    connections_str = connections_str.rstrip()
+
+    cmd = (
+        f'pf run create --flow "{flow_folder}" --data "{flow_input_data}" --name {run_name} '
+        f"--environment-variables PF_WORKER_COUNT='{flow_batch_run_size}' PF_BATCH_METHOD='spawn' "
+        f"--column-mapping {TEXT_CHUNK}='${{data.text_chunk}}' --connections {connections_str} --debug"
+    )
+    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    logger.info(
+        f"Submit batch run successfully. process id {process.pid}. Please wait for the batch run to complete..."
+    )
+    return run_name, process
+
+
+def get_batch_run_output(output_path: Path):
+    logger.info(f"Reading batch run output from '{output_path}'.")
+    # wait for the output file to be created
+    start_time = time.time()
+    while not Path(output_path).is_file():
+        time.sleep(1)
+        # if the log file is not created within 5 minutes, raise an error
+        if time.time() - start_time > 300:
+            raise Exception(f"Output jsonl file '{output_path}' is not created within 5 minutes.")
+
+    output_lines = []
+    try:
+        with open(output_path, "r", encoding="utf-8") as f:
+            output_lines = list(map(json.loads, f))
+    except json.decoder.JSONDecodeError as e:
+        logger.warning(
+            f"Error reading the output file: {e}. It could be that the batch run output is empty. "
+            "Please check your flow and ensure it can run successfully."
+        )
+
+    return [
+        {"question": line["question"], "suggested_answer": line["suggested_answer"], "debug_info": line["debug_info"]}
+        for line in output_lines
+    ]
+
+
+def run_local(
+    documents_folder: str,
+    document_chunk_size: int,
+    document_chunk_overlap: int,
+    document_nodes_file: str,
+    flow_folder: str,
+    flow_batch_run_size: int,
+    output_folder: str,
+    should_skip_split: bool,
+    node_inputs_override: dict,
+):
+    text_chunks_path = document_nodes_file
+    output_folder = Path(output_folder) / datetime.now().strftime("%b-%d-%Y-%H-%M-%S")
+    if not Path(output_folder).is_dir():
+        Path(output_folder).mkdir(parents=True, exist_ok=True)
+
+    if not should_skip_split:
+        text_chunks_path = split_document(document_chunk_size, document_chunk_overlap, documents_folder, output_folder)
+
+    run_name, process = batch_run_flow(flow_folder, text_chunks_path, flow_batch_run_size, node_inputs_override)
+
+    run_folder_path = Path.home() / f".promptflow/.runs/{run_name}"
+    print_progress(run_folder_path / "logs.txt", process)
+    test_data_set = get_batch_run_output(run_folder_path / "outputs.jsonl")
+    # Store intermedian batch run output results
+    jsonl_str = "\n".join(map(json.dumps, test_data_set))
+    batch_run_details_file = Path(output_folder) / DETAILS_FILE_NAME
+    with open(batch_run_details_file, "wt") as text_file:
+        print(f"{jsonl_str}", file=text_file)
+
+    clean_data_output = Path(output_folder) / TEST_DATA_FILE_NAME
+    clean_data(test_data_set, clean_data_output)
+    logger.info(f"More debug info of test data generation can be found in '{batch_run_details_file}'.")
+
+    try:
+        summary_output_file = Path(output_folder) / SUMMARY_FILE_NAME
+        summarize_batch_run_res(
+            gen_details_file_path=batch_run_details_file,
+            document_nodes_file_path=text_chunks_path,
+            output_file_path=summary_output_file,
+        )
+        logger.info(f"Check test data generation summary in '{summary_output_file}'.")
+    except Exception as e:
+        logger.warning(f"Error to analyze batch run results: {e}")
+
+
+def run_cloud(
+    documents_folder: str,
+    document_chunk_size: int,
+    document_chunk_overlap: int,
+    document_nodes_file: str,
+    flow_folder: str,
+    subscription_id: str,
+    resource_group: str,
+    workspace_name: str,
+    aml_cluster: str,
+    prs_instance_count: int,
+    prs_mini_batch_size: int,
+    prs_max_concurrency_per_instance: int,
+    prs_max_retry_count: int,
+    prs_run_invocation_time: int,
+    prs_allowed_failed_count: int,
+    should_skip_split: bool,
+    node_inputs_override: dict,
+):
+    # lazy import azure dependencies
+    try:
+        from azure.ai.ml import Input as V2Input
+        from azure.ai.ml import MLClient, dsl, load_component
+        from azure.ai.ml.entities import RetrySettings
+        from azure.identity import DefaultAzureCredential
+    except ImportError:
+        raise ImportError(
+            "Please install azure dependencies using the following command: "
+            + "`pip install -r requirements_cloud.txt`"
+        )
+
+    @dsl.pipeline(
+        non_pipeline_inputs=[
+            "flow_yml_path",
+            "should_skip_doc_split",
+            "instance_count",
+            "mini_batch_size",
+            "max_concurrency_per_instance",
+            "max_retry_count",
+            "run_invocation_time",
+            "allowed_failed_count",
+        ]
+    )
+    def gen_test_data_pipeline(
+        data_input: V2Input,
+        flow_yml_path: str,
+        should_skip_doc_split: bool,
+        chunk_size=1024,
+        chunk_overlap=200,
+        instance_count=1,
+        mini_batch_size=1,
+        max_concurrency_per_instance=2,
+        max_retry_count=3,
+        run_invocation_time=600,
+        allowed_failed_count=-1,
+    ):
+        from components import clean_data_component, split_document_component, summarize_generation_details_component
+
+        data = (
+            data_input
+            if should_skip_doc_split
+            else split_document_component(
+                documents_folder=data_input, chunk_size=chunk_size, chunk_overlap=chunk_overlap
+            ).outputs.document_node_output
+        )
+        flow_node = load_component(flow_yml_path, params_override=[{"name": "gen_test_data_example_flow"}])(
+            data=data, text_chunk="${data.text_chunk}", connections=node_inputs_override
+        )
+        flow_node.mini_batch_size = mini_batch_size
+        flow_node.max_concurrency_per_instance = max_concurrency_per_instance
+        flow_node.set_resources(instance_count=instance_count)
+        flow_node.retry_settings = RetrySettings(max_retry_count=max_retry_count, timeout=run_invocation_time)
+        flow_node.mini_batch_error_threshold = allowed_failed_count
+        # Should use `mount` mode to ensure PRS complete merge output lines.
+        flow_node.outputs.flow_outputs.mode = "mount"
+        clean_data_component(test_data_set_folder=flow_node.outputs.flow_outputs).outputs.test_data_output
+        summarize_generation_details_component(
+            document_node_output=data, test_data_set_folder=flow_node.outputs.flow_outputs
+        ).outputs.summary_output
+
+    def get_ml_client(subscription_id: str, resource_group: str, workspace_name: str):
+        credential = DefaultAzureCredential(exclude_shared_token_cache_credential=True)
+        return MLClient(
+            credential=credential,
+            subscription_id=subscription_id,
+            resource_group_name=resource_group,
+            workspace_name=workspace_name,
+        )
+
+    ml_client = get_ml_client(subscription_id, resource_group, workspace_name)
+
+    if should_skip_split:
+        data_input = V2Input(path=document_nodes_file, type="uri_file")
+    else:
+        data_input = V2Input(path=documents_folder, type="uri_folder")
+
+    prs_configs = {
+        "instance_count": prs_instance_count,
+        "mini_batch_size": prs_mini_batch_size,
+        "max_concurrency_per_instance": prs_max_concurrency_per_instance,
+        "max_retry_count": prs_max_retry_count,
+        "run_invocation_time": prs_run_invocation_time,
+        "allowed_failed_count": prs_allowed_failed_count,
+    }
+
+    pipeline_with_flow = gen_test_data_pipeline(
+        data_input=data_input,
+        flow_yml_path=os.path.join(flow_folder, "flow.dag.yaml"),
+        should_skip_doc_split=should_skip_split,
+        chunk_size=document_chunk_size,
+        chunk_overlap=document_chunk_overlap,
+        **prs_configs,
+    )
+    pipeline_with_flow.compute = aml_cluster
+    studio_url = ml_client.jobs.create_or_update(pipeline_with_flow).studio_url
+    logger.info(f"Completed to submit pipeline. Experiment Link: {studio_url}")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--cloud", action="store_true", help="Run test data generation at cloud.")
+    args = parser.parse_args()
+
+    if Path(CONFIG_FILE).is_file():
+        with open(CONFIG_FILE, "r") as stream:
+            config = load_yaml(stream)
+    else:
+        raise Exception(
+            f"'{CONFIG_FILE}' does not exist. "
+            + "Please check if you are under the wrong directory or the file is missing."
+        )
+
+    should_skip_split_documents = False
+    document_nodes_file = convert_to_abs_path(config.get("document_nodes_file", None))
+    documents_folder = convert_to_abs_path(config.get("documents_folder", None))
+    flow_folder = convert_to_abs_path(config.get("flow_folder", None))
+    output_folder = convert_to_abs_path(config.get("output_folder", None))
+    validate_path_func = non_padding_path if args.cloud else local_path_exists
+    node_inputs_override = config.get("node_inputs_override", None)
+
+    if document_nodes_file and validate_path_func(document_nodes_file):
+        should_skip_split_documents = True
+    elif not documents_folder or not validate_path_func(documents_folder):
+        raise Exception(
+            "Neither 'documents_folder' nor 'document_nodes_file' is valid.\n"
+            f"documents_folder: '{documents_folder}'\ndocument_nodes_file: '{document_nodes_file}'"
+        )
+
+    if not validate_path_func(flow_folder):
+        raise Exception(f"Invalid flow folder: '{flow_folder}'")
+
+    if args.cloud:
+        logger.info("Start to generate test data at cloud...")
+    else:
+        logger.info("Start to generate test data at local...")
+
+    if should_skip_split_documents:
+        logger.info(
+            "Skip step 1 'Split documents to document nodes' as received document nodes from "
+            f"input file path '{document_nodes_file}'."
+        )
+        if Path(document_nodes_file).is_file():
+            logger.info(f"Collected {count_non_blank_lines(document_nodes_file)} document nodes.")
+
+    if args.cloud:
+        run_cloud(
+            documents_folder,
+            config.get("document_chunk_size", 512),
+            config.get("document_chunk_overlap", 100),
+            document_nodes_file,
+            flow_folder,
+            config["subscription_id"],
+            config["resource_group"],
+            config["workspace_name"],
+            config["aml_cluster"],
+            config.get("prs_instance_count", 2),
+            config.get("prs_mini_batch_size", 1),
+            config.get("prs_max_concurrency_per_instance", 4),
+            config.get("prs_max_retry_count", 3),
+            config.get("prs_run_invocation_time", 800),
+            config.get("prs_allowed_failed_count", -1),
+            should_skip_split_documents,
+            node_inputs_override,
+        )
+    else:
+        run_local(
+            documents_folder,
+            config.get("document_chunk_size", 512),
+            config.get("document_chunk_overlap", 100),
+            document_nodes_file,
+            flow_folder,
+            config.get("flow_batch_run_size", 16),
+            output_folder,
+            should_skip_split_documents,
+            node_inputs_override,
+        )
@@ -0,0 +1,3 @@
+promptflow>=1.7.0
+promptflow-tools
+llama_index==0.9.48
@@ -0,0 +1,4 @@
+promptflow>=1.7.0
+promptflow-tools
+azure-ai-ml==1.15.0
+mldesigner==0.1.0b18