chore: import upstream snapshot with attribution

examples/flex-flows/eval-code-quality/README.md

@@ -0,0 +1,90 @@
+# Eval Code Quality
+A example flow defined using class based entry which leverages model config to evaluate the quality of code snippet.
+
+## Prerequisites
+
+Install promptflow sdk and other dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+## Run flow
+
+- Prepare your Azure OpenAI resource follow this [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) and get your `api_key` if you don't have one.
+
+- Setup connection
+
+Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
+
+Or use CLI to create connection:
+
+```bash
+# Override keys with --set to avoid yaml file changes
+pf connection create --file ../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
+```
+
+Note in [flow.flex.yaml](flow.flex.yaml) we are using connection named `open_ai_connection`.
+```bash
+# show registered connection
+pf connection show --name open_ai_connection
+```
+
+- Run as normal Python file
+```bash
+python code_quality.py
+```
+
+- Test flow
+```bash
+# correct
+pf flow test --flow . --inputs code='print(\"Hello, world!\")' --init init.json
+
+# incorrect
+pf flow test --flow . --inputs code='printf("Hello, world!")' --init init.json
+```
+
+- Create run with multiple lines data
+
+```bash
+pf run create --flow . --init init.json --data ./data.jsonl --stream
+```
+
+Reference [here](https://aka.ms/pf/column-mapping) for default behavior when `column-mapping` not provided in CLI.
+
+- List and show run meta
+
+```bash
+# list created run
+pf run list
+
+# get a sample run name
+
+name=$(pf run list -r 10 | jq '.[] | select(.name | contains("eval_code_quality_")) | .name'| head -n 1 | tr -d '"')
+# show specific run detail
+pf run show --name $name
+
+# show output
+pf run show-details --name $name
+
+# show metrics
+pf run show-metrics --name $name
+
+# visualize run in browser
+pf run visualize --name $name
+```
+
+## Run flow in cloud
+
+- Assume we already have a connection named `open_ai_connection` in workspace.
+
+```bash
+# set default workspace
+az account set -s <your_subscription_id>
+az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
+```
+
+- Create run
+
+```bash
+# run with environment variable reference connection in azureml workspace
+pfazure run create --flow . --init init.json --data ./data.jsonl --stream

examples/flex-flows/eval-code-quality/code_quality.py

@@ -0,0 +1,69 @@
+import json
+
+from typing import TypedDict
+from pathlib import Path
+
+from jinja2 import Template
+
+from promptflow.tracing import trace
+from promptflow.core import AzureOpenAIModelConfiguration
+from promptflow.core._flow import Prompty
+
+BASE_DIR = Path(__file__).absolute().parent
+
+
+@trace
+def load_prompt(jinja2_template: str, code: str, examples: list) -> str:
+    """Load prompt function."""
+    with open(BASE_DIR / jinja2_template, "r", encoding="utf-8") as f:
+        tmpl = Template(f.read(), trim_blocks=True, keep_trailing_newline=True)
+        prompt = tmpl.render(code=code, examples=examples)
+        return prompt
+
+
+class Result(TypedDict):
+    correctness: float
+    readability: float
+    explanation: str
+
+
+class CodeEvaluator:
+    def __init__(self, model_config: AzureOpenAIModelConfiguration):
+        self.model_config = model_config
+
+    def __call__(self, code: str) -> Result:
+        """Evaluate the code based on correctness, readability."""
+        prompty = Prompty.load(
+            source=BASE_DIR / "eval_code_quality.prompty",
+            model={"configuration": self.model_config},
+        )
+        output = prompty(code=code)
+        output = json.loads(output)
+        output = Result(**output)
+        return output
+
+    def __aggregate__(self, line_results: list) -> dict:
+        """Aggregate the results."""
+        total = len(line_results)
+        avg_correctness = sum(int(r["correctness"]) for r in line_results) / total
+        avg_readability = sum(int(r["readability"]) for r in line_results) / total
+        return {
+            "average_correctness": avg_correctness,
+            "average_readability": avg_readability,
+            "total": total,
+        }
+
+
+if __name__ == "__main__":
+    from promptflow.tracing import start_trace
+
+    start_trace()
+    model_config = AzureOpenAIModelConfiguration(
+        connection="open_ai_connection",
+        azure_deployment="gpt-4o",
+    )
+    evaluator = CodeEvaluator(model_config)
+    result = evaluator('print("Hello, world!")')
+    print(result)
+    aggregate_result = evaluator.__aggregate__([result])
+    print(aggregate_result)

examples/flex-flows/eval-code-quality/data.jsonl

@@ -0,0 +1,2 @@
+{"code": "print(\"Hello, world!\")"}
+{"code": "printf(\"Hello, world!\")"}

examples/flex-flows/eval-code-quality/eval_code_quality.prompty

@@ -0,0 +1,38 @@
+---
+name: Evaluate code quality
+description: Evaluate the quality of code snippet.
+model:
+  api: chat
+  configuration:
+    type: azure_openai
+    azure_deployment: gpt-4o
+  parameters:
+    temperature: 0.2
+inputs: 
+  code:
+    type: string
+sample: ${file:sample.json}
+---
+# system:
+You are an AI assistant. 
+You task is to evaluate the code based on correctness, readability.
+Only accepts valid JSON format response without extra prefix or postfix.
+
+# user:
+This correctness value should always be an integer between 1 and 5. So the correctness produced should be 1 or 2 or 3 or 4 or 5.
+This readability value should always be an integer between 1 and 5. So the readability produced should be 1 or 2 or 3 or 4 or 5.
+
+Here are a few examples:
+
+**Example 1**
+Code: print(\"Hello, world!\")
+OUTPUT:
+{
+    "correctness": 5,
+    "readability": 5,
+    "explanation": "The code is correct as it is a simple question and answer format. The readability is also good as the code is short and easy to understand."
+}
+
+For a given code, valuate the code based on correctness, readability:
+Code: {{code}}
+OUTPUT:

examples/flex-flows/eval-code-quality/flow.flex.yaml

@@ -0,0 +1,6 @@
+$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
+# flow is defined as python function
+entry: code_quality:CodeEvaluator
+environment:
+  # image: mcr.microsoft.com/azureml/promptflow/promptflow-python
+  python_requirements_txt: requirements.txt

examples/flex-flows/eval-code-quality/init.json

@@ -0,0 +1,6 @@
+{
+    "model_config": {
+        "connection": "open_ai_connection",
+        "azure_deployment": "gpt-4o"
+    }
+}

examples/flex-flows/eval-code-quality/requirements.txt

@@ -0,0 +1 @@
+promptflow

examples/flex-flows/eval-code-quality/sample.json

@@ -0,0 +1,3 @@
+{
+    "code": "print(\"Hello, world!\")"
+}