wehub/microsoft--promptflow
1
0
Fork 0
Code Issues Pull Requests Actions 11 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
Details
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Details
Publish Promptflow Doc / Build (push) Has been cancelled
Details
Publish Promptflow Doc / Deploy (push) Has been cancelled
Details
Flake8 Lint / flake8 (push) Has been cancelled
Details
Spell check CI / Spell_Check (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:52 +08:00
commit e768098d0e
4004 changed files with 2804145 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/dev/dev_setup.md
+168
View File
@@ -0,0 +1,168 @@
# Dev Setup
## Set up process
Select either Conda or Poetry to set up your development environment.
1. Conda environment setup
- First create a new [conda](https://conda.io/projects/conda/en/latest/user-guide/getting-started.html) environment. Please specify python version as 3.9/3.10/3.11.
`conda create -n <env_name> python=3.9`.
- Activate the env you created.
- In root folder, run `python scripts/dev-setup/main.py` to install the packages and dependencies; if you are using Visual Studio Code, it is recommended to add `--vscode` (which is `python scripts/dev-setup/main.py --vscode`) to enable VS Code to recognize the packages.
2. Poetry environment setup
- Install [poetry](https://python-poetry.org/docs/). Please specify python version as 3.9/3.10/3.11.
- Each folder under [src](../../src/) (except the promptflow folder) is a separate package, so you need to install the dependencies for each package.
- `poetry install -C promptflow-core -E <extra> --with dev,test`
- `poetry install -C promptflow-devkit -E <extra> --with dev,test`
- `poetry install -C promptflow-azure -E <extra> --with dev,test`
## How to run tests
### Set up your secrets
`dev-connections.json.example` is a template about connections provided in `src/promptflow`. You can follow these steps to refer to this template to configure your connection for the test cases:
1. `cd ./src/promptflow`
2. Run the command `cp dev-connections.json.example connections.json`;
3. Replace the values in the json file with your connection info;
4. Set the environment `PROMPTFLOW_CONNECTIONS='connections.json'`;
After above setup process is finished. You can use `pytest` command to run test, for example in root folder you can:
### Run tests via command
1. Conda environment
- Run all tests under a folder: `pytest src/promptflow/tests -v`, `pytest src/promptflow-devkit/tests -v`
- Run a single test: ` pytest src/promptflow/tests/promptflow_test/e2etests/test_executor.py::TestExecutor::test_executor_basic_flow -v`
2. Poetry environment: there is limitation for running tests in src/promptflow folder, you can only run tests under other package folders.
- for example: under the target folder `promptflow-devkit`, you can run `poetry run pytest tests/sdk_cli_test -v`
### Run tests in VSCode
---
#### Conda environment
1. Set up your python interperter
- Open the Command Palette (Ctrl+Shift+P) and select `Python: Select Interpreter`.
![img0](../media/dev_setup/set_up_vscode_0.png)
- Select existing conda env which you created previously.
![img1](../media/dev_setup/set_up_vscode_1.png)
2. Set up your test framework and directory
- Open the Command Palette (Ctrl+Shift+P) and select `Python: Configure Tests`.
![img2](../media/dev_setup/set_up_vscode_2.png)
- Select `pytest` as test framework.
![img3](../media/dev_setup/set_up_vscode_3.png)
- Select `Root directory` as test directory.
![img4](../media/dev_setup/set_up_vscode_4.png)
3. Exclude specific test folders.
You can exclude specific test folders if you don't have some extra dependency to avoid VS Code's test discovery fail.
For example, if you don't have azure dependency, you can exclude `sdk_cli_azure_test`.
Open `.vscode/settings.json`, write `"--ignore=src/promptflow/tests/sdk_cli_azure_test"` to `"python.testing.pytestArgs"`.
![img6](../media/dev_setup/set_up_vscode_6.png)
4. Click the `Run Test` button on the left
![img5](../media/dev_setup/set_up_vscode_5.png)
### Run tests in pycharm
1. Set up your pycharm python interpreter
![img0](../media/dev_setup/set_up_pycharm_0.png)
2. Select existing conda env which you created previously
![img1](../media/dev_setup/set_up_pycharm_1.png)
3. Run test, right-click the test name to run, or click the green arrow button on the left.
![img2](../media/dev_setup/set_up_pycharm_2.png)
---
#### Poetry environment
VSCode could pick up the correct environment automatically if you open vscode/pycharm under the package folders.
There are some limitations currently, intellisense may not work properly in poetry environment.
PyCharm behaves differently from VSCode, it will automatically picks up the correct environment.
## How to write docstring
A clear and consistent API documentation is crucial for the usability and maintainability of our codebase. Please refer to [API Documentation Guidelines](./documentation_guidelines.md) to learn how to write docstring when developing the project.
## How to write tests
- Put all test data/configs under `src/promptflow/tests/test_configs`.
- Write unit tests:
- Flow run: `src/promptflow/tests/sdk_cli_test/unittest/`
- Flow run in azure: `src/promptflow/tests/sdk_cli_azure_test/unittest/`
- Write e2e tests:
- Flow run: `src/promptflow/tests/sdk_cli_test/e2etests/`
- Flow run in azure: `src/promptflow/tests/sdk_cli_azure_test/e2etests/`
- Test file name and the test case name all start with `test_`.
- A basic test example, see [test_connection.py](../../src/promptflow-devkit/tests/sdk_cli_test/e2etests/test_connection.py).
### Test structure
Tests are under corresponding source folder, and test_configs are shared among different test folders:
- src/promptflow/
- test_configs/
- connections/
- datas/
- flows/
- runs/
- wrong_flows/
- wrong_tools/
- src/promptflow-core/
- tests/
- core/ # Basic test with promptflow-core installed.
- e2etests/
- unittests/
- azureml-serving/ # Test with promptflow-core[azureml-serving] installed.
- e2etests/
- unittests/
- executor-service/ # Test with promptflow-core[executor-service] installed.
- e2etests/
- unittests/
- src/promptflow-devkit/
- tests/
- sdk_cli_tests/
- e2etests/
- unittests/
- src/promptflow-azure/
- tests/
- sdk_cli_azure_test/
- e2etests/
- unittests/
Principal #1: Put the tests in the same folder as the code they are testing, to ensure code can work within minor environment requirements.
For example, you write code requires basic `promptflow-core` package, then put the tests in `promptflow-core/tests/core`, DO NOT put it in the promptflow-devkit or promptflow-azure.
Principal #2: Setup separate workflow for tests with extra-requires.
For example, you want to test `promptflow-core[azureml-serving]`, then add a new test folder `promptflow-core/tests/azureml-serving` to test the azure related code,
and add new test steps and environment setup step into `promptflow-core-test.yml` for that folder. DO NOT update the environment of `promptflow-core` basic test directly.
### Record and replay tests
Please refer to [Replay End-to-End Tests](./replay-e2e-test.md) to learn how to record and replay tests.
docs/dev/documentation_guidelines.md
+160
View File
@@ -0,0 +1,160 @@
# Promptflow Reference Documentation Guide
## Overview
This guide describes how to author Python docstrings for promptflow public interfaces. See our doc site at [Promptflow API reference documentation](https://microsoft.github.io/promptflow/reference/python-library-reference/promptflow-tracing/promptflow.html).
## Principles
- **Coverage**: Every public object must have a docstring. For private objects, docstrings are encouraged but not required.
- **Style**: All docstrings should be written in [Sphinx style](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html#the-sphinx-docstring-format) noting all types and if any exceptions are raised.
- **Relevance**: The documentation is up-to-date and relevant to the current version of the product.
- **Clarity**: The documentation is written in clear, concise language that is easy to understand.
- **Consistency**: The documentation has a consistent format and structure, making it easy to navigate and follow.
## How to write the docstring
First please read through [Sphinx style](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html#the-sphinx-docstring-format) to have a basic understanding of sphinx style docstring.
### Write class docstring
Let's start with a class example:
```python
from typing import Dict, Optional, Union
from promptflow.client import PFClient
class MyClass:
"""One-line summary of the class.
More detailed explanation of the class. May include below notes, admonitions, code blocks.
.. note::
Here are some notes to show, with a nested python code block:
.. code-block:: python
from promptflow import MyClass, PFClient
obj = MyClass(PFClient())
.. admonition:: [Title of the admonition]
Here are some admonitions to show.
:param client: Descrition of the client.
:type client: ~promptflow.PFClient
:param param_int: Description of the parameter.
:type param_int: Optional[int]
:param param_str: Description of the parameter.
:type param_str: Optional[str]
:param param_dict: Description of the parameter.
:type param_dict: Optional[Dict[str, str]]
"""
def __init__(
client: PFClient,
param_int: Optional[int] = None,
param_str: Optional[str] = None,
param_dict: Optional[Dict[str, str]] = None,
) -> None:
"""No docstring for __init__, it should be written in class definition above."""
...
```
**Notes**:
1. One-line summary is required. It should be clear and concise.
2. Detailed explanation is encouraged but not required. This part may or may not include notes, admonitions and code blocks.
- The format like `.. note::` is called `directive`. Directives are a mechanism to extend the content of [reStructuredText](https://docutils.sourceforge.io/rst.html). Every directive declares a block of content with specific role. Start a new line with `.. directive_name::` to use the directive.
- The directives used in the sample(`note/admonition/code-block`) should be enough for basic usage of docstring in our project. But you are welcomed to explore more [Directives](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#specific-admonitions).
3. Parameter description and type is required.
- A pair of `:param [ParamName]:` and `:type [ParamName]:` is required.
- If the type is a promptflow public class, use the `full path to the class` and prepend it with a "~". This will create a link when the documentation is rendered on the doc site that will take the user to the class reference documentation for more information.
```text
:param client: Descrition of the client.
:type client: ~promptflow.PFClient
```
- Use `Union/Optional` when appropriate in function declaration. And use the same annotaion after `:type [ParamName]:`
```text
:type param_int: Optional[int]
```
4. For classes, include docstring in definition only. If you include a docstring in both the class definition and the constructor (init method) docstrings, it will show up twice in the reference docs.
5. Constructors (def `__init__`) should return `None`, per [PEP 484 standards](https://peps.python.org/pep-0484/#the-meaning-of-annotations).
6. To create a link for promptflow class on our doc site. `~promptflow.xxx.MyClass` alone only works after `:type [ParamName]` and `:rtype:`. If you want to achieve the same effect in docstring summary, you should use it with `:class:`:
```python
"""
An example to achieve link effect in summary for :class:`~promptflow.xxx.MyClass`
For function, use :meth:`~promptflow.xxx.my_func`
"""
```
7. There are some tricks to highlight the content in your docstring:
- Single backticks (`): Single backticks are used to represent inline code elements within the text. It is typically used to highlight function names, variable names, or any other code elements within the documentation.
- Double backticks(``): Double backticks are typically used to highlight a literal value.
8. If there are any class level constants you don't want to expose to doc site, make sure to add `_` in front of the constant to hide it.
### Write function docstring
```python
from typing import Optional
def my_method(param_int: Optional[int] = None) -> int:
"""One-line summary
Detailed explanations.
:param param_int: Description of the parameter.
:type param_int: int
:raises [ErrorType1]: [ErrorDescription1]
:raises [ErrorType2]: [ErrorDescription2]
:return: Description of the return value.
:rtype: int
"""
...
```
In addition to `class docstring` notes:
1. Function docstring should include return values.
- If return type is promptflow class, we should also use `~promptflow.xxx.[ClassName]`.
2. Function docstring should include exceptions that may be raised in this function.
- If exception type is `PromptflowException`, use `~promptflow.xxx.[ExceptionName]`
- If multiple exceptions are raised, just add new lines of `:raises`, see the example above.
## How to build doc site locally
You can build the documentation site locally to preview the final effect of your docstring on the rendered site. This will provide you with a clear understanding of how your docstring will appear on our site once your changes are merged into the main branch.
1. Setup your dev environment, see [dev_setup](./dev_setup.md) for details. Sphinx will load all source code to process docstring.
- Skip this step if you just want to build the doc site without reference doc, but do remove `-WithReferenceDoc` from the command in step 3.
2. Install `langchain` package since it is used in our code but not covered in `dev_setup`.
3. Open a `powershell`, activate the conda env and navigate to `<repo-root>/scripts/docs` , run `doc_generation.ps1`:
```pwsh
cd scripts\docs
.\doc_generation.ps1 -WithReferenceDoc -WarningAsError
```
- For the first time you execute this command, it will take some time to install `sphinx` dependencies. After the initial installation, next time you can add param `-SkipInstall` to above command to save some time for dependency check.
4. Check warnings/errors in the build log, fix them if any, then build again.
5. Open `scripts/docs/_build/index.html` to preview the local doc site.
## Additional comments
- **Utilities**: The [autoDocstring](https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring) VSCode extension or GitHub Copilot can help autocomplete in this style for you.
- **Advanced principles**
- Accuracy: The documentation accurately reflects the features and functionality of the product.
- Completeness: The documentation covers all relevant features and functionality of the product.
- Demonstration: Every docstring should include an up-to-date code snippet that demonstrates how to use the product effectively.
## References
- [AzureML v2 Reference Documentation Guide](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ml/azure-ai-ml/documentation_guidelines.md)
- [Azure SDK for Python documentation guidelines](https://azure.github.io/azure-sdk/python_documentation.html#docstrings)
- [How to document a Python API](https://review.learn.microsoft.com/en-us/help/onboard/admin/reference/python/documenting-api?branch=main)
docs/dev/replay-e2e-test.md
+58
View File
@@ -0,0 +1,58 @@
# Replay end-to-end tests
* This document introduces replay tests for those located in [sdk_cli_azure_test](../../src/promptflow-azure/tests/sdk_cli_azure_test/e2etests/) and [sdk_cli_test](../../src/promptflow-devkit/tests/sdk_cli_test/e2etests/).
* The primary purpose of replay tests is to avoid the need for credentials, Azure workspaces, OpenAI tokens, and to directly test prompt flow behavior.
* Although there are different techniques behind recording/replaying, there are some common steps to run the tests in replay mode.
* The key handle of replay tests is the environment variable `PROMPT_FLOW_TEST_MODE`.
## How to run tests in replay mode
After cloning the full repo and setting up the proper test environment following [dev_setup.md](./dev_setup.md), run the following command in the root directory of the repo:
1. If you have changed/affected tests in __sdk_cli_test__ : Copy or rename the file [dev-connections.json.example](../../src/promptflow/dev-connections.json.example) to `connections.json` in the same folder.
* There are some python package version requirements for running the replay/record tests. It needs pydantic >= 2.0.0.
2. In your Python environment, set the environment variable `PROMPT_FLOW_TEST_MODE` to `'replay'` and run the test(s).
These tests should work properly without any real connection settings.
## Test modes
There are 3 representative values of the environment variable `PROMPT_FLOW_TEST_MODE`
- `live`: Tests run against the real backend, which is the way traditional end-to-end tests do.
- `record`: Tests run against the real backend, and network traffic will be sanitized (filter sensitive and unnecessary requests/responses) and recorded to local files (recordings).
- `replay`: There is no real network traffic between SDK/CLI and the backend, tests run against local recordings.
## Supported modules
* [promptflow-devkit](../../src/promptflow-devkit)
* [promptflow-azure](../../src/promptflow-azure)
## Update test recordings
To record a test, don’t forget to clone the full repo and set up the proper test environment following [dev_setup.md](./dev_setup.md):
1. Ensure you have installed dev version of promptflow-recording package.
* If it is not installed, run `pip install -e src/promptflow-recording` in the root directory of the repo.
2. Prepare some data.
* If you have changed/affected tests in __sdk_cli_test__: Copy or rename the file [dev-connections.json.example](../../src/promptflow/dev-connections.json.example) to `connections.json` in the same folder.
* If you have changed/affected tests in __sdk_cli_azure_test__: prepare your Azure ML workspace, make sure your Azure CLI logged in, and set the environment variable `PROMPT_FLOW_SUBSCRIPTION_ID`, `PROMPT_FLOW_RESOURCE_GROUP_NAME`, `PROMPT_FLOW_WORKSPACE_NAME` and `PROMPT_FLOW_RUNTIME_NAME` (if needed) pointing to your workspace.
3. Record the test.
* Specify the environment variable `PROMPT_FLOW_TEST_MODE` to `'record'`. If you have a `.env` file, we recommend specifying it there. Here is an example [.env file](../../src/promptflow/.env.example). Then, just run the test that you want to record.
4. Once the test completed.
* If you have changed/affected tests in __sdk_cli_azure_test__: There should be one new YAML file located in [Azure recording folder](../../src/promptflow-recording/recordings/azure/), containing the network traffic of the test.
* If you have changed/affected tests in __sdk_cli_test__: There may be changes in the folder [Local recording folder](../../src/promptflow-recording/recordings/local/). Don’t worry if there are no changes, because similar LLM calls may have been recorded before.
## Techniques behind replay test
### Sdk_cli_azure_test
End-to-end tests for pfazure aim to test the behavior of the PromptFlow SDK/CLI as it interacts with the service. This process can be time-consuming, error-prone, and require credentials (which are unavailable to pull requests from forked repositories); all of these go against our intention for a smooth development experience.
Therefore, we introduce replay tests, which leverage [VCR.py](https://pypi.org/project/vcrpy/) to record all required network traffic to local files and replay during tests. In this way, we avoid the need for credentials, speed up, and stabilize the test process.
### Sdk_cli_test
sdk_cli_test often doesn’t use a real backend. It will directly invokes LLM calls from localhost. Thus the key target of replay tests is to avoid the need for OpenAI tokens. If you have OpenAI / Azure OpenAI tokens yourself, you can try recording the tests. Record Storage will not record your own LLM connection, but only the inputs and outputs of the LLM calls.
There are also limitations. Currently, recorded calls are:
* AzureOpenAI calls
* OpenAI calls
* tool name "fetch_text_content_from_url" and tool name "my_python_tool"
docs/dev/rotate_aoai_keys.md
+8
View File
@@ -0,0 +1,8 @@
Here is a checklist to rotate the AOAI keys:
1. Go to the well known URL of the AOAI service.
2. Check the secondary keys (This is the key used in the following days).
3. Change [promptflow-eastus2euap](https://ml.azure.com/prompts/list?wsid=/subscriptions/96aede12-2f73-41cb-b983-6d11a904839b/resourcegroups/promptflow/providers/Microsoft.MachineLearningServices/workspaces/promptflow-eastus2euap&tid=72f988bf-86f1-41af-91ab-2d7cd011db47#FlowsConnections) Connections
4. Also Change [promptflow-eastus](https://ml.azure.com/prompts/list?wsid=/subscriptions/96aede12-2f73-41cb-b983-6d11a904839b/resourceGroups/promptflow/providers/Microsoft.MachineLearningServices/workspaces/promptflow-eastus&tid=72f988bf-86f1-41af-91ab-2d7cd011db47#FlowsConnections) Connections
5. Save the key in the well known key vault.
6. Save the key in the github secrets, to mask the key.
7. Rotate the DEPRECATED the leaked key in the AOAI service.