wehub/microsoft--promptflow
1
0
Fork 0
Code Issues Pull Requests Actions 11 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
Details
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Details
Publish Promptflow Doc / Build (push) Has been cancelled
Details
Publish Promptflow Doc / Deploy (push) Has been cancelled
Details
Flake8 Lint / flake8 (push) Has been cancelled
Details
Spell check CI / Spell_Check (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:52 +08:00
commit e768098d0e
4004 changed files with 2804145 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/tools-reference/aoai-gpt4-turbo-vision.md
+45
View File
@@ -0,0 +1,45 @@
# Azure OpenAI GPT-4 Turbo with Vision
## Introduction
Azure OpenAI GPT-4 Turbo with Vision tool enables you to leverage your AzureOpenAI GPT-4 Turbo with Vision model deployment to analyze images and provide textual responses to questions about them.
## Prerequisites
- Create AzureOpenAI resources
Create Azure OpenAI resources with [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal)
- Create a GPT-4 Turbo with Vision deployment
Browse to [Azure OpenAI Studio](https://oai.azure.com/) and sign in with the credentials associated with your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.
Under Management select Deployments and Create a GPT-4 Turbo with Vision deployment by selecting model name: `gpt-4` and model version `vision-preview`.
## Connection
Setup connections to provisioned resources in prompt flow.
| Type | Name | API KEY | API Type | API Version |
|-------------|----------|----------|----------|-------------|
| AzureOpenAI | Required | Required | Required | Required |
## Inputs
| Name | Type | Description | Required |
|------------------------|-------------|------------------------------------------------------------------------------------------------|----------|
| connection | AzureOpenAI | the AzureOpenAI connection to be used in the tool | Yes |
| deployment\_name | string | the language model to use | Yes |
| prompt | string | The text prompt that the language model will use to generate it's response. | Yes |
| max\_tokens | integer | the maximum number of tokens to generate in the response. Default is 512. | No |
| temperature | float | the randomness of the generated text. Default is 1. | No |
| stop | list | the stopping sequence for the generated text. Default is null. | No |
| top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No |
| presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No |
| frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0. | No |
| detail | string | configure how the model interprets and processes images, default is "auto". [Read more](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision?tabs=rest%2Csystem-assigned%2Cresource#detail-parameter-settings-in-image-processing-low-high-auto) | No |
## Outputs
| Return Type | Description |
|-------------|------------------------------------------|
| string | The text of one response of conversation |
docs/reference/tools-reference/contentsafety_text_tool.md
+55
View File
@@ -0,0 +1,55 @@
# Content Safety (Text)
Azure Content Safety is a content moderation service developed by Microsoft that help users detect harmful content from different modalities and languages. This tool is a wrapper for the Azure Content Safety Text API, which allows you to detect text content and get moderation results. See the [Azure Content Safety](https://aka.ms/acs-doc) for more information.
## Requirements
- For AzureML users, the tool is installed in default image, you can use the tool without extra installation.
- For local users,
`pip install promptflow-tools`
> [!NOTE]
> Content Safety (Text) tool is now incorporated into the latest `promptflow-tools` package. If you have previously installed the package `promptflow-contentsafety`, please uninstall it to avoid the duplication in your local tool list.
## Prerequisites
- Create an [Azure Content Safety](https://aka.ms/acs-create) resource.
- Add "Azure Content Safety" connection in prompt flow. Fill "API key" field with "Primary key" from "Keys and Endpoint" section of created resource.
## Inputs
You can use the following parameters as inputs for this tool:
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| text | string | The text that need to be moderated. | Yes |
| hate_category | string | The moderation sensitivity for Hate category. You can choose from four options: *disable*, *low_sensitivity*, *medium_sensitivity*, or *high_sensitivity*. The *disable* option means no moderation for hate category. The other three options mean different degrees of strictness in filtering out hate content. The default option is *medium_sensitivity*. | Yes |
| sexual_category | string | The moderation sensitivity for Sexual category. You can choose from four options: *disable*, *low_sensitivity*, *medium_sensitivity*, or *high_sensitivity*. The *disable* option means no moderation for sexual category. The other three options mean different degrees of strictness in filtering out sexual content. The default option is *medium_sensitivity*. | Yes |
| self_harm_category | string | The moderation sensitivity for Self-harm category. You can choose from four options: *disable*, *low_sensitivity*, *medium_sensitivity*, or *high_sensitivity*. The *disable* option means no moderation for self-harm category. The other three options mean different degrees of strictness in filtering out self_harm content. The default option is *medium_sensitivity*. | Yes |
| violence_category | string | The moderation sensitivity for Violence category. You can choose from four options: *disable*, *low_sensitivity*, *medium_sensitivity*, or *high_sensitivity*. The *disable* option means no moderation for violence category. The other three options mean different degrees of strictness in filtering out violence content. The default option is *medium_sensitivity*. | Yes |
For more information, please refer to [Azure Content Safety](https://aka.ms/acs-doc)
## Outputs
The following is an example JSON format response returned by the tool:
<details>
<summary>Output</summary>
```json
{
"action_by_category": {
"Hate": "Accept",
"SelfHarm": "Accept",
"Sexual": "Accept",
"Violence": "Accept"
},
"suggested_action": "Accept"
}
```
</details>
The `action_by_category` field gives you a binary value for each category: *Accept* or *Reject*. This value shows if the text meets the sensitivity level that you set in the request parameters for that category.
The `suggested_action` field gives you an overall recommendation based on the four categories. If any category has a *Reject* value, the `suggested_action` will be *Reject* as well.
docs/reference/tools-reference/embedding_tool.md
+62
View File
@@ -0,0 +1,62 @@
# Embedding
## Introduction
OpenAI's embedding models convert text into dense vector representations for various NLP tasks. See the [OpenAI Embeddings API](https://platform.openai.com/docs/api-reference/embeddings) for more information.
## Prerequisite
Create OpenAI resources:
- **OpenAI**
Sign up account [OpenAI website](https://openai.com/)
Login and [Find personal API key](https://platform.openai.com/account/api-keys)
- **Azure OpenAI (AOAI)**
Create Azure OpenAI resources with [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal)
## **Connections**
Setup connections to provide resources in embedding tool.
| Type | Name | API KEY | API Type | API Version |
|-------------|----------|----------|----------|-------------|
| OpenAI | Required | Required | - | - |
| AzureOpenAI | Required | Required | Required | Required |
## Inputs
| Name | Type | Description | Required |
|------------------------|-------------|-----------------------------------------------------------------------|----------|
| input | string | the input text to embed | Yes |
| connection | string | the connection for the embedding tool use to provide resources | Yes |
| model/deployment_name | string | instance of the text-embedding engine to use. Fill in model name if you use OpenAI connection, or deployment name if use Azure OpenAI connection. | Yes |
## Outputs
| Return Type | Description |
|-------------|------------------------------------------|
| list | The vector representations for inputs |
The following is an example response returned by the embedding tool:
<details>
<summary>Output</summary>
```
[-0.005744616035372019,
-0.007096089422702789,
-0.00563855143263936,
-0.005272455979138613,
-0.02355326898396015,
0.03955197334289551,
-0.014260607771575451,
-0.011810848489403725,
-0.023170066997408867,
-0.014739611186087132,
...]
```
</details>
docs/reference/tools-reference/faiss_index_lookup_tool.md
+87
View File
@@ -0,0 +1,87 @@
# Faiss Index Lookup
Faiss Index Lookup is a tool tailored for querying within a user-provided Faiss-based vector store. In combination with our Large Language Model (LLM) tool, it empowers users to extract contextually relevant information from a domain knowledge base.
## Requirements
- For AzureML users, the tool is installed in default image, you can use the tool without extra installation.
- For local users, if your index is stored in local path,
`pip install promptflow-vectordb`
if your index is stored in Azure storage,
`pip install promptflow-vectordb[azure]`
## Prerequisites
### For AzureML users,
- step 1. Prepare an accessible path on Azure Blob Storage. Here's the guide if a new storage account needs to be created: [Azure Storage Account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal).
- step 2. Create related Faiss-based index files on Azure Blob Storage. We support the LangChain format (index.faiss + index.pkl) for the index files, which can be prepared either by employing our promptflow-vectordb SDK or following the quick guide from [LangChain documentation](https://python.langchain.com/docs/integrations/vectorstores/faiss). Please refer to the instructions of <a href="https://aka.ms/pf-sample-build-faiss-index" target="_blank">An example code for creating Faiss index</a> for building index using promptflow-vectordb SDK.
- step 3. Based on where you put your own index files, the identity used by the promptflow runtime should be granted with certain roles. Please refer to [Steps to assign an Azure role](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-steps):
| Location | Role |
| ---- | ---- |
| workspace datastores or workspace default blob | AzureML Data Scientist |
| other blobs | Storage Blob Data Reader |
### For local users,
- Create Faiss-based index files in local path by only doing step 2 above.
## Inputs
The tool accepts the following inputs:
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| path | string | URL or path for the vector store.<br><br>local path (for local users):<br>`<local_path_to_the_index_folder>`<br><br> Azure blob URL format (with [azure] extra installed):<br>https://`<account_name>`.blob.core.windows.net/`<container_name>`/`<path_and_folder_name>`.<br><br>AML datastore URL format (with [azure] extra installed):<br>azureml://subscriptions/`<your_subscription>`/resourcegroups/`<your_resource_group>`/workspaces/`<your_workspace>`/data/`<data_path>`<br><br>public http/https URL (for public demonstration):<br>http(s)://`<path_and_folder_name>` | Yes |
| vector | list[float] | The target vector to be queried, which can be generated by the LLM tool. | Yes |
| top_k | integer | The count of top-scored entities to return. Default value is 3. | No |
## Outputs
The following is an example for JSON format response returned by the tool, which includes the top-k scored entities. The entity follows a generic schema of vector search result provided by our promptflow-vectordb SDK. For the Faiss Index Search, the following fields are populated:
| Field Name | Type | Description |
| ---- | ---- | ----------- |
| text | string | Text of the entity |
| score | float | Distance between the entity and the query vector |
| metadata | dict | Customized key-value pairs provided by user when create the index |
<details>
<summary>Output</summary>
```json
[
{
"metadata": {
"link": "http://sample_link_0",
"title": "title0"
},
"original_entity": null,
"score": 0,
"text": "sample text #0",
"vector": null
},
{
"metadata": {
"link": "http://sample_link_1",
"title": "title1"
},
"original_entity": null,
"score": 0.05000000447034836,
"text": "sample text #1",
"vector": null
},
{
"metadata": {
"link": "http://sample_link_2",
"title": "title2"
},
"original_entity": null,
"score": 0.20000001788139343,
"text": "sample text #2",
"vector": null
}
]
```
</details>
docs/reference/tools-reference/llm-tool.md
+225
View File
@@ -0,0 +1,225 @@
# LLM
## Introduction
Prompt flow LLM tool enables you to leverage widely used large language models like [OpenAI](https://platform.openai.com/), [Azure OpenAI (AOAI)](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/overview), and models in [Azure AI Studio model catalog](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/model-catalog) for natural language processing.
> [!NOTE]
> The previous version of the LLM tool is now being deprecated. Please upgrade to latest [promptflow-tools](https://pypi.org/project/promptflow-tools/) package to consume new llm tools.
Prompt flow provides a few different LLM APIs:
- **[Completion](https://platform.openai.com/docs/api-reference/completions)**: OpenAI's completion models generate text based on provided prompts.
- **[Chat](https://platform.openai.com/docs/api-reference/chat)**: OpenAI's chat models facilitate interactive conversations with text-based inputs and responses.
## Prerequisite
Create OpenAI resources, Azure OpenAI resources or MaaS deployment with the LLM models (e.g.: llama2, mistral, cohere etc.) in Azure AI Studio model catalog:
- **OpenAI**
Sign up account [OpenAI website](https://openai.com/)
Login and [Find personal API key](https://platform.openai.com/account/api-keys)
- **Azure OpenAI (AOAI)**
Create Azure OpenAI resources with [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal)
- **MaaS deployment**
Create MaaS deployment for models in Azure AI Studio model catalog with [instruction](https://learn.microsoft.com/azure/ai-studio/concepts/deployments-overview)
You can create serverless connection to use this MaaS deployment.
## **Connections**
Setup connections to provisioned resources in prompt flow.
| Type | Name | API KEY | API BASE | API Type | API Version |
|-------------|----------|----------|----------|-----------|-------------|
| OpenAI | Required | Required | - | - | - |
| AzureOpenAI | Required | Required | Required | Required | Required |
| Serverless | Required | Required | Required | - | - |
## Inputs
### Text Completion
| Name | Type | Description | Required |
|------------------------|-------------|-----------------------------------------------------------------------------------------|----------|
| prompt | string | text prompt that the language model will complete | Yes |
| model, deployment_name | string | the language model to use | Yes |
| max\_tokens | integer | the maximum number of tokens to generate in the completion. Default is 16. | No |
| temperature | float | the randomness of the generated text. Default is 1. | No |
| stop | list | the stopping sequence for the generated text. Default is null. | No |
| suffix | string | text appended to the end of the completion | No |
| top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No |
| logprobs | integer | the number of log probabilities to generate. Default is null. | No |
| echo | boolean | value that indicates whether to echo back the prompt in the response. Default is false. | No |
| presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No |
| frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0. | No |
| best\_of | integer | the number of best completions to generate. Default is 1. | No |
| logit\_bias | dictionary | the logit bias for the language model. Default is empty dictionary. | No |
### Chat
| Name | Type | Description | Required |
|------------------------|-------------|------------------------------------------------------------------------------------------------|----------|
| prompt | string | text prompt that the language model will response | Yes |
| model, deployment_name | string | the language model to use | Yes |
| max\_tokens | integer | the maximum number of tokens to generate in the response. Default is inf. | No |
| temperature | float | the randomness of the generated text. Default is 1. | No |
| stop | list | the stopping sequence for the generated text. Default is null. | No |
| top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No |
| presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No |
| frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0.| No |
| logit\_bias | dictionary | the logit bias for the language model. Default is empty dictionary. | No |
| tool\_choice | object | value that controls which tool is called by the model. Default is null. | No |
| tools | list | a list of tools the model may generate JSON inputs for. Default is null. | No |
| response_format | object | an object specifying the format that the model must output. Default is null. | No |
## Outputs
| Return Type | Description |
|-------------|----------------------------------------------------------------------|
| string | The text of one predicted completion or response of conversation |
## How to use LLM Tool?
1. Setup and select the connections to OpenAI resources
2. Configure LLM model api and its parameters
3. Prepare the Prompt with [guidance](./prompt-tool.md#how-to-write-prompt).
## How to write a chat prompt?
_To grasp the fundamentals of creating a chat prompt, begin with [this section](./prompt-tool.md#how-to-write-prompt) for an introductory understanding of jinja._
We offer a method to distinguish between different roles in a chat prompt, such as "system", "user", "assistant" and "tool". The "system", "user", "assistant" roles can have "name" and "content" properties. The "tool" role, however, should have "tool_call_id" and "content" properties. For an example of a tool chat prompt, please refer to [Sample 3](#sample-3).
### Sample 1
```jinja
# system:
You are a helpful assistant.
{% for item in chat_history %}
# user:
{{item.inputs.question}}
# assistant:
{{item.outputs.answer}}
{% endfor %}
# user:
{{question}}
```
In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API.
```
[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "<question-of-chat-history-round-1>"
},
{
"role": "assistant",
"content": "<answer-of-chat-history-round-1>"
},
...
{
"role": "user",
"content": "<question>"
}
]
```
### Sample 2
```jinja
# system:
{# For role naming customization, the following syntax is used #}
## name:
Alice
## content:
You are a bot can tell good jokes.
```
In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API.
```
[
{
"role": "system",
"name": "Alice",
"content": "You are a bot can tell good jokes."
}
]
```
### Sample 3
This sample illustrates how to write a tool chat prompt.
```jinja
# system:
You are a helpful assistant.
# user:
What is the current weather like in Boston?
# assistant:
{# The assistant message with 'tool_calls' must be followed by messages with role 'tool'. #}
## tool_calls:
{{llm_output.tool_calls}}
# tool:
{#
Messages with role 'tool' must be a response to a preceding message with 'tool_calls'.
Additionally, 'tool_call_id's should match ids of assistant message 'tool_calls'.
#}
## tool_call_id:
{{llm_output.tool_calls[0].id}}
## content:
{{tool-answer-of-last-question}}
# user:
{{question}}
```
In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API.
```
[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is the current weather like in Boston?"
},
{
"role": "assistant",
"content": null,
"function_call": null,
"tool_calls": [
{
"id": "<tool-call-id-of-last-question>",
"type": "function",
"function": "<function-to-call-of-last-question>"
}
]
},
{
"role": "tool",
"tool_call_id": "<tool-call-id-of-last-question>",
"content": "<tool-answer-of-last-question>"
}
...
{
"role": "user",
"content": "<question>"
}
]
```
docs/reference/tools-reference/llm-vision-tool.md
+51
View File
@@ -0,0 +1,51 @@
# LLM Vision
## Introduction
Prompt flow LLM vision tool enables you to leverage your AzureOpenAI GPT-4 Turbo or OpenAI's GPT-4 with vision to analyze images and provide textual responses to questions about them.
## Prerequisites
Create OpenAI or Azure OpenAI resources:
- **OpenAI**
Sign up account [OpenAI website](https://openai.com/)
Login and [Find personal API key](https://platform.openai.com/account/api-keys)
- **Azure OpenAI (AOAI)**
Create Azure OpenAI resources with [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal)
Browse to [Azure OpenAI Studio](https://oai.azure.com/) and sign in with the credentials associated with your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.
Under Management select Deployments and Create a GPT-4 Turbo with Vision deployment by selecting model name: `gpt-4` and model version `vision-preview`.
## **Connections**
Setup connections to provisioned resources in prompt flow.
| Type | Name | API KEY | API Type | API Version |
|-------------|----------|----------|----------|-------------|
| OpenAI | Required | Required | - | - |
| AzureOpenAI | Required | Required | Required | Required |
## Inputs
| Name | Type | Description | Required |
|-------------------------|-------------|------------------------------------------------------------------------------------------------|----------|
| model, deployment\_name | string | the language model to use | Yes |
| prompt | string | The text prompt that the language model will use to generate it's response. | Yes |
| max\_tokens | integer | the maximum number of tokens to generate in the response. Default is 512. | No |
| temperature | float | the randomness of the generated text. Default is 1. | No |
| stop | list | the stopping sequence for the generated text. Default is null. | No |
| top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No |
| presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No |
| frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0.| No |
## Outputs
| Return Type | Description |
|-------------|------------------------------------------|
| string | The text of one response of conversation |
docs/reference/tools-reference/open_model_llm_tool.md
+89
View File
@@ -0,0 +1,89 @@
# Open Model LLM
## Introduction
The Open Model LLM tool enables the utilization of a variety of Open Model and Foundational Models, such as [Falcon](https://ml.azure.com/models/tiiuae-falcon-7b/version/4/catalog/registry/azureml) and [Llama 2](https://ml.azure.com/models/Llama-2-7b-chat/version/14/catalog/registry/azureml-meta), for natural language processing in Azure ML Prompt Flow.
Here's how it looks in action on the Visual Studio Code prompt flow extension. In this example, the tool is being used to call a LlaMa-2 chat endpoint and asking "What is CI?".
![Screenshot of the Open Model LLM On VScode Prompt Flow extension](../../media/reference/tools-reference/open_model_llm_on_vscode_promptflow.png)
This prompt flow tool supports two different LLM API types:
- **Chat**: Shown in the example above. The chat API type facilitates interactive conversations with text-based inputs and responses.
- **Completion**: The Completion API type is used to generate single response text completions based on provided prompt input.
## Quick Overview: How do I use Open Model LLM Tool?
1. Choose a Model from the AzureML Model Catalog and get it deployed.
2. Connect to the model deployment.
3. Configure the open model llm tool settings.
4. Prepare the Prompt with [guidance](./prompt-tool.md#how-to-write-prompt).
5. Run the flow.
## Prerequisites: Model Deployment
1. Pick the model which matched your scenario from the [Azure Machine Learning model catalog](https://ml.azure.com/model/catalog).
2. Use the "Deploy" button to deploy the model to a AzureML Online Inference endpoint.
2.1. Use one of the Pay as you go deployment options.
More detailed instructions can be found here [Deploying foundation models to endpoints for inferencing.](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-foundation-models?view=azureml-api-2#deploying-foundation-models-to-endpoints-for-inferencing)
## Prerequisites: Connect to the Model
In order for prompt flow to use your deployed model, you will need to connect to it. There are several ways to connect.
### 1. Endpoint Connections
Once associated to a AzureML or Azure AI Studio workspace, the Open Model LLM tool can use the endpoints on that workspace.
1. **Using AzureML or Azure AI Studio workspaces**: If you are using prompt flow in one of the web page based browsers workspaces, the online endpoints available on that workspace will automatically who up.
2. **Using VScode or Code First**: If you are using prompt flow in VScode or one of the Code First offerings, you will need to connect to the workspace. The Open Model LLM tool uses the azure.identity DefaultAzureCredential client for authorization. One way is through [setting environment credential values](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python).
### 2. Custom Connections
The Open Model LLM tool uses the CustomConnection. Prompt flow supports two types of connections:
1. **Workspace Connections** - These are connections which are stored as secrets on an Azure Machine Learning workspace. While these can be used, in many places, the are commonly created and maintained in the Studio UI.
2. **Local Connections** - These are connections which are stored locally on your machine. These connections are not available in the Studio UX's, but can be used with the VScode extension.
Instructions on how to create a workspace or local Custom Connection [can be found here.](../../how-to-guides/manage-connections.md#create-a-connection)
The required keys to set are:
1. **endpoint_url**
- This value can be found at the previously created Inferencing endpoint.
2. **endpoint_api_key**
- Ensure to set this as a secret value.
- This value can be found at the previously created Inferencing endpoint.
3. **model_family**
- Supported values: LLAMA, DOLLY, GPT2, or FALCON
- This value is dependent on the type of deployment you are targeting.
## Running the Tool: Inputs
The Open Model LLM tool has a number of parameters, some of which are required. Please see the below table for details, you can match these to the screen shot above for visual clarity.
| Name | Type | Description | Required |
|------|------|-------------|----------|
| api | string | This is the API mode and will depend on the model used and the scenario selected. *Supported values: (Completion \| Chat)* | Yes |
| endpoint_name | string | Name of an Online Inferencing Endpoint with a supported model deployed on it. Takes priority over connection. | No |
| temperature | float | The randomness of the generated text. Default is 1. | No |
| max_new_tokens | integer | The maximum number of tokens to generate in the completion. Default is 500. | No |
| top_p | float | The probability of using the top choice from the generated tokens. Default is 1. | No |
| model_kwargs | dictionary | This input is used to provide configuration specific to the model used. For example, the Llama-02 model may use {\"temperature\":0.4}. *Default: {}* | No |
| deployment_name | string | The name of the deployment to target on the Online Inferencing endpoint. If no value is passed, the Inferencing load balancer traffic settings will be used. | No |
| prompt | string | The text prompt that the language model will use to generate it's response. | Yes |
## Outputs
| API | Return Type | Description |
|------------|-------------|------------------------------------------|
| Completion | string | The text of one predicted completion |
| Chat | string | The text of one response int the conversation |
## Deploying to an Online Endpoint
When deploying a flow containing the Open Model LLM tool to an online endpoint, there is an additional step to setup permissions. During deployment through the web pages, there is a choice between System-assigned and User-assigned Identity types. Either way, using the Azure Portal (or a similar functionality), add the "Reader" Job function role to the identity on the Azure Machine Learning workspace or Ai Studio project which is hosting the endpoint. The prompt flow deployment may need to be refreshed.
docs/reference/tools-reference/openai-gpt-4v-tool.md
+44
View File
@@ -0,0 +1,44 @@
# OpenAI GPT-4V
## Introduction
OpenAI GPT-4V tool enables you to leverage OpenAI's GPT-4 with vision, also referred to as GPT-4V or gpt-4-vision-preview in the API, to take images as input and answer questions about them.
## Prerequisites
- Create OpenAI resources
Sign up account [OpenAI website](https://openai.com/)
Login and [Find personal API key](https://platform.openai.com/account/api-keys)
- Get Access to GPT-4 API
To use GPT-4 with vision, you need access to GPT-4 API. Learn more about [How to get access to GPT-4 API](https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4)
## Connection
Setup connections to provisioned resources in prompt flow.
| Type | Name | API KEY |
|-------------|----------|----------|
| OpenAI | Required | Required |
## Inputs
| Name | Type | Description | Required |
|------------------------|-------------|------------------------------------------------------------------------------------------------|----------|
| connection | OpenAI | the OpenAI connection to be used in the tool | Yes |
| model | string | the language model to use, currently only support gpt-4-vision-preview | Yes |
| prompt | string | The text prompt that the language model will use to generate it's response. | Yes |
| max\_tokens | integer | the maximum number of tokens to generate in the response. Default is 512. | No |
| temperature | float | the randomness of the generated text. Default is 1. | No |
| stop | list | the stopping sequence for the generated text. Default is null. | No |
| top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No |
| presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No |
| frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0. | No |
| detail | string | control over how the model processes the image and generates its textual understanding, default is "auto". [Read more](https://platform.openai.com/docs/guides/vision/low-or-high-fidelity-image-understanding) | No |
## Outputs
| Return Type | Description |
|-------------|------------------------------------------|
| string | The text of one response of conversation |
docs/reference/tools-reference/prompt-tool.md
+73
View File
@@ -0,0 +1,73 @@
# Prompt
## Introduction
The Prompt Tool in PromptFlow offers a collection of textual templates that serve as a starting point for creating prompts.
These templates, based on the Jinja2 template engine, facilitate the definition of prompts. The tool proves useful
when prompt tuning is required prior to feeding the prompts into the Language Model (LLM) model in PromptFlow.
## Inputs
| Name | Type | Description | Required |
|--------------------|--------|----------------------------------------------------------|----------|
| prompt | string | The prompt template in Jinja | Yes |
| Inputs | - | List of variables of prompt template and its assignments | - |
## Outputs
The prompt text parsed from the prompt + Inputs
## How to write Prompt?
1. Prepare jinja template. Learn more about [Jinja](https://jinja.palletsprojects.com/en/3.1.x/)
_In below example, the prompt incorporates Jinja templating syntax to dynamically generate the welcome message and personalize it based on the user's name. It also presents a menu of options for the user to choose from. Depending on whether the user_name variable is provided, it either addresses the user by name or uses a generic greeting._
```jinja
Welcome to {{ website_name }}!
{% if user_name %}
Hello, {{ user_name }}!
{% else %}
Hello there!
{% endif %}
Please select an option from the menu below:
1. View your account
2. Update personal information
3. Browse available products
4. Contact customer support
```
2. Assign value for the variables.
_In above example, two variables would be automatically detected and listed in '**Inputs**' section. Please assign values._
### Sample 1
Inputs
| Variable | Type | Sample Value |
|---------------|--------|--------------|
| website_name | string | "Microsoft" |
| user_name | string | "Jane" |
Outputs
```
Welcome to Microsoft! Hello, Jane! Please select an option from the menu below: 1. View your account 2. Update personal information 3. Browse available products 4. Contact customer support
```
### Sample 2
Inputs
| Variable | Type | Sample Value |
|--------------|--------|----------------|
| website_name | string | "Bing" |
| user_name | string | " |
Outputs
```
Welcome to Bing! Hello there! Please select an option from the menu below: 1. View your account 2. Update personal information 3. Browse available products 4. Contact customer support
```
docs/reference/tools-reference/python-tool.md
+112
View File
@@ -0,0 +1,112 @@
# Python
## Introduction
Users are empowered by the Python Tool to offer customized code snippets as self-contained executable nodes in PromptFlow.
Users can effortlessly create Python tools, edit code, and verify results with ease.
## Inputs
| Name | Type | Description | Required |
|--------|--------|------------------------------------------------------|---------|
| Code | string | Python code snippet | Yes |
| Inputs | - | List of tool function parameters and its assignments | - |
### Types
| Type | Python example | Description |
|-----------------------------------------------------|---------------------------------|--------------------------------------------|
| int | param: int | Integer type |
| bool | param: bool | Boolean type |
| string | param: str | String type |
| double | param: float | Double type |
| list | param: list or param: List[T] | List type |
| object | param: dict or param: Dict[K, V] | Object type |
| [Connection](../../concepts/concept-connections.md) | param: CustomConnection | Connection type, will be handled specially |
Parameters with `Connection` type annotation will be treated as connection inputs, which means:
- Promptflow extension will show a selector to select the connection.
- During execution time, promptflow will try to find the connection with the name same from parameter value passed in.
Note that `Union[...]` type annotation is supported **ONLY** for connection type,
for example, `param: Union[CustomConnection, OpenAIConnection]`.
## Outputs
The return of the python tool function.
## How to write Python Tool?
### Guidelines
1. Python Tool Code should consist of a complete Python code, including any necessary module imports.
2. Python Tool Code must contain a function decorated with @tool (tool function), serving as the entry point for execution. The @tool decorator should be applied only once within the snippet.
_Below sample defines python tool "my_python_tool", decorated with @tool_
3. Python tool function parameters must be assigned in 'Inputs' section
_Below sample defines inputs "message" and assign with "world"_
4. Python tool function shall have return
_Below sample returns a concatenated string_
### Code
The snippet below shows the basic structure of a tool function. Promptflow will read the function and extract inputs
from function parameters and type annotations.
```python
from promptflow.core import tool
from promptflow.connections import CustomConnection
# The inputs section will change based on the arguments of the tool function, after you save the code
# Adding type to arguments and return value will help the system show the types properly
# Please update the function name/signature per need
@tool
def my_python_tool(message: str, my_conn: CustomConnection) -> str:
my_conn_dict = dict(my_conn)
# Do some function call with my_conn_dict...
return 'hello ' + message
```
### Inputs
| Name | Type | Sample Value in Flow Yaml | Value passed to function|
|---------|--------|-------------------------| ------------------------|
| message | string | "world" | "world" |
| my_conn | CustomConnection | "my_conn" | CustomConnection object |
Promptflow will try to find the connection named 'my_conn' during execution time.
### outputs
```python
"hello world"
```
### Keyword Arguments Support
Starting from version 1.0.0 of PromptFlow and version 1.4.0 of [Prompt flow for VS Code](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow),
we have introduced support for keyword arguments (kwargs) in the Python tool.
```python
from promptflow.core import tool
@tool
def print_test(normal_input: str, **kwargs):
for key, value in kwargs.items():
print(f"Key {key}'s value is {value}")
return len(kwargs)
```
When you add `kwargs` in your python tool like above code, you can insert variable number of inputs by the `+Add input` button.
![Screenshot of the kwargs On VScode Prompt Flow extension](../../media/reference/tools-reference/python_tool_kwargs.png)
docs/reference/tools-reference/rerank-tool.md
+69
View File
@@ -0,0 +1,69 @@
# Rerank
## Introduction
Rerank is a semantic search tool that improves search quality with a semantic-based reranking system which can contextualize the meaning of a user's query beyond keyword relevance. This tool works best with look up tool as a ranker after the initial retrieval. The list of current supported ranking method is as follows.
| Name | Description |
| --- | --- |
| BM25 | BM25 is an open source ranking algorithm to measure the relevance of documents to a given query |
| Scaled Score Fusion | Scaled Score Fusion calculates a scaled relevance score. |
| Cohere Rerank | Cohere Rerank is the market’s leading reranking model used for semantic search and retrieval-augmented generation (RAG). |
## Requirements
- For AzureML users, the tool is installed in default image, you can use the tool without extra installation.
- For local users,
`pip install promptflow-vectordb`
## Prerequisites
BM25 and Scaled Score Fusion are included as default reranking methods. To use cohere rerank model, you should create serverless deployment to the model, and establish connection between the tool and the resource as follows.
- Add Serverless Model connection. Fill "API base" and "API key" field to your serverless deployment.
## Inputs
| Name | Type | Description | Required |
|------------------------|-------------|-----------------------------------------------------------------------|----------|
| query | string | the question relevant to your input documents | Yes |
| ranker_parameters | string | the type of ranking methods to use | Yes |
| result_groups | object | the list of document chunks to rerank. Normally this is output from lookup | Yes |
| top_k | int | the maximum number of relevant documents to return | No |
## Outputs
| Return Type | Description |
|-------------|------------------------------------------|
| text | text of the entity |
| metadata | metadata like file path and url |
| additional_fields | metadata and rerank score |
<details>
<summary>Output</summary>
```json
[
{
"text": "sample text",
"metadata":
{
"filepath": "sample_file_path",
"metadata_json_string": "meta_json_string"
"title": "",
"url": ""
},
"additional_fields":
{
"filepath": "sample_file_path",
"metadata_json_string": "meta_json_string"
"title": "",
"url": "",
"@promptflow_vectordb.reranker_score": 0.013795365
}
}
]
```
</details>
docs/reference/tools-reference/serp-api-tool.md
+45
View File
@@ -0,0 +1,45 @@
# SerpAPI
## Introduction
The SerpAPI API is a Python tool that provides a wrapper to the [SerpAPI Google Search Engine Results API](https://serpapi.com/search-api) and [SerpApi Bing Search Engine Results API
](https://serpapi.com/bing-search-api).
We could use the tool to retrieve search results from a number of different search engines, including Google and Bing, and you can specify a range of search parameters, such as the search query, location, device type, and more.
## Prerequisite
Sign up at [SERP API homepage](https://serpapi.com/)
## Connection
Connection is the model used to establish connections with Serp API.
| Type | Name | API KEY |
|-------------|----------|----------|
| Serp | Required | Required |
_**API Key** is on SerpAPI account dashboard_
## Inputs
The **serp api** tool supports following parameters:
| Name | Type | Description | Required |
|----------|---------|---------------------------------------------------------------|----------|
| query | string | The search query to be executed. | Yes |
| engine | string | The search engine to use for the search. Default is 'google'. | Yes |
| num | integer | The number of search results to return.Default is 10. | No |
| location | string | The geographic location to execute the search from. | No |
| safe | string | The safe search mode to use for the search. Default is 'off'. | No |
## Outputs
The json representation from serpapi query.
| Engine | Return Type | Output |
|----------|-------------|-------------------------------------------------------|
| google | json | [Sample](https://serpapi.com/search-api#api-examples) |
| bing | json | [Sample](https://serpapi.com/bing-search-api) |
docs/reference/tools-reference/vector_db_lookup_tool.md
+187
View File
@@ -0,0 +1,187 @@
# Vector DB Lookup
Vector DB Lookup is a vector search tool that allows users to search top k similar vectors from vector database. This tool is a wrapper for multiple third-party vector databases. The list of current supported databases is as follows.
| Name | Description |
| --- | --- |
| Azure Cognitive Search | Microsoft's cloud search service with built-in AI capabilities that enrich all types of information to help identify and explore relevant content at scale. |
| Qdrant | Qdrant is a vector similarity search engine that provides a production-ready service with a convenient API to store, search and manage points (i.e. vectors) with an additional payload. |
| Weaviate | Weaviate is an open source vector database that stores both objects and vectors. This allows for combining vector search with structured filtering. |
This tool will support more vector databases.
## Requirements
- For AzureML users, the tool is installed in default image, you can use the tool without extra installation.
- For local users,
`pip install promptflow-vectordb`
## Prerequisites
The tool searches data from a third-party vector database. To use it, you should create resources in advance and establish connection between the tool and the resource.
- **Azure Cognitive Search:**
- Create resource [Azure Cognitive Search](https://learn.microsoft.com/en-us/azure/search/search-create-service-portal).
- Add "Cognitive search" connection. Fill "API key" field with "Primary admin key" from "Keys" section of created resource, and fill "API base" field with the URL, the URL format is `https://{your_serive_name}.search.windows.net`.
- **Qdrant:**
- Follow the [installation](https://qdrant.tech/documentation/quick-start/) to deploy Qdrant to a self-maintained cloud server.
- Add "Qdrant" connection. Fill "API base" with your self-maintained cloud server address and fill "API key" field.
- **Weaviate:**
- Follow the [installation](https://weaviate.io/developers/weaviate/installation) to deploy Weaviate to a self-maintained instance.
- Add "Weaviate" connection. Fill "API base" with your self-maintained instance address and fill "API key" field.
## Inputs
The tool accepts the following inputs:
- **Azure Cognitive Search:**
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| connection | CognitiveSearchConnection | The created connection for accessing to Cognitive Search endpoint. | Yes |
| index_name | string | The index name created in Cognitive Search resource. | Yes |
| text_field | string | The text field name. The returned text field will populate the text of output. | No |
| vector_field | string | The vector field name. The target vector is searched in this vector field. | Yes |
| search_params | dict | The search parameters. It's key-value pairs. Except for parameters in the tool input list mentioned above, additional search parameters can be formed into a JSON object as search_params. For example, use `{"select": ""}` as search_params to select the returned fields, use `{"search": ""}` to perform a [hybrid search](https://learn.microsoft.com/en-us/azure/search/search-get-started-vector#hybrid-search). | No |
| search_filters | dict | The search filters. It's key-value pairs, the input format is like `{"filter": ""}` | No |
| vector | list | The target vector to be queried, which can be generated by Embedding tool. | Yes |
| top_k | int | The count of top-scored entities to return. Default value is 3 | No |
- **Qdrant:**
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| connection | QdrantConnection | The created connection for accessing to Qdrant server. | Yes |
| collection_name | string | The collection name created in self-maintained cloud server. | Yes |
| text_field | string | The text field name. The returned text field will populate the text of output. | No |
| search_params | dict | The search parameters can be formed into a JSON object as search_params. For example, use `{"params": {"hnsw_ef": 0, "exact": false, "quantization": null}}` to set search_params. | No |
| search_filters | dict | The search filters. It's key-value pairs, the input format is like `{"filter": {"should": [{"key": "", "match": {"value": ""}}]}}` | No |
| vector | list | The target vector to be queried, which can be generated by Embedding tool. | Yes |
| top_k | int | The count of top-scored entities to return. Default value is 3 | No |
- **Weaviate:**
| Name | Type | Description | Required |
| ---- | ---- | ----------- | -------- |
| connection | WeaviateConnection | The created connection for accessing to Weaviate. | Yes |
| class_name | string | The class name. | Yes |
| text_field | string | The text field name. The returned text field will populate the text of output. | No |
| vector | list | The target vector to be queried, which can be generated by Embedding tool. | Yes |
| top_k | int | The count of top-scored entities to return. Default value is 3 | No |
## Outputs
The following is an example JSON format response returned by the tool, which includes the top-k scored entities. The entity follows a generic schema of vector search result provided by promptflow-vectordb SDK.
- **Azure Cognitive Search:**
For Azure Cognitive Search, the following fields are populated:
| Field Name | Type | Description |
| ---- | ---- | ----------- |
| original_entity | dict | the original response json from search REST API|
| score | float | @search.score from the original entity, which evaluates the similarity between the entity and the query vector |
| text | string | text of the entity|
| vector | list | vector of the entity|
<details>
<summary>Output</summary>
```json
[
{
"metadata": null,
"original_entity": {
"@search.score": 0.5099789,
"id": "",
"your_text_filed_name": "sample text1",
"your_vector_filed_name": [-0.40517663431890405, 0.5856996257406859, -0.1593078462266455, -0.9776269170785785, -0.6145604369828972],
"your_additional_field_name": ""
},
"score": 0.5099789,
"text": "sample text1",
"vector": [-0.40517663431890405, 0.5856996257406859, -0.1593078462266455, -0.9776269170785785, -0.6145604369828972]
}
]
```
</details>
- **Qdrant:**
For Qdrant, the following fields are populated:
| Field Name | Type | Description |
| ---- | ---- | ----------- |
| original_entity | dict | the original response json from search REST API|
| metadata | dict | payload from the original entity|
| score | float | score from the original entity, which evaluates the similarity between the entity and the query vector|
| text | string | text of the payload|
| vector | list | vector of the entity|
<details>
<summary>Output</summary>
```json
[
{
"metadata": {
"text": "sample text1"
},
"original_entity": {
"id": 1,
"payload": {
"text": "sample text1"
},
"score": 1,
"vector": [0.18257418, 0.36514837, 0.5477226, 0.73029673],
"version": 0
},
"score": 1,
"text": "sample text1",
"vector": [0.18257418, 0.36514837, 0.5477226, 0.73029673]
}
]
```
</details>
- **Weaviate:**
For Weaviate, the following fields are populated:
| Field Name | Type | Description |
| ---- | ---- | ----------- |
| original_entity | dict | the original response json from search REST API|
| score | float | certainty from the original entity, which evaluates the similarity between the entity and the query vector|
| text | string | text in the original entity|
| vector | list | vector of the entity|
<details>
<summary>Output</summary>
```json
[
{
"metadata": null,
"original_entity": {
"_additional": {
"certainty": 1,
"distance": 0,
"vector": [
0.58,
0.59,
0.6,
0.61,
0.62
]
},
"text": "sample text1."
},
"score": 1,
"text": "sample text1.",
"vector": [
0.58,
0.59,
0.6,
0.61,
0.62
]
}
]
```
</details>