wehub-resource-sync
e768098d0e
Flake8 Lint / flake8 (push) Waiting to run
Spell check CI / Spell_Check (push) Waiting to run
tools_continuous_delivery / Private PyPI non-main branch release (push) Has been skipped
tools_continuous_delivery / Private PyPI main branch release (push) Failing after 2m42s
Publish Promptflow Doc / Build (push) Has been cancelled
Publish Promptflow Doc / Deploy (push) Has been cancelled
226 lines
10 KiB
Markdown
226 lines
10 KiB
Markdown
# LLM
|
|
|
|
## Introduction
|
|
Prompt flow LLM tool enables you to leverage widely used large language models like [OpenAI](https://platform.openai.com/), [Azure OpenAI (AOAI)](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/overview), and models in [Azure AI Studio model catalog](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/model-catalog) for natural language processing.
|
|
> [!NOTE]
|
|
> The previous version of the LLM tool is now being deprecated. Please upgrade to latest [promptflow-tools](https://pypi.org/project/promptflow-tools/) package to consume new llm tools.
|
|
|
|
Prompt flow provides a few different LLM APIs:
|
|
- **[Completion](https://platform.openai.com/docs/api-reference/completions)**: OpenAI's completion models generate text based on provided prompts.
|
|
- **[Chat](https://platform.openai.com/docs/api-reference/chat)**: OpenAI's chat models facilitate interactive conversations with text-based inputs and responses.
|
|
|
|
|
|
## Prerequisite
|
|
Create OpenAI resources, Azure OpenAI resources or MaaS deployment with the LLM models (e.g.: llama2, mistral, cohere etc.) in Azure AI Studio model catalog:
|
|
|
|
- **OpenAI**
|
|
|
|
Sign up account [OpenAI website](https://openai.com/)
|
|
|
|
Login and [Find personal API key](https://platform.openai.com/account/api-keys)
|
|
|
|
- **Azure OpenAI (AOAI)**
|
|
|
|
Create Azure OpenAI resources with [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal)
|
|
|
|
- **MaaS deployment**
|
|
|
|
Create MaaS deployment for models in Azure AI Studio model catalog with [instruction](https://learn.microsoft.com/azure/ai-studio/concepts/deployments-overview)
|
|
|
|
You can create serverless connection to use this MaaS deployment.
|
|
|
|
## **Connections**
|
|
|
|
Setup connections to provisioned resources in prompt flow.
|
|
|
|
| Type | Name | API KEY | API BASE | API Type | API Version |
|
|
|-------------|----------|----------|----------|-----------|-------------|
|
|
| OpenAI | Required | Required | - | - | - |
|
|
| AzureOpenAI | Required | Required | Required | Required | Required |
|
|
| Serverless | Required | Required | Required | - | - |
|
|
|
|
|
|
## Inputs
|
|
### Text Completion
|
|
|
|
| Name | Type | Description | Required |
|
|
|------------------------|-------------|-----------------------------------------------------------------------------------------|----------|
|
|
| prompt | string | text prompt that the language model will complete | Yes |
|
|
| model, deployment_name | string | the language model to use | Yes |
|
|
| max\_tokens | integer | the maximum number of tokens to generate in the completion. Default is 16. | No |
|
|
| temperature | float | the randomness of the generated text. Default is 1. | No |
|
|
| stop | list | the stopping sequence for the generated text. Default is null. | No |
|
|
| suffix | string | text appended to the end of the completion | No |
|
|
| top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No |
|
|
| logprobs | integer | the number of log probabilities to generate. Default is null. | No |
|
|
| echo | boolean | value that indicates whether to echo back the prompt in the response. Default is false. | No |
|
|
| presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No |
|
|
| frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0. | No |
|
|
| best\_of | integer | the number of best completions to generate. Default is 1. | No |
|
|
| logit\_bias | dictionary | the logit bias for the language model. Default is empty dictionary. | No |
|
|
|
|
|
|
### Chat
|
|
|
|
|
|
| Name | Type | Description | Required |
|
|
|------------------------|-------------|------------------------------------------------------------------------------------------------|----------|
|
|
| prompt | string | text prompt that the language model will response | Yes |
|
|
| model, deployment_name | string | the language model to use | Yes |
|
|
| max\_tokens | integer | the maximum number of tokens to generate in the response. Default is inf. | No |
|
|
| temperature | float | the randomness of the generated text. Default is 1. | No |
|
|
| stop | list | the stopping sequence for the generated text. Default is null. | No |
|
|
| top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No |
|
|
| presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No |
|
|
| frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0.| No |
|
|
| logit\_bias | dictionary | the logit bias for the language model. Default is empty dictionary. | No |
|
|
| tool\_choice | object | value that controls which tool is called by the model. Default is null. | No |
|
|
| tools | list | a list of tools the model may generate JSON inputs for. Default is null. | No |
|
|
| response_format | object | an object specifying the format that the model must output. Default is null. | No |
|
|
|
|
## Outputs
|
|
|
|
| Return Type | Description |
|
|
|-------------|----------------------------------------------------------------------|
|
|
| string | The text of one predicted completion or response of conversation |
|
|
|
|
|
|
## How to use LLM Tool?
|
|
|
|
1. Setup and select the connections to OpenAI resources
|
|
2. Configure LLM model api and its parameters
|
|
3. Prepare the Prompt with [guidance](./prompt-tool.md#how-to-write-prompt).
|
|
|
|
## How to write a chat prompt?
|
|
|
|
_To grasp the fundamentals of creating a chat prompt, begin with [this section](./prompt-tool.md#how-to-write-prompt) for an introductory understanding of jinja._
|
|
|
|
We offer a method to distinguish between different roles in a chat prompt, such as "system", "user", "assistant" and "tool". The "system", "user", "assistant" roles can have "name" and "content" properties. The "tool" role, however, should have "tool_call_id" and "content" properties. For an example of a tool chat prompt, please refer to [Sample 3](#sample-3).
|
|
|
|
### Sample 1
|
|
```jinja
|
|
# system:
|
|
You are a helpful assistant.
|
|
|
|
{% for item in chat_history %}
|
|
# user:
|
|
{{item.inputs.question}}
|
|
# assistant:
|
|
{{item.outputs.answer}}
|
|
{% endfor %}
|
|
|
|
# user:
|
|
{{question}}
|
|
```
|
|
|
|
In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API.
|
|
|
|
```
|
|
[
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant."
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<question-of-chat-history-round-1>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "<answer-of-chat-history-round-1>"
|
|
},
|
|
...
|
|
{
|
|
"role": "user",
|
|
"content": "<question>"
|
|
}
|
|
]
|
|
```
|
|
|
|
### Sample 2
|
|
```jinja
|
|
# system:
|
|
{# For role naming customization, the following syntax is used #}
|
|
## name:
|
|
Alice
|
|
## content:
|
|
You are a bot can tell good jokes.
|
|
```
|
|
|
|
In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API.
|
|
|
|
```
|
|
[
|
|
{
|
|
"role": "system",
|
|
"name": "Alice",
|
|
"content": "You are a bot can tell good jokes."
|
|
}
|
|
]
|
|
```
|
|
|
|
### Sample 3
|
|
This sample illustrates how to write a tool chat prompt.
|
|
```jinja
|
|
# system:
|
|
You are a helpful assistant.
|
|
|
|
# user:
|
|
What is the current weather like in Boston?
|
|
|
|
# assistant:
|
|
{# The assistant message with 'tool_calls' must be followed by messages with role 'tool'. #}
|
|
## tool_calls:
|
|
{{llm_output.tool_calls}}
|
|
|
|
# tool:
|
|
{#
|
|
Messages with role 'tool' must be a response to a preceding message with 'tool_calls'.
|
|
Additionally, 'tool_call_id's should match ids of assistant message 'tool_calls'.
|
|
#}
|
|
## tool_call_id:
|
|
{{llm_output.tool_calls[0].id}}
|
|
## content:
|
|
{{tool-answer-of-last-question}}
|
|
|
|
# user:
|
|
{{question}}
|
|
```
|
|
|
|
In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API.
|
|
|
|
```
|
|
[
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant."
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "What is the current weather like in Boston?"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": null,
|
|
"function_call": null,
|
|
"tool_calls": [
|
|
{
|
|
"id": "<tool-call-id-of-last-question>",
|
|
"type": "function",
|
|
"function": "<function-to-call-of-last-question>"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"role": "tool",
|
|
"tool_call_id": "<tool-call-id-of-last-question>",
|
|
"content": "<tool-answer-of-last-question>"
|
|
}
|
|
...
|
|
{
|
|
"role": "user",
|
|
"content": "<question>"
|
|
}
|
|
]
|
|
```
|