# LLM ## Introduction Prompt flow LLM tool enables you to leverage widely used large language models like [OpenAI](https://platform.openai.com/), [Azure OpenAI (AOAI)](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/overview), and models in [Azure AI Studio model catalog](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/model-catalog) for natural language processing. > [!NOTE] > The previous version of the LLM tool is now being deprecated. Please upgrade to latest [promptflow-tools](https://pypi.org/project/promptflow-tools/) package to consume new llm tools. Prompt flow provides a few different LLM APIs: - **[Completion](https://platform.openai.com/docs/api-reference/completions)**: OpenAI's completion models generate text based on provided prompts. - **[Chat](https://platform.openai.com/docs/api-reference/chat)**: OpenAI's chat models facilitate interactive conversations with text-based inputs and responses. ## Prerequisite Create OpenAI resources, Azure OpenAI resources or MaaS deployment with the LLM models (e.g.: llama2, mistral, cohere etc.) in Azure AI Studio model catalog: - **OpenAI** Sign up account [OpenAI website](https://openai.com/) Login and [Find personal API key](https://platform.openai.com/account/api-keys) - **Azure OpenAI (AOAI)** Create Azure OpenAI resources with [instruction](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal) - **MaaS deployment** Create MaaS deployment for models in Azure AI Studio model catalog with [instruction](https://learn.microsoft.com/azure/ai-studio/concepts/deployments-overview) You can create serverless connection to use this MaaS deployment. ## **Connections** Setup connections to provisioned resources in prompt flow. | Type | Name | API KEY | API BASE | API Type | API Version | |-------------|----------|----------|----------|-----------|-------------| | OpenAI | Required | Required | - | - | - | | AzureOpenAI | Required | Required | Required | Required | Required | | Serverless | Required | Required | Required | - | - | ## Inputs ### Text Completion | Name | Type | Description | Required | |------------------------|-------------|-----------------------------------------------------------------------------------------|----------| | prompt | string | text prompt that the language model will complete | Yes | | model, deployment_name | string | the language model to use | Yes | | max\_tokens | integer | the maximum number of tokens to generate in the completion. Default is 16. | No | | temperature | float | the randomness of the generated text. Default is 1. | No | | stop | list | the stopping sequence for the generated text. Default is null. | No | | suffix | string | text appended to the end of the completion | No | | top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No | | logprobs | integer | the number of log probabilities to generate. Default is null. | No | | echo | boolean | value that indicates whether to echo back the prompt in the response. Default is false. | No | | presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No | | frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0. | No | | best\_of | integer | the number of best completions to generate. Default is 1. | No | | logit\_bias | dictionary | the logit bias for the language model. Default is empty dictionary. | No | ### Chat | Name | Type | Description | Required | |------------------------|-------------|------------------------------------------------------------------------------------------------|----------| | prompt | string | text prompt that the language model will response | Yes | | model, deployment_name | string | the language model to use | Yes | | max\_tokens | integer | the maximum number of tokens to generate in the response. Default is inf. | No | | temperature | float | the randomness of the generated text. Default is 1. | No | | stop | list | the stopping sequence for the generated text. Default is null. | No | | top_p | float | the probability of using the top choice from the generated tokens. Default is 1. | No | | presence\_penalty | float | value that controls the model's behavior with regards to repeating phrases. Default is 0. | No | | frequency\_penalty | float | value that controls the model's behavior with regards to generating rare phrases. Default is 0.| No | | logit\_bias | dictionary | the logit bias for the language model. Default is empty dictionary. | No | | tool\_choice | object | value that controls which tool is called by the model. Default is null. | No | | tools | list | a list of tools the model may generate JSON inputs for. Default is null. | No | | response_format | object | an object specifying the format that the model must output. Default is null. | No | ## Outputs | Return Type | Description | |-------------|----------------------------------------------------------------------| | string | The text of one predicted completion or response of conversation | ## How to use LLM Tool? 1. Setup and select the connections to OpenAI resources 2. Configure LLM model api and its parameters 3. Prepare the Prompt with [guidance](./prompt-tool.md#how-to-write-prompt). ## How to write a chat prompt? _To grasp the fundamentals of creating a chat prompt, begin with [this section](./prompt-tool.md#how-to-write-prompt) for an introductory understanding of jinja._ We offer a method to distinguish between different roles in a chat prompt, such as "system", "user", "assistant" and "tool". The "system", "user", "assistant" roles can have "name" and "content" properties. The "tool" role, however, should have "tool_call_id" and "content" properties. For an example of a tool chat prompt, please refer to [Sample 3](#sample-3). ### Sample 1 ```jinja # system: You are a helpful assistant. {% for item in chat_history %} # user: {{item.inputs.question}} # assistant: {{item.outputs.answer}} {% endfor %} # user: {{question}} ``` In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API. ``` [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "" }, { "role": "assistant", "content": "" }, ... { "role": "user", "content": "" } ] ``` ### Sample 2 ```jinja # system: {# For role naming customization, the following syntax is used #} ## name: Alice ## content: You are a bot can tell good jokes. ``` In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API. ``` [ { "role": "system", "name": "Alice", "content": "You are a bot can tell good jokes." } ] ``` ### Sample 3 This sample illustrates how to write a tool chat prompt. ```jinja # system: You are a helpful assistant. # user: What is the current weather like in Boston? # assistant: {# The assistant message with 'tool_calls' must be followed by messages with role 'tool'. #} ## tool_calls: {{llm_output.tool_calls}} # tool: {# Messages with role 'tool' must be a response to a preceding message with 'tool_calls'. Additionally, 'tool_call_id's should match ids of assistant message 'tool_calls'. #} ## tool_call_id: {{llm_output.tool_calls[0].id}} ## content: {{tool-answer-of-last-question}} # user: {{question}} ``` In LLM tool, the prompt is transformed to match the [openai messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to openai chat API. ``` [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is the current weather like in Boston?" }, { "role": "assistant", "content": null, "function_call": null, "tool_calls": [ { "id": "", "type": "function", "function": "" } ] }, { "role": "tool", "tool_call_id": "", "content": "" } ... { "role": "user", "content": "" } ] ```