wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
414 lines
12 KiB
Plaintext
414 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6340e329",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/llm/azure_openai.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8d09c269-526b-4858-979d-d77285c25260",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Azure AI model inference\n",
|
|
"\n",
|
|
"This notebook explains how to use `llama-index-llm-azure-inference` package with models deployed with the Azure AI model inference API in Azure AI studio or Azure Machine Learning. The package also support GitHub Models (Preview) endpoints."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "24e9114d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-llms-azure-inference"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d981ca8e",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you're opening this notebook on Google Colab, you will probably need to install LlamaIndex 🦙."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "02fcb2e6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5386685e-fa9e-46b4-8bd9-2cdec2d9903e",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prerequisites"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5a3491e8-11e8-4548-b6b9-d08246c7ef9b",
|
|
"metadata": {},
|
|
"source": [
|
|
"The Azure AI model inference is an API that allows developers to get access to a variety of models hosted on Azure AI using a consistent schema. You can use `llama-index-llms-azure-inference` integration package with models that support this API, including models deployed to Azure AI serverless API endpoints and a subset of models from Managed Inference. To read more about the API specification and the models that support it see [Azure AI model inference API](https://aka.ms/azureai/modelinference).\n",
|
|
"\n",
|
|
"To run this tutorial you need:\n",
|
|
"\n",
|
|
"1. Create an [Azure subscription](https://azure.microsoft.com).\n",
|
|
"2. Create an Azure AI hub resource as explained at [How to create and manage an Azure AI Studio hub](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/create-azure-ai-resource).\n",
|
|
"3. Deploy one model supporting the [Azure AI model inference API](https://aka.ms/azureai/modelinference). In this example we use a `Mistral-Large` deployment. \n",
|
|
"\n",
|
|
" * You can follow the instructions at [Deploy models as serverless APIs](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-serverless).\n",
|
|
"\n",
|
|
"Alternatively, you can use GitHub Models endpoints with this integration, including the free tier experience. Read more about [GitHub models](https://github.com/marketplace/models)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "14050fdc-890d-464c-89ff-9f444672de1d",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Environment Setup"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f7b24c58-b8fe-4687-bfa7-a659c2a86d93",
|
|
"metadata": {},
|
|
"source": [
|
|
"Follow this steps to get the information you need from the model you want to use:\n",
|
|
"\n",
|
|
"1. Go to the [Azure AI Foundry (formerly Azure AI Studio)](https://ai.azure.com/) or [Azure Machine Learning studio](https://ml.azure.com), depending on the product you are using.\n",
|
|
"2. Go to deployments (endpoints in Azure Machine Learning) and select the model you have deployed as indicated in the prerequisites.\n",
|
|
"3. Copy the endpoint URL and the key.\n",
|
|
" \n",
|
|
"> If your model was deployed with Microsoft Entra ID support, you don't need a key."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "03167162",
|
|
"metadata": {},
|
|
"source": [
|
|
"In this scenario, we have placed both the endpoint URL and key in the following environment variables:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "82233222-5a91-473b-b968-10bf8b7105e7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"os.environ[\"AZURE_INFERENCE_ENDPOINT\"] = \"<your-endpoint>\"\n",
|
|
"os.environ[\"AZURE_INFERENCE_CREDENTIAL\"] = \"<your-credential>\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a593031b-c872-4360-8775-dff4844ccead",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Connect to your deployment and endpoint"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ae049806-09b3-46fe-b589-2ae2f33beda9",
|
|
"metadata": {},
|
|
"source": [
|
|
"To use LLMs deployed in Azure AI studio or Azure Machine Learning you need the endpoint and credentials to connect to it. The parameter `model_name` is not required for endpoints serving a single model, like Managed Online Endpoints."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "fd389e2c-a3d5-4b47-acbe-b22b3da17670",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.llms.azure_inference import AzureAICompletionsModel"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1a53c5cc-d0aa-4d02-a2d6-791b2266a925",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = AzureAICompletionsModel(\n",
|
|
" endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"],\n",
|
|
" credential=os.environ[\"AZURE_INFERENCE_CREDENTIAL\"],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "64a41d09-5d11-44f8-b1c7-01321d567b4c",
|
|
"metadata": {},
|
|
"source": [
|
|
"Alternatively, if you endpoint support Microsoft Entra ID, you can use the following code to create the client:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "be72d467-2c15-45f4-b470-0396e41742bc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azure.identity import DefaultAzureCredential\n",
|
|
"\n",
|
|
"llm = AzureAICompletionsModel(\n",
|
|
" endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"],\n",
|
|
" credential=DefaultAzureCredential(),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8944961a",
|
|
"metadata": {},
|
|
"source": [
|
|
"> Note: When using Microsoft Entra ID, make sure that the endpoint was deployed with that authentication method and that you have the required permissions to invoke it."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ed641e58",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you are planning to use asynchronous calling, it's a best practice to use the asynchronous version for the credentials:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8aa5e256",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from azure.identity.aio import (\n",
|
|
" DefaultAzureCredential as DefaultAzureCredentialAsync,\n",
|
|
")\n",
|
|
"\n",
|
|
"llm = AzureAICompletionsModel(\n",
|
|
" endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"],\n",
|
|
" credential=DefaultAzureCredentialAsync(),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e3a6ad14",
|
|
"metadata": {},
|
|
"source": [
|
|
"If your endpoint is serving more than one model, like [GitHub Models](https://github.com/marketplace/models) or Azure AI Services, then you have to indicate the parameter `model_name`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f95b7416",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = AzureAICompletionsModel(\n",
|
|
" endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"],\n",
|
|
" credential=os.environ[\"AZURE_INFERENCE_CREDENTIAL\"],\n",
|
|
" model_name=\"mistral-large\", # change it to the model you want to use\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f414500c",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Use the model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "579ce31c-7b51-471e-bcb5-47da90b3d555",
|
|
"metadata": {},
|
|
"source": [
|
|
"Use the `complete` endpoint for text completion. Ihe `complete` method is still available for model of type `chat-completions`. On those cases, your input text is converted to a message with `role=\"user\"`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e333dd8e-e9cc-4002-b4ae-c38ad1af7543",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = llm.complete(\"The sky is a beautiful blue and\")\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3439845e-4add-4547-a335-b609a18c0e4e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = llm.stream_complete(\"The sky is a beautiful blue and\")\n",
|
|
"for r in response:\n",
|
|
" print(r.delta, end=\"\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "be0182f0-2914-406d-bf0a-8878f6200bdd",
|
|
"metadata": {},
|
|
"source": [
|
|
"Use the `chat` endpoint for conversation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0487b326-3aa9-4eec-9f75-a8ca1ed6b82c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.llms import ChatMessage\n",
|
|
"\n",
|
|
"messages = [\n",
|
|
" ChatMessage(\n",
|
|
" role=\"system\", content=\"You are a pirate with colorful personality.\"\n",
|
|
" ),\n",
|
|
" ChatMessage(role=\"user\", content=\"Hello\"),\n",
|
|
"]\n",
|
|
"\n",
|
|
"response = llm.chat(messages)\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "68d86509-72de-458c-8e8a-32a944c57486",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = llm.stream_chat(messages)\n",
|
|
"for r in response:\n",
|
|
" print(r.delta, end=\"\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "78e1bbec",
|
|
"metadata": {},
|
|
"source": [
|
|
"Rather than adding same parameters to each chat or completion call, you can set them at the client instance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2f61dac8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = AzureAICompletionsModel(\n",
|
|
" endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"],\n",
|
|
" credential=os.environ[\"AZURE_INFERENCE_CREDENTIAL\"],\n",
|
|
" temperature=0.0,\n",
|
|
" model_kwargs={\"top_p\": 1.0},\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8dac4cf8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = llm.complete(\"The sky is a beautiful blue and\")\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a6a2d27",
|
|
"metadata": {},
|
|
"source": [
|
|
"For parameters extra parameters that are not supported by the Azure AI model inference API but that are available in the underlying model, you can use the `model_extras` argument. In the following example, the parameter `safe_prompt`, only available for Mistral models, is being passed."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a7194d8f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = AzureAICompletionsModel(\n",
|
|
" endpoint=os.environ[\"AZURE_INFERENCE_ENDPOINT\"],\n",
|
|
" credential=os.environ[\"AZURE_INFERENCE_CREDENTIAL\"],\n",
|
|
" temperature=0.0,\n",
|
|
" model_kwargs={\"model_extras\": {\"safe_prompt\": True}},\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "20a0155b",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = llm.complete(\"The sky is a beautiful blue and\")\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "301330ae",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Additional resources\n",
|
|
"\n",
|
|
"To learn more about this integration visit [Getting starting with LlamaIndex and Azure AI](https://aka.ms/azureai/llamaindex)."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|