mirrors/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions 1 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:30:30 +08:00
commit 914fea506e
2793 changed files with 802106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
open-models/README.md
+34
View File
@@ -0,0 +1,34 @@
# Open Models
This repository contains examples for deploying and fine-tuning open source models with Vertex AI.
## Notebooks
### Serving
- [get_started_with_model_garden_sdk.ipynb](./get_started_with_model_garden_sdk.ipynb) - This notebook showcase how to deploy and serve open models on Vertex AI using the Model Garden SDK, with simplified setup and streamlined configs.
- [serving/cloud_run_ollama_gemma3_inference.ipynb](./serving/cloud_run_ollama_gemma3_inference.ipynb) - This notebook showcase how to deploy Google Gemma 3 in Cloud Run using Ollama, with the objective to build a simple API for chat.
- [serving/cloud_run_vllm_gemma3_inference.ipynb](./serving/cloud_run_vllm_gemma3_inference.ipynb) - This notebook showcase how to deploy Google Gemma 3 in Cloud Run using vLLM, with the objective to build a simple API for chat.
- [serving/cloud_run_ollama_qwen3_inference.ipynb](./serving/cloud_run_ollama_qwen3_inference.ipynb) - This notebook shows how to deploy Qwen 3 in Cloud Run using Ollama, with the objective to build a simple AI Agent.
- [serving/vertex_ai_ollama_gemma2_rag_agent.ipynb](./serving/vertex_ai_ollama_gemma2_rag_agent.ipynb) - This notebooks provides steps and code to deploy an open source agentic RAG pipeline to Vertex AI Prediction using Ollama and a Gemma 2 model adapter.
- [serving/vertex_ai_pytorch_inference_paligemma_with_custom_handler.ipynb](./serving/vertex_ai_pytorch_inference_paligemma_with_custom_handler.ipynb) - This notebooks provides steps and code to deploy Google PaliGemma with the Hugging Face Python Inference DLC using a custom handler on Vertex AI.
- [serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb](./serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb) - This notebook shows how to deploy Polish Large Language Model (PLLuM) from the Hugging Face Hub on Vertex AI using the Hugging Face Deep Learning Container (DLC) for Pytorch Inference in combination with a custom handler.
- [serving/vertex_ai_text_generation_inference_gemma.ipynb](./serving/vertex_ai_text_generation_inference_gemma.ipynb) - This notebooks provides steps and code to deploy Google Gemma with the Hugging Face DLC for Text Generation Inference (TGI) on Vertex AI.
- [serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb](./serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb) - This notebook showcases how to deploy Gemma 2 from the Hugging Face Hub with multiple LoRA adapters fine-tuned for different purposes such as coding, or SQL using Hugging Face's Text Generation Inference (TGI) Deep Learning Container (DLC) in combination with a custom handler on Vertex AI.
### Fine-tuning
- [fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb](./fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb) - This notebooks provides steps and code to fine-tune Google Gemma with TRL via the Hugging Face PyTorch DLC for Training on Vertex AI.
### Evaluation
- [evaluation/vertex_ai_tgi_gemma_with_genai_evaluation.ipynb](./evaluation/vertex_ai_tgi_gemma_with_genai_evaluation.ipynb) - This notebooks provides steps and code to use the Vertex AI Gen AI Evaluation framework to evaluate Gemma 2 in a summarization task.
- [evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb](./evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb) - This notebooks shows how to use custom judge model to evaluate LLM-based application using the Autorater configuration in Gen AI Eval service.
### Use cases
- [use-cases/model_garden_litellm_inference.ipynb](./use-cases/model_garden_litellm_inference.ipynb) - This notebook showcases how to deploy an open-source model from Vertex AI Model Garden and serve inference through LiteLLM using an OpenAI-compatible API, including support for chat completion and function calling.
- [use-cases/bigquery_ml_llama_inference.ipynb](./use-cases/bigquery_ml_llama_inference.ipynb) - This notebook showcases a simple end-to-end process for extracting entities and performing data analytics using BigQuery in conjunction with an open-source text-generation Large Language Model (LLM). We use Meta's Llama 3.3 70B model as an example.
- [use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb](./use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb) - This notebooks provides steps and code to deploy an open source RAG pipeline to Cloud Run using Ollama and the Gemma 2 model.
- [use-cases/guess_app.ipynb](./use-cases/guess_app.ipynb) - This notebook shows how to build a "Guess Who or What" app using FLUX and Gemini.
- [vertex_ai_deepseek_smolagents.ipynb](./use-cases/vertex_ai_deepseek_smolagents.ipynb) - This notebook showcases how to deploy DeepSeek R1 Distill Qwen 7B from the Hugging Face Hub on Vertex AI using Vertex AI Model Garden. It also shows how to prototype and deploy a simple agent using HuggingFace's smol-agents library on Vertex AI Reasoning Engine.
open-models/agents/qwen3_adk_vertexai.ipynb
+813
View File
@@ -0,0 +1,813 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Building an ADK agent using QWEN 3 on Vertex AI\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/agents/qwen3_adk_vertexai.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fagents%2Fqwen3_adk_vertexai.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/agents/qwen3_adk_vertexai.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/agents/qwen3_adk_vertexai.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/agents/qwen3_adk_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/agents/qwen3_adk_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/agents/qwen3_adk_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/agents/qwen3_adk_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/agents/qwen3_adk_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Firstname Lastname](https://github.com/username) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This tutorial demonstrates how to build an AI agent using Agent Development Kit (ADK) with Qwen3 model deployed on Vertex AI. Also, it shows how to deploy the agent as a scalable, production-ready service with Vertex AI Agent Engine.\n",
"\n",
"What you'll learn how to\n",
"\n",
"\n",
"* Deploy an open-source model like Qwen3 from Vertex AI Model Garden to a dedicated endpoint.\n",
"\n",
"* Build a tool-calling agent using Google's Agent Development Kit (ADK).\n",
"\n",
"* Integrate a custom model endpoint with ADK using the LiteLLM library.\n",
"\n",
"* Define tools and provide instructions for the agent to use them correctly.\n",
"\n",
"* Test the agent's functionality in a local environment.\n",
"\n",
"* Package and deploy the ADK agent to Vertex AI Agent Engine for a scalable, production-ready implementation.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install required packages\n",
"\n",
"First things first, let's get our toolkit ready. We'll need to install the necessary Python packages for interacting with Vertex AI, ADK, and LiteLLM."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet \"google-cloud-aiplatform[adk,agent_engines]>=1.101.0\" \"litellm>=1.73.6.post1\" \"nest-asyncio>=1.6.0\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"import uuid\n",
"\n",
"import vertexai\n",
"\n",
"# Project settings\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"if not PROJECT_ID:\n",
" raise ValueError(\"PROJECT_ID not set. Please set it in the cell above.\")\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"BUCKET_URI = \"[your-bucket-uri]\" # @param {type: \"string\", placeholder: \"[your-bucket-uri]\", isTemplate: true}\n",
"if not BUCKET_URI or BUCKET_URI == \"[your-bucket-uri]\":\n",
" raise ValueError(\"Please set a valid GCS bucket URI in the BUCKET_URI variable.\")\n",
"\n",
"# Set config for adk\n",
"os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID\n",
"os.environ[\"GOOGLE_CLOUD_LOCATION\"] = LOCATION\n",
"os.environ[\"GOOGLE_GENAI_USE_VERTEXAI\"] = \"1\"\n",
"\n",
"# Agent configuration\n",
"APP_NAME = \"weather_agent\"\n",
"USER_ID = \"user_\" + str(uuid.uuid4())\n",
"SESSION_ID = \"session_\" + str(uuid.uuid4())\n",
"MAX_TOKENS = 1000\n",
"TEMPERATURE = 1.0\n",
"\n",
"# Initialize Vertex AI\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries\n",
"\n",
"Now, let's import all the Python libraries and modules we'll be using throughout this tutorial. This includes components from Vertex AI, ADK, LiteLLM, and standard Python libraries.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6fc324893334"
},
"outputs": [],
"source": [
"import functools\n",
"import inspect\n",
"from typing import Callable, Optional, get_type_hints\n",
"\n",
"import google.auth\n",
"import nest_asyncio\n",
"from google import adk\n",
"from google.adk.agents import Agent\n",
"from google.adk.models.lite_llm import LiteLlm\n",
"from google.adk.sessions import InMemorySessionService\n",
"from google.adk.tools import FunctionTool\n",
"from google.cloud import aiplatform\n",
"from google.genai import types\n",
"from litellm import completion\n",
"from vertexai import agent_engines\n",
"from vertexai.preview import model_garden, reasoning_engines\n",
"\n",
"nest_asyncio.apply()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yUr0RADhgshQ"
},
"source": [
"### Helpers\n",
"\n",
"We'll define a few helper functions to simplify running our agent and interacting with it in a chat-like interface. These helpers will allow us to test the agent both in the local runner and on the remote Agent Engine."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Izt5PN6RguyF"
},
"outputs": [],
"source": [
"def python_type_to_json_schema_type(py_type):\n",
" \"\"\"Maps Python types to JSON schema types.\"\"\"\n",
" if py_type in (int, float):\n",
" return \"integer\" if py_type is int else \"number\"\n",
" elif py_type is str:\n",
" return \"string\"\n",
" elif py_type is bool:\n",
" return \"boolean\"\n",
" elif py_type is list:\n",
" return \"array\"\n",
" elif py_type is dict:\n",
" return \"object\"\n",
" return \"string\" # Default to string for unknown types\n",
"\n",
"\n",
"def convert_to_openai_tool(func):\n",
" \"\"\"\n",
" Converts a Python function into an OpenAI tool schema.\n",
" \"\"\"\n",
" # Get the function's signature and docstring\n",
" signature = inspect.signature(func)\n",
" docstring = inspect.getdoc(func) or \"\"\n",
"\n",
" # Get type hints\n",
" type_hints = get_type_hints(func)\n",
"\n",
" # Initialize the schema\n",
" function_schema = {\n",
" \"name\": func.__name__,\n",
" \"description\": docstring.split(\"\\n\\n\")[0],\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {},\n",
" \"required\": [],\n",
" },\n",
" }\n",
"\n",
" # Process each parameter\n",
" for param_name, param in signature.parameters.items():\n",
" param_type = type_hints.get(param_name, str)\n",
" json_type = python_type_to_json_schema_type(param_type)\n",
"\n",
" function_schema[\"parameters\"][\"properties\"][param_name] = {\n",
" \"type\": json_type,\n",
" \"description\": f\"The {param_name} for the function.\",\n",
" }\n",
"\n",
" # Check if the parameter is required\n",
" if param.default is inspect.Parameter.empty:\n",
" function_schema[\"parameters\"][\"required\"].append(param_name)\n",
"\n",
" # Final tool schema structure\n",
" tool_schema = {\"type\": \"function\", \"function\": function_schema}\n",
"\n",
" return tool_schema\n",
"\n",
"\n",
"def get_base_url(endpoint: aiplatform.Endpoint) -> str:\n",
" \"\"\"Get the base URL for the endpoint.\"\"\"\n",
" location = endpoint.resource_name.split(\"/\")[3]\n",
" if endpoint.gca_resource.dedicated_endpoint_enabled:\n",
" return (\n",
" f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}\"\n",
" f\"/v1beta1/{endpoint.resource_name}\"\n",
" )\n",
" return (\n",
" f\"https://{location}-aiplatform.googleapis.com\"\n",
" f\"/v1beta1/{endpoint.resource_name}\"\n",
" )\n",
"\n",
"\n",
"def run_single_turn(\n",
" runner: adk.Runner, query: str, session_id: str, user_id: str\n",
") -> Optional[str]:\n",
" \"\"\"Runs one turn with a local ADK runner.\"\"\"\n",
" content = types.Content(role=\"user\", parts=[types.Part(text=query)])\n",
" events = runner.run(user_id=user_id, session_id=session_id, new_message=content)\n",
" for event in events:\n",
" if event.is_final_response():\n",
" return event.content.parts[0].text\n",
" return None\n",
"\n",
"\n",
"def run_remote_single_turn(\n",
" remote_app: reasoning_engines.templates.adk.AdkApp,\n",
" query: str,\n",
" session_id: str,\n",
" user_id: str,\n",
") -> Optional[str]:\n",
" \"\"\"Runs one turn on a deployed Agent Engine.\"\"\"\n",
" events = remote_app.stream_query(\n",
" user_id=user_id, session_id=session_id, message=query\n",
" )\n",
" final_event = [event for event in events][-1]\n",
" return final_event[\"content\"][\"parts\"][0][\"text\"]\n",
"\n",
"\n",
"def chat_loop(\n",
" turn_handler: Callable[[str, str, str], Optional[str]],\n",
" session_id: str,\n",
" user_id: str,\n",
") -> None:\n",
" \"\"\"Main chat interface loop. It is given the function to handle each turn.\"\"\"\n",
" print(\"\\nStarting chat. Type 'exit' or 'quit' to end.\")\n",
" while True:\n",
" try:\n",
" user_input = input(\"You: \")\n",
" if user_input.lower() in [\"quit\", \"exit\", \"bye\"]:\n",
" print(\"\\nAssistant: Thank you for chatting. Have a great day!\")\n",
" break\n",
"\n",
" response = turn_handler(user_input, session_id, user_id)\n",
" if response:\n",
" print(f\"Assistant: {response}\")\n",
" else:\n",
" print(\"Assistant: I received no response.\")\n",
" except (KeyboardInterrupt, EOFError):\n",
" print(\"\\nAssistant: Chat ended.\")\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QAAOd02fV43F"
},
"source": [
"## Deploy Qwen3 model using Vertex AI Model Garden SDK\n",
"\n",
"Deploy the Qwen3 model from the Vertex AI Model Garden. Model Garden provides a curated collection of foundation models that can be easily deployed to a Vertex AI Endpoint. This gives us a scalable, managed API for our model, which our agent will use as its \"brain.\"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "l764864DiyCR"
},
"outputs": [],
"source": [
"# Load the model from Model Garden\n",
"model = model_garden.OpenModel(\"publishers/qwen/models/qwen3@qwen3-1.7b\")\n",
"\n",
"# Get container specification\n",
"serving_container_spec = model.list_deploy_options()[0].container_spec\n",
"\n",
"# Deploy the model\n",
"endpoint = model.deploy(\n",
" serving_container_spec=serving_container_spec,\n",
" machine_type=\"g2-standard-12\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=1,\n",
" use_dedicated_endpoint=False,\n",
" spot=False,\n",
" deploy_request_timeout=1800\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eDiKHwP4bAUh"
},
"source": [
"## Test the deployed endpoint with a simple request using LiteLlm\n",
"\n",
"Before building the full agent, it's a good practice to ensure we can communicate with our newly deployed Qwen3 endpoint. We'll use the LiteLLM library for this.\n",
"\n",
"LiteLLM provides a simple, unified interface to call over 100 different LLM APIs. It allows us to interact with our custom Vertex AI endpoint using the same syntax as the OpenAI API. This is incredibly useful because the ADK has a built-in LiteLlm model wrapper, making integration seamless.\n",
"\n",
"Get the necessary authentication credentials for our Google Cloud project."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b0aRsdUma_uv"
},
"outputs": [],
"source": [
"# Get credentials\n",
"creds, _ = google.auth.default()\n",
"auth_req = google.auth.transport.requests.Request()\n",
"creds.refresh(auth_req)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BtOuTPiwJ4Ow"
},
"source": [
"Define a simple Python function (get_weather) that we'll later turn into a tool for our agent.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DQVqKlVreqSc"
},
"outputs": [],
"source": [
"# Define a simple tool\n",
"def get_weather(city: str) -> str:\n",
"\n",
" \"\"\"\n",
" Get weather information for a city.\n",
"\n",
" Args:\n",
" city: The city name to get weather for.\n",
"\n",
" Returns:\n",
" Weather information string.\n",
" \"\"\"\n",
" city_lower = city.lower()\n",
"\n",
" # Simple weather simulation\n",
" weather_data = {\n",
" \"san francisco\": \"It's 70°F (21°C) and foggy.\",\n",
" \"sf\": \"It's 70°F (21°C) and foggy.\",\n",
" \"new york\": \"It's 75°F (24°C) and partly cloudy.\",\n",
" \"nyc\": \"It's 75°F (24°C) and partly cloudy.\",\n",
" \"london\": \"It's 60°F (15°C) and rainy.\",\n",
" \"tokyo\": \"It's 72°F (22°C) and sunny.\",\n",
" }\n",
"\n",
" for key, weather in weather_data.items():\n",
" if key in city_lower:\n",
" return weather\n",
"\n",
" return f\"It's 80°F (27°C) and sunny in {city}.\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-QNb9NnlJ7Mh"
},
"source": [
"Use litellm.completion to send a direct request to our Qwen3 endpoint (vertex_ai/openai/{endpoint.name}). This verifies that the endpoint is live and that our connection is configured correctly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KeuJAfKmdBuy"
},
"outputs": [],
"source": [
"response = completion(\n",
" model=f\"vertex_ai/openai/{endpoint.name}\",\n",
" messages=[{\"content\": \"Hello, how are you?/no_think\", \"role\": \"user\"}],\n",
" tools=[\n",
" # Convert the function to an OpenAI tool schema\n",
" convert_to_openai_tool(get_weather)\n",
" ],\n",
" temperature=0.7,\n",
" max_tokens=8192,\n",
" extra_body={\n",
" \"chat_template_kwargs\": {\"enable_thinking\": True},\n",
" },\n",
")\n",
"\n",
"print(response[\"choices\"][0][\"message\"][\"content\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## Create an Agent with ADK\n",
"\n",
"Now that we have a working model endpoint, it's time to build our agent using the Agent Development Kit. An ADK Agent is defined by several key components:\n",
"\n",
"\n",
"* model: The LLM the agent will use. Here, we'll use ADK's LiteLlm wrapper to point to our Qwen3 endpoint.\n",
"* name: A unique identifier for the agent.\n",
"\n",
"* description: A high-level summary of what the agent does.\n",
"* instruction: The core prompt that guides the agent's behavior. This is where we tell the agent how and when to use its tools.\n",
"* tools: A list of functions the agent can call to interact with external systems. We'll use the FunctionTool wrapper to make our get_weather function available to the agent.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7QHxMswxfTLW"
},
"outputs": [],
"source": [
"root_agent = Agent(\n",
" model=LiteLlm(\n",
" model=f\"vertex_ai/openai/{endpoint.name}\",\n",
" ),\n",
" name=\"weather_agent\",\n",
" description=\"Provides weather information for specific cities.\",\n",
" instruction=\"You are a helpful weather assistant. \"\n",
" \"When the user asks for the weather in a specific city, \"\n",
" \"use the 'get_weather' tool to find the information. \"\n",
" \"If the tool returns an error, inform the user politely. \"\n",
" \"If the tool is successful, present the weather report clearly.\"\n",
" \"After using tools, provide a natural language response incorporating the results.\"\n",
" \"/no_think\",\n",
" tools=[FunctionTool(func=get_weather)],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VScXms74fxV5"
},
"source": [
"## Test the agent locally\n",
"\n",
"Before deploying our agent to a production environment, it's crucial to test its logic and behavior locally. The ADK Runner allows us to do this easily.\n",
"\n",
"The Runner orchestrates the interaction between the user, the agent, and the various services. For this local test, we'll use an InMemorySessionService to keep track of the conversation history.\n",
"\n",
"The chat_loop helper function will provide a simple command-line interface to interact with our agent and see how it responds and uses its get_weather tool."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZzFNqH8kf07m"
},
"outputs": [],
"source": [
"# Create a session to store conversations\n",
"session_service = InMemorySessionService()\n",
"\n",
"# Create the specific session where the conversation will happen\n",
"session = await session_service.create_session(\n",
" app_name=APP_NAME, user_id=USER_ID, session_id=SESSION_ID\n",
")\n",
"\n",
"# Create the runner to run the agent\n",
"runner = adk.Runner(\n",
" agent=root_agent, app_name=APP_NAME, session_service=session_service\n",
")\n",
"\n",
"local_turn_handler = functools.partial(run_single_turn, runner)\n",
"\n",
"chat_loop(local_turn_handler, session.id, USER_ID)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6XRsZZpqhFES"
},
"source": [
"## Deploy the agent on Agent Engine\n",
"\n",
"Once we're satisfied with our agent's local performance, the final step is to deploy it as a scalable, production-ready service using Vertex AI Agent Engine. This powerful platform takes our ADK-defined agent and hosts it as a managed endpoint, handling all the underlying infrastructure, scaling, and logging."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VFIInWqpXJs6"
},
"source": [
"### Prepare your agent for Agent Engine\n",
"\n",
"To prepare our agent for deployment, we wrap it in a reasoning_engines.AdkApp. This class serves as the bridge between the ADK framework and the Agent Engine deployment platform.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iJxOA631hH1a"
},
"outputs": [],
"source": [
"app = reasoning_engines.AdkApp(\n",
" agent=root_agent,\n",
" enable_tracing=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WuqACpAXXZnJ"
},
"source": [
"### Send queries to your agent (locally)\n",
"\n",
"Before pushing to the cloud, we can use the AdkApp object to run a final local test. This ensures that the agent is packaged correctly and behaves as expected within the Agent Engine wrapper.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "P74ol0xtXbC-"
},
"outputs": [],
"source": [
"session = app.create_session(user_id=USER_ID)\n",
"\n",
"local_app_turn_handler = functools.partial(run_remote_single_turn, app)\n",
"\n",
"chat_loop(local_app_turn_handler, session.id, USER_ID)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eVjLyb-8X9o3"
},
"source": [
"### Deploy your agent to Agent Engine\n",
"\n",
"Deploying is as simple as calling the agent_engines.create() function. We pass our AdkApp object and specify any Python package requirements. Agent Engine handles the rest: it containerizes the application, provisions the necessary resources, and exposes a secure endpoint for our agent.\n",
"\n",
"Note: The deployment process can take a few minutes.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-1khvUTUX_F2"
},
"outputs": [],
"source": [
"remote_app = agent_engines.create(\n",
" agent_engine=app,\n",
" requirements=[\n",
" \"google-cloud-aiplatform[adk,agent_engines]>=1.101.0\",\n",
" \"litellm==1.73.6.post1\",\n",
" ],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S2Hm49QIYhWd"
},
"source": [
"### Send queries to your agent (remote)\n",
"\n",
"With the agent deployed, we get a remote_app object that points to our live Agent Engine endpoint. We can interact with this remote agent using the exact same chat_loop function, demonstrating the seamless transition from local development to cloud deployment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "crfG5FqcYjzy"
},
"outputs": [],
"source": [
"remote_session = remote_app.create_session(user_id=USER_ID)\n",
"\n",
"remote_app_turn_handler = functools.partial(run_remote_single_turn, remote_app)\n",
"\n",
"chat_loop(remote_app_turn_handler, remote_session.id, USER_ID)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up\n",
"\n",
"To avoid incurring ongoing charges to your Google Cloud account, it's important to clean up the resources you've created. The following code will delete the deployed Agent Engine and the Vertex AI Endpoint for the Qwen3 model.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kJpYPexsYyCD"
},
"outputs": [],
"source": [
"delete_endpoint = False\n",
"delete_agent_engine = False\n",
"\n",
"if delete_endpoint:\n",
" endpoint.delete(force=True)\n",
"\n",
"if delete_agent_engine:\n",
" remote_app.delete(force=True)"
]
}
],
"metadata": {
"colab": {
"name": "qwen3_adk_vertexai.ipynb",
"toc_visible": true
},
"environment": {
"kernel": "python3",
"name": "workbench-notebooks.m130",
"type": "gcloud",
"uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m130"
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": ""
}
},
"nbformat": 4,
"nbformat_minor": 4
}
open-models/benchmarking_eagle_on_vertex_ai.ipynb
+1846
View File
File diff suppressed because it is too large Load Diff
open-models/evaluation/evaluate_open_models_byod.ipynb
+418
View File
@@ -0,0 +1,418 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Qwen 3 evaluation - Bring your own data eval\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/evaluate_open_models_byod.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fevaluation%2Fevaluate_open_models_byod.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/evaluation/evaluate_open_models_byod.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/evaluate_open_models_byod.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/evaluate_open_models_byod.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/evaluate_open_models_byod.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/evaluate_open_models_byod.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/evaluate_open_models_byod.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/evaluate_open_models_byod.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>|"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to use the Vertex AI Evaluation service to compare the performance of a fine-tuned model against its base model on a custom dataset. This \"bring your own data\" approach is essential for assessing models on tasks that are specific to your domain.\n",
"\n",
"You will learn how to:\n",
"* **Set up your environment**: Install the necessary libraries and authenticate with Google Cloud.\n",
"* **Construct an evaluation dataset**: Create a `pandas.DataFrame` containing your prompts and the corresponding responses from two different models (a \"candidate\" and a \"baseline\").\n",
"* **Configure a pairwise evaluation task**: Use the `EvalTask` class from the Vertex AI SDK to set up a `pairwise_summarization_quality` evaluation.\n",
"* **Execute and analyze results**: Run the evaluation, which uses a powerful LLM as an \"auto-rater\" to compare the responses, and interpret the win/loss/tie metrics."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-cloud-aiplatform[evaluation]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"\n",
"\n",
"import vertexai\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1ioxwfhXiGxB"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZgXVDiILiKhZ"
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"import pandas as pd\n",
"from vertexai.evaluation import EvalTask"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## Pairwise comparison for Qwen3 (base vs finetuned)\n",
"\n",
"Now we get to the core of our tutorial. We're going to set up an evaluation to compare two sets of model responses for a medical note summarization task.\n",
"* **Candidate Model**: A fine-tuned model (`qwen_tuned_responses`) that has been specialized for this task. Note the use of precise medical terminology.\n",
"* **Baseline Model**: The original, general-purpose base model (`qwen_responses`). Note that its summaries are simpler and less technical.\n",
"\n",
"Our goal is to quantitatively determine if the fine-tuning resulted in a better model for this specific use case."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1mM0H8DbkYkp"
},
"source": [
"### Construct the Evaluation Dataset\n",
"\n",
"The first step is to gather our data into a structured format. The Vertex AI Evaluation service seamlessly integrates with `pandas.DataFrame`. We will create a DataFrame with three essential columns:\n",
"\n",
"* `prompt`: The input text we want the models to summarize.\n",
"* `response`: The output from our **candidate** model (the one we are testing, in this case, the fine-tuned model).\n",
"* `baseline_model_response`: The output from the model we are comparing against.\n",
"\n",
"This structure is fundamental to pairwise evaluation. For each row, the evaluator will be asked: \"Given this prompt, is `response` or `baseline_model_response` the better summary?\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WzL0IEjWhtmt"
},
"outputs": [],
"source": [
"prompts = [\n",
" \"Please summarize the following patient note: 72-year-old female with a past medical history of atrial fibrillation (not on anticoagulation) presented to the ED with a sudden onset of left-sided weakness and facial droop, which began 2 hours ago. Non-contrast CT head is negative for any acute bleed. NIH Stroke Scale is 12. Patient is a candidate for tPA. Neurology is on board.\",\n",
" \"Please summarize the following patient note: 65-year-old male with a 40-pack-year smoking history and known COPD presents with a 3-day history of increased shortness of breath, wheezing, and productive cough with yellow sputum. On exam, he is in mild respiratory distress with diffuse expiratory wheezes. ABG shows mild respiratory acidosis. He was started on Duonebs, IV Solu-Medrol, and Levaquin.\",\n",
" \"Please summarize the following patient note: 45-year-old male with a history of alcohol abuse presents with severe, constant epigastric pain radiating to the back, associated with nausea and vomiting. Labs are significant for lipase of 3500 U/L. CT abdomen shows peripancreatic fat stranding consistent with acute pancreatitis. Patient is made NPO and started on aggressive IV fluid hydration and pain management.\",\n",
" \"Please summarize the following patient note: 88-year-old female from a nursing home who tripped and fell, now with severe right hip pain and inability to bear weight. Physical exam reveals a shortened and externally rotated right leg. X-ray of the pelvis confirms a displaced right femoral neck fracture. Orthopedics has been consulted for surgical fixation.\",\n",
" \"Please summarize the following patient note: 19-year-old female with Type 1 Diabetes Mellitus presents with a 1-day history of polyuria, polydipsia, and vomiting. Fingerstick glucose is 'high' (>500 mg/dL). VBG shows a pH of 7.15 and bicarbonate of 10 mEq/L. Urine ketones are large. She is being admitted to the ICU for management of diabetic ketoacidosis with an insulin drip and fluid resuscitation.\",\n",
" \"Please summarize the following patient note: 78-year-old male with an indwelling Foley catheter presents from a skilled nursing facility with fever, confusion, and hypotension (BP 85/45). Labs show leukocytosis of 18,000 and a lactic acid of 4.2 mmol/L. Urinalysis is positive for nitrates and leukocyte esterase. Diagnosis is septic shock secondary to a UTI. He received a fluid bolus and was started on broad-spectrum antibiotics (Zosyn).\",\n",
" \"Please summarize the following patient note: A 2-year-old female with a URI for 2 days developed a fever of 104°F and experienced a 2-minute generalized tonic-clonic seizure. In the ED, she is postictal but arousable. Workup is consistent with a simple febrile seizure. Parents were educated on fever control and seizure precautions.\",\n",
" \"Please summarize the following patient note: 34-year-old female with a history of major depressive disorder, non-compliant with her SSRI, is brought in by family due to worsening depression, anhedonia, and active suicidal ideation with a plan. She is being admitted to the inpatient psychiatric unit on a 1:1 observation for safety and medication stabilization.\",\n",
" \"Please summarize the following patient note: 31-year-old female, G1P0 at 34 weeks gestation, presents for a routine check-up and is found to have a blood pressure of 165/110. She also reports headaches and seeing 'spots'. Urine dipstick shows 3+ proteinuria. The patient is being admitted for management of pre-eclampsia with severe features, and will be started on magnesium sulfate for seizure prophylaxis and labetalol for BP control.\",\n",
" \"Please summarize the following patient note: 68-year-old male presents with palpitations and lightheadedness. EKG shows atrial fibrillation with a rapid ventricular response at a rate of 140 bpm. Blood pressure is stable at 110/70. The patient was given a 20mg IV bolus of Cardizem, which converted him to normal sinus rhythm at 80 bpm. He will be started on an Eliquis for stroke prophylaxis.\",\n",
"]\n",
"\n",
"qwen_tuned_responses = [\n",
" \"A 72-year-old female with untreated atrial fibrillation is presenting with an acute ischemic stroke within the tPA window. A non-contrast CT head was negative for hemorrhage and Neurology is proceeding with thrombolytic therapy.\",\n",
" \"A 65-year-old male with a significant smoking history is being treated for an acute exacerbation of COPD, presenting with respiratory distress and mild respiratory acidosis. Management includes bronchodilators, systemic steroids, and antibiotics.\",\n",
" \"A 45-year-old male with a history of alcohol abuse is diagnosed with acute pancreatitis, confirmed by elevated lipase and CT findings. He is being managed supportively with NPO status, IV fluids, and analgesics.\",\n",
" \"An 88-year-old female sustained a displaced right femoral neck fracture after a fall. She is being evaluated by Orthopedics for surgical intervention.\",\n",
" \"A 19-year-old female with T1DM is being admitted to the ICU for treatment of severe diabetic ketoacidosis (DKA), requiring an insulin infusion and aggressive fluid resuscitation.\",\n",
" \"A 78-year-old male is in septic shock secondary to a urosepsis, presenting with hypotension, confusion, and significant lactic acidosis. He is being resuscitated with IV fluids and broad-spectrum antibiotics.\",\n",
" \"A 2-year-old female presented with a simple febrile seizure in the setting of a URI. She is clinically stable and the family was educated on supportive care.\",\n",
" \"A 34-year-old female with a history of MDD is being admitted to inpatient psychiatry for acute suicidal ideation, requiring close observation and medication management.\",\n",
" \"A 31-year-old female at 34 weeks gestation is being admitted for pre-eclampsia with severe features. She is being started on magnesium sulfate for seizure prophylaxis and antihypertensive therapy.\",\n",
" \"A 68-year-old male presented with atrial fibrillation with RVR and was successfully cardioverted with diltiazem (Cardizem). He will be initiated on apixaban (Eliquis) for anticoagulation.\",\n",
"]\n",
"\n",
"qwen_responses = [\n",
" \"A 72-year-old woman came to the hospital because her left side was weak. A CT scan of her head was normal. Her stroke score is 12. Doctors are giving her a special medicine for the stroke.\",\n",
" \"A 65-year-old man who smokes a lot is having trouble breathing. He has a cough and is wheezing. Doctors gave him some medicine to help him breathe and an antibiotic.\",\n",
" \"A 45-year-old man who drinks alcohol has bad stomach pain that goes to his back. His lab tests were very high. A CT scan showed a problem with his pancreas. He can't eat and is getting IV fluids and pain medicine.\",\n",
" \"An 88-year-old woman fell and hurt her hip. She can't walk on it. Her right leg looks shorter. An x-ray showed her hip is broken. A bone doctor will see her.\",\n",
" \"A 19-year-old woman with diabetes is sick and throwing up. Her blood sugar is very high and her blood is acidic. She is going to the ICU to get insulin and fluids.\",\n",
" \"An old man from a nursing home is confused and has a low blood pressure. His white blood cell count is high and he has a high lactic acid. The doctors think he has an infection from his urine and are giving him fluids and antibiotics.\",\n",
" \"A 2-year-old girl had a seizure that lasted for 2 minutes because she had a high fever. She is awake now. The doctors think it was a febrile seizure and told her parents what to do.\",\n",
" \"A 34-year-old woman is very depressed and wants to harm herself. She stopped taking her medicine. She is going to a psychiatric hospital so they can keep her safe and help with her medicine.\",\n",
" \"A 31-year-old pregnant woman has very high blood pressure. She has a headache and is seeing spots. She has protein in her urine. She is going to the hospital to get medicine to prevent seizures and lower her blood pressure.\",\n",
" \"A 68-year-old man's heart was beating very fast and irregularly. He felt dizzy. They gave him a medicine called Cardizem through an IV and his heart went back to normal. He will start a new medicine to prevent strokes.\",\n",
"]\n",
"\n",
"eval_dataset = pd.DataFrame(\n",
" {\n",
" \"prompt\": prompts,\n",
" \"response\": qwen_tuned_responses,\n",
" \"baseline_model_response\": qwen_responses,\n",
" }\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FrEsxxAmlhcA"
},
"outputs": [],
"source": [
"eval_dataset.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "W7pwueCBifIC"
},
"source": [
"### Define the Evaluation Task\n",
"\n",
"With our dataset prepared, we now define the evaluation job using the `EvalTask` class. This object encapsulates all the information needed to run the evaluation on the Vertex AI service.\n",
"\n",
"* `dataset`: The DataFrame we just created.\n",
"* `metrics`: A list of metrics to compute. We use `\"pairwise_summarization_quality\"`. This tells Vertex AI to use its model-based evaluator, configured with specific instructions for judging summary quality (e.g., coherence, accuracy, conciseness).\n",
"* `experiment`: The name of the Vertex AI Experiment to log this run under. Using experiments is a key MLOps practice, helping you organize, track, and compare different evaluation runs over time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fm4xanSWifVb"
},
"outputs": [],
"source": [
"EXPERIMENT_NAME = \"eval-qwen3\"\n",
"\n",
"summarization_eval_task = EvalTask(\n",
" dataset=eval_dataset,\n",
" metrics=[\"pairwise_summarization_quality\"],\n",
" experiment=EXPERIMENT_NAME,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Vr2wzWZ-j9KR"
},
"source": [
"### Run your evaluation\n",
"\n",
"This is where the magic happens. Calling `.evaluate()` sends our dataset and configuration to the Vertex AI backend. A scalable, serverless job spins up, and for each row in our DataFrame, a powerful foundation model (the \"auto-rater\") reads the prompt and both responses and renders a judgment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JnWDqgqmj9Yc"
},
"outputs": [],
"source": [
"eval_result = summarization_eval_task.evaluate()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "89taehcWke7S"
},
"source": [
"### Analyze the Results\n",
"\n",
"The `evaluate()` method returns a result object containing the aggregated metrics. The `.metrics_table` provides a clear, concise `pandas.DataFrame` summarizing the outcome."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7WLQDRsfkKl1"
},
"outputs": [],
"source": [
"eval_result.metrics_table"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "epaxAAxnkmHk"
},
"outputs": [],
"source": [
"delete_experiment = False\n",
"\n",
"if delete_experiment:\n",
" experiment = aiplatform.Experiment(EXPERIMENT_NAME)\n",
" experiment.delete()"
]
}
],
"metadata": {
"colab": {
"name": "evaluate_open_models_byod.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb
+1104
View File
File diff suppressed because it is too large Load Diff
open-models/evaluation/vertex_ai_tgi_gemma_with_genai_evaluation.ipynb
+1272
View File
File diff suppressed because it is too large Load Diff
open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb
+898
View File
@@ -0,0 +1,898 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# MetaMath with Vertex AI Open Source Model Tuning\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Ffine-tuning%2Fget_started_with_oss_tuning_on_vertexai.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/get_started_with_oss_tuning_on_vertexai.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Ivan Nardini](https://github.com/inardini) |\n",
"| [Vincent Truong](https://github.com/vincentkt-google) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to reproduce the core ideas of the **MetaMath** paper by fine-tuning a Llama model on the `MetaMathQA` dataset using Vertex AI's managed service for open-source models.\n",
"\n",
"### Objective\n",
"\n",
"The goal is to leverage the `MetaMathQA` dataset to enhance the mathematical reasoning capabilities of a base Llama model. Specifically, this notebook focuses on reproducing the results for the **7B model** variant discussed in the paper, using the comparable **Llama 3.1 8B model** available on Vertex AI.\n",
"\n",
"We will cover the following steps:\n",
"\n",
"1. **Prepare the Dataset**: Download the `MetaMathQA` dataset and convert it to the required JSON Lines (JSONL) format for Vertex AI.\n",
"2. **Fine-Tune the Model**: Configure and launch a managed fine-tuning job on Vertex AI using a Llama 3.1 8B model.\n",
"3. **Deploy & Evaluate**: Deploy the newly tuned model to a Vertex AI Endpoint and test its mathematical reasoning.\n",
"4. **Compare (Optional)**: Compare our model's output with the official pre-trained MetaMath model from Hugging Face.\n",
"5. **Run Official Evaluation (Advanced)**: Download our tuned model and run the official evaluation scripts from the MetaMath repository.\n",
"\n",
"\n",
"### Citation\n",
"\n",
"```\n",
"@article{yu2023metamath,\n",
" title={MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models},\n",
" author={Yu, Longhui and Jiang, Weisen and Shi, Han and Yu, Jincheng and Liu, Zhengying and Zhang, Yu and Kwok, James T and Li, Zhenguo and Weller, Adrian and Liu, Weiyang},\n",
" journal={arXiv preprint arXiv:2309.12284},\n",
" year={2023}\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started\n",
"\n",
"This initial section handles all the necessary setup, including authenticating your account, installing libraries, and configuring your Google Cloud environment."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sOwtBan9NObx"
},
"source": [
"### (Optional) Choose your runtime.\n",
"\n",
"This tutorial has been designed to compare your tuned version with the MetaMath model by running some tests and the evaluation benchmarks.\n",
"\n",
"To run **ONLY** those optional sections of this tutorial, you need to have access more resources compared to the one are available in free Colab. Consider to create a Workbench instance on Vertex AI using `a2-highgpu-2g (Accelerator Optimized: 2 NVIDIA Tesla A100 GPUs, 24 vCPUs, 170GB RAM)`.\n",
"\n",
"Optionally, you can install `jupyterlab-nvdashboard` to visualize GPU usage metrics within your notebook environment."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK and other required packages\n",
"\n",
"Install the Python libraries needed for this tutorial.\n",
"\n",
" * `google-cloud-aiplatform`: The official SDK for interacting with Vertex AI services like model tuning and deployment.\n",
" * `datasets`: A library from Hugging Face that makes it easy to download and manipulate datasets.\n",
" * `transformers`: A Hugging Face library used for downloading and running the official pre-trained MetaMath model for comparison.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet --force-reinstall google-cloud-aiplatform>=1.129.0\n",
"%pip install --upgrade --quiet --force-reinstall datasets transformers torch sentencepiece accelerate bitsandbytes vllm hf_transfer fraction tqdm numpy fire openai scipy jsonlines pandas pydantic crcmod"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment.\n",
"\n",
"This gives the notebook permission to access your Google Cloud resources on your behalf.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"# import sys\n",
"\n",
"# if \"google.colab\" in sys.modules:\n",
"# from google.colab import auth\n",
"\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
"\n",
"Then, you define essential configuration variables. All resources you create (like models and storage buckets) will be associated with your specified Google Cloud project and region.\n",
"\n",
" * `PROJECT_ID`: Your unique Google Cloud project identifier.\n",
" * `REGION`: The geographic location where your resources will be created (e.g., `us-central1`).\n",
" * `BUCKET_URI`: A unique Google Cloud Storage (GCS) bucket. This will be our central location for storing the training dataset and the resulting model artifacts.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"import vertexai\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = \"europe-west4\"\n",
"\n",
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type: \"string\", placeholder: \"[your-bucket-name]\", isTemplate: true}\n",
"# fmt: on\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\"\n",
"\n",
"# Create the GCS bucket if it doesn't exist\n",
"! gcloud storage buckets create {BUCKET_URI} --project {PROJECT_ID} --location {LOCATION}\n",
"\n",
"# Initialize the Vertex AI SDK. This authenticates our session and sets the default project and location.\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6fc324893334"
},
"outputs": [],
"source": [
"import gc\n",
"import json\n",
"import os\n",
"import time\n",
"import uuid\n",
"\n",
"import torch\n",
"import vertexai\n",
"from datasets import load_dataset\n",
"from google.cloud import aiplatform\n",
"from google.cloud.aiplatform_v1beta1.types import JobState\n",
"from pydantic import BaseModel, Field\n",
"from transformers import pipeline\n",
"from vertexai.preview import model_garden\n",
"from vertexai.tuning import SourceModel, sft"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kh9YeyTMXSA-"
},
"source": [
"### Helpers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "AfWTScHyXTo3"
},
"outputs": [],
"source": [
"def save_to_jsonl(dataset, output_path):\n",
" \"\"\"Save dataset in JSONL format required by Vertex AI.\"\"\"\n",
" with open(output_path, \"w\") as f:\n",
" for example in dataset:\n",
" json.dump(example, f)\n",
" f.write(\"\\n\")\n",
" print(f\"Saved {len(dataset)} examples to {output_path}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KPMJku26fvcR"
},
"source": [
"## Prepare the MetaMathQA Dataset\n",
"\n",
"The core of the MetaMath paper is its unique dataset, `MetaMathQA`. Here, we'll download it, process it into the required format, and upload it to our cloud storage bucket so the Vertex AI tuning service can access it.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "76OvI4emfyLI"
},
"source": [
"### Download and Format the Dataset\n",
"\n",
"The Vertex AI tuning service expects the training data to be in a **JSON Lines (JSONL)** format, where each line is a separate JSON object. For chat models, each object should contain a `\"messages\"` field with a list of conversation turns. We'll download the dataset from Hugging Face and map its `question` and `response` columns to this required format."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bJOxPVu7gAC7"
},
"outputs": [],
"source": [
"# Load the MetaMathQA dataset from the Hugging Face Hub.\n",
"# We'll use the 'GSM8K' configuration, which is a key part of the paper's contribution.\n",
"dataset = load_dataset(\"meta-math/MetaMathQA\")[\"train\"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jr2nwdu0uQwh"
},
"outputs": [],
"source": [
"# Use the .train_test_split() method to create an 80/20 split.\n",
"# 80% of the data will be for training, 20% for validation.\n",
"# This easily satisfies the <25% requirement for the validation set.\n",
"split_dataset = dataset.train_test_split(test_size=0.2, seed=42)\n",
"\n",
"# The result is a dictionary containing the two new splits.\n",
"train_split = split_dataset[\"train\"]\n",
"validation_split = split_dataset[\"test\"]\n",
"\n",
"# Limit validation dataset to less than 5000 rows for Vertex AI requirement\n",
"if len(validation_split) > 5000:\n",
" validation_split = validation_split.shuffle(\n",
" seed=42\n",
" ) # Use a seed for reproducibility\n",
" validation_split = validation_split.select(range(4999))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1T0m86eqf9o6"
},
"outputs": [],
"source": [
"# MetaMath's instruction template\n",
"METAMATH_TEMPLATE = \"\"\"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
"\n",
"### Instruction:\n",
"{instruction}\n",
"\n",
"### Response:\"\"\"\n",
"\n",
"# Define a function to transform each example into the {\"prompt\": ..., \"completion\": ...} structure.\n",
"\n",
"\n",
"def format_for_tuning(example):\n",
" query = example[\"query\"]\n",
" response = example[\"response\"]\n",
"\n",
" instruction = METAMATH_TEMPLATE.format(instruction=query)\n",
"\n",
" # Important: Add space before response for proper tokenization\n",
" return {\n",
" \"messages\": [\n",
" {\"role\": \"user\", \"content\": instruction},\n",
" {\"role\": \"assistant\", \"content\": f\" {response}\"},\n",
" ]\n",
" }\n",
"\n",
"\n",
"# Apply the formatting function to the entire dataset.\n",
"train_formatted_dataset = train_split.map(\n",
" format_for_tuning, remove_columns=train_split.column_names, num_proc=os.cpu_count()\n",
")\n",
"validation_formatted_dataset = validation_split.map(\n",
" format_for_tuning,\n",
" remove_columns=validation_split.column_names,\n",
" num_proc=os.cpu_count(),\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6JFCFh3As8Ka"
},
"outputs": [],
"source": [
"train_file_path = \"metamath_gsm8k_train.jsonl\"\n",
"validation_file_path = \"metamath_gsm8k_validation.jsonl\"\n",
"\n",
"# Write the formatted training data to a local JSONL file.\n",
"save_to_jsonl(train_formatted_dataset, train_file_path)\n",
"save_to_jsonl(validation_formatted_dataset, validation_file_path)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BI3_W51PmqxU"
},
"source": [
"### Upload Dataset to GCS\n",
"\n",
"The Vertex AI tuning service runs on Google's infrastructure and cannot directly access files in this notebook's local environment. Therefore, we must upload our formatted JSONL file to our GCS bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DpzE-ksCm_PE"
},
"outputs": [],
"source": [
"# Define the destination path in your GCS bucket.\n",
"train_file_uri = f\"{BUCKET_URI}/datasets/metamath_gsm8k_train.jsonl\"\n",
"validation_file_uri = f\"{BUCKET_URI}/datasets/metamath_gsm8k_validation.jsonl\"\n",
"\n",
"# Use the gsutil command-line tool to copy the local file to GCS.\n",
"! gcloud storage cp {train_file_path} {train_file_uri}\n",
"! gcloud storage cp {validation_file_path} {validation_file_uri}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Qzk4nMOJpy6q"
},
"source": [
"## Configure and Launch the Fine-Tuning Job\n",
"\n",
"We'll define all the parameters according to the official MetaMath README for our fine-tuning job and then launch it.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0ut8XGrqrTRY"
},
"source": [
"### Define configuration\n",
"\n",
"Set tuning parameters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kVnrHDAhpxH6"
},
"outputs": [],
"source": [
"# This class groups all hyperparameters and provides documentation and default values.\n",
"class MetaMathTuningConfig(BaseModel):\n",
" \"\"\"Configuration settings for the MetaMath fine-tuning job.\"\"\"\n",
"\n",
" base_model: str = Field(\n",
" default=\"meta/llama3_1@llama-3.1-8b\",\n",
" description=\"The base model to fine-tune, corresponding to the 7B model in the paper.\",\n",
" )\n",
" tuning_mode: str = Field(\n",
" default=\"FULL\",\n",
" description=\"The tuning mode. We use 'FULL' to replicate the paper's method for the 7B model.\",\n",
" )\n",
" epochs: int = Field(\n",
" default=3,\n",
" description=\"Number of training epochs, as specified in the MetaMath paper.\",\n",
" )\n",
" learning_rate: float = Field(\n",
" default=2e-5,\n",
" description=\"The learning rate for the optimizer, matching the paper's value for full fine-tuning.\",\n",
" )\n",
"\n",
"\n",
"# Create an instance of the configuration class.\n",
"config = MetaMathTuningConfig()\n",
"\n",
"# Dynamically create paths that depend on runtime variables.\n",
"output_uri = f\"{BUCKET_URI}/tuning-output/{uuid.uuid4()}\"\n",
"model_artifacts_gcs_uri = os.path.join(\n",
" output_uri, \"postprocess/node-0/checkpoints/final\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_-EdCljErt7x"
},
"source": [
"### Launch the full fine-tuning job\n",
"\n",
"This function sends our configuration to the Vertex AI service, which will provision machines and run the training.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FWBih5jfrNaA"
},
"outputs": [],
"source": [
"source_model = SourceModel(base_model=config.base_model)\n",
"\n",
"sft_tuning_job = sft.train(\n",
" source_model=source_model,\n",
" tuning_mode=config.tuning_mode,\n",
" epochs=config.epochs,\n",
" learning_rate=config.learning_rate,\n",
" train_dataset=train_file_uri,\n",
" validation_dataset=validation_file_uri,\n",
" output_uri=output_uri,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EKoGUHLNsTZs"
},
"source": [
"### Monitor the Job\n",
"\n",
"The tuning job runs remotely on Google Cloud. The following code provides a convenient way to check the job's status from within the notebook without having to manually refresh the console. It will print an update every 10 minutes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fMwlgrUtsc-C"
},
"outputs": [],
"source": [
"print(\n",
" \"Monitoring job... This will take several hours. You can safely close this notebook and come back later.\"\n",
")\n",
"\n",
"while sft_tuning_job.state not in [\n",
" JobState.JOB_STATE_CANCELLED,\n",
" JobState.JOB_STATE_FAILED,\n",
" JobState.JOB_STATE_SUCCEEDED,\n",
"]:\n",
" time.sleep(600) # Check status every 10 minutes\n",
" sft_tuning_job.refresh()\n",
" print(f\"Current job state: {sft_tuning_job.state.name!s}\")\n",
"\n",
"print(f\"Job finished with state: {sft_tuning_job.state.name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CIzIsfVMLN5k"
},
"source": [
"## Deploy the Tuned Model\n",
"\n",
"Once the tuning job is complete, the new model \"lives\" as a set of files in your GCS bucket. To use it for inference, we must **deploy** it. This process loads the model onto a server with a GPU and creates a unique API endpoint that we can send prediction requests to."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "V9dym9GbLTNb"
},
"outputs": [],
"source": [
"# Define the hardware for our deployment. An L4 GPU is a cost-effective choice for a model of this size.\n",
"machine_type = \"g2-standard-12\"\n",
"accelerator_type = \"NVIDIA_L4\"\n",
"accelerator_count = 1\n",
"\n",
"# Create a CustomModel object that points to our tuned model artifacts in GCS.\n",
"tuned_model = model_garden.CustomModel(gcs_uri=model_artifacts_gcs_uri)\n",
"\n",
"# Deploy the model. This step provisions the hardware and can take 15-30 minutes.\n",
"endpoint = tuned_model.deploy(\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uMhu2iwPVBpO"
},
"source": [
"## Evaluate and Compare\n",
"\n",
"Now, the exciting part! Let's test our newly tuned model and compare its performance to the official version."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZKk-YQBkVGdO"
},
"source": [
"### Evaluate Tuned Model\n",
"\n",
"We'll send a sample math problem to our deployed endpoint. The prompt format is critical; we use the exact template specified in the official MetaMath repository to ensure the model responds as expected."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fl9PnNwFU0mS"
},
"outputs": [],
"source": [
"# The inference prompt for MetaMath models.\n",
"prompt_template = \"Below is an instruction that describes a task. Write a response that appropriately completes the request.\\n\\n### Instruction:\\n{instruction}\\n\\n### Response: Let's think step by step.\"\n",
"instruction = \"James buys 5 packs of beef that are 4 pounds each. The price of beef is $5.50 per pound. How much did he pay?\"\n",
"\n",
"# Define the prediction request payload.\n",
"# We set a low temperature for more factual, less \"creative\" output.\n",
"instances = [\n",
" {\n",
" \"prompt\": prompt_template.format(instruction=instruction),\n",
" \"max_tokens\": 250,\n",
" \"temperature\": 0.2,\n",
" \"top_p\": 1.0,\n",
" \"top_k\": 1,\n",
" \"raw_response\": True,\n",
" }\n",
"]\n",
"\n",
"# Send the request to our endpoint.\n",
"response = endpoint.predict(instances=instances, use_dedicated_endpoint=True)\n",
"\n",
"print(\"Response from tuned model\")\n",
"for prediction in response.predictions:\n",
" print(prediction)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JHRtxQd3bq06"
},
"source": [
"### (Optional) Compare with the official MetaMath Model\n",
"\n",
"\n",
"To see how well our reproduction worked, we can compare its output to the official `MetaMath-7B-V1.0` model released by the authors on Hugging Face. This provides a valuable benchmark.\n",
"\n",
"**Note**: This step runs a large model locally on the notebook's machine and may require significant RAM, GPU and time to download and generate predictions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PCTnUS6_b71g"
},
"outputs": [],
"source": [
"# Enable hf_transfer for parallel downloads\n",
"os.environ[\"HF_HUB_ENABLE_HF_TRANSFER\"] = \"1\"\n",
"\n",
"# Load the official MetaMath 7B model from Hugging Face.\n",
"official_pipe = pipeline(\n",
" \"text-generation\", model=\"meta-math/MetaMath-7B-V1.0\", device_map=\"auto\"\n",
")\n",
"\n",
"# Use the same prompt and instruction for a fair comparison.\n",
"official_response = official_pipe(\n",
" prompt_template.format(instruction=instruction), max_new_tokens=250, do_sample=False\n",
")\n",
"\n",
"print(\"Response from the MetaMath Model\")\n",
"print(official_response[0][\"generated_text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dI2n45VTpZu_"
},
"source": [
"### Clean runtime\n",
"\n",
"To clear GPU memory we explicitly delete the pipeline, free up unused memory from PyTorch cache and any memory occupied by objects that are no longer referenced."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bWJnyKyhpZu_"
},
"outputs": [],
"source": [
"del official_pipe\n",
"torch.cuda.empty_cache()\n",
"gc.collect()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "t7PHkLLUVH4C"
},
"source": [
"### (Optional) Run the Official Evaluation Scripts\n",
"\n",
"The single-prompt tests above are good for a qualitative check. To get the official `pass@1` benchmark scores reported in the paper, you must run the evaluation scripts from the MetaMath GitHub repository against the full test dataset. This is how academic results are formally measured."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GF5RjQDqFGpK"
},
"source": [
"#### Clone the MetaMath Repository\n",
"This command downloads the evaluation scripts (`eval_gsm8k.py`) and the test data files."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "C6Ix2T0WFbrg"
},
"outputs": [],
"source": [
"!git clone https://github.com/meta-math/MetaMath.git"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VgCS_8peFNSc"
},
"source": [
"#### Download Your Tuned Model from GCS\n",
"The evaluation script runs locally, so it needs the model files on the notebook's machine. We'll copy them from our GCS output directory.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UQGrW2C8Ffzx"
},
"outputs": [],
"source": [
"# Create a local directory to store the model.\n",
"LOCAL_MODEL_PATH = \"./my_tuned_metamath_model\"\n",
"!mkdir -p {LOCAL_MODEL_PATH}\n",
"\n",
"# Copy the model files from GCS to the local path. This can take several minutes.\n",
"!gcloud storage cp --recursive {model_artifacts_gcs_uri}/* {LOCAL_MODEL_PATH}/"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ky7SwphaFvQ1"
},
"source": [
"#### Run the GSM8K Evaluation\n",
"Finally, execute the official evaluation script, pointing it to your locally downloaded model. This will run the model on every question in the GSM8K test set."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fIG1Ek1pF0-_"
},
"outputs": [],
"source": [
"!python MetaMath/eval_gsm8k.py \\\n",
" --model {LOCAL_MODEL_PATH} \\\n",
" --data_file ./MetaMath/data/test/GSM8K_test.jsonl \\\n",
" --tensor_parallel_size 2 \\\n",
" --batch_size 32"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gXQ-q0cRF2LS"
},
"source": [
"After running, the script will output the final `pass@1` accuracy score. You can compare this number directly to the results table in the MetaMath paper to see how well your model performed! Also if you are interested, you can run the same process to measure `math` metrics."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up\n",
"\n",
"To avoid incurring ongoing charges for the deployed model and stored data, you must undeploy the endpoint and delete your GCS artifacts."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f3PTswIxGLy3"
},
"outputs": [],
"source": [
"delete_experiments = True\n",
"delete_endpoint = True\n",
"delete_bucket = True\n",
"\n",
"# Deleting experiment\n",
"if delete_experiments:\n",
" experiment = aiplatform.Experiment.list()[0]\n",
" experiment.delete()\n",
"\n",
"# Deleting the endpoint itself removes the resource configuration.\n",
"if delete_endpoint:\n",
" endpoint = aiplatform.Endpoint.list()[0]\n",
" endpoint.delete(force=True)\n",
"\n",
"# To fully clean up, you should also delete the model artifacts and dataset from your GCS bucket.\n",
"# You can do this via the command line or the Google Cloud Console.\n",
"if delete_bucket:\n",
" !gcloud storage rm --recursive {BUCKET_URI}"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"name": "get_started_with_oss_tuning_on_vertexai.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb
+611
View File
@@ -0,0 +1,611 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Fine-tuning GPT-OSS 20B with Unsloth on Vertex AI Colab Enterprise and Nvidia A100 40GB GPU\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Ffine-tuning%2Fgpt_oss_20B_finetuning_with_unsloth.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/gpt_oss_20B_finetuning_with_unsloth.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Fred Molina](https://github.com/mltuto) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook provides a step-by-step guide to fine-tuning the GPT-OSS 20B model using Unsloth.\n",
" The process involves:\n",
"\n",
" 01. Installing the required libraries.\n",
" 02. Loading the GPT-OSS 20B model.\n",
" 03. Adding LoRA adapters to the model for fine-tuning.\n",
" 04. Preparing the dataset for fine-tuning.\n",
" 05. Fine-tuning the model on the dataset.\n",
"\n",
" **DISCLAIMER**\n",
" This notebook is intended for educational purposes only.\n",
"\n",
" - Date: Aug 2025\n",
" - Not suitable for production environments.\n",
" - Use at your own risk.\n",
" - This notebook is an adaptation of the original Unsloth team Notebook that runs on Colab public with T4 GPUS: https://docs.unsloth.ai/get-started/unsloth-notebooks all credits to them!\n",
" Some minor changes were done in how to install the required packages as Vertex AI Colab Enterprise manages the environments differently than local or Colab public\n",
" \n",
"Requirements:\n",
"A Vertex AI colab enterprise environment running on a Runtime that have a GPU (e.g., NVIDIA A100)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
"\n",
"from google import genai\n",
"\n",
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6fc324893334"
},
"outputs": [],
"source": [
"from IPython.display import Markdown, display\n",
"# 1. Upgrade uv, the fast package installer\n",
"!pip install --upgrade -qqq uv\n",
"# 2. Use uv to install all packages in a single, consolidated command.\n",
"# THIS VERSION FORCES NUMPY to a version < 2.0 to solve the TensorFlow conflict.\n",
"print(\"⏳ Installing all required libraries with NumPy compatibility fix...\")\n",
"!uv pip install --system --upgrade \\\n",
" \"numpy<2.0\" \\\n",
" \"torch>=2.8.0\" \\\n",
" \"triton>=3.4.0\" \\\n",
" \"torchvision==0.23.0\" \\\n",
" \"bitsandbytes==0.46.1\" \\\n",
" \"unsloth @ git+https://github.com/unslothai/unsloth.git@79b46f71b249600488842511c9ee40f27a3989f2\" \\\n",
" \"unsloth_zoo @ git+https://github.com/unslothai/unsloth-zoo@26615eb3021b92abbfc8f895da4cd6803322b658\" \\\n",
" \"peft @ git+https://github.com/huggingface/peft.git@a90003f0edd6353f489f48bd2c35080d27bb6974\" \\\n",
" \"accelerate @ git+https://github.com/huggingface/accelerate.git@23cf4ef8a3b58f016f63eeb158b4aa2c3e79fe6f\" \\\n",
" \"transformers @ git+https://github.com/huggingface/transformers.git@f4d57f2f0cdff0f63ee74a1f16f442dfaf525231\" \\\n",
" \"protobuf<=3.20.3\" \\\n",
" \"setuptools==69.5.1\" \\\n",
" \"wandb==0.21.1\"\n",
"print(\"✅✅✅ Installation complete!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "twtmUqlBSSXR"
},
"outputs": [],
"source": [
"# Restart Notebook Kernel\n",
"import os\n",
"\n",
"os.kill(os.getpid(), 9)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"##OpenAI GPT-OSS 20B finetuning on Vertex AI Colab Enterprise with Unsloth!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "n_9JN_wFSgAr"
},
"outputs": [],
"source": [
"import torch\n",
"from unsloth import FastLanguageModel\n",
"\n",
"max_seq_length = 1024\n",
"dtype = None\n",
"\n",
"model, tokenizer = FastLanguageModel.from_pretrained(\n",
" model_name=\"unsloth/gpt-oss-20b\",\n",
" dtype=dtype, # None for auto detection\n",
" max_seq_length=max_seq_length, # Choose any for long context!\n",
" load_in_4bit=True, # 4 bit quantization to reduce memory\n",
" full_finetuning=False, # [NEW!] We have full finetuning now!\n",
" # token = \"hf_...\", # use one if using gated models\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B_jeQ2mISmdM"
},
"source": [
"We now add LoRA adapters for parameter efficient finetuning - this allows us to only efficiently train 1% of all parameters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sW7jpAQJSg2v"
},
"outputs": [],
"source": [
"model = FastLanguageModel.get_peft_model(\n",
" model,\n",
" r=8, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",
" target_modules=[\n",
" \"q_proj\",\n",
" \"k_proj\",\n",
" \"v_proj\",\n",
" \"o_proj\",\n",
" \"gate_proj\",\n",
" \"up_proj\",\n",
" \"down_proj\",\n",
" ],\n",
" lora_alpha=16,\n",
" lora_dropout=0, # Supports any, but = 0 is optimized\n",
" bias=\"none\", # Supports any, but = \"none\" is optimized\n",
" # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",
" use_gradient_checkpointing=\"unsloth\", # True or \"unsloth\" for very long context\n",
" random_state=3407,\n",
" use_rslora=False, # We support rank stabilized LoRA\n",
" loftq_config=None, # And LoftQ\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L4XmgbNQYKyU"
},
"source": [
"###Data Prep"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D5AWtlaPYOxy"
},
"source": [
"The HuggingFaceH4/Multilingual-Thinking dataset will be utilized as our example. This dataset, available on Hugging Face, contains reasoning chain-of-thought examples derived from user questions that have been translated from English into four other languages. It is also the same dataset referenced in OpenAI's cookbook for fine-tuning. The purpose of using this dataset is to enable the model to learn and develop reasoning capabilities in these four distinct languages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "E_VJ7N7PYH-o"
},
"outputs": [],
"source": [
"def formatting_prompts_func(examples):\n",
" convos = examples[\"messages\"]\n",
" texts = [\n",
" tokenizer.apply_chat_template(\n",
" convo, tokenize=False, add_generation_prompt=False\n",
" )\n",
" for convo in convos\n",
" ]\n",
" return {\n",
" \"text\": texts,\n",
" }\n",
"\n",
"\n",
"from datasets import load_dataset\n",
"\n",
"dataset = load_dataset(\"HuggingFaceH4/Multilingual-Thinking\", split=\"train\")\n",
"dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gX2BBFGJYRYN"
},
"source": [
"To format our dataset, we will apply our version of the GPT OSS prompt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cQ1WXAN7YUaC"
},
"outputs": [],
"source": [
"from unsloth.chat_templates import standardize_sharegpt\n",
"\n",
"dataset = standardize_sharegpt(dataset)\n",
"dataset = dataset.map(\n",
" formatting_prompts_func,\n",
" batched=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hGzLfYvXYWwC"
},
"source": [
"Let's take a look at the dataset, and check what the 1st example shows"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Po_E_X7OYYCz"
},
"outputs": [],
"source": [
"print(dataset[0][\"text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KonBtw8DYanM"
},
"source": [
"What is unique about GPT-OSS is that it uses OpenAI Harmony format which supports conversation structures, reasoning output, and tool calling."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PnpsupSBYd_D"
},
"source": [
"### Train the model\n",
"\n",
"Now let's use Huggingface TRL's SFTTrainer! More docs here: TRL SFT docs. We do 30 steps to speed things up, but you can set num_train_epochs=1 for a full run, and turn off max_steps=None."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "auIEyRD-YYGa"
},
"outputs": [],
"source": [
"from trl import SFTConfig, SFTTrainer\n",
"\n",
"trainer = SFTTrainer(\n",
" model=model,\n",
" tokenizer=tokenizer,\n",
" train_dataset=dataset,\n",
" args=SFTConfig(\n",
" per_device_train_batch_size=1,\n",
" gradient_accumulation_steps=4,\n",
" warmup_steps=5,\n",
" # num_train_epochs = 1, # Set this for 1 full training run.\n",
" max_steps=30,\n",
" learning_rate=2e-4,\n",
" logging_steps=1,\n",
" optim=\"adamw_8bit\",\n",
" weight_decay=0.01,\n",
" lr_scheduler_type=\"linear\",\n",
" seed=3407,\n",
" output_dir=\"outputs\",\n",
" report_to=\"none\",\n",
" ),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kC1IdtxjYnlf"
},
"source": [
"# Show current memory stats"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "24EDdt5mYYJk"
},
"outputs": [],
"source": [
"# @title Show current memory stats\n",
"gpu_stats = torch.cuda.get_device_properties(0)\n",
"start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
"max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
"print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
"print(f\"{start_gpu_memory} GB of memory reserved.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OCoiRZPzYYMW"
},
"outputs": [],
"source": [
"trainer_stats = trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qFI-dVv8Ytx0"
},
"source": [
"# Show final memory and time stats"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SJNr5q_FYYPL"
},
"outputs": [],
"source": [
"# @title Show final memory and time stats\n",
"used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
"used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
"used_percentage = round(used_memory / max_memory * 100, 3)\n",
"lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
"print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
"print(\n",
" f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
")\n",
"print(f\"Peak reserved memory = {used_memory} GB.\")\n",
"print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
"print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
"print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iuIt6vBZYy9u"
},
"source": [
"## Inference\n",
"Let's run the model! You can change the instruction and input - leave the output blank!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bx9XjEPdY2TN"
},
"outputs": [],
"source": [
"messages = [\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a helpful assistant that can solve mathematical problems.\",\n",
" },\n",
" {\"role\": \"user\", \"content\": \"Solve x^5 + 3x^4 - 10 = 3.\"},\n",
"]\n",
"inputs = tokenizer.apply_chat_template(\n",
" messages,\n",
" add_generation_prompt=True,\n",
" return_tensors=\"pt\",\n",
" return_dict=True,\n",
" reasoning_effort=\"medium\",\n",
").to(model.device)\n",
"from transformers import TextStreamer\n",
"\n",
"_ = model.generate(**inputs, max_new_tokens=128, streamer=TextStreamer(tokenizer))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mPzuhAFSY4vL"
},
"source": [
"We just saw how to Fine Tune GPT - OSS 20B with an A100 40GB on Vertex AI Colab Enterprise using Unsloth. Unsloth has a Discord channel If you like Unsloth optimizations, show your support and ⭐️ Star Unsloth on Github ⭐️"
]
}
],
"metadata": {
"colab": {
"name": "gpt_oss_20B_finetuning_with_unsloth.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb
+754
View File
@@ -0,0 +1,754 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Hugging Face DLCs: Fine-tuning Gemma with Transformer Reinforcement Learning (TRL) on Vertex AI\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Ffine-tuning%2Fvertex_ai_trl_fine_tuning_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/fine-tuning/vertex_ai_trl_fine_tuning_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "83b98b0ba19c"
},
"source": [
"![Hugging Face x Google Cloud](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/google-cloud/thumbnail.png)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Alvaro Bartolome](https://github.com/alvarobartt), [Philipp Schmid](https://github.com/philschmid), [Simon Pagezy](https://github.com/pagezyhf), and [Jeff Boudier](https://github.com/jeffboudier) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3608b194054c"
},
"source": [
"> [**Gemma**](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models, developed by Google DeepMind and other teams across Google.\n",
"\n",
"> [**Transformer Reinforcement Learning (TRL)**](https://github.com/huggingface/trl) is a framework developed by Hugging Face to fine-tune and align both transformer language and diffusion models using methods such as Supervised Fine-Tuning (SFT), Reward Modeling (RM), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and others.\n",
"\n",
"> [**Hugging Face DLCs**](https://github.com/huggingface/Google-Cloud-Containers) are pre-built and optimized Deep Learning Containers (DLCs) maintained by Hugging Face and Google Cloud teams to simplify environment configuration for your ML workloads.\n",
"\n",
"> [**Google Vertex AI**](https://cloud.google.com/vertex-ai) is a Machine Learning (ML) platform that lets you train and deploy ML models and AI applications, and customize large language models (LLMs) for use in your AI-powered applications.\n",
"\n",
"This notebook showcases how to fine-tune Google Gemma with Supervised Fine-Tuning (SFT) and Low-Rank Adaptation (LoRA) in a single GPU via a custom job on Vertex AI using the Hugging Face PyTorch Deep Learning Container (DLC) for Training.\n",
"\n",
"By the end of this notebook, you will learn:\n",
"\n",
"- About Hugging Face's TRL and LLM fine-tuning\n",
"- How to create and run a custom container job on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK and other required packages\n",
"\n",
"To run this example, you will only need the [`google-cloud-aiplatform`](https://github.com/googleapis/python-aiplatform) Python SDK and the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) Python package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --user --quiet google-cloud-aiplatform huggingface_hub"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages in this Jupyter environment, if you are on Colab you must restart the runtime. You can do this by running the cell below, which restarts the current kernel. The restart might take a minute or longer. After it's restarted, continue to the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SbmM4z7FOBpM"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9cc332aee5fd"
},
"source": [
"**1. Vertex AI Workbench**\n",
"\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "54c81e1a63e6"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c93d96bc9c43"
},
"outputs": [],
"source": [
"# !gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3e571ce6c56"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "984a0526fb68"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9af3e57f89a"
},
"source": [
"### Authenticate your Hugging Face account\n",
"\n",
"As [`google/gemma-2b`](https://huggingface.co/google/gemma-2b) is a gated model, you need to have a Hugging Face Hub account, and accept the Google's usage license for Gemma. Once that's done, you need to generate a new user access token with read-only access so that the weights can be downloaded from the Hub in the Hugging Face DLC for TGI.\n",
"\n",
"> Note that the user access token can only be generated via [the Hugging Face Hub UI](https://huggingface.co/settings/tokens/new), where you can either select read-only access to your account, or follow the recommendations and generate a fine-grained token with read-only access to [`google/gemma-2b`](https://huggingface.co/google/gemma-2b)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4c31c7272804"
},
"source": [
"Then you can install the `huggingface_hub` that comes with a CLI that will be used for the authentication with the token generated in advance. So that then the token can be safely retrieved via `huggingface_hub.get_token`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8d836e0210fe"
},
"outputs": [],
"source": [
"from huggingface_hub import interpreter_login\n",
"\n",
"interpreter_login()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c71a4314c250"
},
"source": [
"Read more about [Hugging Face Security](https://huggingface.co/docs/hub/en/security), specifically about [Hugging Face User Access Tokens](https://huggingface.co/docs/hub/en/security-tokens)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud environment variables"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c14bf7a1ce9e"
},
"source": [
"You will need to set the following environment variables that are required to run this example."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"if PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
"\n",
"os.environ[\"LOCATION\"] = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d03b04d07dc0"
},
"source": [
"### (Optional) Create a bucket in Google Cloud Storage (GCS)\n",
"\n",
"As the custom job on Vertex AI will need to dump the fine-tuned artifacts somewhere on Google Cloud, you will need to have a bucket available on Google Cloud Storage (GCS), so that you can specify it as the bucket to be used within the custom job, so that anything written within the container in that path is automatically uploaded to GCS."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0ca9b8a1bf63"
},
"outputs": [],
"source": [
"BUCKET_URI = \"[your-bucket-uri]\" # @param {type:\"string\", isTemplate: true}\n",
"if BUCKET_URI == \"[your-bucket-uri]\":\n",
" raise ValueError(\n",
" \"A valid BUCKET_URI (e.g. `gs://path/to/bucket`) needs to be specified\"\n",
" )\n",
"os.environ[\"BUCKET_URI\"] = BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c1a68956ee21"
},
"source": [
"> Uncomment the `gcloud storage buckets create` command below if you need to create a bucket on GCS."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2ff73fc11e33"
},
"outputs": [],
"source": [
"# !gcloud storage buckets create $BUCKET_URI --project $PROJECT_ID --location=$LOCATION --default-storage-class=STANDARD --uniform-bucket-level-access"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4d527513fda4"
},
"source": [
"### Initialize Vertex AI SDK\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com), if not enabled already.\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "865070bd8875"
},
"outputs": [],
"source": [
"aiplatform.init(\n",
" project=os.getenv(\"PROJECT_ID\"),\n",
" location=os.getenv(\"LOCATION\"),\n",
" staging_bucket=os.getenv(\"BUCKET_URI\"),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ee37e1544281"
},
"source": [
"## Requirements"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "877cd3fb2dce"
},
"source": [
"You will need to have the following IAM roles set:\n",
"\n",
"- Artifact Registry Reader (roles/artifactregistry.reader)\n",
"- Vertex AI User (roles/aiplatform.user)\n",
"- Storage Object Creator (roles/storage.objectCreator)\n",
"\n",
"For more information about granting roles, see [Manage access](https://cloud.google.com/iam/docs/granting-changing-revoking-access).\n",
"\n",
"---\n",
"\n",
"You will also need to enable the following APIs (if not enabled already):\n",
"\n",
"- Vertex AI API (aiplatform.googleapis.com)\n",
"- Cloud Storage API (storage-api.googleapis.com)\n",
"- Artifact Registry API (artifactregistry.googleapis.com)\n",
"\n",
"For more information about API enablement, see [Enabling APIs](https://cloud.google.com/apis/docs/getting-started#enabling_apis).\n",
"\n",
"---\n",
"\n",
"You will also need to have a Google Cloud Storage (GCS) bucket where the custom job can read and write artifacts.\n",
"\n",
"---\n",
"\n",
"To access Gemma on Hugging Face, you're required to review and agree to Google's usage license on the Hugging Face Hub for any of the models from the [Gemma release collection](https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b), and the access request will be processed immediately."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## Create a `CustomContainerTrainingJob` on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2d1505636843"
},
"source": [
"You need to define a `CustomContainerTrainingJob` that runs on top of the Hugging Face PyTorch DLC for Training.\n",
"\n",
"The Hugging Face PyTorch DCL for Training comes with most of the Hugging Face Python libraries installed, so as to provide a seamless environment to use the Hugging Face stack on Google Cloud. In this example, as already mentioned, [`trl`](https://github.com/huggingface/trl) will be used to run the Supervised Fine-Tuning (SFT) on top of [`transformers`](https://github.com/huggingface/transformers) for the modelling, [`peft`](https://github.com/huggingface/peft) for the LoRA support and [`bitsandbytes`](https://github.com/huggingface/bitsandbytes) for the quantization support.\n",
"\n",
"Since the Hugging Face PyTorch DLC for Training does not have any pre-defined `CMD` or `ENTRYPOINT`, you will need to set it to `trl sft`; which is the TRL command to run the SFT fine-tuning via the recently released [TRL CLI](https://huggingface.co/docs/trl/en/clis)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f223713c7111"
},
"outputs": [],
"source": [
"job = aiplatform.CustomContainerTrainingJob(\n",
" display_name=\"gemma-2b-sft-lora\",\n",
" container_uri=\"us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-pytorch-training-cu121.2-3.transformers.4-42.ubuntu2204.py310\",\n",
" command=[\"trl\", \"sft\"],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "70ccc1885fbb"
},
"source": [
"## Prepare `CustomContainerTrainingJob` on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61f2d20694d6"
},
"source": [
"Before submitting the `CustomContainerTrainingJob` you need to define the following:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ae5cb15e64c2"
},
"source": [
"### Model to fine-tune\n",
"\n",
"In this case you will be fine-tuning a base model, [`google/gemma-2b`](https://huggingface.co/google/gemma-2b), meaning that the model by itself will just generate text and has not been fined-tuned in advance i.e. no prompt templates, etc."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c914172f34d9"
},
"source": [
"### Model requirements\n",
"\n",
"Fine-tuning a model can be expensive, you can estimate that to full fine-tune an LLM in half precision you would need around four times the disk size of the LLM in GPU VRAM.\n",
"\n",
"For a 2B LLM that takes around 5GiB on disk, it could translate into 20GiB for the half precision fine-tuning; but that could be reduced further by using a quantized optimizer, using LoRA or QLoRA, reducing the batch size, etc. In this case, you will be using an L4 NVIDIA GPU that comes with 24GiB of VRAM so both SFT or SFT + LoRA will work out of the box as those fit within the 24GiB VRAM limit.\n",
"\n",
"Read more about it in [Eleuther AI - Transformer Math 101](https://blog.eleuther.ai/transformer-math/)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8dc06eda3134"
},
"source": [
"### Dataset to fine-tune on\n",
"\n",
"SFT expects either a text-only, a conversational, or a prompt-completion dataset; in this case, a text-only dataset [`timdettmers/openassistant-guanaco`](https://huggingface.co/datasets/timdettmers/openassistant-guanaco) will be used.\n",
"\n",
"This dataset is a subset of the [Open Assistant dataset](https://huggingface.co/datasets/OpenAssistant/oasst1); and this subset only contains the highest-rated paths in the conversation tree, with a total of 9,846 samples."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0ad5dcdcde93"
},
"source": [
"## Submit `CustomContainerTrainingJob` on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7bcc72c900e5"
},
"source": [
"As the `CustomContainerTrainingJob` defines the command `trl sft` the arguments to be provided are listed either in the Python reference at [trl.SFTConfig](https://huggingface.co/docs/trl/en/sft_trainer#trl.SFTConfig) or via the `trl sft --help` command.\n",
"\n",
"Read more about the [TRL CLI](https://huggingface.co/docs/trl/en/clis)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "af41d90a8a96"
},
"outputs": [],
"source": [
"args = [\n",
" # MODEL\n",
" \"--model_name_or_path=google/gemma-2b\",\n",
" \"--torch_dtype=bfloat16\",\n",
" \"--attn_implementation=flash_attention_2\",\n",
" # DATASET\n",
" \"--dataset_name=timdettmers/openassistant-guanaco\",\n",
" \"--dataset_text_field=text\",\n",
" # PEFT\n",
" \"--use_peft\",\n",
" \"--lora_r=16\",\n",
" \"--lora_alpha=32\",\n",
" \"--lora_dropout=0.1\",\n",
" \"--lora_target_modules=all-linear\",\n",
" # TRAINER\n",
" \"--bf16\",\n",
" \"--max_seq_length=1024\",\n",
" \"--per_device_train_batch_size=8\",\n",
" \"--gradient_accumulation_steps=4\",\n",
" \"--gradient_checkpointing\",\n",
" \"--learning_rate=0.0002\",\n",
" \"--lr_scheduler_type=cosine\",\n",
" \"--optim=adamw_bnb_8bit\",\n",
" \"--num_train_epochs=1\",\n",
" \"--logging_steps=10\",\n",
" \"--do_eval\",\n",
" \"--eval_steps=100\",\n",
" \"--report_to=none\",\n",
" f\"--output_dir={os.getenv('BUCKET_URI').replace('gs://', '/gcs/')}/gemma-2b-sft-lora\",\n",
" \"--overwrite_output_dir\",\n",
" \"--seed=42\",\n",
" \"--log_level=debug\",\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "77cdb9afabbf"
},
"source": [
"> It's important to note that since GCS FUSE is used to mount the bucket as a directory within the running container job, the mounted path follows the formatting `/gcs/<BUCKET_NAME>`; meaning that anything the `SFTTrainer` writes there will be automatically uploaded to the GCS Bucket.\n",
">\n",
"> More information in the [Vertex AI Documentation - Prepare training code](https://cloud.google.com/vertex-ai/docs/training/code-requirements)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3bc2dce404dc"
},
"source": [
"Then you need to call the `submit` method on the `aiplatform.CustomContainerTrainingJob`, which is a non-blocking method that will schedule the job without blocking the execution.\n",
"\n",
"The arguments provided to the `submit` method are listed below:\n",
"\n",
"* **`args`** defines the list of arguments to be provided to the `trl sft` command, provided as `trl sft --arg_1=value ...`.\n",
"\n",
"* **`replica_count`** defines the number of replicas to run the job in, for training normally this value will be set to one.\n",
"\n",
"* **`machine_type`**, **`accelerator_type`** and **`accelerator_count`** define the machine i.e. Compute Engine instance, the accelerator (if any), and the number of accelerators (ranging from 1 to 8); respectively. The `machine_type` and the `accelerator_type` are tied together, so you will need to select an instance that supports the accelerator that you are using and vice-versa. More information about the different instances at [Compute Engine Documentation - GPU machine types](https://cloud.google.com/compute/docs/gpus), and about the `accelerator_type` naming at [Vertex AI Documentation - MachineSpec](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/MachineSpec).\n",
"\n",
"* **`base_output_dir`** defines the base directory that will be mounted within the running container from the GCS Bucket, conditioned by the `staging_bucket` argument provided to the `aiplatform.init` initially.\n",
"\n",
"* (optional) **`environment_variables`** defines the environment variables to define within the running container. As you are fine-tuning a gated model i.e. [`google/gemma-2b`](https://huggingface.co/google/gemma-2b), you need to set the `HF_TOKEN` environment variable. Additionally, some other environment variables are defined to set the cache path (`HF_HOME`) and to ensure that the logging messages are streamed to Google Cloud Logs Explorer properly (`TRL_USE_RICH`, `ACCELERATE_LOG_LEVEL`, `TRANSFORMERS_LOG_LEVEL`, and `TQDM_POSITION`).\n",
"\n",
"* (optional) **`timeout`** and **`create_request_timeout`** define the timeouts in seconds before interrupting the job execution or the job creation request (time to allocate required resources and start the execution), respectively."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0e685d739cc6"
},
"outputs": [],
"source": [
"from huggingface_hub import get_token\n",
"\n",
"job.submit(\n",
" args=args,\n",
" replica_count=1,\n",
" machine_type=\"g2-standard-12\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=1,\n",
" base_output_dir=f\"{os.getenv('BUCKET_URI')}/gemma-2b-sft-lora\",\n",
" environment_variables={\n",
" \"HF_HOME\": \"/root/.cache/huggingface\",\n",
" \"HF_TOKEN\": get_token(),\n",
" \"TRL_USE_RICH\": \"0\",\n",
" \"ACCELERATE_LOG_LEVEL\": \"INFO\",\n",
" \"TRANSFORMERS_LOG_LEVEL\": \"INFO\",\n",
" \"TQDM_POSITION\": \"-1\",\n",
" },\n",
" timeout=60 * 60 * 3, # 3 hours (10800s)\n",
" create_request_timeout=60 * 10, # 10 minutes (600s)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "da780e6ea3ff"
},
"source": [
"> The `CustomContainerTrainingJob` will run asynchronously, meaning that the execution won't be blocked and the job will automatically allocate and deallocate the required resources. So once triggered, you can go to Vertex AI in the Google Cloud Console to monitor the job closely."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1a39697ebbdf"
},
"source": [
"## What's next?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1b87bbe001c8"
},
"source": [
"Once the fine-tuning is done, you can already deploy it!\n",
"\n",
"To deploy it, you can either [merge the LoRA adapters into the fine-tuned Google Gemma model](https://huggingface.co/docs/peft/en/developer_guides/model_merging) in advance, or just [deploy the base model and the adapter separately](https://huggingface.co/blog/multi-lora-serving).\n",
"\n",
"As deployment and serving is out of the scope of this example, you can refer to the following examples in [`GoogleCloudPlatform/generative-ai`](https://github.com/GoogleCloudPlatform/generative-ai):\n",
"\n",
"- [Hugging Face DLCs: Serving Gemma with Text Generation Inference (TGI) on Vertex AI](./vertex_ai_text_generation_inference_gemma.ipynb)\n",
"\n",
"Or to the examples available within the [`huggingface/Google-Cloud-Containers`](https://github.com/huggingface/Google-Cloud-Containers) repository:\n",
"\n",
"- [Deploy Gemma 7B with TGI on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/deploy-gemma-on-vertex-ai/vertex-notebook.ipynb)\n",
"- [Deploy Gemma 7B from GCS with TGI on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/deploy-gemma-from-gcs-on-vertex-ai/vertex-notebook.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "392ded651e9e"
},
"source": [
"## References"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "137d82da56f2"
},
"source": [
"- [GitHub Repository - Hugging Face DLCs for Google Cloud](https://github.com/huggingface/Google-Cloud-Containers): contains all the containers developed by the collaboration of both Hugging Face and Google Cloud teams; as well as a lot of examples on both training and inference, covering both CPU and GPU, as well support for most of the models within the Hugging Face Hub.\n",
"- [Google Cloud Documentation - Hugging Face DLCs](https://cloud.google.com/deep-learning-containers/docs/choosing-container#hugging-face): contains a table with the latest released Hugging Face DLCs on Google Cloud.\n",
"- [Google Artifact Registry - Hugging Face DLCs](https://console.cloud.google.com/artifacts/docker/deeplearning-platform-release/us/gcr.io): contains all the DLCs released by Google Cloud that can be used.\n",
"- [Hugging Face Documentation - Google Cloud](https://huggingface.co/docs/google-cloud): contains the official Hugging Face documentation for the Google Cloud DLCs."
]
}
],
"metadata": {
"colab": {
"name": "vertex_ai_trl_fine_tuning_gemma.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/get_started_with_model_garden_sdk.ipynb
+878
View File
@@ -0,0 +1,878 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Vertex AI Model Garden SDK\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/tree/main/open-models/get_started_with_model_garden_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fget_started_with_model_garden_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/get_started_with_model_garden_sdk.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/tree/main/open-models/get_started_with_model_garden_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Ivan Nardini](https://github.com/inardini), [Eliza Huang](https://github.com/lizzij) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"Have you ever tried to deploy open models on Vertex AI? Deploying open models on Vertex AI can sometimes involve navigating numerous API calls and parameters, which can lead to a slower, more error-prone experience.\n",
"\n",
"But what if deploying state-of-the-art open models would be as simple as choosing your model and hitting the 'deploy' button?\n",
"\n",
"The new Vertex AI Model Garden CLI and SDK, powered by the new Deploy API, are designed to do just that. These tools are truly model-centric, providing you with a more consistent, and fluid experience. The SDK simplifies prototyping, removing the need for users to specify container details. The CLI provides a precise and interactive command-line interface for managing models, offering a programmatic alternative to the UI's one-click deployment and enabling scriptable automation.\n",
"\n",
"This tutorial shows how to use the Vertex AI Model Garden SDK for deploying your open models on Vertex AI.\n",
"\n",
"You will learn how to:\n",
"\n",
"- Find the models that you can deploy\n",
"- Deploy your 1st Model Garden model\n",
"- Handle with some advanced usage including setting deployment parameters and error handling\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YGcCmlbabAZZ"
},
"source": [
"orgpolicy.googleapis.com"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --force-reinstall --quiet 'google-cloud-aiplatform>=1.93.1' 'openai' 'google-auth' 'requests'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"import vertexai\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "haaUJ5VaWajl"
},
"source": [
"## Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kCp1oCOyWfZe"
},
"outputs": [],
"source": [
"import base64\n",
"import io\n",
"import os\n",
"\n",
"from PIL import Image\n",
"import google.auth\n",
"import matplotlib.pyplot as plt\n",
"import openai\n",
"from vertexai import model_garden"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fvVKEpROoXW5"
},
"source": [
"## Define helpers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9AoCHkCNoaLG"
},
"outputs": [],
"source": [
"def print_models(data_list: list[str], items_per_line: int = 2) -> None:\n",
" \"\"\"Prints the list with a specified number of items per line with index and emojis,\n",
" and includes the total count.\"\"\"\n",
" print(\"🌟--- Models available ---🌟\")\n",
" print(\"\\n\")\n",
" print(f\"🔢 Total models: {len(data_list)} 🔢\\n\") # Print the count here\n",
"\n",
" for i, item in enumerate(data_list):\n",
" print(f\"✨ {item} \", end=\"\")\n",
" if (i + 1) % items_per_line == 0:\n",
" print()\n",
" else:\n",
" print(\" --- \", end=\"\")\n",
"\n",
" if len(data_list) % items_per_line != 0:\n",
" print()\n",
"\n",
"\n",
"def plot_image_from_bytes(image_bytes_string: str) -> None:\n",
" \"\"\"Plots an image from a base64-encoded bytes string.\"\"\"\n",
" # Decode the base64 string\n",
" image_bytes = base64.b64decode(image_bytes_string)\n",
"\n",
" # Create a BytesIO object\n",
" image_stream = io.BytesIO(image_bytes)\n",
"\n",
" # Open the image using Pillow library\n",
" image = Image.open(image_stream)\n",
"\n",
" # Plot the image using matplotlib\n",
" plt.imshow(image)\n",
" plt.axis(\"off\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dJmOAajdW7qQ"
},
"source": [
"## Find the models that you can deploy\n",
"\n",
"In Vertex AI Model Garden, you can discover and deploy a wide range of open-source models.\n",
"\n",
"Many of these models are directly supported in Vertex AI Model Garden with some pre-configured for optimized deployment on Vertex AI. When the open model is not available in Vertex AI Model Garden or you want to deploy your model from HF hub, you can leverage the Hugging Face gallery which gives you access to more that 1M models.\n",
"\n",
"With Vertex AI Model Garden SDK, you can models that you can deploy and record the model ID to deploy. You can optionally list the supported Hugging Face models in Model Garden and even filter them by model names.\n",
"\n",
"Let's check with Gemma models are available in Vertex AI Model Garden."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2l5LsbqTevUr"
},
"outputs": [],
"source": [
"model_garden_models = model_garden.list_deployable_models(\n",
" model_filter=\"gemma\", list_hf_models=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "z5Fa1spZr2YZ"
},
"outputs": [],
"source": [
"print_models(model_garden_models, items_per_line=3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EWRuFtX4iXW4"
},
"source": [
"To include Gemma models that are available via Hugging Face Gallery, you can enable `list_hf_models` flag."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e9zjeZ-rrPPk"
},
"outputs": [],
"source": [
"deployable_models = model_garden.list_deployable_models(\n",
" model_filter=\"gemma\", list_hf_models=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "On3fzp-erTse"
},
"outputs": [],
"source": [
"print_models(deployable_models)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3BOPydegEkP2"
},
"source": [
"## Deploy your 1st Model Garden model\n",
"\n",
"To deploy an open model, Vertex AI Model Garden SDK provides the `OpenModel` which simplifies the process of deploying these models to Vertex AI for inference.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4qGzyKIPrkFE"
},
"outputs": [],
"source": [
"model_id = \"google/gemma3@gemma-3-1b-it\"\n",
"\n",
"gemma_model = model_garden.OpenModel(model_id)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vSFfpZ7Nxlkt"
},
"source": [
"### Check the deployment configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cA_yIRK-t_2T"
},
"source": [
"After you initiate the model, use `list_deploy_options()` method to discover the verified deployment configurations supported by a specific model.\n",
"\n",
"This is important to verify if you have enough resources to deploy the model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PaOvRoFZt9T1"
},
"outputs": [],
"source": [
"deploy_options = gemma_model.list_deploy_options(concise=True)\n",
"print(deploy_options)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C3OKT16hugDY"
},
"source": [
"### Deploy the model\n",
"\n",
"Now that you know how the model will be deployed, let's use the `deploy()` method to serve the selected open model to a Vertex AI Endpoint. Depending on the model, the deployment would require some minutes.\n",
"\n",
"> **Note**: If the model has an End User License Agreement (EULA), you can accept it using `accept_eula` flag.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0YqJ9pKi0cjz"
},
"outputs": [],
"source": [
"gemma_endpoint = gemma_model.deploy(accept_eula=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tmuwCPShxso1"
},
"source": [
"### Generate predictions\n",
"\n",
"After the model gets deployed, you can use the Vertex AI SDK."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vVHOFm7g_Lud"
},
"outputs": [],
"source": [
"prediction = gemma_endpoint.predict(\n",
" instances=[{\"prompt\": \"Tell me a joke\", \"temperature\": 0.7, \"max_tokens\": 50}]\n",
")\n",
"print(prediction.predictions[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xtsK74K7_K3j"
},
"source": [
"Additionally, you can get your prediction using ChatCompletion API from the OpenAI SDK."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "acGaSDaH9YAp"
},
"outputs": [],
"source": [
"creds, project = google.auth.default()\n",
"auth_req = google.auth.transport.requests.Request()\n",
"creds.refresh(auth_req)\n",
"\n",
"gemma_endpoint_url = f\"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/{gemma_endpoint.resource_name}\"\n",
"\n",
"client = openai.OpenAI(base_url=gemma_endpoint_url, api_key=creds.token)\n",
"\n",
"prediction = client.chat.completions.create(\n",
" model=\"\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Tell me a joke\"}],\n",
" temperature=0.7,\n",
" max_tokens=50,\n",
")\n",
"\n",
"print(prediction.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sai-h7DryWKi"
},
"source": [
"## Advanced configuration and error handling\n",
"\n",
"The Vertex AI Model Garden SDK provides a customizable and reliable interface to deploy your open models.\n",
"\n",
"You can specify advanced deployment configurations. And it provides a solid error handling to ensure robust and stable deployments, providing informative messages when problems arise.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KRAuXc76v4pY"
},
"source": [
"### Specify optional deployement parameters\n",
"\n",
"With the `deploy()` method, you can configure various deployment settings, including:\n",
"\n",
"- Compute Resources: Machine type, number of replicas (min/max), accelerator type and count.\n",
"\n",
"- Infrastructure: Spot VMs, reservation affinity, dedicated endpoints.\n",
"\n",
"- Serving Container: Specify custom container images, ports, health checks, and environment variables for advanced customization.\n",
"\n",
"Let's deploy a diffusion model with some additional deployment parameters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Pf3egfPPv_E6"
},
"outputs": [],
"source": [
"sd_model = model_garden.OpenModel(\"stabilityai/stable-diffusion-xl-base-1.0\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-c6-ocHPKca0"
},
"source": [
"You deploy the model to a Vertex AI Endpoint."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GCnpQHQZwBFk"
},
"outputs": [],
"source": [
"sd_endpoint = sd_model.deploy(\n",
" machine_type=\"g2-standard-4\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=1,\n",
" min_replica_count=1,\n",
" max_replica_count=1,\n",
" endpoint_display_name=\"sd-endpoint\",\n",
" model_display_name=\"sd-model\",\n",
" deploy_request_timeout=3 * 60 * 60,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fS1FFbbPWiLN"
},
"source": [
"Generate your image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VjoOS7irWqx0"
},
"outputs": [],
"source": [
"prediction = sd_endpoint.predict(instances=[\"A cat\"])\n",
"plot_image_from_bytes(prediction.predictions[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p6_t8l_R07QJ"
},
"source": [
"### Error Handling"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vdmv4qVOVVrY"
},
"source": [
"#### Model not found\n",
"\n",
"You're trying to deploy a model named `google/some-model@some-version`. This string represents the model's identifier and version within the Model Garden (or potentially another registry). However, Vertex AI cannot find a model matching that identifier and version.\n",
"\n",
"The `model.deploy()` call will raise a Python exception. This is crucial because it immediately signals that something went wrong."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dNwuhhO61DvW"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"google/some-model@some-version\")\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xl8deqpvVyH1"
},
"source": [
"#### Malformed model name\n",
"\n",
"You're attempting to deploy a model using the identifier `publisher/google/some-model@some-version`. The problem here isn't that the model doesn't exist, but rather that the format of the model name itself is incorrect according to the Vertex AI Model Garden's expected structure.\n",
"\n",
"Similar to the \"Model Not Found\" case, `model.deploy()` will raise a Python exception. This time, it will be an exception indicating an invalid input."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wtgkO-Rd15kR"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"publisher/google/some-model@some-version\")\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "O2uSTNqpWYRP"
},
"source": [
"#### Insufficient quota\n",
"\n",
"You are trying to deploy the model `meta/llama3_1@llama-3.1-8b-instruct`. This time, the model name is correctly formatted, and the model exists. However, your Google Cloud project has insufficient quota to deploy this specific model.\n",
"\n",
"> Quotas are limits set by Google Cloud to control resource usage and prevent accidental overspending. Relevant quotas could include Compute Engine Quotas which are limits on the number of CPUs, GPUs, or the amount of RAM you can use in a given region. As well as Vertex AI Quota which are specific limits on the number of model deployments, prediction requests, or other Vertex AI-specific operations.\n",
"\n",
"As with the previous cases, the `model.deploy()` call will raise a Python exception. This indicates that the deployment failed due to a quota issue."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "L0iTgub52Osg"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\n",
" \"publishers/deepseek-ai/models/deepseek-r1@deepseek-r1\"\n",
" )\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S-_Y3_SrXWV1"
},
"source": [
"#### Organization policy\n",
"\n",
"You're attempting to deploy the model `stabilityai/stable-diffusion-xl-base-1.0`. The model name is correct, the model exists, and you have sufficient quota. However, an organizational policy prevents you from deploying this specific model.\n",
"\n",
"> Organizational policies are rules set by your Google Cloud organization's administrator to control which resources and services can be used within the organization. These policies are often used for security, compliance, and cost control.\n",
"\n",
"The `model.deploy()` call will raise a Python exception like the one below:\n",
"```\n",
"---------------------------------------------------------------------------\n",
"_InactiveRpcError Traceback (most recent call last)\n",
"/usr/local/lib/python3.11/dist-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)\n",
" 75 try:\n",
"---> 76 return callable_(*args, **kwargs)\n",
" 77 except grpc.RpcError as exc:\n",
"\n",
"6 frames\n",
"_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:\n",
"\tstatus = StatusCode.FAILED_PRECONDITION\n",
"\tdetails = \"Organization Policy constraint `constraints/vertexai.allowedModels` violated for `projects/your-project-id` attempting to use a disallowed Gen AI model stable-diffusion-xl-base-1.0. Please contact your organization administrator to fix this violation. For more info, see https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access.\"\n",
"\tdebug_error_string = \"UNKNOWN:Error received from peer ipv4:your-ipv4 {created_time:\"2025-01-16T20:52:11.244017603+00:00\", grpc_status:9, grpc_message:\"Organization Policy constraint `constraints/vertexai.allowedModels` violated for `projects/your-project-id` attempting to use a disallowed Gen AI model stable-diffusion-xl-base-1.0. Please contact your organization administrator to fix this violation. For more info, see https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access.\"}\"\n",
">\n",
"\n",
"The above exception was the direct cause of the following exception:\n",
"\n",
"FailedPrecondition Traceback (most recent call last)\n",
"/usr/local/lib/python3.11/dist-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)\n",
" 76 return callable_(*args, **kwargs)\n",
" 77 except grpc.RpcError as exc:\n",
"---> 78 raise exceptions.from_grpc_error(exc) from exc\n",
" 79\n",
" 80 return error_remapped_callable\n",
"\n",
"FailedPrecondition: 400 Organization Policy constraint `constraints/vertexai.allowedModels` violated for `projects/your-project-id` attempting to use a disallowed Gen AI model stable-diffusion-xl-base-1.0. Please contact your organization administrator to fix this violation. For more info, see https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access.\n",
"\n",
"```\n",
"\n",
"This exception will indicate that the deployment was blocked due to a policy violation."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D6ZikIbpYPk5"
},
"source": [
"#### Missing EULA (End-User License Agreement)\n",
"\n",
"You're trying to deploy the model `publishers/meta/models/llama3-2@llama-3.2-90b-vision`. The model name is correct, the model exists, you have sufficient quota, and no organizational policies are blocking it.\n",
"\n",
"However, you haven't accepted the model's EULA. Many models, especially those from third-party publishers (like Meta in this case), require you to agree to their terms of service before you can use them.\n",
"\n",
"The `model.deploy()` call will raise a Python exception. This exception will indicate that the deployment failed because the EULA hasn't been accepted.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PP-ZZUH85LzV"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\n",
" \"publishers/meta/models/llama3-2@llama-3.2-90b-vision\"\n",
" )\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MCoNmRJubrSi"
},
"source": [
"#### Request Hugging Face gated model without access token\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j7KbEY8aPoaE"
},
"source": [
"You are trying to deploy `black-forest-labs/flux.1-dev`, which, is a gated model on the Hugging Face Hub. This means you need a Hugging Face Hub account, acceptance of the model's license, and a valid read-only access token to download the model weights. In this scenario, you're missing the necessary access token.\n",
"\n",
"The `model.deploy()` call will raise a Python exception. The exception will signal that authentication with the Hugging Face Hub failed due to a missing token."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ktX2GNEy6hP4"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"black-forest-labs/FLUX.1-dev\")\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VK67i0I2fBvb"
},
"source": [
"#### Request Hugging Face gated model with invalid access token\n",
"\n",
"You're attempting to deploy the gated model `black-forest-labs/FLUX.1-dev`. You're explicitly providing a Hugging Face access token via the hugging_face_access_token argument to the deploy method. However, the token `invalid-token` is, as the name suggests, not valid.\n",
"\n",
"The `model.deploy()` call will definitely raise an exception. The exception type will be similar to the previous missing token case, indicating an authentication failure with the Hugging Face Hub."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qTcDSSgG8NiR"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"black-forest-labs/FLUX.1-dev\")\n",
" endpoint = model.deploy(hugging_face_access_token=\"invalid-token\")\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N5UrKKMifeGi"
},
"outputs": [],
"source": [
"delete_endpoints = True\n",
"\n",
"if delete_endpoints:\n",
" gemma_endpoint.delete(force=True)\n",
" sd_endpoint.delete(force=True)"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_model_garden_sdk.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/get_started_with_model_garden_sdk_custom_import.ipynb
+586
View File
@@ -0,0 +1,586 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Import, Deploy, and Serve custom open models on Vertex AI using Vertex AI Model Garden SDK.\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fget_started_with_model_garden_sdk_custom_import.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk_custom_import.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Ivan Nardini](https://github.com/inardini) |\n",
"| [Eliza Huang](https://github.com/lizzij) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"The Vertex AI Model Garden allows you to not only use Google's state-of-the-art models but also to bring your own. \n",
"\n",
"This tutorial shows how to take an open-source model from Hugging Face, importing it into Vertex AI, and serving it on a scalable, production-ready endpoint using Vertex AI Model Garden SDK Import Custom Model Weights feature. \n",
"\n",
"This feature gives you the flexibility to use a vast ecosystem of open models while leveraging the Google Cloud's enterprise-grade infrastructure on Vertex AI for serving models.\n",
"\n",
"### What You'll Learn\n",
"\n",
" * How to efficiently transfer model artifacts from Hugging Face Hub directly to Google Cloud Storage (GCS).\n",
" * How to register your model with the Vertex AI Model Registry using a GCS URI.\n",
" * How to deploy the custom model to a dedicated Vertex AI Endpoint with a single command.\n",
" * How to run inference on your newly deployed model.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install required packages\n",
"\n",
"Install the necessary Python packages for this tutorial.\n",
"\n",
"*Note: The hf_transfer extra enables faster downloads using Rust-based transfer. This can speed up large model downloads by 2-5x*\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --force-reinstall --quiet 'google-cloud-aiplatform>=1.105.0' 'openai' 'google-auth' 'requests' 'huggingface_hub[hf_transfer]'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"# import sys\n",
"\n",
"# if \"google.colab\" in sys.modules:\n",
"# from google.colab import auth\n",
"\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"import vertexai\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type: \"string\", placeholder: \"[your-bucket-name]\", isTemplate: true}\n",
"# fmt: on\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\"\n",
"\n",
"! gcloud storage buckets create {BUCKET_URI} --project={PROJECT_ID} --location={LOCATION}\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4oWLaELWrRPz"
},
"source": [
"### Authenticate your HuggingFace account\n",
"\n",
"To download models from Hugging Face, especially gated ones, you need to authenticate.\n",
"\n",
"The `interpreter_login` function provides an easy way to do this within a notebook environment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BQcw4H0srWNE"
},
"outputs": [],
"source": [
"from huggingface_hub import interpreter_login\n",
"\n",
"interpreter_login()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6fc324893334"
},
"outputs": [],
"source": [
"import shutil\n",
"from pathlib import Path\n",
"\n",
"import google.auth\n",
"import openai\n",
"from google.cloud import storage\n",
"from google.cloud.storage import transfer_manager\n",
"from huggingface_hub import snapshot_download\n",
"from tqdm.auto import tqdm\n",
"from vertexai.preview import model_garden"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "w0Lwlu7DM3Bw"
},
"source": [
"### Helpers\n",
"\n",
"Before Vertex AI can deploy a custom model, its artifacts must be accessible in a Google Cloud Storage bucket. This helper function, `transfer_model`, automates the entire process.\n",
"\n",
"It handles:\n",
"\n",
"1. **Fast Downloads**: Enables `hf_transfer` for accelerated downloads from Hugging Face.\n",
"2. **Local Staging**: Downloads the model files to a temporary local directory.\n",
"3. **Efficient Uploads**: Uses the `transfer_manager` to upload files to GCS in parallel chunks, which is much faster for large model weights.\n",
"4. **Automatic Cleanup**: Removes the temporary local files after the upload is complete."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Fi0qLHMxrgfc"
},
"outputs": [],
"source": [
"def transfer_model(\n",
" model_id: str,\n",
" bucket_name: str,\n",
" location: str,\n",
" artifact_path: str = \"\",\n",
" exclude_patterns: list | None = None,\n",
" chunk_size: int = 32 * 1024 * 1024, # 32 MiB\n",
" workers: int = 8,\n",
") -> str:\n",
" \"\"\"Download model from HuggingFace and upload to GCS with parallel transfers.\"\"\"\n",
" # Defaults\n",
" exclude_patterns = exclude_patterns or [\n",
" \"*.bin\",\n",
" \"*.pth\",\n",
" \"*.gguf\",\n",
" \".gitattributes\",\n",
" ]\n",
" artifact_path = artifact_path or model_id.split(\"/\")[-1]\n",
" local_dir = Path(f\"tmp/{model_id.replace('/', '--')}\")\n",
"\n",
" # Enable fast transfers\n",
" os.environ[\"HF_HUB_ENABLE_HF_TRANSFER\"] = \"1\"\n",
"\n",
" try:\n",
" # Download from HuggingFace\n",
" print(f\"📥 Downloading {model_id}...\")\n",
" snapshot_download(\n",
" repo_id=model_id,\n",
" local_dir=str(local_dir),\n",
" ignore_patterns=exclude_patterns,\n",
" resume_download=True,\n",
" )\n",
"\n",
" # Setup GCS\n",
" client = storage.Client()\n",
" bucket = client.bucket(bucket_name)\n",
"\n",
" # Create bucket if needed\n",
" if not bucket.exists():\n",
" print(f\"📦 Creating bucket {bucket_name}...\")\n",
" bucket = client.create_bucket(bucket_name, location=location)\n",
"\n",
" # Get files to upload\n",
" files = [f for f in local_dir.rglob(\"*\") if f.is_file()]\n",
" total_size = sum(f.stat().st_size for f in files) / (1024**3)\n",
"\n",
" print(f\"☁️ Uploading {len(files)} files ({total_size:.2f} GB) to GCS...\")\n",
"\n",
" # Upload files with parallel chunks\n",
" for file_path in tqdm(files, desc=\"Files\", unit=\"file\"):\n",
" blob_name = f\"{artifact_path}/{file_path.relative_to(local_dir)}\"\n",
" blob = bucket.blob(blob_name)\n",
"\n",
" # Use parallel upload for large files (>32MB), regular for small files\n",
" if file_path.stat().st_size > chunk_size:\n",
" transfer_manager.upload_chunks_concurrently(\n",
" str(file_path), blob, chunk_size=chunk_size, max_workers=workers\n",
" )\n",
" else:\n",
" blob.upload_from_filename(str(file_path))\n",
"\n",
" gcs_path = f\"gs://{bucket_name}/{artifact_path}\"\n",
" print(f\"✅ Done! Model available at {gcs_path}\")\n",
" return gcs_path\n",
"\n",
" finally:\n",
" # Cleanup\n",
" if local_dir.exists():\n",
" shutil.rmtree(local_dir)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HX7UhfWdsN3w"
},
"source": [
"## Transfer Your Model from Hugging Face to GCS\n",
"\n",
"Now, let's use the function to transfer a fine-tuned Gemma model from Hugging Face to our GCS bucket.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uqf_RmUwv7z5"
},
"outputs": [],
"source": [
"# The Hugging Face model we want to import.\n",
"hf_model_id = \"xsanskarx/thinkygemma-4b\"\n",
"\n",
"# This command will download the model and upload it to your GCS bucket.\n",
"# This may take several minutes depending on the model size and your connection.\n",
"imported_custom_model_uri = transfer_model(hf_model_id, BUCKET_NAME, LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hUhWp6uHwa7s"
},
"source": [
"## Import and Deploy the Model with Model Garden\n",
"\n",
"With the model artifacts in GCS, we can now use the `model_garden` SDK to register and deploy it.\n",
"\n",
"First, we create a `CustomModel` object, pointing it to the GCS URI where our model is stored."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TAKZG1sBwfRs"
},
"outputs": [],
"source": [
"model = model_garden.CustomModel(\n",
" gcs_uri=imported_custom_model_uri,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "215f72a4a900"
},
"source": [
"After you initiate the model, use `list_deploy_options()` method to discover the verified deployment configurations supported by a specific model.\n",
"\n",
"This is important to verify if you have enough resources to deploy the model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0a490035154e"
},
"outputs": [],
"source": [
"deploy_options = model.list_deploy_options()\n",
"print(deploy_options)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6f2815215036"
},
"source": [
"Next, we use the suggested configuration to set the serving infrastructure for our model's endpoint and deploy the model.\n",
"\n",
"The real magic happens with the **`model.deploy()`** command. This single line of code abstracts away a complex series of operations:\n",
"\n",
" * It creates a new **Vertex AI Model** resource from your GCS artifacts.\n",
" * It provisions the specified compute resources.\n",
" * It creates a **Vertex AI Endpoint**.\n",
" * It deploys the model to that endpoint, making it ready to serve inference requests."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "18b2o2qx0jIh"
},
"outputs": [],
"source": [
"# Define the machine resources for the serving endpoint.\n",
"# g2-standard-24 and 2 L4 GPUs is a robust configuration for many ~7B models.\n",
"MACHINE_TYPE = \"g2-standard-24\"\n",
"ACCELERATOR_TYPE = \"NVIDIA_L4\"\n",
"ACCELERATOR_COUNT = 2\n",
"\n",
"# Deploy the model. This will create a Vertex AI Endpoint and deploy the model to it.\n",
"# This step can take 15-20 minutes as it provisions the necessary hardware.\n",
"endpoint = model.deploy(\n",
" machine_type=MACHINE_TYPE,\n",
" accelerator_type=ACCELERATOR_TYPE,\n",
" accelerator_count=ACCELERATOR_COUNT,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oynBjezZvWIq"
},
"source": [
"## Run Inference on Your Deployed Model\n",
"\n",
"Once the deployment is complete, your model is live and ready to serve predictions. You can interact with the endpoint using the `predict()` method.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0LHF_UR_vd39"
},
"outputs": [],
"source": [
"response = endpoint.predict(\n",
" instances=[{\"prompt\": \"how many r does strawberry have?\"}],\n",
" use_dedicated_endpoint=True,\n",
")\n",
"print(response.predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tmuwCPShxso1"
},
"source": [
"You can also get your prediction using ChatCompletion API from the OpenAI SDK."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "acGaSDaH9YAp"
},
"outputs": [],
"source": [
"creds, project = google.auth.default()\n",
"auth_req = google.auth.transport.requests.Request()\n",
"creds.refresh(auth_req)\n",
"\n",
"endpoint_url = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1beta1/{endpoint.resource_name}\"\n",
"\n",
"client = openai.OpenAI(base_url=endpoint_url, api_key=creds.token)\n",
"\n",
"prediction = client.chat.completions.create(\n",
" model=\"\", messages=[{\"role\": \"user\", \"content\": \"Tell me a joke\"}], temperature=0.7\n",
")\n",
"\n",
"print(prediction.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LE7mv18CvvZs"
},
"source": [
"## Cleaning up\n",
"\n",
"To avoid incurring ongoing charges to your Google Cloud account, it's important to clean up the resources you've created. The following commands will undeploy the model from the endpoint and then delete the endpoint itself.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "r-LKGlR2v32X"
},
"outputs": [],
"source": [
"delete_endpoint = True\n",
"delete_bucket = True\n",
"\n",
"if delete_endpoint:\n",
" endpoint.delete(force=True)\n",
"\n",
"if delete_bucket:\n",
" !gcloud storage rm --recursive {BUCKET_URI}"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_model_garden_sdk_custom_import.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/get_started_with_model_garden_terraform_deployment.ipynb
+1480
View File
File diff suppressed because it is too large Load Diff
open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb
+790
View File
@@ -0,0 +1,790 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Handling Reasoning with MaaS Models on Vertex AI using vLLM\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fget_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "da996843f0dc"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Ivan Nardini](https://github.com/inardini)|"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This tutorial explains how to use reasoning capabilities with Model-as-a-Service (MaaS) models on Google Cloud Vertex AI. These models leverage vLLM's reasoning infrastructure to provide both reasoning steps and final conclusions in their outputs."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## What is Reasoning in LLMs?\n",
"\n",
"Reasoning models like DeepSeek R1 are trained to go through a \"thinking process\" before providing an answer. Depending on the inference framework you use, they usually return:\n",
"\n",
"- **`reasoning_content`**: The internal reasoning steps that led to the conclusion \n",
"- **`content`**: The final answer or conclusion"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "q3GWbxFSDTLV"
},
"source": [
"## Supported MaaS Models on Vertex AI\n",
"\n",
"Currently, Vertex AI supports the following reasoning-capable MaaS models:\n",
"\n",
"| Model | Model ID | Location | Reasoning Support |\n",
"| :---- | :---- | :---- | :---- |\n",
"| DeepSeek R1 | `deepseek-ai/deepseek-r1-0528-maas` | us-central1 | ✅ Full reasoning |\n",
"| DeepSeek v3.1 | `deepseek-ai/deepseek-v3.1-maas` | us-west2 | ✅ With thinking parameter |\n",
"| GPT-OSS 20B | `openai/gpt-oss-20b-maas` | us-central1 | ✅ Full reasoning |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet openai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5303c05f7aa6"
},
"source": [
"### Import libraries\n",
"\n",
"Let's import our dependencies. We're grabbing google.auth to handle the login and, of course, the `openai` library that we'll use to actually build and send our prompts."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6fc324893334"
},
"outputs": [],
"source": [
"import openai\n",
"from google.auth import default\n",
"from google.auth.transport.requests import Request"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B19k4HYgE1m7"
},
"source": [
"### Set Up Authentication\n",
"\n",
"Here's where we actually grab the auth token. `default()` finds the credentials you provided earlier, and `credentials.refresh()` trades them in for a fresh, ready-to-use access token.\n",
"\n",
"This token is our ticket for accessing the Vertex AI API."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jdh-25I7E728"
},
"outputs": [],
"source": [
"# Get default credentials\n",
"credentials, project = default(\n",
" scopes=[\"https://www.googleapis.com/auth/cloud-platform\"]\n",
")\n",
"request = Request()\n",
"credentials.refresh(request)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BJT35G4aF0or"
},
"source": [
"## Basic Usage"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L-hPeruXF6Ev"
},
"source": [
"### Initialize the OpenAI Client\n",
"\n",
"We're creating an instance of the `openai` client and we're setting the `base_url` to our own Vertex AI endpoint. Then, for the `api_key`, we just pass in that Google Cloud access token we just generated. And just like that, the openai SDK is now a client for Vertex AI.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jW3ZT-7LF5RU"
},
"outputs": [],
"source": [
"MODEL_LOCATION = \"us-central1\"\n",
"\n",
"# Configure the Vertex AI endpoint\n",
"vertex_endpoint_url = f\"https://{MODEL_LOCATION}-aiplatform.googleapis.com/v1beta1/projects/{PROJECT_ID}/locations/{MODEL_LOCATION}/endpoints/openapi\"\n",
"\n",
"# Create OpenAI client with Vertex AI endpoint\n",
"client = openai.OpenAI(\n",
" base_url=vertex_endpoint_url,\n",
" api_key=credentials.token,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TGqyPKuIHcVb"
},
"source": [
"### Make a Request with Reasoning\n",
"\n",
"Time to ask our first question! We're calling the deepseek-r1 model and asking it to \"Think step by step.\"\n",
"\n",
"Check out the response. The model literally shows its work inside <think> tags before giving the final answer. This is great for debugging or for building apps where you want to show users how the AI got to its conclusion."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CUWQl7DdHcKx"
},
"outputs": [],
"source": [
"response = client.chat.completions.create(\n",
" model=\"deepseek-ai/deepseek-r1-0528-maas\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"How many r's are in strawberry? Think step by step\",\n",
" },\n",
" ],\n",
" stream=False,\n",
")\n",
"\n",
"response"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OQ52-Jswe-o1"
},
"source": [
"As you saw, DeepSeek model gives us everything in one big string. It is up to you to unpack that. It's just some simple Python string-slicing to pull out the text between the `<think>` tags. This lets you separate the \"behind-the-scenes\" reasoning from the clean, final answer you'd show to a user.\n",
"\n",
"> **Note**: You will see later that, for GPT-OSS family, reasoning & final response are populate in `reasoning_content` and `content` respectively."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7C5nVFCoe8eL"
},
"outputs": [],
"source": [
"response_text = response.choices[0].message.content\n",
"\n",
"start_tag = \"<think>\"\n",
"end_tag = \"</think>\"\n",
"\n",
"start_index = response_text.find(start_tag)\n",
"end_index = response_text.find(end_tag)\n",
"\n",
"reasoning_content = \"\"\n",
"if start_index != -1 and end_index != -1 and start_index < end_index:\n",
" reasoning_content = response_text[start_index + len(start_tag) : end_index].strip()\n",
" print(\"Extracted Reasoning:\\n\", reasoning_content)\n",
"else:\n",
" print(\"No reasoning content found in the expected format.\")\n",
"\n",
"# The rest of the content after </think>\n",
"final_answer = response_text[end_index + len(end_tag) :].strip()\n",
"print(\"\\nExtracted Final Answer:\\n\", final_answer)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JN0Ws9hbISra"
},
"source": [
"## Advanced Features"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lMkKweOkIW-h"
},
"source": [
"### Controlling Reasoning Behavior\n",
"\n",
"At this point, we know how the model can think but we can be more explicit about wanting it to think.\n",
"\n",
"To achieve that, we pass an `extra_body` parameter with `{\"thinking\": True}`. It's a good way to be sure the model knows you want to see its thought process.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2nAMpsBAJWp-"
},
"source": [
"#### For DeepSeek Models\n",
"\n",
"Use the `thinking` parameter to enable/disable reasoning."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gFJJnvGQIuOM"
},
"outputs": [],
"source": [
"MODEL_LOCATION = \"us-west2\"\n",
"\n",
"# Configure the Vertex AI endpoint\n",
"vertex_endpoint_url = (\n",
" f\"https://{MODEL_LOCATION}-aiplatform.googleapis.com/v1beta1/\"\n",
" f\"projects/{PROJECT_ID}/locations/{MODEL_LOCATION}/endpoints/openapi\"\n",
")\n",
"\n",
"# Create OpenAI client with Vertex AI endpoint\n",
"client = openai.OpenAI(\n",
" base_url=vertex_endpoint_url,\n",
" api_key=credentials.token,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "J2_BF-TfIhVT"
},
"outputs": [],
"source": [
"# Enable reasoning explicitly\n",
"response = client.chat.completions.create(\n",
" model=\"deepseek-ai/deepseek-v3.1-maas\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"How many r's are in strawberry? Think step by step\",\n",
" },\n",
" ],\n",
" extra_body={\"chat_template_kwargs\": {\"thinking\": True}},\n",
")\n",
"\n",
"\n",
"response"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HsgIXuQMI6TU"
},
"source": [
"#### For GPT-OSS Models\n",
"\n",
"Now let's chat with the GPT-OSS model. This one is slightly different. It has a `reasoning_effort` parameter you can set to control reasoning depth.\n",
"\n",
"Also the response object returns separated thoughts into a `reasoning_content` field and the final answer into content."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CpaCUShdI-rx"
},
"outputs": [],
"source": [
"MODEL_LOCATION = \"us-central1\"\n",
"\n",
"# Configure the Vertex AI endpoint\n",
"vertex_endpoint_url = (\n",
" f\"https://{MODEL_LOCATION}-aiplatform.googleapis.com/v1beta1/\"\n",
" f\"projects/{PROJECT_ID}/locations/{MODEL_LOCATION}/endpoints/openapi\"\n",
")\n",
"\n",
"# Create OpenAI client with Vertex AI endpoint\n",
"client = openai.OpenAI(\n",
" base_url=vertex_endpoint_url,\n",
" api_key=credentials.token,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "YK-beET3I-rx"
},
"outputs": [],
"source": [
"# Enable reasoning explicitly\n",
"response = client.chat.completions.create(\n",
" model=\"openai/gpt-oss-20b-maas\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"How many r's are in strawberry? Think step by step\",\n",
" },\n",
" ],\n",
" reasoning_effort=\"high\", # Options: \"low\", \"medium\", \"high\"\n",
")\n",
"\n",
"\n",
"response"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WLpUxvtTJaLT"
},
"source": [
"### Streaming Responses with Reasoning\n",
"\n",
"By setting `stream=True`, you get the response back token-by-token.\n",
"\n",
"This example below loops through the chunks as they arrive and prints them out. You can see the model \"thinking\" in real-time as the `reasoning_content` streams in, followed by the final answer."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "IvSZ_yXjJa8n"
},
"outputs": [],
"source": [
"stream = client.chat.completions.create(\n",
" model=\"openai/gpt-oss-20b-maas\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"How many r's are in strawberry? Think step by step\",\n",
" },\n",
" ],\n",
" stream=True,\n",
")\n",
"\n",
"reasoning_content = \"\"\n",
"final_content = \"\"\n",
"\n",
"for chunk in stream:\n",
" # Add a check for chunk.choices and chunk.choices[0]\n",
" if chunk.choices and chunk.choices[0].delta:\n",
" # Check for reasoning content\n",
" if hasattr(chunk.choices[0].delta, \"reasoning_content\"):\n",
" reasoning_chunk = chunk.choices[0].delta.reasoning_content\n",
" if reasoning_chunk:\n",
" reasoning_content += reasoning_chunk\n",
" print(f\"\\nThinking: {reasoning_chunk}\", end=\"\", flush=True)\n",
"\n",
" # Check for final content\n",
" elif hasattr(chunk.choices[0].delta, \"content\"):\n",
" content_chunk = chunk.choices[0].delta.content\n",
" if content_chunk:\n",
" final_content += content_chunk\n",
" print(f\"\\nAnswer: {content_chunk}\", end=\"\", flush=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uuyNSfO6LkOS"
},
"source": [
"### Structured Output with Reasoning\n",
"\n",
"How many times have you asked an LLM for JSON and gotten back a broken string? With `guided_json`, you define a Pydantic schema, pass it to the model, and it's guaranteed to return valid JSON that matches that schema.\n",
"\n",
"And you still get the `reasoning_content` to see how it came up with the JSON fields and values.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "M-jCttlmLmUB"
},
"outputs": [],
"source": [
"from pydantic import BaseModel\n",
"\n",
"class Person(BaseModel):\n",
" name: str\n",
" age: int\n",
"\n",
"\n",
"json_schema = Person.model_json_schema()\n",
"\n",
"response = client.chat.completions.create(\n",
" model=\"openai/gpt-oss-20b-maas\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Generate a JSON with a person's details\"}],\n",
" extra_body={\"guided_json\": json_schema},\n",
")\n",
"\n",
"print(\"Reasoning:\", response.choices[0].message.reasoning_content)\n",
"print(\"JSON Output:\", response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GvlKNgCMMCbW"
},
"source": [
"## Complete Example: Comparing Models\n",
"\n",
"Let's wrap it up with a model showdown that brings everything together. It loops through all the models we've looked at, makes sure to set up the client for the correct region for each one, and asks them all the same question.\n",
"\n",
"It's a perfect template for running your own model evaluations to see which one works best for your specific use case."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mOSIiE384Hg-"
},
"outputs": [],
"source": [
"# Test different models\n",
"models = [\n",
" {\"id\": \"deepseek-ai/deepseek-r1-0528-maas\", \"location\": \"us-central1\"},\n",
" {\"id\": \"deepseek-ai/deepseek-v3.1-maas\", \"location\": \"us-west2\"},\n",
" {\"id\": \"openai/gpt-oss-20b-maas\", \"location\": \"us-central1\"},\n",
"]\n",
"\n",
"question = \"What is 9.11 vs 9.8? Which is greater?\"\n",
"\n",
"for model_info in models:\n",
" # Create client for each model's location\n",
" vertex_endpoint_url = (\n",
" f\"https://{model_info['location']}-aiplatform.googleapis.com/v1beta1/\"\n",
" f\"projects/{PROJECT_ID}/locations/{model_info['location']}/endpoints/openapi\"\n",
" )\n",
"\n",
" client = openai.OpenAI(\n",
" base_url=vertex_endpoint_url,\n",
" api_key=credentials.token,\n",
" )\n",
"\n",
" # Make request\n",
" response = client.chat.completions.create(\n",
" model=model_info[\"id\"],\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant\"},\n",
" {\"role\": \"user\", \"content\": question},\n",
" ],\n",
" stream=False,\n",
" )\n",
"\n",
" print(f\"\\n{'=' * 50}\")\n",
" print(f\"Model: {model_info['id']}\")\n",
" print(f\"{'=' * 50}\")\n",
"\n",
" # Check if reasoning_content exists before accessing it\n",
" if (\n",
" hasattr(response.choices[0].message, \"reasoning_content\")\n",
" and response.choices[0].message.reasoning_content\n",
" ):\n",
" print(f\"Reasoning: {response.choices[0].message.reasoning_content[:200]}...\")\n",
" print(f\"Answer: {response.choices[0].message.content}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "B26hbgJnMeIq"
},
"source": [
"## Best Practices"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IwVE0xVP-ZxP"
},
"source": [
"### Token Management\n",
"\n",
"- Reasoning content adds to token usage \n",
"- Monitor `response.usage` for token counts \n",
"- Consider disabling reasoning for simple queries"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EkyQ-ZlE-eFf"
},
"source": [
"### Error Handling\n",
"\n",
"```py\n",
"try:\n",
" response = client.chat.completions.create(...)\n",
"except Exception as e:\n",
" # Refresh token if expired\n",
" credentials.refresh(request)\n",
" client.api_key = credentials.token\n",
" response = client.chat.completions.create(...)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gg-8cKWI-hoF"
},
"source": [
"### Performance Optimization\n",
"\n",
"- Use streaming for long reasoning chains \n",
"- Cache credentials to avoid repeated authentication \n",
"- Choose appropriate model locations for latency"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hnzZiOyf-ks-"
},
"source": [
"### 4. When to Use Reasoning\n",
"\n",
"✅ **Use reasoning for:**\n",
"\n",
"- Complex problem-solving \n",
"- Mathematical calculations \n",
"- Step-by-step analysis \n",
"- Debugging assistance\n",
"\n",
"❌ **Skip reasoning for:**\n",
"\n",
"- Simple factual queries \n",
"- Quick responses \n",
"- High-volume requests\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XmTCXYPKMYe2"
},
"source": [
"## Conclusion\n",
"\n",
"MaaS models on Vertex AI with vLLM reasoning provide powerful capabilities for transparent AI decision-making. By understanding how to properly configure and use these models, you can build applications that not only provide answers but also explain their reasoning process.\n",
"\n",
"## Additional Resources\n",
"\n",
"- [Vertex AI Open models documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/maas/use-open-models) \n",
"- [vLLM Reasoning Documentation](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#reasoning-with-llms) \n",
"- [OpenAI Python SDK](https://github.com/openai/openai-python)\n"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_oss_maas_reasoning_open_ai_sdk.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/cloud_run_ollama_gemma3_inference.ipynb
+582
View File
@@ -0,0 +1,582 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Serving Gemma 3 with Ollama on Cloud Run\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fcloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "83b98b0ba19c"
},
"source": [
"<img src=\"https://ollama.com/public/ollama.png\" height=\"200px\" alignment=\"center\"/>\n",
"<img src=\"https://cloud.google.com/static/architecture/images/ac-page-icons/card_google_cloud_partner.svg\" height=\"200px\">\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Vlad Kolesnikov](https://github.com/vladkol) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Gemma 3**](https://ai.google.dev/gemma) is a new generation of open models developed by Google. It is a collection of lightweight, state-of-the-art open models built from the same research and technology that powers our Gemini 2.0 models. Gemma 3 comes in a range of sizes (1B, 4B, 12B and 27B), allowing you to choose the best model for your specific hardware and performance needs. Gemma 3 models are available through platforms like Google AI Studio, Kaggle, and Hugging Face.\n",
"\n",
"> **[Cloud Run](https://cloud.google.com/run)**:\n",
"It's a serverless platform by Google Cloud for running containerized applications. It automatically scales and manages infrastructure, supporting various programming languages. Cloud Run now offers GPU acceleration for AI/ML workloads. With 30 seconds to the first token, Cloud Run is a perfect platform for serving lightweight models like Gemma.\n",
"\n",
"> **Note:** GPU support in Cloud Run is in preview. To use the GPU feature, you must request `Total Nvidia L4 GPU allocation, per project per region` quota under Cloud Run in the [Quotas and system limits page](https://cloud.google.com/run/quotas#increase).\n",
"\n",
"\n",
"> **[Ollama](ollama.com)**: is an open-source tool for easily running and deploying large language models locally. It offers simple management and usage of LLMs on personal computers or servers.\n",
"\n",
"This notebook showcase how to deploy [Google Gemma 3](https://developers.googleblog.com/en/introducing-gemma3) in Cloud Run, with the objective to build a simple API for chat or RAG applications.\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"1. Deploy Google Gemma 3 as an OpenAI-compatible API on Cloud Run using Ollama.\n",
"2. Build a custom container with Ollama to deploy any Large Language Model (LLM) of your choice.\n",
"3. Make requests to an API hosted on Cloud Run."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aOiPjM5DEPhK"
},
"source": [
"### Install Google Cloud SDK\n",
"\n",
"Make sure you Google Cloud SDK is installed (try running `gcloud version`) or [install it](https://cloud.google.com/sdk/docs/install) before executing this notebook.\n",
"\n",
"> If you are running in Colab or Vertex AI workbench, you have Google Cloud SDK installed."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HfAVa08RDDJB"
},
"source": [
"### Choose a model, a project, and a region to host the model\n",
"\n",
"Choose a Gemma 3 model to use, a Google Cloud project to host your Cloud Run service, and a region to host it in.\n",
"\n",
"If you don't have a project yet:\n",
"\n",
"1. [Create a project](https://console.cloud.google.com/projectcreate) in the Google Cloud Console.\n",
"2. Copy your `Project ID` from the project's [Settings page](https://console.cloud.google.com/iam-admin/settings).\n",
"\n",
"The project must have `Total Nvidia L4 GPU allocation, per project per region` quota allocated in the selected region.\n",
"To make sure it's available, check Cloud Run in the [Quotas and system limits page](https://console.cloud.google.com/iam-admin/quotas)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "TV0pbqJHDDJB"
},
"outputs": [],
"source": [
"# { display-mode: \"form\", run: \"auto\" }\n",
"\n",
"MODEL = \"gemma3:4b\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"REGION = \"us-central1\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"if PROJECT_ID == \"[your-project-id]\" or not PROJECT_ID:\n",
" print(\"Please specify your project id in PROJECT_ID variable.\")\n",
" raise KeyboardInterrupt\n",
"\n",
"MODEL_NAME_ESCAPED = MODEL.translate(str.maketrans(\".:/\", \"---\"))\n",
"SERVICE_NAME = f\"ollama--{MODEL_NAME_ESCAPED}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Run the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xc8Jm1P3Y7fs"
},
"outputs": [],
"source": [
"!gcloud auth print-identity-token -q &> /dev/null || gcloud auth login --project=\"{PROJECT_ID}\" --update-adc --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l728UOEPDDJB"
},
"source": [
"## Prepare container image\n",
"\n",
"First, let's create a Docker file for a container with the model embedded into it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "glBn9gPKDDJB"
},
"outputs": [],
"source": [
"%%writefile Dockerfile\n",
"\n",
"FROM ollama/ollama:0.6.0\n",
"\n",
"ARG MODEL\n",
"\n",
"# Set the model name\n",
"ENV MODEL=$MODEL\n",
"\n",
"# Set the host and port to listen on\n",
"ENV OLLAMA_HOST 0.0.0.0:8080\n",
"\n",
"# Set the directory to store model weight files\n",
"ENV OLLAMA_MODELS /models\n",
"\n",
"# Reduce the verbosity of the logs\n",
"ENV OLLAMA_DEBUG false\n",
"\n",
"# Do not unload model weights from the GPU\n",
"ENV OLLAMA_KEEP_ALIVE -1\n",
"\n",
"# Start the ollama server and download the model weights\n",
"RUN ollama serve & sleep 5 && ollama pull $MODEL\n",
"\n",
"# At startup time we start the server and run a dummy request\n",
"# to request the model to be loaded in the GPU memory\n",
"ENTRYPOINT [\"/bin/sh\"]\n",
"CMD [\"-c\", \"ollama serve & (ollama run $MODEL 'Say one word' &) && wait\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T4gS8ovMDDJB"
},
"source": [
"Second, we create a Cloud Build file to use for building and pushing our container image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1dV1_cMDDDJB"
},
"outputs": [],
"source": [
"%%writefile cloudbuild.yaml\n",
"\n",
"steps:\n",
"- name: 'gcr.io/cloud-builders/docker'\n",
" id: build\n",
" entrypoint: 'bash'\n",
" args:\n",
" - -c\n",
" - |\n",
" docker buildx build --tag=${_IMAGE} --build-arg MODEL=${_MODEL} .\n",
"\n",
"images: [\"${_IMAGE}\"]\n",
"\n",
"substitutions:\n",
" _IMAGE: '${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_AR_REPO_NAME}/${_SERVICE_NAME}'\n",
"\n",
"options:\n",
" dynamicSubstitutions: true\n",
" machineType: \"E2_HIGHCPU_32\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vbDiABJcDDJC"
},
"source": [
"## Build Container Image and Deploy Cloud Run Service\n",
"\n",
"We are ready to build our container image and deploy Cloud Run service.\n",
"\n",
"The script below performs the following actions:\n",
"\n",
"* Enables necessary APIs.\n",
"* Creates an Artifact Repository for the image.\n",
"* Creates a Service Account for the service.\n",
"* Submits a Cloud Build job to create and push the container image.\n",
"* Deploys the Cloud Run service.\n",
"\n",
"> The script may take 10-45 minutes to finish.\n",
"\n",
"Note the following important flags in Cloud Build deployment command:\n",
"\n",
"* `--concurrency 4` is set to match the value of the environment variable `OLLAMA_NUM_PARALLEL`.\n",
"* `--gpu 1` with `--gpu-type nvidia-l4` assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.\n",
"`--no-allow-authenticated` restricts unauthenticated access to the service.\n",
"By keeping the service private, you can rely on Cloud Run's built-in [Identity and Access Management (IAM)](https://cloud.google.com/iam) authentication for service-to-service communication.\n",
"* `--no-cpu-throttling` is required for enabling GPU.\n",
"* `--service-account` the service identity of the service.\n",
"* `--max-instances` sets maximum number of instances of the service.\n",
"It has to be equal to or lower than your project's NVIDIA L4 GPU (`Total Nvidia L4 GPU allocation, per project per region`) quota.\n",
"\n",
"For optimal GPU utilization, increase `--concurrency`, keeping it within twice the value of `OLLAMA_NUM_PARALLEL`.\n",
"While this leads to request queuing in Ollama, it can help improve utilization:\n",
"Ollama instances can immediately process requests from their queue, and the queues help absorb traffic spikes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TXg7IYU1DDJC"
},
"outputs": [],
"source": [
"%%writefile deploy.sh\n",
"\n",
"PROJECT_ID=$1\n",
"REGION=$2\n",
"MODEL_ID=\"${3}\"\n",
"SERVICE_NAME=\"${4}\"\n",
"AR_REPO_NAME=\"ollama-repo\"\n",
"SERVICE_ACCOUNT=\"ollama-cloud-run-sa\"\n",
"SERVICE_ACCOUNT_ADDRESS=\"${SERVICE_ACCOUNT}@$PROJECT_ID.iam.gserviceaccount.com\"\n",
"MAX_INSTANCES=1 # Adjust this value to match your Cloud Run L4 GPU quota (\"Total Nvidia L4 GPU allocation, per project per region\", NvidiaL4GpuAllocPerProjectRegion, run.googleapis.com/nvidia_l4_gpu_allocation)\n",
"\n",
"echo \"Enabling APIs in project ${PROJECT_ID}.\"\n",
"gcloud services enable run.googleapis.com \\\n",
" cloudbuild.googleapis.com \\\n",
" artifactregistry.googleapis.com \\\n",
" --project ${PROJECT_ID} \\\n",
" --quiet\n",
"\n",
"set -e\n",
"\n",
"# Creating the service account if doesn't exist.\n",
"sa_list=$(gcloud iam service-accounts list --quiet --format 'value(email)' --project $PROJECT_ID --filter=email:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com 2>/dev/null)\n",
"if [ -z \"${sa_list}\" ]; then\n",
" echo \"Creating Service Account ${SERVICE_ACCOUNT}.\"\n",
" gcloud iam service-accounts create $SERVICE_ACCOUNT \\\n",
" --project ${PROJECT_ID} \\\n",
" --display-name=\"${SERVICE_ACCOUNT} - Cloud Run Service Account\"\n",
"fi\n",
"\n",
"# Creating the Artifacts Repository if doesn't exist\n",
"repo_list=$(gcloud artifacts repositories list --format 'value(name)' --filter=name=\"projects/${PROJECT_ID}/locations/${REGION}/repositories/${AR_REPO_NAME}\" --project ${PROJECT_ID} --quiet --location ${REGION} 2>/dev/null)\n",
"if [ -z \"${repo_list}\" ]; then\n",
" echo \"Creating Artifact Registry ${AR_REPO_NAME}.\"\n",
" gcloud artifacts repositories create $AR_REPO_NAME \\\n",
" --repository-format docker \\\n",
" --location ${REGION} \\\n",
" --project=${PROJECT_ID}\n",
"fi\n",
"\n",
"echo \"Building container image.\"\n",
"gcloud builds submit --config=cloudbuild.yaml --project=${PROJECT_ID} . \\\n",
" --suppress-logs \\\n",
" --substitutions \\\n",
" _AR_REPO_NAME=$AR_REPO_NAME,_REGION=$REGION,_SERVICE_NAME=$SERVICE_NAME,_MODEL=$MODEL_ID\n",
"rm -f cloudbuild.yaml\n",
"rm -f Dockerfile\n",
"\n",
"echo \"Deploying Service ${SERVICE_NAME}.\"\n",
"gcloud beta run deploy $SERVICE_NAME \\\n",
" --project=${PROJECT_ID} \\\n",
" --image=${REGION}-docker.pkg.dev/$PROJECT_ID/$AR_REPO_NAME/$SERVICE_NAME \\\n",
" --service-account $SERVICE_ACCOUNT_ADDRESS \\\n",
" --cpu=8 \\\n",
" --memory=32Gi \\\n",
" --gpu=1 --gpu-type=nvidia-l4 \\\n",
" --concurrency 4 \\\n",
" --set-env-vars OLLAMA_NUM_PARALLEL=4 \\\n",
" --region ${REGION} \\\n",
" --no-allow-unauthenticated \\\n",
" --max-instances ${MAX_INSTANCES} \\\n",
" --no-cpu-throttling \\\n",
" --timeout 1h\n",
"\n",
"SERVICE_URL=$(gcloud run services describe ${SERVICE_NAME} --project=${PROJECT_ID} --region $REGION --format 'value(status.url)' --quiet)\n",
"echo \"✅ Success!\"\n",
"echo \"🚀 Service URL: ${SERVICE_URL}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e6L2dVGyOAxB"
},
"outputs": [],
"source": [
"!/bin/bash ./deploy.sh \"{PROJECT_ID}\" \"{REGION}\" \"{MODEL}\" \"{SERVICE_NAME}\" && rm -f ./deploy.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dgaJ62rmDDJC"
},
"source": [
"## Test the deployed service\n",
"\n",
"Now, let's test the service you deployed.\n",
"\n",
"First, simply by using `cURL`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iX7LmWwGDDJC"
},
"outputs": [],
"source": [
"%%bash -s $MODEL $SERVICE_NAME $PROJECT_ID $REGION\n",
"\n",
"PROMPT=\"Hello!\"\n",
"SERVICE_URL=$(gcloud run services describe ${2} --project ${3} --region ${4} --format 'value(status.url)' --quiet)\n",
"AUTH_TOKEN=$(gcloud auth print-identity-token -q)\n",
"\n",
"curl -s -X POST \"${SERVICE_URL}/api/generate\" \\\n",
"-H \"Authorization: Bearer ${AUTH_TOKEN}\" \\\n",
"-H \"Content-Type: application/json\" \\\n",
"-d '{ \"model\": \"'${1}'\", \"prompt\": \"'${PROMPT}'\", \"max_tokens\": 100, \"stream\": false}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "63oQqBNmDDJC"
},
"source": [
"### Ollama Python Library\n",
"\n",
"You can also use Ollama Python Library to make requests to the service you deployed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "h8G3te5pDDJC"
},
"outputs": [],
"source": [
"# Install Ollama Python Library\n",
"%pip install ollama -q"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "X-2TbV6tDDJC"
},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"from ollama import Client\n",
"\n",
"identity_token = (\n",
" subprocess.check_output(\"gcloud auth print-identity-token -q\", shell=True)\n",
" .decode()\n",
" .strip()\n",
")\n",
"service_url = (\n",
" subprocess.check_output(\n",
" (\n",
" \"gcloud run services describe \"\n",
" f\"{SERVICE_NAME} --project={PROJECT_ID} \"\n",
" f\"--region={REGION} \"\n",
" \"--format='value(status.url)' -q\"\n",
" ),\n",
" shell=True,\n",
" )\n",
" .decode()\n",
" .strip()\n",
")\n",
"client = Client(host=service_url, headers={\"Authorization\": f\"Bearer {identity_token}\"})\n",
"stream = client.chat(\n",
" model=MODEL,\n",
" messages=[{\"role\": \"user\", \"content\": \"Why is the sky blue?\"}],\n",
" stream=True,\n",
")\n",
"for chunk in stream:\n",
" print(chunk[\"message\"][\"content\"], end=\"\", flush=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FFL2V_ClDDJD"
},
"source": [
"## Conclusion\n",
"Congratulations! 💎 Now you know how to deploy Gemma 3 with Ollama to Cloud Run powered by a GPU!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s1blF2ziDDJD"
},
"source": [
"To delete the Cloud Run service you created, you can uncomment and run the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VbhAz7-9DDJD"
},
"outputs": [],
"source": [
"# !gcloud run services delete $SERVICE_NAME --project $PROJECT_ID --region $LOCATION --quiet"
]
}
],
"metadata": {
"colab": {
"name": "cloud_run_ollama_gemma3_inference.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/cloud_run_ollama_qwen3_inference.ipynb
+736
View File
@@ -0,0 +1,736 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Running Qwen 3 with Ollama in Cloud Run for Agents\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fcloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Vlad Kolesnikov](https://github.com/vladkol) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b1455cd3766f"
},
"source": [
"<style>\n",
"td, th {\n",
" border: none!important;\n",
"}\n",
"</style>\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://camo.githubusercontent.com/8793b3b4014d538b367ec8819dcca85e79cb8d910c808fa7849e3cd85e2ebe79/68747470733a2f2f7169616e77656e2d7265732e6f73732d616363656c65726174652d6f766572736561732e616c6979756e63732e636f6d2f6c6f676f5f7177656e332e706e67\" width=\"100px\"/>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://ollama.com/public/ollama.png\" height=\"50px\"/>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://google.github.io/adk-docs/assets/agent-development-kit.png\" height=\"50px\">\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://www.gstatic.com/bricks/image/f2e0986a2802c0b6c4be7f1355599d5aadfb21a63b7e9643d96697ff9334a1e1.svg\" height=\"50px\">\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Qwen 3**](https://qwenlm.github.io/blog/qwen3/) is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models.\n",
"It supports thinking and function calling.\n",
"\n",
"> **[Cloud Run](https://cloud.google.com/run)**:\n",
"It's a serverless platform by Google Cloud for running containerized applications. It automatically scales and manages infrastructure, supporting various programming languages. Cloud Run now offers GPU acceleration for AI/ML workloads. With 30 seconds to the first token, Cloud Run is a perfect platform for serving lightweight models.\n",
"\n",
"> **Note:** [GPU support in Cloud Run](https://cloud.google.com/run/docs/configuring/services/gpu) is Generally Available.\n",
"To use the GPU feature, your project must have `Total Nvidia L4 GPU allocation without zonal redundancy, per project per region`.\n",
"\n",
"> **[Ollama](ollama.com)**: is an open-source tool for easily running and deploying large language models locally. It offers simple management and usage of LLMs on personal computers or servers.\n",
"\n",
"This notebook showcase how to deploy [Qwen 3](https://developers.googleblog.com/en/introducing-gemma3) in Cloud Run,\n",
"with the objective to an API for running AI Agents built with Google [Agent Development Kit](https://google.github.io/adk-docs/).\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"1. Deploy Qwen 3 as an OpenAI-compatible API on Cloud Run using Ollama.\n",
"2. Build a custom container with Ollama to deploy any Large Language Model (LLM) of your choice.\n",
"3. Make requests to an API hosted on Cloud Run.\n",
"4. Create and run an Agent that uses Qwen 3.\n",
"\n",
"We will build an agent using [Agent Development Kit](https://google.github.io/adk-docs/) - a flexible and modular model-agnostic framework for developing and deploying AI agents."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aOiPjM5DEPhK"
},
"source": [
"### Install Google Cloud CLI\n",
"\n",
"Make sure you Google Cloud CLI is installed (try running `gcloud version`) or [install it](https://cloud.google.com/sdk/docs/install) before executing this notebook.\n",
"\n",
"> If you are running in Colab or Vertex AI workbench, you already have Google Cloud CLI installed."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b0b84cb331d5"
},
"source": [
"### Install Agent Development Kit and other required packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "330f4fbf7da9"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai google-adk litellm"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HfAVa08RDDJB"
},
"source": [
"### Choose a model, a project, and a region to host the model\n",
"\n",
"[Choose a Qwen 3 model](https://ollama.com/library/qwen3) to use, a Google Cloud project to host your Cloud Run service, and a region to host it in.\n",
"Ollama offers multiple sizes with different quantization.\n",
"In this notebook, we use [Qwen3:8b](https://ollama.com/library/qwen3:8b) with `Q4_K_M` quantization.\n",
"\n",
"For Google Cloud project, if you don't have a project yet:\n",
"\n",
"1. [Create a project](https://console.cloud.google.com/projectcreate) in the Google Cloud Console.\n",
"2. Copy your `Project ID` from the project's [Settings page](https://console.cloud.google.com/iam-admin/settings).\n",
"\n",
"The project must have `Total Nvidia L4 GPU allocation without zonal redundancy, per project per region` quota allocated in the selected region.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "TV0pbqJHDDJB"
},
"outputs": [],
"source": [
"# { display-mode: \"form\", run: \"auto\" }\n",
"\n",
"MODEL = \"qwen3:8b\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"REGION = \"us-central1\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"if PROJECT_ID == \"[your-project-id]\" or not PROJECT_ID:\n",
" print(\"Please specify your project id in PROJECT_ID variable.\")\n",
" raise KeyboardInterrupt\n",
"\n",
"MODEL_NAME_ESCAPED = MODEL.translate(str.maketrans(\".:/\", \"---\"))\n",
"SERVICE_NAME = f\"ollama--{MODEL_NAME_ESCAPED}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6c36c31ee2de"
},
"outputs": [],
"source": [
"### Python dependency imports\n",
"import os\n",
"import subprocess\n",
"from datetime import datetime"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Run the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xc8Jm1P3Y7fs"
},
"outputs": [],
"source": [
"!gcloud auth print-identity-token -q &> /dev/null || gcloud auth login --project=\"{PROJECT_ID}\" --update-adc --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l728UOEPDDJB"
},
"source": [
"## Prepare serving container\n",
"\n",
"First, let's create a Docker file for a container with the model embedded into it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f56f9803255a"
},
"outputs": [],
"source": [
"!rm -f Dockerfile\n",
"!echo \"ARG _MODEL=\\\"{MODEL}\\\"\" > Dockerfile"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "glBn9gPKDDJB"
},
"outputs": [],
"source": [
"%%writefile -a Dockerfile\n",
"\n",
"FROM ollama/ollama:latest\n",
"ARG _MODEL\n",
"\n",
"# Set the host and port to listen on\n",
"ENV OLLAMA_HOST 0.0.0.0:8080\n",
"\n",
"# Set the directory to store model weight files\n",
"ENV OLLAMA_MODELS /models\n",
"\n",
"# Reduce the verbosity of the logs\n",
"ENV OLLAMA_DEBUG false\n",
"\n",
"# Do not unload model weights from the GPU\n",
"ENV OLLAMA_KEEP_ALIVE -1\n",
"\n",
"# Start the ollama server and download the model weights\n",
"RUN ollama serve & sleep 5 && ollama pull $_MODEL\n",
"\n",
"# At startup time we start the server and run a dummy request\n",
"# to request the model to be loaded in the GPU memory\n",
"ENTRYPOINT [\"/bin/sh\"]\n",
"CMD [\"-c\", \"ollama serve & (ollama run $_MODEL 'Say one word' &) && wait\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vbDiABJcDDJC"
},
"source": [
"## Build Container Image and Deploy Cloud Run Service\n",
"\n",
"We are ready to build our container image and deploy Cloud Run service.\n",
"\n",
"The script below performs the following actions:\n",
"\n",
"* Enables necessary APIs.\n",
"* Creates an Artifact Repository for the image.\n",
"* Creates a Service Account for the service.\n",
"* Submits a Cloud Build job to create and push the container image.\n",
"* Deploys the Cloud Run service.\n",
"\n",
"> The script may take 10-45 minutes to finish.\n",
"\n",
"Note the following important flags in Cloud Build deployment command:\n",
"\n",
"* `--concurrency 4` is set to match the value of the environment variable `OLLAMA_NUM_PARALLEL`.\n",
"* `--gpu 1` with `--gpu-type nvidia-l4` assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.\n",
"* `--no-gpu-zonal-redundancy` allows using the default GPU quota.\n",
"* `--no-allow-authenticated` restricts unauthenticated access to the service.\n",
"By keeping the service private, you can rely on Cloud Run's built-in [Identity and Access Management (IAM)](https://cloud.google.com/iam) authentication for service-to-service communication.\n",
"* `--no-cpu-throttling` is required for enabling GPU.\n",
"* `--service-account` the service identity of the service.\n",
"* `--max-instances` sets maximum number of instances of the service.\n",
"It has to be equal to or lower than your project's NVIDIA L4 GPU quota.\n",
"\n",
"For optimal GPU utilization, increase `--concurrency`, keeping it within twice the value of `OLLAMA_NUM_PARALLEL`.\n",
"While this leads to request queuing in Ollama, it can help improve utilization:\n",
"Ollama instances can immediately process requests from their queue, and the queues help absorb traffic spikes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TXg7IYU1DDJC"
},
"outputs": [],
"source": [
"%%writefile deploy.sh\n",
"\n",
"PROJECT_ID=$1\n",
"REGION=$2\n",
"MODEL_ID=\"${3}\"\n",
"SERVICE_NAME=\"${4}\"\n",
"SERVICE_ACCOUNT=\"ollama-cloud-run-sa\"\n",
"SERVICE_ACCOUNT_ADDRESS=\"${SERVICE_ACCOUNT}@$PROJECT_ID.iam.gserviceaccount.com\"\n",
"MAX_INSTANCES=1 # Adjust this value to match your Cloud Run L4 GPU quota\n",
"\n",
"echo \"Enabling APIs in project ${PROJECT_ID}.\"\n",
"gcloud services enable run.googleapis.com \\\n",
" cloudbuild.googleapis.com \\\n",
" artifactregistry.googleapis.com \\\n",
" --project ${PROJECT_ID} \\\n",
" --quiet\n",
"\n",
"set -e\n",
"\n",
"# Creating the service account if doesn't exist.\n",
"sa_list=$(gcloud iam service-accounts list --quiet --format 'value(email)' --project $PROJECT_ID --filter=email:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com 2>/dev/null)\n",
"if [ -z \"${sa_list}\" ]; then\n",
" echo \"Creating Service Account ${SERVICE_ACCOUNT}.\"\n",
" gcloud iam service-accounts create $SERVICE_ACCOUNT \\\n",
" --project ${PROJECT_ID} \\\n",
" --display-name=\"${SERVICE_ACCOUNT} - Cloud Run Service Account\"\n",
"fi\n",
"\n",
"echo \"Deploying Service ${SERVICE_NAME}. It will take a few minutes...\"\n",
"gcloud beta run deploy $SERVICE_NAME \\\n",
" --source . \\\n",
" --project=${PROJECT_ID} \\\n",
" --service-account $SERVICE_ACCOUNT_ADDRESS \\\n",
" --cpu=8 \\\n",
" --memory=32Gi \\\n",
" --gpu=1 \\\n",
" --gpu-type=nvidia-l4 \\\n",
" --concurrency 4 \\\n",
" --set-env-vars OLLAMA_NUM_PARALLEL=4 \\\n",
" --region ${REGION} \\\n",
" --no-allow-unauthenticated \\\n",
" --max-instances ${MAX_INSTANCES} \\\n",
" --no-cpu-throttling \\\n",
" --timeout 1h \\\n",
" --no-gpu-zonal-redundancy \\\n",
" --quiet \\\n",
" --no-user-output-enabled\n",
"\n",
"rm -f ./Dockerfile # Cleanup\n",
"\n",
"SERVICE_URL=$(gcloud run services describe ${SERVICE_NAME} --project=${PROJECT_ID} --region $REGION --format 'value(status.url)' --quiet)\n",
"echo \"✅ Success!\"\n",
"echo \"🚀 Service URL: ${SERVICE_URL}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e6L2dVGyOAxB"
},
"outputs": [],
"source": [
"!/bin/bash ./deploy.sh \"{PROJECT_ID}\" \"{REGION}\" \"{MODEL}\" \"{SERVICE_NAME}\" && rm -f ./deploy.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dgaJ62rmDDJC"
},
"source": [
"### Test the deployed service\n",
"\n",
"Now, let's test the service you deployed.\n",
"\n",
"First, simply by using `cURL`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iX7LmWwGDDJC"
},
"outputs": [],
"source": [
"%%bash -s $MODEL $SERVICE_NAME $PROJECT_ID $REGION\n",
"\n",
"PROMPT=\"Hello!\"\n",
"SERVICE_URL=$(gcloud run services describe ${2} --project ${3} --region ${4} --format 'value(status.url)' --quiet)\n",
"AUTH_TOKEN=$(gcloud auth print-identity-token -q)\n",
"\n",
"curl -s -X POST \"${SERVICE_URL}/api/generate\" \\\n",
"-H \"Authorization: Bearer ${AUTH_TOKEN}\" \\\n",
"-H \"Content-Type: application/json\" \\\n",
"-d '{ \"model\": \"'${1}'\", \"prompt\": \"'${PROMPT}'\", \"max_tokens\": 100, \"stream\": false}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2b60e54c901d"
},
"source": [
"## Create an AI Agent with Qwen 3"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1e1f3cb538ed"
},
"source": [
"#### Retrieve an Identity Token\n",
"\n",
"Cloud Run with authentication by\n",
"[Google Cloud IAM](https://cloud.google.com/iam/docs/) requires an [identity token](https://cloud.google.com/docs/authentication/get-id-token) in every request's authentication header."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "X-2TbV6tDDJC"
},
"outputs": [],
"source": [
"auth_id_token = (\n",
" subprocess.check_output(\"gcloud auth print-identity-token -q\", shell=True)\n",
" .decode()\n",
" .strip()\n",
")\n",
"service_url = (\n",
" subprocess.check_output(\n",
" (\n",
" \"gcloud run services describe \"\n",
" f\"{SERVICE_NAME} --project={PROJECT_ID} \"\n",
" f\"--region={REGION} \"\n",
" \"--format='value(status.url)' -q\"\n",
" ),\n",
" shell=True,\n",
" )\n",
" .decode()\n",
" .strip()\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2f5ae9755e13"
},
"source": [
"#### Create and run an Agent\n",
"\n",
"We create a simple agent that can answer questions about current time.\n",
"\n",
"First, we make a tool that returns current user's time with time zone offset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d66169bd7a12"
},
"outputs": [],
"source": [
"def get_current_time() -> str:\n",
" \"\"\"Returns user's local time and timezone offset.\n",
"\n",
" Returns:\n",
" str: Time in ISO format with user's timezone offset.\n",
" \"\"\"\n",
" return datetime.now().astimezone().isoformat()\n",
"\n",
"\n",
"print(get_current_time())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b1a1ae76f3bb"
},
"source": [
"Now, we create an agent that can use our tool to answer user's question."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d7c2b8951050"
},
"outputs": [],
"source": [
"from google.adk import Runner\n",
"from google.adk.agents import Agent\n",
"from google.adk.artifacts import InMemoryArtifactService\n",
"from google.adk.models.lite_llm import LiteLlm\n",
"from google.adk.sessions import InMemorySessionService, Session\n",
"from google.genai import types\n",
"\n",
"os.environ[\"OLLAMA_API_BASE\"] = service_url # still required for LiteLlm to work\n",
"\n",
"root_agent = Agent(\n",
" name=\"time_agent\",\n",
" model=LiteLlm(\n",
" model=f\"openai/{MODEL}\",\n",
" api_base=service_url + \"/v1\",\n",
" api_key=auth_id_token,\n",
" temperature=0.1, # for stable function calling\n",
" ),\n",
" description=(\"Agent to answer questions about current time.\"),\n",
" # Agent instructions with `/no_think` for Qwen 3 to run faster.\n",
" instruction=\"\"\"\n",
" You are a helpful agent who can answer user questions\n",
" about the current local time.\n",
" Always output date and time is human-readable form.\n",
" /no_think\n",
" \"\"\",\n",
" tools=[get_current_time],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0763595c0eff"
},
"source": [
"Initialize the runtime."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2d8631d9da3f"
},
"outputs": [],
"source": [
"app_name = \"my_app\"\n",
"user_id_1 = \"user1\"\n",
"session_service = InMemorySessionService()\n",
"artifact_service = InMemoryArtifactService()\n",
"runner = Runner(\n",
" app_name=app_name,\n",
" agent=root_agent,\n",
" artifact_service=artifact_service,\n",
" session_service=session_service,\n",
")\n",
"new_session = await session_service.create_session(app_name=app_name, user_id=user_id_1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "64b9387644c6"
},
"source": [
"And run our agent!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f09d7a5e0ca8"
},
"outputs": [],
"source": [
"async def run_prompt(session: Session, new_message: str):\n",
" content = types.Content(role=\"user\", parts=[types.Part.from_text(text=new_message)])\n",
" print(\"** User says:\", new_message)\n",
" async for event in runner.run_async(\n",
" user_id=user_id_1,\n",
" session_id=session.id,\n",
" new_message=content,\n",
" ):\n",
" if not event.content or not event.content.parts:\n",
" continue\n",
" print(f\"** {event.author}:\")\n",
" for part in event.content.parts:\n",
" if part.function_call and part.function_call.name:\n",
" print(\n",
" f\"\\t#### Calling `{part.function_call.name}` \"\n",
" f\"with args: {part.function_call.args}\"\n",
" )\n",
" elif part.function_response and part.function_response.response:\n",
" print(f\"\\t### Function call result: {part.function_response.response}\")\n",
" elif part.text and part.text.strip():\n",
" print(f\"\\t{part.text.strip()}\")\n",
"\n",
"\n",
"QUESTION = \"What time is it now?\"\n",
"await run_prompt(new_session, QUESTION) # if not Jupyter, wrap in asyncio.run"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FFL2V_ClDDJD"
},
"source": [
"## Conclusion\n",
"Congratulations! You can now use Qwen 3 for running your AI Agents built with Agent Development Kit in Cloud Run!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s1blF2ziDDJD"
},
"source": [
"To delete the Cloud Run service you created, you can uncomment and run the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VbhAz7-9DDJD"
},
"outputs": [],
"source": [
"# !gcloud run services delete $SERVICE_NAME --project $PROJECT_ID --region $LOCATION --quiet"
]
}
],
"metadata": {
"colab": {
"name": "cloud_run_ollama_qwen3_inference.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/cloud_run_vllm_gemma3_inference.ipynb
+685
View File
@@ -0,0 +1,685 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Serving Gemma 3 with vLLM on Cloud Run\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fcloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "83b98b0ba19c"
},
"source": [
"<img src=\"https://docs.vllm.ai/en/latest/_images/vllm-logo-text-light.png\" width=\"200px\" alignment=\"center\"/>\n",
"<img src=\"https://cloud.google.com/static/architecture/images/ac-page-icons/card_google_cloud_partner.svg\" width=\"100px\">\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Vlad Kolesnikov](https://github.com/vladkol) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Gemma 3**](https://ai.google.dev/gemma) is a new generation of open models developed by Google. It is a collection of lightweight, state-of-the-art open models built from the same research and technology that powers our Gemini 2.0 models. Gemma 3 comes in a range of sizes (1B, 4B, 12B and 27B), allowing you to choose the best model for your specific hardware and performance needs. Gemma 3 models are available through platforms like Google AI Studio, Kaggle, and Hugging Face.\n",
"\n",
"> **[Cloud Run](https://cloud.google.com/run)**:\n",
"It's a serverless platform by Google Cloud for running containerized applications. It automatically scales and manages infrastructure, supporting various programming languages. Cloud Run now offers GPU acceleration for AI/ML workloads. With 30 seconds to the first token, Cloud Run is a perfect platform for serving lightweight models like Gemma.\n",
"\n",
"> **Note:** GPU support in Cloud Run is in preview. To use the GPU feature, you must request `Total Nvidia L4 GPU allocation, per project per region` quota under Cloud Run in the [Quotas and system limits page](https://cloud.google.com/run/quotas#increase).\n",
"\n",
"\n",
"> **[vLLM](https://docs.vllm.ai/)**: is a fast and easy-to-use library for LLM inference and serving. It provides high-throughput serving, optimized model execution, and tensor parallelism.\n",
"\n",
"This notebook showcase how to deploy [Google Gemma 3](https://developers.googleblog.com/en/introducing-gemma3) in Cloud Run, with the objective to build a simple API for chat or RAG applications.\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"1. Deploy Google Gemma 3 as an OpenAI-compatible API on Cloud Run using vLLM.\n",
"2. Build a custom container with vLLM to deploy any Large Language Model (LLM) of your choice.\n",
"3. Make requests to an API hosted on Cloud Run."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Cloud SDK and HuggingFace Hub dependencies\n",
"\n",
"Make sure you Google Cloud SDK is installed (try running `gcloud version`) or [install it](https://cloud.google.com/sdk/docs/install) before executing this notebook.\n",
"\n",
"> If you are running in Colab or Vertex AI workbench, you have Google Cloud SDK installed.\n",
"\n",
"You also need to install `huggingface_hub` Python package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Tm31a7_cDbeu"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet huggingface_hub"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MajzyxmFDbeu"
},
"source": [
"### Choose a model, a project, and a region to host the model\n",
"\n",
"Choose a Gemma 3 model to use, a Google Cloud project to host your Cloud Run service, and a region to host it in.\n",
"\n",
"If you don't have a project yet:\n",
"\n",
"1. [Create a project](https://console.cloud.google.com/projectcreate) in the Google Cloud Console.\n",
"2. Copy your `Project ID` from the project's [Settings page](https://console.cloud.google.com/iam-admin/settings).\n",
"\n",
"The project must have `Total Nvidia L4 GPU allocation, per project per region` quota allocated in the selected region.\n",
"To make sure it's available, check Cloud Run in the [Quotas and system limits page](https://console.cloud.google.com/iam-admin/quotas)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "Do_Z5bVNDbeu"
},
"outputs": [],
"source": [
"# { display-mode: \"form\", run: \"auto\" }\n",
"\n",
"MODEL = \"google/gemma-3-4b-it\" # @param {type:\"string\"}\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"REGION = \"us-central1\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"if PROJECT_ID == \"[your-project-id]\" or not PROJECT_ID:\n",
" print(\"Please specify your project id in PROJECT_ID variable.\")\n",
" raise KeyboardInterrupt\n",
"\n",
"SERVICE_NAME = f\"vllm--{MODEL.replace('.', '-').replace('/','--')}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Run the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c93d96bc9c43"
},
"outputs": [],
"source": [
"!gcloud auth print-identity-token -q &> /dev/null || gcloud auth login --project=\"{PROJECT_ID}\" --update-adc --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9af3e57f89a"
},
"source": [
"### Authenticate your Hugging Face account\n",
"\n",
"As Google Gemma (https://huggingface.co/google/gemma-3-4b-it) are gated models, you need to have a Hugging Face Hub account, and accept the Google's usage license for Gemma. Once that's done, you need to generate a new user access token with read-only access so that the weights can be downloaded from the Hub.\n",
"\n",
"> Note that the user access token can only be generated via [the Hugging Face Hub UI](https://huggingface.co/settings/tokens/new), where you can either select read-only access to your account, or follow the recommendations and generate a fine-grained token with read-only access to [`google/gemma-3-4b-it`](https://huggingface.co/google/gemma-3-4b-it).\n",
">\n",
"> Even you had an access token before accepting the Gemma license, you need to generate a new one after accepting it.\n",
"\n",
"Then you can use `huggingface_hub` for the authentication with the token generated in advance. So that then the token can be safely retrieved via `huggingface_hub.get_token`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8d836e0210fe"
},
"outputs": [],
"source": [
"from huggingface_hub import get_token, interpreter_login\n",
"\n",
"interpreter_login(new_session=False)\n",
"HF_TOKEN = get_token()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c71a4314c250"
},
"source": [
"Read more about [Hugging Face Security](https://huggingface.co/docs/hub/en/security), specifically about [Hugging Face User Access Tokens](https://huggingface.co/docs/hub/en/security-tokens)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gwSj4-j9Dbev"
},
"source": [
"## Prepare container image\n",
"\n",
"First, let's create a Docker file for a container with the model embedded into it.\n",
"\n",
"It will use your Hugging Face Hub secret at build time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8Pt7jloHDbev"
},
"outputs": [],
"source": [
"%%writefile Dockerfile\n",
"\n",
"FROM us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250312_0916_RC01\n",
"\n",
"ARG MODEL\n",
"ENV HF_MODEL=$MODEL\n",
"ENV MODEL_ID=$MODEL\n",
"\n",
"ENV HF_HOME=/model-cache\n",
"# Download the model to the cache\n",
"RUN --mount=type=secret,id=HF_TOKEN HF_TOKEN=$(cat /run/secrets/HF_TOKEN) \\\n",
" huggingface-cli download ${HF_MODEL}\n",
"# Switch to offline mode to make sure vLLM always uses the cached model\n",
"ENV HF_HUB_OFFLINE=1\n",
"\n",
"ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \\\n",
" --port ${PORT:-8080} \\\n",
" --model ${HF_MODEL} \\\n",
" --max-num-seqs=4 \\\n",
" --max-model-len 32768"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vZG5JfZRDbev"
},
"source": [
"Second, we create a Cloud Build file to use for building and pushing our container image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cRiA0VArDbev"
},
"outputs": [],
"source": [
"%%writefile cloudbuild.yaml\n",
"\n",
"steps:\n",
"- name: 'gcr.io/cloud-builders/docker'\n",
" id: build\n",
" entrypoint: 'bash'\n",
" secretEnv: ['HF_TOKEN']\n",
" args:\n",
" - -c\n",
" - |\n",
" SECRET_TOKEN=\"$$HF_TOKEN\" docker buildx build --tag=${_IMAGE} --build-arg MODEL=${_MODEL} --secret id=HF_TOKEN .\n",
"\n",
"availableSecrets:\n",
" secretManager:\n",
" - versionName: 'projects/${PROJECT_ID}/secrets/${_HF_TOKEN_SECRET_NAME}/versions/latest'\n",
" env: 'HF_TOKEN'\n",
"\n",
"images: [\"${_IMAGE}\"]\n",
"\n",
"substitutions:\n",
" _IMAGE: '${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_AR_REPO_NAME}/${_SERVICE_NAME}'\n",
"\n",
"options:\n",
" dynamicSubstitutions: true\n",
" machineType: \"E2_HIGHCPU_32\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9Vppnp8kDbev"
},
"source": [
"## Build Container Image and Deploy Cloud Run Service\n",
"\n",
"We are ready to build our container image and deploy Cloud Run service.\n",
"\n",
"The script below performs the following actions:\n",
"\n",
"* Enables necessary APIs.\n",
"* Creates an Artifact Repository for the image.\n",
"* Creates a Service Account for the service.\n",
"* Create a secret in Secret Manager that Cloud Build will use for building the container image.\n",
"* Assigns Service Account permissions to access the secret.\n",
"* Submits a Cloud Build job to create and push the container image.\n",
"* Deploys the Cloud Run service.\n",
"\n",
"> The script may take 15-45 minutes to finish.\n",
"\n",
"Note the following important flags in Cloud Build deployment command:\n",
"\n",
"* `--gpu 1` with `--gpu-type nvidia-l4` assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.\n",
"`--no-allow-authenticated` restricts unauthenticated access to the service.\n",
"By keeping the service private, you can rely on Cloud Run's built-in [Identity and Access Management (IAM)](https://cloud.google.com/iam) authentication for service-to-service communication.\n",
"* `--no-cpu-throttling` is required for enabling GPU.\n",
"* `--service-account` the service identity of the service.\n",
"* `--max-instances` sets maximum number of instances of the service.\n",
"It has to be equal to or lower than your project's NVIDIA L4 GPU (`Total Nvidia L4 GPU allocation, per project per region`) quota."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xsbvKy-_Dbev"
},
"outputs": [],
"source": [
"%%writefile deploy.sh\n",
"\n",
"PROJECT_ID=$1\n",
"REGION=$2\n",
"MODEL_ID=\"${3}\"\n",
"SERVICE_NAME=\"${5}\"\n",
"AR_REPO_NAME=\"vllm-repo\"\n",
"SERVICE_ACCOUNT=\"vllm-cloud-run-sa\"\n",
"SERVICE_ACCOUNT_ADDRESS=\"${SERVICE_ACCOUNT}@$PROJECT_ID.iam.gserviceaccount.com\"\n",
"HF_TOKEN_SECRET_NAME=\"HF_TOKEN_SECRET\"\n",
"HF_TOKEN=$4\n",
"MAX_INSTANCES=1 # Adjust this value to match your Cloud Run L4 GPU quota (\"Total Nvidia L4 GPU allocation, per project per region\", NvidiaL4GpuAllocPerProjectRegion, run.googleapis.com/nvidia_l4_gpu_allocation)\n",
"\n",
"echo \"Enabling APIs in project ${PROJECT_ID}.\"\n",
"gcloud services enable run.googleapis.com \\\n",
" cloudbuild.googleapis.com \\\n",
" secretmanager.googleapis.com \\\n",
" artifactregistry.googleapis.com \\\n",
" --project ${PROJECT_ID} \\\n",
" --quiet\n",
"\n",
"set -e\n",
"\n",
"# Creating the service account if doesn't exist.\n",
"sa_list=$(gcloud iam service-accounts list --quiet --format 'value(email)' --project $PROJECT_ID --filter=email:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com 2>/dev/null)\n",
"if [ -z \"${sa_list}\" ]; then\n",
" echo \"Creating Service Account ${SERVICE_ACCOUNT}.\"\n",
" gcloud iam service-accounts create $SERVICE_ACCOUNT \\\n",
" --project ${PROJECT_ID} \\\n",
" --display-name=\"${SERVICE_ACCOUNT} - Cloud Run Service Account\"\n",
"fi\n",
"\n",
"# Creating or updating the secret.\n",
"secrets_list=$(gcloud secrets list --project=\"${PROJECT_ID}\" --quiet --format 'value(name)' --filter=name:\"${HF_TOKEN_SECRET_NAME}\" 2>/dev/null)\n",
"if [ -z \"${secrets_list}\" ]; then\n",
" echo \"Creating Secret ${HF_TOKEN_SECRET_NAME}.\"\n",
" echo \"${HF_TOKEN}\" | gcloud secrets create \"${HF_TOKEN_SECRET_NAME}\" --project=${PROJECT_ID} --data-file=-\n",
"else\n",
" echo \"Updating Secret ${HF_TOKEN_SECRET_NAME}.\"\n",
" echo \"${HF_TOKEN}\" | gcloud secrets versions add \"${HF_TOKEN_SECRET_NAME}\" --project=${PROJECT_ID} --data-file=-\n",
"fi\n",
"\n",
"# Getting Cloud Build service account and giving it access to the secret\n",
"echo \"Applying Cloud Build account permissions.\"\n",
"cloud_build_sa=$(gcloud builds get-default-service-account --format 'value(serviceAccountEmail)' --project ${PROJECT_ID} --quiet)\n",
"cloud_build_sa_email=\"${cloud_build_sa##*/}\"\n",
"gcloud secrets add-iam-policy-binding \"${HF_TOKEN_SECRET_NAME}\" \\\n",
" --member serviceAccount:\"${cloud_build_sa_email}\" \\\n",
" --role='roles/secretmanager.secretAccessor' \\\n",
" --project=${PROJECT_ID}\n",
"\n",
"# Creating the Artifacts Repository if doesn't exist\n",
"repo_list=$(gcloud artifacts repositories list --format 'value(name)' --filter=name=\"projects/${PROJECT_ID}/locations/${REGION}/repositories/${AR_REPO_NAME}\" --project ${PROJECT_ID} --quiet --location ${REGION} 2>/dev/null)\n",
"if [ -z \"${repo_list}\" ]; then\n",
" echo \"Creating Artifact Registry ${AR_REPO_NAME}.\"\n",
" gcloud artifacts repositories create $AR_REPO_NAME \\\n",
" --repository-format docker \\\n",
" --location ${REGION} \\\n",
" --project=${PROJECT_ID}\n",
"fi\n",
"\n",
"echo \"Building container image.\"\n",
"gcloud builds submit --config=cloudbuild.yaml --project=${PROJECT_ID} . \\\n",
" --suppress-logs \\\n",
" --substitutions \\\n",
" _AR_REPO_NAME=$AR_REPO_NAME,_REGION=$REGION,_SERVICE_NAME=$SERVICE_NAME,_MODEL=$MODEL_ID,_HF_TOKEN_SECRET_NAME=$HF_TOKEN_SECRET_NAME\n",
"rm -f cloudbuild.yaml\n",
"rm -f Dockerfile\n",
"\n",
"echo \"Deploying Service ${SERVICE_NAME}.\"\n",
"gcloud beta run deploy $SERVICE_NAME \\\n",
" --project=${PROJECT_ID} \\\n",
" --image=${REGION}-docker.pkg.dev/$PROJECT_ID/$AR_REPO_NAME/$SERVICE_NAME \\\n",
" --service-account $SERVICE_ACCOUNT_ADDRESS \\\n",
" --cpu=8 \\\n",
" --memory=32Gi \\\n",
" --gpu=1 --gpu-type=nvidia-l4 \\\n",
" --region ${REGION} \\\n",
" --no-allow-unauthenticated \\\n",
" --max-instances ${MAX_INSTANCES} \\\n",
" --no-cpu-throttling \\\n",
" --timeout 1h\n",
"\n",
"SERVICE_URL=$(gcloud run services describe ${SERVICE_NAME} --project=${PROJECT_ID} --region $REGION --format 'value(status.url)' --quiet)\n",
"echo \"✅ Success!\"\n",
"echo \"🚀 Service URL: ${SERVICE_URL}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eff7e1585597"
},
"outputs": [],
"source": [
"!/bin/bash ./deploy.sh \"{PROJECT_ID}\" \"{REGION}\" \"{MODEL}\" \"{HF_TOKEN}\" \"{SERVICE_NAME}\" && rm -f ./deploy.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WzSFgfuyDbew"
},
"source": [
"## Test the deployed service\n",
"\n",
"Now, let's test the service you deployed.\n",
"\n",
"First, simply by using `cURL`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1LDyxRmADbew"
},
"outputs": [],
"source": [
"%%bash -s {MODEL} {SERVICE_NAME} {PROJECT_ID} {REGION}\n",
"\n",
"PROMPT=\"Why is the sky blue?\"\n",
"SERVICE_URL=$(gcloud run services describe ${2} --project ${3} --region ${4} --format 'value(status.url)' --quiet)\n",
"AUTH_TOKEN=$(gcloud auth print-identity-token -q)\n",
"\n",
"curl \"${SERVICE_URL}/v1/chat/completions\" \\\n",
" -s \\\n",
" -X POST \\\n",
" -H \"Content-Type: application/json\" \\\n",
" -H \"Authorization: Bearer ${AUTH_TOKEN}\" \\\n",
" -d @- <<EOF | python3 -m json.tool\n",
" {\n",
" \"model\": \"$1\",\n",
" \"messages\": [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"${PROMPT}\"\n",
" }\n",
" ]\n",
" }\n",
"EOF"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bc1P8IcrDbew"
},
"source": [
"### OpenAI Python API library\n",
"\n",
"vLLM serving container provides [OpenAI-compatible endpoint](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) you've just used. Let's test it with OpenAI Python API library."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "L5bVcdITDbew"
},
"outputs": [],
"source": [
"# Install OpenAI library\n",
"%pip install openai -q"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1xNVHL-6Dbew"
},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"from IPython.display import Image, Markdown, display\n",
"from openai import OpenAI\n",
"\n",
"prompt = \"What's in this image?\"\n",
"image_url = \"https://upload.wikimedia.org/wikipedia/commons/thumb/b/b6/Felis_catus-cat_on_snow.jpg/320px-Felis_catus-cat_on_snow.jpg\"\n",
"\n",
"identity_token = (\n",
" subprocess.check_output(\"gcloud auth print-identity-token -q\", shell=True)\n",
" .decode()\n",
" .strip()\n",
")\n",
"service_url = (\n",
" subprocess.check_output(\n",
" (\n",
" \"gcloud run services describe \"\n",
" f\"{SERVICE_NAME} --project={PROJECT_ID} \"\n",
" f\"--region={REGION} \"\n",
" \"--format='value(status.url)' -q\"\n",
" ),\n",
" shell=True,\n",
" )\n",
" .decode()\n",
" .strip()\n",
")\n",
"\n",
"client = OpenAI(\n",
" api_key=\"EMPTY\",\n",
" base_url=f\"{service_url}/v1\",\n",
")\n",
"chat_response = client.chat.completions.create(\n",
" model=MODEL,\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"text\", \"text\": prompt},\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\n",
" \"url\": image_url,\n",
" },\n",
" },\n",
" ],\n",
" }\n",
" ],\n",
" temperature=0.5,\n",
" extra_headers=f\"Bearer {identity_token}\",\n",
")\n",
"print(f\"Prompt: {prompt}\")\n",
"display(Image(url=image_url))\n",
"Markdown(chat_response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lLLwC3E3Dbew"
},
"source": [
"## Conclusion\n",
"Congratulations! Now you know how to deploy Gemma 3 with vLLM to Cloud Run powered by a GPU!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TKNPU6JADbew"
},
"source": [
"To delete the Cloud Run service you created, you can uncomment and run the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rxfg1S7pDbew"
},
"outputs": [],
"source": [
"# !gcloud run services delete $SERVICE_NAME --project $PROJECT_ID --region $LOCATION --quiet"
]
}
],
"metadata": {
"colab": {
"name": "cloud_run_vllm_gemma3_inference.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/get_started_with_vllm_lora_serving_on_vertex_ai.ipynb
+1265
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_ollama_gemma2_rag_agent.ipynb
+1645
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_pytorch_inference_paligemma_with_custom_handler.ipynb
+1384
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb
+1287
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb
+884
View File
@@ -0,0 +1,884 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Hugging Face DLCs: Serving Gemma with Text Generation Inference (TGI) on Vertex AI\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fvertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "83b98b0ba19c"
},
"source": [
"![Hugging Face x Google Cloud](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/google-cloud/thumbnail.png)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Alvaro Bartolome](https://github.com/alvarobartt), [Philipp Schmid](https://github.com/philschmid), [Simon Pagezy](https://github.com/pagezyhf), and [Jeff Boudier](https://github.com/jeffboudier) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Gemma**](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models, developed by Google DeepMind and other teams across Google.\n",
"\n",
"> [**Text Generation Inference (TGI)**](https://github.com/huggingface/text-generation-inference) is a toolkit developed by Hugging Face for deploying and serving LLMs, with high performance text generation.\n",
"\n",
"> [**Hugging Face DLCs**](https://github.com/huggingface/Google-Cloud-Containers) are pre-built and optimized Deep Learning Containers (DLCs) maintained by Hugging Face and Google Cloud teams to simplify environment configuration for your ML workloads.\n",
"\n",
"> [**Google Vertex AI**](https://cloud.google.com/vertex-ai) is a Machine Learning (ML) platform that lets you train and deploy ML models and AI applications, and customize large language models (LLMs) for use in your AI-powered applications.\n",
"\n",
"This notebook showcases how to deploy Google Gemma from the Hugging Face Hub on Vertex AI using the Hugging Face Deep Learning Container (DLC) for Text Generation Inference (TGI).\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"- Register any LLM from the Hugging Face Hub on Vertex AI\n",
"- Deploy an LLM on Vertex AI\n",
"- Send online predictions on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK and other required packages\n",
"\n",
"To run this example, you will only need the [`google-cloud-aiplatform`](https://github.com/googleapis/python-aiplatform) Python SDK and the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) Python package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --user --quiet google-cloud-aiplatform huggingface_hub"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages in this Jupyter environment, if you are on Colab you must restart the runtime. You can do this by running the cell below, which restarts the current kernel. The restart might take a minute or longer. After it's restarted, continue to the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SbmM4z7FOBpM"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9cc332aee5fd"
},
"source": [
"**1. Vertex AI Workbench**\n",
"\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "54c81e1a63e6"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c93d96bc9c43"
},
"outputs": [],
"source": [
"# !gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3e571ce6c56"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "984a0526fb68"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9af3e57f89a"
},
"source": [
"### Authenticate your Hugging Face account\n",
"\n",
"As [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it) is a gated model, you need to have a Hugging Face Hub account, and accept the Google's usage license for Gemma. Once that's done, you need to generate a new user access token with read-only access so that the weights can be downloaded from the Hub in the Hugging Face DLC for TGI.\n",
"\n",
"> Note that the user access token can only be generated via [the Hugging Face Hub UI](https://huggingface.co/settings/tokens/new), where you can either select read-only access to your account, or follow the recommendations and generate a fine-grained token with read-only access to [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4c31c7272804"
},
"source": [
"Then you can install the `huggingface_hub` that comes with a CLI that will be used for the authentication with the token generated in advance. So that then the token can be safely retrieved via `huggingface_hub.get_token`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8d836e0210fe"
},
"outputs": [],
"source": [
"from huggingface_hub import interpreter_login\n",
"\n",
"interpreter_login()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c71a4314c250"
},
"source": [
"Read more about [Hugging Face Security](https://huggingface.co/docs/hub/en/security), specifically about [Hugging Face User Access Tokens](https://huggingface.co/docs/hub/en/security-tokens)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com), if not enabled already.\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"if PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ee37e1544281"
},
"source": [
"## Requirements"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "877cd3fb2dce"
},
"source": [
"You will need to have the following IAM roles set:\n",
"\n",
"- Artifact Registry Reader (roles/artifactregistry.reader)\n",
"- Vertex AI User (roles/aiplatform.user)\n",
"\n",
"For more information about granting roles, see [Manage access](https://cloud.google.com/iam/docs/granting-changing-revoking-access).\n",
"\n",
"---\n",
"\n",
"You will also need to enable the following APIs (if not enabled already):\n",
"\n",
"- Vertex AI API (aiplatform.googleapis.com)\n",
"- Artifact Registry API (artifactregistry.googleapis.com)\n",
"\n",
"For more information about API enablement, see [Enabling APIs](https://cloud.google.com/apis/docs/getting-started#enabling_apis).\n",
"\n",
"---\n",
"\n",
"To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license on the Hugging Face Hub for any of the models from the [Gemma release collection](https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b), and the access request will be processed inmediately."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## Register Google Gemma on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "28d24c673821"
},
"source": [
"To serve Gemma with Text Generation Inference (TGI) on Vertex AI, you start importing the model on Vertex AI Model Registry.\n",
"\n",
"The Vertex AI Model Registry is a central repository where you can manage the lifecycle of your ML models. From the Model Registry, you have an overview of your models so you can better organize, track, and train new versions. When you have a model version you would like to deploy, you can assign it to an endpoint directly from the registry, or using aliases, deploy models to an endpoint. The models on the Vertex AI Model Registry just contain the model configuration and not the weights per se, as it's not a storage.\n",
"\n",
"Before going into the code to upload or import a model on Vertex AI, let's quickly review the arguments provided to the `aiplatform.Model.upload` method:\n",
"\n",
"* **`display_name`** is the name that will be shown in the Vertex AI Model Registry.\n",
"\n",
"* **`serving_container_image_uri`** is the location of the Hugging Face DLC for TGI that will be used for serving the model.\n",
"\n",
"* **`serving_container_environment_variables`** are the environment variables that will be used during the container runtime, so these are aligned with the environment variables defined by `text-generation-inference`, which are analog to the [`text-generation-launcher` arguments](https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/launcher). Additionally, the Hugging Face DLCs for TGI also capture the `AIP_` environment variables from Vertex AI as in [Vertex AI Documentation - Custom container requirements for prediction](https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements).\n",
"\n",
" * `MODEL_ID` is the identifier of the model in the Hugging Face Hub. To explore all the supported models you can check [the Hugging Face Hub](https://huggingface.co/models?sort=trending&other=text-generation-inference).\n",
" * `NUM_SHARD` is the number of shards to use if you don't want to use all GPUs on a given machine e.g. if you have two GPUs but you just want to use one for TGI then `NUM_SHARD=1`, otherwise it matches the `CUDA_VISIBLE_DEVICES`.\n",
" * `MAX_INPUT_TOKENS` is the maximum allowed input length (expressed in number of tokens), the larger it is, the larger the prompt can be, but also more memory will be consumed.\n",
" * `MAX_TOTAL_TOKENS` is the most important value to set as it defines the \"memory budget\" of running clients requests, the larger this value, the larger amount each request will be in your RAM and the less effective batching can be.\n",
" * `MAX_BATCH_PREFILL_TOKENS` limits the number of tokens for the prefill operation, as it takes the most memory and is compute bound, it is interesting to limit the number of requests that can be sent.\n",
" * `HUGGING_FACE_HUB_TOKEN` is the Hugging Face Hub token, required as [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it) is a gated model.\n",
"\n",
"* (optional) **`serving_container_ports`** is the port where the Vertex AI endpoint |will be exposed, by default 8080.\n",
"\n",
"For more information on the supported `aiplatform.Model.upload` arguments, check [its Python reference](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Model#google_cloud_aiplatform_Model_upload)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ef487ce082f3"
},
"outputs": [],
"source": [
"from huggingface_hub import get_token\n",
"\n",
"model = aiplatform.Model.upload(\n",
" display_name=\"google--gemma-7b-it\",\n",
" serving_container_image_uri=\"us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu121.2-2.ubuntu2204.py310\",\n",
" serving_container_environment_variables={\n",
" \"MODEL_ID\": \"google/gemma-7b-it\",\n",
" \"NUM_SHARD\": \"1\",\n",
" \"MAX_INPUT_TOKENS\": \"512\",\n",
" \"MAX_TOTAL_TOKENS\": \"1024\",\n",
" \"MAX_BATCH_PREFILL_TOKENS\": \"1512\",\n",
" \"HUGGING_FACE_HUB_TOKEN\": get_token(),\n",
" },\n",
" serving_container_ports=[8080],\n",
")\n",
"model.wait()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c427c0a87016"
},
"source": [
"## Deploy Google Gemma on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ef4cd160dc09"
},
"source": [
"After the model is registered on Vertex AI, you can deploy the model to an endpoint.\n",
"\n",
"You need to first deploy a model to an endpoint before that model can be used to serve online predictions. Deploying a model associates physical resources with the model so it can serve online predictions with low latency.\n",
"\n",
"Before going into the code to deploy a model to an endpoint, let's quickly review the arguments provided to the `aiplatform.Model.deploy` method:\n",
"\n",
"- **`endpoint`** is the endpoint to deploy the model to, which is optional, and by default will be set to the model display name with the `_endpoint` suffix.\n",
"- **`machine_type`**, **`accelerator_type`** and **`accelerator_count`** are arguments that define which instance to use, and additionally, the accelerator to use and the number of accelerators, respectively. The `machine_type` and the `accelerator_type` are tied together, so you will need to select an instance that supports the accelerator that you are using and vice-versa. More information about the different instances at [Compute Engine Documentation - GPU machine types](https://cloud.google.com/compute/docs/gpus), and about the `accelerator_type` naming at [Vertex AI Documentation - MachineSpec](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/MachineSpec).\n",
"\n",
"For more information on the supported `aiplatform.Model.deploy` arguments, you can check [its Python reference](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Model#google_cloud_aiplatform_Model_deploy)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e777427015db"
},
"outputs": [],
"source": [
"deployed_model = model.deploy(\n",
" endpoint=aiplatform.Endpoint.create(display_name=\"google--gemma-7b-it-endpoint\"),\n",
" machine_type=\"g2-standard-4\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=1,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5f2380d3ea60"
},
"source": [
"> Note that the model deployment on Vertex AI can take around 15 to 25 minutes; most of the time being the allocation / reservation of the resources, setting up the network and security, and such."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9b3fd1898241"
},
"source": [
"## Online predictions on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2aa9cee03bd0"
},
"source": [
"Once the model is deployed on Vertex AI, you can run the online predictions using the `aiplatform.Endpoint.predict` method, which will send the requests to the running endpoint in the `/predict` route specified within the container following Vertex AI I/O payload formatting.\n",
"\n",
"As you are serving a `text-generation` model fine-tuned for instruction-following, you will need to make sure that the chat template, if any, is applied correctly to the input conversation; meaning that `transformers` need to be installed so as to instantiate the `tokenizer` for [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it) and run the `apply_chat_template` method over the input conversation before sending the input within the payload to the Vertex AI endpoint.\n",
"\n",
"> Note that the Messages API will be supported on Vertex AI on upcoming TGI releases, starting on 2.3, meaning that at the time of writing this post, the prompts need to be formatted before sending the request if you want to achieve nice results (assuming you are using an intruction-following model and not a base model)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d46c0a866ffc"
},
"outputs": [],
"source": [
"%pip install --upgrade --user --quiet transformers jinja2"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7e1ab8de0b5c"
},
"source": [
"After the installation is complete, the following snippet will apply the chat template to the conversation:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "82fd1842f0cc"
},
"outputs": [],
"source": [
"from huggingface_hub import get_token\n",
"from transformers import AutoTokenizer\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(\"google/gemma-7b-it\", token=get_token())\n",
"\n",
"messages = [\n",
" {\"role\": \"user\", \"content\": \"What's Deep Learning?\"},\n",
"]\n",
"\n",
"inputs = tokenizer.apply_chat_template(\n",
" messages,\n",
" tokenize=False,\n",
" add_generation_prompt=True,\n",
")\n",
"# <bos><start_of_turn>user\\nWhat's Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d0522c9b9bc2"
},
"source": [
"Which is what you will be sending within the payload to the deployed Vertex AI Endpoint, as well as [the generation parameters](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient.text_generation)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "79e393e97ddb"
},
"source": [
"### Via Python"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1a404fd19a08"
},
"source": [
"#### Within the same session\n",
"\n",
"To run the online prediction via the Vertex AI SDK, you can simply use the `predict` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4ab63079a69a"
},
"outputs": [],
"source": [
"output = deployed_model.predict(\n",
" instances=[\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat's Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": True,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 1.0,\n",
" },\n",
" },\n",
" ]\n",
")\n",
"output"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b28efb473c7e"
},
"source": [
"#### From a different session\n",
"\n",
"To run the online prediction from a different session, you can run the following snippet."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e003ab03f30f"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"if PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)\n",
"\n",
"endpoint_display_name = (\n",
" \"google--gemma-7b-it-endpoint\" # TODO: change to your endpoint display name\n",
")\n",
"\n",
"# Iterates over all the Vertex AI Endpoints within the current project and keeps the first match (if any), otherwise set to None\n",
"ENDPOINT_ID = next(\n",
" (\n",
" endpoint.name\n",
" for endpoint in aiplatform.Endpoint.list()\n",
" if endpoint.display_name == endpoint_display_name\n",
" ),\n",
" None,\n",
")\n",
"assert ENDPOINT_ID, (\n",
" \"`ENDPOINT_ID` is not set, please make sure that the `endpoint_display_name` is correct at \"\n",
" f\"https://console.cloud.google.com/vertex-ai/online-prediction/endpoints?project={os.getenv('PROJECT_ID')}\"\n",
")\n",
"\n",
"endpoint = aiplatform.Endpoint(\n",
" f\"projects/{PROJECT_ID}/locations/{LOCATION}/endpoints/{ENDPOINT_ID}\"\n",
")\n",
"output = endpoint.predict(\n",
" instances=[\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat's Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": True,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 0.7,\n",
" },\n",
" },\n",
" ],\n",
")\n",
"output"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7cc0b19ab641"
},
"source": [
"### Via gcloud"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8fed7007c31c"
},
"source": [
"You can also send the requests using the `gcloud` CLI via the `gcloud ai endpoints` command.\n",
"\n",
"> Note that, before proceeding, you should either replace the values or set the following environment variables in advance from the Python variables set in the example, as follows:\n",
">\n",
"> ```python\n",
"> import os\n",
"> os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
"> os.environ[\"LOCATION\"] = LOCATION\n",
"> os.environ[\"ENDPOINT_NAME\"] = \"google--gemma-7b-it-endpoint\"\n",
"> ```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7fb1547d409f"
},
"outputs": [],
"source": [
"%%bash\n",
"ENDPOINT_ID=$(gcloud ai endpoints list \\\n",
" --project=$PROJECT_ID \\\n",
" --region=$LOCATION \\\n",
" --filter=\"display_name=$ENDPOINT_NAME\" \\\n",
" --format=\"value(name)\" \\\n",
" | cut -d'/' -f6)\n",
"\n",
"echo '{\n",
" \"instances\": [\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat'\\''s Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": true,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 1.0\n",
" }\n",
" }\n",
" ]\n",
"}' | gcloud ai endpoints predict $ENDPOINT_ID \\\n",
" --project=$PROJECT_ID \\\n",
" --region=$LOCATION \\\n",
" --json-request=\"-\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7e588a65759e"
},
"source": [
"### Via cURL"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cf3de731bcf4"
},
"source": [
"Alternatively, you can also send the requests via `cURL`.\n",
"\n",
"> Note that, before proceeding, you should either replace the values or set the following environment variables in advance from the Python variables set in the example, as follows:\n",
">\n",
"> ```python\n",
"> import os\n",
"> os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
"> os.environ[\"LOCATION\"] = LOCATION\n",
"> os.environ[\"ENDPOINT_NAME\"] = \"google--gemma-7b-it-endpoint\"\n",
"> ```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a71da4e1152b"
},
"outputs": [],
"source": [
"%%bash\n",
"ENDPOINT_ID=$(gcloud ai endpoints list \\\n",
" --project=$PROJECT_ID \\\n",
" --region=$LOCATION \\\n",
" --filter=\"display_name=$ENDPOINT_NAME\" \\\n",
" --format=\"value(name)\" \\\n",
" | cut -d'/' -f6)\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://$LOCATION-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/$LOCATION/endpoints/$ENDPOINT_ID:predict \\\n",
" -d '{\n",
" \"instances\": [\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat'\\''s Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": true,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 1.0\n",
" }\n",
" }\n",
" ]\n",
" }'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"Finally, you can already release the resources that you've created as follows, to avoid unnecessary costs:\n",
"\n",
"- `deployed_model.undeploy_all` to undeploy the model from all the endpoints.\n",
"- `deployed_model.delete` to delete the endpoint/s where the model was deployed gracefully, after the `undeploy_all` method.\n",
"- `model.delete` to delete the model from the registry."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b074ffdbd0c4"
},
"outputs": [],
"source": [
"deployed_model.undeploy_all()\n",
"deployed_model.delete()\n",
"model.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6bc4da2fcc35"
},
"source": [
"Alternatively, you can also remove those from the Google Cloud Console following the steps:\n",
"\n",
"- Go to Vertex AI in Google Cloud\n",
"- Go to Deploy and use -> Online prediction\n",
"- Click on the endpoint and then on the deployed model/s to \"Undeploy model from endpoint\"\n",
"- Then go back to the endpoint list and remove the endpoint\n",
"- Finally, go to Deploy and use -> Model Registry, and remove the model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "137d82da56f2"
},
"source": [
"## References\n",
"\n",
"- [GitHub Repository - Hugging Face DLCs for Google Cloud](https://github.com/huggingface/Google-Cloud-Containers): contains all the containers developed by the collaboration of both Hugging Face and Google Cloud teams; as well as a lot of examples on both training and inference, covering both CPU and GPU, as well support for most of the models within the Hugging Face Hub.\n",
"- [Google Cloud Documentation - Hugging Face DLCs](https://cloud.google.com/deep-learning-containers/docs/choosing-container#hugging-face): contains a table with the latest released Hugging Face DLCs on Google Cloud.\n",
"- [Google Artifact Registry - Hugging Face DLCs](https://console.cloud.google.com/artifacts/docker/deeplearning-platform-release/us/gcr.io): contains all the DLCs released by Google Cloud that can be used.\n",
"- [Hugging Face Documentation - Google Cloud](https://huggingface.co/docs/google-cloud): contains the official Hugging Face documentation for the Google Cloud DLCs."
]
}
],
"metadata": {
"colab": {
"name": "vertex_ai_text_generation_inference_gemma.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb
+1537
View File
File diff suppressed because it is too large Load Diff
open-models/use-cases/bigquery_ml_llama_inference.ipynb
+930
View File
@@ -0,0 +1,930 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Use Any OSS Gen AI Model Against Your BigQuery Data\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/open-models/use-cases/bigquery_ml_llama_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fuse-cases%2Fbigquery_ml_llama_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/bigquery/import?url=https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/bigquery/v1/32px.svg\" alt=\"BigQuery Studio logo\"><br> Open in BigQuery Studio\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/bigquery_ml_llama_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Jasper Xu](https://github.com/ZehaoXU), [Luis Lopez](https://github.com/luseloso) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook showcases a simple end-to-end process for extracting entities and performing data analytics using BigQuery in conjunction with an open-source text-generation Large Language Model (LLM). We use Meta's Llama 3.3 70B model as an example, and the process involves:\n",
"\n",
"* Deploy the Llama 3.3 70B model on Vertex AI.\n",
"* Configure the necessary setup, including downloading sample data and enabling BigQuery access to Vertex AI.\n",
"* Create a [remote model](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-remote-model-open) in BigQuery against the Llama 3.3 70B\n",
"* Employ the [ML.GENERATE_TEXT](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-text) function to extract structured information from medical transcripts.\n",
"* Analyzing the resulting structured data to gain insights.\n",
"\n",
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"* BigQuery\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), [BigQuery pricing](https://cloud.google.com/bigquery/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"## Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Deploy the Llama 3.3 70B Model on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"#### Request H100 quota\n",
"\n",
"By default, the quota for H100 deployment `Custom model serving per region` is 0. If you prefer to use H100 for higher throughput (recommended), you need to request for H100 quota following the instructions at [\"Request a higher quota\"](https://cloud.google.com/docs/quota/view-manage#requesting_higher_quota). We recommend starting with a small quota request, for example setting it to 8.\n",
"\n",
"If you want to run predictions with A100 80GB or H100 GPUs, we recommend using the regions listed below. \n",
"\n",
"| Machine Type | Accelerator Type | Recommended Regions |\n",
"| --- | --- | --- |\n",
"| a2-ultragpu-1g | 1 NVIDIA_A100_80GB | us-central1, us-east4, europe-west4, asia-southeast1, us-east4 |\n",
"| a3-highgpu-2g | 2 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n",
"| a3-highgpu-4g | 4 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n",
"| a3-highgpu-8g | 8 NVIDIA_H100_80GB | us-central1, us-east5, europe-west4, us-west1, asia-southeast1 |\n",
"\n",
"**NOTE:** Make sure you have associated quota in selected regions. Click the links to see your current quota for each GPU type: [Nvidia A100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_a100_80gb_gpus), [Nvidia H100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_h100_gpus).\n",
"\n",
"Alternatively, you can choose to use Nvidia L4 GPUs for this deployment.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ABhBcw0bloIu"
},
"source": [
"#### Setup Google Cloud project\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"\n",
"2. Point the code to your Project ID. \n",
"\n",
"\n",
"3. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the same region as where the notebook was launched. Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\").\n",
"\n",
"\n",
"4. Set your preferred region.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1f81f6289064"
},
"source": [
"**Note:** You do not need to fill out the `PROJECT_ID`, `BUCKET_URI`, or `REGION` as we will pick these values from `os.environ[\"GOOGLE_CLOUD_PROJECT\"]`, `os.environ[\"GOOGLE_CLOUD_REGION\"]` and the code will create a temporary bucket. \n",
"\n",
"If you would like to specify specific values, replace the empty strings below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fda9a1639038"
},
"outputs": [],
"source": [
"# fmt: off\n",
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
"BUCKET_URI = \"gs://\" # @param {type:\"string\"}\n",
"REGION = \"\" # @param {type:\"string\"}\n",
"# fmt: on"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SPdyTYbDjeHw"
},
"outputs": [],
"source": [
"# Import the necessary packages\n",
"\n",
"# Upgrade Vertex AI SDK.\n",
"% pip install --upgrade --quiet 'google-cloud-aiplatform>=1.64.0'\n",
"! git clone https://github.com/GoogleCloudPlatform/vertex-ai-samples.git\n",
"\n",
"import datetime\n",
"import importlib\n",
"import os\n",
"import uuid\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"common_util = importlib.import_module(\n",
" \"vertex-ai-samples.community-content.vertex_model_garden.model_oss.notebook_util.common_util\"\n",
")\n",
"\n",
"models, endpoints = {}, {}\n",
"\n",
"# Get the default cloud project id.\n",
"if not PROJECT_ID:\n",
" PROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\n",
"\n",
"# Get the default region for launching jobs.\n",
"if not REGION:\n",
" REGION = os.environ[\"GOOGLE_CLOUD_REGION\"]\n",
"\n",
"# Enable the Vertex AI API and Compute Engine API, if not already.\n",
"print(\"Enabling Vertex AI API and Compute Engine API.\")\n",
"! gcloud services enable aiplatform.googleapis.com compute.googleapis.com --project {PROJECT_ID}\n",
"\n",
"# Cloud Storage bucket for storing the experiment artifacts.\n",
"# A unique GCS bucket will be created for the purpose of this notebook. If you\n",
"# prefer using your own GCS bucket, change the value yourself below.\n",
"now = datetime.datetime.now().strftime(\"%Y%m%d%H%M%S\")\n",
"BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n",
"\n",
"if BUCKET_URI is None or BUCKET_URI.strip() == \"\" or BUCKET_URI == \"gs://\":\n",
" BUCKET_URI = f\"gs://{PROJECT_ID}-tmp-{now}-{str(uuid.uuid4())[:4]}\"\n",
" BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n",
" ! gsutil mb -l {REGION} {BUCKET_URI}\n",
"else:\n",
" assert BUCKET_URI.startswith(\"gs://\"), \"BUCKET_URI must start with `gs://`.\"\n",
" shell_output = ! gsutil ls -Lb {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n",
" bucket_region = shell_output[0].strip().lower()\n",
" if bucket_region != REGION:\n",
" raise ValueError(\n",
" \"Bucket region %s is different from notebook region %s\"\n",
" % (bucket_region, REGION)\n",
" )\n",
"print(f\"Using this GCS Bucket: {BUCKET_URI}\")\n",
"\n",
"STAGING_BUCKET = os.path.join(BUCKET_URI, \"temporal\")\n",
"MODEL_BUCKET = os.path.join(BUCKET_URI, \"llama3-3\")\n",
"\n",
"\n",
"# Initialize Vertex AI API.\n",
"print(\"Initializing Vertex AI API.\")\n",
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=STAGING_BUCKET)\n",
"\n",
"# Gets the default SERVICE_ACCOUNT.\n",
"shell_output = ! gcloud projects describe $PROJECT_ID\n",
"project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n",
"SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n",
"print(\"Using this default Service Account:\", SERVICE_ACCOUNT)\n",
"\n",
"\n",
"# Provision permissions to the SERVICE_ACCOUNT with the GCS bucket\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.admin $BUCKET_NAME\n",
"\n",
"! gcloud config set project $PROJECT_ID\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/storage.admin\"\n",
"! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/aiplatform.user\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c55789278f88"
},
"source": [
"# Access Llama 3.3 models on Vertex AI for serving\n",
"The original models from Meta are converted into the Hugging Face format for serving in Vertex AI.\n",
"Accept the model agreement to access the models:\n",
"1. Open the [Llama 3.3 model card](https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama3-3) from [Vertex AI Model Garden](https://cloud.google.com/model-garden).\n",
"2. Review and accept the agreement in the pop-up window on the model card page. If you have previously accepted the model agreement, there will not be a pop-up window on the model card page and this step is not needed.\n",
"3. After accepting the agreement of Llama 3.3, a `gs://` URI containing Llama 3.3 models will be shared.\n",
"4. Paste the URI in the `VERTEX_AI_MODEL_GARDEN_LLAMA_3_3` field below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7b8ae65ff28c"
},
"outputs": [],
"source": [
"# fmt: off\n",
"VERTEX_AI_MODEL_GARDEN_LLAMA_3_3 = \"gs://vertex-model-garden-public-us/llama3.3\" # @param {type:\"string\", isTemplate:true}\n",
"# fmt: on\n",
"assert VERTEX_AI_MODEL_GARDEN_LLAMA_3_3, (\n",
" \"Click the agreement of Llama 3.3 in Vertex AI Model Garden, and get the GCS path of Llama 3.3 model artifacts.\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KCsVQ2qolxSD"
},
"source": [
"### Deploy Llama 3.3 70B Instruct with vLLM\n",
"\n",
"The following step will register the model in Vertex AI and deploy it to an endpoint for serving. \n",
"\n",
"\n",
"**NOTE:** It usually takes 15~30 minutes to complete. \n",
"\n",
"To monitor deployment progress, navigate to [Vertex AI](https://console.cloud.google.com/vertex-ai) -> Online Prediction -> View logs for the Llama 3.3 endpoint."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bVxM9XiEjt1I"
},
"outputs": [],
"source": [
"# @markdown This section uploads Llama 3.3 to Model Registry and deploys it to a Vertex AI Endpoint. It takes ~30 minutes.\n",
"\n",
"# @markdown The serving efficiency of L4 GPUs is inferior to that of H100 GPUs, but L4 GPUs are nevertheless good serving solutions if you do not have H100 quota.\n",
"\n",
"# @markdown H100 is hard to get for now. It's recommended to use the deployment button in the model card. You can still try to deploy H100 endpoint through the notebook, but there is a chance that resource is not available.\n",
"\n",
"# @markdown Set the model to deploy.\n",
"\n",
"# fmt: off\n",
"base_model_name = \"Llama-3.3-70B-Instruct\" # @param [\"Llama-3.3-70B-Instruct\"] {isTemplate:true}\n",
"# fmt: on\n",
"model_id = os.path.join(VERTEX_AI_MODEL_GARDEN_LLAMA_3_3, base_model_name)\n",
"ENABLE_DYNAMIC_LORA = False\n",
"hf_model_id = \"meta-llama/\" + base_model_name\n",
"\n",
"accelerator_type = \"NVIDIA_L4\" # @param [\"NVIDIA_H100_80GB\", \"NVIDIA_L4\"]\n",
"\n",
"# The pre-built serving docker images.\n",
"VLLM_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20241001_0916_RC00\"\n",
"\n",
"use_dedicated_endpoint = False\n",
"# @markdown Find Vertex AI prediction supported accelerators and regions at https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
"if accelerator_type == \"NVIDIA_L4\":\n",
" machine_type = \"g2-standard-96\"\n",
" accelerator_count = 8\n",
" max_loras = 1\n",
"elif accelerator_type == \"NVIDIA_H100_80GB\":\n",
" machine_type = \"a3-highgpu-4g\"\n",
" accelerator_count = 4\n",
" max_loras = 1\n",
"else:\n",
" raise ValueError(\n",
" f\"Recommended GPU setting not found for: {accelerator_type} and {base_model_name}.\"\n",
" )\n",
"\n",
"common_util.check_quota(\n",
" project_id=PROJECT_ID,\n",
" region=REGION,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" is_for_training=False,\n",
")\n",
"\n",
"gpu_memory_utilization = 0.95\n",
"max_model_len = 8192 # Maximum context length.\n",
"\n",
"\n",
"def deploy_model_vllm(\n",
" model_name: str,\n",
" model_id: str,\n",
" publisher: str,\n",
" publisher_model_id: str,\n",
" service_account: str,\n",
" base_model_id: str = None,\n",
" machine_type: str = \"g2-standard-8\",\n",
" accelerator_type: str = \"NVIDIA_L4\",\n",
" accelerator_count: int = 1,\n",
" gpu_memory_utilization: float = 0.9,\n",
" max_model_len: int = 4096,\n",
" dtype: str = \"auto\",\n",
" enable_trust_remote_code: bool = False,\n",
" enforce_eager: bool = False,\n",
" enable_lora: bool = False,\n",
" enable_chunked_prefill: bool = False,\n",
" enable_prefix_cache: bool = False,\n",
" host_prefix_kv_cache_utilization_target: float = 0.0,\n",
" max_loras: int = 1,\n",
" max_cpu_loras: int = 8,\n",
" use_dedicated_endpoint: bool = False,\n",
" max_num_seqs: int = 256,\n",
" model_type: str = None,\n",
") -> tuple[aiplatform.Model, aiplatform.Endpoint]:\n",
" \"\"\"Deploys trained models with vLLM into Vertex AI.\"\"\"\n",
" endpoint = aiplatform.Endpoint.create(\n",
" display_name=f\"{model_name}-endpoint\",\n",
" dedicated_endpoint_enabled=use_dedicated_endpoint,\n",
" )\n",
"\n",
" if not base_model_id:\n",
" base_model_id = model_id\n",
"\n",
" # See https://docs.vllm.ai/en/latest/models/engine_args.html for a list of possible arguments with descriptions.\n",
" vllm_args = [\n",
" \"python\",\n",
" \"-m\",\n",
" \"vllm.entrypoints.api_server\",\n",
" \"--host=0.0.0.0\",\n",
" \"--port=8080\",\n",
" f\"--model={model_id}\",\n",
" f\"--tensor-parallel-size={accelerator_count}\",\n",
" \"--swap-space=16\",\n",
" f\"--gpu-memory-utilization={gpu_memory_utilization}\",\n",
" f\"--max-model-len={max_model_len}\",\n",
" f\"--dtype={dtype}\",\n",
" f\"--max-loras={max_loras}\",\n",
" f\"--max-cpu-loras={max_cpu_loras}\",\n",
" f\"--max-num-seqs={max_num_seqs}\",\n",
" \"--disable-log-stats\",\n",
" ]\n",
"\n",
" if enable_trust_remote_code:\n",
" vllm_args.append(\"--trust-remote-code\")\n",
"\n",
" if enforce_eager:\n",
" vllm_args.append(\"--enforce-eager\")\n",
"\n",
" if enable_lora:\n",
" vllm_args.append(\"--enable-lora\")\n",
"\n",
" if enable_chunked_prefill:\n",
" vllm_args.append(\"--enable-chunked-prefill\")\n",
"\n",
" if enable_prefix_cache:\n",
" vllm_args.append(\"--enable-prefix-caching\")\n",
"\n",
" if 0 < host_prefix_kv_cache_utilization_target < 1:\n",
" vllm_args.append(\n",
" f\"--host-prefix-kv-cache-utilization-target={host_prefix_kv_cache_utilization_target}\"\n",
" )\n",
"\n",
" if model_type:\n",
" vllm_args.append(f\"--model-type={model_type}\")\n",
"\n",
" env_vars = {\n",
" \"MODEL_ID\": base_model_id,\n",
" \"DEPLOY_SOURCE\": \"notebook\",\n",
" }\n",
"\n",
" # HF_TOKEN is not a compulsory field and may not be defined.\n",
" try:\n",
" if HF_TOKEN:\n",
" env_vars[\"HF_TOKEN\"] = HF_TOKEN\n",
" except NameError:\n",
" pass\n",
"\n",
" model = aiplatform.Model.upload(\n",
" display_name=model_name,\n",
" serving_container_image_uri=VLLM_DOCKER_URI,\n",
" serving_container_args=vllm_args,\n",
" serving_container_ports=[8080],\n",
" serving_container_predict_route=\"/generate\",\n",
" serving_container_health_route=\"/ping\",\n",
" serving_container_environment_variables=env_vars,\n",
" serving_container_shared_memory_size_mb=(16 * 1024), # 16 GB\n",
" serving_container_deployment_timeout=7200,\n",
" model_garden_source_model_name=(\n",
" f\"publishers/{publisher}/models/{publisher_model_id}\"\n",
" ),\n",
" )\n",
" print(\n",
" f\"Deploying {model_name} on {machine_type} with {accelerator_count} {accelerator_type} GPU(s).\"\n",
" )\n",
" model.deploy(\n",
" endpoint=endpoint,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" deploy_request_timeout=1800,\n",
" service_account=service_account,\n",
" system_labels={\n",
" \"NOTEBOOK_NAME\": \"model_garden_pytorch_llama3_3_deployment.ipynb\",\n",
" },\n",
" )\n",
" print(\"endpoint_name:\", endpoint.name)\n",
"\n",
" return model, endpoint\n",
"\n",
"\n",
"models[\"vllm_gpu\"], endpoints[\"vllm_gpu\"] = deploy_model_vllm(\n",
" model_name=common_util.get_job_name_with_datetime(prefix=\"llama3-3-serve\"),\n",
" model_id=model_id,\n",
" publisher=\"meta\",\n",
" publisher_model_id=\"llama3-3\",\n",
" base_model_id=hf_model_id,\n",
" service_account=SERVICE_ACCOUNT,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" gpu_memory_utilization=gpu_memory_utilization,\n",
" max_model_len=max_model_len,\n",
" max_loras=max_loras,\n",
" enforce_eager=True,\n",
" enable_lora=ENABLE_DYNAMIC_LORA,\n",
" enable_chunked_prefill=not ENABLE_DYNAMIC_LORA,\n",
" use_dedicated_endpoint=use_dedicated_endpoint,\n",
" model_type=\"llama3.1\",\n",
")\n",
"# @markdown Click \"Show Code\" to see more details."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s8bAhMq8wglq"
},
"source": [
"#### Get the endpoint ID\n",
"\n",
"Once the deployment is successful, we need to grab the endpoint. We can reconstruct the String from our previous step"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xIWHZNzkwEIt"
},
"outputs": [],
"source": [
"ENDPOINT_ID = f\"https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoints['vllm_gpu'].name}\"\n",
"print(ENDPOINT_ID)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sFqr3PYnoxUf"
},
"source": [
"## Data Analytics in BigQuery"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SNeGovGko2Aw"
},
"source": [
"### Setup"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RXDi9glXtUBN"
},
"source": [
"#### Create a new dataset\n",
"\n",
"This will house any tables created throughout this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VxcDQUwxta8v"
},
"outputs": [],
"source": [
"!bq mk --location=us --dataset --project_id={PROJECT_ID} demo_dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M26CPteyth0R"
},
"source": [
"#### Create a Cloud resource connection\n",
"\n",
"[Cloud resource connections](https://cloud.google.com/bigquery/docs/connections-api-intro#cloud-resource-connections) enable BigQuery to access other Cloud services, like Cloud Storage and Vertex AI."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rZB9N43itty-"
},
"outputs": [],
"source": [
"!bq mk --connection --connection_type=CLOUD_RESOURCE --location=us --project_id={PROJECT_ID} \"demo_conn\"\n",
"!bq show --location=us --connection --project_id={PROJECT_ID} \"demo_conn\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jd-6mfdot8x7"
},
"source": [
"#### Add permissions to Cloud resource connection service account\n",
"\n",
"The Cloud resource connection is associated with a service account. \n",
"\n",
"The following cell enables the service account to access the Vertex AI service.\n",
"\n",
"**Note:** Copy the service account ID from the prior cell and input it below. It will look like `your-copied-service-account@gcp-sa-bigquery-condel.iam.gserviceaccount.com`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NiJUgWYYuC6n"
},
"outputs": [],
"source": [
"# fmt: off\n",
"connection_service_account = \"your-copied-service-account@gcp-sa-bigquery-condel.iam.gserviceaccount.com\" # @param {type: \"string\"}\n",
"# fmt: on\n",
"connection_member = f\"serviceAccount:{connection_service_account}\"\n",
"\n",
"\n",
"!gcloud projects add-iam-policy-binding {PROJECT_ID} --member={connection_member} --role='roles/aiplatform.user' --condition=None --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tEhojixwvGrt"
},
"source": [
"### Load medical transcripts into BigQuery\n",
"\n",
"For this demo, we will use a small medical transcripts dataset which has unstructured, varied raw transcripts capturing history, diagnosis and treatment provided of patients visiting a medical facility. \n",
"\n",
"The dataset contains 164 rows, each with an average of 814 input tokens.\n",
"\n",
"Let's load this dataset into a BigQuery table and peek one example row:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "x2D6NnFovOdb"
},
"outputs": [],
"source": [
"%%bigquery --project $PROJECT_ID\n",
"\n",
"LOAD DATA OVERWRITE\n",
" demo_dataset.medical_transcript\n",
"FROM FILES( format='NEWLINE_DELIMITED_JSON',\n",
" uris = ['gs://cloud-samples-data/vertex-ai/model-evaluation/peft_eval_sample.jsonl'] );\n",
"\n",
"SELECT * FROM demo_dataset.medical_transcript LIMIT 10;"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7UbctAXm0XaA"
},
"source": [
"### Create a remote model\n",
"\n",
"In this next step we will register a Vertex AI endpoint as a remote model in BigQuery. Then, you use the [ML.GENERATE_TEXT function](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-predict) to make predictions using the remote model.\n",
"\n",
"You can use remote models when a model is too large to import into BigQuery. They are also useful when you want to have a single point of inference for online, batch, and micro-batch use cases.\n",
"\n",
"**Note**: If you get an error in this step is because the service account permissions have yet finished propagating, please wait a minute and retry\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BsMbwJc10okx"
},
"outputs": [],
"source": [
"from google.cloud import bigquery\n",
"\n",
"client = bigquery.Client(project=PROJECT_ID)\n",
"\n",
"query = f\"\"\"\n",
"CREATE OR REPLACE MODEL demo_dataset.llama_3_3\n",
"REMOTE WITH CONNECTION\n",
" `{PROJECT_ID}.us.demo_conn`\n",
"OPTIONS(\n",
" endpoint='{ENDPOINT_ID}'\n",
");\n",
"\"\"\"\n",
"\n",
"query_job = client.query(query) # API request\n",
"query_job.result() # Waits for the query to complete\n",
"\n",
"print(\"Remote model `demo_dataset.llama_3_3` created or replaced successfully.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jJ9KjfU52gf3"
},
"source": [
"### Extract structured entity from medical transcripts\n",
"\n",
"We will now use our Llama model to extract structured data from the unstructured transcripts with [ML.GENERATE_TEXT](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-generate-text). Say we want to extract the patient's age, gender and list of diseases for each entry. We can do so with a SQL statement like the following and save the derived insights to a table. We include the information we want to extract and its schema in the model prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-fd4S6Wp2ari"
},
"outputs": [],
"source": [
"%%bigquery --project $PROJECT_ID\n",
"\n",
"CREATE TEMP FUNCTION ExtractOutput(s STRING)\n",
"RETURNS STRING\n",
"AS (\n",
" SUBSTR(s, INSTR(s, \"Output:\")+8)\n",
");\n",
"\n",
"\n",
"CREATE OR REPLACE TABLE demo_dataset.medical_transcript_analysis AS (\n",
"SELECT\n",
" ExtractOutput(ml_generate_text_llm_result) AS generated_text, * EXCEPT(ml_generate_text_llm_result)\n",
"FROM\n",
" ML.GENERATE_TEXT( MODEL `demo_dataset.llama_3_3`,\n",
" (\n",
" SELECT\n",
" CONCAT('Extract the Gender, Age (in years), and Disease information from the following medical transcript. Return **only** a JSON in the following schema: \\n{ \"Age\": Int, \"Gender\": \"String\", \"Disease\": [\"String\"]}. \\nIf Age, Gender, or Disease information is not found, return `null` for that field. Summarize the disease(s) in 1 to 5 words. If the patient has multiple diseases, include them in a comma-separated list within the \"Disease\" field. \\n**Do not include any other text or labels in your response.**. \\n', input_text) AS prompt\n",
" FROM\n",
" demo_dataset.medical_transcript\n",
" ),\n",
" STRUCT(\n",
" 0 AS temperature,\n",
" 0.001 AS top_p,\n",
" 1 AS top_k,\n",
" 128 AS max_output_tokens,\n",
" TRUE AS flatten_json_output))\n",
");\n",
"\n",
"\n",
"SELECT * FROM demo_dataset.medical_transcript_analysis;"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "voR8CEbx4IHv"
},
"source": [
"### Analyze structured outputs for insights\n",
"\n",
"We can now perform all sorts of analytics on this data. For example, let us answer `What are the most common diseases in females ages 30+ in our sample?`. Using a simple SQL query like below we see that 'Hypertension', 'Arthritis' and 'Hyperlipidemia' are most common."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8UU1N1z85i_q"
},
"outputs": [],
"source": [
"%%bigquery --project $PROJECT_ID\n",
"\n",
"WITH\n",
" parsed_data AS (\n",
" SELECT\n",
" JSON_EXTRACT_SCALAR(generated_text, '$.Gender') AS gender,\n",
" CAST(JSON_EXTRACT_SCALAR(generated_text, '$.Age') AS INT64) AS age,\n",
" JSON_EXTRACT_ARRAY(generated_text, '$.Disease') AS diseases,\n",
" FROM\n",
" demo_dataset.medical_transcript_analysis)\n",
"\n",
"SELECT\n",
" disease,\n",
" count(*) AS occurrence\n",
"FROM\n",
" parsed_data, UNNEST(diseases) AS disease\n",
"WHERE\n",
" LOWER(gender) = 'female'\n",
" AND age >= 30\n",
"GROUP BY\n",
" disease\n",
"ORDER BY\n",
" occurrence DESC\n",
"LIMIT 3;"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up\n",
"\n",
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial on both Vertex AI and BigQuery side:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eQ3LpU9763Lv"
},
"source": [
"### Delete the Vertex AI model and endpoint\n",
"Delete the experiment models and endpoints to recycle the resources and avoid unnecessary continuous charges that may incur."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_5iltjKv68NJ"
},
"outputs": [],
"source": [
"# Undeploy model and delete endpoint.\n",
"for endpoint in endpoints.values():\n",
" endpoint.delete(force=True)\n",
"\n",
"# Delete models.\n",
"for model in models.values():\n",
" model.delete()\n",
"\n",
"# fmt: off\n",
"delete_bucket = True # @param {type:\"boolean\"}\n",
"# fmt: on\n",
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_NAME"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gwlpFUem7S4M"
},
"source": [
"### Delete BigQuery dataset and connnection"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d7beBMcw7U_J"
},
"outputs": [],
"source": [
"!gcloud projects remove-iam-policy-binding $PROJECT_ID --member=serviceAccount:$SERVICE_ACCOUNT --role=\"roles/aiplatform.user\n",
"!gcloud projects remove-iam-policy-binding $PROJECT_ID --member=serviceAccount:$SERVICE_ACCOUNT --role=\"roles/storage.admin\n",
"!bq rm -r -f $PROJECT_ID:demo_dataset\n",
"!bq rm --connection --project_id=$PROJECT_ID --location=us demo_conn"
]
}
],
"metadata": {
"colab": {
"name": "bigquery_ml_llama_inference.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb
+1166
View File
File diff suppressed because it is too large Load Diff
open-models/use-cases/deploy_llama3_vllm_gke_cloud_function.ipynb
+1116
View File
File diff suppressed because it is too large Load Diff
open-models/use-cases/guess_app.ipynb
+784
View File
@@ -0,0 +1,784 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Guess who or what app using Hugging Face Deep Learning container model on Vertex AI\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/guess_app.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fuse-cases%2Fguess_app.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/use-cases/guess_app.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/guess_app.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/guess_app.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/guess_app.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/guess_app.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/guess_app.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/guess_app.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Ivan Nardini](https://github.com/inardini) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook shows how you can use Vertex AI and Hugging Face Deep Learning container to create a simple \"Guess who or what\" application which combines an image generation open model with Gemini to answer and represent subjects of Guess who or what riddles."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK and other required packages\n",
"\n",
"To run this example, you will only need the [`google-cloud-aiplatform`](https://github.com/googleapis/python-aiplatform) Python SDK and the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) Python package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --user --quiet google-cloud-aiplatform huggingface_hub gradio"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages in this Jupyter environment, if you are on Colab you must restart the runtime. You can do this by running the cell below, which restarts the current kernel. The restart might take a minute or longer. After it's restarted, continue to the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SbmM4z7FOBpM"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9cc332aee5fd"
},
"source": [
"**1. Vertex AI Workbench**\n",
"\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "54c81e1a63e6"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c93d96bc9c43"
},
"outputs": [],
"source": [
"# !gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3e571ce6c56"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "984a0526fb68"
},
"outputs": [],
"source": [
"from google.colab import auth\n",
"\n",
"auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9af3e57f89a"
},
"source": [
"### Authenticate your Hugging Face account"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4c31c7272804"
},
"source": [
"Then you can install the `huggingface_hub` that comes with a CLI that will be used for the authentication with the token generated in advance. So that then the token can be safely retrieved via `huggingface_hub.get_token`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8d836e0210fe"
},
"outputs": [],
"source": [
"from huggingface_hub import interpreter_login\n",
"\n",
"interpreter_login()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c71a4314c250"
},
"source": [
"Read more about [Hugging Face Security](https://huggingface.co/docs/hub/en/security), specifically about [Hugging Face User Access Tokens](https://huggingface.co/docs/hub/en/security-tokens)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com), if not enabled already.\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"if PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ee37e1544281"
},
"source": [
"### Requirements"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "877cd3fb2dce"
},
"source": [
"To successfully deploy your Hugging Face model on Vertex AI, you need to have the following IAM roles set:\n",
"\n",
"- Artifact Registry Reader (`roles/artifactregistry.reader`)\n",
"- Vertex AI User (`roles/aiplatform.user`)\n",
"\n",
"For more information about granting roles, see [Manage access](https://cloud.google.com/iam/docs/granting-changing-revoking-access).\n",
"\n",
"---\n",
"\n",
"You also need to enable the following APIs (if not enabled already):\n",
"\n",
"- Vertex AI API (`aiplatform.googleapis.com`)\n",
"- Artifact Registry API (`artifactregistry.googleapis.com`)\n",
"\n",
"For more information about API enablement, see [Enabling APIs](https://cloud.google.com/apis/docs/getting-started#enabling_apis).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2pNoqLHVWG6X"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "V41joDcRWIzY"
},
"outputs": [],
"source": [
"import base64\n",
"import io\n",
"import os\n",
"\n",
"from IPython.display import display\n",
"from PIL import Image\n",
"from google.cloud.aiplatform import Endpoint\n",
"import gradio as gr\n",
"from huggingface_hub import get_token\n",
"import vertexai\n",
"from vertexai.generative_models import GenerationConfig, GenerativeModel, SafetySetting"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2qxTWPNu1p-d"
},
"source": [
"# Build a \"Guess what or who\" app\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dJsGNCv4T6R-"
},
"source": [
"## Register FLUX on Vertex AI\n",
"\n",
"To deploy a \"Guess what or who\" application using a Hugging Face model like [FLUX](https://console.cloud.google.com/vertex-ai/publishers/black-forest-labs/model-garden/flux1-schnell) for image generation on Vertex AI, you first register the chosen model within the [Vertex AI Model Registry](https://cloud.google.com/vertex-ai/docs/model-registry/introduction). This registry serves as a central repository for managing your models on Vertex AI.\n",
"\n",
"Registering a model involves specifying a serving container image and corresponding environment variables, which vary depending on the chosen model. For instance, for FLUX, you'll use a regular PyTorch Inference Deep Learning Container. You can find the appropriate container URI and further details in the [Google Cloud Deep Learning Containers documentation](https://huggingface.co/docs/google-cloud/en/containers/introduction). "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Pc-Owc_7WiZb"
},
"outputs": [],
"source": [
"flux_model = aiplatform.Model.upload(\n",
" display_name=\"flux--generate\",\n",
" serving_container_image_uri=\"us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu121.2-2.ubuntu2204.py310\",\n",
" serving_container_environment_variables={\n",
" \"HF_MODEL_ID\": \"black-forest-labs/FLUX.1-dev\",\n",
" \"HF_TASK\": \"text-to-image\",\n",
" \"HF_TOKEN\": get_token(),\n",
" },\n",
")\n",
"flux_model.wait()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c427c0a87016"
},
"source": [
"## Deploy Flux on Vertex AI\n",
"\n",
"After successfully registering the model, you can then deploy it to a Vertex AI Endpoint according to your preferred deployment configuration, making it ready for image generation.\n",
"\n",
"This deployment creates a new instance of the FLUX model in Vertex AI Prediction, the managed inference service on Vertex AI."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sxI3SsHxVYn4"
},
"outputs": [],
"source": [
"endpoint = aiplatform.Endpoint.create(display_name=\"flux--generate-endpoint\")\n",
"\n",
"deployed_flux_model = flux_model.deploy(\n",
" endpoint=endpoint,\n",
" machine_type=\"g2-standard-48\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=4,\n",
" sync=False,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8lIr8uko04T_"
},
"source": [
"## Generate predictions\n",
"\n",
"After the model gets successfully deployed, you can test it by submitting a prediction request."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ERBKKCPylaTt"
},
"source": [
"### Generate an image using FLUX"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Hxw6otB3SBZE"
},
"outputs": [],
"source": [
"response = deployed_flux_model.predict(\n",
" instances=[\"a image of a cat riding a horse in illustration style\"],\n",
" parameters={\n",
" \"width\": 512,\n",
" \"height\": 512,\n",
" \"num_inference_steps\": 8,\n",
" \"guidance_scale\": 3.5,\n",
" },\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wAGzijoxOqvr"
},
"source": [
"### Get the generated image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-9vR2U6b12sK"
},
"outputs": [],
"source": [
"image = Image.open(io.BytesIO(base64.b64decode(response.predictions[0])))\n",
"display(image)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9b3fd1898241"
},
"source": [
"## Play \"Guess who or what\"\n",
"\n",
"With the image generation open model deployed in a Vertex AI Endpoint, you are now able to build your \"Guess who or what\" Gen AI application.\n",
"\n",
"In this scenario, you use Gradio to quickly build a web application for your Gen AI models."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wkBl-jwH0_AJ"
},
"outputs": [],
"source": [
"# Get enviroment variables\n",
"PROJECT_ID = os.environ.get(\"PROJECT_ID\", PROJECT_ID)\n",
"LOCATION = os.environ.get(\"LOCATION\", LOCATION)\n",
"MODEL_ID = os.environ.get(\"MODEL_ID\", \"gemini-2.0-flash\")\n",
"ENDPOINT_NAME = os.environ.get(\"ENDPOINT\", endpoint.resource_name)\n",
"\n",
"# Initialize Vertex AI SDK\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
"\n",
"# Initialize Gemini model\n",
"MODEL = GenerativeModel(\"gemini-2.0-flash\")\n",
"ENDPOINT = Endpoint(ENDPOINT_NAME)\n",
"\n",
"# Common generation config and safety settings\n",
"GENERATION_CONFIG = GenerationConfig(\n",
" candidate_count=1, max_output_tokens=8192, temperature=0\n",
")\n",
"\n",
"SAFETY_SETTINGS = [\n",
" SafetySetting(\n",
" category=SafetySetting.HarmCategory.HARM_CATEGORY_HATE_SPEECH,\n",
" threshold=SafetySetting.HarmBlockThreshold.OFF,\n",
" ),\n",
" SafetySetting(\n",
" category=SafetySetting.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,\n",
" threshold=SafetySetting.HarmBlockThreshold.OFF,\n",
" ),\n",
" SafetySetting(\n",
" category=SafetySetting.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,\n",
" threshold=SafetySetting.HarmBlockThreshold.OFF,\n",
" ),\n",
" SafetySetting(\n",
" category=SafetySetting.HarmCategory.HARM_CATEGORY_HARASSMENT,\n",
" threshold=SafetySetting.HarmBlockThreshold.OFF,\n",
" ),\n",
"]\n",
"\n",
"\n",
"def generate_gemini_content(prompt_template: str, **kwargs) -> str:\n",
" \"\"\"Generate content using Gemini model with given prompt template.\"\"\"\n",
" prompt = prompt_template.format(**kwargs)\n",
" response = MODEL.generate_content(\n",
" [prompt],\n",
" generation_config=GENERATION_CONFIG,\n",
" safety_settings=SAFETY_SETTINGS,\n",
" stream=False,\n",
" )\n",
" return response.text\n",
"\n",
"\n",
"def generate_subject(riddle: str) -> str:\n",
" \"\"\"Extract the subject/answer from a given riddle using Gemini model.\"\"\"\n",
" riddle_solver_prompt_template = \"\"\"\n",
" You are the best riddle solver. Given a riddle, your goal is solve it and only indicate the subject of the riddle.\n",
" RIDDLE: {riddle}\n",
" SUBJECT:\n",
" \"\"\"\n",
" subject = generate_gemini_content(riddle_solver_prompt_template, riddle=riddle)\n",
" return subject.replace(\"SUBJECT:\", \"\").strip()\n",
"\n",
"\n",
"def generate_prompt(subject: str) -> str:\n",
" \"\"\"Generate an image generation prompt for a given subject.\"\"\"\n",
" image_gen_prompt_template = \"\"\"\n",
" You are a professional prompt engineer. Given a subject, prototype the most appropriate prompt to best visualize the subject.\n",
" Only return the preferred prompt.\n",
" SUBJECT: {subject}\n",
" PROMPT:\n",
" \"\"\"\n",
" return generate_gemini_content(image_gen_prompt_template, subject=subject)\n",
"\n",
"\n",
"def generate_image(image_gen_prompt: str) -> Image.Image:\n",
" \"\"\"Generate an image based on the provided prompt.\"\"\"\n",
" response = ENDPOINT.predict(\n",
" instances=[image_gen_prompt],\n",
" parameters={\n",
" \"width\": 512,\n",
" \"height\": 512,\n",
" \"num_inference_steps\": 8,\n",
" \"guidance_scale\": 3.5,\n",
" },\n",
" )\n",
" return Image.open(io.BytesIO(base64.b64decode(response.predictions[0])))\n",
"\n",
"\n",
"def guess_game(riddle: str) -> tuple[Image.Image, str, str]:\n",
" \"\"\"Run the complete riddle-to-image game flow.\"\"\"\n",
" answer = generate_subject(riddle)\n",
" prompt = generate_prompt(answer)\n",
" image = generate_image(prompt)\n",
" return image, answer, prompt\n",
"\n",
"\n",
"def increment_counter(counter: int) -> tuple[int, Image.Image | None, str, str, str]:\n",
" \"\"\"Increment the game counter\"\"\"\n",
" return counter + 1, None, \"\", \"\", \"\"\n",
"\n",
"\n",
"def reset_game(counter: int) -> tuple[int, Image.Image | None, str, str, str]:\n",
" \"\"\"Reset the game state.\"\"\"\n",
" return counter, None, \"\", \"\", \"\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5vzru9pIXxIT"
},
"source": [
"## Build the app\n",
"\n",
"Put together functions you define to build your Gradio app."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OpZ0mbmzXirY"
},
"outputs": [],
"source": [
"# Create main application block with Ocean theme\n",
"with gr.Blocks(theme=gr.themes.Ocean()) as app:\n",
"\n",
" # Header row containing title and counter state\n",
" with gr.Row():\n",
" gr.Markdown(\"# Guess What Game ❓\")\n",
" counter_state = gr.State(value=0)\n",
"\n",
" # Input row for user's riddle/description\n",
" with gr.Row():\n",
" prompt_input = gr.Textbox(label=\"Describe someone or something 💬 \")\n",
"\n",
" # Submit button to generate image and answer\n",
" submit_btn = gr.Button(\"Submit\")\n",
"\n",
" # Output row displaying generated content\n",
" with gr.Row():\n",
" image_prompt = gr.Textbox(\n",
" label=\"Generated Image Prompt with Gemini 🎨 \", visible=True\n",
" )\n",
" image_output = gr.Image(label=\"Generated Image 🖼️ \")\n",
" answer_output = gr.Textbox(\n",
" label=\"Generated Answer with Gemini 🌌 \", interactive=False\n",
" )\n",
"\n",
" # Game control buttons row\n",
" with gr.Row():\n",
" correct_btn = gr.Button(\"+1 Correct\")\n",
" reset_btn = gr.Button(\"Reset\")\n",
"\n",
" # Display for tracking correct guesses\n",
" counter_display = gr.Number(value=0, label=\"Correct Guesses 👍\", interactive=False)\n",
"\n",
" # Event handler for submit button\n",
" submit_btn.click(\n",
" guess_game,\n",
" inputs=[prompt_input],\n",
" outputs=[image_output, answer_output, image_prompt],\n",
" )\n",
"\n",
" # Event handler for correct button\n",
" correct_btn.click(\n",
" increment_counter,\n",
" inputs=[counter_state],\n",
" outputs=[\n",
" counter_state,\n",
" image_output,\n",
" answer_output,\n",
" image_prompt,\n",
" prompt_input,\n",
" ],\n",
" ).then(lambda x: x, inputs=[counter_state], outputs=[counter_display])\n",
"\n",
" # Event handler for reset button\n",
" reset_btn.click(\n",
" reset_game,\n",
" inputs=[counter_state],\n",
" outputs=[\n",
" counter_state,\n",
" image_output,\n",
" answer_output,\n",
" image_prompt,\n",
" prompt_input,\n",
" ],\n",
" ).then(lambda x: x, inputs=[counter_state], outputs=[counter_display])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sbSdpVjoOSgD"
},
"source": [
"### Launch the app\n",
"\n",
"You are now ready to launch your \"Guess who or what\" Gradio app."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cwUjdcP23Qmw"
},
"outputs": [],
"source": [
"app.launch()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iRfRH5fQc1IL"
},
"outputs": [],
"source": [
"delete_app = True\n",
"delete_endpoint = True\n",
"delete_model = True\n",
"\n",
"if delete_app:\n",
" app.close()\n",
"\n",
"if delete_endpoint:\n",
" deployed_flux_model.delete()\n",
"\n",
"if delete_model:\n",
" flux_model.delete()"
]
}
],
"metadata": {
"colab": {
"name": "guess_app.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/use-cases/model_garden_litellm_inference.ipynb
+667
View File
@@ -0,0 +1,667 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Serving Open-Source LLMs on Vertex AI with LiteLLM and OpenAI-Compatible APIs\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/tree/main/open-models/use-cases/model_garden_litellm_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fuse-cases%2Fmodel_garden_litellm_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/use-cases/model_garden_litellm_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/tree/main/open-models/use-cases/model_garden_litellm_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/model_garden_litellm_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Eliza Huang](https://github.com/lizzij) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to:\n",
"- Deploy an open-source LLM (e.g. DeepSeek, Qwen, LLaMA, Gemma) via [Vertex AI Model Garden](https://cloud.google.com/vertex-ai/docs/generative-ai/model-garden).\n",
"- Serve the model with a public Vertex AI endpoint.\n",
"- Connect it to [LiteLLM](https://docs.litellm.ai/) using an OpenAI-compatible schema for chat completion and function calling.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --force-reinstall --quiet 'google-cloud-aiplatform>=1.93.1' 'litellm' 'openai' 'google-auth' 'requests'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"import vertexai\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = os.environ.get(\"GOOGLE_CLOUD_PROJECT\")\n",
" if PROJECT_ID:\n",
" PROJECT_ID = str(PROJECT_ID)\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "haaUJ5VaWajl"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kCp1oCOyWfZe"
},
"outputs": [],
"source": [
"import os\n",
"import re\n",
"\n",
"from vertexai import model_garden"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fvVKEpROoXW5"
},
"source": [
"### Define helpers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9AoCHkCNoaLG"
},
"outputs": [],
"source": [
"def print_models(data_list: list[str], items_per_line: int = 2) -> None:\n",
" \"\"\"Prints the list with a specified number of items per line with index and emojis,\n",
" and includes the total count.\"\"\"\n",
" print(\"🌟--- Models available ---🌟\")\n",
" print(\"\\n\")\n",
" print(f\"🔢 Total models: {len(data_list)} 🔢\\n\") # Print the count here\n",
"\n",
" for i, item in enumerate(data_list):\n",
" print(f\"✨ {item} \", end=\"\")\n",
" if (i + 1) % items_per_line == 0:\n",
" print()\n",
" else:\n",
" print(\" --- \", end=\"\")\n",
"\n",
" if len(data_list) % items_per_line != 0:\n",
" print()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dJmOAajdW7qQ"
},
"source": [
"## Find the models that you can deploy\n",
"\n",
"In Vertex AI Model Garden, you can discover and deploy a wide range of open-source models.\n",
"\n",
"Many of these models are directly supported in Vertex AI Model Garden with some pre-configured for optimized deployment on Vertex AI. When the open model is not available in Vertex AI Model Garden or you want to deploy your model from HF hub, you can leverage the Hugging Face gallery which gives you access to more that 1M models.\n",
"\n",
"With the Vertex AI Model Garden SDK, you can browse and deploy supported models, including those from the Hugging Face gallery. You can also filter by model name or type to find the best fit for your use case.\n",
"\n",
"Let's explore which Llama models are available in Vertex AI Model Garden."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2l5LsbqTevUr"
},
"outputs": [],
"source": [
"model_garden_models = model_garden.list_deployable_models(\n",
" model_filter=\"Llama\", list_hf_models=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "z5Fa1spZr2YZ"
},
"outputs": [],
"source": [
"print_models(model_garden_models, items_per_line=3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EWRuFtX4iXW4"
},
"source": [
"To include Llama models that are available via Hugging Face Gallery, you can enable `list_hf_models` flag."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e9zjeZ-rrPPk"
},
"outputs": [],
"source": [
"deployable_models = model_garden.list_deployable_models(\n",
" model_filter=\"llama\", list_hf_models=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "On3fzp-erTse"
},
"outputs": [],
"source": [
"print_models(deployable_models)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3BOPydegEkP2"
},
"source": [
"## Deploy your 1st Model Garden model\n",
"\n",
"In this section, you'll deploy an open-source model from Vertex AI Model Garden to a public endpoint. You can use the Cloud Console or Vertex AI SDK for deployment.\n",
"\n",
"For more advanced workflows (e.g., custom hardware, model tuning, or shared endpoints), refer to the [official SDK notebook](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4qGzyKIPrkFE"
},
"outputs": [],
"source": [
"model_id = \"meta/llama3_1@llama-3.1-8b-instruct\"\n",
"\n",
"model = model_garden.OpenModel(model_id)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vSFfpZ7Nxlkt"
},
"source": [
"### Check the deployment configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cA_yIRK-t_2T"
},
"source": [
"After you initiate the model, use `list_deploy_options()` method to discover the verified deployment configurations supported by a specific model.\n",
"\n",
"This is important to verify if you have enough resources to deploy the model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PaOvRoFZt9T1"
},
"outputs": [],
"source": [
"deploy_options = model.list_deploy_options(concise=True)\n",
"print(deploy_options)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C3OKT16hugDY"
},
"source": [
"### Deploy the model\n",
"\n",
"Now that you know how the model will be deployed, let's use the `deploy()` method to serve the selected open model to a Vertex AI Endpoint. Depending on the model, the deployment would require some minutes.\n",
"\n",
"> **Note**: If the model has an End User License Agreement (EULA), you can accept it using `accept_eula` flag.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0YqJ9pKi0cjz"
},
"outputs": [],
"source": [
"endpoint = model.deploy(\n",
" accept_eula=True,\n",
" serving_container_image_uri=\"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20241016_0916_RC00_maas\",\n",
" machine_type=\"g2-standard-12\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=1,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sai-h7DryWKi"
},
"source": [
"## LiteLLM Inference"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Set up LiteLLM\n",
"\n",
"Configure [LiteLLM](https://docs.litellm.ai/) to route OpenAI-compatible API calls (e.g. chat, function calling) to your Vertex AI endpoint. Use the `vertex_ai/openai/{endpoint_id}` format to enable OpenAI-style behavior. Other formats like `vertex_ai/gemini/{endpoint}` are also supported — see the full list [here](https://litellm-api.up.railway.app/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def _setup_model_garden_endpoint(endpoint_resource_name: str) -> str:\n",
" \"\"\"Sets up the model garden endpoint by extracting project, location, and endpoint ID.\n",
"\n",
" Args:\n",
" endpoint_resource_name: The full resource name of the endpoint,\n",
" e.g., \"projects/{project}/locations/{location}/endpoints/{endpoint}\".\n",
"\n",
" Returns:\n",
" A formatted string for the endpoint, e.g., \"vertex_ai/openai/{endpoint_id}\".\n",
"\n",
" Raises:\n",
" ValueError: If the endpoint_resource_name does not match the expected format.\n",
" \"\"\"\n",
" _MODEL_GARDEN_ENDPOINT_REGEX = r\"^projects/([^/]+)/locations/([^/]+)/endpoints/([^/]+)$\"\n",
"\n",
" match = re.fullmatch(_MODEL_GARDEN_ENDPOINT_REGEX, endpoint_resource_name)\n",
" if not match:\n",
" raise ValueError(\n",
" f\"Invalid model garden endpoint: '{endpoint_resource_name}'. \"\n",
" \"Please use the format 'projects/{{project}}/locations/{{location}}/endpoints/{{endpoint}}'.\"\n",
" )\n",
"\n",
" project_id, location_id, endpoint_id = match.groups()\n",
"\n",
" os.environ.setdefault(\"GOOGLE_GENAI_USE_VERTEXAI\", \"True\")\n",
" os.environ[\"VERTEXAI_PROJECT\"] = project_id\n",
" os.environ[\"VERTEXAI_LOCATION\"] = location_id\n",
" os.environ[\"LITELLM_LOG\"] = \"DEBUG\" # Consider if this is always desired in production\n",
"\n",
" return f\"vertex_ai/openai/{endpoint_id}\"\n",
"\n",
"deployed_model = _setup_model_garden_endpoint(endpoint.resource_name)\n",
"print(f\"The current model is: {deployed_model}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Chat Completion via LiteLLM\n",
"\n",
"This section demonstrates how to send role-based chat messages to your Vertex AI–hosted model using LiteLLM’s `completion` API. For examples and formatting guidance, refer to the [LiteLLM input documentation](https://docs.litellm.ai/completion/input)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from litellm import completion\n",
"\n",
"response = completion(\n",
" model=deployed_model,\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"What's the capital of Japan?\"}\n",
" ]\n",
")\n",
"\n",
"print(response[\"choices\"][0][\"message\"][\"content\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Function Calling via LiteLLM\n",
"\n",
"Here we use OpenAI-style function calling with LiteLLM and a Vertex AI–hosted model. You’ll define a function schema, and the model will return structured arguments suitable for downstream execution. For reference, see [`test_parallel_function_call`](https://docs.litellm.ai/docs/completion/function_call#full-code---parallel-function-calling-with-gpt-35-turbo-1106) in the LiteLLM documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import google.auth\n",
"\n",
"creds, project = google.auth.default()\n",
"auth_req = google.auth.transport.requests.Request()\n",
"creds.refresh(auth_req)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import litellm\n",
"import json\n",
"\n",
"# set openai api key\n",
"import os\n",
"os.environ['OPENAI_API_KEY'] = creds.token\n",
"\n",
"# Example dummy function hard coded to return the same weather\n",
"# In production, this could be your backend API or an external API\n",
"def get_current_weather(location, unit=\"fahrenheit\"):\n",
" \"\"\"Get the current weather in a given location\"\"\"\n",
" if \"tokyo\" in location.lower():\n",
" return json.dumps({\"location\": \"Tokyo\", \"temperature\": \"10\", \"unit\": \"celsius\"})\n",
" elif \"san francisco\" in location.lower():\n",
" return json.dumps({\"location\": \"San Francisco\", \"temperature\": \"72\", \"unit\": \"fahrenheit\"})\n",
" elif \"paris\" in location.lower():\n",
" return json.dumps({\"location\": \"Paris\", \"temperature\": \"22\", \"unit\": \"celsius\"})\n",
" else:\n",
" return json.dumps({\"location\": location, \"temperature\": \"unknown\"})\n",
"\n",
"\n",
"def test_parallel_function_call():\n",
" try:\n",
" # Step 1: send the conversation and available functions to the model\n",
" messages = [{\"role\": \"user\", \"content\": \"What's the weather like in San Francisco, Tokyo, and Paris?\"}]\n",
" tools = [\n",
" {\n",
" \"type\": \"function\",\n",
" \"function\": {\n",
" \"name\": \"get_current_weather\",\n",
" \"description\": \"Get the current weather in a given location\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"location\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
" },\n",
" \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n",
" },\n",
" \"required\": [\"location\"],\n",
" },\n",
" },\n",
" }\n",
" ]\n",
" response = litellm.completion(\n",
" model=deployed_model,\n",
" messages=messages,\n",
" tools=tools,\n",
" tool_choice=\"auto\", # auto is default, but we'll be explicit\n",
" )\n",
" print(\"\\nFirst LLM Response:\\n\", response)\n",
" response_message = response.choices[0].message\n",
" tool_calls = response_message.tool_calls\n",
"\n",
" print(\"\\nLength of tool calls\", len(tool_calls))\n",
"\n",
" # Step 2: check if the model wanted to call a function\n",
" if tool_calls:\n",
" # Step 3: call the function\n",
" # Note: the JSON response may not always be valid; be sure to handle errors\n",
" available_functions = {\n",
" \"get_current_weather\": get_current_weather,\n",
" } # only one function in this example, but you can have multiple\n",
" messages.append(response_message) # extend conversation with assistant's reply\n",
"\n",
" # Step 4: send the info for each function call and function response to the model\n",
" for tool_call in tool_calls:\n",
" function_name = tool_call.function.name\n",
" function_to_call = available_functions[function_name]\n",
" function_args = json.loads(tool_call.function.arguments)\n",
" function_response = function_to_call(\n",
" location=function_args.get(\"location\"),\n",
" unit=function_args.get(\"unit\"),\n",
" )\n",
" messages.append(\n",
" {\n",
" \"tool_call_id\": tool_call.id,\n",
" \"role\": \"tool\",\n",
" \"name\": function_name,\n",
" \"content\": function_response,\n",
" }\n",
" ) # extend conversation with function response\n",
" second_response = litellm.completion(\n",
" model=deployed_model,\n",
" messages=messages,\n",
" ) # get a new response from the model where it can see the function response\n",
" print(\"\\nSecond LLM response:\\n\", second_response)\n",
" return second_response\n",
" except Exception as e:\n",
" print(f\"Error occurred: {e}\")\n",
"\n",
"test_parallel_function_call()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N5UrKKMifeGi"
},
"outputs": [],
"source": [
"delete_endpoints = True\n",
"\n",
"if delete_endpoints:\n",
" endpoint.delete(force=True)"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_model_garden_sdk.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/use-cases/vertex_ai_deepseek_smolagents.ipynb
+1498
View File
File diff suppressed because it is too large Load Diff