mirrors/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions 1 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:30:30 +08:00
commit 914fea506e
2793 changed files with 802106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
open-models/serving/cloud_run_ollama_gemma3_inference.ipynb
+582
View File
@@ -0,0 +1,582 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Serving Gemma 3 with Ollama on Cloud Run\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fcloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "83b98b0ba19c"
},
"source": [
"<img src=\"https://ollama.com/public/ollama.png\" height=\"200px\" alignment=\"center\"/>\n",
"<img src=\"https://cloud.google.com/static/architecture/images/ac-page-icons/card_google_cloud_partner.svg\" height=\"200px\">\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Vlad Kolesnikov](https://github.com/vladkol) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Gemma 3**](https://ai.google.dev/gemma) is a new generation of open models developed by Google. It is a collection of lightweight, state-of-the-art open models built from the same research and technology that powers our Gemini 2.0 models. Gemma 3 comes in a range of sizes (1B, 4B, 12B and 27B), allowing you to choose the best model for your specific hardware and performance needs. Gemma 3 models are available through platforms like Google AI Studio, Kaggle, and Hugging Face.\n",
"\n",
"> **[Cloud Run](https://cloud.google.com/run)**:\n",
"It's a serverless platform by Google Cloud for running containerized applications. It automatically scales and manages infrastructure, supporting various programming languages. Cloud Run now offers GPU acceleration for AI/ML workloads. With 30 seconds to the first token, Cloud Run is a perfect platform for serving lightweight models like Gemma.\n",
"\n",
"> **Note:** GPU support in Cloud Run is in preview. To use the GPU feature, you must request `Total Nvidia L4 GPU allocation, per project per region` quota under Cloud Run in the [Quotas and system limits page](https://cloud.google.com/run/quotas#increase).\n",
"\n",
"\n",
"> **[Ollama](ollama.com)**: is an open-source tool for easily running and deploying large language models locally. It offers simple management and usage of LLMs on personal computers or servers.\n",
"\n",
"This notebook showcase how to deploy [Google Gemma 3](https://developers.googleblog.com/en/introducing-gemma3) in Cloud Run, with the objective to build a simple API for chat or RAG applications.\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"1. Deploy Google Gemma 3 as an OpenAI-compatible API on Cloud Run using Ollama.\n",
"2. Build a custom container with Ollama to deploy any Large Language Model (LLM) of your choice.\n",
"3. Make requests to an API hosted on Cloud Run."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aOiPjM5DEPhK"
},
"source": [
"### Install Google Cloud SDK\n",
"\n",
"Make sure you Google Cloud SDK is installed (try running `gcloud version`) or [install it](https://cloud.google.com/sdk/docs/install) before executing this notebook.\n",
"\n",
"> If you are running in Colab or Vertex AI workbench, you have Google Cloud SDK installed."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HfAVa08RDDJB"
},
"source": [
"### Choose a model, a project, and a region to host the model\n",
"\n",
"Choose a Gemma 3 model to use, a Google Cloud project to host your Cloud Run service, and a region to host it in.\n",
"\n",
"If you don't have a project yet:\n",
"\n",
"1. [Create a project](https://console.cloud.google.com/projectcreate) in the Google Cloud Console.\n",
"2. Copy your `Project ID` from the project's [Settings page](https://console.cloud.google.com/iam-admin/settings).\n",
"\n",
"The project must have `Total Nvidia L4 GPU allocation, per project per region` quota allocated in the selected region.\n",
"To make sure it's available, check Cloud Run in the [Quotas and system limits page](https://console.cloud.google.com/iam-admin/quotas)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "TV0pbqJHDDJB"
},
"outputs": [],
"source": [
"# { display-mode: \"form\", run: \"auto\" }\n",
"\n",
"MODEL = \"gemma3:4b\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"REGION = \"us-central1\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"if PROJECT_ID == \"[your-project-id]\" or not PROJECT_ID:\n",
" print(\"Please specify your project id in PROJECT_ID variable.\")\n",
" raise KeyboardInterrupt\n",
"\n",
"MODEL_NAME_ESCAPED = MODEL.translate(str.maketrans(\".:/\", \"---\"))\n",
"SERVICE_NAME = f\"ollama--{MODEL_NAME_ESCAPED}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Run the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xc8Jm1P3Y7fs"
},
"outputs": [],
"source": [
"!gcloud auth print-identity-token -q &> /dev/null || gcloud auth login --project=\"{PROJECT_ID}\" --update-adc --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l728UOEPDDJB"
},
"source": [
"## Prepare container image\n",
"\n",
"First, let's create a Docker file for a container with the model embedded into it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "glBn9gPKDDJB"
},
"outputs": [],
"source": [
"%%writefile Dockerfile\n",
"\n",
"FROM ollama/ollama:0.6.0\n",
"\n",
"ARG MODEL\n",
"\n",
"# Set the model name\n",
"ENV MODEL=$MODEL\n",
"\n",
"# Set the host and port to listen on\n",
"ENV OLLAMA_HOST 0.0.0.0:8080\n",
"\n",
"# Set the directory to store model weight files\n",
"ENV OLLAMA_MODELS /models\n",
"\n",
"# Reduce the verbosity of the logs\n",
"ENV OLLAMA_DEBUG false\n",
"\n",
"# Do not unload model weights from the GPU\n",
"ENV OLLAMA_KEEP_ALIVE -1\n",
"\n",
"# Start the ollama server and download the model weights\n",
"RUN ollama serve & sleep 5 && ollama pull $MODEL\n",
"\n",
"# At startup time we start the server and run a dummy request\n",
"# to request the model to be loaded in the GPU memory\n",
"ENTRYPOINT [\"/bin/sh\"]\n",
"CMD [\"-c\", \"ollama serve & (ollama run $MODEL 'Say one word' &) && wait\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T4gS8ovMDDJB"
},
"source": [
"Second, we create a Cloud Build file to use for building and pushing our container image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1dV1_cMDDDJB"
},
"outputs": [],
"source": [
"%%writefile cloudbuild.yaml\n",
"\n",
"steps:\n",
"- name: 'gcr.io/cloud-builders/docker'\n",
" id: build\n",
" entrypoint: 'bash'\n",
" args:\n",
" - -c\n",
" - |\n",
" docker buildx build --tag=${_IMAGE} --build-arg MODEL=${_MODEL} .\n",
"\n",
"images: [\"${_IMAGE}\"]\n",
"\n",
"substitutions:\n",
" _IMAGE: '${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_AR_REPO_NAME}/${_SERVICE_NAME}'\n",
"\n",
"options:\n",
" dynamicSubstitutions: true\n",
" machineType: \"E2_HIGHCPU_32\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vbDiABJcDDJC"
},
"source": [
"## Build Container Image and Deploy Cloud Run Service\n",
"\n",
"We are ready to build our container image and deploy Cloud Run service.\n",
"\n",
"The script below performs the following actions:\n",
"\n",
"* Enables necessary APIs.\n",
"* Creates an Artifact Repository for the image.\n",
"* Creates a Service Account for the service.\n",
"* Submits a Cloud Build job to create and push the container image.\n",
"* Deploys the Cloud Run service.\n",
"\n",
"> The script may take 10-45 minutes to finish.\n",
"\n",
"Note the following important flags in Cloud Build deployment command:\n",
"\n",
"* `--concurrency 4` is set to match the value of the environment variable `OLLAMA_NUM_PARALLEL`.\n",
"* `--gpu 1` with `--gpu-type nvidia-l4` assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.\n",
"`--no-allow-authenticated` restricts unauthenticated access to the service.\n",
"By keeping the service private, you can rely on Cloud Run's built-in [Identity and Access Management (IAM)](https://cloud.google.com/iam) authentication for service-to-service communication.\n",
"* `--no-cpu-throttling` is required for enabling GPU.\n",
"* `--service-account` the service identity of the service.\n",
"* `--max-instances` sets maximum number of instances of the service.\n",
"It has to be equal to or lower than your project's NVIDIA L4 GPU (`Total Nvidia L4 GPU allocation, per project per region`) quota.\n",
"\n",
"For optimal GPU utilization, increase `--concurrency`, keeping it within twice the value of `OLLAMA_NUM_PARALLEL`.\n",
"While this leads to request queuing in Ollama, it can help improve utilization:\n",
"Ollama instances can immediately process requests from their queue, and the queues help absorb traffic spikes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TXg7IYU1DDJC"
},
"outputs": [],
"source": [
"%%writefile deploy.sh\n",
"\n",
"PROJECT_ID=$1\n",
"REGION=$2\n",
"MODEL_ID=\"${3}\"\n",
"SERVICE_NAME=\"${4}\"\n",
"AR_REPO_NAME=\"ollama-repo\"\n",
"SERVICE_ACCOUNT=\"ollama-cloud-run-sa\"\n",
"SERVICE_ACCOUNT_ADDRESS=\"${SERVICE_ACCOUNT}@$PROJECT_ID.iam.gserviceaccount.com\"\n",
"MAX_INSTANCES=1 # Adjust this value to match your Cloud Run L4 GPU quota (\"Total Nvidia L4 GPU allocation, per project per region\", NvidiaL4GpuAllocPerProjectRegion, run.googleapis.com/nvidia_l4_gpu_allocation)\n",
"\n",
"echo \"Enabling APIs in project ${PROJECT_ID}.\"\n",
"gcloud services enable run.googleapis.com \\\n",
" cloudbuild.googleapis.com \\\n",
" artifactregistry.googleapis.com \\\n",
" --project ${PROJECT_ID} \\\n",
" --quiet\n",
"\n",
"set -e\n",
"\n",
"# Creating the service account if doesn't exist.\n",
"sa_list=$(gcloud iam service-accounts list --quiet --format 'value(email)' --project $PROJECT_ID --filter=email:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com 2>/dev/null)\n",
"if [ -z \"${sa_list}\" ]; then\n",
" echo \"Creating Service Account ${SERVICE_ACCOUNT}.\"\n",
" gcloud iam service-accounts create $SERVICE_ACCOUNT \\\n",
" --project ${PROJECT_ID} \\\n",
" --display-name=\"${SERVICE_ACCOUNT} - Cloud Run Service Account\"\n",
"fi\n",
"\n",
"# Creating the Artifacts Repository if doesn't exist\n",
"repo_list=$(gcloud artifacts repositories list --format 'value(name)' --filter=name=\"projects/${PROJECT_ID}/locations/${REGION}/repositories/${AR_REPO_NAME}\" --project ${PROJECT_ID} --quiet --location ${REGION} 2>/dev/null)\n",
"if [ -z \"${repo_list}\" ]; then\n",
" echo \"Creating Artifact Registry ${AR_REPO_NAME}.\"\n",
" gcloud artifacts repositories create $AR_REPO_NAME \\\n",
" --repository-format docker \\\n",
" --location ${REGION} \\\n",
" --project=${PROJECT_ID}\n",
"fi\n",
"\n",
"echo \"Building container image.\"\n",
"gcloud builds submit --config=cloudbuild.yaml --project=${PROJECT_ID} . \\\n",
" --suppress-logs \\\n",
" --substitutions \\\n",
" _AR_REPO_NAME=$AR_REPO_NAME,_REGION=$REGION,_SERVICE_NAME=$SERVICE_NAME,_MODEL=$MODEL_ID\n",
"rm -f cloudbuild.yaml\n",
"rm -f Dockerfile\n",
"\n",
"echo \"Deploying Service ${SERVICE_NAME}.\"\n",
"gcloud beta run deploy $SERVICE_NAME \\\n",
" --project=${PROJECT_ID} \\\n",
" --image=${REGION}-docker.pkg.dev/$PROJECT_ID/$AR_REPO_NAME/$SERVICE_NAME \\\n",
" --service-account $SERVICE_ACCOUNT_ADDRESS \\\n",
" --cpu=8 \\\n",
" --memory=32Gi \\\n",
" --gpu=1 --gpu-type=nvidia-l4 \\\n",
" --concurrency 4 \\\n",
" --set-env-vars OLLAMA_NUM_PARALLEL=4 \\\n",
" --region ${REGION} \\\n",
" --no-allow-unauthenticated \\\n",
" --max-instances ${MAX_INSTANCES} \\\n",
" --no-cpu-throttling \\\n",
" --timeout 1h\n",
"\n",
"SERVICE_URL=$(gcloud run services describe ${SERVICE_NAME} --project=${PROJECT_ID} --region $REGION --format 'value(status.url)' --quiet)\n",
"echo \"✅ Success!\"\n",
"echo \"🚀 Service URL: ${SERVICE_URL}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e6L2dVGyOAxB"
},
"outputs": [],
"source": [
"!/bin/bash ./deploy.sh \"{PROJECT_ID}\" \"{REGION}\" \"{MODEL}\" \"{SERVICE_NAME}\" && rm -f ./deploy.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dgaJ62rmDDJC"
},
"source": [
"## Test the deployed service\n",
"\n",
"Now, let's test the service you deployed.\n",
"\n",
"First, simply by using `cURL`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iX7LmWwGDDJC"
},
"outputs": [],
"source": [
"%%bash -s $MODEL $SERVICE_NAME $PROJECT_ID $REGION\n",
"\n",
"PROMPT=\"Hello!\"\n",
"SERVICE_URL=$(gcloud run services describe ${2} --project ${3} --region ${4} --format 'value(status.url)' --quiet)\n",
"AUTH_TOKEN=$(gcloud auth print-identity-token -q)\n",
"\n",
"curl -s -X POST \"${SERVICE_URL}/api/generate\" \\\n",
"-H \"Authorization: Bearer ${AUTH_TOKEN}\" \\\n",
"-H \"Content-Type: application/json\" \\\n",
"-d '{ \"model\": \"'${1}'\", \"prompt\": \"'${PROMPT}'\", \"max_tokens\": 100, \"stream\": false}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "63oQqBNmDDJC"
},
"source": [
"### Ollama Python Library\n",
"\n",
"You can also use Ollama Python Library to make requests to the service you deployed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "h8G3te5pDDJC"
},
"outputs": [],
"source": [
"# Install Ollama Python Library\n",
"%pip install ollama -q"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "X-2TbV6tDDJC"
},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"from ollama import Client\n",
"\n",
"identity_token = (\n",
" subprocess.check_output(\"gcloud auth print-identity-token -q\", shell=True)\n",
" .decode()\n",
" .strip()\n",
")\n",
"service_url = (\n",
" subprocess.check_output(\n",
" (\n",
" \"gcloud run services describe \"\n",
" f\"{SERVICE_NAME} --project={PROJECT_ID} \"\n",
" f\"--region={REGION} \"\n",
" \"--format='value(status.url)' -q\"\n",
" ),\n",
" shell=True,\n",
" )\n",
" .decode()\n",
" .strip()\n",
")\n",
"client = Client(host=service_url, headers={\"Authorization\": f\"Bearer {identity_token}\"})\n",
"stream = client.chat(\n",
" model=MODEL,\n",
" messages=[{\"role\": \"user\", \"content\": \"Why is the sky blue?\"}],\n",
" stream=True,\n",
")\n",
"for chunk in stream:\n",
" print(chunk[\"message\"][\"content\"], end=\"\", flush=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FFL2V_ClDDJD"
},
"source": [
"## Conclusion\n",
"Congratulations! 💎 Now you know how to deploy Gemma 3 with Ollama to Cloud Run powered by a GPU!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s1blF2ziDDJD"
},
"source": [
"To delete the Cloud Run service you created, you can uncomment and run the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VbhAz7-9DDJD"
},
"outputs": [],
"source": [
"# !gcloud run services delete $SERVICE_NAME --project $PROJECT_ID --region $LOCATION --quiet"
]
}
],
"metadata": {
"colab": {
"name": "cloud_run_ollama_gemma3_inference.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/cloud_run_ollama_qwen3_inference.ipynb
+736
View File
@@ -0,0 +1,736 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Running Qwen 3 with Ollama in Cloud Run for Agents\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fcloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_ollama_qwen3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Vlad Kolesnikov](https://github.com/vladkol) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b1455cd3766f"
},
"source": [
"<style>\n",
"td, th {\n",
" border: none!important;\n",
"}\n",
"</style>\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://camo.githubusercontent.com/8793b3b4014d538b367ec8819dcca85e79cb8d910c808fa7849e3cd85e2ebe79/68747470733a2f2f7169616e77656e2d7265732e6f73732d616363656c65726174652d6f766572736561732e616c6979756e63732e636f6d2f6c6f676f5f7177656e332e706e67\" width=\"100px\"/>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://ollama.com/public/ollama.png\" height=\"50px\"/>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://google.github.io/adk-docs/assets/agent-development-kit.png\" height=\"50px\">\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <img src=\"https://www.gstatic.com/bricks/image/f2e0986a2802c0b6c4be7f1355599d5aadfb21a63b7e9643d96697ff9334a1e1.svg\" height=\"50px\">\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Qwen 3**](https://qwenlm.github.io/blog/qwen3/) is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models.\n",
"It supports thinking and function calling.\n",
"\n",
"> **[Cloud Run](https://cloud.google.com/run)**:\n",
"It's a serverless platform by Google Cloud for running containerized applications. It automatically scales and manages infrastructure, supporting various programming languages. Cloud Run now offers GPU acceleration for AI/ML workloads. With 30 seconds to the first token, Cloud Run is a perfect platform for serving lightweight models.\n",
"\n",
"> **Note:** [GPU support in Cloud Run](https://cloud.google.com/run/docs/configuring/services/gpu) is Generally Available.\n",
"To use the GPU feature, your project must have `Total Nvidia L4 GPU allocation without zonal redundancy, per project per region`.\n",
"\n",
"> **[Ollama](ollama.com)**: is an open-source tool for easily running and deploying large language models locally. It offers simple management and usage of LLMs on personal computers or servers.\n",
"\n",
"This notebook showcase how to deploy [Qwen 3](https://developers.googleblog.com/en/introducing-gemma3) in Cloud Run,\n",
"with the objective to an API for running AI Agents built with Google [Agent Development Kit](https://google.github.io/adk-docs/).\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"1. Deploy Qwen 3 as an OpenAI-compatible API on Cloud Run using Ollama.\n",
"2. Build a custom container with Ollama to deploy any Large Language Model (LLM) of your choice.\n",
"3. Make requests to an API hosted on Cloud Run.\n",
"4. Create and run an Agent that uses Qwen 3.\n",
"\n",
"We will build an agent using [Agent Development Kit](https://google.github.io/adk-docs/) - a flexible and modular model-agnostic framework for developing and deploying AI agents."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aOiPjM5DEPhK"
},
"source": [
"### Install Google Cloud CLI\n",
"\n",
"Make sure you Google Cloud CLI is installed (try running `gcloud version`) or [install it](https://cloud.google.com/sdk/docs/install) before executing this notebook.\n",
"\n",
"> If you are running in Colab or Vertex AI workbench, you already have Google Cloud CLI installed."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b0b84cb331d5"
},
"source": [
"### Install Agent Development Kit and other required packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "330f4fbf7da9"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai google-adk litellm"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HfAVa08RDDJB"
},
"source": [
"### Choose a model, a project, and a region to host the model\n",
"\n",
"[Choose a Qwen 3 model](https://ollama.com/library/qwen3) to use, a Google Cloud project to host your Cloud Run service, and a region to host it in.\n",
"Ollama offers multiple sizes with different quantization.\n",
"In this notebook, we use [Qwen3:8b](https://ollama.com/library/qwen3:8b) with `Q4_K_M` quantization.\n",
"\n",
"For Google Cloud project, if you don't have a project yet:\n",
"\n",
"1. [Create a project](https://console.cloud.google.com/projectcreate) in the Google Cloud Console.\n",
"2. Copy your `Project ID` from the project's [Settings page](https://console.cloud.google.com/iam-admin/settings).\n",
"\n",
"The project must have `Total Nvidia L4 GPU allocation without zonal redundancy, per project per region` quota allocated in the selected region.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "TV0pbqJHDDJB"
},
"outputs": [],
"source": [
"# { display-mode: \"form\", run: \"auto\" }\n",
"\n",
"MODEL = \"qwen3:8b\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"REGION = \"us-central1\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"if PROJECT_ID == \"[your-project-id]\" or not PROJECT_ID:\n",
" print(\"Please specify your project id in PROJECT_ID variable.\")\n",
" raise KeyboardInterrupt\n",
"\n",
"MODEL_NAME_ESCAPED = MODEL.translate(str.maketrans(\".:/\", \"---\"))\n",
"SERVICE_NAME = f\"ollama--{MODEL_NAME_ESCAPED}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6c36c31ee2de"
},
"outputs": [],
"source": [
"### Python dependency imports\n",
"import os\n",
"import subprocess\n",
"from datetime import datetime"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Run the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xc8Jm1P3Y7fs"
},
"outputs": [],
"source": [
"!gcloud auth print-identity-token -q &> /dev/null || gcloud auth login --project=\"{PROJECT_ID}\" --update-adc --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l728UOEPDDJB"
},
"source": [
"## Prepare serving container\n",
"\n",
"First, let's create a Docker file for a container with the model embedded into it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f56f9803255a"
},
"outputs": [],
"source": [
"!rm -f Dockerfile\n",
"!echo \"ARG _MODEL=\\\"{MODEL}\\\"\" > Dockerfile"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "glBn9gPKDDJB"
},
"outputs": [],
"source": [
"%%writefile -a Dockerfile\n",
"\n",
"FROM ollama/ollama:latest\n",
"ARG _MODEL\n",
"\n",
"# Set the host and port to listen on\n",
"ENV OLLAMA_HOST 0.0.0.0:8080\n",
"\n",
"# Set the directory to store model weight files\n",
"ENV OLLAMA_MODELS /models\n",
"\n",
"# Reduce the verbosity of the logs\n",
"ENV OLLAMA_DEBUG false\n",
"\n",
"# Do not unload model weights from the GPU\n",
"ENV OLLAMA_KEEP_ALIVE -1\n",
"\n",
"# Start the ollama server and download the model weights\n",
"RUN ollama serve & sleep 5 && ollama pull $_MODEL\n",
"\n",
"# At startup time we start the server and run a dummy request\n",
"# to request the model to be loaded in the GPU memory\n",
"ENTRYPOINT [\"/bin/sh\"]\n",
"CMD [\"-c\", \"ollama serve & (ollama run $_MODEL 'Say one word' &) && wait\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vbDiABJcDDJC"
},
"source": [
"## Build Container Image and Deploy Cloud Run Service\n",
"\n",
"We are ready to build our container image and deploy Cloud Run service.\n",
"\n",
"The script below performs the following actions:\n",
"\n",
"* Enables necessary APIs.\n",
"* Creates an Artifact Repository for the image.\n",
"* Creates a Service Account for the service.\n",
"* Submits a Cloud Build job to create and push the container image.\n",
"* Deploys the Cloud Run service.\n",
"\n",
"> The script may take 10-45 minutes to finish.\n",
"\n",
"Note the following important flags in Cloud Build deployment command:\n",
"\n",
"* `--concurrency 4` is set to match the value of the environment variable `OLLAMA_NUM_PARALLEL`.\n",
"* `--gpu 1` with `--gpu-type nvidia-l4` assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.\n",
"* `--no-gpu-zonal-redundancy` allows using the default GPU quota.\n",
"* `--no-allow-authenticated` restricts unauthenticated access to the service.\n",
"By keeping the service private, you can rely on Cloud Run's built-in [Identity and Access Management (IAM)](https://cloud.google.com/iam) authentication for service-to-service communication.\n",
"* `--no-cpu-throttling` is required for enabling GPU.\n",
"* `--service-account` the service identity of the service.\n",
"* `--max-instances` sets maximum number of instances of the service.\n",
"It has to be equal to or lower than your project's NVIDIA L4 GPU quota.\n",
"\n",
"For optimal GPU utilization, increase `--concurrency`, keeping it within twice the value of `OLLAMA_NUM_PARALLEL`.\n",
"While this leads to request queuing in Ollama, it can help improve utilization:\n",
"Ollama instances can immediately process requests from their queue, and the queues help absorb traffic spikes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TXg7IYU1DDJC"
},
"outputs": [],
"source": [
"%%writefile deploy.sh\n",
"\n",
"PROJECT_ID=$1\n",
"REGION=$2\n",
"MODEL_ID=\"${3}\"\n",
"SERVICE_NAME=\"${4}\"\n",
"SERVICE_ACCOUNT=\"ollama-cloud-run-sa\"\n",
"SERVICE_ACCOUNT_ADDRESS=\"${SERVICE_ACCOUNT}@$PROJECT_ID.iam.gserviceaccount.com\"\n",
"MAX_INSTANCES=1 # Adjust this value to match your Cloud Run L4 GPU quota\n",
"\n",
"echo \"Enabling APIs in project ${PROJECT_ID}.\"\n",
"gcloud services enable run.googleapis.com \\\n",
" cloudbuild.googleapis.com \\\n",
" artifactregistry.googleapis.com \\\n",
" --project ${PROJECT_ID} \\\n",
" --quiet\n",
"\n",
"set -e\n",
"\n",
"# Creating the service account if doesn't exist.\n",
"sa_list=$(gcloud iam service-accounts list --quiet --format 'value(email)' --project $PROJECT_ID --filter=email:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com 2>/dev/null)\n",
"if [ -z \"${sa_list}\" ]; then\n",
" echo \"Creating Service Account ${SERVICE_ACCOUNT}.\"\n",
" gcloud iam service-accounts create $SERVICE_ACCOUNT \\\n",
" --project ${PROJECT_ID} \\\n",
" --display-name=\"${SERVICE_ACCOUNT} - Cloud Run Service Account\"\n",
"fi\n",
"\n",
"echo \"Deploying Service ${SERVICE_NAME}. It will take a few minutes...\"\n",
"gcloud beta run deploy $SERVICE_NAME \\\n",
" --source . \\\n",
" --project=${PROJECT_ID} \\\n",
" --service-account $SERVICE_ACCOUNT_ADDRESS \\\n",
" --cpu=8 \\\n",
" --memory=32Gi \\\n",
" --gpu=1 \\\n",
" --gpu-type=nvidia-l4 \\\n",
" --concurrency 4 \\\n",
" --set-env-vars OLLAMA_NUM_PARALLEL=4 \\\n",
" --region ${REGION} \\\n",
" --no-allow-unauthenticated \\\n",
" --max-instances ${MAX_INSTANCES} \\\n",
" --no-cpu-throttling \\\n",
" --timeout 1h \\\n",
" --no-gpu-zonal-redundancy \\\n",
" --quiet \\\n",
" --no-user-output-enabled\n",
"\n",
"rm -f ./Dockerfile # Cleanup\n",
"\n",
"SERVICE_URL=$(gcloud run services describe ${SERVICE_NAME} --project=${PROJECT_ID} --region $REGION --format 'value(status.url)' --quiet)\n",
"echo \"✅ Success!\"\n",
"echo \"🚀 Service URL: ${SERVICE_URL}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e6L2dVGyOAxB"
},
"outputs": [],
"source": [
"!/bin/bash ./deploy.sh \"{PROJECT_ID}\" \"{REGION}\" \"{MODEL}\" \"{SERVICE_NAME}\" && rm -f ./deploy.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dgaJ62rmDDJC"
},
"source": [
"### Test the deployed service\n",
"\n",
"Now, let's test the service you deployed.\n",
"\n",
"First, simply by using `cURL`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iX7LmWwGDDJC"
},
"outputs": [],
"source": [
"%%bash -s $MODEL $SERVICE_NAME $PROJECT_ID $REGION\n",
"\n",
"PROMPT=\"Hello!\"\n",
"SERVICE_URL=$(gcloud run services describe ${2} --project ${3} --region ${4} --format 'value(status.url)' --quiet)\n",
"AUTH_TOKEN=$(gcloud auth print-identity-token -q)\n",
"\n",
"curl -s -X POST \"${SERVICE_URL}/api/generate\" \\\n",
"-H \"Authorization: Bearer ${AUTH_TOKEN}\" \\\n",
"-H \"Content-Type: application/json\" \\\n",
"-d '{ \"model\": \"'${1}'\", \"prompt\": \"'${PROMPT}'\", \"max_tokens\": 100, \"stream\": false}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2b60e54c901d"
},
"source": [
"## Create an AI Agent with Qwen 3"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1e1f3cb538ed"
},
"source": [
"#### Retrieve an Identity Token\n",
"\n",
"Cloud Run with authentication by\n",
"[Google Cloud IAM](https://cloud.google.com/iam/docs/) requires an [identity token](https://cloud.google.com/docs/authentication/get-id-token) in every request's authentication header."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "X-2TbV6tDDJC"
},
"outputs": [],
"source": [
"auth_id_token = (\n",
" subprocess.check_output(\"gcloud auth print-identity-token -q\", shell=True)\n",
" .decode()\n",
" .strip()\n",
")\n",
"service_url = (\n",
" subprocess.check_output(\n",
" (\n",
" \"gcloud run services describe \"\n",
" f\"{SERVICE_NAME} --project={PROJECT_ID} \"\n",
" f\"--region={REGION} \"\n",
" \"--format='value(status.url)' -q\"\n",
" ),\n",
" shell=True,\n",
" )\n",
" .decode()\n",
" .strip()\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2f5ae9755e13"
},
"source": [
"#### Create and run an Agent\n",
"\n",
"We create a simple agent that can answer questions about current time.\n",
"\n",
"First, we make a tool that returns current user's time with time zone offset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d66169bd7a12"
},
"outputs": [],
"source": [
"def get_current_time() -> str:\n",
" \"\"\"Returns user's local time and timezone offset.\n",
"\n",
" Returns:\n",
" str: Time in ISO format with user's timezone offset.\n",
" \"\"\"\n",
" return datetime.now().astimezone().isoformat()\n",
"\n",
"\n",
"print(get_current_time())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b1a1ae76f3bb"
},
"source": [
"Now, we create an agent that can use our tool to answer user's question."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d7c2b8951050"
},
"outputs": [],
"source": [
"from google.adk import Runner\n",
"from google.adk.agents import Agent\n",
"from google.adk.artifacts import InMemoryArtifactService\n",
"from google.adk.models.lite_llm import LiteLlm\n",
"from google.adk.sessions import InMemorySessionService, Session\n",
"from google.genai import types\n",
"\n",
"os.environ[\"OLLAMA_API_BASE\"] = service_url # still required for LiteLlm to work\n",
"\n",
"root_agent = Agent(\n",
" name=\"time_agent\",\n",
" model=LiteLlm(\n",
" model=f\"openai/{MODEL}\",\n",
" api_base=service_url + \"/v1\",\n",
" api_key=auth_id_token,\n",
" temperature=0.1, # for stable function calling\n",
" ),\n",
" description=(\"Agent to answer questions about current time.\"),\n",
" # Agent instructions with `/no_think` for Qwen 3 to run faster.\n",
" instruction=\"\"\"\n",
" You are a helpful agent who can answer user questions\n",
" about the current local time.\n",
" Always output date and time is human-readable form.\n",
" /no_think\n",
" \"\"\",\n",
" tools=[get_current_time],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0763595c0eff"
},
"source": [
"Initialize the runtime."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2d8631d9da3f"
},
"outputs": [],
"source": [
"app_name = \"my_app\"\n",
"user_id_1 = \"user1\"\n",
"session_service = InMemorySessionService()\n",
"artifact_service = InMemoryArtifactService()\n",
"runner = Runner(\n",
" app_name=app_name,\n",
" agent=root_agent,\n",
" artifact_service=artifact_service,\n",
" session_service=session_service,\n",
")\n",
"new_session = await session_service.create_session(app_name=app_name, user_id=user_id_1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "64b9387644c6"
},
"source": [
"And run our agent!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f09d7a5e0ca8"
},
"outputs": [],
"source": [
"async def run_prompt(session: Session, new_message: str):\n",
" content = types.Content(role=\"user\", parts=[types.Part.from_text(text=new_message)])\n",
" print(\"** User says:\", new_message)\n",
" async for event in runner.run_async(\n",
" user_id=user_id_1,\n",
" session_id=session.id,\n",
" new_message=content,\n",
" ):\n",
" if not event.content or not event.content.parts:\n",
" continue\n",
" print(f\"** {event.author}:\")\n",
" for part in event.content.parts:\n",
" if part.function_call and part.function_call.name:\n",
" print(\n",
" f\"\\t#### Calling `{part.function_call.name}` \"\n",
" f\"with args: {part.function_call.args}\"\n",
" )\n",
" elif part.function_response and part.function_response.response:\n",
" print(f\"\\t### Function call result: {part.function_response.response}\")\n",
" elif part.text and part.text.strip():\n",
" print(f\"\\t{part.text.strip()}\")\n",
"\n",
"\n",
"QUESTION = \"What time is it now?\"\n",
"await run_prompt(new_session, QUESTION) # if not Jupyter, wrap in asyncio.run"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FFL2V_ClDDJD"
},
"source": [
"## Conclusion\n",
"Congratulations! You can now use Qwen 3 for running your AI Agents built with Agent Development Kit in Cloud Run!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s1blF2ziDDJD"
},
"source": [
"To delete the Cloud Run service you created, you can uncomment and run the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VbhAz7-9DDJD"
},
"outputs": [],
"source": [
"# !gcloud run services delete $SERVICE_NAME --project $PROJECT_ID --region $LOCATION --quiet"
]
}
],
"metadata": {
"colab": {
"name": "cloud_run_ollama_qwen3_inference.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/cloud_run_vllm_gemma3_inference.ipynb
+685
View File
@@ -0,0 +1,685 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Serving Gemma 3 with vLLM on Cloud Run\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fcloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/cloud_run_vllm_gemma3_inference.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "83b98b0ba19c"
},
"source": [
"<img src=\"https://docs.vllm.ai/en/latest/_images/vllm-logo-text-light.png\" width=\"200px\" alignment=\"center\"/>\n",
"<img src=\"https://cloud.google.com/static/architecture/images/ac-page-icons/card_google_cloud_partner.svg\" width=\"100px\">\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Vlad Kolesnikov](https://github.com/vladkol) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Gemma 3**](https://ai.google.dev/gemma) is a new generation of open models developed by Google. It is a collection of lightweight, state-of-the-art open models built from the same research and technology that powers our Gemini 2.0 models. Gemma 3 comes in a range of sizes (1B, 4B, 12B and 27B), allowing you to choose the best model for your specific hardware and performance needs. Gemma 3 models are available through platforms like Google AI Studio, Kaggle, and Hugging Face.\n",
"\n",
"> **[Cloud Run](https://cloud.google.com/run)**:\n",
"It's a serverless platform by Google Cloud for running containerized applications. It automatically scales and manages infrastructure, supporting various programming languages. Cloud Run now offers GPU acceleration for AI/ML workloads. With 30 seconds to the first token, Cloud Run is a perfect platform for serving lightweight models like Gemma.\n",
"\n",
"> **Note:** GPU support in Cloud Run is in preview. To use the GPU feature, you must request `Total Nvidia L4 GPU allocation, per project per region` quota under Cloud Run in the [Quotas and system limits page](https://cloud.google.com/run/quotas#increase).\n",
"\n",
"\n",
"> **[vLLM](https://docs.vllm.ai/)**: is a fast and easy-to-use library for LLM inference and serving. It provides high-throughput serving, optimized model execution, and tensor parallelism.\n",
"\n",
"This notebook showcase how to deploy [Google Gemma 3](https://developers.googleblog.com/en/introducing-gemma3) in Cloud Run, with the objective to build a simple API for chat or RAG applications.\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"1. Deploy Google Gemma 3 as an OpenAI-compatible API on Cloud Run using vLLM.\n",
"2. Build a custom container with vLLM to deploy any Large Language Model (LLM) of your choice.\n",
"3. Make requests to an API hosted on Cloud Run."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Cloud SDK and HuggingFace Hub dependencies\n",
"\n",
"Make sure you Google Cloud SDK is installed (try running `gcloud version`) or [install it](https://cloud.google.com/sdk/docs/install) before executing this notebook.\n",
"\n",
"> If you are running in Colab or Vertex AI workbench, you have Google Cloud SDK installed.\n",
"\n",
"You also need to install `huggingface_hub` Python package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Tm31a7_cDbeu"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet huggingface_hub"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MajzyxmFDbeu"
},
"source": [
"### Choose a model, a project, and a region to host the model\n",
"\n",
"Choose a Gemma 3 model to use, a Google Cloud project to host your Cloud Run service, and a region to host it in.\n",
"\n",
"If you don't have a project yet:\n",
"\n",
"1. [Create a project](https://console.cloud.google.com/projectcreate) in the Google Cloud Console.\n",
"2. Copy your `Project ID` from the project's [Settings page](https://console.cloud.google.com/iam-admin/settings).\n",
"\n",
"The project must have `Total Nvidia L4 GPU allocation, per project per region` quota allocated in the selected region.\n",
"To make sure it's available, check Cloud Run in the [Quotas and system limits page](https://console.cloud.google.com/iam-admin/quotas)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "Do_Z5bVNDbeu"
},
"outputs": [],
"source": [
"# { display-mode: \"form\", run: \"auto\" }\n",
"\n",
"MODEL = \"google/gemma-3-4b-it\" # @param {type:\"string\"}\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"REGION = \"us-central1\" # @param {type:\"string\", isTemplate: true}\n",
"\n",
"if PROJECT_ID == \"[your-project-id]\" or not PROJECT_ID:\n",
" print(\"Please specify your project id in PROJECT_ID variable.\")\n",
" raise KeyboardInterrupt\n",
"\n",
"SERVICE_NAME = f\"vllm--{MODEL.replace('.', '-').replace('/','--')}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Run the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c93d96bc9c43"
},
"outputs": [],
"source": [
"!gcloud auth print-identity-token -q &> /dev/null || gcloud auth login --project=\"{PROJECT_ID}\" --update-adc --quiet"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9af3e57f89a"
},
"source": [
"### Authenticate your Hugging Face account\n",
"\n",
"As Google Gemma (https://huggingface.co/google/gemma-3-4b-it) are gated models, you need to have a Hugging Face Hub account, and accept the Google's usage license for Gemma. Once that's done, you need to generate a new user access token with read-only access so that the weights can be downloaded from the Hub.\n",
"\n",
"> Note that the user access token can only be generated via [the Hugging Face Hub UI](https://huggingface.co/settings/tokens/new), where you can either select read-only access to your account, or follow the recommendations and generate a fine-grained token with read-only access to [`google/gemma-3-4b-it`](https://huggingface.co/google/gemma-3-4b-it).\n",
">\n",
"> Even you had an access token before accepting the Gemma license, you need to generate a new one after accepting it.\n",
"\n",
"Then you can use `huggingface_hub` for the authentication with the token generated in advance. So that then the token can be safely retrieved via `huggingface_hub.get_token`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8d836e0210fe"
},
"outputs": [],
"source": [
"from huggingface_hub import get_token, interpreter_login\n",
"\n",
"interpreter_login(new_session=False)\n",
"HF_TOKEN = get_token()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c71a4314c250"
},
"source": [
"Read more about [Hugging Face Security](https://huggingface.co/docs/hub/en/security), specifically about [Hugging Face User Access Tokens](https://huggingface.co/docs/hub/en/security-tokens)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gwSj4-j9Dbev"
},
"source": [
"## Prepare container image\n",
"\n",
"First, let's create a Docker file for a container with the model embedded into it.\n",
"\n",
"It will use your Hugging Face Hub secret at build time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8Pt7jloHDbev"
},
"outputs": [],
"source": [
"%%writefile Dockerfile\n",
"\n",
"FROM us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250312_0916_RC01\n",
"\n",
"ARG MODEL\n",
"ENV HF_MODEL=$MODEL\n",
"ENV MODEL_ID=$MODEL\n",
"\n",
"ENV HF_HOME=/model-cache\n",
"# Download the model to the cache\n",
"RUN --mount=type=secret,id=HF_TOKEN HF_TOKEN=$(cat /run/secrets/HF_TOKEN) \\\n",
" huggingface-cli download ${HF_MODEL}\n",
"# Switch to offline mode to make sure vLLM always uses the cached model\n",
"ENV HF_HUB_OFFLINE=1\n",
"\n",
"ENTRYPOINT python3 -m vllm.entrypoints.openai.api_server \\\n",
" --port ${PORT:-8080} \\\n",
" --model ${HF_MODEL} \\\n",
" --max-num-seqs=4 \\\n",
" --max-model-len 32768"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vZG5JfZRDbev"
},
"source": [
"Second, we create a Cloud Build file to use for building and pushing our container image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cRiA0VArDbev"
},
"outputs": [],
"source": [
"%%writefile cloudbuild.yaml\n",
"\n",
"steps:\n",
"- name: 'gcr.io/cloud-builders/docker'\n",
" id: build\n",
" entrypoint: 'bash'\n",
" secretEnv: ['HF_TOKEN']\n",
" args:\n",
" - -c\n",
" - |\n",
" SECRET_TOKEN=\"$$HF_TOKEN\" docker buildx build --tag=${_IMAGE} --build-arg MODEL=${_MODEL} --secret id=HF_TOKEN .\n",
"\n",
"availableSecrets:\n",
" secretManager:\n",
" - versionName: 'projects/${PROJECT_ID}/secrets/${_HF_TOKEN_SECRET_NAME}/versions/latest'\n",
" env: 'HF_TOKEN'\n",
"\n",
"images: [\"${_IMAGE}\"]\n",
"\n",
"substitutions:\n",
" _IMAGE: '${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_AR_REPO_NAME}/${_SERVICE_NAME}'\n",
"\n",
"options:\n",
" dynamicSubstitutions: true\n",
" machineType: \"E2_HIGHCPU_32\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9Vppnp8kDbev"
},
"source": [
"## Build Container Image and Deploy Cloud Run Service\n",
"\n",
"We are ready to build our container image and deploy Cloud Run service.\n",
"\n",
"The script below performs the following actions:\n",
"\n",
"* Enables necessary APIs.\n",
"* Creates an Artifact Repository for the image.\n",
"* Creates a Service Account for the service.\n",
"* Create a secret in Secret Manager that Cloud Build will use for building the container image.\n",
"* Assigns Service Account permissions to access the secret.\n",
"* Submits a Cloud Build job to create and push the container image.\n",
"* Deploys the Cloud Run service.\n",
"\n",
"> The script may take 15-45 minutes to finish.\n",
"\n",
"Note the following important flags in Cloud Build deployment command:\n",
"\n",
"* `--gpu 1` with `--gpu-type nvidia-l4` assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.\n",
"`--no-allow-authenticated` restricts unauthenticated access to the service.\n",
"By keeping the service private, you can rely on Cloud Run's built-in [Identity and Access Management (IAM)](https://cloud.google.com/iam) authentication for service-to-service communication.\n",
"* `--no-cpu-throttling` is required for enabling GPU.\n",
"* `--service-account` the service identity of the service.\n",
"* `--max-instances` sets maximum number of instances of the service.\n",
"It has to be equal to or lower than your project's NVIDIA L4 GPU (`Total Nvidia L4 GPU allocation, per project per region`) quota."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xsbvKy-_Dbev"
},
"outputs": [],
"source": [
"%%writefile deploy.sh\n",
"\n",
"PROJECT_ID=$1\n",
"REGION=$2\n",
"MODEL_ID=\"${3}\"\n",
"SERVICE_NAME=\"${5}\"\n",
"AR_REPO_NAME=\"vllm-repo\"\n",
"SERVICE_ACCOUNT=\"vllm-cloud-run-sa\"\n",
"SERVICE_ACCOUNT_ADDRESS=\"${SERVICE_ACCOUNT}@$PROJECT_ID.iam.gserviceaccount.com\"\n",
"HF_TOKEN_SECRET_NAME=\"HF_TOKEN_SECRET\"\n",
"HF_TOKEN=$4\n",
"MAX_INSTANCES=1 # Adjust this value to match your Cloud Run L4 GPU quota (\"Total Nvidia L4 GPU allocation, per project per region\", NvidiaL4GpuAllocPerProjectRegion, run.googleapis.com/nvidia_l4_gpu_allocation)\n",
"\n",
"echo \"Enabling APIs in project ${PROJECT_ID}.\"\n",
"gcloud services enable run.googleapis.com \\\n",
" cloudbuild.googleapis.com \\\n",
" secretmanager.googleapis.com \\\n",
" artifactregistry.googleapis.com \\\n",
" --project ${PROJECT_ID} \\\n",
" --quiet\n",
"\n",
"set -e\n",
"\n",
"# Creating the service account if doesn't exist.\n",
"sa_list=$(gcloud iam service-accounts list --quiet --format 'value(email)' --project $PROJECT_ID --filter=email:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com 2>/dev/null)\n",
"if [ -z \"${sa_list}\" ]; then\n",
" echo \"Creating Service Account ${SERVICE_ACCOUNT}.\"\n",
" gcloud iam service-accounts create $SERVICE_ACCOUNT \\\n",
" --project ${PROJECT_ID} \\\n",
" --display-name=\"${SERVICE_ACCOUNT} - Cloud Run Service Account\"\n",
"fi\n",
"\n",
"# Creating or updating the secret.\n",
"secrets_list=$(gcloud secrets list --project=\"${PROJECT_ID}\" --quiet --format 'value(name)' --filter=name:\"${HF_TOKEN_SECRET_NAME}\" 2>/dev/null)\n",
"if [ -z \"${secrets_list}\" ]; then\n",
" echo \"Creating Secret ${HF_TOKEN_SECRET_NAME}.\"\n",
" echo \"${HF_TOKEN}\" | gcloud secrets create \"${HF_TOKEN_SECRET_NAME}\" --project=${PROJECT_ID} --data-file=-\n",
"else\n",
" echo \"Updating Secret ${HF_TOKEN_SECRET_NAME}.\"\n",
" echo \"${HF_TOKEN}\" | gcloud secrets versions add \"${HF_TOKEN_SECRET_NAME}\" --project=${PROJECT_ID} --data-file=-\n",
"fi\n",
"\n",
"# Getting Cloud Build service account and giving it access to the secret\n",
"echo \"Applying Cloud Build account permissions.\"\n",
"cloud_build_sa=$(gcloud builds get-default-service-account --format 'value(serviceAccountEmail)' --project ${PROJECT_ID} --quiet)\n",
"cloud_build_sa_email=\"${cloud_build_sa##*/}\"\n",
"gcloud secrets add-iam-policy-binding \"${HF_TOKEN_SECRET_NAME}\" \\\n",
" --member serviceAccount:\"${cloud_build_sa_email}\" \\\n",
" --role='roles/secretmanager.secretAccessor' \\\n",
" --project=${PROJECT_ID}\n",
"\n",
"# Creating the Artifacts Repository if doesn't exist\n",
"repo_list=$(gcloud artifacts repositories list --format 'value(name)' --filter=name=\"projects/${PROJECT_ID}/locations/${REGION}/repositories/${AR_REPO_NAME}\" --project ${PROJECT_ID} --quiet --location ${REGION} 2>/dev/null)\n",
"if [ -z \"${repo_list}\" ]; then\n",
" echo \"Creating Artifact Registry ${AR_REPO_NAME}.\"\n",
" gcloud artifacts repositories create $AR_REPO_NAME \\\n",
" --repository-format docker \\\n",
" --location ${REGION} \\\n",
" --project=${PROJECT_ID}\n",
"fi\n",
"\n",
"echo \"Building container image.\"\n",
"gcloud builds submit --config=cloudbuild.yaml --project=${PROJECT_ID} . \\\n",
" --suppress-logs \\\n",
" --substitutions \\\n",
" _AR_REPO_NAME=$AR_REPO_NAME,_REGION=$REGION,_SERVICE_NAME=$SERVICE_NAME,_MODEL=$MODEL_ID,_HF_TOKEN_SECRET_NAME=$HF_TOKEN_SECRET_NAME\n",
"rm -f cloudbuild.yaml\n",
"rm -f Dockerfile\n",
"\n",
"echo \"Deploying Service ${SERVICE_NAME}.\"\n",
"gcloud beta run deploy $SERVICE_NAME \\\n",
" --project=${PROJECT_ID} \\\n",
" --image=${REGION}-docker.pkg.dev/$PROJECT_ID/$AR_REPO_NAME/$SERVICE_NAME \\\n",
" --service-account $SERVICE_ACCOUNT_ADDRESS \\\n",
" --cpu=8 \\\n",
" --memory=32Gi \\\n",
" --gpu=1 --gpu-type=nvidia-l4 \\\n",
" --region ${REGION} \\\n",
" --no-allow-unauthenticated \\\n",
" --max-instances ${MAX_INSTANCES} \\\n",
" --no-cpu-throttling \\\n",
" --timeout 1h\n",
"\n",
"SERVICE_URL=$(gcloud run services describe ${SERVICE_NAME} --project=${PROJECT_ID} --region $REGION --format 'value(status.url)' --quiet)\n",
"echo \"✅ Success!\"\n",
"echo \"🚀 Service URL: ${SERVICE_URL}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eff7e1585597"
},
"outputs": [],
"source": [
"!/bin/bash ./deploy.sh \"{PROJECT_ID}\" \"{REGION}\" \"{MODEL}\" \"{HF_TOKEN}\" \"{SERVICE_NAME}\" && rm -f ./deploy.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WzSFgfuyDbew"
},
"source": [
"## Test the deployed service\n",
"\n",
"Now, let's test the service you deployed.\n",
"\n",
"First, simply by using `cURL`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1LDyxRmADbew"
},
"outputs": [],
"source": [
"%%bash -s {MODEL} {SERVICE_NAME} {PROJECT_ID} {REGION}\n",
"\n",
"PROMPT=\"Why is the sky blue?\"\n",
"SERVICE_URL=$(gcloud run services describe ${2} --project ${3} --region ${4} --format 'value(status.url)' --quiet)\n",
"AUTH_TOKEN=$(gcloud auth print-identity-token -q)\n",
"\n",
"curl \"${SERVICE_URL}/v1/chat/completions\" \\\n",
" -s \\\n",
" -X POST \\\n",
" -H \"Content-Type: application/json\" \\\n",
" -H \"Authorization: Bearer ${AUTH_TOKEN}\" \\\n",
" -d @- <<EOF | python3 -m json.tool\n",
" {\n",
" \"model\": \"$1\",\n",
" \"messages\": [\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"${PROMPT}\"\n",
" }\n",
" ]\n",
" }\n",
"EOF"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bc1P8IcrDbew"
},
"source": [
"### OpenAI Python API library\n",
"\n",
"vLLM serving container provides [OpenAI-compatible endpoint](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) you've just used. Let's test it with OpenAI Python API library."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "L5bVcdITDbew"
},
"outputs": [],
"source": [
"# Install OpenAI library\n",
"%pip install openai -q"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1xNVHL-6Dbew"
},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"from IPython.display import Image, Markdown, display\n",
"from openai import OpenAI\n",
"\n",
"prompt = \"What's in this image?\"\n",
"image_url = \"https://upload.wikimedia.org/wikipedia/commons/thumb/b/b6/Felis_catus-cat_on_snow.jpg/320px-Felis_catus-cat_on_snow.jpg\"\n",
"\n",
"identity_token = (\n",
" subprocess.check_output(\"gcloud auth print-identity-token -q\", shell=True)\n",
" .decode()\n",
" .strip()\n",
")\n",
"service_url = (\n",
" subprocess.check_output(\n",
" (\n",
" \"gcloud run services describe \"\n",
" f\"{SERVICE_NAME} --project={PROJECT_ID} \"\n",
" f\"--region={REGION} \"\n",
" \"--format='value(status.url)' -q\"\n",
" ),\n",
" shell=True,\n",
" )\n",
" .decode()\n",
" .strip()\n",
")\n",
"\n",
"client = OpenAI(\n",
" api_key=\"EMPTY\",\n",
" base_url=f\"{service_url}/v1\",\n",
")\n",
"chat_response = client.chat.completions.create(\n",
" model=MODEL,\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": [\n",
" {\"type\": \"text\", \"text\": prompt},\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\n",
" \"url\": image_url,\n",
" },\n",
" },\n",
" ],\n",
" }\n",
" ],\n",
" temperature=0.5,\n",
" extra_headers=f\"Bearer {identity_token}\",\n",
")\n",
"print(f\"Prompt: {prompt}\")\n",
"display(Image(url=image_url))\n",
"Markdown(chat_response.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lLLwC3E3Dbew"
},
"source": [
"## Conclusion\n",
"Congratulations! Now you know how to deploy Gemma 3 with vLLM to Cloud Run powered by a GPU!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TKNPU6JADbew"
},
"source": [
"To delete the Cloud Run service you created, you can uncomment and run the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rxfg1S7pDbew"
},
"outputs": [],
"source": [
"# !gcloud run services delete $SERVICE_NAME --project $PROJECT_ID --region $LOCATION --quiet"
]
}
],
"metadata": {
"colab": {
"name": "cloud_run_vllm_gemma3_inference.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/get_started_with_vllm_lora_serving_on_vertex_ai.ipynb
+1265
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_ollama_gemma2_rag_agent.ipynb
+1645
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_pytorch_inference_paligemma_with_custom_handler.ipynb
+1384
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_pytorch_inference_pllum_with_custom_handler.ipynb
+1287
View File
File diff suppressed because it is too large Load Diff
open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb
+884
View File
@@ -0,0 +1,884 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Hugging Face DLCs: Serving Gemma with Text Generation Inference (TGI) on Vertex AI\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fserving%2Fvertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/serving/vertex_ai_text_generation_inference_gemma.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "83b98b0ba19c"
},
"source": [
"![Hugging Face x Google Cloud](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/google-cloud/thumbnail.png)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Alvaro Bartolome](https://github.com/alvarobartt), [Philipp Schmid](https://github.com/philschmid), [Simon Pagezy](https://github.com/pagezyhf), and [Jeff Boudier](https://github.com/jeffboudier) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ccd500ae19b5"
},
"source": [
"## Overview"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"> [**Gemma**](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open models built from the same research and technology used to create the Gemini models, developed by Google DeepMind and other teams across Google.\n",
"\n",
"> [**Text Generation Inference (TGI)**](https://github.com/huggingface/text-generation-inference) is a toolkit developed by Hugging Face for deploying and serving LLMs, with high performance text generation.\n",
"\n",
"> [**Hugging Face DLCs**](https://github.com/huggingface/Google-Cloud-Containers) are pre-built and optimized Deep Learning Containers (DLCs) maintained by Hugging Face and Google Cloud teams to simplify environment configuration for your ML workloads.\n",
"\n",
"> [**Google Vertex AI**](https://cloud.google.com/vertex-ai) is a Machine Learning (ML) platform that lets you train and deploy ML models and AI applications, and customize large language models (LLMs) for use in your AI-powered applications.\n",
"\n",
"This notebook showcases how to deploy Google Gemma from the Hugging Face Hub on Vertex AI using the Hugging Face Deep Learning Container (DLC) for Text Generation Inference (TGI).\n",
"\n",
"By the end of this notebook, you will learn how to:\n",
"\n",
"- Register any LLM from the Hugging Face Hub on Vertex AI\n",
"- Deploy an LLM on Vertex AI\n",
"- Send online predictions on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK and other required packages\n",
"\n",
"To run this example, you will only need the [`google-cloud-aiplatform`](https://github.com/googleapis/python-aiplatform) Python SDK and the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) Python package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --user --quiet google-cloud-aiplatform huggingface_hub"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages in this Jupyter environment, if you are on Colab you must restart the runtime. You can do this by running the cell below, which restarts the current kernel. The restart might take a minute or longer. After it's restarted, continue to the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SbmM4z7FOBpM"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9cc332aee5fd"
},
"source": [
"**1. Vertex AI Workbench**\n",
"\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "54c81e1a63e6"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c93d96bc9c43"
},
"outputs": [],
"source": [
"# !gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3e571ce6c56"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "984a0526fb68"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9af3e57f89a"
},
"source": [
"### Authenticate your Hugging Face account\n",
"\n",
"As [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it) is a gated model, you need to have a Hugging Face Hub account, and accept the Google's usage license for Gemma. Once that's done, you need to generate a new user access token with read-only access so that the weights can be downloaded from the Hub in the Hugging Face DLC for TGI.\n",
"\n",
"> Note that the user access token can only be generated via [the Hugging Face Hub UI](https://huggingface.co/settings/tokens/new), where you can either select read-only access to your account, or follow the recommendations and generate a fine-grained token with read-only access to [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4c31c7272804"
},
"source": [
"Then you can install the `huggingface_hub` that comes with a CLI that will be used for the authentication with the token generated in advance. So that then the token can be safely retrieved via `huggingface_hub.get_token`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8d836e0210fe"
},
"outputs": [],
"source": [
"from huggingface_hub import interpreter_login\n",
"\n",
"interpreter_login()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c71a4314c250"
},
"source": [
"Read more about [Hugging Face Security](https://huggingface.co/docs/hub/en/security), specifically about [Hugging Face User Access Tokens](https://huggingface.co/docs/hub/en/security-tokens)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com), if not enabled already.\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"if PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ee37e1544281"
},
"source": [
"## Requirements"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "877cd3fb2dce"
},
"source": [
"You will need to have the following IAM roles set:\n",
"\n",
"- Artifact Registry Reader (roles/artifactregistry.reader)\n",
"- Vertex AI User (roles/aiplatform.user)\n",
"\n",
"For more information about granting roles, see [Manage access](https://cloud.google.com/iam/docs/granting-changing-revoking-access).\n",
"\n",
"---\n",
"\n",
"You will also need to enable the following APIs (if not enabled already):\n",
"\n",
"- Vertex AI API (aiplatform.googleapis.com)\n",
"- Artifact Registry API (artifactregistry.googleapis.com)\n",
"\n",
"For more information about API enablement, see [Enabling APIs](https://cloud.google.com/apis/docs/getting-started#enabling_apis).\n",
"\n",
"---\n",
"\n",
"To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license on the Hugging Face Hub for any of the models from the [Gemma release collection](https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b), and the access request will be processed inmediately."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## Register Google Gemma on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "28d24c673821"
},
"source": [
"To serve Gemma with Text Generation Inference (TGI) on Vertex AI, you start importing the model on Vertex AI Model Registry.\n",
"\n",
"The Vertex AI Model Registry is a central repository where you can manage the lifecycle of your ML models. From the Model Registry, you have an overview of your models so you can better organize, track, and train new versions. When you have a model version you would like to deploy, you can assign it to an endpoint directly from the registry, or using aliases, deploy models to an endpoint. The models on the Vertex AI Model Registry just contain the model configuration and not the weights per se, as it's not a storage.\n",
"\n",
"Before going into the code to upload or import a model on Vertex AI, let's quickly review the arguments provided to the `aiplatform.Model.upload` method:\n",
"\n",
"* **`display_name`** is the name that will be shown in the Vertex AI Model Registry.\n",
"\n",
"* **`serving_container_image_uri`** is the location of the Hugging Face DLC for TGI that will be used for serving the model.\n",
"\n",
"* **`serving_container_environment_variables`** are the environment variables that will be used during the container runtime, so these are aligned with the environment variables defined by `text-generation-inference`, which are analog to the [`text-generation-launcher` arguments](https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/launcher). Additionally, the Hugging Face DLCs for TGI also capture the `AIP_` environment variables from Vertex AI as in [Vertex AI Documentation - Custom container requirements for prediction](https://cloud.google.com/vertex-ai/docs/predictions/custom-container-requirements).\n",
"\n",
" * `MODEL_ID` is the identifier of the model in the Hugging Face Hub. To explore all the supported models you can check [the Hugging Face Hub](https://huggingface.co/models?sort=trending&other=text-generation-inference).\n",
" * `NUM_SHARD` is the number of shards to use if you don't want to use all GPUs on a given machine e.g. if you have two GPUs but you just want to use one for TGI then `NUM_SHARD=1`, otherwise it matches the `CUDA_VISIBLE_DEVICES`.\n",
" * `MAX_INPUT_TOKENS` is the maximum allowed input length (expressed in number of tokens), the larger it is, the larger the prompt can be, but also more memory will be consumed.\n",
" * `MAX_TOTAL_TOKENS` is the most important value to set as it defines the \"memory budget\" of running clients requests, the larger this value, the larger amount each request will be in your RAM and the less effective batching can be.\n",
" * `MAX_BATCH_PREFILL_TOKENS` limits the number of tokens for the prefill operation, as it takes the most memory and is compute bound, it is interesting to limit the number of requests that can be sent.\n",
" * `HUGGING_FACE_HUB_TOKEN` is the Hugging Face Hub token, required as [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it) is a gated model.\n",
"\n",
"* (optional) **`serving_container_ports`** is the port where the Vertex AI endpoint |will be exposed, by default 8080.\n",
"\n",
"For more information on the supported `aiplatform.Model.upload` arguments, check [its Python reference](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Model#google_cloud_aiplatform_Model_upload)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ef487ce082f3"
},
"outputs": [],
"source": [
"from huggingface_hub import get_token\n",
"\n",
"model = aiplatform.Model.upload(\n",
" display_name=\"google--gemma-7b-it\",\n",
" serving_container_image_uri=\"us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu121.2-2.ubuntu2204.py310\",\n",
" serving_container_environment_variables={\n",
" \"MODEL_ID\": \"google/gemma-7b-it\",\n",
" \"NUM_SHARD\": \"1\",\n",
" \"MAX_INPUT_TOKENS\": \"512\",\n",
" \"MAX_TOTAL_TOKENS\": \"1024\",\n",
" \"MAX_BATCH_PREFILL_TOKENS\": \"1512\",\n",
" \"HUGGING_FACE_HUB_TOKEN\": get_token(),\n",
" },\n",
" serving_container_ports=[8080],\n",
")\n",
"model.wait()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c427c0a87016"
},
"source": [
"## Deploy Google Gemma on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ef4cd160dc09"
},
"source": [
"After the model is registered on Vertex AI, you can deploy the model to an endpoint.\n",
"\n",
"You need to first deploy a model to an endpoint before that model can be used to serve online predictions. Deploying a model associates physical resources with the model so it can serve online predictions with low latency.\n",
"\n",
"Before going into the code to deploy a model to an endpoint, let's quickly review the arguments provided to the `aiplatform.Model.deploy` method:\n",
"\n",
"- **`endpoint`** is the endpoint to deploy the model to, which is optional, and by default will be set to the model display name with the `_endpoint` suffix.\n",
"- **`machine_type`**, **`accelerator_type`** and **`accelerator_count`** are arguments that define which instance to use, and additionally, the accelerator to use and the number of accelerators, respectively. The `machine_type` and the `accelerator_type` are tied together, so you will need to select an instance that supports the accelerator that you are using and vice-versa. More information about the different instances at [Compute Engine Documentation - GPU machine types](https://cloud.google.com/compute/docs/gpus), and about the `accelerator_type` naming at [Vertex AI Documentation - MachineSpec](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/MachineSpec).\n",
"\n",
"For more information on the supported `aiplatform.Model.deploy` arguments, you can check [its Python reference](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.Model#google_cloud_aiplatform_Model_deploy)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e777427015db"
},
"outputs": [],
"source": [
"deployed_model = model.deploy(\n",
" endpoint=aiplatform.Endpoint.create(display_name=\"google--gemma-7b-it-endpoint\"),\n",
" machine_type=\"g2-standard-4\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=1,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5f2380d3ea60"
},
"source": [
"> Note that the model deployment on Vertex AI can take around 15 to 25 minutes; most of the time being the allocation / reservation of the resources, setting up the network and security, and such."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9b3fd1898241"
},
"source": [
"## Online predictions on Vertex AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2aa9cee03bd0"
},
"source": [
"Once the model is deployed on Vertex AI, you can run the online predictions using the `aiplatform.Endpoint.predict` method, which will send the requests to the running endpoint in the `/predict` route specified within the container following Vertex AI I/O payload formatting.\n",
"\n",
"As you are serving a `text-generation` model fine-tuned for instruction-following, you will need to make sure that the chat template, if any, is applied correctly to the input conversation; meaning that `transformers` need to be installed so as to instantiate the `tokenizer` for [`google/gemma-7b-it`](https://huggingface.co/google/gemma-7b-it) and run the `apply_chat_template` method over the input conversation before sending the input within the payload to the Vertex AI endpoint.\n",
"\n",
"> Note that the Messages API will be supported on Vertex AI on upcoming TGI releases, starting on 2.3, meaning that at the time of writing this post, the prompts need to be formatted before sending the request if you want to achieve nice results (assuming you are using an intruction-following model and not a base model)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d46c0a866ffc"
},
"outputs": [],
"source": [
"%pip install --upgrade --user --quiet transformers jinja2"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7e1ab8de0b5c"
},
"source": [
"After the installation is complete, the following snippet will apply the chat template to the conversation:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "82fd1842f0cc"
},
"outputs": [],
"source": [
"from huggingface_hub import get_token\n",
"from transformers import AutoTokenizer\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained(\"google/gemma-7b-it\", token=get_token())\n",
"\n",
"messages = [\n",
" {\"role\": \"user\", \"content\": \"What's Deep Learning?\"},\n",
"]\n",
"\n",
"inputs = tokenizer.apply_chat_template(\n",
" messages,\n",
" tokenize=False,\n",
" add_generation_prompt=True,\n",
")\n",
"# <bos><start_of_turn>user\\nWhat's Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d0522c9b9bc2"
},
"source": [
"Which is what you will be sending within the payload to the deployed Vertex AI Endpoint, as well as [the generation parameters](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient.text_generation)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "79e393e97ddb"
},
"source": [
"### Via Python"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1a404fd19a08"
},
"source": [
"#### Within the same session\n",
"\n",
"To run the online prediction via the Vertex AI SDK, you can simply use the `predict` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4ab63079a69a"
},
"outputs": [],
"source": [
"output = deployed_model.predict(\n",
" instances=[\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat's Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": True,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 1.0,\n",
" },\n",
" },\n",
" ]\n",
")\n",
"output"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b28efb473c7e"
},
"source": [
"#### From a different session\n",
"\n",
"To run the online prediction from a different session, you can run the following snippet."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e003ab03f30f"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google.cloud import aiplatform\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
"if PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION)\n",
"\n",
"endpoint_display_name = (\n",
" \"google--gemma-7b-it-endpoint\" # TODO: change to your endpoint display name\n",
")\n",
"\n",
"# Iterates over all the Vertex AI Endpoints within the current project and keeps the first match (if any), otherwise set to None\n",
"ENDPOINT_ID = next(\n",
" (\n",
" endpoint.name\n",
" for endpoint in aiplatform.Endpoint.list()\n",
" if endpoint.display_name == endpoint_display_name\n",
" ),\n",
" None,\n",
")\n",
"assert ENDPOINT_ID, (\n",
" \"`ENDPOINT_ID` is not set, please make sure that the `endpoint_display_name` is correct at \"\n",
" f\"https://console.cloud.google.com/vertex-ai/online-prediction/endpoints?project={os.getenv('PROJECT_ID')}\"\n",
")\n",
"\n",
"endpoint = aiplatform.Endpoint(\n",
" f\"projects/{PROJECT_ID}/locations/{LOCATION}/endpoints/{ENDPOINT_ID}\"\n",
")\n",
"output = endpoint.predict(\n",
" instances=[\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat's Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": True,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 0.7,\n",
" },\n",
" },\n",
" ],\n",
")\n",
"output"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7cc0b19ab641"
},
"source": [
"### Via gcloud"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8fed7007c31c"
},
"source": [
"You can also send the requests using the `gcloud` CLI via the `gcloud ai endpoints` command.\n",
"\n",
"> Note that, before proceeding, you should either replace the values or set the following environment variables in advance from the Python variables set in the example, as follows:\n",
">\n",
"> ```python\n",
"> import os\n",
"> os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
"> os.environ[\"LOCATION\"] = LOCATION\n",
"> os.environ[\"ENDPOINT_NAME\"] = \"google--gemma-7b-it-endpoint\"\n",
"> ```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7fb1547d409f"
},
"outputs": [],
"source": [
"%%bash\n",
"ENDPOINT_ID=$(gcloud ai endpoints list \\\n",
" --project=$PROJECT_ID \\\n",
" --region=$LOCATION \\\n",
" --filter=\"display_name=$ENDPOINT_NAME\" \\\n",
" --format=\"value(name)\" \\\n",
" | cut -d'/' -f6)\n",
"\n",
"echo '{\n",
" \"instances\": [\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat'\\''s Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": true,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 1.0\n",
" }\n",
" }\n",
" ]\n",
"}' | gcloud ai endpoints predict $ENDPOINT_ID \\\n",
" --project=$PROJECT_ID \\\n",
" --region=$LOCATION \\\n",
" --json-request=\"-\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7e588a65759e"
},
"source": [
"### Via cURL"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cf3de731bcf4"
},
"source": [
"Alternatively, you can also send the requests via `cURL`.\n",
"\n",
"> Note that, before proceeding, you should either replace the values or set the following environment variables in advance from the Python variables set in the example, as follows:\n",
">\n",
"> ```python\n",
"> import os\n",
"> os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
"> os.environ[\"LOCATION\"] = LOCATION\n",
"> os.environ[\"ENDPOINT_NAME\"] = \"google--gemma-7b-it-endpoint\"\n",
"> ```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a71da4e1152b"
},
"outputs": [],
"source": [
"%%bash\n",
"ENDPOINT_ID=$(gcloud ai endpoints list \\\n",
" --project=$PROJECT_ID \\\n",
" --region=$LOCATION \\\n",
" --filter=\"display_name=$ENDPOINT_NAME\" \\\n",
" --format=\"value(name)\" \\\n",
" | cut -d'/' -f6)\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://$LOCATION-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/$LOCATION/endpoints/$ENDPOINT_ID:predict \\\n",
" -d '{\n",
" \"instances\": [\n",
" {\n",
" \"inputs\": \"<bos><start_of_turn>user\\nWhat'\\''s Deep Learning?<end_of_turn>\\n<start_of_turn>model\\n\",\n",
" \"parameters\": {\n",
" \"max_new_tokens\": 20,\n",
" \"do_sample\": true,\n",
" \"top_p\": 0.95,\n",
" \"temperature\": 1.0\n",
" }\n",
" }\n",
" ]\n",
" }'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f6f17f9aff65"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"Finally, you can already release the resources that you've created as follows, to avoid unnecessary costs:\n",
"\n",
"- `deployed_model.undeploy_all` to undeploy the model from all the endpoints.\n",
"- `deployed_model.delete` to delete the endpoint/s where the model was deployed gracefully, after the `undeploy_all` method.\n",
"- `model.delete` to delete the model from the registry."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b074ffdbd0c4"
},
"outputs": [],
"source": [
"deployed_model.undeploy_all()\n",
"deployed_model.delete()\n",
"model.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6bc4da2fcc35"
},
"source": [
"Alternatively, you can also remove those from the Google Cloud Console following the steps:\n",
"\n",
"- Go to Vertex AI in Google Cloud\n",
"- Go to Deploy and use -> Online prediction\n",
"- Click on the endpoint and then on the deployed model/s to \"Undeploy model from endpoint\"\n",
"- Then go back to the endpoint list and remove the endpoint\n",
"- Finally, go to Deploy and use -> Model Registry, and remove the model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "137d82da56f2"
},
"source": [
"## References\n",
"\n",
"- [GitHub Repository - Hugging Face DLCs for Google Cloud](https://github.com/huggingface/Google-Cloud-Containers): contains all the containers developed by the collaboration of both Hugging Face and Google Cloud teams; as well as a lot of examples on both training and inference, covering both CPU and GPU, as well support for most of the models within the Hugging Face Hub.\n",
"- [Google Cloud Documentation - Hugging Face DLCs](https://cloud.google.com/deep-learning-containers/docs/choosing-container#hugging-face): contains a table with the latest released Hugging Face DLCs on Google Cloud.\n",
"- [Google Artifact Registry - Hugging Face DLCs](https://console.cloud.google.com/artifacts/docker/deeplearning-platform-release/us/gcr.io): contains all the DLCs released by Google Cloud that can be used.\n",
"- [Hugging Face Documentation - Google Cloud](https://huggingface.co/docs/google-cloud): contains the official Hugging Face documentation for the Google Cloud DLCs."
]
}
],
"metadata": {
"colab": {
"name": "vertex_ai_text_generation_inference_gemma.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
open-models/serving/vertex_ai_tgi_gemma_multi_lora_adapters_deployment.ipynb
+1537
View File
File diff suppressed because it is too large Load Diff