wehub-resource-sync
1167 lines
41 KiB
Plaintext
1167 lines
41 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2024 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JAPoU8Sm5E6e"
|
|
},
|
|
"source": [
|
|
"# Cloud Run GPU Inference: Gemma 2 RAG Q&A with Ollama and LangChain\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fuse-cases%2Fcloud_run_ollama_gemma2_rag_qa.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td> \n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\">\n",
|
|
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/use-cases/cloud_run_ollama_gemma2_rag_qa.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a> "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "84f0f73a0f76"
|
|
},
|
|
"source": [
|
|
"| | |\n",
|
|
"|-|-|\n",
|
|
"| Author(s) | [Elia Secchi](https://github.com/eliasecchig/) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"> **[Cloud Run](https://cloud.google.com/run)**:\n",
|
|
"It's a serverless platform by Google Cloud for running containerized applications. It automatically scales and manages infrastructure, supporting various programming languages. Cloud Run now offers GPU acceleration for AI/ML workloads.\n",
|
|
"\n",
|
|
"> **Note:** GPU support in Cloud Run is a guarded feature. Before running this notebook, make sure your Google Cloud project is enabled. You can do that by visiting this page [g.co/cloudrun/gpu](https://g.co/cloudrun/gpu).\n",
|
|
"\n",
|
|
"\n",
|
|
"> **[Ollama](ollama.com)**: is an open-source tool for easily running and deploying large language models locally. It offers simple management and usage of LLMs on personal computers or servers.\n",
|
|
"\n",
|
|
"This notebook showcase how to deploy [Google Gemma 2](https://blog.google/technology/developers/google-gemma-2/) in Cloud Run, with the objective to build a simple RAG Q&A application.\n",
|
|
"\n",
|
|
"By the end of this notebook, you will learn how to:\n",
|
|
"\n",
|
|
"1. Deploy Google Gemma 2 on Cloud Run using Ollama\n",
|
|
"2. Implement a Retrieval-Augmented Generation (RAG) application with Gemma 2 and Ollama\n",
|
|
"3. Build a custom container with Ollama to deploy any Large Language Model (LLM) of your choice\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"### Required roles\n",
|
|
"\n",
|
|
"To get the permissions that you need to complete the tutorial, ask your administrator to grant you the following IAM roles on your project:\n",
|
|
"\n",
|
|
"1. Artifact Registry Administrator (`roles/artifactregistry.admin`)\n",
|
|
"2. Cloud Build Editor (`roles/cloudbuild.builds.editor`)\n",
|
|
"3. Cloud Run Admin (`roles/run.developer`)\n",
|
|
"4. Service Account User (`roles/iam.serviceAccountUser`)\n",
|
|
"5. Service Usage Consumer (`roles/serviceusage.serviceUsageConsumer`)\n",
|
|
"6. Storage Admin (`roles/storage.admin`)\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"For more information about granting roles, see [Manage access](https://cloud.google.com/iam/docs/granting-changing-revoking-access)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "FYbo7iEPluZQ"
|
|
},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "No17Cw5hgx12"
|
|
},
|
|
"source": [
|
|
"### Install Vertex AI SDK and other required packages"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --user --quiet google-cloud-aiplatform langchain-community langchainhub langchain_google_vertexai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "R5Xep4W9lq-Z"
|
|
},
|
|
"source": [
|
|
"### Restart runtime\n",
|
|
"\n",
|
|
"To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
|
|
"\n",
|
|
"The restart might take a minute or longer. After it's restarted, continue to the next step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "XRvKdaPDTznN"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import IPython\n",
|
|
"\n",
|
|
"app = IPython.Application.instance()\n",
|
|
"app.kernel.do_shutdown(True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "SbmM4z7FOBpM"
|
|
},
|
|
"source": [
|
|
"<div class=\"alert alert-block alert-warning\">\n",
|
|
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
|
|
"</div>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dmWOrTJ3gx13"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)\n",
|
|
"\n",
|
|
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NyKGtVQjgx13"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!gcloud auth login --update-adc --quiet"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DF4l8DTdWgPY"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
|
|
"\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Nqwi-5ufWp_B"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Use the environment variable if the user doesn't provide Project ID.\n",
|
|
"import os\n",
|
|
"\n",
|
|
"import vertexai\n",
|
|
"\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\", isTemplate: true}\n",
|
|
"if PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
|
|
"\n",
|
|
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8pB4NiQAMzgt"
|
|
},
|
|
"source": [
|
|
"### Fetch your Google Cloud project number"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Y54slycDMjHK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PROJECT_NUMBER = get_ipython().getoutput('gcloud projects describe $PROJECT_ID --format=\"value(projectNumber)\"')[0]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "EdvJRUWRNGHE"
|
|
},
|
|
"source": [
|
|
"## Deploy Ollama with Cloud Run"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5J5rY6YhxTRl"
|
|
},
|
|
"source": [
|
|
"## Build your container\n",
|
|
"\n",
|
|
"For deploying Gemma 2 in Cloud Run, create a container that packages the Ollama server and the Gemma 2 model.\n",
|
|
"\n",
|
|
"To build the container, you can use [Cloud Build](https://cloud.google.com/build), a serverless CI/CD platform which allows developers to easily build software.\n",
|
|
"\n",
|
|
"> For optimal startup time and improved scalability, it's recommended to store model weights for Gemma 2 (9B) and similarly sized models directly in the container image.\n",
|
|
"However, consider the storage requirements of larger models as they might be impractical to store in the container image. Refer to [Best practices: AI inference on Cloud Run with GPUs](https://cloud.google.com/run/docs/configuring/services/gpu-best-practices#loading-storing-models-tradeoff) for an overview of the trade-offs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "IprOEAAN1sBQ"
|
|
},
|
|
"source": [
|
|
"### Create Artifact Registry repository\n",
|
|
"\n",
|
|
"To build a container you will need to first create a repository in Google Cloud Artifact Registry:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "p5hXDtoYsCEB"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"AR_REPOSITORY_NAME = \"cr-gpu-repo\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "z1ZBM9PDrbdM"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!gcloud artifacts repositories create $AR_REPOSITORY_NAME \\\n",
|
|
" --repository-format=docker \\\n",
|
|
" --location=$LOCATION \\\n",
|
|
" --project=$PROJECT_ID"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "IDMpuXEu2thu"
|
|
},
|
|
"source": [
|
|
"### Create a Dockerfile\n",
|
|
"\n",
|
|
"You will then need to create a Dockerfile which defines the build steps of the container.\n",
|
|
"\n",
|
|
"You can customize the model used by modifying the `MODEL_NAME` variable. \n",
|
|
"Explore the [Ollama library](https://ollama.com/library) for a comprehensive list of available models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "IcPKeFPNQZzI"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"MODEL_NAME = \"gemma2:9b\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Vi9T53CScWdn"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"dockerfile_content = f\"\"\"\n",
|
|
"FROM ollama/ollama\n",
|
|
"\n",
|
|
"# Set the host and port to listen on\n",
|
|
"ENV OLLAMA_HOST 0.0.0.0:8080\n",
|
|
"\n",
|
|
"# Set the directory to store model weight files\n",
|
|
"ENV OLLAMA_MODELS /models\n",
|
|
"\n",
|
|
"# Reduce the verbosity of the logs\n",
|
|
"ENV OLLAMA_DEBUG false\n",
|
|
"\n",
|
|
"# Do not unload model weights from the GPU\n",
|
|
"ENV OLLAMA_KEEP_ALIVE -1\n",
|
|
"\n",
|
|
"# Choose the model to load. Ollama defaults to 4-bit quantized weights\n",
|
|
"ENV MODEL {MODEL_NAME}\n",
|
|
"\n",
|
|
"# Start the ollama server and download the model weights\n",
|
|
"RUN ollama serve & sleep 5 && ollama pull $MODEL\n",
|
|
"\n",
|
|
"# At startup time we start the server and run a dummy request\n",
|
|
"# to request the model to be loaded in the GPU memory\n",
|
|
"ENTRYPOINT [\"/bin/sh\"]\n",
|
|
"CMD [\"-c\", \"ollama serve & (ollama run $MODEL 'Say one word' &) && wait\"]\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"# Write the Dockerfile\n",
|
|
"with open(\"Dockerfile\", \"w\") as f:\n",
|
|
" f.write(dockerfile_content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5RnaYx2p235W"
|
|
},
|
|
"source": [
|
|
"### Trigger Cloud Build\n",
|
|
"\n",
|
|
"You are now ready to trigger the container build process!\n",
|
|
"We will use the `gcloud builds submit` command, using a `e2-highcpu-32` machine to optimize build time. We use e2-highcpu-32 machines because multiple cores allow for parallel downloads, significantly speeding up the build process.\n",
|
|
"\n",
|
|
"Cloud Build pricing is based on build minutes consumed. See [the pricing page](https://cloud.google.com/build/pricing) for details\n",
|
|
"\n",
|
|
"The operation will take ~10 minutes for completion."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "k2aooaREsT-F"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"CONTAINER_URI = (\n",
|
|
" f\"{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{AR_REPOSITORY_NAME}/ollama-gemma-2\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "CU8n7kk5OeP8"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!gcloud builds submit --tag $CONTAINER_URI --project $PROJECT_ID --machine-type e2-highcpu-32"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "xd_Zfz9c3cZy"
|
|
},
|
|
"source": [
|
|
"You can now use the container you just built to deploy a new Cloud Run service!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "3YGGFLB-JElj"
|
|
},
|
|
"source": [
|
|
"### Deploy container in Cloud Run\n",
|
|
"\n",
|
|
"You are now ready for deployment! Cloud Run offers multiple deployment methods, including Console, gcloud CLI, Cloud Code, Terraform, YAML, and Client Libraries. Explore all the options in the [official documentation](https://cloud.google.com/run/docs/deploying#service).\n",
|
|
"\n",
|
|
"For quick prototyping, you can start with the gcloud CLI `gcloud run deploy` command. This convenient command-line tool provides a straightforward way to get your container running on Cloud Run. Learn more about its features and usage in the [gcloud CLI reference](https://cloud.google.com/sdk/gcloud/reference/run/deploy)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "8e6kybbhp3Na"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"SERVICE_NAME = \"ollama-gemma-2\" # @param {type:\"string\"}"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "kDkLl8AFKKD0"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!gcloud beta run deploy $SERVICE_NAME \\\n",
|
|
" --project $PROJECT_ID \\\n",
|
|
" --region $LOCATION \\\n",
|
|
" --image $CONTAINER_URI \\\n",
|
|
" --concurrency 4 \\\n",
|
|
" --cpu 8 \\\n",
|
|
" --gpu 1 \\\n",
|
|
" --gpu-type nvidia-l4 \\\n",
|
|
" --max-instances 7 \\\n",
|
|
" --memory 32Gi \\\n",
|
|
" --no-allow-unauthenticated \\\n",
|
|
" --no-cpu-throttling \\\n",
|
|
" --timeout=600"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "e1afbaee64a4"
|
|
},
|
|
"source": [
|
|
"*Expect a slower initial deployment as the container image is being pulled for the first time.*"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8IRTamcobASG"
|
|
},
|
|
"source": [
|
|
"### Setting concurrency for optimal performance\n",
|
|
"\n",
|
|
"In Cloud Run, [concurrency](https://cloud.google.com/run/docs/about-concurrency) defines the maximum number of requests that can be processed simultaneously by a given instance.\n",
|
|
"\n",
|
|
"For this sample we set a `concurrency` value equal to 4.\n",
|
|
"\n",
|
|
"As part of your use case you might need to experiment with different concurrency settings to find the best latency vs throughput tradeoff.\n",
|
|
"\n",
|
|
"Refer to the following documentation pages to know more about performance optimizations:\n",
|
|
"- [Setting concurrency for optimal performance in Cloud Run](https://cloud.google.com/run/docs/tutorials/gpu-gemma2-with-ollama#set-concurrency-for-performance)\n",
|
|
"- [GPU performance best practices](https://cloud.google.com/run/docs/configuring/services/gpu-best-practices)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XSrXJkabGdjw"
|
|
},
|
|
"source": [
|
|
"## Invoking Gemma 2 in Cloud Run\n",
|
|
"\n",
|
|
"We are now ready to send some requests to Gemma!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Vrx30A8jKwrY"
|
|
},
|
|
"source": [
|
|
"### Fetch identity token\n",
|
|
"\n",
|
|
"Once deployed to Cloud Run, to invoke Gemma 2, we will need to fetch an Identity token to perform authentication. See the relative documentation to discover more about [authentication in Cloud Run](https://cloud.google.com/run/docs/authenticating/overview).\n",
|
|
"\n",
|
|
"In the appendix of this sample, you'll find a helper function that supports the automatic refresh of the [Identity Token](https://cloud.google.com/docs/authentication/token-types#id), which expires every hour by default."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "qSa5aZCPuLlU"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"ID_TOKEN = get_ipython().getoutput('gcloud auth print-identity-token -q')[0]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "UmA-haVjOA6U"
|
|
},
|
|
"source": [
|
|
"### Setup the Service URL"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "LOVfy893tvcl"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"SERVICE_URL = f\"https://{SERVICE_NAME}-{PROJECT_NUMBER}.{LOCATION}.run.app\" # type: ignore"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XbOtGicVLNgD"
|
|
},
|
|
"source": [
|
|
"## Invoking Gemma\n",
|
|
"\n",
|
|
"You are ready to test the model you just deployed! The [Ollama API docs](https://github.com/ollama/ollama/blob/main/docs/api.md) are a great resource to learn more about the different endpoints and how to interact with your model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "iMI0nlXVT20t"
|
|
},
|
|
"source": [
|
|
"#### Invoke through CURL request\n",
|
|
"You can invoke Gemma and Cloud Run in many ways. For example, you can send an HTTP CURL request to Cloud Run:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "4b1c47642f7e"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"ENDPOINT_URL = f\"{SERVICE_URL}/api/generate\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NsixJcaBP2q4"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%bash -s \"$ENDPOINT_URL\" \"$ID_TOKEN\" \"$MODEL_NAME\" \n",
|
|
"ENDPOINT_URL=$1\n",
|
|
"ID_TOKEN=$2\n",
|
|
"MODEL_NAME=$3\n",
|
|
"\n",
|
|
"curl -s -X POST \"${ENDPOINT_URL}\" \\\n",
|
|
"-H \"Authorization: Bearer ${ID_TOKEN}\" \\\n",
|
|
"-H \"Content-Type: application/json\" \\\n",
|
|
"-d '{ \"model\": \"'${MODEL_NAME}'\", \"prompt\": \"Hi\", \"max_tokens\": 100, \"stream\": false}'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "41657205b738"
|
|
},
|
|
"source": [
|
|
"#### Invoke with a Python POST Request\n",
|
|
"\n",
|
|
"You can also invoke the model using a POST request with Python's popular `requests` library. [Learn more about the `requests` library here.](https://requests.readthedocs.io/en/latest/) "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "2e8c87dfd38b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import requests\n",
|
|
"\n",
|
|
"headers = {\"Authorization\": f\"Bearer {ID_TOKEN}\", \"Content-Type\": \"application/json\"} # type: ignore\n",
|
|
"\n",
|
|
"data = {\n",
|
|
" \"model\": MODEL_NAME,\n",
|
|
" \"prompt\": \"Hi, I am using python!\",\n",
|
|
" \"max_tokens\": 100,\n",
|
|
" \"stream\": False,\n",
|
|
"}\n",
|
|
"\n",
|
|
"response = requests.post(ENDPOINT_URL, headers=headers, json=data)\n",
|
|
"\n",
|
|
"print(response.text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "bFoe0NVOT6DD"
|
|
},
|
|
"source": [
|
|
"#### Invoke Ollama with Python integrations\n",
|
|
"\n",
|
|
"Popular Generative AI orchestration frameworks like [LangChain](https://www.langchain.com) and [LlamaIndex](https://www.llamaindex.ai/) offer direct integration with Ollama:\n",
|
|
"- [LangChain integration](https://python.langchain.com/v0.2/docs/integrations/llms/ollama/)\n",
|
|
"- [LlamaIndex integration](https://docs.llamaindex.ai/en/stable/api_reference/llms/ollama/)\n",
|
|
"\n",
|
|
"As part of this sample, we will be using the LangChain integration to perform different calls and build a sample RAG chain."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "vZyZqnnNaeWw"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "gQDWB66Vadlx"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import google.auth\n",
|
|
"from langchain.schema import BaseMessage, Document\n",
|
|
"from langchain_community.chat_models import ChatOllama\n",
|
|
"from langchain_community.document_loaders import WebBaseLoader\n",
|
|
"from langchain_community.vectorstores import SKLearnVectorStore\n",
|
|
"from langchain_core.output_parsers import StrOutputParser\n",
|
|
"from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
|
|
"from langchain_core.runnables import RunnablePassthrough\n",
|
|
"from langchain_google_vertexai import VertexAIEmbeddings\n",
|
|
"from langchain_text_splitters import CharacterTextSplitter"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "_hnaKrZftjbT"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = ChatOllama(\n",
|
|
" model=MODEL_NAME,\n",
|
|
" base_url=SERVICE_URL,\n",
|
|
" num_predict=300,\n",
|
|
" headers={\"Authorization\": f\"Bearer {ID_TOKEN}\"}, # type: ignore\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "9GYGr76T7aYF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# You can perform a synchronous invocation through the `.invoke` method\n",
|
|
"\n",
|
|
"llm.invoke(\"Hi!\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yVqxaDylWjck"
|
|
},
|
|
"source": [
|
|
"Or invoke through the generation of a stream through the `.stream` **method**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "2twtoa6a4_ui"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# You can also generate a stream through the `.stream` method\n",
|
|
"\n",
|
|
"for m in llm.stream(\"Hi!\"):\n",
|
|
" print(m)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "bqr_QPso7shY"
|
|
},
|
|
"source": [
|
|
"## RAG Q&A Chain with Gemma 2 and Cloud Run\n",
|
|
"\n",
|
|
"We can leverage the LangChain integration to create a sample RAG application with Gemma, Cloud Run, [Vertex AI Embedding](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings) for generating embeddings and [FAISS vector store](https://python.langchain.com/v0.2/docs/integrations/vectorstores/faiss/) for document retrieval.\n",
|
|
"\n",
|
|
"Through RAG, we will ask Gemma 2 to answer questions about the [Cloud Run documentation page](https://cloud.google.com/run/docs/overview/what-is-cloud-run)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "wHq8zpG5a4u9"
|
|
},
|
|
"source": [
|
|
"### Setup embedding model and retriever\n",
|
|
"\n",
|
|
"We are ready to setup our embedding model and retriever."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tI66kos7a8B4"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"credentials, _ = google.auth.default(quota_project_id=PROJECT_ID)\n",
|
|
"embeddings = VertexAIEmbeddings(\n",
|
|
" project=PROJECT_ID, model_name=\"text-embedding-005\", credentials=credentials\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "xxCu8MST6oWK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"loader = WebBaseLoader(\"https://cloud.google.com/run/docs/overview/what-is-cloud-run\")\n",
|
|
"docs = loader.load()\n",
|
|
"documents = CharacterTextSplitter(chunk_size=800, chunk_overlap=100).split_documents(\n",
|
|
" docs\n",
|
|
")\n",
|
|
"\n",
|
|
"vector = SKLearnVectorStore.from_documents(documents, embeddings)\n",
|
|
"retriever = vector.as_retriever()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rCH9n2UEzwMv"
|
|
},
|
|
"source": [
|
|
"### RAG Chain Definition\n",
|
|
"\n",
|
|
"We will define now our RAG Chain.\n",
|
|
"\n",
|
|
"The RAG chain works as follows:\n",
|
|
"\n",
|
|
"1. The user's query and conversation history are passed to the `query_rewrite_chain` to generate a rewritten query optimized for semantic search.\n",
|
|
"2. The rewritten query is used by the `retriever` to fetch relevant documents.\n",
|
|
"3. The retrieved documents are formatted into a single string.\n",
|
|
"4. The formatted documents, along with the original user messages, are passed to the LLM with instructions to generate an answer based on the provided context.\n",
|
|
"5. The LLM's response is parsed and returned as the final answer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "x1nPO00s4r4o"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"answer_generation_template = ChatPromptTemplate.from_messages(\n",
|
|
" [\n",
|
|
" (\n",
|
|
" \"system\",\n",
|
|
" \"You are an assistant for question answering-tasks. \"\n",
|
|
" \"Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. \"\n",
|
|
" \"{context}\",\n",
|
|
" ),\n",
|
|
" MessagesPlaceholder(variable_name=\"messages\"),\n",
|
|
" ]\n",
|
|
")\n",
|
|
"query_rewrite_template = ChatPromptTemplate.from_messages(\n",
|
|
" [\n",
|
|
" (\n",
|
|
" \"system\",\n",
|
|
" \"Rewrite a query to a semantic search engine using the current conversation. \"\n",
|
|
" \"Provide only the rewritten query as output.\",\n",
|
|
" ),\n",
|
|
" MessagesPlaceholder(variable_name=\"messages\"),\n",
|
|
" ]\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "jm309i0r0Lwv"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"query_rewrite_chain = query_rewrite_template | llm\n",
|
|
"\n",
|
|
"\n",
|
|
"def extract_query(messages: list[BaseMessage]) -> str:\n",
|
|
" return query_rewrite_chain.invoke(messages).content\n",
|
|
"\n",
|
|
"\n",
|
|
"def format_docs(docs: list[Document]) -> str:\n",
|
|
" return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
|
|
"\n",
|
|
"\n",
|
|
"rag_chain = (\n",
|
|
" {\n",
|
|
" \"context\": extract_query | retriever | format_docs,\n",
|
|
" \"messages\": RunnablePassthrough(),\n",
|
|
" }\n",
|
|
" | answer_generation_template\n",
|
|
" | llm\n",
|
|
" | StrOutputParser()\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rRFtc7eo0Tn7"
|
|
},
|
|
"source": [
|
|
"### Testing the RAG Chain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "MIfd_Wwa7RUe"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"rag_chain.invoke([(\"human\", \"What features does Cloud Run offer?\")])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "f0994074042d"
|
|
},
|
|
"source": [
|
|
"Now, let's use a specific question from the documentation to explore how RAG addresses potential gaps in the model's knowledge."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ec5895d100f5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"QUESTION = \"List all the different Cloud Run integrations\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "e0abfae19b32"
|
|
},
|
|
"source": [
|
|
"First, we'll ask the LLM directly:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "4fb7c3e2426e"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(llm.invoke(QUESTION).content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "5575573f8174"
|
|
},
|
|
"source": [
|
|
"Then, we'll ask the same question using the RAG chain:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AszY1ke5bWC9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(rag_chain.invoke([(\"human\", QUESTION)]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "c98450b91f72"
|
|
},
|
|
"source": [
|
|
"We can notice how RAG chain provides a more accurate and comprehensive answer than the LLM by leveraging the [source documentation](https://cloud.google.com/run/docs/overview/what-is-cloud-run). "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "88c5f72f1981"
|
|
},
|
|
"source": [
|
|
"## Conclusion\n",
|
|
"Congratulations. Now you know how to deploy an open model to Cloud Run powered by a GPU! Specifically, you deployed a Gemma 2 model to Cloud Run with a GPU, as part of a RAG application powered by LangChain. You were able to ask answers from Gemma 2 about a documentation page.\n",
|
|
"\n",
|
|
"For more information about your identity tokens expiring and how to refresh your tokens, see the next section below \"Appendix: Handling Identity Token Expiration\".\n",
|
|
"\n",
|
|
"To clean up the resources you created in this section, see the section at the bottom \"Cleaning up\"."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "A5VPH4gP3igf"
|
|
},
|
|
"source": [
|
|
"## Appendix: Handling Identity Token Expiration\n",
|
|
"\n",
|
|
"When deploying a Generative AI application Google Cloud Run, you'll often need to authenticate your requests using Identity Tokens.\n",
|
|
"\n",
|
|
"These tokens will expire hourly, requiring a mechanism for automatic refresh to ensure uninterrupted operation.\n",
|
|
"\n",
|
|
"The following helper classes provide an example of how to deal with token refresh. \n",
|
|
"It leverages:\n",
|
|
"1. The `google.auth` library to handle the authentication process and automatically refresh the token when necessary\n",
|
|
"2. ChatOllama's [auth parameter](https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.ollama.ChatOllama.html#langchain_community.chat_models.ollama.ChatOllama.auth) for passing an authentication callable\n",
|
|
"\n",
|
|
"\n",
|
|
"See the following resources for more information on authentication:\n",
|
|
"* [Identity Token Overview](https://cloud.google.com/docs/authentication/token-types#id)\n",
|
|
"* [Google Cloud Run Authentication Documentation](https://cloud.google.com/run/docs/authenticating/overview)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "e76fe84f0d63"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import time\n",
|
|
"\n",
|
|
"import google.auth\n",
|
|
"from google.auth.credentials import Credentials\n",
|
|
"from google.auth.exceptions import DefaultCredentialsError\n",
|
|
"import google.auth.transport.requests\n",
|
|
"import google.oauth2.id_token\n",
|
|
"from requests.auth import AuthBase\n",
|
|
"from requests.models import PreparedRequest\n",
|
|
"\n",
|
|
"\n",
|
|
"class GoogleCloudAuth(AuthBase):\n",
|
|
" def __init__(self, url: str, token_lifetime: int = 3600):\n",
|
|
" self.url: str = url\n",
|
|
" self.token: str | None = None\n",
|
|
" self.expiry_time: float = 0\n",
|
|
" self.token_lifetime: int = token_lifetime\n",
|
|
" self.creds: Credentials\n",
|
|
" self.creds, _ = google.auth.default()\n",
|
|
"\n",
|
|
" def __call__(self, r: PreparedRequest) -> PreparedRequest:\n",
|
|
" r.headers[\"Authorization\"] = f\"Bearer {self.get_token()}\"\n",
|
|
" return r\n",
|
|
"\n",
|
|
" def get_token(self) -> str | None:\n",
|
|
" if time.time() >= self.expiry_time:\n",
|
|
" self.refresh_token()\n",
|
|
" return self.token\n",
|
|
"\n",
|
|
" def refresh_token(self) -> None:\n",
|
|
" \"\"\"\n",
|
|
" Retrieves an ID token, attempting to use default credentials first,\n",
|
|
" and falling back to fetching a service-to-service new token if necessary.\n",
|
|
" See more on Cloud Run authentication at this link:\n",
|
|
" https://cloud.google.com/run/docs/authenticating/service-to-service\n",
|
|
" Args:\n",
|
|
" url: The URL to use for the token request.\n",
|
|
" \"\"\"\n",
|
|
" auth_req = google.auth.transport.requests.Request()\n",
|
|
" try:\n",
|
|
" self.token = google.oauth2.id_token.fetch_id_token(auth_req, self.url)\n",
|
|
" except DefaultCredentialsError:\n",
|
|
" self.creds.refresh(auth_req)\n",
|
|
" self.token = self.creds.id_token\n",
|
|
" self.expiry_time = time.time() + self.token_lifetime"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "d96d74e2bf40"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = ChatOllama(\n",
|
|
" auth=GoogleCloudAuth(url=SERVICE_URL),\n",
|
|
" model=MODEL_NAME,\n",
|
|
" base_url=SERVICE_URL,\n",
|
|
" num_predict=300,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "d0aff0566961"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm.invoke(\"Hi, testing a request\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "LjgBf0M4Mokn"
|
|
},
|
|
"source": [
|
|
"You can now use the `invoke` function as usual, with the token being refreshed automatically every hour."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "270LFAC0P3i4"
|
|
},
|
|
"source": [
|
|
"## Cleaning up\n",
|
|
"To clean up all Google Cloud resources, you can run the following cell to delete the Cloud Run service you created."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "XJRe7I7KP2HM"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Delete the Cloud Run service deployed above\n",
|
|
"\n",
|
|
"!gcloud run services delete $SERVICE_NAME --project $PROJECT_ID --region $LOCATION --quiet"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "cloud_run_ollama_gemma2_rag_qa.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|