wehub-resource-sync
767 lines
27 KiB
Plaintext
767 lines
27 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ur8xi4C7S06n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2024 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "JAPoU8Sm5E6e"
|
|
},
|
|
"source": [
|
|
"# Intro to Context Caching with the Gemini API\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/context-caching/intro_context_caching.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fcontext-caching%2Fintro_context_caching.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/context-caching/intro_context_caching.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/context-caching/intro_context_caching.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/context-caching/intro_context_caching.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/context-caching/intro_context_caching.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/context-caching/intro_context_caching.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/context-caching/intro_context_caching.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/context-caching/intro_context_caching.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "84f0f73a0f76"
|
|
},
|
|
"source": [
|
|
"| Authors |\n",
|
|
"| --- |\n",
|
|
"| [Eric Dong](https://github.com/gericdong) |\n",
|
|
"| [Holt Skinner](https://github.com/holtskinner) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tvgnzT1CKxrO"
|
|
},
|
|
"source": [
|
|
"## Overview\n",
|
|
"\n",
|
|
"The Gemini API provides the context caching feature to store frequently used input tokens in a dedicated cache and use them for subsequent requests, eliminating the need to repeatedly pass the same set of tokens to a model. This feature can help reduce the number of tokens sent to the model, thereby lowering the cost of requests.\n",
|
|
"\n",
|
|
"The Gemini API offers two different caching mechanisms:\n",
|
|
"\n",
|
|
"- Implicit caching (automatic, no cost saving guarantee)\n",
|
|
"- Explicit caching (manual, cost saving guarantee)\n",
|
|
"\n",
|
|
"For more information, refer to the [context caching overview](https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview) page.\n",
|
|
"\n",
|
|
"### Objectives\n",
|
|
"\n",
|
|
"In this tutorial, you learn how to use implicit caching and explicit caching with the Google Gen AI SDK in Vertex AI.\n",
|
|
"\n",
|
|
"You will complete the following tasks:\n",
|
|
"\n",
|
|
"- Use implicit caching\n",
|
|
"- Use explicit caching\n",
|
|
" - Create a context cache\n",
|
|
" - Retrieve and use a context cache\n",
|
|
" - Use context caching in Chat\n",
|
|
" - Update the expire time of a context cache\n",
|
|
" - Delete a context cache"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "61RBz8LLbxCR"
|
|
},
|
|
"source": [
|
|
"## Get started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "No17Cw5hgx12"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK for Python"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tFy3H3aPgx12"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dmWOrTJ3gx13"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment (Colab only)\n",
|
|
"\n",
|
|
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NyKGtVQjgx13"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DF4l8DTdWgPY"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information and create client\n",
|
|
"\n",
|
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Nqwi-5ufWp_B"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
|
|
"\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "EdvJRUWRNGHE"
|
|
},
|
|
"source": [
|
|
"## Code Examples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "vHwJCyNF6u0O"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mginH0QC6u0O"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from IPython.display import Markdown, display\n",
|
|
"from google import genai\n",
|
|
"from google.genai.types import (\n",
|
|
" Content,\n",
|
|
" CreateCachedContentConfig,\n",
|
|
" GenerateContentConfig,\n",
|
|
" Part,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "a6f6d2cbf665"
|
|
},
|
|
"source": [
|
|
"### Create a client"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "3870ef96f984"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "27e7a9b7ac0d"
|
|
},
|
|
"source": [
|
|
"### Use a supported model\n",
|
|
"\n",
|
|
"See context caching [supported models](https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview#supported_models)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "bb92e043e0b3"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL_ID = \"gemini-3.5-flash\" # @param [\"gemini-3.5-flash\", \"gemini-2.5-flash\", \"gemini-2.5-pro\"] {\"allow-input\":true, isTemplate: true}\n",
|
|
"# fmt: on"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "548cbdd322d5"
|
|
},
|
|
"source": [
|
|
"## Implicit caching\n",
|
|
"\n",
|
|
"Implicit caching directly passes cache cost savings to developers without the need to create an explicit cache. Now, when you send a request to one of the Gemini 2.5 models, if the request shares a common prefix as one of previous requests, then it's eligible for a cache hit.\n",
|
|
"\n",
|
|
"**Note** that implicit caching is enabled by default for all Gemini 3 and 2.5 models but cost savings only apply to Gemini 2.5 models. The minimum input token count for implicit caching is 2,048 for 2.5 Flash and 2.5 Pro."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "2f2e8b5f2e1c"
|
|
},
|
|
"source": [
|
|
"### Re-enable caching\n",
|
|
"\n",
|
|
"By default, Google foundation models cache inputs for Gemini models. If you disabled caching and want to re-enable it, run the following curl command. To run this command, a user must be granted the Vertex AI administrator role `roles/aiplatform.admin`.\n",
|
|
"\n",
|
|
"For more information about enabling and disabling data caching, see [documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance#enabling-disabling-caching)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "f0d7ce598e4d"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"os.environ[\"API_ENDPOINT\"] = (\n",
|
|
" f\"{LOCATION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}\"\n",
|
|
")\n",
|
|
"os.environ[\"PROJECT_ID\"] = PROJECT_ID"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "25af7a380ff3"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%bash\n",
|
|
"\n",
|
|
"# Enable caching\n",
|
|
"curl -X PATCH \\\n",
|
|
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
|
|
" -H \"Content-Type: application/json\" \\\n",
|
|
" https://${API_ENDPOINT}/cacheConfig \\\n",
|
|
" -d '{\n",
|
|
" \"name\": \"projects/${PROJECT_ID}/cacheConfig\",\n",
|
|
" \"disableCache\": false\n",
|
|
" }'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "a09770b117e7"
|
|
},
|
|
"source": [
|
|
"### Send a request with a large and common content\n",
|
|
"\n",
|
|
"To increase the chance of an implicit cache hit:\n",
|
|
"\n",
|
|
"- Put large and common contents at the beginning of your prompt\n",
|
|
"- Send requests with similar prefix in a short amount of time\n",
|
|
"\n",
|
|
"In this example, you send a request with an image at the beginning of your prompt and add a user's request/question at the end of the prompt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "4939e2bc9bcf"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Part.from_uri(\n",
|
|
" file_uri=\"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/a-man-and-a-dog.png\",\n",
|
|
" mime_type=\"image/png\",\n",
|
|
" ),\n",
|
|
" \"Describe this image.\",\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "68d0aaeec4aa"
|
|
},
|
|
"source": [
|
|
"### Send requests with similar prefixes\n",
|
|
"\n",
|
|
"To demonstrate the implicit cache hit, repeat the same request multiple times, and print out the `cached_content_token_count` in the usage metadata which indicates how many tokens in the request were cached."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "7dd92db38ad9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"NUM_ATTEMPTS = 5 # @param {type: \"integer\"}\n",
|
|
"\n",
|
|
"for i in range(NUM_ATTEMPTS):\n",
|
|
" response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Part.from_uri(\n",
|
|
" file_uri=\"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/a-man-and-a-dog.png\",\n",
|
|
" mime_type=\"image/png\",\n",
|
|
" ),\n",
|
|
" \"Write a short and engaging blog post based on this image.\",\n",
|
|
" ],\n",
|
|
" )\n",
|
|
"\n",
|
|
" cached_token_count = response.usage_metadata.cached_content_token_count or 0\n",
|
|
"\n",
|
|
" print(f\"#{i + 1} Attempt\")\n",
|
|
" print(f\"Input tokens: {response.usage_metadata.prompt_token_count}\")\n",
|
|
" print(f\"Cached tokens: {cached_token_count}\")\n",
|
|
" print(f\"Output tokens: {response.usage_metadata.candidates_token_count}\")\n",
|
|
" print(f\"Total tokens: {response.usage_metadata.total_token_count}\")\n",
|
|
" print()\n",
|
|
"\n",
|
|
" if cached_token_count > 0:\n",
|
|
" print(response.usage_metadata.cache_tokens_details)\n",
|
|
" print(\"Cached content found, exiting loop.\")\n",
|
|
" break"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "baf87e6e3b64"
|
|
},
|
|
"source": [
|
|
"## Explicit caching\n",
|
|
"\n",
|
|
"Using the explicit caching feature, you can pass some content to the model once, cache the input tokens, and then refer to the cached tokens for subsequent requests. The minimum input token count for context caching is 2,048 for all models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "PAGPrpYagu2z"
|
|
},
|
|
"source": [
|
|
"### Create a context cache\n",
|
|
"\n",
|
|
"Create a `CachedContent` object specifying the prompt you want to use, including the file and other fields you wish to cache.\n",
|
|
"\n",
|
|
"**Notes**\n",
|
|
"- Caches are model specific. You cannot use a cache made with a different model as their tokenization might be slightly different.\n",
|
|
"- The default expiration time of a context cache is 60 minutes. You can specify a different expiration time using the `ttl` (time to live) or the `expire_time` property.\n",
|
|
"\n",
|
|
"This example shows how to create a context cache using two large research papers stored in a Cloud Storage bucket, and set the `ttl` to 600s.\n",
|
|
"\n",
|
|
"- Paper 1: [Gemini: A Family of Highly Capable Multimodal Models](https://arxiv.org/abs/2312.11805)\n",
|
|
"- Paper 2: [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://arxiv.org/abs/2403.05530)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "UmJA6AvVujyZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"system_instruction = \"\"\"\n",
|
|
"You are an expert researcher who has years of experience in conducting systematic literature surveys and meta-analyses of different topics.\n",
|
|
"You pride yourself on incredible accuracy and attention to detail. You always stick to the facts in the sources provided, and never make up new facts.\n",
|
|
"Now look at the research paper below, and answer the following questions in 1-2 sentences.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"cached_content = client.caches.create(\n",
|
|
" model=MODEL_ID,\n",
|
|
" config=CreateCachedContentConfig(\n",
|
|
" contents=[\n",
|
|
" Content(\n",
|
|
" role=\"user\",\n",
|
|
" parts=[\n",
|
|
" Part.from_uri(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/pdf/2312.11805v3.pdf\",\n",
|
|
" mime_type=\"application/pdf\",\n",
|
|
" ),\n",
|
|
" Part.from_uri(\n",
|
|
" file_uri=\"gs://cloud-samples-data/generative-ai/pdf/2403.05530.pdf\",\n",
|
|
" mime_type=\"application/pdf\",\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" system_instruction=system_instruction,\n",
|
|
" ttl=\"600s\",\n",
|
|
" ),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "7e1dKGSLDg2q"
|
|
},
|
|
"source": [
|
|
"You can access the properties of the cached content as example below. You can use its `name` or `resource_name` to reference the contents of the context cache.\n",
|
|
"\n",
|
|
"**Note**: The `name` of the context cache is also referred to as cache ID."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "uRJPRtkKDk2b"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(cached_content.name)\n",
|
|
"print(cached_content.model)\n",
|
|
"print(cached_content.create_time)\n",
|
|
"print(cached_content.expire_time)\n",
|
|
"print(cached_content.usage_metadata)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "d-f5gTEaCPkN"
|
|
},
|
|
"source": [
|
|
"### Retrieve a context cache\n",
|
|
"\n",
|
|
"You can use the property `name` to reference the contents of the context cache. For example:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "68e807d418e8"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"new_cached_content = client.caches.get(name=cached_content.name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "6f0c98e451f8"
|
|
},
|
|
"source": [
|
|
"### Use a context cache"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "RQ1zMmFQ1BNj"
|
|
},
|
|
"source": [
|
|
"To use the context cache, you provide the `cached_content` resource name in the `config` parameter of the `generate_content()` method.\n",
|
|
"\n",
|
|
"Then you can query the model with a prompt, and the cached content will be used as a prefix to the prompt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "EPVyJIW1BaVj"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.generate_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=\"What is the research goal shared by these research papers?\",\n",
|
|
" config=GenerateContentConfig(\n",
|
|
" cached_content=cached_content.name,\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "c525ede329b3"
|
|
},
|
|
"source": [
|
|
"You can check cached_content_token_count in the usage metadata which indicates how many tokens in the request were cached."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "13afac780407"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(f\"Input tokens: {response.usage_metadata.prompt_token_count}\")\n",
|
|
"print(f\"Cached tokens: {response.usage_metadata.cached_content_token_count or 0}\")\n",
|
|
"print(f\"Output tokens: {response.usage_metadata.candidates_token_count}\")\n",
|
|
"print(f\"Total tokens: {response.usage_metadata.total_token_count}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tX7vHiybEWeJ"
|
|
},
|
|
"source": [
|
|
"### Use context caching in chat\n",
|
|
"\n",
|
|
"You can use the context cache in a multi-turn chat session.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "FYNZS5o0FoGR"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"chat = client.chats.create(\n",
|
|
" model=MODEL_ID,\n",
|
|
" config=GenerateContentConfig(\n",
|
|
" cached_content=cached_content.name,\n",
|
|
" ),\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "5U-6wGSFFx51"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"\n",
|
|
"How do the approaches to responsible AI development and mitigation strategies in Gemini 1.5 evolve from those in Gemini 1.0?\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response = chat.send_message(prompt)\n",
|
|
"\n",
|
|
"display(Markdown(response.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "FFO_JgKNeCpK"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = \"\"\"\n",
|
|
"Given the advancements presented in Gemini 1.5, what are the key future research directions identified in both papers\n",
|
|
"for further improving multimodal AI models?\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response = chat.send_message(prompt)\n",
|
|
"\n",
|
|
"print(response.text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ORsGDdXXLwHK"
|
|
},
|
|
"source": [
|
|
"### Update the expiration time of a context cache\n",
|
|
"\n",
|
|
"The default expiration time of a context cache is 60 minutes. To update the expiration time, update one of the following properties:\n",
|
|
"\n",
|
|
"`ttl` - The number of seconds that the cache lives after it's created or after the `ttl` is updated before it expires.\n",
|
|
"\n",
|
|
"`expire_time` - A Timestamp that specifies the absolute date and time when the context cache expires."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "WyiZoZHKI2Jr"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cached_content = client.caches.update(\n",
|
|
" name=cached_content.name,\n",
|
|
" config=CreateCachedContentConfig(\n",
|
|
" system_instruction=system_instruction,\n",
|
|
" ttl=\"3600s\",\n",
|
|
" ),\n",
|
|
")\n",
|
|
"\n",
|
|
"print(cached_content.expire_time)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "chd6_8YRxdIu"
|
|
},
|
|
"source": [
|
|
"### Delete a context cache\n",
|
|
"\n",
|
|
"You can remove content from the cache using the delete operation."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "XGzgTk6YzgSt"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"client.caches.delete(name=cached_content.name)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "e0b4f3814f59"
|
|
},
|
|
"source": [
|
|
"## Next Steps\n",
|
|
"\n",
|
|
"- Learn more about context caching on the [context caching overview](https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview) page."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "intro_context_caching.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|