wehub-resource-sync
933 lines
36 KiB
Plaintext
933 lines
36 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ijGzTHJJUCPY"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2024 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "NDsTUvKjwHBW"
|
|
},
|
|
"source": [
|
|
"# Introduction to Gemini Multimodal Embeddings\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_multimodal_embeddings.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fembeddings%2Fintro_multimodal_embeddings.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/embeddings/intro_multimodal_embeddings.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_multimodal_embeddings.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://goo.gle/4fVmfkB\">\n",
|
|
" <img width=\"32px\" src=\"https://cdn.qwiklabs.com/assets/gcp_cloud-e3a77215f0b8bfa9b3f611c0d2208c7e8708ed31.svg\" alt=\"Google Cloud logo\"><br> Open in Cloud Skills Boost\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_multimodal_embeddings.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_multimodal_embeddings.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_multimodal_embeddings.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_multimodal_embeddings.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_multimodal_embeddings.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "4uoRmYQsKBgl"
|
|
},
|
|
"source": [
|
|
"| Authors |\n",
|
|
"| --- |\n",
|
|
"| [Lavi Nigam](https://github.com/lavinigam-gcp) |\n",
|
|
"| [Kaz Sato](https://github.com/kazunori279) |\n",
|
|
"| [Tengxin Li](https://github.com/tengxin-li) |\n",
|
|
"| [Eric Dong](www.linkedin.com/in/edong186) |\n",
|
|
"| [Holt Skinner](https://github.com/holtskinner) |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "RQT500QqVPIb"
|
|
},
|
|
"source": [
|
|
"### Objectives\n",
|
|
"\n",
|
|
"In this notebook, you will explore:\n",
|
|
"* Multimodal Embeddings with Gemini Embeddings Model (Texts, Images, Video, Audio, and PDFs)\n",
|
|
"* Setting embeddings dimensionality (truncation)\n",
|
|
"* Building various use cases with embeddings:\n",
|
|
" - Product search with text and images\n",
|
|
" - Semantic Similarity Analysis with text\n",
|
|
" - Multimodal Document Similarity Analysis with PDFs\n",
|
|
" - Using `task_type` for optimized Retrieval Augmented Generation (RAG)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "76476d2db2c0"
|
|
},
|
|
"source": [
|
|
"## Gemini Multimodal Embeddings\n",
|
|
"\n",
|
|
"**[Gemini Embedding 2](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/embedding-2)** is Google's multimodal embedding model for high-performance embedding generation, specifically designed to power complex retrieval and advanced analytics.\n",
|
|
"\n",
|
|
"The model processes a wide array of input modalities including **text, images, documents, audio, and video** to generate dense, 3072-dimensional vectors. These embeddings are positioned within a unified semantic space, ensuring that disparate data types with similar conceptual meanings are represented by mathematically proximal vectors.\n",
|
|
"\n",
|
|
"This notebook demonstrates how to use the [Multimodal Embeddings API](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/embeddings/get-multimodal-embeddings) for generating high-dimensional vector representations.\n",
|
|
"\n",
|
|
"**Use cases**\n",
|
|
"\n",
|
|
"* **Image classification & search:** Search relevant or similar images, or classify images based on labels.\n",
|
|
"* **Video content search:** Search relevant videos using text queries or similarity search.\n",
|
|
"* **Recommendations:** Generate product or advertisement recommendations based on multimodal data.\n",
|
|
"* **Semantic Similarity:** Compare the meaning of text fragments beyond keyword matching.\n",
|
|
"* **Document Analysis:** Identify documents with conceptually related content, regardless of formatting or layout.\n",
|
|
"* **Retrieval Augmented Generation (RAG):** Optimize retrieval for LLMs by using task-specific embeddings."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DXJpXzKrh2rJ"
|
|
},
|
|
"source": [
|
|
"## Getting Started"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "N5afkyDMSBW5"
|
|
},
|
|
"source": [
|
|
"### Install Google Gen AI SDK and other required packages\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "kc4WxYmLSBW5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet google-genai numpy pandas seaborn scikit-learn pymupdf"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "FtsU9Bw9h2rL"
|
|
},
|
|
"source": [
|
|
"### Authenticate your notebook environment\n",
|
|
"\n",
|
|
"If you are running this notebook in **Google Colab**, run the cell below to authenticate your account."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "GpYEyLsOh2rL"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "O1vKZZoEh2rL"
|
|
},
|
|
"source": [
|
|
"### Set Google Cloud project information\n",
|
|
"\n",
|
|
"To get started using Agent Platform, you must have an existing Google Cloud project and [enable the Agent Platform API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
|
"\n",
|
|
"Learn more about [setting up a project](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects) and a [development environment](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "gJqZ76rJh2rM"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"from google import genai\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"LOCATION = \"global\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"\n",
|
|
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
|
|
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
|
|
"\n",
|
|
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "BuQwwRiniVFG"
|
|
},
|
|
"source": [
|
|
"### Import libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "rtMowvm-yQ97"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import ast\n",
|
|
"\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np\n",
|
|
"import pandas as pd\n",
|
|
"import pymupdf\n",
|
|
"import seaborn as sns\n",
|
|
"from IPython.display import Image, Markdown, Video, display\n",
|
|
"from google.genai.types import Content, EmbedContentConfig, Part\n",
|
|
"from sklearn.metrics.pairwise import cosine_similarity\n",
|
|
"\n",
|
|
"pd.options.mode.chained_assignment = None"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "r-TX_R_xh2rM"
|
|
},
|
|
"source": [
|
|
"### Load Embedding Model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "SvMwSRJJh2rM"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# fmt: off\n",
|
|
"MODEL_ID = \"gemini-embedding-2\" # @param [\"gemini-embedding-2\"] {type: \"string\"}\n",
|
|
"# fmt: on"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "2176030e88a0"
|
|
},
|
|
"source": [
|
|
"## Generate Text Embeddings"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "aoiSW47MCZ0n"
|
|
},
|
|
"source": [
|
|
"Call the `embed_content` method with the model to generate text embeddings. You can embed a list of multiple prompts with one API call for efficiency."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "beaMkDH9zlr2"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" \"How do I get a driver's license?\",\n",
|
|
" \"What is the meaning of life?\",\n",
|
|
" \"How do I renew my driver's license?\",\n",
|
|
" \"How do I change my address on my driver's license?\",\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"for i, embedding in enumerate(response.embeddings):\n",
|
|
" print(f\"Embedding {i} length: {len(embedding.values)}\")\n",
|
|
" print(f\"First five values: {embedding.values[:5]}\\n\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "YXwxpeWqCZ0n"
|
|
},
|
|
"source": [
|
|
"### Set embeddings dimensionality (Truncation)\n",
|
|
"\n",
|
|
"The model is able to compress information into the earlier dimensions of the vector, allowing it to support lower embedding dimensions. Specify `output_dimensionality` to truncate the output.\n",
|
|
"\n",
|
|
"- **Default:** 3072 dimensions.\n",
|
|
"- **Reduced Dimensions:** You can specify a lower value (e.g., 768 or even 10) to significantly reduce storage costs and increase search speed with minimal loss in accuracy."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "iJzFSO07CZ0n"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"text = [\"Hello world\"]\n",
|
|
"\n",
|
|
"# Truncated to 768 dimensions\n",
|
|
"response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=text,\n",
|
|
" config=EmbedContentConfig(output_dimensionality=768),\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"Truncated embedding length: {len(response.embeddings[0].values)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "lNOoRm1rbC6X"
|
|
},
|
|
"source": [
|
|
"## Generate Multimodal Embeddings\n",
|
|
"\n",
|
|
"The model supports the following formats:\n",
|
|
"\n",
|
|
"- **Text:** Supports up to 8,192 tokens.\n",
|
|
"- **Image:** Maximum of 6 images per request. Supported formats: PNG, JPEG.\n",
|
|
"- **PDF:** Maximum of 6 pages.\n",
|
|
"- **Audio:** Maximum duration of 80 seconds. Supported formats: MP3, WAV.\n",
|
|
"- **Video:** Maximum duration of 128 seconds. Supported formats: MP4, MOV."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "7a2d7f3e3a33"
|
|
},
|
|
"source": [
|
|
"### Embed Images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "10kzx4YoVjMU"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -O cookies.png https://storage.googleapis.com/cloud-samples-data/generative-ai/image/cookies.png -q\n",
|
|
"\n",
|
|
"with open(\"cookies.png\", \"rb\") as f:\n",
|
|
" image_bytes = f.read()\n",
|
|
"\n",
|
|
"response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Part.from_bytes(data=image_bytes, mime_type=\"image/png\"),\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"Image embedding length: {len(response.embeddings[0].values)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Diw5oZjHb8jb"
|
|
},
|
|
"source": [
|
|
"#### Embedding Aggregation\n",
|
|
"\n",
|
|
"When working with multimodal content, how you structure your input affects the embedding output:\n",
|
|
"\n",
|
|
"* **Single content entry:** Submitting multiple parts (e.g., text and an image) within a single `Content` object produces one **aggregated** embedding for all modalities within that entry.\n",
|
|
"* **Multiple entries:** Sending multiple entries in the `contents` array returns separate embeddings for each entry.\n",
|
|
"* **Post-level representation:** For complex objects like social media posts with multiple media items, we recommend aggregating separate embeddings (for example, by averaging) to create a coherent post-level representation.\n",
|
|
"\n",
|
|
"Example of aggregated embedding (text + image):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "MeMdtUSncGDo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Content(\n",
|
|
" parts=[\n",
|
|
" Part(text=\"An image of cookies\"),\n",
|
|
" Part.from_bytes(data=image_bytes, mime_type=\"image/png\"),\n",
|
|
" ]\n",
|
|
" )\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"# This produces exactly ONE embedding\n",
|
|
"print(f\"Aggregated embedding length: {len(response.embeddings[0].values)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "cLZWZQGIcWp9"
|
|
},
|
|
"source": [
|
|
"### Embed Audio"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "fvwqRSg0cnxq"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"AUDIO_URL = \"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav\"\n",
|
|
"\n",
|
|
"response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Part.from_uri(file_uri=AUDIO_URL, mime_type=\"audio/wav\"),\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"Audio embedding length: {len(response.embeddings[0].values)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "fdcec1fefb20"
|
|
},
|
|
"source": [
|
|
"### Embed Video"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "QeGOaYxGwtdp"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"VIDEO_URL = (\n",
|
|
" \"https://storage.googleapis.com/cloud-samples-data/generative-ai/video/animals.mp4\"\n",
|
|
")\n",
|
|
"\n",
|
|
"response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Part.from_uri(file_uri=VIDEO_URL, mime_type=\"video/mp4\"),\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"Video embedding length: {len(response.embeddings[0].values)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "OAuk7MC5dbxE"
|
|
},
|
|
"source": [
|
|
"### Embed PDFs\n",
|
|
"\n",
|
|
"PDF documents can be embedded directly. The model processes the visual and text content of each page.\n",
|
|
"\n",
|
|
"**[IMPORTANT]** Page Limit Constraint: Currently, the model supports a maximum of **6 pages** per PDF. If your document is longer, you must truncate the file before processing."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "I-SMq-3xeLb_"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PDF_URL = \"https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/w9.pdf\"\n",
|
|
"!wget -q $PDF_URL -O sample.pdf\n",
|
|
"\n",
|
|
"def truncate_pdf_bytes(file_path: str, max_pages: int = 6) -> bytes:\n",
|
|
" with pymupdf.open(file_path) as doc:\n",
|
|
" if doc.page_count <= max_pages:\n",
|
|
" return doc.tobytes()\n",
|
|
" doc.select(range(max_pages))\n",
|
|
" return doc.tobytes(garbage=3, deflate=True)\n",
|
|
"\n",
|
|
"\n",
|
|
"pdf_bytes = truncate_pdf_bytes(\"sample.pdf\")\n",
|
|
"\n",
|
|
"response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[\n",
|
|
" Part.from_bytes(data=pdf_bytes, mime_type=\"application/pdf\"),\n",
|
|
" ],\n",
|
|
")\n",
|
|
"\n",
|
|
"print(f\"PDF embedding length: {len(response.embeddings[0].values)}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DXJpXzKrh2rK"
|
|
},
|
|
"source": [
|
|
"## Use Cases"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "73CFiID6WNo8"
|
|
},
|
|
"source": [
|
|
"### Find product images based on text search query"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "nigd4xO7Vow5"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def load_and_prepare_data(url: str) -> pd.DataFrame:\n",
|
|
" df = pd.read_csv(url)\n",
|
|
" df[\"image_embeddings\"] = df[\"image_embeddings\"].apply(\n",
|
|
" lambda x: np.array(ast.literal_eval(x))\n",
|
|
" )\n",
|
|
" df[\"public_url\"] = (\n",
|
|
" df[\"gcs_path\"]\n",
|
|
" .str.replace(\"gs://\", \"https://storage.googleapis.com/\", regex=False)\n",
|
|
" .str.replace(\" \", \"%20\", regex=False)\n",
|
|
" )\n",
|
|
" return df\n",
|
|
"\n",
|
|
"\n",
|
|
"# get product list with pre-computed image embeddings\n",
|
|
"product_image_list = load_and_prepare_data(\n",
|
|
" \"https://storage.googleapis.com/github-repo/embeddings/getting_started_embeddings/image_data_with_embeddings.csv\"\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"def get_text_embedding(text: str, dimension: int = 768):\n",
|
|
" response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=text,\n",
|
|
" config=EmbedContentConfig(output_dimensionality=dimension),\n",
|
|
" )\n",
|
|
" return np.array(response.embeddings[0].values)\n",
|
|
"\n",
|
|
"\n",
|
|
"def show_similar_images(query_emb: np.ndarray, df: pd.DataFrame, top_n: int = 5):\n",
|
|
" # Vectorized Dot Product (Matrix-Vector Multiplication)\n",
|
|
" # Stack all embeddings into one (N, Dimension) matrix\n",
|
|
" embeddings_matrix = np.stack(df[\"image_embeddings\"].values)\n",
|
|
"\n",
|
|
" # Calculate all scores at once using the @ operator (dot product)\n",
|
|
" df[\"score\"] = embeddings_matrix @ query_emb\n",
|
|
"\n",
|
|
" # Get Top N results efficiently\n",
|
|
" top_results = df.nlargest(top_n, \"score\")\n",
|
|
"\n",
|
|
" # Display results\n",
|
|
" print(top_results[[\"score\", \"title\"]])\n",
|
|
" for url in top_results[\"public_url\"]:\n",
|
|
" display(Image(url=url, width=200))\n",
|
|
"\n",
|
|
"\n",
|
|
"query_emb = get_text_embedding(\"something related to dinosaurs theme\")\n",
|
|
"show_similar_images(query_emb, product_image_list)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "533eeb913f05"
|
|
},
|
|
"source": [
|
|
"### Find videos based on text search query"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "19d37f510975"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"video_list = pd.read_csv(\n",
|
|
" \"https://storage.googleapis.com/github-repo/embeddings/getting_started_embeddings/video_data_with_embeddings.csv\"\n",
|
|
")\n",
|
|
"print(f\"Items in the video list: {len(video_list)}\")\n",
|
|
"video_list.head()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "9e92424d3b42"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def print_similar_videos(query_emb: list[float], data_frame: pd.DataFrame):\n",
|
|
" video_embs = data_frame[\"video_embeddings\"]\n",
|
|
" scores = [np.dot(eval(video_emb), query_emb) for video_emb in video_embs]\n",
|
|
" data_frame[\"score\"] = scores\n",
|
|
" data_frame = data_frame.sort_values(by=\"score\", ascending=False)\n",
|
|
"\n",
|
|
" # print results\n",
|
|
" print(data_frame.head()[[\"score\", \"file_name\"]])\n",
|
|
" url = data_frame.iloc[0][\"gcs_path\"]\n",
|
|
" display(Video(url.replace(\"gs://\", \"https://storage.googleapis.com/\")))\n",
|
|
"\n",
|
|
"\n",
|
|
"query_emb = get_text_embedding(\"A music concert\")\n",
|
|
"print_similar_videos(query_emb, video_list)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "71e4a82831d2"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"query_emb = get_text_embedding(\"A person doing TaiChi\")\n",
|
|
"print_similar_videos(query_emb, video_list)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rJ3aikvGCZ0o"
|
|
},
|
|
"source": [
|
|
"### Semantic Similarity Analysis\n",
|
|
"\n",
|
|
"Semantic similarity analysis transforms data into numerical vectors that encapsulate their underlying meaning. In a high-dimensional vector space, semantically related items are represented by vectors that are mathematically close to one another.\n",
|
|
"\n",
|
|
"This technique allows for comparison beyond keyword matching, capturing intent and context. For example:\n",
|
|
"* **Thematic Clustering:** Grouping content by topic regardless of format (e.g., a \"programming\" text and an \"explainer video\" on Python).\n",
|
|
"* **[Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity):** Measuring the angular distance between vectors to determine how \"similar\" two items are, where a score closer to 1.0 indicates higher semantic proximity."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "hEJvtYrNCZ0o"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"text_examples = [\n",
|
|
" \"I really enjoyed last night's movie\",\n",
|
|
" \"we watched a lot of acrobatic scenes yesterday\",\n",
|
|
" \"I had fun writing my first program in Python\",\n",
|
|
" \"huge sense of relief when my .py script finally ran without error\",\n",
|
|
" \"Oh Romeo, Romeo, wherefore art thou Romeo?\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"df = pd.DataFrame(text_examples, columns=[\"text\"])\n",
|
|
"df[\"embeddings\"] = df.apply(lambda x: get_text_embedding(x.text), axis=1)\n",
|
|
"\n",
|
|
"cos_sim_array = cosine_similarity(list(df.embeddings.values))\n",
|
|
"sim_df = pd.DataFrame(cos_sim_array, index=text_examples, columns=text_examples)\n",
|
|
"\n",
|
|
"plt.figure(figsize=(8, 6))\n",
|
|
"ax = sns.heatmap(sim_df, annot=True, cmap=\"crest\")\n",
|
|
"ax.xaxis.tick_top()\n",
|
|
"ax.set_xticklabels(text_examples, rotation=90)\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yA_Gs7VECZ0o"
|
|
},
|
|
"source": [
|
|
"### Multimodal Document Similarity Analysis\n",
|
|
"\n",
|
|
"By measuring the angular distance (Cosine Similarity) between these vectors, you can identify documents with conceptually related content, regardless of differences in formatting or layout."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "DFpdJtnlCZ0o"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -O google-q3-2025-report.pdf https://s206.q4cdn.com/479360582/files/doc_financials/2025/q3/2025q3-alphabet-earnings-release.pdf -q\n",
|
|
"!wget -O google-q4-2025-report.pdf https://s206.q4cdn.com/479360582/files/doc_financials/2025/q4/2025q4-alphabet-earnings-release.pdf -q\n",
|
|
"!wget -O meta-q4-2025-report.pdf https://s21.q4cdn.com/399680738/files/doc_financials/2025/q4/Earnings-Presentation-Q4-2025-FINAL.pdf -q\n",
|
|
"!wget -O gemini-2.5-report.pdf https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf -q\n",
|
|
"\n",
|
|
"docs = {\n",
|
|
" \"Google Q3 2025\": \"google-q3-2025-report.pdf\",\n",
|
|
" \"Google Q4 2025\": \"google-q4-2025-report.pdf\",\n",
|
|
" \"Meta Q4 2025\": \"meta-q4-2025-report.pdf\",\n",
|
|
" \"Gemini 2.5 Tech Report\": \"gemini-2.5-report.pdf\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"\n",
|
|
"def get_pdf_embedding(file_path):\n",
|
|
" pdf_bytes = truncate_pdf_bytes(file_path, max_pages=6)\n",
|
|
" response = client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=[Part.from_bytes(data=pdf_bytes, mime_type=\"application/pdf\")],\n",
|
|
" )\n",
|
|
" return response.embeddings[0].values\n",
|
|
"\n",
|
|
"\n",
|
|
"doc_embeddings = {title: get_pdf_embedding(path) for title, path in docs.items()}\n",
|
|
"\n",
|
|
"titles = list(doc_embeddings.keys())\n",
|
|
"embeddings = list(doc_embeddings.values())\n",
|
|
"sim_matrix = cosine_similarity(embeddings)\n",
|
|
"\n",
|
|
"sim_docs_df = pd.DataFrame(sim_matrix, index=titles, columns=titles)\n",
|
|
"\n",
|
|
"plt.figure(figsize=(8, 6))\n",
|
|
"sns.heatmap(sim_docs_df, annot=True, cmap=\"crest\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "GneVOB1yPMXg"
|
|
},
|
|
"source": [
|
|
"### Using `task_type` for different scenarios\n",
|
|
"\n",
|
|
"When generating embeddings, especially for complex systems like Retrieval Augmented Generation (RAG), a one-size-fits-all approach can sometimes lead to lower-quality results. For instance, a question like \"Why is the sky blue?\" and its answer, \"The scattering of sunlight causes the blue color,\" have distinct meanings as statements, and a general-purpose embedding model might not recognize their strong relationship.\n",
|
|
"\n",
|
|
"To solve this, you can specify a `task_type` when creating embeddings. This instructs the model to produce vectors that are specifically optimized for your intended use case, which can significantly improve performance while also saving time and cost.\n",
|
|
"\n",
|
|
"* **`SEMANTIC_SIMILARITY`**: Compares how similar two pieces of text are in meaning.\n",
|
|
"* **`RETRIEVAL_QUERY`** and **`RETRIEVAL_DOCUMENT`**: Used for search systems. Use `QUERY` for the user's question and `DOCUMENT` for the content you are searching through. This is the foundation for building effective semantic search and RAG systems.\n",
|
|
"* **`CLASSIFICATION`**: Used when you need to sort text into specific, pre-defined categories (like labeling emails as \"Spam\" or \"Not Spam\").\n",
|
|
"* **`CLUSTERING`**: Groups similar texts together to find hidden patterns or topics when you don't have pre-set labels."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "cVg9Ifh8jsBc"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"docs_df = pd.read_json(\n",
|
|
" \"https://storage.googleapis.com/github-repo/embeddings/google-car.json\"\n",
|
|
")\n",
|
|
"\n",
|
|
"# Generate retrieval embeddings for documents\n",
|
|
"docs_df[\"embeddings\"] = docs_df.apply(\n",
|
|
" lambda x: (\n",
|
|
" client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=x.contents,\n",
|
|
" config=EmbedContentConfig(task_type=\"RETRIEVAL_DOCUMENT\"),\n",
|
|
" )\n",
|
|
" .embeddings[0]\n",
|
|
" .values\n",
|
|
" ),\n",
|
|
" axis=1,\n",
|
|
")\n",
|
|
"\n",
|
|
"\n",
|
|
"def query_docs(query, df):\n",
|
|
" query_emb = (\n",
|
|
" client.models.embed_content(\n",
|
|
" model=MODEL_ID,\n",
|
|
" contents=query,\n",
|
|
" config=EmbedContentConfig(task_type=\"RETRIEVAL_QUERY\"),\n",
|
|
" )\n",
|
|
" .embeddings[0]\n",
|
|
" .values\n",
|
|
" )\n",
|
|
"\n",
|
|
" scores = np.dot(np.stack(df.embeddings), query_emb)\n",
|
|
" best_idx = np.argmax(scores)\n",
|
|
" return df.contents.iloc[best_idx]\n",
|
|
"\n",
|
|
"\n",
|
|
"question = \"how to make the fan speed stronger?\"\n",
|
|
"\n",
|
|
"context = query_docs(question, docs_df)\n",
|
|
"\n",
|
|
"print(f\"Best matching passage: {context}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "bRDVjDIdHfdt"
|
|
},
|
|
"source": [
|
|
"Now you can do one augmented generation (the last step of the RAG process) using the best passage found by the first step, but still having custom answers for users instead of simply pasting large documents chunks directly:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "OaaRuN89HiOX"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"final_answer_prompt = f\"\"\"\n",
|
|
" Your Role: You are a friendly AI assistant. Your purpose is to explain information to users who are not experts.\n",
|
|
"\n",
|
|
" Your Task: Use the provided \"Source Text\" below to answer the user's question.\n",
|
|
"\n",
|
|
" Guidelines for your Response:\n",
|
|
"\n",
|
|
" Be Clear and Simple: Explain any complicated ideas in easy-to-understand terms. Avoid jargon.\n",
|
|
" Be Friendly: Write in a warm, conversational, and approachable tone.\n",
|
|
" Be Thorough: Construct a complete and detailed answer in full sentences, using all the relevant information from the source text.\n",
|
|
" Stay on Topic: If the source text does not contain the answer, you must state that the information is not available in the provided material. Do not use outside knowledge.\n",
|
|
"\n",
|
|
" QUESTION: {question}\n",
|
|
" PASSAGE: {context}\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"# fmt: off\n",
|
|
"GEMINI_MODEL_ID = \"gemini-3.5-flash\" # @param [\"gemini-2.5-flash-lite\", \"gemini-2.5-flash\", \"gemini-3.5-flash\", \"gemini-3.1-flash-lite\", \"gemini-3.1-pro-preview\"] {\"allow-input\":true, isTemplate: true}\n",
|
|
"# fmt: on\n",
|
|
"\n",
|
|
"final_answer = client.models.generate_content(\n",
|
|
" model=GEMINI_MODEL_ID,\n",
|
|
" contents=final_answer_prompt,\n",
|
|
")\n",
|
|
"\n",
|
|
"display(Markdown(final_answer.text))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "IddiO0T-wdno"
|
|
},
|
|
"source": [
|
|
"## What's next?\n",
|
|
"\n",
|
|
"- Learn how to store the vectors (embeddings) into Vector Search: [Notebook](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-intro.ipynb)\n",
|
|
"- Learn how to tune the embeddings with your own data: [Notebook](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/intro_embeddings_tuning.ipynb)\n",
|
|
"- Check out the [Gemini Embedding 2 documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/embedding-2) for detailed reference."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "intro_multimodal_embeddings.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|