wehub-resource-sync
2089 lines
74 KiB
Plaintext
2089 lines
74 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "YJy8qKC5Zwmx"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Copyright 2025 Google LLC\n",
|
|
"#\n",
|
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
|
"# you may not use this file except in compliance with the License.\n",
|
|
"# You may obtain a copy of the License at\n",
|
|
"#\n",
|
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
|
"#\n",
|
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
|
"# See the License for the specific language governing permissions and\n",
|
|
"# limitations under the License."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "A8dLtX2lZ6ah"
|
|
},
|
|
"source": [
|
|
"# Vector Search 2.0 Public Preview Quickstart\n",
|
|
"\n",
|
|
"<table align=\"left\">\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-quickstart.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fembeddings%2Fvector-search-2-quickstart.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://console.cloud.google.com/agent-platform/workbench/instances?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/embeddings/vector-search-2-quickstart.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://storage.googleapis.com/github-repo/workbench-icon.svg\" alt=\"Workbench logo\"><br> Open in Workbench\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
" <td style=\"text-align: center\">\n",
|
|
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-quickstart.ipynb\">\n",
|
|
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
|
" </a>\n",
|
|
" </td>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"<div style=\"clear: both;\"></div>\n",
|
|
"\n",
|
|
"<p>\n",
|
|
"<b>Share to:</b>\n",
|
|
"\n",
|
|
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
|
|
"</a>\n",
|
|
"\n",
|
|
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/embeddings/vector-search-2-quickstart.ipynb\" target=\"_blank\">\n",
|
|
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
|
|
"</a>\n",
|
|
"</p>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "3zQRr7StaHj3"
|
|
},
|
|
"source": [
|
|
"## Objectives\n",
|
|
"\n",
|
|
"In this notebook, you will learn how to get started with the Vector Search 2.0 public preview API.\n",
|
|
"\n",
|
|
"### **Warning: delete your objects after the tutorial**\n",
|
|
"\n",
|
|
"In case you are using your own Cloud project, please make sure to delete all the Collection and any associated Indexes after finishing this tutorial. Otherwise the remaining assets would incur unexpected costs.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8rNNxNcaaQRb"
|
|
},
|
|
"source": [
|
|
"## Prerequisites\n",
|
|
"\n",
|
|
"This tutorial requires a Google Cloud project that is linked with a billing account. To create a new project, take a look at [this document](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/start/cloud-environment) to create a project and setup a billing account for it.\n",
|
|
"To get the permissions that you need to give a service account access to enable APIs and interact with Agent Platform resources, ask your administrator to grant you the [Security Admin](https://cloud.google.com/iam/docs/roles-permissions/iam#iam.securityAdmin) (`roles/iam.securityAdmin`) IAM role on your project. For more information about granting roles, see [Manage access to projects, folders, and organizations](https://cloud.google.com/iam/docs/granting-changing-revoking-access).[link text](https://)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "ZBbY9yJvbuL1"
|
|
},
|
|
"source": [
|
|
"## Install the Vector Search SDK"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "69744e32"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install google-cloud-vectorsearch"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "bQ5q_KMkbXSi"
|
|
},
|
|
"source": [
|
|
"## Environment variables\n",
|
|
"\n",
|
|
"Set environment variables for your project and location."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "VrZJvAovvN7D"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"PROJECT_ID = \"your-project-id\" # @param {type:\"string\"}\n",
|
|
"LOCATION = \"us-central1\" # @param {type:\"string\"}"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nitj2X54bgQC"
|
|
},
|
|
"source": [
|
|
"## Authentication\n",
|
|
"\n",
|
|
"On Colab, run the following to authenticate calls to the Vector Search APIs:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "zQKHCit5vo-o"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sys\n",
|
|
"\n",
|
|
"# Additional authentication is required for Google Colab\n",
|
|
"if \"google.colab\" in sys.modules:\n",
|
|
" # Authenticate user to Google Cloud\n",
|
|
" from google.colab import auth\n",
|
|
"\n",
|
|
" auth.authenticate_user()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "WycjwYB2Djp1"
|
|
},
|
|
"source": [
|
|
"## Enable APIs\n",
|
|
"\n",
|
|
"Run the following commands to enable APIs for Vector Search and, if using Auto-Embeddings or Semantic Search, the Agent Platform API with this Google Cloud project.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "e3VZ0UNJDkg-"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"! gcloud services enable vectorsearch.googleapis.com aiplatform.googleapis.com --project \"{PROJECT_ID}\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "9gxfSP7sbIVU"
|
|
},
|
|
"source": [
|
|
"## Initialize Clients"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "zfiSJTIJvzI1"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from google.cloud import vectorsearch_v1beta\n",
|
|
"\n",
|
|
"vector_search_service_client = vectorsearch_v1beta.VectorSearchServiceClient()\n",
|
|
"data_object_service_client = vectorsearch_v1beta.DataObjectServiceClient()\n",
|
|
"data_object_search_service_client = vectorsearch_v1beta.DataObjectSearchServiceClient()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "AOQkdB5mYtbU"
|
|
},
|
|
"source": [
|
|
"## Create Collection"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "AOTKBWwjYvJj"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import getpass\n",
|
|
"from datetime import datetime\n",
|
|
"\n",
|
|
"collection_id = f\"movies-demo-{getpass.getuser()}-{datetime.now().strftime('%m-%d-%y')}\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Nv1wUdXvY4kZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.CreateCollectionRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}\",\n",
|
|
" collection_id=collection_id,\n",
|
|
" collection={\n",
|
|
" \"data_schema\": {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"year\": {\"type\": \"number\"},\n",
|
|
" \"genre\": {\"type\": \"string\"},\n",
|
|
" \"director\": {\"type\": \"string\"},\n",
|
|
" \"title\": {\"type\": \"string\"},\n",
|
|
" },\n",
|
|
" },\n",
|
|
" \"vector_schema\": {\n",
|
|
" \"plot_embedding\": {\"dense_vector\": {\"dimensions\": 3}},\n",
|
|
" \"soundtrack_embedding\": {\"dense_vector\": {\"dimensions\": 5}},\n",
|
|
" \"genre_embedding\": {\n",
|
|
" \"dense_vector\": {\n",
|
|
" \"dimensions\": 4,\n",
|
|
" \"vertex_embedding_config\": {\n",
|
|
" # If a data object is created without a supplied value for genre_embedding, it will be\n",
|
|
" # auto-generated based on this config.\n",
|
|
" \"model_id\": \"text-embedding-004\",\n",
|
|
" \"text_template\": (\"Movie: {title} Genre: {genre} Year: {year}\"),\n",
|
|
" \"task_type\": \"RETRIEVAL_DOCUMENT\",\n",
|
|
" },\n",
|
|
" }\n",
|
|
" },\n",
|
|
" \"sparse_embedding\": {\"sparse_vector\": {}},\n",
|
|
" },\n",
|
|
" },\n",
|
|
")\n",
|
|
"operation = vector_search_service_client.create_collection(request=request)\n",
|
|
"operation.result()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "mBji6kevajeO"
|
|
},
|
|
"source": [
|
|
"## Get Collection"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mHT5ef44akz3"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.GetCollectionRequest(\n",
|
|
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\"\n",
|
|
")\n",
|
|
"vector_search_service_client.get_collection(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "6t8c1DP-a_CX"
|
|
},
|
|
"source": [
|
|
"## List Collections"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "b-6jO3ifv4M-"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.ListCollectionsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}\"\n",
|
|
")\n",
|
|
"vector_search_service_client.list_collections(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "c3n0ofiYbSOR"
|
|
},
|
|
"source": [
|
|
"## Generate Sample Data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "BE4UEvwbbU_t"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import math\n",
|
|
"import random\n",
|
|
"\n",
|
|
"\n",
|
|
"def normalize(v):\n",
|
|
" norm = math.sqrt(sum(x * x for x in v))\n",
|
|
" return [x / norm for x in v]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Sparse embedding generation\n",
|
|
"\n",
|
|
"\n",
|
|
"VOCAB_SIZE = 50000 # Imagine a vocabulary of 50,000 possible items\n",
|
|
"NON_ZERO_COUNT = 7 # We want to represent an item that has 7 active features\n",
|
|
"VALUE_RANGE = (1, 10) # Let's say the values represent ratings from 1 to 10\n",
|
|
"\n",
|
|
"\n",
|
|
"def generate_sparse_embedding(\n",
|
|
" vocabulary_size: int, num_non_zero: int, value_range: tuple = (1, 100)\n",
|
|
") -> tuple[list[int], list[int]]:\n",
|
|
" \"\"\"Generates a random sparse representation with unique indices and corresponding values.\n",
|
|
"\n",
|
|
" Args:\n",
|
|
" vocabulary_size (int): The total size of the embedding space (e.g.,\n",
|
|
" 10000).\n",
|
|
" num_non_zero (int): The number of non-zero elements to generate.\n",
|
|
" value_range (tuple): A tuple (min, max) for the range of random integer\n",
|
|
" values.\n",
|
|
"\n",
|
|
" Returns:\n",
|
|
" tuple[list[int], list[int]]: A tuple containing two lists:\n",
|
|
" - A sorted list of unique indices.\n",
|
|
" - A list of corresponding random values.\n",
|
|
" \"\"\"\n",
|
|
" # Ensure we don't try to pick more unique items than available\n",
|
|
" if num_non_zero > vocabulary_size:\n",
|
|
" raise ValueError(\n",
|
|
" \"Number of non-zero elements cannot exceed the vocabulary size.\"\n",
|
|
" )\n",
|
|
"\n",
|
|
" # --- Generate Indices ---\n",
|
|
" # Pick a random sample of unique indices from the full range of the vocabulary.\n",
|
|
" # range(vocabulary_size) creates a sequence from 0 to vocabulary_size - 1.\n",
|
|
" indices = random.sample(range(vocabulary_size), num_non_zero)\n",
|
|
"\n",
|
|
" # It's a common and good practice for sparse formats to have sorted indices.\n",
|
|
" indices.sort()\n",
|
|
"\n",
|
|
" # --- Generate Values ---\n",
|
|
" # For each index, generate a corresponding random integer value.\n",
|
|
" min_val, max_val = value_range\n",
|
|
" values = [random.randint(min_val, max_val) for _ in range(num_non_zero)]\n",
|
|
"\n",
|
|
" return indices, values\n",
|
|
"\n",
|
|
"\n",
|
|
"movies = [\n",
|
|
" {\n",
|
|
" \"id\": \"the-shawshank-redemption\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Shawshank Redemption\",\n",
|
|
" \"genre\": \"Drama\",\n",
|
|
" \"year\": 1994,\n",
|
|
" \"director\": \"Frank Darabont\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-godfather\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Godfather\",\n",
|
|
" \"genre\": \"Crime\",\n",
|
|
" \"year\": 1972,\n",
|
|
" \"director\": \"Francis Ford Coppola\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-dark-knight\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Dark Knight\",\n",
|
|
" \"genre\": \"Action\",\n",
|
|
" \"year\": 2008,\n",
|
|
" \"director\": \"Christopher Nolan\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"pulp-fiction\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Pulp Fiction\",\n",
|
|
" \"genre\": \"Crime\",\n",
|
|
" \"year\": 1994,\n",
|
|
" \"director\": \"Quentin Tarantino\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"schindlers-list\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Schindler's List\",\n",
|
|
" \"genre\": \"Biography\",\n",
|
|
" \"year\": 1993,\n",
|
|
" \"director\": \"Steven Spielberg\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"twelve-angry-men\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"12 Angry Men\",\n",
|
|
" \"genre\": \"Drama\",\n",
|
|
" \"year\": 1957,\n",
|
|
" \"director\": \"Sidney Lumet\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-lord-of-the-rings-the-return-of-the-king\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Lord of the Rings: The Return of the King\",\n",
|
|
" \"genre\": \"Adventure\",\n",
|
|
" \"year\": 2003,\n",
|
|
" \"director\": \"Peter Jackson\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"spirited-away\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Spirited Away\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 2001,\n",
|
|
" \"director\": \"Hayao Miyazaki\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"parasite\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Parasite\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 2019,\n",
|
|
" \"director\": \"Bong Joon-ho\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-matrix\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Matrix\",\n",
|
|
" \"genre\": \"Sci-Fi\",\n",
|
|
" \"year\": 1999,\n",
|
|
" \"director\": \"The Wachowskis\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"inception\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Inception\",\n",
|
|
" \"genre\": \"Sci-Fi\",\n",
|
|
" \"year\": 2010,\n",
|
|
" \"director\": \"Christopher Nolan\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"interstellar\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Interstellar\",\n",
|
|
" \"genre\": \"Sci-Fi\",\n",
|
|
" \"year\": 2014,\n",
|
|
" \"director\": \"Christopher Nolan\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-silence-of-the-lambs\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Silence of the Lambs\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 1991,\n",
|
|
" \"director\": \"Jonathan Demme\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"psycho\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Psycho\",\n",
|
|
" \"genre\": \"Horror\",\n",
|
|
" \"year\": 1960,\n",
|
|
" \"director\": \"Alfred Hitchcock\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-green-mile\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Green Mile\",\n",
|
|
" \"genre\": \"Drama\",\n",
|
|
" \"year\": 1999,\n",
|
|
" \"director\": \"Frank Darabont\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"forrest-gump\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Forrest Gump\",\n",
|
|
" \"genre\": \"Drama\",\n",
|
|
" \"year\": 1994,\n",
|
|
" \"director\": \"Robert Zemeckis\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"fight-club\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Fight Club\",\n",
|
|
" \"genre\": \"Drama\",\n",
|
|
" \"year\": 1999,\n",
|
|
" \"director\": \"David Fincher\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-lion-king\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Lion King\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 1994,\n",
|
|
" \"director\": \"Roger Allers\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"beauty-and-the-beast\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Beauty and the Beast\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 1991,\n",
|
|
" \"director\": \"Gary Trousdale\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"toy-story\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Toy Story\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 1995,\n",
|
|
" \"director\": \"John Lasseter\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"goodfellas\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Goodfellas\",\n",
|
|
" \"genre\": \"Crime\",\n",
|
|
" \"year\": 1990,\n",
|
|
" \"director\": \"Martin Scorsese\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"seven\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Seven\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 1995,\n",
|
|
" \"director\": \"David Fincher\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"se7en\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Se7en\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 1995,\n",
|
|
" \"director\": \"David Fincher\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"city-of-god\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"City of God\",\n",
|
|
" \"genre\": \"Crime\",\n",
|
|
" \"year\": 2002,\n",
|
|
" \"director\": \"Fernando Meirelles\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-departed\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Departed\",\n",
|
|
" \"genre\": \"Crime\",\n",
|
|
" \"year\": 2006,\n",
|
|
" \"director\": \"Martin Scorsese\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"oldboy\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Oldboy\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 2003,\n",
|
|
" \"director\": \"Park Chan-wook\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"memento\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Memento\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 2000,\n",
|
|
" \"director\": \"Christopher Nolan\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"shutter-island\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Shutter Island\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 2010,\n",
|
|
" \"director\": \"Martin Scorsese\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-usual-suspects\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Usual Suspects\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 1995,\n",
|
|
" \"director\": \"Bryan Singer\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"gone-girl\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Gone Girl\",\n",
|
|
" \"genre\": \"Thriller\",\n",
|
|
" \"year\": 2014,\n",
|
|
" \"director\": \"David Fincher\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-sixth-sense\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Sixth Sense\",\n",
|
|
" \"genre\": \"Horror\",\n",
|
|
" \"year\": 1999,\n",
|
|
" \"director\": \"M. Night Shyamalan\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-others\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Others\",\n",
|
|
" \"genre\": \"Horror\",\n",
|
|
" \"year\": 2001,\n",
|
|
" \"director\": \"Alejandro Amenábar\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-ring\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Ring\",\n",
|
|
" \"genre\": \"Horror\",\n",
|
|
" \"year\": 2002,\n",
|
|
" \"director\": \"Gore Verbinski\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-exorcist\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Exorcist\",\n",
|
|
" \"genre\": \"Horror\",\n",
|
|
" \"year\": 1973,\n",
|
|
" \"director\": \"William Friedkin\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"singin-in-the-rain\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Singin' in the Rain\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1952,\n",
|
|
" \"director\": \"Stanley Donen\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-sound-of-music\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Sound of Music\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1965,\n",
|
|
" \"director\": \"Robert Wise\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"west-side-story\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"West Side Story\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1961,\n",
|
|
" \"director\": \"Robert Wise\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"seven-samurai\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Seven Samurai\",\n",
|
|
" \"genre\": \"Adventure\",\n",
|
|
" \"year\": 1954,\n",
|
|
" \"director\": \"Akira Kurosawa\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"my-neighbor-totoro\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"My Neighbor Totoro\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 1988,\n",
|
|
" \"director\": \"Hayao Miyazaki\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"howls-moving-castle\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Howl's Moving Castle\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 2004,\n",
|
|
" \"director\": \"Hayao Miyazaki\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"ponyo\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Ponyo\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 2008,\n",
|
|
" \"director\": \"Hayao Miyazaki\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-secret-world-of-arrietty\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Secret World of Arrietty\",\n",
|
|
" \"genre\": \"Animation\",\n",
|
|
" \"year\": 2010,\n",
|
|
" \"director\": \"Hiromasa Yonebayashi\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"oklahoma\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Oklahoma!\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1955,\n",
|
|
" \"director\": \"Fred Zinnemann\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"the-king-and-i\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The King and I\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1956,\n",
|
|
" \"director\": \"Walter Lang\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"my-fair-lady\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"My Fair Lady\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1964,\n",
|
|
" \"director\": \"George Cukor\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"cabaret\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Cabaret\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1972,\n",
|
|
" \"director\": \"Bob Fosse\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"grease\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Grease\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 1978,\n",
|
|
" \"director\": \"Randal Kleiser\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"chicago\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Chicago\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 2002,\n",
|
|
" \"director\": \"Rob Marshall\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"hairspray\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Hairspray\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 2007,\n",
|
|
" \"director\": \"Adam Shankman\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"id\": \"les-miserables\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"Les Misérables\",\n",
|
|
" \"genre\": \"Musical\",\n",
|
|
" \"year\": 2012,\n",
|
|
" \"director\": \"Tom Hooper\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
"]\n",
|
|
"\n",
|
|
"for movie in movies:\n",
|
|
" movie[\"vectors\"] = {}\n",
|
|
" movie[\"vectors\"][\"plot_embedding\"] = {\n",
|
|
" \"dense\": {\"values\": normalize([random.random() for _ in range(3)])}\n",
|
|
" }\n",
|
|
" movie[\"vectors\"][\"genre_embedding\"] = {\n",
|
|
" \"dense\": {\"values\": normalize([random.random() for _ in range(4)])}\n",
|
|
" }\n",
|
|
"\n",
|
|
" # Create a cluster for \"Musical\" movies in soundtrack_embedding\n",
|
|
" if movie[\"data\"][\"genre\"] == \"Musical\":\n",
|
|
" movie[\"vectors\"][\"soundtrack_embedding\"] = {\n",
|
|
" \"dense\": {\n",
|
|
" \"values\": normalize([0.9 + random.random() * 0.1 for _ in range(5)])\n",
|
|
" }\n",
|
|
" }\n",
|
|
" else:\n",
|
|
" movie[\"vectors\"][\"soundtrack_embedding\"] = {\n",
|
|
" \"dense\": {\"values\": normalize([random.random() for _ in range(5)])}\n",
|
|
" }\n",
|
|
" indices, values = generate_sparse_embedding(\n",
|
|
" vocabulary_size=VOCAB_SIZE,\n",
|
|
" num_non_zero=NON_ZERO_COUNT,\n",
|
|
" value_range=VALUE_RANGE,\n",
|
|
" )\n",
|
|
" movie[\"vectors\"][\"sparse_embedding\"] = {\n",
|
|
" \"sparse\": {\"values\": values, \"indices\": indices}\n",
|
|
" }"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "z8JMlDDgbkOB"
|
|
},
|
|
"source": [
|
|
"## Populate Data Objects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "VKBjHXWzb3bA"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"movies[0]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "O_u_MqM1bnJO"
|
|
},
|
|
"source": [
|
|
"### Create Data Object"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Lyl9n6tSblgk"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.CreateDataObjectRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" data_object_id=movies[0][\"id\"],\n",
|
|
" data_object={\n",
|
|
" \"data\": movies[0][\"data\"],\n",
|
|
" \"vectors\": movies[0][\"vectors\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"data_object_service_client.create_data_object(request=request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "DZNvLmpkc_aZ"
|
|
},
|
|
"source": [
|
|
"### Batch Create Data Objects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Jrh2I_u2dEn9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"rest_of_movies_batch_request = []\n",
|
|
"for movie in movies[1:-1]:\n",
|
|
" rest_of_movies_batch_request.append(\n",
|
|
" {\n",
|
|
" \"data_object_id\": movie[\"id\"],\n",
|
|
" \"data_object\": {\"data\": movie[\"data\"], \"vectors\": movie[\"vectors\"]},\n",
|
|
" }\n",
|
|
" )\n",
|
|
"request = vectorsearch_v1beta.BatchCreateDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" requests=rest_of_movies_batch_request,\n",
|
|
")\n",
|
|
"data_object_service_client.batch_create_data_objects(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "i8yPvGcUTmys"
|
|
},
|
|
"source": [
|
|
"### Import Data Objects from GCS file(s)\n",
|
|
"\n",
|
|
"The import will fail if the collection already has an ANN index."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8XlDRUHBTyiv"
|
|
},
|
|
"source": [
|
|
"#### Prepare GCS data for import\n",
|
|
"\n",
|
|
"The example below writes one data object to a JSON file in the specified GCS bucket."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "fyMHek32lq4w"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import json\n",
|
|
"import os\n",
|
|
"\n",
|
|
"IMPORT_BUCKET = f\"my-movie-demo-import-{PROJECT_ID}\"\n",
|
|
"\n",
|
|
"# Directory must only contain import data.\n",
|
|
"IMPORT_DIRECTORY = \"import-data/\"\n",
|
|
"IMPORT_FILE = \"movies.json\"\n",
|
|
"\n",
|
|
"# Error directory must be empty.\n",
|
|
"IMPORT_ERROR_DIRECTORY = \"import-errors/\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "H40kxvcOFU_J"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import copy\n",
|
|
"\n",
|
|
"movie_to_import = copy.deepcopy(movies[-1])\n",
|
|
"movie_to_import[\"vectors\"] = {\n",
|
|
" key: vector_info[\"dense\"][\"values\"]\n",
|
|
" if isinstance(vector_info, dict) and \"dense\" in vector_info\n",
|
|
" else vector_info\n",
|
|
" for key, vector_info in movie_to_import[\"vectors\"].items()\n",
|
|
"}\n",
|
|
"movie_string = json.dumps(movie_to_import)\n",
|
|
"movie_string"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "BCyLpyeZToET"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"blob_name = os.path.join(IMPORT_DIRECTORY, IMPORT_FILE)\n",
|
|
"gcs_uri = f\"gs://{IMPORT_BUCKET}/{blob_name}\"\n",
|
|
"try:\n",
|
|
" from google.cloud import storage\n",
|
|
"\n",
|
|
" storage_client = storage.Client(project=PROJECT_ID)\n",
|
|
" bucket = storage_client.bucket(IMPORT_BUCKET)\n",
|
|
" # To programmatically create the GCS bucket, uncomment the following line:\n",
|
|
" # storage_client.create_bucket(bucket)\n",
|
|
" blob = bucket.blob(blob_name)\n",
|
|
" blob.upload_from_string(movie_string)\n",
|
|
" print(f\"Successfully uploaded movie data to {gcs_uri}\")\n",
|
|
"except Exception as e:\n",
|
|
" print(f\"Could not write to GCS using google-cloud-storage. Error: {e}\")\n",
|
|
" print(f\"Please manually copy the following JSON string to {gcs_uri}\")\n",
|
|
" print(movie_string)\n",
|
|
" print(\"Sample command:\")\n",
|
|
" print(f\"echo '{movie_string}' | gcloud storage cp - {gcs_uri}\")\n",
|
|
" print(\"(the bucket must already exist)\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "jzmySijQT1wM"
|
|
},
|
|
"source": [
|
|
"#### Perform the import"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "8tjGeAs_T7oF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"contents_uri = f\"gs://{IMPORT_BUCKET}/{IMPORT_DIRECTORY}\"\n",
|
|
"error_uri = f\"gs://{IMPORT_BUCKET}/{IMPORT_ERROR_DIRECTORY}\"\n",
|
|
"print(f\"Importing from {contents_uri}\")\n",
|
|
"print(f\"Errors will be written to {error_uri}\")\n",
|
|
"request = vectorsearch_v1beta.ImportDataObjectsRequest(\n",
|
|
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" gcs_import={\n",
|
|
" \"contents_uri\": contents_uri,\n",
|
|
" \"error_uri\": error_uri,\n",
|
|
" },\n",
|
|
")\n",
|
|
"import_lro = vector_search_service_client.import_data_objects(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "1caMIuc6mcSA"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(f\"Waiting for import LRO: {import_lro.operation.name}\")\n",
|
|
"import_lro.result()\n",
|
|
"print(\"Import LRO complete.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "BILAS9zXHA6u"
|
|
},
|
|
"source": [
|
|
"## Get Data Object"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "tSiB9WP8HCQi"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.GetDataObjectRequest(\n",
|
|
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}/dataObjects/{movies[0]['id']}\",\n",
|
|
")\n",
|
|
"data_object_service_client.get_data_object(request=request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "zfFqCJEOINub"
|
|
},
|
|
"source": [
|
|
"## Update Data Object"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "l5-hDOVVINDs"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.UpdateDataObjectRequest(\n",
|
|
" data_object={\n",
|
|
" \"name\": f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}/dataObjects/{movies[0]['id']}\",\n",
|
|
" \"data\": {\"title\": f\"{movies[0]['data']['title']} (updated)\"},\n",
|
|
" \"vectors\": {\"plot_embedding\": {\"dense\": {\"values\": [1.0, 1.0, 1.0]}}},\n",
|
|
" }\n",
|
|
")\n",
|
|
"data_object_service_client.update_data_object(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "MNw5doXKIXsf"
|
|
},
|
|
"source": [
|
|
"## Batch Update Data Object"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Rvg5EFsdIZtT"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"movies[9][\"data\"][\"title\"] = movies[9][\"data\"][\"title\"] + \" updated\"\n",
|
|
"movies[10][\"data\"][\"title\"] = movies[10][\"data\"][\"title\"] + \" updated\"\n",
|
|
"movies[11][\"data\"][\"title\"] = movies[11][\"data\"][\"title\"] + \" updated\"\n",
|
|
"collection_name = (\n",
|
|
" f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\"\n",
|
|
")\n",
|
|
"requests = [\n",
|
|
" {\n",
|
|
" \"data_object\": {\n",
|
|
" \"name\": f\"{collection_name}/dataObjects/{movies[9]['id']}\",\n",
|
|
" \"data\": movies[9][\"data\"],\n",
|
|
" \"vectors\": movies[9][\"vectors\"],\n",
|
|
" }\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"data_object\": {\n",
|
|
" \"name\": f\"{collection_name}/dataObjects/{movies[10]['id']}\",\n",
|
|
" \"data\": movies[10][\"data\"],\n",
|
|
" \"vectors\": movies[10][\"vectors\"],\n",
|
|
" }\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"data_object\": {\n",
|
|
" \"name\": f\"{collection_name}/dataObjects/{movies[11]['id']}\",\n",
|
|
" \"data\": movies[11][\"data\"],\n",
|
|
" \"vectors\": movies[11][\"vectors\"],\n",
|
|
" }\n",
|
|
" },\n",
|
|
"]\n",
|
|
"request = vectorsearch_v1beta.BatchUpdateDataObjectsRequest(\n",
|
|
" parent=collection_name,\n",
|
|
" requests=requests,\n",
|
|
")\n",
|
|
"data_object_service_client.batch_update_data_objects(request)\n",
|
|
"# scifi = data_object_search_service_client.query_data_objects(\n",
|
|
"# request={\"parent\": collection_name, \"filter\": {\"genre\": {\"$eq\": \"Sci-Fi\"}}}\n",
|
|
"# )\n",
|
|
"# print([m.data[\"title\"] for m in scifi])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yvlHwQjgIs7v"
|
|
},
|
|
"source": [
|
|
"## Create Data Object with Auto-Embeddings\n",
|
|
"\n",
|
|
"This requires the user has already enabled the Vertex Prediction API, as described in the `Enable APIs` section above."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "R2KXXql2Iue9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"movie_without_embedding = {\n",
|
|
" \"id\": \"the-matrix-2\",\n",
|
|
" \"data\": {\n",
|
|
" \"title\": \"The Matrix\",\n",
|
|
" \"genre\": \"Sci-Fi\",\n",
|
|
" \"year\": 2003,\n",
|
|
" \"director\": \"The Wachowskis\",\n",
|
|
" },\n",
|
|
"}\n",
|
|
"movies.append(movie_without_embedding)\n",
|
|
"request = vectorsearch_v1beta.CreateDataObjectRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" data_object_id=movie_without_embedding[\"id\"],\n",
|
|
" data_object={\"data\": movie_without_embedding[\"data\"], \"vectors\": {}},\n",
|
|
")\n",
|
|
"data_object_service_client.create_data_object(request=request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "B_-c9HImI6r9"
|
|
},
|
|
"source": [
|
|
"## Query (list) data objects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "gq8lt1vwJZZg"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"paged_response = data_object_search_service_client.query_data_objects(\n",
|
|
" vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" page_size=2,\n",
|
|
" output_fields={\n",
|
|
" \"data_fields\": \"*\",\n",
|
|
" \"vector_fields\": \"*\",\n",
|
|
" \"metadata_fields\": \"*\",\n",
|
|
" },\n",
|
|
" )\n",
|
|
")\n",
|
|
"page1 = next(paged_response.pages)\n",
|
|
"next_page_token_1 = page1.next_page_token\n",
|
|
"page1.data_objects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "iPlmRQgBJftf"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Page 2\n",
|
|
"paged_response_2 = data_object_search_service_client.query_data_objects(\n",
|
|
" vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" page_size=2,\n",
|
|
" page_token=next_page_token_1,\n",
|
|
" output_fields={\n",
|
|
" \"data_fields\": \"*\",\n",
|
|
" \"vector_fields\": \"*\",\n",
|
|
" \"metadata_fields\": \"*\",\n",
|
|
" },\n",
|
|
" )\n",
|
|
")\n",
|
|
"page2 = next(paged_response_2.pages)\n",
|
|
"page2.data_objects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "4NYnmElNJkEk"
|
|
},
|
|
"source": [
|
|
"## Query with filters"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "oHizcC4HJnMZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"thrillers_request = vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" filter={\"genre\": {\"$eq\": \"Thriller\"}},\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
")\n",
|
|
"thrillers = data_object_search_service_client.query_data_objects(thrillers_request)\n",
|
|
"print([m.data[\"title\"] for m in thrillers])\n",
|
|
"\n",
|
|
"thrillers_since_1995_request = vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" filter={\"$and\": [{\"genre\": {\"$eq\": \"Thriller\"}}, {\"year\": {\"$gte\": 1995}}]},\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
")\n",
|
|
"thrillers_since_1995 = data_object_search_service_client.query_data_objects(\n",
|
|
" thrillers_since_1995_request\n",
|
|
")\n",
|
|
"list(thrillers_since_1995)\n",
|
|
"\n",
|
|
"nested_conditionals_request = vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" filter={\n",
|
|
" \"$or\": [\n",
|
|
" {\"director\": {\"$eq\": \"Akira Kurosawa\"}},\n",
|
|
" {\n",
|
|
" \"$and\": [\n",
|
|
" {\"director\": {\"$eq\": \"David Fincher\"}},\n",
|
|
" {\"genre\": {\"$ne\": \"Thriller\"}},\n",
|
|
" ]\n",
|
|
" },\n",
|
|
" ]\n",
|
|
" },\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
")\n",
|
|
"nested_conditionals = data_object_search_service_client.query_data_objects(\n",
|
|
" nested_conditionals_request\n",
|
|
")\n",
|
|
"list(nested_conditionals)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "VdQbEYvVKl5K"
|
|
},
|
|
"source": [
|
|
"## Query with aggregates"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "daWCB6x7KpLE"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"aggregate_request = vectorsearch_v1beta.AggregateDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" aggregate=\"COUNT\",\n",
|
|
")\n",
|
|
"data_object_search_service_client.aggregate_data_objects(aggregate_request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "uf4IcjeDKryZ"
|
|
},
|
|
"source": [
|
|
"## Search"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "l0IlxHlcKuT9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"search_request_1 = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"genre_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=normalize([0.1, 0.2, 0.3, 0.4])),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(\n",
|
|
" data_fields=[\"*\"], vector_fields=[\"*\"], metadata_fields=[\"*\"]\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"results = data_object_search_service_client.search_data_objects(search_request_1)\n",
|
|
"for result in results:\n",
|
|
" print(result.data_object)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "1gEgArXcK-l4"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"search_request_2 = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"soundtrack_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(\n",
|
|
" values=normalize([0.1, 0.1, 0.1, 0.1, 0.1])\n",
|
|
" ),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(\n",
|
|
" data_fields=[\"*\"], vector_fields=[\"*\"], metadata_fields=[\"*\"]\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"results = data_object_search_service_client.search_data_objects(search_request_2)\n",
|
|
"for result in results:\n",
|
|
" print(result.data_object)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "v7woZlw5L_3w"
|
|
},
|
|
"source": [
|
|
"## Search with filters"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "fjvJiGrsMBw8"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"search_filter_request_1 = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"plot_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=normalize([0.3, 0.4, 0.5])),\n",
|
|
" filter={\"genre\": {\"$eq\": \"Thriller\"}},\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(\n",
|
|
" data_fields=[\"*\"], vector_fields=[\"*\"], metadata_fields=[\"*\"]\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"results = data_object_search_service_client.search_data_objects(search_filter_request_1)\n",
|
|
"list(results)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ZZ4xQQyDMD-q"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"search_filter_request_2 = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"plot_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=normalize([0.3, 0.4, 0.5])),\n",
|
|
" filter={\n",
|
|
" \"$and\": [\n",
|
|
" {\"genre\": {\"$eq\": \"Thriller\"}},\n",
|
|
" {\"director\": {\"$eq\": \"David Fincher\"}},\n",
|
|
" ]\n",
|
|
" },\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(\n",
|
|
" data_fields=[\"*\"], vector_fields=[\"*\"], metadata_fields=[\"*\"]\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"results = data_object_search_service_client.search_data_objects(search_filter_request_2)\n",
|
|
"list(results)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "QVCS25RBMNjc"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"search_filter_request_3 = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"plot_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=normalize([0.3, 0.4, 0.5])),\n",
|
|
" filter={\n",
|
|
" \"$and\": [\n",
|
|
" {\"genre\": {\"$eq\": \"Thriller\"}},\n",
|
|
" {\"director\": {\"$eq\": \"David Fincher\"}},\n",
|
|
" {\"title\": {\"$ne\": \"Seven\"}},\n",
|
|
" ]\n",
|
|
" },\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(\n",
|
|
" data_fields=[\"*\"], vector_fields=[\"*\"], metadata_fields=[\"*\"]\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"results = data_object_search_service_client.search_data_objects(search_filter_request_3)\n",
|
|
"list(results)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "Q-AbPHrEMQaf"
|
|
},
|
|
"source": [
|
|
"## Semantic Search"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "Qtpqr_MVMRrw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"semantic_search_request = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" semantic_search=vectorsearch_v1beta.SemanticSearch(\n",
|
|
" search_text=\"Wonderful genre of a Wonderful movie\",\n",
|
|
" search_field=\"genre_embedding\",\n",
|
|
" task_type=\"RETRIEVAL_QUERY\",\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(\n",
|
|
" data_fields=[\"*\"], vector_fields=[\"*\"], metadata_fields=[\"*\"]\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"results = data_object_search_service_client.search_data_objects(semantic_search_request)\n",
|
|
"for result in results:\n",
|
|
" print(result.data_object)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "6Nc9O5FHMvY2"
|
|
},
|
|
"source": [
|
|
"## Text Search"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "jx3lejMPMwVJ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"text_search_request = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" text_search=vectorsearch_v1beta.TextSearch(\n",
|
|
" search_text=\"king OR castle\",\n",
|
|
" data_field_names=[\"title\"],\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(\n",
|
|
" data_fields=[\"*\"], vector_fields=[\"*\"], metadata_fields=[\"*\"]\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"results = data_object_search_service_client.search_data_objects(text_search_request)\n",
|
|
"for result in results:\n",
|
|
" print(result.data_object)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "3X2TuDEnNvm2"
|
|
},
|
|
"source": [
|
|
"## Batch Search"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "5fM5JcoBNyAH"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"batch_request_1 = vectorsearch_v1beta.BatchSearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" searches=[\n",
|
|
" vectorsearch_v1beta.Search(\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"genre_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=[0.1, 0.2, 0.3, 0.4]),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" vectorsearch_v1beta.Search(\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"soundtrack_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(\n",
|
|
" values=[0.1, 0.1, 0.1, 0.1, 0.1]\n",
|
|
" ),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" vectorsearch_v1beta.Search(\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"plot_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=[0.3, 0.4, 0.5]),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
")\n",
|
|
"data_object_search_service_client.batch_search_data_objects(batch_request_1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "bvhd0j-PN0u0"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"batch_request_2 = vectorsearch_v1beta.BatchSearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" searches=[\n",
|
|
" vectorsearch_v1beta.Search(\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"genre_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=[0.1, 0.2, 0.3, 0.4]),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" vectorsearch_v1beta.Search(\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"soundtrack_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(\n",
|
|
" values=[0.1, 0.1, 0.1, 0.1, 0.1]\n",
|
|
" ),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" vectorsearch_v1beta.Search(\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"plot_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=[0.3, 0.4, 0.5]),\n",
|
|
" top_k=5,\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
" )\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
" combine=vectorsearch_v1beta.BatchSearchDataObjectsRequest.CombineResultsOptions(\n",
|
|
" ranker=vectorsearch_v1beta.Ranker(\n",
|
|
" rrf=vectorsearch_v1beta.ReciprocalRankFusion(weights=[1.0, 1.0, 1.0])\n",
|
|
" )\n",
|
|
" ),\n",
|
|
")\n",
|
|
"data_object_search_service_client.batch_search_data_objects(batch_request_2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "PHFGTj_5Q7uK"
|
|
},
|
|
"source": [
|
|
"## Create ANN Index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "C4zKyCJSQ-mM"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.CreateIndexRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" index_id=\"plot_index\",\n",
|
|
" index={\n",
|
|
" \"index_field\": \"plot_embedding\",\n",
|
|
" \"filter_fields\": [\"year\", \"genre\"],\n",
|
|
" \"store_fields\": [\"title\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"dense_index_lro = vector_search_service_client.create_index(request)\n",
|
|
"dense_index_operation_name = dense_index_lro.operation.name\n",
|
|
"dense_index_operation_name"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "dyE1N0tgS1iG"
|
|
},
|
|
"source": [
|
|
"## Create Sparse ANN Index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "u5LVYZXdS4CF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.CreateIndexRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" index_id=\"sparse_index\",\n",
|
|
" index={\n",
|
|
" \"index_field\": \"sparse_embedding\",\n",
|
|
" \"filter_fields\": [\"year\", \"genre\"],\n",
|
|
" \"store_fields\": [\"title\"],\n",
|
|
" },\n",
|
|
")\n",
|
|
"sparse_index_lro = vector_search_service_client.create_index(request)\n",
|
|
"sparse_index_operation_name = sparse_index_lro.operation.name\n",
|
|
"sparse_index_operation_name"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "8kmxG_AHTLhI"
|
|
},
|
|
"source": [
|
|
"## Poll LROs\n",
|
|
"\n",
|
|
"Index creation operations typically take several minutes or more to complete. The progress can be polled via the operation LROs:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "8Fvw-J5fTEZP"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(f\"Waiting for dense index LRO: {dense_index_lro.operation.name}\")\n",
|
|
"dense_index_lro.result()\n",
|
|
"print(\"Dense index ready.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "No0wKmPaTBIe"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(f\"Waiting for sparse index LRO: {sparse_index_lro.operation.name}\")\n",
|
|
"sparse_index_lro.result()\n",
|
|
"print(\"Sparse index ready.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "nMM49A5nTRTW"
|
|
},
|
|
"source": [
|
|
"## Get ANN Index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "HK2SugXwTVKI"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.GetIndexRequest(\n",
|
|
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}/indexes/plot_index\"\n",
|
|
")\n",
|
|
"vector_search_service_client.get_index(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "kd9Xc1TyTXZ3"
|
|
},
|
|
"source": [
|
|
"## List ANN Indexes"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "_ibj1MuQTYZo"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.ListIndexesRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\"\n",
|
|
")\n",
|
|
"vector_search_service_client.list_indexes(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "YUvdsaasksj6"
|
|
},
|
|
"source": [
|
|
"## Delete Indexes"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "LvhXbWXDktmR"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.DeleteIndexRequest(\n",
|
|
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}/indexes/plot_index\"\n",
|
|
")\n",
|
|
"delete_index_lro = vector_search_service_client.delete_index(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "JDh8zTeNlNUT"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.DeleteIndexRequest(\n",
|
|
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}/indexes/sparse_index\"\n",
|
|
")\n",
|
|
"delete_sparse_index_lro = vector_search_service_client.delete_index(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "N7WsQEwIlNgH"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(f\"Waiting for dense index deletion LRO: {delete_index_lro.operation.name}\")\n",
|
|
"delete_index_lro.result()\n",
|
|
"print(\"Dense index deleted.\")\n",
|
|
"\n",
|
|
"print(\n",
|
|
" f\"Waiting for sparse index deletion LRO: {delete_sparse_index_lro.operation.name}\"\n",
|
|
")\n",
|
|
"delete_sparse_index_lro.result()\n",
|
|
"print(\"Sparse index deleted.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "0MFmsip8N91c"
|
|
},
|
|
"source": [
|
|
"## Delete data objects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "mnVQI1gHOANw"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" filter={\"$or\": [{\"title\": {\"$eq\": \"Seven\"}}, {\"title\": {\"$eq\": \"Se7en\"}}]},\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
")\n",
|
|
"data_object_search_service_client.query_data_objects(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "fEqfffm1OC3L"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"delete_request = vectorsearch_v1beta.DeleteDataObjectRequest(\n",
|
|
" name=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}/dataObjects/seven\"\n",
|
|
")\n",
|
|
"data_object_service_client.delete_data_object(delete_request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "2MSDfzgVOFWF"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.SearchDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" vector_search=vectorsearch_v1beta.VectorSearch(\n",
|
|
" search_field=\"plot_embedding\",\n",
|
|
" vector=vectorsearch_v1beta.DenseVector(values=normalize([0.3, 0.4, 0.5])),\n",
|
|
" filter={\n",
|
|
" \"$and\": [\n",
|
|
" {\"genre\": {\"$eq\": \"Thriller\"}},\n",
|
|
" {\"director\": {\"$eq\": \"David Fincher\"}},\n",
|
|
" ]\n",
|
|
" },\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
" ),\n",
|
|
")\n",
|
|
"data_object_search_service_client.search_data_objects(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "-sO3OV4COZEg"
|
|
},
|
|
"source": [
|
|
"## Batch Delete data objects"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NVZhkhTuOdFZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" filter={\"$or\": [{\"genre\": {\"$eq\": \"Sci-Fi\"}}]},\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
")\n",
|
|
"data_object_search_service_client.query_data_objects(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "NSy_ZmtBOhhZ"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"batch_delete_request = vectorsearch_v1beta.BatchDeleteDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" requests=[\n",
|
|
" vectorsearch_v1beta.DeleteDataObjectRequest(\n",
|
|
" name=f\"{collection_name}/dataObjects/the-matrix\"\n",
|
|
" ),\n",
|
|
" vectorsearch_v1beta.DeleteDataObjectRequest(\n",
|
|
" name=f\"{collection_name}/dataObjects/inception\"\n",
|
|
" ),\n",
|
|
" ],\n",
|
|
")\n",
|
|
"data_object_service_client.batch_delete_data_objects(batch_delete_request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "69_O9WbDOnKs"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"request = vectorsearch_v1beta.QueryDataObjectsRequest(\n",
|
|
" parent=f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\",\n",
|
|
" filter={\"$or\": [{\"genre\": {\"$eq\": \"Sci-Fi\"}}]},\n",
|
|
" output_fields=vectorsearch_v1beta.OutputFields(data_fields=[\"*\"]),\n",
|
|
")\n",
|
|
"data_object_search_service_client.query_data_objects(request)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "-aE0ZTxOOruh"
|
|
},
|
|
"source": [
|
|
"## Clean up Collection\n",
|
|
"\n",
|
|
"Please also delete any ANN indexes, as described in the `Delete Indexes` section above."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "ezl_ZrdOOuaD"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"collection_name = (\n",
|
|
" f\"projects/{PROJECT_ID}/locations/{LOCATION}/collections/{collection_id}\"\n",
|
|
")\n",
|
|
"for movie in movies:\n",
|
|
" request = vectorsearch_v1beta.DeleteDataObjectRequest(\n",
|
|
" name=f\"{collection_name}/dataObjects/{movie['id']}\"\n",
|
|
" )\n",
|
|
" try:\n",
|
|
" data_object_service_client.delete_data_object(request)\n",
|
|
" except:\n",
|
|
" pass\n",
|
|
"request = vectorsearch_v1beta.DeleteCollectionRequest(name=collection_name)\n",
|
|
"vector_search_service_client.delete_collection(request).result()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"name": "vector-search-2-quickstart.ipynb",
|
|
"toc_visible": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|