wehub/openai--openai-cookbook
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Rebuild Cookbook Website / deploy (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:41:49 +08:00
commit 327604cc89
3065 changed files with 860207 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb
+510
View File
@@ -0,0 +1,510 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using PolarDB-PG as a vector database for OpenAI embeddings\n",
"\n",
"This notebook guides you step by step on using PolarDB-PG as a vector database for OpenAI embeddings.\n",
"\n",
"This notebook presents an end-to-end process of:\n",
"1. Using precomputed embeddings created by OpenAI API.\n",
"2. Storing the embeddings in a cloud instance of PolarDB-PG.\n",
"3. Converting raw text query to an embedding with OpenAI API.\n",
"4. Using PolarDB-PG to perform the nearest neighbour search in the created collection.\n",
"\n",
"### What is PolarDB-PG\n",
"\n",
"[PolarDB-PG](https://www.alibabacloud.com/help/en/polardb/latest/what-is-polardb-2) is a high-performance vector database that adopts a read-write separation architecture. It is a cloud-native database managed by Alibaba Cloud, 100% compatible with PostgreSQL, and highly compatible with Oracle syntax. It supports processing massive vector data storage and queries, and greatly improves the efficiency of vector calculations through optimization of underlying execution algorithms, providing users with fast, elastic, high-performance, massive storage, and secure and reliable vector database services. Additionally, PolarDB-PG also supports multi-dimensional and multi-modal spatiotemporal information engines and geographic information engines.At the same time, PolarDB-PG is equipped with complete OLAP functionality and service level agreements, which has been recognized and used by many users;\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"### Deployment options\n",
"\n",
"- Using [PolarDB-PG Cloud Vector Database](https://www.alibabacloud.com/product/polardb-for-postgresql). [Click here](https://www.alibabacloud.com/product/polardb-for-postgresql?spm=a3c0i.147400.6791778070.243.9f204881g5cjpP) to fast deploy it."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"For the purposes of this exercise we need to prepare a couple of things:\n",
"\n",
"1. PolarDB-PG cloud server instance.\n",
"2. The 'psycopg2' library to interact with the vector database. Any other postgresql client library is ok.\n",
"3. An [OpenAI API key](https://beta.openai.com/account/api-keys)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We might validate if the server was launched successfully by running a simple curl command:"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install requirements\n",
"\n",
"This notebook obviously requires the `openai` and `psycopg2` packages, but there are also some other additional libraries we will use. The following command installs them all:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install openai psycopg2 pandas wget"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Prepare your OpenAI API key\n",
"The OpenAI API key is used for vectorization of the documents and queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.\n",
"\n",
"Once you get your key, please add it to your environment variables as OPENAI_API_KEY.\n",
"\n",
"If you have any doubts about setting the API key through environment variables, please refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print(\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print(\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to PolarDB\n",
"First add it to your environment variables. or you can just change the \"psycopg2.connect\" parameters below\n",
"\n",
"Connecting to a running instance of PolarDB server is easy with the official Python library:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import psycopg2\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"PGHOST\"] = \"your_host\"\n",
"# os.environ[\"PGPORT\"] \"5432\"),\n",
"# os.environ[\"PGDATABASE\"] \"postgres\"),\n",
"# os.environ[\"PGUSER\"] \"user\"),\n",
"# os.environ[\"PGPASSWORD\"] \"password\"),\n",
"\n",
"connection = psycopg2.connect(\n",
" host=os.environ.get(\"PGHOST\", \"localhost\"),\n",
" port=os.environ.get(\"PGPORT\", \"5432\"),\n",
" database=os.environ.get(\"PGDATABASE\", \"postgres\"),\n",
" user=os.environ.get(\"PGUSER\", \"user\"),\n",
" password=os.environ.get(\"PGPASSWORD\", \"password\")\n",
")\n",
"\n",
"# Create a new cursor object\n",
"cursor = connection.cursor()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can test the connection by running any available method:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Connection successful!\n"
]
}
],
"source": [
"# Execute a simple query to test the connection\n",
"cursor.execute(\"SELECT 1;\")\n",
"result = cursor.fetchone()\n",
"\n",
"# Check the query result\n",
"if result == (1,):\n",
" print(\"Connection successful!\")\n",
"else:\n",
" print(\"Connection failed.\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded.zip'"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The downloaded file has to be then extracted:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n"
]
}
],
"source": [
"import zipfile\n",
"import os\n",
"import re\n",
"import tempfile\n",
"\n",
"current_directory = os.getcwd()\n",
"zip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\n",
"output_directory = os.path.join(current_directory, \"../../data\")\n",
"\n",
"with zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(output_directory)\n",
"\n",
"\n",
"# check the csv file exist\n",
"file_name = \"vector_database_wikipedia_articles_embedded.csv\"\n",
"data_directory = os.path.join(current_directory, \"../../data\")\n",
"file_path = os.path.join(data_directory, file_name)\n",
"\n",
"\n",
"if os.path.exists(file_path):\n",
" print(f\"The file {file_name} exists in the data directory.\")\n",
"else:\n",
" print(f\"The file {file_name} does not exist in the data directory.\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index data\n",
"\n",
"PolarDB stores data in __relation__ where each object is described by at least one vector. Our relation will be called **articles** and each object will be described by both **title** and **content** vectors. \n",
"\n",
"We will start with creating a relation and create a vector index on both **title** and **content**, and then we will fill it with our precomputed embeddings."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"create_table_sql = '''\n",
"CREATE TABLE IF NOT EXISTS public.articles (\n",
" id INTEGER NOT NULL,\n",
" url TEXT,\n",
" title TEXT,\n",
" content TEXT,\n",
" title_vector vector(1536),\n",
" content_vector vector(1536),\n",
" vector_id INTEGER\n",
");\n",
"\n",
"ALTER TABLE public.articles ADD PRIMARY KEY (id);\n",
"'''\n",
"\n",
"# SQL statement for creating indexes\n",
"create_indexes_sql = '''\n",
"CREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n",
"\n",
"CREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n",
"'''\n",
"\n",
"# Execute the SQL statements\n",
"cursor.execute(create_table_sql)\n",
"cursor.execute(create_indexes_sql)\n",
"\n",
"# Commit the changes\n",
"connection.commit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"\n",
"# Path to your local CSV file\n",
"csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n",
"\n",
"# Define a generator function to process the file line by line\n",
"def process_file(file_path):\n",
" with open(file_path, 'r') as file:\n",
" for line in file:\n",
" yield line\n",
"\n",
"# Create a StringIO object to store the modified lines\n",
"modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n",
"\n",
"# Create the COPY command for the copy_expert method\n",
"copy_command = '''\n",
"COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\n",
"FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n",
"'''\n",
"\n",
"# Execute the COPY command using the copy_expert method\n",
"cursor.copy_expert(copy_command, modified_lines)\n",
"\n",
"# Commit the changes\n",
"connection.commit()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Count:25000\n"
]
}
],
"source": [
"# Check the collection size to make sure all the points have been stored\n",
"count_sql = \"\"\"select count(*) from public.articles;\"\"\"\n",
"cursor.execute(count_sql)\n",
"result = cursor.fetchone()\n",
"print(f\"Count:{result[0]}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search data\n",
"\n",
"Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"def query_polardb(query, collection_name, vector_name=\"title_vector\", top_k=20):\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.Embedding.create(\n",
" input=query,\n",
" model=\"text-embedding-3-small\",\n",
" )[\"data\"][0][\"embedding\"]\n",
"\n",
" # Convert the embedded_query to PostgreSQL compatible format\n",
" embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n",
"\n",
" # Create SQL query\n",
" query_sql = f\"\"\"\n",
" SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n",
" FROM {collection_name}\n",
" ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n",
" LIMIT {top_k};\n",
" \"\"\"\n",
" # Execute the query\n",
" cursor.execute(query_sql)\n",
" results = cursor.fetchall()\n",
"\n",
" return results"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art (Score: 0.5)\n",
"2. Western Europe (Score: 0.485)\n",
"3. Renaissance art (Score: 0.479)\n",
"4. Pop art (Score: 0.472)\n",
"5. Northern Europe (Score: 0.461)\n",
"6. Hellenistic art (Score: 0.457)\n",
"7. Modernist literature (Score: 0.447)\n",
"8. Art film (Score: 0.44)\n",
"9. Central Europe (Score: 0.439)\n",
"10. European (Score: 0.437)\n",
"11. Art (Score: 0.437)\n",
"12. Byzantine art (Score: 0.436)\n",
"13. Postmodernism (Score: 0.434)\n",
"14. Eastern Europe (Score: 0.433)\n",
"15. Europe (Score: 0.433)\n",
"16. Cubism (Score: 0.432)\n",
"17. Impressionism (Score: 0.432)\n",
"18. Bauhaus (Score: 0.431)\n",
"19. Surrealism (Score: 0.429)\n",
"20. Expressionism (Score: 0.429)\n"
]
}
],
"source": [
"import openai\n",
"\n",
"query_results = query_polardb(\"modern art in Europe\", \"Articles\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn (Score: 0.489)\n",
"2. Wars of Scottish Independence (Score: 0.474)\n",
"3. 1651 (Score: 0.457)\n",
"4. First War of Scottish Independence (Score: 0.452)\n",
"5. Robert I of Scotland (Score: 0.445)\n",
"6. 841 (Score: 0.441)\n",
"7. 1716 (Score: 0.441)\n",
"8. 1314 (Score: 0.429)\n",
"9. 1263 (Score: 0.428)\n",
"10. William Wallace (Score: 0.426)\n",
"11. Stirling (Score: 0.419)\n",
"12. 1306 (Score: 0.419)\n",
"13. 1746 (Score: 0.418)\n",
"14. 1040s (Score: 0.414)\n",
"15. 1106 (Score: 0.412)\n",
"16. 1304 (Score: 0.411)\n",
"17. David II of Scotland (Score: 0.408)\n",
"18. Braveheart (Score: 0.407)\n",
"19. 1124 (Score: 0.406)\n",
"20. July 27 (Score: 0.405)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_results = query_polardb(\"Famous battles in Scottish history\", \"Articles\", \"content_vector\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/README.md
+32
View File
@@ -0,0 +1,32 @@
# Vector Databases
This section of the OpenAI Cookbook showcases many of the vector databases available to support your semantic search use cases.
Vector databases can be a great accompaniment for knowledge retrieval applications, which reduce hallucinations by providing the LLM with the relevant context to answer questions.
Each provider has their own named directory, with a standard notebook to introduce you to using our API with their product, and any supplementary notebooks they choose to add to showcase their functionality.
## Guides & deep dives
- [AnalyticDB](https://www.alibabacloud.com/help/en/analyticdb-for-postgresql/latest/get-started-with-analyticdb-for-postgresql)
- [Cassandra/Astra DB](https://docs.datastax.com/en/astra-serverless/docs/vector-search/qandasimsearch-quickstart.html)
- [Azure AI Search](https://learn.microsoft.com/azure/search/search-get-started-vector)
- [Azure SQL Database](https://learn.microsoft.com/azure/azure-sql/database/ai-artificial-intelligence-intelligent-applications?view=azuresql)
- [Chroma](https://docs.trychroma.com/getting-started)
- [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html)
- [Hologres](https://www.alibabacloud.com/help/en/hologres/latest/procedure-to-use-hologres)
- [Kusto](https://learn.microsoft.com/en-us/azure/data-explorer/web-query-data)
- [Milvus](https://milvus.io/docs/example_code.md)
- [MyScale](https://docs.myscale.com/en/quickstart/)
- [MongoDB](https://www.mongodb.com/products/platform/atlas-vector-search)
- [Neon Postgres](https://neon.tech/docs/ai/ai-intro)
- [Pinecone](https://docs.pinecone.io/docs/quickstart)
- [PolarDB](https://www.alibabacloud.com/help/en/polardb/latest/quick-start)
- [Qdrant](https://qdrant.tech/documentation/quick-start/)
- [Redis](https://github.com/RedisVentures/simple-vecsim-intro)
- [SingleStoreDB](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/)
- [Supabase](https://supabase.com/docs/guides/ai)
- [Tembo](https://tembo.io/docs/product/stacks/ai/vectordb)
- [Typesense](https://typesense.org/docs/guide/)
- [Vespa AI](https://vespa.ai/)
- [Weaviate](https://weaviate.io/developers/weaviate/quickstart)
- [Zilliz](https://docs.zilliz.com/docs/quick-start-1)
examples/vector_databases/SingleStoreDB/OpenAI_wikipedia_semantic_search.ipynb
+563
View File
@@ -0,0 +1,563 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c2b98618",
"metadata": {},
"source": [
"# Intro\n",
"This notebook is an example on how you can use SingleStoreDB vector storage and functions to build an interactive Q&A application with ChatGPT. If you start a [Trial](https://www.singlestore.com/cloud-trial/) in SingleStoreDB, you can find the same notebook in our sample notebooks with native connection."
]
},
{
"cell_type": "markdown",
"id": "55b58478",
"metadata": {},
"source": [
"## First let's talk directly to ChatGPT and try and get back a response"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "661cd7c3",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.1.2\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.11 -m pip install --upgrade pip\u001b[0m\n"
]
}
],
"source": [
"!pip install openai --quiet\n"
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "61468873",
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"GPT_MODEL = \"gpt-3.5-turbo\"\n"
]
},
{
"cell_type": "markdown",
"id": "3778d23e",
"metadata": {},
"source": [
"## Let's connect to OpenAI and see the result we get when asking for a date beyond 2021"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "3f654b3f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"I'm sorry, I cannot provide information about events that have not occurred yet. The Winter Olympics 2022 will be held in Beijing, China from February 4 to 20, 2022. The curling events will take place during this time and the results will not be known until after the competition has concluded.\n"
]
}
],
"source": [
"openai.api_key = 'OPENAI API KEY'\n",
"\n",
"response = openai.ChatCompletion.create(\n",
" model=GPT_MODEL,\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"Who won the gold medal for curling in Olymics 2022?\"},\n",
" ]\n",
")\n",
"\n",
"print(response['choices'][0]['message']['content'])\n"
]
},
{
"cell_type": "markdown",
"id": "a9c15d6d",
"metadata": {},
"source": [
"# Get the data about Winter Olympics and provide the information to ChatGPT as context"
]
},
{
"cell_type": "markdown",
"id": "c5247835",
"metadata": {},
"source": [
"## 1. Setup"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "0948696c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.1.2\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.11 -m pip install --upgrade pip\u001b[0m\n"
]
}
],
"source": [
"!pip install matplotlib plotly.express scikit-learn tabulate tiktoken wget --quiet\n"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "1e36f5d8",
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import os\n",
"import wget\n",
"import ast\n"
]
},
{
"cell_type": "markdown",
"id": "ba9b8ae2",
"metadata": {},
"source": [
"## Step 1 - Grab the data from CSV and prepare it"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "ce3897b4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"File downloaded successfully.\n"
]
}
],
"source": [
"# download pre-chunked text and pre-computed embeddings\n",
"# this file is ~200 MB, so may take a minute depending on your connection speed\n",
"embeddings_path = \"https://cdn.openai.com/API/examples/data/winter_olympics_2022.csv\"\n",
"file_path = \"winter_olympics_2022.csv\"\n",
"\n",
"if not os.path.exists(file_path):\n",
" wget.download(embeddings_path, file_path)\n",
" print(\"File downloaded successfully.\")\n",
"else:\n",
" print(\"File already exists in the local file system.\")\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "082e9545",
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(\n",
" \"winter_olympics_2022.csv\"\n",
")\n",
"\n",
"# convert embeddings from CSV str type back to list type\n",
"df['embedding'] = df['embedding'].apply(ast.literal_eval)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1768fa60",
"metadata": {},
"outputs": [],
"source": [
"df\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "37791a10",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 6059 entries, 0 to 6058\n",
"Data columns (total 2 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 text 6059 non-null object\n",
" 1 embedding 6059 non-null object\n",
"dtypes: object(2)\n",
"memory usage: 94.8+ KB\n"
]
}
],
"source": [
"df.info(show_counts=True)\n"
]
},
{
"cell_type": "markdown",
"id": "c4e7feb6",
"metadata": {},
"source": [
"## 2. Set up SingleStore DB"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "81571781",
"metadata": {},
"outputs": [],
"source": [
"import singlestoredb as s2\n",
"\n",
"conn = s2.connect(\"<user>:<Password>@<host>:3306/\")\n",
"\n",
"cur = conn.cursor()\n"
]
},
{
"cell_type": "code",
"execution_count": 70,
"id": "e1b3fc6f",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 70,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Create database\n",
"stmt = \"\"\"\n",
" CREATE DATABASE IF NOT EXISTS winter_wikipedia2;\n",
"\"\"\"\n",
"\n",
"cur.execute(stmt)\n"
]
},
{
"cell_type": "code",
"execution_count": 71,
"id": "e49c728c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 71,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#create table\n",
"stmt = \"\"\"\n",
"CREATE TABLE IF NOT EXISTS winter_wikipedia2.winter_olympics_2022 (\n",
" id INT PRIMARY KEY,\n",
" text TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci,\n",
" embedding BLOB\n",
");\"\"\"\n",
"\n",
"cur.execute(stmt)\n"
]
},
{
"cell_type": "markdown",
"id": "8f10e57e",
"metadata": {},
"source": [
"## 3. Populate the Table with our dataframe df and use JSON_ARRAY_PACK to compact it"
]
},
{
"cell_type": "code",
"execution_count": 72,
"id": "98424a33",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 8.79 s, sys: 4.63 s, total: 13.4 s\n",
"Wall time: 11min 4s\n"
]
}
],
"source": [
"%%time\n",
"\n",
"# Prepare the statement\n",
"stmt = \"\"\"\n",
" INSERT INTO winter_wikipedia2.winter_olympics_2022 (\n",
" id,\n",
" text,\n",
" embedding\n",
" )\n",
" VALUES (\n",
" %s,\n",
" %s,\n",
" JSON_ARRAY_PACK_F64(%s)\n",
" )\n",
"\"\"\"\n",
"\n",
"# Convert the DataFrame to a NumPy record array\n",
"record_arr = df.to_records(index=True)\n",
"\n",
"# Set the batch size\n",
"batch_size = 1000\n",
"\n",
"# Iterate over the rows of the record array in batches\n",
"for i in range(0, len(record_arr), batch_size):\n",
" batch = record_arr[i:i+batch_size]\n",
" values = [(row[0], row[1], str(row[2])) for row in batch]\n",
" cur.executemany(stmt, values)\n"
]
},
{
"cell_type": "markdown",
"id": "3afeb4ec",
"metadata": {},
"source": [
"## 4. Do a semantic search with the same question from above and use the response to send to OpenAI again\n"
]
},
{
"cell_type": "code",
"execution_count": 73,
"id": "b2b79750",
"metadata": {},
"outputs": [],
"source": [
"from utils.embeddings_utils import get_embedding\n",
"\n",
"def strings_ranked_by_relatedness(\n",
" query: str,\n",
" df: pd.DataFrame,\n",
" relatedness_fn=lambda x, y: 1 - spatial.distance.cosine(x, y),\n",
" top_n: int = 100\n",
") -> tuple:\n",
" \"\"\"Returns a list of strings and relatednesses, sorted from most related to least.\"\"\"\n",
"\n",
" # Get the embedding of the query.\n",
" query_embedding_response = get_embedding(query, EMBEDDING_MODEL)\n",
"\n",
" # Create the SQL statement.\n",
" stmt = \"\"\"\n",
" SELECT\n",
" text,\n",
" DOT_PRODUCT_F64(JSON_ARRAY_PACK_F64(%s), embedding) AS score\n",
" FROM winter_wikipedia2.winter_olympics_2022\n",
" ORDER BY score DESC\n",
" LIMIT %s\n",
" \"\"\"\n",
"\n",
" # Execute the SQL statement.\n",
" results = cur.execute(stmt, [str(query_embedding_response), top_n])\n",
"\n",
" # Fetch the results\n",
" results = cur.fetchall()\n",
"\n",
" strings = []\n",
" relatednesses = []\n",
"\n",
" for row in results:\n",
" strings.append(row[0])\n",
" relatednesses.append(row[1])\n",
"\n",
" # Return the results.\n",
" return strings[:top_n], relatednesses[:top_n]\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "804f2659",
"metadata": {},
"outputs": [],
"source": [
"from tabulate import tabulate\n",
"\n",
"strings, relatednesses = strings_ranked_by_relatedness(\n",
" \"curling gold medal\",\n",
" df,\n",
" top_n=5\n",
")\n",
"\n",
"for string, relatedness in zip(strings, relatednesses):\n",
" print(f\"{relatedness=:.3f}\")\n",
" print(tabulate([[string]], headers=['Result'], tablefmt='fancy_grid'))\n"
]
},
{
"cell_type": "markdown",
"id": "3a03fd7f",
"metadata": {},
"source": [
"## 5. Send the right context to ChatGPT for a more accurate answer"
]
},
{
"cell_type": "code",
"execution_count": 75,
"id": "13265651",
"metadata": {},
"outputs": [],
"source": [
"import tiktoken\n",
"\n",
"def num_tokens(text: str, model: str = GPT_MODEL) -> int:\n",
" \"\"\"Return the number of tokens in a string.\"\"\"\n",
" encoding = tiktoken.encoding_for_model(model)\n",
" return len(encoding.encode(text))\n",
"\n",
"\n",
"def query_message(\n",
" query: str,\n",
" df: pd.DataFrame,\n",
" model: str,\n",
" token_budget: int\n",
") -> str:\n",
" \"\"\"Return a message for GPT, with relevant source texts pulled from SingleStoreDB.\"\"\"\n",
" strings, relatednesses = strings_ranked_by_relatedness(query, df, \"winter_olympics_2022\")\n",
" introduction = 'Use the below articles on the 2022 Winter Olympics to answer the subsequent question. If the answer cannot be found in the articles, write \"I could not find an answer.\"'\n",
" question = f\"\\n\\nQuestion: {query}\"\n",
" message = introduction\n",
" for string in strings:\n",
" next_article = f'\\n\\nWikipedia article section:\\n\"\"\"\\n{string}\\n\"\"\"'\n",
" if (\n",
" num_tokens(message + next_article + question, model=model)\n",
" > token_budget\n",
" ):\n",
" break\n",
" else:\n",
" message += next_article\n",
" return message + question\n",
"\n",
"\n",
"def ask(\n",
" query: str,\n",
" df: pd.DataFrame = df,\n",
" model: str = GPT_MODEL,\n",
" token_budget: int = 4096 - 500,\n",
" print_message: bool = False,\n",
") -> str:\n",
" \"\"\"Answers a query using GPT and a table of relevant texts and embeddings in SingleStoreDB.\"\"\"\n",
" message = query_message(query, df, model=model, token_budget=token_budget)\n",
" if print_message:\n",
" print(message)\n",
" messages = [\n",
" {\"role\": \"system\", \"content\": \"You answer questions about the 2022 Winter Olympics.\"},\n",
" {\"role\": \"user\", \"content\": message},\n",
" ]\n",
" response = openai.ChatCompletion.create(\n",
" model=model,\n",
" messages=messages,\n",
" temperature=0\n",
" )\n",
" response_message = response[\"choices\"][0][\"message\"][\"content\"]\n",
" return response_message\n"
]
},
{
"cell_type": "markdown",
"id": "c9128b90",
"metadata": {},
"source": [
"## 6. Get an answer from Chat GPT"
]
},
{
"cell_type": "code",
"execution_count": 76,
"id": "d295286a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(\"There were three curling events at the 2022 Winter Olympics: men's, women's, \"\n",
" 'and mixed doubles. The gold medalists for each event are:\\n'\n",
" '\\n'\n",
" \"- Men's: Sweden (Niklas Edin, Oskar Eriksson, Rasmus Wranå, Christoffer \"\n",
" 'Sundgren, Daniel Magnusson)\\n'\n",
" \"- Women's: Great Britain (Eve Muirhead, Vicky Wright, Jennifer Dodds, Hailey \"\n",
" 'Duff, Mili Smith)\\n'\n",
" '- Mixed doubles: Italy (Stefania Constantini, Amos Mosaner)')\n"
]
}
],
"source": [
"from pprint import pprint\n",
"\n",
"answer = ask('Who won the gold medal for curling in Olymics 2022?')\n",
"\n",
"pprint(answer)\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.11.0 64-bit",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
},
"vscode": {
"interpreter": {
"hash": "5c7b89af1651d0b8571dde13640ecdccf7d5a6204171d6ab33e7c296e100e08a"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/SingleStoreDB/README.md
+15
View File
@@ -0,0 +1,15 @@
**[SingleStoreDB](https://singlestore.com)** has first-class support for vector search through our [Vector Functions](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions.html). Our vector database subsystem, first made available in 2017 and subsequently enhanced, allows extremely fast nearest-neighbor search to find objects that are semantically similar, easily using SQL.
SingleStoreDB supports vectors and vector similarity search using dot_product (for cosine similarity) and euclidean_distance functions. These functions are used by our customers for applications including face recognition, visual product photo search and text-based semantic search. With the explosion of generative AI technology, these capabilities form a firm foundation for text-based AI chatbots.
But remember, SingleStoreDB is a high-performance, scalable, modern SQL DBMS that supports multiple data models including structured data, semi-structured data based on JSON, time-series, full text, spatial, key-value and of course vector data. Start powering your next intelligent application with SingleStoreDB today!
![SingleStore Open AI](https://user-images.githubusercontent.com/8846480/236985121-48980956-fdc5-49c8-b006-f3a412142676.png)
## Example
This folder contains examples of using SingleStoreDB and OpenAI together. We will keep adding more scenarios so stay tuned!
| Name | Description |
| --- | --- |
| [OpenAI wikipedia semantic search](./OpenAI_wikipedia_semantic_search.ipynb) | Improve ChatGPT accuracy through SingleStoreDB semantic Search in QA |
examples/vector_databases/analyticdb/Getting_started_with_AnalyticDB_and_OpenAI.ipynb
+589
View File
@@ -0,0 +1,589 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using AnalyticDB as a vector database for OpenAI embeddings\n",
"\n",
"This notebook guides you step by step on using AnalyticDB as a vector database for OpenAI embeddings.\n",
"\n",
"This notebook presents an end-to-end process of:\n",
"1. Using precomputed embeddings created by OpenAI API.\n",
"2. Storing the embeddings in a cloud instance of AnalyticDB.\n",
"3. Converting raw text query to an embedding with OpenAI API.\n",
"4. Using AnalyticDB to perform the nearest neighbour search in the created collection.\n",
"\n",
"### What is AnalyticDB\n",
"\n",
"[AnalyticDB](https://www.alibabacloud.com/help/en/analyticdb-for-postgresql/latest/product-introduction-overview) is a high-performance distributed vector database. Fully compatible with PostgreSQL syntax, you can effortlessly utilize it. AnalyticDB is Alibaba Cloud managed cloud-native database with strong-performed vector compute engine. Absolute out-of-box experience allow to scale into billions of data vectors processing with rich features including indexing algorithms, structured & non-structured data features, realtime update, distance metrics, scalar filtering, time travel searches etc. Also equipped with full OLAP database functionality and SLA commitment for production usage promise;\n",
"\n",
"### Deployment options\n",
"\n",
"- Using [AnalyticDB Cloud Vector Database](https://www.alibabacloud.com/help/zh/analyticdb-for-postgresql/latest/overview-2). [Click here](https://www.alibabacloud.com/product/hybriddb-postgresql) to fast deploy it.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"For the purposes of this exercise we need to prepare a couple of things:\n",
"\n",
"1. AnalyticDB cloud server instance.\n",
"2. The 'psycopg2' library to interact with the vector database. Any other postgresql client library is ok.\n",
"3. An [OpenAI API key](https://beta.openai.com/account/api-keys).\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We might validate if the server was launched successfully by running a simple curl command:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install requirements\n",
"\n",
"This notebook obviously requires the `openai` and `psycopg2` packages, but there are also some other additional libraries we will use. The following command installs them all:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.718972Z",
"start_time": "2023-02-16T12:04:30.434820Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"! pip install openai psycopg2 pandas wget"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare your OpenAI API key\n",
"\n",
"The OpenAI API key is used for vectorization of the documents and queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.730338Z",
"start_time": "2023-02-16T12:05:05.723351Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print(\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print(\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to AnalyticDB\n",
"First add it to your environment variables. or you can just change the \"psycopg2.connect\" parameters below\n",
"\n",
"Connecting to a running instance of AnalyticDB server is easy with the official Python library:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.827143Z",
"start_time": "2023-02-16T12:05:05.733771Z"
}
},
"outputs": [],
"source": [
"import os\n",
"import psycopg2\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"PGHOST\"] = \"your_host\"\n",
"# os.environ[\"PGPORT\"] \"5432\"),\n",
"# os.environ[\"PGDATABASE\"] \"postgres\"),\n",
"# os.environ[\"PGUSER\"] \"user\"),\n",
"# os.environ[\"PGPASSWORD\"] \"password\"),\n",
"\n",
"connection = psycopg2.connect(\n",
" host=os.environ.get(\"PGHOST\", \"localhost\"),\n",
" port=os.environ.get(\"PGPORT\", \"5432\"),\n",
" database=os.environ.get(\"PGDATABASE\", \"postgres\"),\n",
" user=os.environ.get(\"PGUSER\", \"user\"),\n",
" password=os.environ.get(\"PGPASSWORD\", \"password\")\n",
")\n",
"\n",
"# Create a new cursor object\n",
"cursor = connection.cursor()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can test the connection by running any available method:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.848488Z",
"start_time": "2023-02-16T12:05:06.832612Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Connection successful!\n"
]
}
],
"source": [
"\n",
"# Execute a simple query to test the connection\n",
"cursor.execute(\"SELECT 1;\")\n",
"result = cursor.fetchone()\n",
"\n",
"# Check the query result\n",
"if result == (1,):\n",
" print(\"Connection successful!\")\n",
"else:\n",
" print(\"Connection failed.\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:37.371951Z",
"start_time": "2023-02-16T12:05:06.851634Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100% [......................................................................] 698933052 / 698933052"
]
},
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded.zip'"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The downloaded file has to be then extracted:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:06:01.538851Z",
"start_time": "2023-02-16T12:05:37.376042Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n"
]
}
],
"source": [
"import zipfile\n",
"import os\n",
"import re\n",
"import tempfile\n",
"\n",
"current_directory = os.getcwd()\n",
"zip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\n",
"output_directory = os.path.join(current_directory, \"../../data\")\n",
"\n",
"with zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(output_directory)\n",
"\n",
"\n",
"# check the csv file exist\n",
"file_name = \"vector_database_wikipedia_articles_embedded.csv\"\n",
"data_directory = os.path.join(current_directory, \"../../data\")\n",
"file_path = os.path.join(data_directory, file_name)\n",
"\n",
"\n",
"if os.path.exists(file_path):\n",
" print(f\"The file {file_name} exists in the data directory.\")\n",
"else:\n",
" print(f\"The file {file_name} does not exist in the data directory.\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index data\n",
"\n",
"AnalyticDB stores data in __relation__ where each object is described by at least one vector. Our relation will be called **articles** and each object will be described by both **title** and **content** vectors. \\\n",
"\n",
"We will start with creating a relation and create a vector index on both **title** and **content**, and then we will fill it with our precomputed embeddings."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:17:36.366066Z",
"start_time": "2023-02-16T12:17:35.486872Z"
}
},
"outputs": [],
"source": [
"create_table_sql = '''\n",
"CREATE TABLE IF NOT EXISTS public.articles (\n",
" id INTEGER NOT NULL,\n",
" url TEXT,\n",
" title TEXT,\n",
" content TEXT,\n",
" title_vector REAL[],\n",
" content_vector REAL[],\n",
" vector_id INTEGER\n",
");\n",
"\n",
"ALTER TABLE public.articles ADD PRIMARY KEY (id);\n",
"'''\n",
"\n",
"# SQL statement for creating indexes\n",
"create_indexes_sql = '''\n",
"CREATE INDEX ON public.articles USING ann (content_vector) WITH (distancemeasure = l2, dim = '1536', pq_segments = '64', hnsw_m = '100', pq_centers = '2048');\n",
"\n",
"CREATE INDEX ON public.articles USING ann (title_vector) WITH (distancemeasure = l2, dim = '1536', pq_segments = '64', hnsw_m = '100', pq_centers = '2048');\n",
"'''\n",
"\n",
"# Execute the SQL statements\n",
"cursor.execute(create_table_sql)\n",
"cursor.execute(create_indexes_sql)\n",
"\n",
"# Commit the changes\n",
"connection.commit()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:37.518210Z",
"start_time": "2023-02-16T12:17:36.368564Z"
},
"pycharm": {
"is_executing": true
},
"scrolled": false
},
"outputs": [],
"source": [
"import io\n",
"\n",
"# Path to your local CSV file\n",
"csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n",
"\n",
"# Define a generator function to process the file line by line\n",
"def process_file(file_path):\n",
" with open(file_path, 'r') as file:\n",
" for line in file:\n",
" # Replace '[' with '{' and ']' with '}'\n",
" modified_line = line.replace('[', '{').replace(']', '}')\n",
" yield modified_line\n",
"\n",
"# Create a StringIO object to store the modified lines\n",
"modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n",
"\n",
"# Create the COPY command for the copy_expert method\n",
"copy_command = '''\n",
"COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\n",
"FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n",
"'''\n",
"\n",
"# Execute the COPY command using the copy_expert method\n",
"cursor.copy_expert(copy_command, modified_lines)\n",
"\n",
"# Commit the changes\n",
"connection.commit()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.675202Z",
"start_time": "2023-02-16T12:30:40.655654Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Count:25000\n"
]
}
],
"source": [
"# Check the collection size to make sure all the points have been stored\n",
"count_sql = \"\"\"select count(*) from public.articles;\"\"\"\n",
"cursor.execute(count_sql)\n",
"result = cursor.fetchone()\n",
"print(f\"Count:{result[0]}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search data\n",
"\n",
"Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:38.024370Z",
"start_time": "2023-02-16T12:30:37.712816Z"
}
},
"outputs": [],
"source": [
"def query_analyticdb(query, collection_name, vector_name=\"title_vector\", top_k=20):\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.Embedding.create(\n",
" input=query,\n",
" model=\"text-embedding-3-small\",\n",
" )[\"data\"][0][\"embedding\"]\n",
"\n",
" # Convert the embedded_query to PostgreSQL compatible format\n",
" embedded_query_pg = \"{\" + \",\".join(map(str, embedded_query)) + \"}\"\n",
"\n",
" # Create SQL query\n",
" query_sql = f\"\"\"\n",
" SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::real[]) AS similarity\n",
" FROM {collection_name}\n",
" ORDER BY {vector_name} <-> '{embedded_query_pg}'::real[]\n",
" LIMIT {top_k};\n",
" \"\"\"\n",
" # Execute the query\n",
" cursor.execute(query_sql)\n",
" results = cursor.fetchall()\n",
"\n",
" return results"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:39.379566Z",
"start_time": "2023-02-16T12:30:38.031041Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art (Score: 0.75)\n",
"2. Western Europe (Score: 0.735)\n",
"3. Renaissance art (Score: 0.728)\n",
"4. Pop art (Score: 0.721)\n",
"5. Northern Europe (Score: 0.71)\n",
"6. Hellenistic art (Score: 0.706)\n",
"7. Modernist literature (Score: 0.694)\n",
"8. Art film (Score: 0.687)\n",
"9. Central Europe (Score: 0.685)\n",
"10. European (Score: 0.683)\n",
"11. Art (Score: 0.683)\n",
"12. Byzantine art (Score: 0.682)\n",
"13. Postmodernism (Score: 0.68)\n",
"14. Eastern Europe (Score: 0.679)\n",
"15. Europe (Score: 0.678)\n",
"16. Cubism (Score: 0.678)\n",
"17. Impressionism (Score: 0.677)\n",
"18. Bauhaus (Score: 0.676)\n",
"19. Surrealism (Score: 0.674)\n",
"20. Expressionism (Score: 0.674)\n"
]
}
],
"source": [
"import openai\n",
"\n",
"query_results = query_analyticdb(\"modern art in Europe\", \"Articles\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.652676Z",
"start_time": "2023-02-16T12:30:39.382555Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn (Score: 0.739)\n",
"2. Wars of Scottish Independence (Score: 0.723)\n",
"3. 1651 (Score: 0.705)\n",
"4. First War of Scottish Independence (Score: 0.699)\n",
"5. Robert I of Scotland (Score: 0.692)\n",
"6. 841 (Score: 0.688)\n",
"7. 1716 (Score: 0.688)\n",
"8. 1314 (Score: 0.674)\n",
"9. 1263 (Score: 0.673)\n",
"10. William Wallace (Score: 0.671)\n",
"11. Stirling (Score: 0.663)\n",
"12. 1306 (Score: 0.662)\n",
"13. 1746 (Score: 0.661)\n",
"14. 1040s (Score: 0.656)\n",
"15. 1106 (Score: 0.654)\n",
"16. 1304 (Score: 0.653)\n",
"17. David II of Scotland (Score: 0.65)\n",
"18. Braveheart (Score: 0.649)\n",
"19. 1124 (Score: 0.648)\n",
"20. July 27 (Score: 0.646)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_results = query_analyticdb(\"Famous battles in Scottish history\", \"Articles\", \"content_vector\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.7"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
examples/vector_databases/analyticdb/QA_with_Langchain_AnalyticDB_and_OpenAI.ipynb
+453
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb
+738
View File
@@ -0,0 +1,738 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Azure AI Search as a vector database for OpenAI embeddings"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook provides step by step instuctions on using Azure AI Search (f.k.a Azure Cognitive Search) as a vector database with OpenAI embeddings. Azure AI Search is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications.\n",
"\n",
"## Prerequistites:\n",
"For the purposes of this exercise you must have the following:\n",
"- [Azure AI Search Service](https://learn.microsoft.com/azure/search/)\n",
"- [OpenAI Key](https://platform.openai.com/account/api-keys) or [Azure OpenAI credentials](https://learn.microsoft.com/azure/cognitive-services/openai/)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install wget\n",
"! pip install azure-search-documents \n",
"! pip install azure-identity\n",
"! pip install openai"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import required libraries"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import json \n",
"import wget\n",
"import pandas as pd\n",
"import zipfile\n",
"from openai import AzureOpenAI\n",
"from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
"from azure.core.credentials import AzureKeyCredential \n",
"from azure.search.documents import SearchClient, SearchIndexingBufferedSender \n",
"from azure.search.documents.indexes import SearchIndexClient \n",
"from azure.search.documents.models import (\n",
" QueryAnswerType,\n",
" QueryCaptionType,\n",
" QueryType,\n",
" VectorizedQuery,\n",
")\n",
"from azure.search.documents.indexes.models import (\n",
" HnswAlgorithmConfiguration,\n",
" HnswParameters,\n",
" SearchField,\n",
" SearchableField,\n",
" SearchFieldDataType,\n",
" SearchIndex,\n",
" SemanticConfiguration,\n",
" SemanticField,\n",
" SemanticPrioritizedFields,\n",
" SemanticSearch,\n",
" SimpleField,\n",
" VectorSearch,\n",
" VectorSearchAlgorithmKind,\n",
" VectorSearchAlgorithmMetric,\n",
" VectorSearchProfile,\n",
")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configure OpenAI settings\n",
"\n",
"This section guides you through setting up authentication for Azure OpenAI, allowing you to securely interact with the service using either Azure Active Directory (AAD) or an API key. Before proceeding, ensure you have your Azure OpenAI endpoint and credentials ready. For detailed instructions on setting up AAD with Azure OpenAI, refer to the [official documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/managed-identity).\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"endpoint: str = \"YOUR_AZURE_OPENAI_ENDPOINT\"\n",
"api_key: str = \"YOUR_AZURE_OPENAI_KEY\"\n",
"api_version: str = \"2023-05-15\"\n",
"deployment = \"YOUR_AZURE_OPENAI_DEPLOYMENT_NAME\"\n",
"credential = DefaultAzureCredential()\n",
"token_provider = get_bearer_token_provider(\n",
" credential, \"https://cognitiveservices.azure.com/.default\"\n",
")\n",
"\n",
"# Set this flag to True if you are using Azure Active Directory\n",
"use_aad_for_aoai = True \n",
"\n",
"if use_aad_for_aoai:\n",
" # Use Azure Active Directory (AAD) authentication\n",
" client = AzureOpenAI(\n",
" azure_endpoint=endpoint,\n",
" api_version=api_version,\n",
" azure_ad_token_provider=token_provider,\n",
" )\n",
"else:\n",
" # Use API key authentication\n",
" client = AzureOpenAI(\n",
" api_key=api_key,\n",
" api_version=api_version,\n",
" azure_endpoint=endpoint,\n",
" )"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configure Azure AI Search Vector Store settings\n",
"This section explains how to set up the Azure AI Search client for integrating with the Vector Store feature. You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# Configuration\n",
"search_service_endpoint: str = \"YOUR_AZURE_SEARCH_ENDPOINT\"\n",
"search_service_api_key: str = \"YOUR_AZURE_SEARCH_ADMIN_KEY\"\n",
"index_name: str = \"azure-ai-search-openai-cookbook-demo\"\n",
"\n",
"# Set this flag to True if you are using Azure Active Directory\n",
"use_aad_for_search = True \n",
"\n",
"if use_aad_for_search:\n",
" # Use Azure Active Directory (AAD) authentication\n",
" credential = DefaultAzureCredential()\n",
"else:\n",
" # Use API key authentication\n",
" credential = AzureKeyCredential(search_service_api_key)\n",
"\n",
"# Initialize the SearchClient with the selected authentication method\n",
"search_client = SearchClient(\n",
" endpoint=search_service_endpoint, index_name=index_name, credential=credential\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded.zip'"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\", \"r\") as zip_ref:\n",
" zip_ref.extractall(\"../../data\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df = pd.read_csv(\"../../data/vector_database_wikipedia_articles_embedded.csv\")\n",
"\n",
"# Read vectors from strings back into a list using json.loads\n",
"article_df[\"title_vector\"] = article_df.title_vector.apply(json.loads)\n",
"article_df[\"content_vector\"] = article_df.content_vector.apply(json.loads)\n",
"article_df[\"vector_id\"] = article_df[\"vector_id\"].apply(str)\n",
"article_df.head()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create an index\n",
"This code snippet demonstrates how to define and create a search index using the `SearchIndexClient` from the Azure AI Search Python SDK. The index incorporates both vector search and semantic ranker capabilities. For more details, visit our documentation on how to [Create a Vector Index](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index?.tabs=config-2023-11-01%2Crest-2023-11-01%2Cpush%2Cportal-check-index)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"azure-ai-search-openai-cookbook-demo created\n"
]
}
],
"source": [
"# Initialize the SearchIndexClient\n",
"index_client = SearchIndexClient(\n",
" endpoint=search_service_endpoint, credential=credential\n",
")\n",
"\n",
"# Define the fields for the index\n",
"fields = [\n",
" SimpleField(name=\"id\", type=SearchFieldDataType.String),\n",
" SimpleField(name=\"vector_id\", type=SearchFieldDataType.String, key=True),\n",
" SimpleField(name=\"url\", type=SearchFieldDataType.String),\n",
" SearchableField(name=\"title\", type=SearchFieldDataType.String),\n",
" SearchableField(name=\"text\", type=SearchFieldDataType.String),\n",
" SearchField(\n",
" name=\"title_vector\",\n",
" type=SearchFieldDataType.Collection(SearchFieldDataType.Single),\n",
" vector_search_dimensions=1536,\n",
" vector_search_profile_name=\"my-vector-config\",\n",
" ),\n",
" SearchField(\n",
" name=\"content_vector\",\n",
" type=SearchFieldDataType.Collection(SearchFieldDataType.Single),\n",
" vector_search_dimensions=1536,\n",
" vector_search_profile_name=\"my-vector-config\",\n",
" ),\n",
"]\n",
"\n",
"# Configure the vector search configuration\n",
"vector_search = VectorSearch(\n",
" algorithms=[\n",
" HnswAlgorithmConfiguration(\n",
" name=\"my-hnsw\",\n",
" kind=VectorSearchAlgorithmKind.HNSW,\n",
" parameters=HnswParameters(\n",
" m=4,\n",
" ef_construction=400,\n",
" ef_search=500,\n",
" metric=VectorSearchAlgorithmMetric.COSINE,\n",
" ),\n",
" )\n",
" ],\n",
" profiles=[\n",
" VectorSearchProfile(\n",
" name=\"my-vector-config\",\n",
" algorithm_configuration_name=\"my-hnsw\",\n",
" )\n",
" ],\n",
")\n",
"\n",
"# Configure the semantic search configuration\n",
"semantic_search = SemanticSearch(\n",
" configurations=[\n",
" SemanticConfiguration(\n",
" name=\"my-semantic-config\",\n",
" prioritized_fields=SemanticPrioritizedFields(\n",
" title_field=SemanticField(field_name=\"title\"),\n",
" keywords_fields=[SemanticField(field_name=\"url\")],\n",
" content_fields=[SemanticField(field_name=\"text\")],\n",
" ),\n",
" )\n",
" ]\n",
")\n",
"\n",
"# Create the search index with the vector search and semantic search configurations\n",
"index = SearchIndex(\n",
" name=index_name,\n",
" fields=fields,\n",
" vector_search=vector_search,\n",
" semantic_search=semantic_search,\n",
")\n",
"\n",
"# Create or update the index\n",
"result = index_client.create_or_update_index(index)\n",
"print(f\"{result.name} created\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Uploading Data to Azure AI Search Index\n",
"\n",
"The following code snippet outlines the process of uploading a batch of documents—specifically, Wikipedia articles with pre-computed embeddings—from a pandas DataFrame to an Azure AI Search index. For a detailed guide on data import strategies and best practices, refer to [Data Import in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-data-import).\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Uploaded 25000 documents in total\n"
]
}
],
"source": [
"from azure.core.exceptions import HttpResponseError\n",
"\n",
"# Convert the 'id' and 'vector_id' columns to string so one of them can serve as our key field\n",
"article_df[\"id\"] = article_df[\"id\"].astype(str)\n",
"article_df[\"vector_id\"] = article_df[\"vector_id\"].astype(str)\n",
"# Convert the DataFrame to a list of dictionaries\n",
"documents = article_df.to_dict(orient=\"records\")\n",
"\n",
"# Create a SearchIndexingBufferedSender\n",
"batch_client = SearchIndexingBufferedSender(\n",
" search_service_endpoint, index_name, credential\n",
")\n",
"\n",
"try:\n",
" # Add upload actions for all documents in a single call\n",
" batch_client.upload_documents(documents=documents)\n",
"\n",
" # Manually flush to send any remaining documents in the buffer\n",
" batch_client.flush()\n",
"except HttpResponseError as e:\n",
" print(f\"An error occurred: {e}\")\n",
"finally:\n",
" # Clean up resources\n",
" batch_client.close()\n",
"\n",
"print(f\"Uploaded {len(documents)} documents in total\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If your dataset didn't already contain pre-computed embeddings, you can create embeddings by using the below function using the `openai` python library. You'll also notice the same function and model are being used to generate query embeddings for performing vector searches."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Content: April is the fourth month of the year in the Julian and Gregorian calendars, and comes between March\n",
"Content vector generated\n"
]
}
],
"source": [
"# Example function to generate document embedding\n",
"def generate_embeddings(text, model):\n",
" # Generate embeddings for the provided text using the specified model\n",
" embeddings_response = client.embeddings.create(model=model, input=text)\n",
" # Extract the embedding data from the response\n",
" embedding = embeddings_response.data[0].embedding\n",
" return embedding\n",
"\n",
"\n",
"first_document_content = documents[0][\"text\"]\n",
"print(f\"Content: {first_document_content[:100]}\")\n",
"\n",
"content_vector = generate_embeddings(first_document_content, deployment)\n",
"print(\"Content vector generated\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Perform a vector similarity search"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Title: Documenta\n",
"Score: 0.8599451\n",
"URL: https://simple.wikipedia.org/wiki/Documenta\n",
"\n",
"Title: Museum of Modern Art\n",
"Score: 0.85260946\n",
"URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art\n",
"\n",
"Title: Expressionism\n",
"Score: 0.852354\n",
"URL: https://simple.wikipedia.org/wiki/Expressionism\n",
"\n"
]
}
],
"source": [
"# Pure Vector Search\n",
"query = \"modern art in Europe\"\n",
" \n",
"search_client = SearchClient(search_service_endpoint, index_name, credential) \n",
"vector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields=\"content_vector\")\n",
" \n",
"results = search_client.search( \n",
" search_text=None, \n",
" vector_queries= [vector_query], \n",
" select=[\"title\", \"text\", \"url\"] \n",
")\n",
" \n",
"for result in results: \n",
" print(f\"Title: {result['title']}\") \n",
" print(f\"Score: {result['@search.score']}\") \n",
" print(f\"URL: {result['url']}\\n\") "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Perform a Hybrid Search\n",
"Hybrid search combines the capabilities of traditional keyword-based search with vector-based similarity search to provide more relevant and contextual results. This approach is particularly useful when dealing with complex queries that benefit from understanding the semantic meaning behind the text.\n",
"\n",
"The provided code snippet demonstrates how to execute a hybrid search query:"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Title: Wars of Scottish Independence\n",
"Score: 0.03306011110544205\n",
"URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence\n",
"\n",
"Title: Battle of Bannockburn\n",
"Score: 0.022253260016441345\n",
"URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn\n",
"\n",
"Title: Scottish\n",
"Score: 0.016393441706895828\n",
"URL: https://simple.wikipedia.org/wiki/Scottish\n",
"\n"
]
}
],
"source": [
"# Hybrid Search\n",
"query = \"Famous battles in Scottish history\" \n",
" \n",
"search_client = SearchClient(search_service_endpoint, index_name, credential) \n",
"vector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields=\"content_vector\")\n",
" \n",
"results = search_client.search( \n",
" search_text=query, \n",
" vector_queries= [vector_query], \n",
" select=[\"title\", \"text\", \"url\"],\n",
" top=3\n",
")\n",
" \n",
"for result in results: \n",
" print(f\"Title: {result['title']}\") \n",
" print(f\"Score: {result['@search.score']}\") \n",
" print(f\"URL: {result['url']}\\n\") "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Perform a Hybrid Search with Reranking (powered by Bing)\n",
"[Semantic ranker](https://learn.microsoft.com/azure/search/semantic-search-overview) measurably improves search relevance by using language understanding to rerank search results. Additionally, you can get extractive captions, answers, and highlights. "
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Semantic Answer: Advancements During the industrial revolution, new technology brought many changes. For example:<em> Canals</em> were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced.\n",
"Semantic Answer Score: 0.90478515625\n",
"\n",
"Title: Industrial Revolution\n",
"Reranker Score: 3.408700942993164\n",
"URL: https://simple.wikipedia.org/wiki/Industrial%20Revolution\n",
"Caption: Advancements During the industrial revolution, new technology brought many changes. For example: Canals were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced.\n",
"\n",
"Title: Printing\n",
"Reranker Score: 1.603400707244873\n",
"URL: https://simple.wikipedia.org/wiki/Printing\n",
"Caption: Machines to speed printing, cheaper paper, automatic stitching and binding all arrived in the 19th century during the industrial revolution. What had once been done by a few men by hand was now done by limited companies on huge machines. The result was much lower prices, and a much wider readership.\n",
"\n",
"Title: Industrialisation\n",
"Reranker Score: 1.3238357305526733\n",
"URL: https://simple.wikipedia.org/wiki/Industrialisation\n",
"Caption: <em>Industrialisation</em> (or<em> industrialization)</em> is a process that happens in countries when they start to use machines to do work that was once done by people.<em> Industrialisation changes</em> the things people do.<em> Industrialisation</em> caused towns to grow larger. Many people left farming to take higher paid jobs in factories in towns.\n",
"\n"
]
}
],
"source": [
"# Semantic Hybrid Search\n",
"query = \"What were the key technological advancements during the Industrial Revolution?\"\n",
"\n",
"search_client = SearchClient(search_service_endpoint, index_name, credential)\n",
"vector_query = VectorizedQuery(\n",
" vector=generate_embeddings(query, deployment),\n",
" k_nearest_neighbors=3,\n",
" fields=\"content_vector\",\n",
")\n",
"\n",
"results = search_client.search(\n",
" search_text=query,\n",
" vector_queries=[vector_query],\n",
" select=[\"title\", \"text\", \"url\"],\n",
" query_type=QueryType.SEMANTIC,\n",
" semantic_configuration_name=\"my-semantic-config\",\n",
" query_caption=QueryCaptionType.EXTRACTIVE,\n",
" query_answer=QueryAnswerType.EXTRACTIVE,\n",
" top=3,\n",
")\n",
"\n",
"semantic_answers = results.get_answers()\n",
"for answer in semantic_answers:\n",
" if answer.highlights:\n",
" print(f\"Semantic Answer: {answer.highlights}\")\n",
" else:\n",
" print(f\"Semantic Answer: {answer.text}\")\n",
" print(f\"Semantic Answer Score: {answer.score}\\n\")\n",
"\n",
"for result in results:\n",
" print(f\"Title: {result['title']}\")\n",
" print(f\"Reranker Score: {result['@search.reranker_score']}\")\n",
" print(f\"URL: {result['url']}\")\n",
" captions = result[\"@search.captions\"]\n",
" if captions:\n",
" caption = captions[0]\n",
" if caption.highlights:\n",
" print(f\"Caption: {caption.highlights}\\n\")\n",
" else:\n",
" print(f\"Caption: {caption.text}\\n\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb
+885
View File
@@ -0,0 +1,885 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "46589cdf-1ab6-4028-b07c-08b75acd98e5",
"metadata": {},
"source": [
"# Philosophy with Vector Embeddings, OpenAI and Astra DB\n",
"\n",
"### AstraPy version"
]
},
{
"cell_type": "markdown",
"id": "b3496d07-f473-4008-9133-1a54b818c8d3",
"metadata": {},
"source": [
"In this quickstart you will learn how to build a \"philosophy quote finder & generator\" using OpenAI's vector embeddings and DataStax [Astra DB](https://docs.datastax.com/en/astra/home/astra.html) as the vector store for data persistence.\n",
"\n",
"The basic workflow of this notebook is outlined below. You will evaluate and store the vector embeddings for a number of quotes by famous philosophers, use them to build a powerful search engine and, after that, even a generator of new quotes!\n",
"\n",
"The notebook exemplifies some of the standard usage patterns of vector search -- while showing how easy is it to get started with [Astra DB](https://docs.datastax.com/en/astra/home/astra.html).\n",
"\n",
"For a background on using vector search and text embeddings to build a question-answering system, please check out this excellent hands-on notebook: [Question answering using embeddings](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb).\n",
"\n",
"Table of contents:\n",
"- Setup\n",
"- Create vector collection\n",
"- Connect to OpenAI\n",
"- Load quotes into the Vector Store\n",
"- Use case 1: **quote search engine**\n",
"- Use case 2: **quote generator**\n",
"- Cleanup"
]
},
{
"cell_type": "markdown",
"id": "cddf17cc-eef4-4021-b72a-4d3832a9b4a7",
"metadata": {},
"source": [
"### How it works\n",
"\n",
"**Indexing**\n",
"\n",
"Each quote is made into an embedding vector with OpenAI's `Embedding`. These are saved in the Vector Store for later use in searching. Some metadata, including the author's name and a few other pre-computed tags, are stored alongside, to allow for search customization.\n",
"\n",
"![1_vector_indexing](https://user-images.githubusercontent.com/14221764/282422016-1d540607-eed4-4240-9c3d-22ee3a3bc90f.png)\n",
"\n",
"**Search**\n",
"\n",
"To find a quote similar to the provided search quote, the latter is made into an embedding vector on the fly, and this vector is used to query the store for similar vectors ... i.e. similar quotes that were previously indexed. The search can optionally be constrained by additional metadata (\"find me quotes by Spinoza similar to this one ...\").\n",
"\n",
"![2_vector_search](https://user-images.githubusercontent.com/14221764/282422033-0a1297c4-63bb-4e04-b120-dfd98dc1a689.png)\n",
"\n",
"The key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n",
"\n",
"The sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space. Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n",
"\n",
"So, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n",
"\n",
"![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)\n",
"\n",
"**Generation**\n",
"\n",
"Given a suggestion (a topic or a tentative quote), the search step is performed, and the first returned results (quotes) are fed into an LLM prompt which asks the generative model to invent a new text along the lines of the passed examples _and_ the initial suggestion.\n",
"\n",
"![4_quote_generation](https://user-images.githubusercontent.com/14221764/282422050-2e209ff5-07d6-41ac-99ac-f442e090b3bb.png)"
]
},
{
"cell_type": "markdown",
"id": "10493f44-565d-4f23-8bfd-1a7335392c2b",
"metadata": {},
"source": [
"## Setup"
]
},
{
"cell_type": "markdown",
"id": "44a14f95-4683-4d0c-a251-0df7b43ca975",
"metadata": {},
"source": [
"Install and import the necessary dependencies:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "39afdb74-56e4-44ff-9c72-ab2669780113",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!pip install --quiet \"astrapy>=0.6.0\" \"openai>=1.0.0\" datasets"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9ca6f5c6-30b4-4518-a816-5c732a60e339",
"metadata": {},
"outputs": [],
"source": [
"from getpass import getpass\n",
"from collections import Counter\n",
"\n",
"from astrapy.db import AstraDB\n",
"import openai\n",
"from datasets import load_dataset"
]
},
{
"cell_type": "markdown",
"id": "9cb99e33-5cb7-416f-8dca-da18e0cb108d",
"metadata": {},
"source": [
"### Connection parameters"
]
},
{
"cell_type": "markdown",
"id": "65a8edc1-4633-491b-9ed3-11163ec24e46",
"metadata": {},
"source": [
"Please retrieve your database credentials on your Astra dashboard ([info](https://docs.datastax.com/en/astra/astra-db-vector/)): you will supply them momentarily.\n",
"\n",
"Example values:\n",
"\n",
"- API Endpoint: `https://01234567-89ab-cdef-0123-456789abcdef-us-east1.apps.astra.datastax.com`\n",
"- Token: `AstraCS:6gBhNmsk135...`"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "ca5a2f5d-3ff2-43d6-91c0-4a52c0ecd06a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Please enter your API Endpoint: https://4f835778-ec78-42b0-9ae3-29e3cf45b596-us-east1.apps.astra.datastax.com\n",
"Please enter your Token ········\n"
]
}
],
"source": [
"ASTRA_DB_API_ENDPOINT = input(\"Please enter your API Endpoint:\")\n",
"ASTRA_DB_APPLICATION_TOKEN = getpass(\"Please enter your Token\")"
]
},
{
"cell_type": "markdown",
"id": "f8c4e5ec-2ab2-4d41-b3ec-c946469fed8b",
"metadata": {},
"source": [
"### Instantiate an Astra DB client"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "1b526e55-ad2c-413d-94b1-cf651afefd02",
"metadata": {},
"outputs": [],
"source": [
"astra_db = AstraDB(\n",
" api_endpoint=ASTRA_DB_API_ENDPOINT,\n",
" token=ASTRA_DB_APPLICATION_TOKEN,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "60829851-bd48-4461-9243-974f76304933",
"metadata": {},
"source": [
"## Create vector collection"
]
},
{
"cell_type": "markdown",
"id": "cbcd19dc-0580-42c2-8d45-1cef52050a59",
"metadata": {},
"source": [
"The only parameter to specify, other than the collection name, is the dimension of the vectors you'll store. Other parameters, notably the similarity metric to use for searches, are optional."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "8db837dc-cd49-41e2-8b5d-edb17ccc470e",
"metadata": {},
"outputs": [],
"source": [
"coll_name = \"philosophers_astra_db\"\n",
"collection = astra_db.create_collection(coll_name, dimension=1536)"
]
},
{
"cell_type": "markdown",
"id": "da86f91a-88a6-4997-b0f8-9da0816f8ece",
"metadata": {},
"source": [
"## Connect to OpenAI"
]
},
{
"cell_type": "markdown",
"id": "a6b664b5-fd84-492e-a7bd-4dda3863b48a",
"metadata": {},
"source": [
"### Set up your secret key"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "37fe7653-dd64-4494-83e1-5702ec41725c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Please enter your OpenAI API Key: ········\n"
]
}
],
"source": [
"OPENAI_API_KEY = getpass(\"Please enter your OpenAI API Key: \")"
]
},
{
"cell_type": "markdown",
"id": "847f2821-7f3f-4dcd-8e0c-49aa397e36f4",
"metadata": {},
"source": [
"### A test call for embeddings\n",
"\n",
"Quickly check how one can get the embedding vectors for a list of input texts:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "6bf89454-9a55-4202-ab6b-ea15b2048f3d",
"metadata": {},
"outputs": [],
"source": [
"client = openai.OpenAI(api_key=OPENAI_API_KEY)\n",
"embedding_model_name = \"text-embedding-3-small\"\n",
"\n",
"result = client.embeddings.create(\n",
" input=[\n",
" \"This is a sentence\",\n",
" \"A second sentence\"\n",
" ],\n",
" model=embedding_model_name,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "e2841934-7b2a-4a00-b112-b0865c9ec593",
"metadata": {},
"source": [
"_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "50a8e6f0-0aa7-4ffc-94e9-702b68566815",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"len(result.data) = 2\n",
"result.data[1].embedding = [-0.0108176339417696, 0.0013546717818826437, 0.00362232...\n",
"len(result.data[1].embedding) = 1536\n"
]
}
],
"source": [
"print(f\"len(result.data) = {len(result.data)}\")\n",
"print(f\"result.data[1].embedding = {str(result.data[1].embedding)[:55]}...\")\n",
"print(f\"len(result.data[1].embedding) = {len(result.data[1].embedding)}\")"
]
},
{
"cell_type": "markdown",
"id": "d7f09c42-fff3-4aa2-922b-043739b4b06a",
"metadata": {},
"source": [
"## Load quotes into the Vector Store"
]
},
{
"cell_type": "markdown",
"id": "cf0f3d58-74c2-458b-903d-3d12e61b7846",
"metadata": {},
"source": [
"Get a dataset with the quotes. _(We adapted and augmented the data from [this Kaggle dataset](https://www.kaggle.com/datasets/mertbozkurt5/quotes-by-philosophers), ready to use in this demo.)_"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "aa68f038-3240-4e22-b7c6-a5f214eda381",
"metadata": {},
"outputs": [],
"source": [
"philo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]"
]
},
{
"cell_type": "markdown",
"id": "ab6b08b1-e3db-4c7c-9d7c-2ada7c8bc71d",
"metadata": {},
"source": [
"A quick inspection:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "10b629cf-efd7-434a-9dc6-7f38f35f7cc8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"An example entry:\n",
"{'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n"
]
}
],
"source": [
"print(\"An example entry:\")\n",
"print(philo_dataset[16])"
]
},
{
"cell_type": "markdown",
"id": "9badaa4d-80ea-462c-bb00-1909c6435eea",
"metadata": {},
"source": [
"Check the dataset size:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "1b33ac73-f8f2-4b64-8a27-178ac76886a9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total: 450 quotes. By author:\n",
" aristotle : 50 quotes\n",
" schopenhauer : 50 quotes\n",
" spinoza : 50 quotes\n",
" hegel : 50 quotes\n",
" freud : 50 quotes\n",
" nietzsche : 50 quotes\n",
" sartre : 50 quotes\n",
" plato : 50 quotes\n",
" kant : 50 quotes\n"
]
}
],
"source": [
"author_count = Counter(entry[\"author\"] for entry in philo_dataset)\n",
"print(f\"Total: {len(philo_dataset)} quotes. By author:\")\n",
"for author, count in author_count.most_common():\n",
" print(f\" {author:<20}: {count} quotes\")"
]
},
{
"cell_type": "markdown",
"id": "062157d1-d262-4735-b06c-f3112575b4cc",
"metadata": {},
"source": [
"### Write to the vector collection\n",
"\n",
"You will compute the embeddings for the quotes and save them into the Vector Store, along with the text itself and the metadata you'll use later.\n",
"\n",
"To optimize speed and reduce the calls, you'll perform batched calls to the embedding OpenAI service.\n",
"\n",
"To store the quote objects, you will use the `insert_many` method of the collection (one call per batch). When preparing the documents for insertion you will choose suitable field names -- keep in mind, however, that the embedding vector must be the fixed special `$vector` field."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "6ab84ccb-3363-4bdc-9484-0d68c25a58ff",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Starting to store entries: [20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][10]\n",
"Finished storing entries.\n"
]
}
],
"source": [
"BATCH_SIZE = 20\n",
"\n",
"num_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)\n",
"\n",
"quotes_list = philo_dataset[\"quote\"]\n",
"authors_list = philo_dataset[\"author\"]\n",
"tags_list = philo_dataset[\"tags\"]\n",
"\n",
"print(\"Starting to store entries: \", end=\"\")\n",
"for batch_i in range(num_batches):\n",
" b_start = batch_i * BATCH_SIZE\n",
" b_end = (batch_i + 1) * BATCH_SIZE\n",
" # compute the embedding vectors for this batch\n",
" b_emb_results = client.embeddings.create(\n",
" input=quotes_list[b_start : b_end],\n",
" model=embedding_model_name,\n",
" )\n",
" # prepare the documents for insertion\n",
" b_docs = []\n",
" for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):\n",
" if tags_list[entry_idx]:\n",
" tags = {\n",
" tag: True\n",
" for tag in tags_list[entry_idx].split(\";\")\n",
" }\n",
" else:\n",
" tags = {}\n",
" b_docs.append({\n",
" \"quote\": quotes_list[entry_idx],\n",
" \"$vector\": emb_result.embedding,\n",
" \"author\": authors_list[entry_idx],\n",
" \"tags\": tags,\n",
" })\n",
" # write to the vector collection\n",
" collection.insert_many(b_docs)\n",
" print(f\"[{len(b_docs)}]\", end=\"\")\n",
"\n",
"print(\"\\nFinished storing entries.\")"
]
},
{
"cell_type": "markdown",
"id": "db3ee629-b6b9-4a77-8c58-c3b93403a6a6",
"metadata": {},
"source": [
"## Use case 1: **quote search engine**"
]
},
{
"cell_type": "markdown",
"id": "db3b12b3-2557-4826-af5a-16e6cd9a4531",
"metadata": {},
"source": [
"For the quote-search functionality, you need first to make the input quote into a vector, and then use it to query the store (besides handling the optional metadata into the search call, that is).\n",
"\n",
"Encapsulate the search-engine functionality into a function for ease of re-use. At its core is the `vector_find` method of the collection:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "d6fcf182-3ab7-4d28-9472-dce35cc38182",
"metadata": {},
"outputs": [],
"source": [
"def find_quote_and_author(query_quote, n, author=None, tags=None):\n",
" query_vector = client.embeddings.create(\n",
" input=[query_quote],\n",
" model=embedding_model_name,\n",
" ).data[0].embedding\n",
" filter_clause = {}\n",
" if author:\n",
" filter_clause[\"author\"] = author\n",
" if tags:\n",
" filter_clause[\"tags\"] = {}\n",
" for tag in tags:\n",
" filter_clause[\"tags\"][tag] = True\n",
" #\n",
" results = collection.vector_find(\n",
" query_vector,\n",
" limit=n,\n",
" filter=filter_clause,\n",
" fields=[\"quote\", \"author\"]\n",
" )\n",
" return [\n",
" (result[\"quote\"], result[\"author\"])\n",
" for result in results\n",
" ]"
]
},
{
"cell_type": "markdown",
"id": "2539262d-100b-4e8d-864d-e9c612a73e91",
"metadata": {},
"source": [
"### Putting search to test"
]
},
{
"cell_type": "markdown",
"id": "3634165c-0882-4281-bc60-ab96261a500d",
"metadata": {},
"source": [
"Passing just a quote:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "6722c2c0-3e54-4738-80ce-4d1149e95414",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n",
" 'schopenhauer'),\n",
" ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n",
" 'aristotle'),\n",
" ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n",
" 'freud')]"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"find_quote_and_author(\"We struggle all our life for nothing\", 3)"
]
},
{
"cell_type": "markdown",
"id": "50828e4c-9bb5-4489-9fe9-87da5fbe1f18",
"metadata": {},
"source": [
"Search restricted to an author:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "da9c705f-5c12-42b3-a038-202f89a3c6da",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('To live is to suffer, to survive is to find some meaning in the suffering.',\n",
" 'nietzsche'),\n",
" ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n",
" 'nietzsche')]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"find_quote_and_author(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")"
]
},
{
"cell_type": "markdown",
"id": "4a3857ea-6dfe-489a-9b86-4e5e0534960f",
"metadata": {},
"source": [
"Search constrained to a tag (out of those saved earlier with the quotes):"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "abcfaec9-8f42-4789-a5ed-1073fa2932c2",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('He who seeks equality between unequals seeks an absurdity.', 'spinoza'),\n",
" ('The people are that part of the state that does not know what it wants.',\n",
" 'hegel')]"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"find_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])"
]
},
{
"cell_type": "markdown",
"id": "746fe38f-139f-44a6-a225-a63e40d3ddf5",
"metadata": {},
"source": [
"### Cutting out irrelevant results\n",
"\n",
"The vector similarity search generally returns the vectors that are closest to the query, even if that means results that might be somewhat irrelevant if there's nothing better.\n",
"\n",
"To keep this issue under control, you can get the actual \"similarity\" between the query and each result, and then implement a cutoff on it, effectively discarding results that are beyond that threshold.\n",
"Tuning this threshold correctly is not an easy problem: here, we'll just show you the way.\n",
"\n",
"To get a feeling on how this works, try the following query and play with the choice of quote and threshold to compare the results. Note that the similarity is returned as the special `$similarity` field in each result document - and it will be returned by default, unless you pass `include_similarity = False` to the search method.\n",
"\n",
"_Note (for the mathematically inclined): this value is **a rescaling between zero and one** of the cosine difference between the vectors, i.e. of the scalar product divided by the product of the norms of the two vectors. In other words, this is 0 for opposite-facing vectors and +1 for parallel vectors. For other measures of similarity (cosine is the default), check the `metric` parameter in `AstraDB.create_collection` and the [documentation on allowed values](https://docs.datastax.com/en/astra-serverless/docs/develop/dev-with-json.html#metric-types)._"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "b9b43721-a3b0-4ac4-b730-7a6aeec52e70",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3 quotes within the threshold:\n",
" 0. [similarity=0.927] \"The assumption that animals are without rights, and the illusion that ...\"\n",
" 1. [similarity=0.922] \"Animals are in possession of themselves; their soul is in possession o...\"\n",
" 2. [similarity=0.920] \"At his best, man is the noblest of all animals; separated from law and...\"\n"
]
}
],
"source": [
"quote = \"Animals are our equals.\"\n",
"# quote = \"Be good.\"\n",
"# quote = \"This teapot is strange.\"\n",
"\n",
"metric_threshold = 0.92\n",
"\n",
"quote_vector = client.embeddings.create(\n",
" input=[quote],\n",
" model=embedding_model_name,\n",
").data[0].embedding\n",
"\n",
"results_full = collection.vector_find(\n",
" quote_vector,\n",
" limit=8,\n",
" fields=[\"quote\"]\n",
")\n",
"results = [res for res in results_full if res[\"$similarity\"] >= metric_threshold]\n",
"\n",
"print(f\"{len(results)} quotes within the threshold:\")\n",
"for idx, result in enumerate(results):\n",
" print(f\" {idx}. [similarity={result['$similarity']:.3f}] \\\"{result['quote'][:70]}...\\\"\")"
]
},
{
"cell_type": "markdown",
"id": "71871251-169f-4d3f-a687-65f836a9a8fe",
"metadata": {},
"source": [
"## Use case 2: **quote generator**"
]
},
{
"cell_type": "markdown",
"id": "b0a9cd63-a131-4819-bf41-c8ffa0b1e1ca",
"metadata": {},
"source": [
"For this task you need another component from OpenAI, namely an LLM to generate the quote for us (based on input obtained by querying the Vector Store).\n",
"\n",
"You also need a template for the prompt that will be filled for the generate-quote LLM completion task."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "a6dd366d-665a-45fd-917b-b6b5312b0865",
"metadata": {},
"outputs": [],
"source": [
"completion_model_name = \"gpt-3.5-turbo\"\n",
"\n",
"generation_prompt_template = \"\"\"\"Generate a single short philosophical quote on the given topic,\n",
"similar in spirit and form to the provided actual example quotes.\n",
"Do not exceed 20-30 words in your quote.\n",
"\n",
"REFERENCE TOPIC: \"{topic}\"\n",
"\n",
"ACTUAL EXAMPLES:\n",
"{examples}\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"id": "53073a9e-16de-4e49-9e97-ff31b9b250c2",
"metadata": {},
"source": [
"Like for search, this functionality is best wrapped into a handy function (which internally uses search):"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "397e6ebd-b30e-413b-be63-81a62947a7b8",
"metadata": {},
"outputs": [],
"source": [
"def generate_quote(topic, n=2, author=None, tags=None):\n",
" quotes = find_quote_and_author(query_quote=topic, n=n, author=author, tags=tags)\n",
" if quotes:\n",
" prompt = generation_prompt_template.format(\n",
" topic=topic,\n",
" examples=\"\\n\".join(f\" - {quote[0]}\" for quote in quotes),\n",
" )\n",
" # a little logging:\n",
" print(\"** quotes found:\")\n",
" for q, a in quotes:\n",
" print(f\"** - {q} ({a})\")\n",
" print(\"** end of logging\")\n",
" #\n",
" response = client.chat.completions.create(\n",
" model=completion_model_name,\n",
" messages=[{\"role\": \"user\", \"content\": prompt}],\n",
" temperature=0.7,\n",
" max_tokens=320,\n",
" )\n",
" return response.choices[0].message.content.replace('\"', '').strip()\n",
" else:\n",
" print(\"** no quotes found.\")\n",
" return None"
]
},
{
"cell_type": "markdown",
"id": "c13f8488-899b-4d4c-a069-73643a778200",
"metadata": {},
"source": [
"_Note: similar to the case of the embedding computation, the code for the Chat Completion API would be slightly different for OpenAI prior to v1.0._"
]
},
{
"cell_type": "markdown",
"id": "63bcc157-e5d4-43ef-8028-d4dcc8a72b9c",
"metadata": {},
"source": [
"#### Putting quote generation to test"
]
},
{
"cell_type": "markdown",
"id": "fe6b3f38-089d-486d-b32c-e665c725faa8",
"metadata": {},
"source": [
"Just passing a text (a \"quote\", but one can actually just suggest a topic since its vector embedding will still end up at the right place in the vector space):"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "806ba758-8988-410e-9eeb-b9c6799e6b25",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"** quotes found:\n",
"** - Happiness is the reward of virtue. (aristotle)\n",
"** - Our moral virtues benefit mainly other people; intellectual virtues, on the other hand, benefit primarily ourselves; therefore the former make us universally popular, the latter unpopular. (schopenhauer)\n",
"** end of logging\n",
"\n",
"A new generated quote:\n",
"True politics lies in the virtuous pursuit of justice, for it is through virtue that we build a better world for all.\n"
]
}
],
"source": [
"q_topic = generate_quote(\"politics and virtue\")\n",
"print(\"\\nA new generated quote:\")\n",
"print(q_topic)"
]
},
{
"cell_type": "markdown",
"id": "ca032d30-4538-4d0b-aea1-731fb32d2d4b",
"metadata": {},
"source": [
"Use inspiration from just a single philosopher:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "7c2e2d4e-865f-4b2d-80cd-a695271415d9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"** quotes found:\n",
"** - Because Christian morality leaves animals out of account, they are at once outlawed in philosophical morals; they are mere 'things,' mere means to any ends whatsoever. They can therefore be used for vivisection, hunting, coursing, bullfights, and horse racing, and can be whipped to death as they struggle along with heavy carts of stone. Shame on such a morality that is worthy of pariahs, and that fails to recognize the eternal essence that exists in every living thing, and shines forth with inscrutable significance from all eyes that see the sun! (schopenhauer)\n",
"** - The assumption that animals are without rights, and the illusion that our treatment of them has no moral significance, is a positively outrageous example of Western crudity and barbarity. Universal compassion is the only guarantee of morality. (schopenhauer)\n",
"** end of logging\n",
"\n",
"A new generated quote:\n",
"Excluding animals from ethical consideration reveals a moral blindness that allows for their exploitation and suffering. True morality embraces universal compassion.\n"
]
}
],
"source": [
"q_topic = generate_quote(\"animals\", author=\"schopenhauer\")\n",
"print(\"\\nA new generated quote:\")\n",
"print(q_topic)"
]
},
{
"cell_type": "markdown",
"id": "4bd8368a-9e23-49a5-8694-921728ea9656",
"metadata": {},
"source": [
"## Cleanup\n",
"\n",
"If you want to remove all resources used for this demo, run this cell (_warning: this will irreversibly delete the collection and its data!_):"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "1eb0fd16-7e15-4742-8fc5-94d9eeeda620",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'status': {'ok': 1}}"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"astra_db.delete_collection(coll_name)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.18"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb
+1430
View File
File diff suppressed because it is too large Load Diff
examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb
+1165
View File
File diff suppressed because it is too large Load Diff
examples/vector_databases/cassandra_astradb/README.md
+37
View File
@@ -0,0 +1,37 @@
# RAG with Astra DB and Cassandra
The demos in this directory show how to use the Vector
Search capabilities available today in **DataStax Astra DB**, a serverless
Database-as-a-Service built on Apache Cassandra®.
These example notebooks demonstrate implementation of
the same GenAI standard RAG workload with different libraries and APIs.
To use [Astra DB](https://docs.datastax.com/en/astra/home/astra.html)
with its HTTP API interface, head to the "AstraPy" notebook (`astrapy`
is the Python client to interact with the database).
If you prefer CQL access to the database (either with
[Astra DB](https://docs.datastax.com/en/astra-serverless/docs/vector-search/overview.html)
or a Cassandra cluster
[supporting vector search](https://cassandra.apache.org/doc/trunk/cassandra/vector-search/overview.html)),
check the "CQL" or "CassIO" notebooks -- they differ in the level of abstraction you get to work at.
If you want to know more about Astra DB and its Vector Search capabilities,
head over to [datastax.com](https://docs.datastax.com/en/astra/home/astra.html).
### Example notebooks
The following examples show how easily OpenAI and DataStax Astra DB can
work together to power vector-based AI applications. You can run them either
with your local Jupyter engine or as Colab notebooks:
| Use case | Target database | Framework | Notebook | Google Colab |
| -------- | --------------- | --------- | -------- | ------------ |
| Search/generate quotes | Astra DB | AstraPy | [Notebook](./Philosophical_Quotes_AstraPy.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb) |
| Search/generate quotes | Cassandra / Astra DB through CQL | CassIO | [Notebook](./Philosophical_Quotes_cassIO.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb) |
| Search/generate quotes | Cassandra / Astra DB through CQL | Plain Cassandra language | [Notebook](./Philosophical_Quotes_CQL.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb) |
### Vector similarity, visual representation
![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)
examples/vector_databases/cassandra_astradb/sources/philo_quotes.json
+2590
View File
File diff suppressed because it is too large Load Diff
examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb
+895
View File
@@ -0,0 +1,895 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Chroma for Embeddings Search\n",
"\n",
"This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"### What is a Vector Database\n",
"\n",
"A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
"\n",
"### Why use a Vector Database\n",
"\n",
"Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
"\n",
"\n",
"### Demo Flow\n",
"The demo flow is:\n",
"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **Chroma**:\n",
" - *Setup*: Here we'll set up the Python client for Chroma. For more details go [here](https://docs.trychroma.com/usage-guide)\n",
" - *Index Data*: We'll create collections with vectors for __titles__ and __content__\n",
" - *Search Data*: We'll run a few searches to confirm it works\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "e2b59250",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Import the required libraries and set the embedding model that we'd like to use."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "8d8810f9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting openai\n",
" Obtaining dependency information for openai from https://files.pythonhosted.org/packages/67/78/7588a047e458cb8075a4089d721d7af5e143ff85a2388d4a28c530be0494/openai-0.27.8-py3-none-any.whl.metadata\n",
" Downloading openai-0.27.8-py3-none-any.whl.metadata (13 kB)\n",
"Collecting requests>=2.20 (from openai)\n",
" Obtaining dependency information for requests>=2.20 from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata\n",
" Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)\n",
"Collecting tqdm (from openai)\n",
" Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)\n",
"Collecting aiohttp (from openai)\n",
" Obtaining dependency information for aiohttp from https://files.pythonhosted.org/packages/fa/9e/49002fde2a97d7df0e162e919c31cf13aa9f184537739743d1239edd0e67/aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl.metadata (7.7 kB)\n",
"Collecting charset-normalizer<4,>=2 (from requests>=2.20->openai)\n",
" Obtaining dependency information for charset-normalizer<4,>=2 from https://files.pythonhosted.org/packages/ec/a7/96835706283d63fefbbbb4f119d52f195af00fc747e67cc54397c56312c8/charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Using cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (31 kB)\n",
"Collecting idna<4,>=2.5 (from requests>=2.20->openai)\n",
" Using cached idna-3.4-py3-none-any.whl (61 kB)\n",
"Collecting urllib3<3,>=1.21.1 (from requests>=2.20->openai)\n",
" Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/9b/81/62fd61001fa4b9d0df6e31d47ff49cfa9de4af03adecf339c7bc30656b37/urllib3-2.0.4-py3-none-any.whl.metadata\n",
" Downloading urllib3-2.0.4-py3-none-any.whl.metadata (6.6 kB)\n",
"Collecting certifi>=2017.4.17 (from requests>=2.20->openai)\n",
" Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)\n",
"Collecting attrs>=17.3.0 (from aiohttp->openai)\n",
" Using cached attrs-23.1.0-py3-none-any.whl (61 kB)\n",
"Collecting multidict<7.0,>=4.5 (from aiohttp->openai)\n",
" Using cached multidict-6.0.4-cp310-cp310-macosx_11_0_arm64.whl (29 kB)\n",
"Collecting async-timeout<5.0,>=4.0.0a3 (from aiohttp->openai)\n",
" Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
"Collecting yarl<2.0,>=1.0 (from aiohttp->openai)\n",
" Using cached yarl-1.9.2-cp310-cp310-macosx_11_0_arm64.whl (62 kB)\n",
"Collecting frozenlist>=1.1.1 (from aiohttp->openai)\n",
" Obtaining dependency information for frozenlist>=1.1.1 from https://files.pythonhosted.org/packages/67/6a/55a49da0fa373ac9aa49ccd5b6393ecc183e2a0904d9449ea3ee1163e0b1/frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.2 kB)\n",
"Collecting aiosignal>=1.1.2 (from aiohttp->openai)\n",
" Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
"Using cached openai-0.27.8-py3-none-any.whl (73 kB)\n",
"Using cached requests-2.31.0-py3-none-any.whl (62 kB)\n",
"Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl (343 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m343.9/343.9 kB\u001b[0m \u001b[31m11.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hUsing cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl (124 kB)\n",
"Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl (46 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hDownloading urllib3-2.0.4-py3-none-any.whl (123 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m123.9/123.9 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hInstalling collected packages: urllib3, tqdm, multidict, idna, frozenlist, charset-normalizer, certifi, attrs, async-timeout, yarl, requests, aiosignal, aiohttp, openai\n",
"Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 async-timeout-4.0.2 attrs-23.1.0 certifi-2023.5.7 charset-normalizer-3.2.0 frozenlist-1.4.0 idna-3.4 multidict-6.0.4 openai-0.27.8 requests-2.31.0 tqdm-4.65.0 urllib3-2.0.4 yarl-1.9.2\n",
"Note: you may need to restart the kernel to use updated packages.\n",
"Collecting chromadb\n",
" Obtaining dependency information for chromadb from https://files.pythonhosted.org/packages/47/b7/41d975f02818c965cdb8a119cab5a38cfb08e0c1abb18efebe9a373ea97b/chromadb-0.4.2-py3-none-any.whl.metadata\n",
" Downloading chromadb-0.4.2-py3-none-any.whl.metadata (6.9 kB)\n",
"Collecting pandas>=1.3 (from chromadb)\n",
" Obtaining dependency information for pandas>=1.3 from https://files.pythonhosted.org/packages/4a/f6/f620ca62365d83e663a255a41b08d2fc2eaf304e0b8b21bb6d62a7390fe3/pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata (18 kB)\n",
"Requirement already satisfied: requests>=2.28 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (2.31.0)\n",
"Collecting pydantic<2.0,>=1.9 (from chromadb)\n",
" Obtaining dependency information for pydantic<2.0,>=1.9 from https://files.pythonhosted.org/packages/79/3e/6b4d0fb2174beceac9a991ba8e67158b45c35faca9ea4545ae32d47096cd/pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata (148 kB)\n",
"Collecting chroma-hnswlib==0.7.1 (from chromadb)\n",
" Obtaining dependency information for chroma-hnswlib==0.7.1 from https://files.pythonhosted.org/packages/a5/d5/54947127f5cb2a1fcef40877fb3e6044495eec0a158ba0956babe4ab2a77/chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata\n",
" Using cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata (252 bytes)\n",
"Collecting fastapi<0.100.0,>=0.95.2 (from chromadb)\n",
" Obtaining dependency information for fastapi<0.100.0,>=0.95.2 from https://files.pythonhosted.org/packages/73/eb/03b691afa0b5ffa1e93ed34f97ec1e7855c758efbdcfb16c209af0b0506b/fastapi-0.99.1-py3-none-any.whl.metadata\n",
" Using cached fastapi-0.99.1-py3-none-any.whl.metadata (23 kB)\n",
"Collecting uvicorn[standard]>=0.18.3 (from chromadb)\n",
" Obtaining dependency information for uvicorn[standard]>=0.18.3 from https://files.pythonhosted.org/packages/5d/07/b9eac057f7efa56900640a233c1ed63db83568322c6bcbabe98f741d5289/uvicorn-0.23.1-py3-none-any.whl.metadata\n",
" Using cached uvicorn-0.23.1-py3-none-any.whl.metadata (6.2 kB)\n",
"Collecting numpy>=1.21.6 (from chromadb)\n",
" Obtaining dependency information for numpy>=1.21.6 from https://files.pythonhosted.org/packages/1b/cd/9e8313ffd849626c836fffd7881296a74f53a7739bd9ce7a6e22b1fc843b/numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.6 kB)\n",
"Collecting posthog>=2.4.0 (from chromadb)\n",
" Using cached posthog-3.0.1-py2.py3-none-any.whl (37 kB)\n",
"Requirement already satisfied: typing-extensions>=4.5.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.7.1)\n",
"Collecting pulsar-client>=3.1.0 (from chromadb)\n",
" Obtaining dependency information for pulsar-client>=3.1.0 from https://files.pythonhosted.org/packages/43/85/ab0455008ce3335a1c75a7c500fd8921ab166f34821fa67dc91ae9687a40/pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata\n",
" Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata (1.0 kB)\n",
"Collecting onnxruntime>=1.14.1 (from chromadb)\n",
" Obtaining dependency information for onnxruntime>=1.14.1 from https://files.pythonhosted.org/packages/cf/06/0c6e355b9ddbebc34d0e21bc5be1e4bd2c124ebd9030525838fa6e65eaa8/onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (4.0 kB)\n",
"Collecting tokenizers>=0.13.2 (from chromadb)\n",
" Using cached tokenizers-0.13.3-cp310-cp310-macosx_12_0_arm64.whl (3.9 MB)\n",
"Collecting pypika>=0.48.9 (from chromadb)\n",
" Using cached PyPika-0.48.9-py2.py3-none-any.whl\n",
"Requirement already satisfied: tqdm>=4.65.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.65.0)\n",
"Collecting overrides>=7.3.1 (from chromadb)\n",
" Using cached overrides-7.3.1-py3-none-any.whl (17 kB)\n",
"Collecting importlib-resources (from chromadb)\n",
" Obtaining dependency information for importlib-resources from https://files.pythonhosted.org/packages/29/d1/bed03eca30aa05aaf6e0873de091f9385c48705c4a607c2dfe3edbe543e8/importlib_resources-6.0.0-py3-none-any.whl.metadata\n",
" Using cached importlib_resources-6.0.0-py3-none-any.whl.metadata (4.2 kB)\n",
"Collecting starlette<0.28.0,>=0.27.0 (from fastapi<0.100.0,>=0.95.2->chromadb)\n",
" Obtaining dependency information for starlette<0.28.0,>=0.27.0 from https://files.pythonhosted.org/packages/58/f8/e2cca22387965584a409795913b774235752be4176d276714e15e1a58884/starlette-0.27.0-py3-none-any.whl.metadata\n",
" Using cached starlette-0.27.0-py3-none-any.whl.metadata (5.8 kB)\n",
"Collecting coloredlogs (from onnxruntime>=1.14.1->chromadb)\n",
" Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
"Collecting flatbuffers (from onnxruntime>=1.14.1->chromadb)\n",
" Obtaining dependency information for flatbuffers from https://files.pythonhosted.org/packages/6f/12/d5c79ee252793ffe845d58a913197bfa02ae9a0b5c9bc3dc4b58d477b9e7/flatbuffers-23.5.26-py2.py3-none-any.whl.metadata\n",
" Using cached flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)\n",
"Requirement already satisfied: packaging in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (23.1)\n",
"Collecting protobuf (from onnxruntime>=1.14.1->chromadb)\n",
" Obtaining dependency information for protobuf from https://files.pythonhosted.org/packages/cb/d3/a164038605494d49acc4f9cda1c0bc200b96382c53edd561387263bb181d/protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata\n",
" Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata (540 bytes)\n",
"Collecting sympy (from onnxruntime>=1.14.1->chromadb)\n",
" Using cached sympy-1.12-py3-none-any.whl (5.7 MB)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pandas>=1.3->chromadb) (2.8.2)\n",
"Collecting pytz>=2020.1 (from pandas>=1.3->chromadb)\n",
" Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)\n",
"Collecting tzdata>=2022.1 (from pandas>=1.3->chromadb)\n",
" Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)\n",
"Requirement already satisfied: six>=1.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb) (1.16.0)\n",
"Collecting monotonic>=1.5 (from posthog>=2.4.0->chromadb)\n",
" Using cached monotonic-1.6-py2.py3-none-any.whl (8.2 kB)\n",
"Collecting backoff>=1.10.0 (from posthog>=2.4.0->chromadb)\n",
" Using cached backoff-2.2.1-py3-none-any.whl (15 kB)\n",
"Requirement already satisfied: certifi in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pulsar-client>=3.1.0->chromadb) (2023.5.7)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.2.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (2.0.4)\n",
"Collecting click>=7.0 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/1a/70/e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/click-8.1.6-py3-none-any.whl.metadata\n",
" Using cached click-8.1.6-py3-none-any.whl.metadata (3.0 kB)\n",
"Collecting h11>=0.8 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Using cached h11-0.14.0-py3-none-any.whl (58 kB)\n",
"Collecting httptools>=0.5.0 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Obtaining dependency information for httptools>=0.5.0 from https://files.pythonhosted.org/packages/8f/71/d535e9f6967958d21b8fe1baeb7efb6304b86e8fcff44d0bda8690e0aec9/httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata\n",
" Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata (3.6 kB)\n",
"Collecting python-dotenv>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Using cached python_dotenv-1.0.0-py3-none-any.whl (19 kB)\n",
"Collecting pyyaml>=5.1 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Obtaining dependency information for pyyaml>=5.1 from https://files.pythonhosted.org/packages/5b/07/10033a403b23405a8fc48975444463d3d10a5c2736b7eb2550b07b367429/PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
" Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.1 kB)\n",
"Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Using cached uvloop-0.17.0-cp310-cp310-macosx_10_9_universal2.whl (2.1 MB)\n",
"Collecting watchfiles>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Using cached watchfiles-0.19.0-cp37-abi3-macosx_11_0_arm64.whl (388 kB)\n",
"Collecting websockets>=10.4 (from uvicorn[standard]>=0.18.3->chromadb)\n",
" Using cached websockets-11.0.3-cp310-cp310-macosx_11_0_arm64.whl (121 kB)\n",
"Collecting anyio<5,>=3.4.0 (from starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n",
" Obtaining dependency information for anyio<5,>=3.4.0 from https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl.metadata\n",
" Using cached anyio-3.7.1-py3-none-any.whl.metadata (4.7 kB)\n",
"Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime>=1.14.1->chromadb)\n",
" Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
"Collecting mpmath>=0.19 (from sympy->onnxruntime>=1.14.1->chromadb)\n",
" Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)\n",
"Collecting sniffio>=1.1 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n",
" Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)\n",
"Collecting exceptiongroup (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n",
" Obtaining dependency information for exceptiongroup from https://files.pythonhosted.org/packages/fe/17/f43b7c9ccf399d72038042ee72785c305f6c6fdc6231942f8ab99d995742/exceptiongroup-1.1.2-py3-none-any.whl.metadata\n",
" Using cached exceptiongroup-1.1.2-py3-none-any.whl.metadata (6.1 kB)\n",
"Downloading chromadb-0.4.2-py3-none-any.whl (399 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m399.3/399.3 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hUsing cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl (195 kB)\n",
"Using cached fastapi-0.99.1-py3-none-any.whl (58 kB)\n",
"Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)\n",
"Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl (6.1 MB)\n",
"Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)\n",
"Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl (10.8 MB)\n",
"Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl (2.5 MB)\n",
"Using cached importlib_resources-6.0.0-py3-none-any.whl (31 kB)\n",
"Using cached click-8.1.6-py3-none-any.whl (97 kB)\n",
"Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl (237 kB)\n",
"Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl (169 kB)\n",
"Using cached starlette-0.27.0-py3-none-any.whl (66 kB)\n",
"Using cached flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)\n",
"Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl (400 kB)\n",
"Using cached uvicorn-0.23.1-py3-none-any.whl (59 kB)\n",
"Using cached anyio-3.7.1-py3-none-any.whl (80 kB)\n",
"Using cached exceptiongroup-1.1.2-py3-none-any.whl (14 kB)\n",
"Installing collected packages: tokenizers, pytz, pypika, mpmath, monotonic, flatbuffers, websockets, uvloop, tzdata, sympy, sniffio, pyyaml, python-dotenv, pydantic, pulsar-client, protobuf, overrides, numpy, importlib-resources, humanfriendly, httptools, h11, exceptiongroup, click, backoff, uvicorn, posthog, pandas, coloredlogs, chroma-hnswlib, anyio, watchfiles, starlette, onnxruntime, fastapi, chromadb\n",
"Successfully installed anyio-3.7.1 backoff-2.2.1 chroma-hnswlib-0.7.1 chromadb-0.4.2 click-8.1.6 coloredlogs-15.0.1 exceptiongroup-1.1.2 fastapi-0.99.1 flatbuffers-23.5.26 h11-0.14.0 httptools-0.6.0 humanfriendly-10.0 importlib-resources-6.0.0 monotonic-1.6 mpmath-1.3.0 numpy-1.25.1 onnxruntime-1.15.1 overrides-7.3.1 pandas-2.0.3 posthog-3.0.1 protobuf-4.23.4 pulsar-client-3.2.0 pydantic-1.10.11 pypika-0.48.9 python-dotenv-1.0.0 pytz-2023.3 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 tzdata-2023.3 uvicorn-0.23.1 uvloop-0.17.0 watchfiles-0.19.0 websockets-11.0.3\n",
"Note: you may need to restart the kernel to use updated packages.\n",
"Collecting wget\n",
" Using cached wget-3.2.zip (10 kB)\n",
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25hBuilding wheels for collected packages: wget\n",
" Building wheel for wget (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=b2d83c5fcdeab398d0a4e9808a470bbf725fffea4a6130e731c6097b9561005b\n",
" Stored in directory: /Users/antontroynikov/Library/Caches/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769\n",
"Successfully built wget\n",
"Installing collected packages: wget\n",
"Successfully installed wget-3.2\n",
"Note: you may need to restart the kernel to use updated packages.\n",
"Requirement already satisfied: numpy in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (1.25.1)\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"# Make sure the OpenAI library is installed\n",
"%pip install openai\n",
"\n",
"# We'll need to install the Chroma client\n",
"%pip install chromadb\n",
"\n",
"# Install wget to pull zip file\n",
"%pip install wget\n",
"\n",
"# Install numpy for data manipulation\n",
"%pip install numpy"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "5be94df6",
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"import pandas as pd\n",
"import os\n",
"import wget\n",
"from ast import literal_eval\n",
"\n",
"# Chroma's client library for Python\n",
"import chromadb\n",
"\n",
"# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"\n",
"# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that we've prepared previous to this session."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "5dff8b55",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded.zip'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "21097972",
"metadata": {},
"outputs": [],
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "70bbd8ba",
"metadata": {},
"outputs": [],
"source": [
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "1721e45d",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "960b82af",
"metadata": {},
"outputs": [],
"source": [
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
"\n",
"# Set vector_id to be a string\n",
"article_df['vector_id'] = article_df['vector_id'].apply(str)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "a334ab8b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 25000 entries, 0 to 24999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 25000 non-null int64 \n",
" 1 url 25000 non-null object\n",
" 2 title 25000 non-null object\n",
" 3 text 25000 non-null object\n",
" 4 title_vector 25000 non-null object\n",
" 5 content_vector 25000 non-null object\n",
" 6 vector_id 25000 non-null object\n",
"dtypes: int64(1), object(6)\n",
"memory usage: 1.3+ MB\n"
]
}
],
"source": [
"article_df.info(show_counts=True)"
]
},
{
"cell_type": "markdown",
"id": "81bf5349",
"metadata": {},
"source": [
"# Chroma\n",
"\n",
"We'll index these embedded documents in a vector database and search them. The first option we'll look at is **Chroma**, an easy to use open-source self-hosted in-memory vector database, designed for working with embeddings together with LLMs. \n",
"\n",
"In this section, we will:\n",
"- Instantiate the Chroma client\n",
"- Create collections for each class of embedding \n",
"- Query each collection "
]
},
{
"cell_type": "markdown",
"id": "37d1f693",
"metadata": {},
"source": [
"### Instantiate the Chroma client\n",
"\n",
"Create the Chroma client. By default, Chroma is ephemeral and runs in memory. \n",
"However, you can easily set up a persistent configuration which writes to disk."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "159d9646",
"metadata": {},
"outputs": [],
"source": [
"chroma_client = chromadb.EphemeralClient() # Equivalent to chromadb.Client(), ephemeral.\n",
"# Uncomment for persistent client\n",
"# chroma_client = chromadb.PersistentClient()"
]
},
{
"cell_type": "markdown",
"id": "5cd61943",
"metadata": {},
"source": [
"### Create collections\n",
"\n",
"Chroma collections allow you to store and filter with arbitrary metadata, making it easy to query subsets of the embedded data. \n",
"\n",
"Chroma is already integrated with OpenAI's embedding functions. The best way to use them is on construction of a collection, as follows.\n",
"Alternatively, you can 'bring your own embeddings'. More information can be found [here](https://docs.trychroma.com/embeddings)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "ad2d1bce",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n",
"\n",
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
" print (\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print (\"OPENAI_API_KEY environment variable not found\")\n",
"\n",
"\n",
"embedding_function = OpenAIEmbeddingFunction(api_key=os.environ.get('OPENAI_API_KEY'), model_name=EMBEDDING_MODEL)\n",
"\n",
"wikipedia_content_collection = chroma_client.create_collection(name='wikipedia_content', embedding_function=embedding_function)\n",
"wikipedia_title_collection = chroma_client.create_collection(name='wikipedia_titles', embedding_function=embedding_function)"
]
},
{
"cell_type": "markdown",
"id": "02887b52",
"metadata": {},
"source": [
"### Populate the collections\n",
"\n",
"Chroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. \n",
"\n",
"For this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. "
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "84885fec",
"metadata": {},
"outputs": [],
"source": [
"# Add the content vectors\n",
"wikipedia_content_collection.add(\n",
" ids=article_df.vector_id.tolist(),\n",
" embeddings=article_df.content_vector.tolist(),\n",
")\n",
"\n",
"# Add the title vectors\n",
"wikipedia_title_collection.add(\n",
" ids=article_df.vector_id.tolist(),\n",
" embeddings=article_df.title_vector.tolist(),\n",
")"
]
},
{
"cell_type": "markdown",
"id": "79122c6b",
"metadata": {},
"source": [
"### Search the collections\n",
"\n",
"Chroma handles embedding queries for you if an embedding function is set, like in this example."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "273b8b4c",
"metadata": {},
"outputs": [],
"source": [
"def query_collection(collection, query, max_results, dataframe):\n",
" results = collection.query(query_texts=query, n_results=max_results, include=['distances']) \n",
" df = pd.DataFrame({\n",
" 'id':results['ids'][0], \n",
" 'score':results['distances'][0],\n",
" 'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],\n",
" 'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],\n",
" })\n",
" \n",
" return df"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "e84cf47f",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>score</th>\n",
" <th>title</th>\n",
" <th>content</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>23266</td>\n",
" <td>0.249646</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11777</th>\n",
" <td>15436</td>\n",
" <td>0.271688</td>\n",
" <td>Hellenistic art</td>\n",
" <td>The art of the Hellenistic time (from 400 B.C....</td>\n",
" </tr>\n",
" <tr>\n",
" <th>12178</th>\n",
" <td>23265</td>\n",
" <td>0.279306</td>\n",
" <td>Byzantine art</td>\n",
" <td>Byzantine art is a form of Christian Greek art...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>13215</th>\n",
" <td>11777</td>\n",
" <td>0.294415</td>\n",
" <td>Art film</td>\n",
" <td>Art films are a type of movie that is very dif...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>15436</th>\n",
" <td>22108</td>\n",
" <td>0.305937</td>\n",
" <td>Renaissance art</td>\n",
" <td>Many of the most famous and best-loved works o...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id score title \\\n",
"2 23266 0.249646 Art \n",
"11777 15436 0.271688 Hellenistic art \n",
"12178 23265 0.279306 Byzantine art \n",
"13215 11777 0.294415 Art film \n",
"15436 22108 0.305937 Renaissance art \n",
"\n",
" content \n",
"2 Art is a creative activity that expresses imag... \n",
"11777 The art of the Hellenistic time (from 400 B.C.... \n",
"12178 Byzantine art is a form of Christian Greek art... \n",
"13215 Art films are a type of movie that is very dif... \n",
"15436 Many of the most famous and best-loved works o... "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"title_query_result = query_collection(\n",
" collection=wikipedia_title_collection,\n",
" query=\"modern art in Europe\",\n",
" max_results=10,\n",
" dataframe=article_df\n",
")\n",
"title_query_result.head()"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "f4db910a",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>score</th>\n",
" <th>title</th>\n",
" <th>content</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>2923</th>\n",
" <td>13135</td>\n",
" <td>0.261328</td>\n",
" <td>1651</td>\n",
" <td>\\n\\nEvents \\n January 1 – Charles II crowned K...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3694</th>\n",
" <td>13571</td>\n",
" <td>0.277058</td>\n",
" <td>Stirling</td>\n",
" <td>Stirling () is a city in the middle of Scotlan...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6248</th>\n",
" <td>2923</td>\n",
" <td>0.294823</td>\n",
" <td>841</td>\n",
" <td>\\n\\nEvents \\n June 25: Battle of Fontenay – Lo...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6297</th>\n",
" <td>13568</td>\n",
" <td>0.300756</td>\n",
" <td>1746</td>\n",
" <td>\\n\\nEvents \\n January 8 – Bonnie Prince Charli...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11702</th>\n",
" <td>11708</td>\n",
" <td>0.307572</td>\n",
" <td>William Wallace</td>\n",
" <td>William Wallace was a Scottish knight who foug...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id score title \\\n",
"2923 13135 0.261328 1651 \n",
"3694 13571 0.277058 Stirling \n",
"6248 2923 0.294823 841 \n",
"6297 13568 0.300756 1746 \n",
"11702 11708 0.307572 William Wallace \n",
"\n",
" content \n",
"2923 \\n\\nEvents \\n January 1 – Charles II crowned K... \n",
"3694 Stirling () is a city in the middle of Scotlan... \n",
"6248 \\n\\nEvents \\n June 25: Battle of Fontenay – Lo... \n",
"6297 \\n\\nEvents \\n January 8 – Bonnie Prince Charli... \n",
"11702 William Wallace was a Scottish knight who foug... "
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"content_query_result = query_collection(\n",
" collection=wikipedia_content_collection,\n",
" query=\"Famous battles in Scottish history\",\n",
" max_results=10,\n",
" dataframe=article_df\n",
")\n",
"content_query_result.head()"
]
},
{
"cell_type": "markdown",
"id": "a03e7645",
"metadata": {},
"source": [
"Now that you've got a basic embeddings search running, you can [hop over to the Chroma docs](https://docs.trychroma.com/usage-guide#using-where-filters) to learn more about how to add filters to your query, update/delete data in your collections, and deploy Chroma."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "chroma_openai",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb
+994
View File
@@ -0,0 +1,994 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Robust Question Answering with Chroma and OpenAI \n",
"\n",
"This notebook guides you step-by-step through answering questions about a collection of data, using [Chroma](https://trychroma.com), an open-source embeddings database, along with OpenAI's [text embeddings](https://platform.openai.com/docs/guides/embeddings/use-cases) and [chat completion](https://platform.openai.com/docs/guides/chat) API's. \n",
"\n",
"Additionally, this notebook demonstrates some of the tradeoffs in making a question answering system more robust. As we shall see, *simple querying doesn't always create the best results*! \n",
"\n",
"## Question Answering with LLMs\n",
"\n",
"Large language models (LLMs) like OpenAI's ChatGPT can be used to answer questions about data that the model may not have been trained on, or have access to. For example;\n",
"\n",
"- Personal data like e-mails and notes\n",
"- Highly specialized data like archival or legal documents\n",
"- Newly created data like recent news stories\n",
"\n",
"In order to overcome this limitation, we can use a data store which is amenable to querying in natural language, just like the LLM itself. An embeddings store like Chroma represents documents as [embeddings](https://openai.com/blog/introducing-text-and-code-embeddings), alongside the documents themselves. \n",
"\n",
"By embedding a text query, Chroma can find relevant documents, which we can then pass to the LLM to answer our question. We'll show detailed examples and variants of this approach. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Setup and preliminaries\n",
"\n",
"First we make sure the python dependencies we need are installed. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"%pip install -qU openai chromadb pandas"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)\n",
"\n",
"You can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OpenAI client is ready\n"
]
}
],
"source": [
"import os\n",
"from openai import OpenAI\n",
"\n",
"# Uncomment the following line to set the environment variable in the notebook\n",
"# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n",
"\n",
"api_key = os.getenv(\"OPENAI_API_KEY\")\n",
"\n",
"if api_key:\n",
" client = OpenAI(api_key=api_key)\n",
" print(\"OpenAI client is ready\")\n",
"else:\n",
" print(\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Set the model for all API calls\n",
"OPENAI_MODEL = \"gpt-4o\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Dataset\n",
"\n",
"Throughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. \n",
"\n",
"Having the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>claim</th>\n",
" <th>evidence</th>\n",
" <th>cited_doc_ids</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>0-dimensional biomaterials show inductive prop...</td>\n",
" <td>{}</td>\n",
" <td>[31715818]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>3</td>\n",
" <td>1,000 genomes project enables mapping of genet...</td>\n",
" <td>{'14717500': [{'sentences': [2, 5], 'label': '...</td>\n",
" <td>[14717500]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>5</td>\n",
" <td>1/2000 in UK have abnormal PrP positivity.</td>\n",
" <td>{'13734012': [{'sentences': [4], 'label': 'SUP...</td>\n",
" <td>[13734012]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>13</td>\n",
" <td>5% of perinatal mortality is due to low birth ...</td>\n",
" <td>{}</td>\n",
" <td>[1606628]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>36</td>\n",
" <td>A deficiency of vitamin B12 increases blood le...</td>\n",
" <td>{}</td>\n",
" <td>[5152028, 11705328]</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id claim \\\n",
"0 1 0-dimensional biomaterials show inductive prop... \n",
"1 3 1,000 genomes project enables mapping of genet... \n",
"2 5 1/2000 in UK have abnormal PrP positivity. \n",
"3 13 5% of perinatal mortality is due to low birth ... \n",
"4 36 A deficiency of vitamin B12 increases blood le... \n",
"\n",
" evidence cited_doc_ids \n",
"0 {} [31715818] \n",
"1 {'14717500': [{'sentences': [2, 5], 'label': '... [14717500] \n",
"2 {'13734012': [{'sentences': [4], 'label': 'SUP... [13734012] \n",
"3 {} [1606628] \n",
"4 {} [5152028, 11705328] "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Load the claim dataset\n",
"import pandas as pd\n",
"\n",
"data_path = '../../data'\n",
"\n",
"claim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)\n",
"claim_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Just asking the model\n",
"\n",
"ChatGPT was trained on a large amount of scientific information. As a baseline, we'd like to understand what the model already knows without any further context. This will allow us to calibrate overall performance. \n",
"\n",
"We construct an appropriate prompt, with some example facts, then query the model with each claim in the dataset. We ask the model to assess a claim as 'True', 'False', or 'NEE' if there is not enough evidence one way or the other. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def build_prompt(claim):\n",
" return [\n",
" {\"role\": \"system\", \"content\": \"I will ask you to assess a scientific claim. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"},\n",
" {\"role\": \"user\", \"content\": f\"\"\"\n",
"Example:\n",
"\n",
"Claim:\n",
"0-dimensional biomaterials show inductive properties.\n",
"\n",
"Assessment:\n",
"False\n",
"\n",
"Claim:\n",
"1/2000 in UK have abnormal PrP positivity.\n",
"\n",
"Assessment:\n",
"True\n",
"\n",
"Claim:\n",
"Aspirin inhibits the production of PGE2.\n",
"\n",
"Assessment:\n",
"False\n",
"\n",
"End of examples. Assess the following claim:\n",
"\n",
"Claim:\n",
"{claim}\n",
"\n",
"Assessment:\n",
"\"\"\"}\n",
" ]\n",
"\n",
"\n",
"def assess_claims(claims):\n",
" responses = []\n",
" # Query the OpenAI API\n",
" for claim in claims:\n",
" response = client.chat.completions.create(\n",
" model=OPENAI_MODEL,\n",
" messages=build_prompt(claim),\n",
" max_tokens=3,\n",
" )\n",
" # Strip any punctuation or whitespace from the response\n",
" responses.append(response.choices[0].message.content.strip('., '))\n",
"\n",
" return responses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We sample 50 claims from the dataset"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# Let's take a look at 50 claims\n",
"samples = claim_df.sample(50)\n",
"\n",
"claims = samples['claim'].tolist()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"def get_groundtruth(evidence):\n",
" groundtruth = []\n",
" for e in evidence:\n",
" # Evidence is empty\n",
" if len(e) == 0:\n",
" groundtruth.append('NEE')\n",
" else:\n",
" # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT\n",
" if list(e.values())[0][0]['label'] == 'SUPPORT':\n",
" groundtruth.append('True')\n",
" else:\n",
" groundtruth.append('False')\n",
" return groundtruth"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"evidence = samples['evidence'].tolist()\n",
"groundtruth = get_groundtruth(evidence)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table. "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"def confusion_matrix(inferred, groundtruth):\n",
" assert len(inferred) == len(groundtruth)\n",
" confusion = {\n",
" 'True': {'True': 0, 'False': 0, 'NEE': 0},\n",
" 'False': {'True': 0, 'False': 0, 'NEE': 0},\n",
" 'NEE': {'True': 0, 'False': 0, 'NEE': 0},\n",
" }\n",
" for i, g in zip(inferred, groundtruth):\n",
" confusion[i][g] += 1\n",
"\n",
" # Pretty print the confusion matrix\n",
" print('\\tGroundtruth')\n",
" print('\\tTrue\\tFalse\\tNEE')\n",
" for i in confusion:\n",
" print(i, end='\\t')\n",
" for g in confusion[i]:\n",
" print(confusion[i][g], end='\\t')\n",
" print()\n",
"\n",
" return confusion"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We ask the model to directly assess the claims, without additional context. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\tGroundtruth\n",
"\tTrue\tFalse\tNEE\n",
"True\t9\t3\t15\t\n",
"False\t0\t3\t2\t\n",
"NEE\t8\t6\t4\t\n"
]
},
{
"data": {
"text/plain": [
"{'True': {'True': 9, 'False': 3, 'NEE': 15},\n",
" 'False': {'True': 0, 'False': 3, 'NEE': 2},\n",
" 'NEE': {'True': 8, 'False': 6, 'NEE': 4}}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gpt_inferred = assess_claims(claims)\n",
"confusion_matrix(gpt_inferred, groundtruth)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Results\n",
"\n",
"From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Adding context \n",
"\n",
"We now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we load the text corpus. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>doc_id</th>\n",
" <th>title</th>\n",
" <th>abstract</th>\n",
" <th>structured</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>4983</td>\n",
" <td>Microstructural development of human newborn c...</td>\n",
" <td>[Alterations of the architecture of cerebral w...</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>5836</td>\n",
" <td>Induction of myelodysplasia by myeloid-derived...</td>\n",
" <td>[Myelodysplastic syndromes (MDS) are age-depen...</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>7912</td>\n",
" <td>BC1 RNA, the transcript from a master gene for...</td>\n",
" <td>[ID elements are short interspersed elements (...</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>18670</td>\n",
" <td>The DNA Methylome of Human Peripheral Blood Mo...</td>\n",
" <td>[DNA methylation plays an important role in bi...</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>19238</td>\n",
" <td>The human myelin basic protein gene is include...</td>\n",
" <td>[Two human Golli (for gene expressed in the ol...</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" doc_id title \\\n",
"0 4983 Microstructural development of human newborn c... \n",
"1 5836 Induction of myelodysplasia by myeloid-derived... \n",
"2 7912 BC1 RNA, the transcript from a master gene for... \n",
"3 18670 The DNA Methylome of Human Peripheral Blood Mo... \n",
"4 19238 The human myelin basic protein gene is include... \n",
"\n",
" abstract structured \n",
"0 [Alterations of the architecture of cerebral w... False \n",
"1 [Myelodysplastic syndromes (MDS) are age-depen... False \n",
"2 [ID elements are short interspersed elements (... False \n",
"3 [DNA methylation plays an important role in bi... False \n",
"4 [Two human Golli (for gene expressed in the ol... False "
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Load the corpus into a dataframe\n",
"corpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)\n",
"corpus_df.head()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loading the corpus into Chroma\n",
"\n",
"The next step is to load the corpus into Chroma. Given an embedding function, Chroma will automatically handle embedding each document, and will store it alongside its text and metadata, making it simple to query."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We instantiate a (ephemeral) Chroma client, and create a collection for the SciFact title and abstract corpus. \n",
"Chroma can also be instantiated in a persisted configuration; learn more at the [Chroma docs](https://docs.trychroma.com/usage-guide?lang=py). "
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"import chromadb\n",
"from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n",
"\n",
"# We initialize an embedding function, and provide it to the collection.\n",
"embedding_function = OpenAIEmbeddingFunction(api_key=os.getenv(\"OPENAI_API_KEY\"))\n",
"\n",
"chroma_client = chromadb.Client() # Ephemeral by default\n",
"scifact_corpus_collection = chroma_client.create_collection(name='scifact_corpus', embedding_function=embedding_function)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we load the corpus into Chroma. Because this data loading is memory intensive, we recommend using a batched loading scheme in batches of 50-1000. For this example it should take just over one minute for the entire corpus. It's being embedded in the background, automatically, using the `embedding_function` we specified earlier."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"batch_size = 100\n",
"\n",
"for i in range(0, len(corpus_df), batch_size):\n",
" batch_df = corpus_df[i:i+batch_size]\n",
" scifact_corpus_collection.add(\n",
" ids=batch_df['doc_id'].apply(lambda x: str(x)).tolist(), # Chroma takes string IDs.\n",
" documents=(batch_df['title'] + '. ' + batch_df['abstract'].apply(lambda x: ' '.join(x))).to_list(), # We concatenate the title and abstract.\n",
" metadatas=[{\"structured\": structured} for structured in batch_df['structured'].to_list()] # We also store the metadata, though we don't use it in this example.\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Retrieving context\n",
"\n",
"Next we retrieve documents from the corpus which may be relevant to each claim in our sample. We want to provide these as context to the LLM for evaluating the claims. We retrieve the 3 most relevant documents for each claim, according to the embedding distance. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"claim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We create a new prompt, this time taking into account the additional context we retrieve from the corpus. "
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"def build_prompt_with_context(claim, context):\n",
" return [{'role': 'system', 'content': \"I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"},\n",
" {'role': 'user', 'content': f\"\"\"\"\n",
"The evidence is the following:\n",
"\n",
"{' '.join(context)}\n",
"\n",
"Assess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text.\n",
"\n",
"Claim:\n",
"{claim}\n",
"\n",
"Assessment:\n",
"\"\"\"}]\n",
"\n",
"\n",
"def assess_claims_with_context(claims, contexts):\n",
" responses = []\n",
" # Query the OpenAI API\n",
" for claim, context in zip(claims, contexts):\n",
" # If no evidence is provided, return NEE\n",
" if len(context) == 0:\n",
" responses.append('NEE')\n",
" continue\n",
" response = client.chat.completions.create(\n",
" model=OPENAI_MODEL,\n",
" messages=build_prompt_with_context(claim=claim, context=context),\n",
" max_tokens=3,\n",
" )\n",
" # Strip any punctuation or whitespace from the response\n",
" responses.append(response.choices[0].message.content.strip('., '))\n",
"\n",
" return responses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then ask the model to evaluate the claims with the retrieved context. "
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\tGroundtruth\n",
"\tTrue\tFalse\tNEE\n",
"True\t13\t1\t4\t\n",
"False\t1\t10\t2\t\n",
"NEE\t3\t1\t15\t\n"
]
},
{
"data": {
"text/plain": [
"{'True': {'True': 13, 'False': 1, 'NEE': 4},\n",
" 'False': {'True': 1, 'False': 10, 'NEE': 2},\n",
" 'NEE': {'True': 3, 'False': 1, 'NEE': 15}}"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])\n",
"confusion_matrix(gpt_with_context_evaluation, groundtruth)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Results\n",
"\n",
"We see that the model performs better overall, and is now significantly better at correctly identifying false claims. Additionally, most NEE cases are also correctly identified now.\n",
"\n",
"Taking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point. "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Filtering context on relevance\n",
"\n",
"Along with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. \n",
"\n",
"If, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence. "
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"def filter_query_result(query_result, distance_threshold=0.25):\n",
"# For each query result, retain only the documents whose distance is below the threshold\n",
" for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):\n",
" for i in range(len(ids)-1, -1, -1):\n",
" if distances[i] > distance_threshold:\n",
" ids.pop(i)\n",
" docs.pop(i)\n",
" distances.pop(i)\n",
" return query_result\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"filtered_claim_query_result = filter_query_result(claim_query_result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we assess the claims using this cleaner context. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\tGroundtruth\n",
"\tTrue\tFalse\tNEE\n",
"True\t9\t0\t1\t\n",
"False\t0\t7\t0\t\n",
"NEE\t8\t5\t20\t\n"
]
},
{
"data": {
"text/plain": [
"{'True': {'True': 9, 'False': 0, 'NEE': 1},\n",
" 'False': {'True': 0, 'False': 7, 'NEE': 0},\n",
" 'NEE': {'True': 8, 'False': 5, 'NEE': 20}}"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])\n",
"confusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Results\n",
"\n",
"\n",
"The model now assesses many fewer claims as True or False when there is not enough evidence present. However, it also is now much more cautious, tending to label most items as not enough evidence, biasing away from certainty. Most claims are now assessed as having not enough evidence, because a large fraction of them are filtered out by the distance threshold. It's possible to tune the distance threshold to find the optimal operating point, but this can be difficult, and is dataset and embedding model dependent. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Hypothetical Document Embeddings: Using hallucinations productively\n",
"\n",
"We want to be able to retrieve relevant documents, without retrieving less relevant ones which might confuse the model. One way to accomplish this is to improve the retrieval query. \n",
"\n",
"Until now, we have queried the dataset using _claims_ which are single sentence statements, while the corpus contains _abstracts_ describing a scientific paper. Intuitively, while these might be related, there are significant differences in their structure and meaning. These differences are encoded by the embedding model, and so influence the distances between the query and the most relevant results. \n",
"\n",
"We can overcome this by leveraging the power of LLMs to generate relevant text. While the facts might be hallucinated, the content and structure of the documents the models generate is more similar to the documents in our corpus, than the queries are. This could lead to better queries and hence better results. \n",
"\n",
"This approach is called [Hypothetical Document Embeddings (HyDE)](https://arxiv.org/abs/2212.10496), and has been shown to be quite good at the retrieval task. It should help us bring more relevant information into the context, without polluting it. \n",
"\n",
"TL;DR:\n",
"- you get much better matches when you embed whole abstracts rather than single sentences\n",
"- but claims are usually single sentences\n",
"- So HyDE shows that using GPT3 to expand claims into hallucinated abstracts and then searching based on those abstracts works (claims -> abstracts -> results) better than searching directly (claims -> results)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we use in-context examples to prompt the model to generate documents similar to what's in the corpus, for each claim we want to assess. "
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"def build_hallucination_prompt(claim):\n",
" return [{'role': 'system', 'content': \"\"\"I will ask you to write an abstract for a scientific paper which supports or refutes a given claim. It should be written in scientific language, include a title. Output only one abstract, then stop.\n",
"\n",
" An Example:\n",
"\n",
" Claim:\n",
" A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects.\n",
"\n",
" Abstract:\n",
" BACKGROUND The heritable haemoglobinopathy alpha(+)-thalassaemia is caused by the reduced synthesis of alpha-globin chains that form part of normal adult haemoglobin (Hb). Individuals homozygous for alpha(+)-thalassaemia have microcytosis and an increased erythrocyte count. Alpha(+)-thalassaemia homozygosity confers considerable protection against severe malaria, including severe malarial anaemia (SMA) (Hb concentration < 50 g/l), but does not influence parasite count. We tested the hypothesis that the erythrocyte indices associated with alpha(+)-thalassaemia homozygosity provide a haematological benefit during acute malaria.\n",
" METHODS AND FINDINGS Data from children living on the north coast of Papua New Guinea who had participated in a case-control study of the protection afforded by alpha(+)-thalassaemia against severe malaria were reanalysed to assess the genotype-specific reduction in erythrocyte count and Hb levels associated with acute malarial disease. We observed a reduction in median erythrocyte count of approximately 1.5 x 10(12)/l in all children with acute falciparum malaria relative to values in community children (p < 0.001). We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).\n",
" CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.\n",
"\n",
" End of example.\n",
"\n",
" \"\"\"}, {'role': 'user', 'content': f\"\"\"\"\n",
" Perform the task for the following claim.\n",
"\n",
" Claim:\n",
" {claim}\n",
"\n",
" Abstract:\n",
" \"\"\"}]\n",
"\n",
"\n",
"def hallucinate_evidence(claims):\n",
" responses = []\n",
" # Query the OpenAI API\n",
" for claim in claims:\n",
" response = client.chat.completions.create(\n",
" model=OPENAI_MODEL,\n",
" messages=build_hallucination_prompt(claim),\n",
" )\n",
" responses.append(response.choices[0].message.content)\n",
" return responses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We hallucinate a document for each claim.\n",
"\n",
"*NB: This can take a while, about 7m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. "
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"hallucinated_evidence = hallucinate_evidence(claims)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. "
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"hallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)\n",
"filtered_hallucinated_query_result = filter_query_result(hallucinated_query_result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We then ask the model to assess the claims, using the new context. "
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\tGroundtruth\n",
"\tTrue\tFalse\tNEE\n",
"True\t13\t0\t3\t\n",
"False\t1\t10\t1\t\n",
"NEE\t3\t2\t17\t\n"
]
},
{
"data": {
"text/plain": [
"{'True': {'True': 13, 'False': 0, 'NEE': 3},\n",
" 'False': {'True': 1, 'False': 10, 'NEE': 1},\n",
" 'NEE': {'True': 3, 'False': 2, 'NEE': 17}}"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])\n",
"confusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Results\n",
"\n",
"Combining HyDE with a simple distance threshold leads to a significant improvement. The model no longer biases assessing claims as True, nor toward their not being enough evidence. It also correctly assesses when there isn't enough evidence more often."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Conclusion\n",
"\n",
"Equipping LLMs with a context based on a corpus of documents is a powerful technique for bringing the general reasoning and natural language interactions of LLMs to your own data. However, it's important to know that naive query and retrieval may not produce the best possible results! Ultimately understanding the data will help get the most out of the retrieval based question-answering approach. \n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb
+836
View File
@@ -0,0 +1,836 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "Ol5OkztZqoAW"
},
"source": [
"# Question Answering with LangChain, Deep Lake, & OpenAI\n",
"\n",
"This notebook shows how to implement a question answering system with LangChain, [Deep Lake](https://activeloop.ai/) as a vector store and OpenAI embeddings. We will take the following steps to achieve this:\n",
"\n",
"1. Load a Deep Lake text dataset\n",
"2. Initialize a [Deep Lake vector store with LangChain](https://docs.activeloop.ai/tutorials/vector-store/deep-lake-vector-store-in-langchain)\n",
"3. Add text to the vector store\n",
"4. Run queries on the database\n",
"5. Done!\n",
"\n",
"You can also follow other tutorials such as question answering over any type of data (PDFs, json, csv, text): [chatting with any data](https://www.activeloop.ai/resources/data-chad-an-ai-app-with-lang-chain-deep-lake-to-chat-with-any-data/) stored in Deep Lake, [code understanding](https://www.activeloop.ai/resources/lang-chain-gpt-4-for-code-understanding-twitter-algorithm/), or [question answering over PDFs](https://www.activeloop.ai/resources/ultimate-guide-to-lang-chain-deep-lake-build-chat-gpt-to-answer-questions-on-your-financial-data/), or [recommending songs](https://www.activeloop.ai/resources/3-ways-to-build-a-recommendation-engine-for-songs-with-lang-chain/)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6uKh5KahrBs3"
},
"source": [
"## Install requirements\n",
"Let's install the following packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cPsdluAqqnRH"
},
"outputs": [],
"source": [
"!pip install deeplake langchain openai tiktoken"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IUm1NzURrGte"
},
"source": [
"## Authentication\n",
"Provide your OpenAI API key here:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Q_-OiwJzrJ8m",
"outputId": "b11b0d5c-cbd4-469d-95d1-fcd7149bd493"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"··········\n"
]
}
],
"source": [
"import getpass\n",
"import os\n",
"\n",
"os.environ['OPENAI_API_KEY'] = getpass.getpass()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ok-hgiotrLmS"
},
"source": [
"## Load a Deep Lake text dataset\n",
"We will use a 20000 sample subset of the [cohere-wikipedia-22](https://app.activeloop.ai/davitbun/cohere-wikipedia-22) dataset for this example."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "cIj5g4smrwOm",
"outputId": "6315bd53-8a2f-40ef-b2f5-2687c90b2231"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\\"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Opening dataset in read-only mode as you don't have write permissions.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"-"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/activeloop/cohere-wikipedia-22-sample\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"|"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"hub://activeloop/cohere-wikipedia-22-sample loaded successfully.\n",
"\n",
"Dataset(path='hub://activeloop/cohere-wikipedia-22-sample', read_only=True, tensors=['ids', 'metadata', 'text'])\n",
"\n",
" tensor htype shape dtype compression\n",
" ------- ------- ------- ------- ------- \n",
" ids text (20000, 1) str None \n",
" metadata json (20000, 1) str None \n",
" text text (20000, 1) str None \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r \r\r\r"
]
}
],
"source": [
"import deeplake\n",
"\n",
"ds = deeplake.load(\"hub://activeloop/cohere-wikipedia-22-sample\")\n",
"ds.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oY6FHqovHPfJ"
},
"source": [
"Let's take a look at a few samples:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "IWPYDrtUHPEr",
"outputId": "91e1b13e-abd0-4709-f65c-87986e90181a"
},
"outputs": [
{
"data": {
"text/plain": [
"['The 24-hour clock is a way of telling the time in which the day runs from midnight to midnight and is divided into 24 hours, numbered from 0 to 23. It does not use a.m. or p.m. This system is also referred to (only in the US and the English speaking parts of Canada) as military time or (only in the United Kingdom and now very rarely) as continental time. In some parts of the world, it is called railway time. Also, the international standard notation of time (ISO 8601) is based on this format.',\n",
" 'A time in the 24-hour clock is written in the form hours:minutes (for example, 01:23), or hours:minutes:seconds (01:23:45). Numbers under 10 have a zero in front (called a leading zero); e.g. 09:07. Under the 24-hour clock system, the day begins at midnight, 00:00, and the last minute of the day begins at 23:59 and ends at 24:00, which is identical to 00:00 of the following day. 12:00 can only be mid-day. Midnight is called 24:00 and is used to mean the end of the day and 00:00 is used to mean the beginning of the day. For example, you would say \"Tuesday at 24:00\" and \"Wednesday at 00:00\" to mean exactly the same time.',\n",
" 'However, the US military prefers not to say 24:00 - they do not like to have two names for the same thing, so they always say \"23:59\", which is one minute before midnight.']"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds[:3].text.data()[\"value\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JRFPjoDaGcSa"
},
"source": [
"## LangChain's Deep Lake vector store\n",
"Let's define a `dataset_path`, this is where your Deep Lake vector store will house the text embeddings."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "Klobw6_T257K"
},
"outputs": [],
"source": [
"dataset_path = 'wikipedia-embeddings-deeplake'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IW6BZubFGgu2"
},
"source": [
"We will setup OpenAI's `text-embedding-3-small` as our embedding function and initialize a Deep Lake vector store at `dataset_path`..."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ykE3HgSl5mcg",
"outputId": "dde4d6bb-6c82-473e-f37d-3f03a358ee8b"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\r\r\r\r"
]
}
],
"source": [
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
"from langchain.vectorstores import DeepLake\n",
"\n",
"embedding = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n",
"db = DeepLake(dataset_path, embedding=embedding, overwrite=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6mt2S1XpGj-D"
},
"source": [
"... and populate it with samples, one batch at a time, using the `add_texts` method."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 275,
"referenced_widgets": [
"30a05f9f55ae454ba75137634896e82a",
"0add33db728844a59c1ffa53e18fab98",
"26bf0f01ac414ab0b0da34971ba8cbdf",
"b595729257c34311a1c21b103a20bbb8",
"6a75dce7a6b84148a0515e30f116ee07",
"1dbe1466e8ba47b1898864ca5aa22f30",
"90c56b9af48d480b93c027032e44c9dd",
"06099626b6e34bf6acf06e53673d08e7",
"b8af7a2bffad44cea5264191b5079995",
"d397a65b169647588cf2eaf8342dde5e",
"2f9e6758a17441359021a6b66cff1dea"
]
},
"id": "hFJTvNGE53lS",
"outputId": "200e3808-1309-4520-9b42-6b59cfc506e6"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "30a05f9f55ae454ba75137634896e82a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"creating embeddings: 0%| | 0/1 [00:00<?, ?it/s]\u001b[A\n",
"creating embeddings: 100%|██████████| 1/1 [00:02<00:00, 2.11s/it]\n",
"\n",
"100%|██████████| 10/10 [00:00<00:00, 462.42it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset(path='wikipedia-embeddings-deeplake', tensors=['text', 'metadata', 'embedding', 'id'])\n",
"\n",
" tensor htype shape dtype compression\n",
" ------- ------- ------- ------- ------- \n",
" text text (10, 1) str None \n",
" metadata json (10, 1) str None \n",
" embedding embedding (10, 1536) float32 None \n",
" id text (10, 1) str None \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"\r\r"
]
}
],
"source": [
"from tqdm.auto import tqdm\n",
"\n",
"batch_size = 100\n",
"\n",
"nsamples = 10 # for testing. Replace with len(ds) to append everything\n",
"for i in tqdm(range(0, nsamples, batch_size)):\n",
" # find end of batch\n",
" i_end = min(nsamples, i + batch_size)\n",
"\n",
" batch = ds[i:i_end]\n",
" id_batch = batch.ids.data()[\"value\"]\n",
" text_batch = batch.text.data()[\"value\"]\n",
" meta_batch = batch.metadata.data()[\"value\"]\n",
"\n",
" db.add_texts(text_batch, metadatas=meta_batch, ids=id_batch)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4AwidW4MGnNH"
},
"source": [
"## Run user queries on the database\n",
"The underlying Deep Lake dataset object is accessible through `db.vectorstore.dataset`, and the data structure can be summarized using `db.vectorstore.summary()`, which shows 4 tensors with 10 samples:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "RSp2aF9nGrgj",
"outputId": "d8d370ff-52ee-42ed-ceb2-c48e8c4ada8f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset(path='wikipedia-embeddings-deeplake', tensors=['text', 'metadata', 'embedding', 'id'])\n",
"\n",
" tensor htype shape dtype compression\n",
" ------- ------- ------- ------- ------- \n",
" text text (10, 1) str None \n",
" metadata json (10, 1) str None \n",
" embedding embedding (10, 1536) float32 None \n",
" id text (10, 1) str None \n"
]
}
],
"source": [
"db.vectorstore.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NMcH9pRsGrUW"
},
"source": [
"We will now setup QA on our vector store with GPT-3.5-Turbo as our LLM."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "ywS3cL5oUHGL"
},
"outputs": [],
"source": [
"from langchain.chains import RetrievalQA\n",
"from langchain.chat_models import ChatOpenAI\n",
"\n",
"# Re-load the vector store in case it's no longer initialized\n",
"# db = DeepLake(dataset_path = dataset_path, embedding_function=embedding)\n",
"\n",
"qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(model='gpt-3.5-turbo'), chain_type=\"stuff\", retriever=db.as_retriever())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZysWCch7Gwf_"
},
"source": [
"Let's try running a prompt and check the output. Internally, this API performs an embedding search to find the most relevant data to feed into the LLM context."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 36
},
"id": "7VaBJgKrFOXu",
"outputId": "951fd6d7-d749-4fd9-9c7b-2ee4c422f65a"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'The military prefers not to say 24:00 because they do not like to have two names for the same thing. Instead, they always say \"23:59\", which is one minute before midnight.'"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query = 'Why does the military not say 24:00?'\n",
"qa.run(query)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xX0lLg9xG0Rk"
},
"source": [
"Et voila!"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"06099626b6e34bf6acf06e53673d08e7": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"0add33db728844a59c1ffa53e18fab98": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_1dbe1466e8ba47b1898864ca5aa22f30",
"placeholder": "​",
"style": "IPY_MODEL_90c56b9af48d480b93c027032e44c9dd",
"value": "100%"
}
},
"1dbe1466e8ba47b1898864ca5aa22f30": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"26bf0f01ac414ab0b0da34971ba8cbdf": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "FloatProgressModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "FloatProgressModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "ProgressView",
"bar_style": "success",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_06099626b6e34bf6acf06e53673d08e7",
"max": 1,
"min": 0,
"orientation": "horizontal",
"style": "IPY_MODEL_b8af7a2bffad44cea5264191b5079995",
"value": 1
}
},
"2f9e6758a17441359021a6b66cff1dea": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"30a05f9f55ae454ba75137634896e82a": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HBoxModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HBoxModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HBoxView",
"box_style": "",
"children": [
"IPY_MODEL_0add33db728844a59c1ffa53e18fab98",
"IPY_MODEL_26bf0f01ac414ab0b0da34971ba8cbdf",
"IPY_MODEL_b595729257c34311a1c21b103a20bbb8"
],
"layout": "IPY_MODEL_6a75dce7a6b84148a0515e30f116ee07"
}
},
"6a75dce7a6b84148a0515e30f116ee07": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"90c56b9af48d480b93c027032e44c9dd": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "DescriptionStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "DescriptionStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"description_width": ""
}
},
"b595729257c34311a1c21b103a20bbb8": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "HTMLModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "HTMLModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/controls",
"_view_module_version": "1.5.0",
"_view_name": "HTMLView",
"description": "",
"description_tooltip": null,
"layout": "IPY_MODEL_d397a65b169647588cf2eaf8342dde5e",
"placeholder": "​",
"style": "IPY_MODEL_2f9e6758a17441359021a6b66cff1dea",
"value": " 1/1 [00:04&lt;00:00, 4.45s/it]"
}
},
"b8af7a2bffad44cea5264191b5079995": {
"model_module": "@jupyter-widgets/controls",
"model_module_version": "1.5.0",
"model_name": "ProgressStyleModel",
"state": {
"_model_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_model_name": "ProgressStyleModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "StyleView",
"bar_color": null,
"description_width": ""
}
},
"d397a65b169647588cf2eaf8342dde5e": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "1.2.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "1.2.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "1.2.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"overflow_x": null,
"overflow_y": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
}
}
}
},
"nbformat": 4,
"nbformat_minor": 0
}
examples/vector_databases/elasticsearch/README.md
+31
View File
@@ -0,0 +1,31 @@
# Elasticsearch
Elasticsearch is a popular search/analytics engine and [vector database](https://www.elastic.co/elasticsearch/vector-database).
Elasticsearch offers an efficient way to create, store, and search vector embeddings at scale.
For technical details, refer to the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html).
The [`elasticsearch-labs`](https://github.com/elastic/elasticsearch-labs) repo contains executable Python notebooks, sample apps, and resources for testing out the Elastic platform.
## OpenAI cookbook notebooks 📒
Check out our notebooks in this repo for working with OpenAI, using Elasticsearch as your vector database.
### [Semantic search](https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-semantic-search.ipynb)
In this notebook you'll learn how to:
- Index the OpenAI Wikipedia embeddings dataset into Elasticsearch
- Encode a question with the `openai ada-02` model
- Perform a semantic search
<hr>
### [Retrieval augmented generation](https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb)
This notebooks builds on the semantic search notebook by:
- Selecting the top hit from a semantic search
- Sending that result to the OpenAI [Chat Completions](https://platform.openai.com/docs/guides/gpt/chat-completions-api) API endpoint for retrieval augmented generation (RAG)
examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb
+892
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/elasticsearch/elasticsearch-semantic-search.ipynb
+812
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb
+866
View File
@@ -0,0 +1,866 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using Hologres as a vector database for OpenAI embeddings\n",
"\n",
"This notebook guides you step by step on using Hologres as a vector database for OpenAI embeddings.\n",
"\n",
"This notebook presents an end-to-end process of:\n",
"1. Using precomputed embeddings created by OpenAI API.\n",
"2. Storing the embeddings in a cloud instance of Hologres.\n",
"3. Converting raw text query to an embedding with OpenAI API.\n",
"4. Using Hologres to perform the nearest neighbour search in the created collection.\n",
"5. Provide large language models with the search results as context in prompt engineering\n",
"\n",
"### What is Hologres\n",
"\n",
"[Hologres](https://www.alibabacloud.com/help/en/hologres/latest/what-is-hologres) is a unified real-time data warehousing service developed by Alibaba Cloud. You can use Hologres to write, update, process, and analyze large amounts of data in real time. Hologres supports standard SQL syntax, is compatible with PostgreSQL, and supports most PostgreSQL functions. Hologres supports online analytical processing (OLAP) and ad hoc analysis for up to petabytes of data, and provides high-concurrency and low-latency online data services. Hologres supports fine-grained isolation of multiple workloads and enterprise-level security capabilities. Hologres is deeply integrated with MaxCompute, Realtime Compute for Apache Flink, and DataWorks, and provides full-stack online and offline data warehousing solutions for enterprises.\n",
"\n",
"Hologres provides vector database functionality by adopting [Proxima](https://www.alibabacloud.com/help/en/hologres/latest/vector-processing).\n",
"\n",
"Proxima is a high-performance software library developed by Alibaba DAMO Academy. It allows you to search for the nearest neighbors of vectors. Proxima provides higher stability and performance than similar open source software such as Facebook AI Similarity Search (Faiss). Proxima provides basic modules that have leading performance and effects in the industry and allows you to search for similar images, videos, or human faces. Hologres is deeply integrated with Proxima to provide a high-performance vector search service.\n",
"\n",
"### Deployment options\n",
"\n",
"- [Click here](https://www.alibabacloud.com/product/hologres) to fast deploy [Hologres data warehouse](https://www.alibabacloud.com/help/en/hologres/latest/getting-started).\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"For the purposes of this exercise we need to prepare a couple of things:\n",
"\n",
"1. Hologres cloud server instance.\n",
"2. The 'psycopg2-binary' library to interact with the vector database. Any other postgresql client library is ok.\n",
"3. An [OpenAI API key](https://beta.openai.com/account/api-keys).\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We might validate if the server was launched successfully by running a simple curl command:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install requirements\n",
"\n",
"This notebook obviously requires the `openai` and `psycopg2-binary` packages, but there are also some other additional libraries we will use. The following command installs them all:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.718972Z",
"start_time": "2023-02-16T12:04:30.434820Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"! pip install openai psycopg2-binary pandas wget"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare your OpenAI API key\n",
"\n",
"The OpenAI API key is used for vectorization of the documents and queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.730338Z",
"start_time": "2023-02-16T12:05:05.723351Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print(\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print(\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to Hologres\n",
"First add it to your environment variables. or you can just change the \"psycopg2.connect\" parameters below\n",
"\n",
"Connecting to a running instance of Hologres server is easy with the official Python library:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.827143Z",
"start_time": "2023-02-16T12:05:05.733771Z"
}
},
"outputs": [],
"source": [
"import os\n",
"import psycopg2\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"PGHOST\"] = \"your_host\"\n",
"# os.environ[\"PGPORT\"] \"5432\"),\n",
"# os.environ[\"PGDATABASE\"] \"postgres\"),\n",
"# os.environ[\"PGUSER\"] \"user\"),\n",
"# os.environ[\"PGPASSWORD\"] \"password\"),\n",
"\n",
"connection = psycopg2.connect(\n",
" host=os.environ.get(\"PGHOST\", \"localhost\"),\n",
" port=os.environ.get(\"PGPORT\", \"5432\"),\n",
" database=os.environ.get(\"PGDATABASE\", \"postgres\"),\n",
" user=os.environ.get(\"PGUSER\", \"user\"),\n",
" password=os.environ.get(\"PGPASSWORD\", \"password\")\n",
")\n",
"connection.set_session(autocommit=True)\n",
"\n",
"# Create a new cursor object\n",
"cursor = connection.cursor()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can test the connection by running any available method:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.848488Z",
"start_time": "2023-02-16T12:05:06.832612Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Connection successful!\n"
]
}
],
"source": [
"\n",
"# Execute a simple query to test the connection\n",
"cursor.execute(\"SELECT 1;\")\n",
"result = cursor.fetchone()\n",
"\n",
"# Check the query result\n",
"if result == (1,):\n",
" print(\"Connection successful!\")\n",
"else:\n",
" print(\"Connection failed.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:37.371951Z",
"start_time": "2023-02-16T12:05:06.851634Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The downloaded file has to be then extracted:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:06:01.538851Z",
"start_time": "2023-02-16T12:05:37.376042Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n"
]
}
],
"source": [
"import zipfile\n",
"import os\n",
"import re\n",
"import tempfile\n",
"\n",
"current_directory = os.getcwd()\n",
"zip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\n",
"output_directory = os.path.join(current_directory, \"../../data\")\n",
"\n",
"with zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(output_directory)\n",
"\n",
"\n",
"# check the csv file exist\n",
"file_name = \"vector_database_wikipedia_articles_embedded.csv\"\n",
"data_directory = os.path.join(current_directory, \"../../data\")\n",
"file_path = os.path.join(data_directory, file_name)\n",
"\n",
"\n",
"if os.path.exists(file_path):\n",
" print(f\"The file {file_name} exists in the data directory.\")\n",
"else:\n",
" print(f\"The file {file_name} does not exist in the data directory.\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Archive: vector_database_wikipedia_articles_embedded.zip\n",
"-rw-r--r--@ 1 geng staff 1.7G Jan 31 01:19 vector_database_wikipedia_articles_embedded.csv\n"
]
}
],
"source": [
"!unzip -n vector_database_wikipedia_articles_embedded.zip\n",
"!ls -lh vector_database_wikipedia_articles_embedded.csv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Take a look at the data."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24995</th>\n",
" <td>98295</td>\n",
" <td>https://simple.wikipedia.org/wiki/Geneva</td>\n",
" <td>Geneva</td>\n",
" <td>Geneva (, , , , ) is the second biggest cit...</td>\n",
" <td>[-0.015773078426718712, 0.01737344264984131, 0...</td>\n",
" <td>[0.008000412955880165, 0.02008531428873539, 0....</td>\n",
" <td>24995</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24996</th>\n",
" <td>98316</td>\n",
" <td>https://simple.wikipedia.org/wiki/Concubinage</td>\n",
" <td>Concubinage</td>\n",
" <td>Concubinage is the state of a woman in a relat...</td>\n",
" <td>[-0.00519518880173564, 0.005898841191083193, 0...</td>\n",
" <td>[-0.01736736111342907, -0.002740012714639306, ...</td>\n",
" <td>24996</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24997</th>\n",
" <td>98318</td>\n",
" <td>https://simple.wikipedia.org/wiki/Mistress%20%...</td>\n",
" <td>Mistress (lover)</td>\n",
" <td>A mistress is a man's long term female sexual ...</td>\n",
" <td>[-0.023164259269833565, -0.02052430994808674, ...</td>\n",
" <td>[-0.017878392711281776, -0.0004517830966506153...</td>\n",
" <td>24997</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24998</th>\n",
" <td>98326</td>\n",
" <td>https://simple.wikipedia.org/wiki/Eastern%20Front</td>\n",
" <td>Eastern Front</td>\n",
" <td>Eastern Front can be one of the following:\\n\\n...</td>\n",
" <td>[-0.00681863259524107, 0.002171179046854377, 8...</td>\n",
" <td>[-0.0019235472427681088, -0.004023272544145584...</td>\n",
" <td>24998</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24999</th>\n",
" <td>98327</td>\n",
" <td>https://simple.wikipedia.org/wiki/Italian%20Ca...</td>\n",
" <td>Italian Campaign</td>\n",
" <td>Italian Campaign can mean the following:\\n\\nTh...</td>\n",
" <td>[-0.014151256531476974, -0.008553029969334602,...</td>\n",
" <td>[-0.011758845299482346, -0.01346028596162796, ...</td>\n",
" <td>24999</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>25000 rows × 7 columns</p>\n",
"</div>"
],
"text/plain": [
" id url \n",
"0 1 https://simple.wikipedia.org/wiki/April \\\n",
"1 2 https://simple.wikipedia.org/wiki/August \n",
"2 6 https://simple.wikipedia.org/wiki/Art \n",
"3 8 https://simple.wikipedia.org/wiki/A \n",
"4 9 https://simple.wikipedia.org/wiki/Air \n",
"... ... ... \n",
"24995 98295 https://simple.wikipedia.org/wiki/Geneva \n",
"24996 98316 https://simple.wikipedia.org/wiki/Concubinage \n",
"24997 98318 https://simple.wikipedia.org/wiki/Mistress%20%... \n",
"24998 98326 https://simple.wikipedia.org/wiki/Eastern%20Front \n",
"24999 98327 https://simple.wikipedia.org/wiki/Italian%20Ca... \n",
"\n",
" title text \n",
"0 April April is the fourth month of the year in the J... \\\n",
"1 August August (Aug.) is the eighth month of the year ... \n",
"2 Art Art is a creative activity that expresses imag... \n",
"3 A A or a is the first letter of the English alph... \n",
"4 Air Air refers to the Earth's atmosphere. Air is a... \n",
"... ... ... \n",
"24995 Geneva Geneva (, , , , ) is the second biggest cit... \n",
"24996 Concubinage Concubinage is the state of a woman in a relat... \n",
"24997 Mistress (lover) A mistress is a man's long term female sexual ... \n",
"24998 Eastern Front Eastern Front can be one of the following:\\n\\n... \n",
"24999 Italian Campaign Italian Campaign can mean the following:\\n\\nTh... \n",
"\n",
" title_vector \n",
"0 [0.001009464613161981, -0.020700545981526375, ... \\\n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"... ... \n",
"24995 [-0.015773078426718712, 0.01737344264984131, 0... \n",
"24996 [-0.00519518880173564, 0.005898841191083193, 0... \n",
"24997 [-0.023164259269833565, -0.02052430994808674, ... \n",
"24998 [-0.00681863259524107, 0.002171179046854377, 8... \n",
"24999 [-0.014151256531476974, -0.008553029969334602,... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 \n",
"... ... ... \n",
"24995 [0.008000412955880165, 0.02008531428873539, 0.... 24995 \n",
"24996 [-0.01736736111342907, -0.002740012714639306, ... 24996 \n",
"24997 [-0.017878392711281776, -0.0004517830966506153... 24997 \n",
"24998 [-0.0019235472427681088, -0.004023272544145584... 24998 \n",
"24999 [-0.011758845299482346, -0.01346028596162796, ... 24999 \n",
"\n",
"[25000 rows x 7 columns]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas, json\n",
"data = pandas.read_csv('../../data/vector_database_wikipedia_articles_embedded.csv')\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1536 1536\n"
]
}
],
"source": [
"title_vector_length = len(json.loads(data['title_vector'].iloc[0]))\n",
"content_vector_length = len(json.loads(data['content_vector'].iloc[0]))\n",
"\n",
"print(title_vector_length, content_vector_length)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create table and proxima vector index"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Hologres stores data in __tables__ where each object is described by at least one vector. Our table will be called **articles** and each object will be described by both **title** and **content** vectors.\n",
"\n",
"We will start with creating a table and create proxima indexes on both **title** and **content**, and then we will fill it with our precomputed embeddings."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"cursor.execute('CREATE EXTENSION IF NOT EXISTS proxima;')\n",
"create_proxima_table_sql = '''\n",
"BEGIN;\n",
"DROP TABLE IF EXISTS articles;\n",
"CREATE TABLE articles (\n",
" id INT PRIMARY KEY NOT NULL,\n",
" url TEXT,\n",
" title TEXT,\n",
" content TEXT,\n",
" title_vector float4[] check(\n",
" array_ndims(title_vector) = 1 and \n",
" array_length(title_vector, 1) = 1536\n",
" ), -- define the vectors\n",
" content_vector float4[] check(\n",
" array_ndims(content_vector) = 1 and \n",
" array_length(content_vector, 1) = 1536\n",
" ),\n",
" vector_id INT\n",
");\n",
"\n",
"-- Create indexes for the vector fields.\n",
"call set_table_property(\n",
" 'articles',\n",
" 'proxima_vectors', \n",
" '{\n",
" \"title_vector\":{\"algorithm\":\"Graph\",\"distance_method\":\"Euclidean\",\"builder_params\":{\"min_flush_proxima_row_count\" : 10}},\n",
" \"content_vector\":{\"algorithm\":\"Graph\",\"distance_method\":\"Euclidean\",\"builder_params\":{\"min_flush_proxima_row_count\" : 10}}\n",
" }'\n",
"); \n",
"\n",
"COMMIT;\n",
"'''\n",
"\n",
"# Execute the SQL statements (will autocommit)\n",
"cursor.execute(create_proxima_table_sql)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Upload data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's upload the data to the Hologres cloud instance using [COPY statement](https://www.alibabacloud.com/help/en/hologres/latest/use-the-copy-statement-to-import-or-export-data). This might take 5-10 minutes according to the network bandwidth."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"\n",
"# Path to the unzipped CSV file\n",
"csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n",
"\n",
"# In SQL, arrays are surrounded by {}, rather than []\n",
"def process_file(file_path):\n",
" with open(file_path, 'r') as file:\n",
" for line in file:\n",
" # Replace '[' with '{' and ']' with '}'\n",
" modified_line = line.replace('[', '{').replace(']', '}')\n",
" yield modified_line\n",
"\n",
"# Create a StringIO object to store the modified lines\n",
"modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n",
"\n",
"# Create the COPY command for the copy_expert method\n",
"copy_command = '''\n",
"COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\n",
"FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n",
"'''\n",
"\n",
"# Execute the COPY command using the copy_expert method\n",
"cursor.copy_expert(copy_command, modified_lines)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The proxima index will be built in the background. We can do searching during this period but the query will be slow without the vector index. Use this command to wait for finish building the index."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"cursor.execute('vacuum articles;')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.675202Z",
"start_time": "2023-02-16T12:30:40.655654Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Count:25000\n"
]
}
],
"source": [
"# Check the collection size to make sure all the points have been stored\n",
"count_sql = \"select count(*) from articles;\"\n",
"cursor.execute(count_sql)\n",
"result = cursor.fetchone()\n",
"print(f\"Count:{result[0]}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search data\n",
"\n",
"Once the data is uploaded we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:38.024370Z",
"start_time": "2023-02-16T12:30:37.712816Z"
}
},
"outputs": [],
"source": [
"import openai\n",
"def query_knn(query, table_name, vector_name=\"title_vector\", top_k=20):\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.Embedding.create(\n",
" input=query,\n",
" model=\"text-embedding-3-small\",\n",
" )[\"data\"][0][\"embedding\"]\n",
"\n",
" # Convert the embedded_query to PostgreSQL compatible format\n",
" embedded_query_pg = \"{\" + \",\".join(map(str, embedded_query)) + \"}\"\n",
"\n",
" # Create SQL query\n",
" query_sql = f\"\"\"\n",
" SELECT id, url, title, pm_approx_euclidean_distance({vector_name},'{embedded_query_pg}'::float4[]) AS distance\n",
" FROM {table_name}\n",
" ORDER BY distance\n",
" LIMIT {top_k};\n",
" \"\"\"\n",
" # Execute the query\n",
" cursor.execute(query_sql)\n",
" results = cursor.fetchall()\n",
"\n",
" return results"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:39.379566Z",
"start_time": "2023-02-16T12:30:38.031041Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art (Score: 0.501)\n",
"2. Western Europe (Score: 0.485)\n",
"3. Renaissance art (Score: 0.479)\n",
"4. Pop art (Score: 0.472)\n",
"5. Northern Europe (Score: 0.461)\n",
"6. Hellenistic art (Score: 0.458)\n",
"7. Modernist literature (Score: 0.447)\n",
"8. Art film (Score: 0.44)\n",
"9. Central Europe (Score: 0.439)\n",
"10. Art (Score: 0.437)\n",
"11. European (Score: 0.437)\n",
"12. Byzantine art (Score: 0.436)\n",
"13. Postmodernism (Score: 0.435)\n",
"14. Eastern Europe (Score: 0.433)\n",
"15. Cubism (Score: 0.433)\n",
"16. Europe (Score: 0.432)\n",
"17. Impressionism (Score: 0.432)\n",
"18. Bauhaus (Score: 0.431)\n",
"19. Surrealism (Score: 0.429)\n",
"20. Expressionism (Score: 0.429)\n"
]
}
],
"source": [
"query_results = query_knn(\"modern art in Europe\", \"Articles\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.652676Z",
"start_time": "2023-02-16T12:30:39.382555Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn (Score: 0.489)\n",
"2. Wars of Scottish Independence (Score: 0.474)\n",
"3. 1651 (Score: 0.457)\n",
"4. First War of Scottish Independence (Score: 0.452)\n",
"5. Robert I of Scotland (Score: 0.445)\n",
"6. 841 (Score: 0.441)\n",
"7. 1716 (Score: 0.441)\n",
"8. 1314 (Score: 0.429)\n",
"9. 1263 (Score: 0.428)\n",
"10. William Wallace (Score: 0.426)\n",
"11. Stirling (Score: 0.419)\n",
"12. 1306 (Score: 0.419)\n",
"13. 1746 (Score: 0.418)\n",
"14. 1040s (Score: 0.414)\n",
"15. 1106 (Score: 0.412)\n",
"16. 1304 (Score: 0.411)\n",
"17. David II of Scotland (Score: 0.408)\n",
"18. Braveheart (Score: 0.407)\n",
"19. 1124 (Score: 0.406)\n",
"20. July 27 (Score: 0.405)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_results = query_knn(\"Famous battles in Scottish history\", \"Articles\", \"content_vector\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb
+1
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/kusto/README.md
+34
View File
@@ -0,0 +1,34 @@
# Kusto as a Vector database
[Azure Data Explorer aka Kusto](https://azure.microsoft.com/en-us/products/data-explorer) is a cloud-based data analytics service that enables users to perform advanced analytics on large datasets in real-time. It is particularly well-suited for handling large volumes of data, making it an excellent choice for storing and searching vectors.
Kusto supports a special data type called dynamic, which can store unstructured data such as arrays and properties bag. [Dynamic data type](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/scalar-data-types/dynamic) is perfect for storing vector values. You can further augment the vector value by storing metadata related to the original object as separate columns in your table.
Kusto also supports in-built function [series_cosine_similarity_fl](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/functions-library/series-cosine-similarity-fl) to perform vector similarity searches.
[Get started](https://aka.ms/kustofree) with Kusto for free.
![Kusto_Vector](./images/kusto_vector_db.png)
## Getting started with Kusto and Open AI embedding
### Demo Scenario
![Wiki_embeddings](./images/wiki_embeddings.png)
![semantic_search_flow](./images/semantic_search_user_flow.png)
If you’d like to try this demo, please follow the instructions in the [Notebook](Getting_started_with_kusto_and_openai_embeddings.ipynb).
It will allow you to -
1. Use precomputed embeddings created by OpenAI API.
2. Store the embeddings in Kusto.
3. Convert raw text query to an embedding with OpenAI API.
4. Use Kusto to perform cosine similarity search in the stored embeddings.
examples/vector_databases/kusto/images/kusto_vector_db.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 63 KiB

examples/vector_databases/kusto/images/semantic_search_user_flow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 33 KiB

examples/vector_databases/kusto/images/wiki_embeddings.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

examples/vector_databases/milvus/Filtered_search_with_Milvus_and_OpenAI.ipynb
+444
View File
@@ -0,0 +1,444 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Filtered Search with Milvus and OpenAI\n",
"### Finding your next movie\n",
"\n",
"In this notebook we will be going over generating embeddings of movie descriptions with OpenAI and using those embeddings within Milvus to find relevant movies. To narrow our search results and try something new, we are going to be using filtering to do metadata searches. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 8 thousand movie entries.\n",
"\n",
"Lets begin by first downloading the required libraries for this notebook:\n",
"- `openai` is used for communicating with the OpenAI embedding service\n",
"- `pymilvus` is used for communicating with the Milvus server\n",
"- `datasets` is used for downloading the dataset\n",
"- `tqdm` is used for the progress bars\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install openai pymilvus datasets tqdm"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"With the required packages installed we can get started. Lets begin by launching the Milvus service. The file being run is the `docker-compose.yaml` found in the folder of this file. This command launches a Milvus standalone instance which we will use for this test. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"E0317 14:06:38.344884000 140704629352640 fork_posix.cc:76] Other threads are currently calling into gRPC, skipping fork() handlers\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/0\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠋ Container milvus-etcd Creating 0.0s\n",
"\u001b[0m\u001b[37m ⠋ Container milvus-minio Creating 0.0s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 1/3\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠙ Container milvus-etcd Creating 0.1s\n",
"\u001b[0m\u001b[37m ⠙ Container milvus-minio Creating 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/3\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.2s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.2s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.3s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.3s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.4s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.4s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.5s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.5s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.6s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.6s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.7s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.8s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.9s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.0s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.0s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.0s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.0s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.0s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.2s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.0s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.0s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.4s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.0s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.5s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 4/4\u001b[0m\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.0s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Started 1.6s\n",
"\u001b[0m\u001b[?25h"
]
}
],
"source": [
"! docker compose up -d"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"With Milvus running we can setup our global variables:\n",
"- HOST: The Milvus host address\n",
"- PORT: The Milvus port number\n",
"- COLLECTION_NAME: What to name the collection within Milvus\n",
"- DIMENSION: The dimension of the embeddings\n",
"- OPENAI_ENGINE: Which embedding model to use\n",
"- openai.api_key: Your OpenAI account key\n",
"- INDEX_PARAM: The index settings to use for the collection\n",
"- QUERY_PARAM: The search parameters to use\n",
"- BATCH_SIZE: How many movies to embed and insert at once"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"HOST = 'localhost'\n",
"PORT = 19530\n",
"COLLECTION_NAME = 'movie_search'\n",
"DIMENSION = 1536\n",
"OPENAI_ENGINE = 'text-embedding-3-small'\n",
"openai.api_key = 'sk-your_key'\n",
"\n",
"INDEX_PARAM = {\n",
" 'metric_type':'L2',\n",
" 'index_type':\"HNSW\",\n",
" 'params':{'M': 8, 'efConstruction': 64}\n",
"}\n",
"\n",
"QUERY_PARAM = {\n",
" \"metric_type\": \"L2\",\n",
" \"params\": {\"ef\": 64},\n",
"}\n",
"\n",
"BATCH_SIZE = 1000"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n",
"\n",
"# Connect to Milvus Database\n",
"connections.connect(host=HOST, port=PORT)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"# Remove collection if it already exists\n",
"if utility.has_collection(COLLECTION_NAME):\n",
" utility.drop_collection(COLLECTION_NAME)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"# Create collection which includes the id, title, and embedding.\n",
"fields = [\n",
" FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n",
" FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='type', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='release_year', dtype=DataType.INT64),\n",
" FieldSchema(name='rating', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n",
"]\n",
"schema = CollectionSchema(fields=fields)\n",
"collection = Collection(name=COLLECTION_NAME, schema=schema)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"# Create the index on the collection and load it.\n",
"collection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\n",
"collection.load()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dataset\n",
"With Milvus up and running we can begin grabbing our data. Hugging Face Datasets is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Milvus along with its title, type, release_year and rating."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Found cached dataset csv (/Users/filiphaltmayer/.cache/huggingface/datasets/hugginglearners___csv/hugginglearners--netflix-shows-03475319fc65a05a/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317)\n"
]
}
],
"source": [
"import datasets\n",
"\n",
"# Download the dataset \n",
"dataset = datasets.load_dataset('hugginglearners/netflix-shows', split='train')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Insert the Data\n",
"Now that we have our data on our machine we can begin embedding it and inserting it into Milvus. The embedding function takes in text and returns the embeddings in a list format. "
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"# Simple function that converts the texts to embeddings\n",
"def embed(texts):\n",
" embeddings = openai.Embedding.create(\n",
" input=texts,\n",
" engine=OPENAI_ENGINE\n",
" )\n",
" return [x['embedding'] for x in embeddings['data']]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaning batch if it exists. "
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 8807/8807 [00:31<00:00, 276.82it/s]\n"
]
}
],
"source": [
"from tqdm import tqdm\n",
"\n",
"data = [\n",
" [], # title\n",
" [], # type\n",
" [], # release_year\n",
" [], # rating\n",
" [], # description\n",
"]\n",
"\n",
"# Embed and insert in batches\n",
"for i in tqdm(range(0, len(dataset))):\n",
" data[0].append(dataset[i]['title'] or '')\n",
" data[1].append(dataset[i]['type'] or '')\n",
" data[2].append(dataset[i]['release_year'] or -1)\n",
" data[3].append(dataset[i]['rating'] or '')\n",
" data[4].append(dataset[i]['description'] or '')\n",
" if len(data[0]) % BATCH_SIZE == 0:\n",
" data.append(embed(data[4]))\n",
" collection.insert(data)\n",
" data = [[],[],[],[],[]]\n",
"\n",
"# Embed and insert the remainder \n",
"if len(data[0]) != 0:\n",
" data.append(embed(data[4]))\n",
" collection.insert(data)\n",
" data = [[],[],[],[],[]]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Query the Database\n",
"With our data safely inserted in Milvus, we can now perform a query. The query takes in a tuple of the movie description you are searching for an the filter to use. More info about the filter can be found [here](https://milvus.io/docs/boolean.md). The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating, and description of the result movies. "
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Description: movie about a fluffly animal Expression: release_year < 2019 and rating like \"PG%\"\n",
"Results:\n",
"\tRank: 1 Score: 0.30083978176116943 Title: The Lamb\n",
"\t\tType: Movie Release Year: 2017 Rating: PG\n",
"A big-dreaming donkey escapes his menial existence and befriends some free-spirited\n",
"animal pals in this imaginative retelling of the Nativity Story.\n",
"\n",
"\tRank: 2 Score: 0.33528298139572144 Title: Puss in Boots\n",
"\t\tType: Movie Release Year: 2011 Rating: PG\n",
"The fabled feline heads to the Land of Giants with friends Humpty Dumpty and Kitty\n",
"Softpaws on a quest to nab its greatest treasure: the Golden Goose.\n",
"\n",
"\tRank: 3 Score: 0.33528298139572144 Title: Puss in Boots\n",
"\t\tType: Movie Release Year: 2011 Rating: PG\n",
"The fabled feline heads to the Land of Giants with friends Humpty Dumpty and Kitty\n",
"Softpaws on a quest to nab its greatest treasure: the Golden Goose.\n",
"\n",
"\tRank: 4 Score: 0.3414868116378784 Title: Show Dogs\n",
"\t\tType: Movie Release Year: 2018 Rating: PG\n",
"A rough and tough police dog must go undercover with an FBI agent as a prim and proper\n",
"pet at a dog show to save a baby panda from an illegal sale.\n",
"\n",
"\tRank: 5 Score: 0.3414868116378784 Title: Show Dogs\n",
"\t\tType: Movie Release Year: 2018 Rating: PG\n",
"A rough and tough police dog must go undercover with an FBI agent as a prim and proper\n",
"pet at a dog show to save a baby panda from an illegal sale.\n",
"\n"
]
}
],
"source": [
"import textwrap\n",
"\n",
"def query(query, top_k = 5):\n",
" text, expr = query\n",
" res = collection.search(embed(text), anns_field='embedding', expr = expr, param=QUERY_PARAM, limit = top_k, output_fields=['title', 'type', 'release_year', 'rating', 'description'])\n",
" for i, hit in enumerate(res):\n",
" print('Description:', text, 'Expression:', expr)\n",
" print('Results:')\n",
" for ii, hits in enumerate(hit):\n",
" print('\\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))\n",
" print('\\t\\t' + 'Type:', hits.entity.get('type'), 'Release Year:', hits.entity.get('release_year'), 'Rating:', hits.entity.get('rating'))\n",
" print(textwrap.fill(hits.entity.get('description'), 88))\n",
" print()\n",
"\n",
"my_query = ('movie about a fluffly animal', 'release_year < 2019 and rating like \\\"PG%\\\"')\n",
"\n",
"query(my_query)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "haystack",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/milvus/Getting_started_with_Milvus_and_OpenAI.ipynb
+575
View File
@@ -0,0 +1,575 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Getting Started with Milvus and OpenAI\n",
"### Finding your next book\n",
"\n",
"In this notebook we will be going over generating embeddings of book descriptions with OpenAI and using those embeddings within Milvus to find relevant books. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 1 million title-description pairs.\n",
"\n",
"Lets begin by first downloading the required libraries for this notebook:\n",
"- `openai` is used for communicating with the OpenAI embedding service\n",
"- `pymilvus` is used for communicating with the Milvus server\n",
"- `datasets` is used for downloading the dataset\n",
"- `tqdm` is used for the progress bars\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
"Requirement already satisfied: openai in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (0.27.2)\n",
"Requirement already satisfied: pymilvus in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (2.2.2)\n",
"Requirement already satisfied: datasets in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (2.10.1)\n",
"Requirement already satisfied: tqdm in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (4.64.1)\n",
"Requirement already satisfied: aiohttp in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from openai) (3.8.4)\n",
"Requirement already satisfied: requests>=2.20 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from openai) (2.28.2)\n",
"Requirement already satisfied: pandas>=1.2.4 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.5.3)\n",
"Requirement already satisfied: ujson<=5.4.0,>=2.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (5.1.0)\n",
"Requirement already satisfied: mmh3<=3.0.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (3.0.0)\n",
"Requirement already satisfied: grpcio<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)\n",
"Requirement already satisfied: grpcio-tools<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)\n",
"Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.12.1)\n",
"Requirement already satisfied: dill<0.3.7,>=0.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.3.6)\n",
"Requirement already satisfied: xxhash in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (3.2.0)\n",
"Requirement already satisfied: pyyaml>=5.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (5.4.1)\n",
"Requirement already satisfied: fsspec[http]>=2021.11.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (2023.1.0)\n",
"Requirement already satisfied: packaging in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (23.0)\n",
"Requirement already satisfied: numpy>=1.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (1.23.5)\n",
"Requirement already satisfied: multiprocess in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.70.14)\n",
"Requirement already satisfied: pyarrow>=6.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (10.0.1)\n",
"Requirement already satisfied: responses<0.19 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.18.0)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (6.0.4)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.3)\n",
"Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (4.0.2)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.8.2)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.1)\n",
"Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (3.0.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (22.2.0)\n",
"Requirement already satisfied: six>=1.5.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio<=1.48.0,>=1.47.0->pymilvus) (1.16.0)\n",
"Requirement already satisfied: protobuf<4.0dev,>=3.12.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (3.20.1)\n",
"Requirement already satisfied: setuptools in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (65.6.3)\n",
"Requirement already satisfied: filelock in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.9.0)\n",
"Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (4.5.0)\n",
"Requirement already satisfied: python-dateutil>=2.8.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2022.7.1)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (1.26.14)\n",
"Requirement already satisfied: idna<4,>=2.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (3.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.7)\n"
]
}
],
"source": [
"! pip install openai pymilvus datasets tqdm"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"With the required packages installed we can get started. Lets begin by launching the Milvus service. The file being run is the `docker-compose.yaml` found in the folder of this file. This command launches a Milvus standalone instance which we will use for this test. "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 0/0\n",
"\u001b[37m ⠋ Network milvus Creating 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 1/1\u001b[0m\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠋ Container milvus-minio Creating 0.1s\n",
"\u001b[0m\u001b[37m ⠋ Container milvus-etcd Creating 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 1/3\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠙ Container milvus-minio Creating 0.2s\n",
"\u001b[0m\u001b[37m ⠙ Container milvus-etcd Creating 0.2s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 1/3\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠹ Container milvus-minio Creating 0.3s\n",
"\u001b[0m\u001b[37m ⠹ Container milvus-etcd Creating 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 3/3\u001b[0m\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Created 0.3s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Created 0.3s\n",
"\u001b[0m\u001b[37m ⠋ Container milvus-standalone Creating 0.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Created 0.3s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Created 0.3s\n",
"\u001b[0m\u001b[37m ⠙ Container milvus-standalone Creating 0.2s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 4/4\u001b[0m\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Created 0.3s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Created 0.3s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.7s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.8s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 0.9s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 0.9s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.0s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.0s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.2s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.2s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.3s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.3s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.4s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.4s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.5s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.5s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.6s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.6s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 2/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-etcd Starting 1.7s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-minio Starting 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Created 0.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.6s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.7s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.8s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 1.9s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 2.0s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 2.1s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 2.2s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 2.3s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 2.4s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 2.5s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 3/4\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[37m ⠿ Container milvus-standalone Starting 2.6s\n",
"\u001b[0m\u001b[?25h\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 4/4\u001b[0m\n",
"\u001b[34m ⠿ Network milvus Created 0.1s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-minio Started 1.8s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-etcd Started 1.7s\n",
"\u001b[0m\u001b[34m ⠿ Container milvus-standalone Started 2.6s\n",
"\u001b[0m\u001b[?25h"
]
}
],
"source": [
"! docker compose up -d"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"With Milvus running we can setup our global variables:\n",
"- HOST: The Milvus host address\n",
"- PORT: The Milvus port number\n",
"- COLLECTION_NAME: What to name the collection within Milvus\n",
"- DIMENSION: The dimension of the embeddings\n",
"- OPENAI_ENGINE: Which embedding model to use\n",
"- openai.api_key: Your OpenAI account key\n",
"- INDEX_PARAM: The index settings to use for the collection\n",
"- QUERY_PARAM: The search parameters to use\n",
"- BATCH_SIZE: How many texts to embed and insert at once"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"HOST = 'localhost'\n",
"PORT = 19530\n",
"COLLECTION_NAME = 'book_search'\n",
"DIMENSION = 1536\n",
"OPENAI_ENGINE = 'text-embedding-3-small'\n",
"openai.api_key = 'sk-your_key'\n",
"\n",
"INDEX_PARAM = {\n",
" 'metric_type':'L2',\n",
" 'index_type':\"HNSW\",\n",
" 'params':{'M': 8, 'efConstruction': 64}\n",
"}\n",
"\n",
"QUERY_PARAM = {\n",
" \"metric_type\": \"L2\",\n",
" \"params\": {\"ef\": 64},\n",
"}\n",
"\n",
"BATCH_SIZE = 1000"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Milvus\n",
"This segment deals with Milvus and setting up the database for this use case. Within Milvus we need to setup a collection and index the collection. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n",
"\n",
"# Connect to Milvus Database\n",
"connections.connect(host=HOST, port=PORT)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# Remove collection if it already exists\n",
"if utility.has_collection(COLLECTION_NAME):\n",
" utility.drop_collection(COLLECTION_NAME)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# Create collection which includes the id, title, and embedding.\n",
"fields = [\n",
" FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n",
" FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n",
"]\n",
"schema = CollectionSchema(fields=fields)\n",
"collection = Collection(name=COLLECTION_NAME, schema=schema)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# Create the index on the collection and load it.\n",
"collection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\n",
"collection.load()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dataset\n",
"With Milvus up and running we can begin grabbing our data. Hugging Face Datasets is a hub that holds many different user datasets, and for this example we are using Skelebor's book dataset. This dataset contains title-description pairs for over 1 million books. We are going to embed each description and store it within Milvus along with its title. "
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"Found cached dataset parquet (/Users/filiphaltmayer/.cache/huggingface/datasets/Skelebor___parquet/Skelebor--book_titles_and_descriptions_en_clean-3596935b1d8a7747/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
]
}
],
"source": [
"import datasets\n",
"\n",
"# Download the dataset and only use the `train` portion (file is around 800Mb)\n",
"dataset = datasets.load_dataset('Skelebor/book_titles_and_descriptions_en_clean', split='train')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Insert the Data\n",
"Now that we have our data on our machine we can begin embedding it and inserting it into Milvus. The embedding function takes in text and returns the embeddings in a list format. "
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"# Simple function that converts the texts to embeddings\n",
"def embed(texts):\n",
" embeddings = openai.Embedding.create(\n",
" input=texts,\n",
" engine=OPENAI_ENGINE\n",
" )\n",
" return [x['embedding'] for x in embeddings['data']]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This next step does the actual inserting. Due to having so many datapoints, if you want to immidiately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough. "
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" 0%| | 1999/1032335 [00:06<57:22, 299.31it/s] \n"
]
},
{
"ename": "KeyboardInterrupt",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[18], line 13\u001b[0m\n\u001b[1;32m 11\u001b[0m data[\u001b[39m1\u001b[39m]\u001b[39m.\u001b[39mappend(dataset[i][\u001b[39m'\u001b[39m\u001b[39mdescription\u001b[39m\u001b[39m'\u001b[39m])\n\u001b[1;32m 12\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mlen\u001b[39m(data[\u001b[39m0\u001b[39m]) \u001b[39m%\u001b[39m BATCH_SIZE \u001b[39m==\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[0;32m---> 13\u001b[0m data\u001b[39m.\u001b[39mappend(embed(data[\u001b[39m1\u001b[39;49m]))\n\u001b[1;32m 14\u001b[0m collection\u001b[39m.\u001b[39minsert(data)\n\u001b[1;32m 15\u001b[0m data \u001b[39m=\u001b[39m [[],[]]\n",
"Cell \u001b[0;32mIn[17], line 3\u001b[0m, in \u001b[0;36membed\u001b[0;34m(texts)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39membed\u001b[39m(texts):\n\u001b[0;32m----> 3\u001b[0m embeddings \u001b[39m=\u001b[39m openai\u001b[39m.\u001b[39;49mEmbedding\u001b[39m.\u001b[39;49mcreate(\n\u001b[1;32m 4\u001b[0m \u001b[39minput\u001b[39;49m\u001b[39m=\u001b[39;49mtexts,\n\u001b[1;32m 5\u001b[0m engine\u001b[39m=\u001b[39;49mOPENAI_ENGINE\n\u001b[1;32m 6\u001b[0m )\n\u001b[1;32m 7\u001b[0m \u001b[39mreturn\u001b[39;00m [x[\u001b[39m'\u001b[39m\u001b[39membedding\u001b[39m\u001b[39m'\u001b[39m] \u001b[39mfor\u001b[39;00m x \u001b[39min\u001b[39;00m embeddings[\u001b[39m'\u001b[39m\u001b[39mdata\u001b[39m\u001b[39m'\u001b[39m]]\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/openai/api_resources/embedding.py:33\u001b[0m, in \u001b[0;36mEmbedding.create\u001b[0;34m(cls, *args, **kwargs)\u001b[0m\n\u001b[1;32m 31\u001b[0m \u001b[39mwhile\u001b[39;00m \u001b[39mTrue\u001b[39;00m:\n\u001b[1;32m 32\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m---> 33\u001b[0m response \u001b[39m=\u001b[39m \u001b[39msuper\u001b[39;49m()\u001b[39m.\u001b[39;49mcreate(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 35\u001b[0m \u001b[39m# If a user specifies base64, we'll just return the encoded string.\u001b[39;00m\n\u001b[1;32m 36\u001b[0m \u001b[39m# This is only for the default case.\u001b[39;00m\n\u001b[1;32m 37\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m user_provided_encoding_format:\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/openai/api_resources/abstract/engine_api_resource.py:153\u001b[0m, in \u001b[0;36mEngineAPIResource.create\u001b[0;34m(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)\u001b[0m\n\u001b[1;32m 127\u001b[0m \u001b[39m@classmethod\u001b[39m\n\u001b[1;32m 128\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mcreate\u001b[39m(\n\u001b[1;32m 129\u001b[0m \u001b[39mcls\u001b[39m,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 136\u001b[0m \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mparams,\n\u001b[1;32m 137\u001b[0m ):\n\u001b[1;32m 138\u001b[0m (\n\u001b[1;32m 139\u001b[0m deployment_id,\n\u001b[1;32m 140\u001b[0m engine,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 150\u001b[0m api_key, api_base, api_type, api_version, organization, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mparams\n\u001b[1;32m 151\u001b[0m )\n\u001b[0;32m--> 153\u001b[0m response, _, api_key \u001b[39m=\u001b[39m requestor\u001b[39m.\u001b[39;49mrequest(\n\u001b[1;32m 154\u001b[0m \u001b[39m\"\u001b[39;49m\u001b[39mpost\u001b[39;49m\u001b[39m\"\u001b[39;49m,\n\u001b[1;32m 155\u001b[0m url,\n\u001b[1;32m 156\u001b[0m params\u001b[39m=\u001b[39;49mparams,\n\u001b[1;32m 157\u001b[0m headers\u001b[39m=\u001b[39;49mheaders,\n\u001b[1;32m 158\u001b[0m stream\u001b[39m=\u001b[39;49mstream,\n\u001b[1;32m 159\u001b[0m request_id\u001b[39m=\u001b[39;49mrequest_id,\n\u001b[1;32m 160\u001b[0m request_timeout\u001b[39m=\u001b[39;49mrequest_timeout,\n\u001b[1;32m 161\u001b[0m )\n\u001b[1;32m 163\u001b[0m \u001b[39mif\u001b[39;00m stream:\n\u001b[1;32m 164\u001b[0m \u001b[39m# must be an iterator\u001b[39;00m\n\u001b[1;32m 165\u001b[0m \u001b[39massert\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39misinstance\u001b[39m(response, OpenAIResponse)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/openai/api_requestor.py:216\u001b[0m, in \u001b[0;36mAPIRequestor.request\u001b[0;34m(self, method, url, params, headers, files, stream, request_id, request_timeout)\u001b[0m\n\u001b[1;32m 205\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mrequest\u001b[39m(\n\u001b[1;32m 206\u001b[0m \u001b[39mself\u001b[39m,\n\u001b[1;32m 207\u001b[0m method,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 214\u001b[0m request_timeout: Optional[Union[\u001b[39mfloat\u001b[39m, Tuple[\u001b[39mfloat\u001b[39m, \u001b[39mfloat\u001b[39m]]] \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m,\n\u001b[1;32m 215\u001b[0m ) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m Tuple[Union[OpenAIResponse, Iterator[OpenAIResponse]], \u001b[39mbool\u001b[39m, \u001b[39mstr\u001b[39m]:\n\u001b[0;32m--> 216\u001b[0m result \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mrequest_raw(\n\u001b[1;32m 217\u001b[0m method\u001b[39m.\u001b[39;49mlower(),\n\u001b[1;32m 218\u001b[0m url,\n\u001b[1;32m 219\u001b[0m params\u001b[39m=\u001b[39;49mparams,\n\u001b[1;32m 220\u001b[0m supplied_headers\u001b[39m=\u001b[39;49mheaders,\n\u001b[1;32m 221\u001b[0m files\u001b[39m=\u001b[39;49mfiles,\n\u001b[1;32m 222\u001b[0m stream\u001b[39m=\u001b[39;49mstream,\n\u001b[1;32m 223\u001b[0m request_id\u001b[39m=\u001b[39;49mrequest_id,\n\u001b[1;32m 224\u001b[0m request_timeout\u001b[39m=\u001b[39;49mrequest_timeout,\n\u001b[1;32m 225\u001b[0m )\n\u001b[1;32m 226\u001b[0m resp, got_stream \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_interpret_response(result, stream)\n\u001b[1;32m 227\u001b[0m \u001b[39mreturn\u001b[39;00m resp, got_stream, \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mapi_key\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/openai/api_requestor.py:516\u001b[0m, in \u001b[0;36mAPIRequestor.request_raw\u001b[0;34m(self, method, url, params, supplied_headers, files, stream, request_id, request_timeout)\u001b[0m\n\u001b[1;32m 514\u001b[0m _thread_context\u001b[39m.\u001b[39msession \u001b[39m=\u001b[39m _make_session()\n\u001b[1;32m 515\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m--> 516\u001b[0m result \u001b[39m=\u001b[39m _thread_context\u001b[39m.\u001b[39;49msession\u001b[39m.\u001b[39;49mrequest(\n\u001b[1;32m 517\u001b[0m method,\n\u001b[1;32m 518\u001b[0m abs_url,\n\u001b[1;32m 519\u001b[0m headers\u001b[39m=\u001b[39;49mheaders,\n\u001b[1;32m 520\u001b[0m data\u001b[39m=\u001b[39;49mdata,\n\u001b[1;32m 521\u001b[0m files\u001b[39m=\u001b[39;49mfiles,\n\u001b[1;32m 522\u001b[0m stream\u001b[39m=\u001b[39;49mstream,\n\u001b[1;32m 523\u001b[0m timeout\u001b[39m=\u001b[39;49mrequest_timeout \u001b[39mif\u001b[39;49;00m request_timeout \u001b[39melse\u001b[39;49;00m TIMEOUT_SECS,\n\u001b[1;32m 524\u001b[0m )\n\u001b[1;32m 525\u001b[0m \u001b[39mexcept\u001b[39;00m requests\u001b[39m.\u001b[39mexceptions\u001b[39m.\u001b[39mTimeout \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 526\u001b[0m \u001b[39mraise\u001b[39;00m error\u001b[39m.\u001b[39mTimeout(\u001b[39m\"\u001b[39m\u001b[39mRequest timed out: \u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(e)) \u001b[39mfrom\u001b[39;00m \u001b[39me\u001b[39;00m\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/requests/sessions.py:587\u001b[0m, in \u001b[0;36mSession.request\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m 582\u001b[0m send_kwargs \u001b[39m=\u001b[39m {\n\u001b[1;32m 583\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mtimeout\u001b[39m\u001b[39m\"\u001b[39m: timeout,\n\u001b[1;32m 584\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mallow_redirects\u001b[39m\u001b[39m\"\u001b[39m: allow_redirects,\n\u001b[1;32m 585\u001b[0m }\n\u001b[1;32m 586\u001b[0m send_kwargs\u001b[39m.\u001b[39mupdate(settings)\n\u001b[0;32m--> 587\u001b[0m resp \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49msend(prep, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49msend_kwargs)\n\u001b[1;32m 589\u001b[0m \u001b[39mreturn\u001b[39;00m resp\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/requests/sessions.py:701\u001b[0m, in \u001b[0;36mSession.send\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m 698\u001b[0m start \u001b[39m=\u001b[39m preferred_clock()\n\u001b[1;32m 700\u001b[0m \u001b[39m# Send the request\u001b[39;00m\n\u001b[0;32m--> 701\u001b[0m r \u001b[39m=\u001b[39m adapter\u001b[39m.\u001b[39;49msend(request, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 703\u001b[0m \u001b[39m# Total elapsed time of the request (approximately)\u001b[39;00m\n\u001b[1;32m 704\u001b[0m elapsed \u001b[39m=\u001b[39m preferred_clock() \u001b[39m-\u001b[39m start\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/requests/adapters.py:489\u001b[0m, in \u001b[0;36mHTTPAdapter.send\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m 487\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m 488\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m chunked:\n\u001b[0;32m--> 489\u001b[0m resp \u001b[39m=\u001b[39m conn\u001b[39m.\u001b[39;49murlopen(\n\u001b[1;32m 490\u001b[0m method\u001b[39m=\u001b[39;49mrequest\u001b[39m.\u001b[39;49mmethod,\n\u001b[1;32m 491\u001b[0m url\u001b[39m=\u001b[39;49murl,\n\u001b[1;32m 492\u001b[0m body\u001b[39m=\u001b[39;49mrequest\u001b[39m.\u001b[39;49mbody,\n\u001b[1;32m 493\u001b[0m headers\u001b[39m=\u001b[39;49mrequest\u001b[39m.\u001b[39;49mheaders,\n\u001b[1;32m 494\u001b[0m redirect\u001b[39m=\u001b[39;49m\u001b[39mFalse\u001b[39;49;00m,\n\u001b[1;32m 495\u001b[0m assert_same_host\u001b[39m=\u001b[39;49m\u001b[39mFalse\u001b[39;49;00m,\n\u001b[1;32m 496\u001b[0m preload_content\u001b[39m=\u001b[39;49m\u001b[39mFalse\u001b[39;49;00m,\n\u001b[1;32m 497\u001b[0m decode_content\u001b[39m=\u001b[39;49m\u001b[39mFalse\u001b[39;49;00m,\n\u001b[1;32m 498\u001b[0m retries\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mmax_retries,\n\u001b[1;32m 499\u001b[0m timeout\u001b[39m=\u001b[39;49mtimeout,\n\u001b[1;32m 500\u001b[0m )\n\u001b[1;32m 502\u001b[0m \u001b[39m# Send the request.\u001b[39;00m\n\u001b[1;32m 503\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 504\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mhasattr\u001b[39m(conn, \u001b[39m\"\u001b[39m\u001b[39mproxy_pool\u001b[39m\u001b[39m\"\u001b[39m):\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/urllib3/connectionpool.py:703\u001b[0m, in \u001b[0;36mHTTPConnectionPool.urlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m 700\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_prepare_proxy(conn)\n\u001b[1;32m 702\u001b[0m \u001b[39m# Make the request on the httplib connection object.\u001b[39;00m\n\u001b[0;32m--> 703\u001b[0m httplib_response \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_make_request(\n\u001b[1;32m 704\u001b[0m conn,\n\u001b[1;32m 705\u001b[0m method,\n\u001b[1;32m 706\u001b[0m url,\n\u001b[1;32m 707\u001b[0m timeout\u001b[39m=\u001b[39;49mtimeout_obj,\n\u001b[1;32m 708\u001b[0m body\u001b[39m=\u001b[39;49mbody,\n\u001b[1;32m 709\u001b[0m headers\u001b[39m=\u001b[39;49mheaders,\n\u001b[1;32m 710\u001b[0m chunked\u001b[39m=\u001b[39;49mchunked,\n\u001b[1;32m 711\u001b[0m )\n\u001b[1;32m 713\u001b[0m \u001b[39m# If we're going to release the connection in ``finally:``, then\u001b[39;00m\n\u001b[1;32m 714\u001b[0m \u001b[39m# the response doesn't need to know about the connection. Otherwise\u001b[39;00m\n\u001b[1;32m 715\u001b[0m \u001b[39m# it will also try to release it and we'll have a double-release\u001b[39;00m\n\u001b[1;32m 716\u001b[0m \u001b[39m# mess.\u001b[39;00m\n\u001b[1;32m 717\u001b[0m response_conn \u001b[39m=\u001b[39m conn \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m release_conn \u001b[39melse\u001b[39;00m \u001b[39mNone\u001b[39;00m\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/urllib3/connectionpool.py:449\u001b[0m, in \u001b[0;36mHTTPConnectionPool._make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m 444\u001b[0m httplib_response \u001b[39m=\u001b[39m conn\u001b[39m.\u001b[39mgetresponse()\n\u001b[1;32m 445\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mBaseException\u001b[39;00m \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 446\u001b[0m \u001b[39m# Remove the TypeError from the exception chain in\u001b[39;00m\n\u001b[1;32m 447\u001b[0m \u001b[39m# Python 3 (including for exceptions like SystemExit).\u001b[39;00m\n\u001b[1;32m 448\u001b[0m \u001b[39m# Otherwise it looks like a bug in the code.\u001b[39;00m\n\u001b[0;32m--> 449\u001b[0m six\u001b[39m.\u001b[39;49mraise_from(e, \u001b[39mNone\u001b[39;49;00m)\n\u001b[1;32m 450\u001b[0m \u001b[39mexcept\u001b[39;00m (SocketTimeout, BaseSSLError, SocketError) \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 451\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_raise_timeout(err\u001b[39m=\u001b[39me, url\u001b[39m=\u001b[39murl, timeout_value\u001b[39m=\u001b[39mread_timeout)\n",
"File \u001b[0;32m<string>:3\u001b[0m, in \u001b[0;36mraise_from\u001b[0;34m(value, from_value)\u001b[0m\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/urllib3/connectionpool.py:444\u001b[0m, in \u001b[0;36mHTTPConnectionPool._make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m 441\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mTypeError\u001b[39;00m:\n\u001b[1;32m 442\u001b[0m \u001b[39m# Python 3\u001b[39;00m\n\u001b[1;32m 443\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m--> 444\u001b[0m httplib_response \u001b[39m=\u001b[39m conn\u001b[39m.\u001b[39;49mgetresponse()\n\u001b[1;32m 445\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mBaseException\u001b[39;00m \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 446\u001b[0m \u001b[39m# Remove the TypeError from the exception chain in\u001b[39;00m\n\u001b[1;32m 447\u001b[0m \u001b[39m# Python 3 (including for exceptions like SystemExit).\u001b[39;00m\n\u001b[1;32m 448\u001b[0m \u001b[39m# Otherwise it looks like a bug in the code.\u001b[39;00m\n\u001b[1;32m 449\u001b[0m six\u001b[39m.\u001b[39mraise_from(e, \u001b[39mNone\u001b[39;00m)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/http/client.py:1377\u001b[0m, in \u001b[0;36mHTTPConnection.getresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 1375\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m 1376\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m-> 1377\u001b[0m response\u001b[39m.\u001b[39;49mbegin()\n\u001b[1;32m 1378\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mConnectionError\u001b[39;00m:\n\u001b[1;32m 1379\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mclose()\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/http/client.py:320\u001b[0m, in \u001b[0;36mHTTPResponse.begin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 318\u001b[0m \u001b[39m# read until we get a non-100 response\u001b[39;00m\n\u001b[1;32m 319\u001b[0m \u001b[39mwhile\u001b[39;00m \u001b[39mTrue\u001b[39;00m:\n\u001b[0;32m--> 320\u001b[0m version, status, reason \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_read_status()\n\u001b[1;32m 321\u001b[0m \u001b[39mif\u001b[39;00m status \u001b[39m!=\u001b[39m CONTINUE:\n\u001b[1;32m 322\u001b[0m \u001b[39mbreak\u001b[39;00m\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/http/client.py:281\u001b[0m, in \u001b[0;36mHTTPResponse._read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 280\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m_read_status\u001b[39m(\u001b[39mself\u001b[39m):\n\u001b[0;32m--> 281\u001b[0m line \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mfp\u001b[39m.\u001b[39;49mreadline(_MAXLINE \u001b[39m+\u001b[39;49m \u001b[39m1\u001b[39;49m), \u001b[39m\"\u001b[39m\u001b[39miso-8859-1\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m 282\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mlen\u001b[39m(line) \u001b[39m>\u001b[39m _MAXLINE:\n\u001b[1;32m 283\u001b[0m \u001b[39mraise\u001b[39;00m LineTooLong(\u001b[39m\"\u001b[39m\u001b[39mstatus line\u001b[39m\u001b[39m\"\u001b[39m)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/socket.py:704\u001b[0m, in \u001b[0;36mSocketIO.readinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m 702\u001b[0m \u001b[39mwhile\u001b[39;00m \u001b[39mTrue\u001b[39;00m:\n\u001b[1;32m 703\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m--> 704\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_sock\u001b[39m.\u001b[39;49mrecv_into(b)\n\u001b[1;32m 705\u001b[0m \u001b[39mexcept\u001b[39;00m timeout:\n\u001b[1;32m 706\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_timeout_occurred \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/ssl.py:1242\u001b[0m, in \u001b[0;36mSSLSocket.recv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m 1238\u001b[0m \u001b[39mif\u001b[39;00m flags \u001b[39m!=\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m 1239\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\n\u001b[1;32m 1240\u001b[0m \u001b[39m\"\u001b[39m\u001b[39mnon-zero flags not allowed in calls to recv_into() on \u001b[39m\u001b[39m%s\u001b[39;00m\u001b[39m\"\u001b[39m \u001b[39m%\u001b[39m\n\u001b[1;32m 1241\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m\u001b[39m__class__\u001b[39m)\n\u001b[0;32m-> 1242\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mread(nbytes, buffer)\n\u001b[1;32m 1243\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 1244\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39msuper\u001b[39m()\u001b[39m.\u001b[39mrecv_into(buffer, nbytes, flags)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/ssl.py:1100\u001b[0m, in \u001b[0;36mSSLSocket.read\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m 1098\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m 1099\u001b[0m \u001b[39mif\u001b[39;00m buffer \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m-> 1100\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_sslobj\u001b[39m.\u001b[39;49mread(\u001b[39mlen\u001b[39;49m, buffer)\n\u001b[1;32m 1101\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 1102\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_sslobj\u001b[39m.\u001b[39mread(\u001b[39mlen\u001b[39m)\n",
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
]
}
],
"source": [
"from tqdm import tqdm\n",
"\n",
"data = [\n",
" [], # title\n",
" [], # description\n",
"]\n",
"\n",
"# Embed and insert in batches\n",
"for i in tqdm(range(0, len(dataset))):\n",
" data[0].append(dataset[i]['title'])\n",
" data[1].append(dataset[i]['description'])\n",
" if len(data[0]) % BATCH_SIZE == 0:\n",
" data.append(embed(data[1]))\n",
" collection.insert(data)\n",
" data = [[],[]]\n",
"\n",
"# Embed and insert the remainder \n",
"if len(data[0]) != 0:\n",
" data.append(embed(data[1]))\n",
" collection.insert(data)\n",
" data = [[],[]]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Query the Database\n",
"With our data safely inserted in Milvus, we can now perform a query. The query takes in a string or a list of strings and searches them. The resuts print out your provided description and the results that include the result score, the result title, and the result book description. "
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"import textwrap\n",
"\n",
"def query(queries, top_k = 5):\n",
" if type(queries) != list:\n",
" queries = [queries]\n",
" res = collection.search(embed(queries), anns_field='embedding', param=QUERY_PARAM, limit = top_k, output_fields=['title', 'description'])\n",
" for i, hit in enumerate(res):\n",
" print('Description:', queries[i])\n",
" print('Results:')\n",
" for ii, hits in enumerate(hit):\n",
" print('\\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))\n",
" print(textwrap.fill(hits.entity.get('description'), 88))\n",
" print()"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"RPC error: [search], <MilvusException: (code=1, message=code: UnexpectedError, reason: code: CollectionNotExists, reason: can't find collection: book_search)>, <Time:{'RPC start': '2023-03-17 14:22:18.368461', 'RPC error': '2023-03-17 14:22:18.382086'}>\n"
]
},
{
"ename": "MilvusException",
"evalue": "<MilvusException: (code=1, message=code: UnexpectedError, reason: code: CollectionNotExists, reason: can't find collection: book_search)>",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mMilvusException\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[32], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m query(\u001b[39m'\u001b[39;49m\u001b[39mBook about a k-9 from europe\u001b[39;49m\u001b[39m'\u001b[39;49m)\n",
"Cell \u001b[0;32mIn[31], line 6\u001b[0m, in \u001b[0;36mquery\u001b[0;34m(queries, top_k)\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mtype\u001b[39m(queries) \u001b[39m!=\u001b[39m \u001b[39mlist\u001b[39m:\n\u001b[1;32m 5\u001b[0m queries \u001b[39m=\u001b[39m [queries]\n\u001b[0;32m----> 6\u001b[0m res \u001b[39m=\u001b[39m collection\u001b[39m.\u001b[39;49msearch(embed(queries), anns_field\u001b[39m=\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39membedding\u001b[39;49m\u001b[39m'\u001b[39;49m, param\u001b[39m=\u001b[39;49mQUERY_PARAM, limit \u001b[39m=\u001b[39;49m top_k, output_fields\u001b[39m=\u001b[39;49m[\u001b[39m'\u001b[39;49m\u001b[39mtitle\u001b[39;49m\u001b[39m'\u001b[39;49m, \u001b[39m'\u001b[39;49m\u001b[39mdescription\u001b[39;49m\u001b[39m'\u001b[39;49m])\n\u001b[1;32m 7\u001b[0m \u001b[39mfor\u001b[39;00m i, hit \u001b[39min\u001b[39;00m \u001b[39menumerate\u001b[39m(res):\n\u001b[1;32m 8\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m'\u001b[39m\u001b[39mDescription:\u001b[39m\u001b[39m'\u001b[39m, queries[i])\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/orm/collection.py:614\u001b[0m, in \u001b[0;36mCollection.search\u001b[0;34m(self, data, anns_field, param, limit, expr, partition_names, output_fields, timeout, round_decimal, **kwargs)\u001b[0m\n\u001b[1;32m 611\u001b[0m \u001b[39mraise\u001b[39;00m DataTypeNotMatchException(message\u001b[39m=\u001b[39mExceptionsMessage\u001b[39m.\u001b[39mExprType \u001b[39m%\u001b[39m \u001b[39mtype\u001b[39m(expr))\n\u001b[1;32m 613\u001b[0m conn \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_get_connection()\n\u001b[0;32m--> 614\u001b[0m res \u001b[39m=\u001b[39m conn\u001b[39m.\u001b[39;49msearch(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_name, data, anns_field, param, limit, expr,\n\u001b[1;32m 615\u001b[0m partition_names, output_fields, round_decimal, timeout\u001b[39m=\u001b[39;49mtimeout,\n\u001b[1;32m 616\u001b[0m schema\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_schema_dict, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 617\u001b[0m \u001b[39mif\u001b[39;00m kwargs\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39m_async\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mFalse\u001b[39;00m):\n\u001b[1;32m 618\u001b[0m \u001b[39mreturn\u001b[39;00m SearchFuture(res)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:109\u001b[0m, in \u001b[0;36merror_handler.<locals>.wrapper.<locals>.handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 107\u001b[0m record_dict[\u001b[39m\"\u001b[39m\u001b[39mRPC error\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(datetime\u001b[39m.\u001b[39mdatetime\u001b[39m.\u001b[39mnow())\n\u001b[1;32m 108\u001b[0m LOGGER\u001b[39m.\u001b[39merror(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mRPC error: [\u001b[39m\u001b[39m{\u001b[39;00minner_name\u001b[39m}\u001b[39;00m\u001b[39m], \u001b[39m\u001b[39m{\u001b[39;00me\u001b[39m}\u001b[39;00m\u001b[39m, <Time:\u001b[39m\u001b[39m{\u001b[39;00mrecord_dict\u001b[39m}\u001b[39;00m\u001b[39m>\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[0;32m--> 109\u001b[0m \u001b[39mraise\u001b[39;00m e\n\u001b[1;32m 110\u001b[0m \u001b[39mexcept\u001b[39;00m grpc\u001b[39m.\u001b[39mFutureTimeoutError \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 111\u001b[0m record_dict[\u001b[39m\"\u001b[39m\u001b[39mgRPC timeout\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(datetime\u001b[39m.\u001b[39mdatetime\u001b[39m.\u001b[39mnow())\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:105\u001b[0m, in \u001b[0;36merror_handler.<locals>.wrapper.<locals>.handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 103\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m 104\u001b[0m record_dict[\u001b[39m\"\u001b[39m\u001b[39mRPC start\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(datetime\u001b[39m.\u001b[39mdatetime\u001b[39m.\u001b[39mnow())\n\u001b[0;32m--> 105\u001b[0m \u001b[39mreturn\u001b[39;00m func(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 106\u001b[0m \u001b[39mexcept\u001b[39;00m MilvusException \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 107\u001b[0m record_dict[\u001b[39m\"\u001b[39m\u001b[39mRPC error\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(datetime\u001b[39m.\u001b[39mdatetime\u001b[39m.\u001b[39mnow())\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:136\u001b[0m, in \u001b[0;36mtracing_request.<locals>.wrapper.<locals>.handler\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 134\u001b[0m \u001b[39mif\u001b[39;00m req_id:\n\u001b[1;32m 135\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mset_onetime_request_id(req_id)\n\u001b[0;32m--> 136\u001b[0m ret \u001b[39m=\u001b[39m func(\u001b[39mself\u001b[39;49m, \u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 137\u001b[0m \u001b[39mreturn\u001b[39;00m ret\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:85\u001b[0m, in \u001b[0;36mretry_on_rpc_failure.<locals>.wrapper.<locals>.handler\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 83\u001b[0m back_off \u001b[39m=\u001b[39m \u001b[39mmin\u001b[39m(back_off \u001b[39m*\u001b[39m back_off_multiplier, max_back_off)\n\u001b[1;32m 84\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m---> 85\u001b[0m \u001b[39mraise\u001b[39;00m e\n\u001b[1;32m 86\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 87\u001b[0m \u001b[39mraise\u001b[39;00m e\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:50\u001b[0m, in \u001b[0;36mretry_on_rpc_failure.<locals>.wrapper.<locals>.handler\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 48\u001b[0m \u001b[39mwhile\u001b[39;00m \u001b[39mTrue\u001b[39;00m:\n\u001b[1;32m 49\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m---> 50\u001b[0m \u001b[39mreturn\u001b[39;00m func(\u001b[39mself\u001b[39;49m, \u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 51\u001b[0m \u001b[39mexcept\u001b[39;00m grpc\u001b[39m.\u001b[39mRpcError \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 52\u001b[0m \u001b[39m# DEADLINE_EXCEEDED means that the task wat not completed\u001b[39;00m\n\u001b[1;32m 53\u001b[0m \u001b[39m# UNAVAILABLE means that the service is not reachable currently\u001b[39;00m\n\u001b[1;32m 54\u001b[0m \u001b[39m# Reference: https://grpc.github.io/grpc/python/grpc.html#grpc-status-code\u001b[39;00m\n\u001b[1;32m 55\u001b[0m \u001b[39mif\u001b[39;00m e\u001b[39m.\u001b[39mcode() \u001b[39m!=\u001b[39m grpc\u001b[39m.\u001b[39mStatusCode\u001b[39m.\u001b[39mDEADLINE_EXCEEDED \u001b[39mand\u001b[39;00m e\u001b[39m.\u001b[39mcode() \u001b[39m!=\u001b[39m grpc\u001b[39m.\u001b[39mStatusCode\u001b[39m.\u001b[39mUNAVAILABLE:\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:472\u001b[0m, in \u001b[0;36mGrpcHandler.search\u001b[0;34m(self, collection_name, data, anns_field, param, limit, expression, partition_names, output_fields, round_decimal, timeout, schema, **kwargs)\u001b[0m\n\u001b[1;32m 467\u001b[0m requests \u001b[39m=\u001b[39m Prepare\u001b[39m.\u001b[39msearch_requests_with_expr(collection_name, data, anns_field, param, limit, schema,\n\u001b[1;32m 468\u001b[0m expression, partition_names, output_fields, round_decimal,\n\u001b[1;32m 469\u001b[0m \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[1;32m 471\u001b[0m auto_id \u001b[39m=\u001b[39m schema[\u001b[39m\"\u001b[39m\u001b[39mauto_id\u001b[39m\u001b[39m\"\u001b[39m]\n\u001b[0;32m--> 472\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_execute_search_requests(requests, timeout, round_decimal\u001b[39m=\u001b[39;49mround_decimal, auto_id\u001b[39m=\u001b[39;49mauto_id, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:441\u001b[0m, in \u001b[0;36mGrpcHandler._execute_search_requests\u001b[0;34m(self, requests, timeout, **kwargs)\u001b[0m\n\u001b[1;32m 439\u001b[0m \u001b[39mif\u001b[39;00m kwargs\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39m_async\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mFalse\u001b[39;00m):\n\u001b[1;32m 440\u001b[0m \u001b[39mreturn\u001b[39;00m SearchFuture(\u001b[39mNone\u001b[39;00m, \u001b[39mNone\u001b[39;00m, \u001b[39mTrue\u001b[39;00m, pre_err)\n\u001b[0;32m--> 441\u001b[0m \u001b[39mraise\u001b[39;00m pre_err\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:432\u001b[0m, in \u001b[0;36mGrpcHandler._execute_search_requests\u001b[0;34m(self, requests, timeout, **kwargs)\u001b[0m\n\u001b[1;32m 429\u001b[0m response \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_stub\u001b[39m.\u001b[39mSearch(request, timeout\u001b[39m=\u001b[39mtimeout)\n\u001b[1;32m 431\u001b[0m \u001b[39mif\u001b[39;00m response\u001b[39m.\u001b[39mstatus\u001b[39m.\u001b[39merror_code \u001b[39m!=\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[0;32m--> 432\u001b[0m \u001b[39mraise\u001b[39;00m MilvusException(response\u001b[39m.\u001b[39mstatus\u001b[39m.\u001b[39merror_code, response\u001b[39m.\u001b[39mstatus\u001b[39m.\u001b[39mreason)\n\u001b[1;32m 434\u001b[0m raws\u001b[39m.\u001b[39mappend(response)\n\u001b[1;32m 435\u001b[0m round_decimal \u001b[39m=\u001b[39m kwargs\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mround_decimal\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39m-\u001b[39m\u001b[39m1\u001b[39m)\n",
"\u001b[0;31mMilvusException\u001b[0m: <MilvusException: (code=1, message=code: UnexpectedError, reason: code: CollectionNotExists, reason: can't find collection: book_search)>"
]
}
],
"source": [
"query('Book about a k-9 from europe')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "haystack",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/milvus/docker-compose.yaml
+52
View File
@@ -0,0 +1,52 @@
version: '3.5'
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.5
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2022-03-17T06-34-49Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "9001:9001"
- "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:latest
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
networks:
default:
name: milvus
examples/vector_databases/mongodb_atlas/README.md
+6
View File
@@ -0,0 +1,6 @@
# MongoDB Atlas Vector Search
[Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is a fully managed service that simplifies the process of effectively indexing high-dimensional vector data within MongoDB and being able to perform fast vector similarity searches. With Atlas Vector Search, you can use MongoDB as a standalone vector database for a new project or augment your existing MongoDB collections with vector search functionality. With Atlas Vector Search, you can use the powerful capabilities of vector search in any major public cloud (AWS, Azure, GCP) and achieve massive scalability and data security out of the box while being enterprise-ready with provisions like FedRamp, SoC2 compliance.
Documentation - [link](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/)
examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb
+404
View File
@@ -0,0 +1,404 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "flQYAhphJC5m"
},
"source": [
"\n",
"This notebook demonstrates how to build a semantic search application using OpenAI and [MongoDB Atlas vector search](https://www.mongodb.com/products/platform/atlas-vector-search)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "iYMn0dXXdFbY",
"outputId": "98dab421-f11b-40b8-8f82-6de42b25725a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collecting pymongo\n",
" Downloading pymongo-4.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m677.1/677.1 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting openai\n",
" Downloading openai-1.3.3-py3-none-any.whl (220 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m220.3/220.3 kB\u001b[0m \u001b[31m24.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting dnspython<3.0.0,>=1.16.0 (from pymongo)\n",
" Downloading dnspython-2.4.2-py3-none-any.whl (300 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m300.4/300.4 kB\u001b[0m \u001b[31m29.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: anyio<4,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n",
"Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)\n",
"Collecting httpx<1,>=0.23.0 (from openai)\n",
" Downloading httpx-0.25.1-py3-none-any.whl (75 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.0/75.0 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai) (1.10.13)\n",
"Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.1)\n",
"Requirement already satisfied: typing-extensions<5,>=4.5 in /usr/local/lib/python3.10/dist-packages (from openai) (4.5.0)\n",
"Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (3.4)\n",
"Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.3.0)\n",
"Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.1.3)\n",
"Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2023.7.22)\n",
"Collecting httpcore (from httpx<1,>=0.23.0->openai)\n",
" Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore->httpx<1,>=0.23.0->openai)\n",
" Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hInstalling collected packages: h11, dnspython, pymongo, httpcore, httpx, openai\n",
"\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
"llmx 0.0.15a0 requires cohere, which is not installed.\n",
"llmx 0.0.15a0 requires tiktoken, which is not installed.\u001b[0m\u001b[31m\n",
"\u001b[0mSuccessfully installed dnspython-2.4.2 h11-0.14.0 httpcore-1.0.2 httpx-0.25.1 openai-1.3.3 pymongo-4.6.0\n"
]
}
],
"source": [
"!pip install pymongo openai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vLuKFvTwJXpu"
},
"source": [
"# Step 1: Setup the environment\n",
"\n",
"There are 2 pre-requisites for this:\n",
"\n",
"1. **MongoDB Atlas cluster**: To create a forever free MongoDB Atlas cluster, first, you need to create a MongoDB Atlas account if you don't already have one. Visit the [MongoDB Atlas website](https://www.mongodb.com/atlas/database) and click on “Register.” Visit the [MongoDB Atlas](https://account.mongodb.com/account/login) dashboard and set up your cluster. In order to take advantage of the `$vectorSearch` operator in an aggregation pipeline, you need to run MongoDB Atlas 6.0.11 or higher. This tutorial can be built using a free cluster. When you’re setting up your deployment, you’ll be prompted to set up a database user and rules for your network connection. Please ensure you save your username and password somewhere safe and have the correct IP address rules in place so your cluster can connect properly. If you need more help getting started, check out our [tutorial on MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial).\n",
"\n",
"2. **OpenAI API key** To create your OpenAI key, you'll need to create an account. Once you have that, visit the [OpenAI platform](https://platform.openai.com/). Click on your profile icon in the top right of the screen to get the dropdown menu and select “View API keys”.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "qJHHIIKjIFUZ",
"outputId": "57ad72d4-8afb-4e34-aad1-1fea6eb3645b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MongoDB Atlas Cluster URI:··········\n",
"OpenAI API Key:··········\n"
]
}
],
"source": [
"import getpass\n",
"\n",
"MONGODB_ATLAS_CLUSTER_URI = getpass.getpass(\"MongoDB Atlas Cluster URI:\")\n",
"OPENAI_API_KEY = getpass.getpass(\"OpenAI API Key:\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Sarx9wdxb4Rr"
},
"source": [
"Note: After executing the step above you will be prompted to enter the credentials."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sk1xXoyxMfil"
},
"source": [
"For this tutorial, we will be using the\n",
"[MongoDB sample dataset](https://www.mongodb.com/docs/atlas/sample-data/). Load the sample dataset using the Atlas UI. We'll be using the “sample_mflix” database, which contains a “movies” collection where each document contains fields like title, plot, genres, cast, directors, etc.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "k-G6WhNFdIvW"
},
"outputs": [],
"source": [
"import openai\n",
"import pymongo\n",
"\n",
"client = pymongo.MongoClient(MONGODB_ATLAS_CLUSTER_URI)\n",
"db = client.sample_mflix\n",
"collection = db.movies\n",
"\n",
"openai.api_key = OPENAI_API_KEY"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "On9e13ASwReq"
},
"outputs": [],
"source": [
"ATLAS_VECTOR_SEARCH_INDEX_NAME = \"default\"\n",
"EMBEDDING_FIELD_NAME = \"embedding_openai_nov19_23\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X-9gl2s-uGtw"
},
"source": [
"# Step 2: Setup embeddings generation function"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BMnE4BxSOCtH"
},
"outputs": [],
"source": [
"model = \"text-embedding-3-small\"\n",
"def generate_embedding(text: str) -> list[float]:\n",
" return openai.embeddings.create(input = [text], model=model).data[0].embedding\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "snSjiSNKwX6Z"
},
"source": [
"# Step 3: Create and store embeddings\n",
"\n",
"Each document in the sample dataset sample_mflix.movies corresponds to a movie; we will execute an operation to create a vector embedding for the data in the \"plot\" field and store it in the database. Creating vector embeddings using OpenAI embeddings endpoint is necessary for performing a similarity search based on intent."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "t4i9gQM2xUFF",
"outputId": "ae558b67-9b06-4c83-c52a-a8047ecd40d5"
},
"outputs": [
{
"data": {
"text/plain": [
"BulkWriteResult({'writeErrors': [], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 50, 'nModified': 50, 'nRemoved': 0, 'upserted': []}, acknowledged=True)"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from pymongo import ReplaceOne\n",
"\n",
"# Update the collection with the embeddings\n",
"requests = []\n",
"\n",
"for doc in collection.find({'plot':{\"$exists\": True}}).limit(500):\n",
" doc[EMBEDDING_FIELD_NAME] = generate_embedding(doc['plot'])\n",
" requests.append(ReplaceOne({'_id': doc['_id']}, doc))\n",
"\n",
"collection.bulk_write(requests)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ShPbxQPaPvHD"
},
"source": [
"After executing the above, the documents in \"movies\" collection will contain an additional field of \"embedding\", as defined by the `EMBEDDDING_FIELD_NAME` variable, apart from already existing fields like title, plot, genres, cast, directors, etc."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Coq0tyjXyNIu"
},
"source": [
"Note: We are restricting this to just 500 documents in the interest of time. If you want to do this over the entire dataset of 23,000+ documents in our sample_mflix database, it will take a little while. Alternatively, you can use the [sample_mflix.embedded_movies collection](https://www.mongodb.com/docs/atlas/sample-data/sample-mflix/#sample_mflix.embedded_movies) which includes a pre-populated `plot_embedding` field that contains embeddings created using OpenAI's `text-embedding-3-small` embedding model that you can use with the Atlas Search vector search feature.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rCRCK6QOskqo"
},
"source": [
"# Step 4: Create a vector search index\n",
"\n",
"We will create Atlas Vector Search Index on this collection which will allow us to perform the Approximate KNN search, which powers the semantic search.\n",
"We will cover 2 ways to create this index - Atlas UI and using MongoDB python driver.\n",
"\n",
"(Optional) [Documentation: Create a Vector Search Index ](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ymRTaFb1X5Tq"
},
"source": [
"Now head over to [Atlas UI](cloud.mongodb.com) and create an Atlas Vector Search index using the steps descibed [here](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-tutorial/#create-the-atlas-vector-search-index). The 'dimensions' field with value 1536, corresponds to openAI text-embedding-ada002.\n",
"\n",
"Use the definition given below in the JSON editor on the Atlas UI.\n",
"\n",
"```\n",
"{\n",
" \"mappings\": {\n",
" \"dynamic\": true,\n",
" \"fields\": {\n",
" \"embedding\": {\n",
" \"dimensions\": 1536,\n",
" \"similarity\": \"dotProduct\",\n",
" \"type\": \"knnVector\"\n",
" }\n",
" }\n",
" }\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2l5BzUgncjiq"
},
"source": [
"(Optional) Alternatively, we can use [pymongo driver to create these vector search indexes programatically](https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.create_search_index)\n",
"The python command given in the cell below will create the index (this only works for the most recent version of the Python Driver for MongoDB and MongoDB server version 7.0+ Atlas cluster)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"id": "54OWgiaPcmD0",
"outputId": "2cb9d1d8-4515-49ad-9fe7-5b4fa3c6c86b"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'default'"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"collection.create_search_index(\n",
" {\"definition\":\n",
" {\"mappings\": {\"dynamic\": True, \"fields\": {\n",
" EMBEDDING_FIELD_NAME : {\n",
" \"dimensions\": 1536,\n",
" \"similarity\": \"dotProduct\",\n",
" \"type\": \"knnVector\"\n",
" }}}},\n",
" \"name\": ATLAS_VECTOR_SEARCH_INDEX_NAME\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6V9QKgm8caNb"
},
"source": [
"# Step 5: Query your data\n",
"\n",
"The results for the query here finds movies which have semantically similar plots to the text captured in the query string, rather than being based on the keyword search.\n",
"\n",
"(Optional) [Documentation: Run Vector Search Queries](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "34tib9TrMPg4"
},
"outputs": [],
"source": [
"\n",
"def query_results(query, k):\n",
" results = collection.aggregate([\n",
" {\n",
" '$vectorSearch': {\n",
" \"index\": ATLAS_VECTOR_SEARCH_INDEX_NAME,\n",
" \"path\": EMBEDDING_FIELD_NAME,\n",
" \"queryVector\": generate_embedding(query),\n",
" \"numCandidates\": 50,\n",
" \"limit\": 5,\n",
" }\n",
" }\n",
" ])\n",
" return results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"id": "kTqrip-hWULK"
},
"outputs": [],
"source": [
"query=\"imaginary characters from outerspace at war with earthlings\"\n",
"movies = query_results(query, 5)\n",
"\n",
"for movie in movies:\n",
" print(f'Movie Name: {movie[\"title\"]},\\nMovie Plot: {movie[\"plot\"]}\\n')"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb
+378
View File
@@ -0,0 +1,378 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using MyScale as a vector database for OpenAI embeddings\n",
"\n",
"This notebook provides a step-by-step guide on using MyScale as a vector database for OpenAI embeddings. The process includes:\n",
"\n",
"1. Utilizing precomputed embeddings generated by OpenAI API.\n",
"2. Storing these embeddings in a cloud instance of MyScale.\n",
"3. Converting raw text query to an embedding using OpenAI API.\n",
"4. Leveraging MyScale to perform nearest neighbor search within the created collection.\n",
"\n",
"### What is MyScale\n",
"\n",
"[MyScale](https://myscale.com) is a database built on Clickhouse that combines vector search and SQL analytics to offer a high-performance, streamlined, and fully managed experience. It's designed to facilitate joint queries and analyses on both structured and vector data, with comprehensive SQL support for all data processing.\n",
"\n",
"\n",
"### Deployment options\n",
"\n",
"- Deploy and execute vector search with SQL on your cluster within two minutes by using [MyScale Console](https://console.myscale.com).\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"To follow this guide, you will need to have the following:\n",
"\n",
"1. A MyScale cluster deployed by following the [quickstart guide](https://docs.myscale.com/en/quickstart/).\n",
"2. The 'clickhouse-connect' library to interact with MyScale.\n",
"3. An [OpenAI API key](https://beta.openai.com/account/api-keys) for vectorization of queries."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install requirements\n",
"\n",
"This notebook requires the `openai`, `clickhouse-connect`, as well as some other dependencies. Use the following command to install them:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.718972Z",
"start_time": "2023-02-16T12:04:30.434820Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"! pip install openai clickhouse-connect wget pandas"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare your OpenAI API key\n",
"\n",
"To use the OpenAI API, you'll need to set up an API key. If you don't have one already, you can obtain it from [OpenAI](https://platform.openai.com/account/api-keys)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.730338Z",
"start_time": "2023-02-16T12:05:05.723351Z"
}
},
"outputs": [],
"source": [
"import openai\n",
"\n",
"# get API key from on OpenAI website\n",
"openai.api_key = \"OPENAI_API_KEY\"\n",
"\n",
"# check we have authenticated\n",
"openai.Engine.list()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to MyScale\n",
"\n",
"Follow the [connections details](https://docs.myscale.com/en/cluster-management/) section to retrieve the cluster host, username, and password information from the MyScale console, and use it to create a connection to your cluster as shown below:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.827143Z",
"start_time": "2023-02-16T12:05:05.733771Z"
}
},
"outputs": [],
"source": [
"import clickhouse_connect\n",
"\n",
"# initialize client\n",
"client = clickhouse_connect.get_client(host='YOUR_CLUSTER_HOST', port=8443, username='YOUR_USERNAME', password='YOUR_CLUSTER_PASSWORD')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We need to load the dataset of precomputed vector embeddings for Wikipedia articles provided by OpenAI. Use the `wget` package to download the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:37.371951Z",
"start_time": "2023-02-16T12:05:06.851634Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After the download is complete, extract the file using the `zipfile` package:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:06:01.538851Z",
"start_time": "2023-02-16T12:05:37.376042Z"
}
},
"outputs": [],
"source": [
"import zipfile\n",
"\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\", \"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can load the data from `vector_database_wikipedia_articles_embedded.csv` into a Pandas DataFrame:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"from ast import literal_eval\n",
"\n",
"# read data from csv\n",
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n",
"article_df = article_df[['id', 'url', 'title', 'text', 'content_vector']]\n",
"\n",
"# read vectors from strings back into a list\n",
"article_df[\"content_vector\"] = article_df.content_vector.apply(literal_eval)\n",
"article_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index data\n",
"\n",
"We will create an SQL table called `articles` in MyScale to store the embeddings data. The table will include a vector index with a cosine distance metric and a constraint for the length of the embeddings. Use the following code to create and insert data into the articles table:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:17:36.366066Z",
"start_time": "2023-02-16T12:17:35.486872Z"
}
},
"outputs": [],
"source": [
"# create articles table with vector index\n",
"embedding_len=len(article_df['content_vector'][0]) # 1536\n",
"\n",
"client.command(f\"\"\"\n",
"CREATE TABLE IF NOT EXISTS default.articles\n",
"(\n",
" id UInt64,\n",
" url String,\n",
" title String,\n",
" text String,\n",
" content_vector Array(Float32),\n",
" CONSTRAINT cons_vector_len CHECK length(content_vector) = {embedding_len},\n",
" VECTOR INDEX article_content_index content_vector TYPE HNSWFLAT('metric_type=Cosine')\n",
")\n",
"ENGINE = MergeTree ORDER BY id\n",
"\"\"\")\n",
"\n",
"# insert data into the table in batches\n",
"from tqdm.auto import tqdm\n",
"\n",
"batch_size = 100\n",
"total_records = len(article_df)\n",
"\n",
"# upload data in batches\n",
"data = article_df.to_records(index=False).tolist()\n",
"column_names = article_df.columns.tolist() \n",
"\n",
"for i in tqdm(range(0, total_records, batch_size)):\n",
" i_end = min(i + batch_size, total_records)\n",
" client.insert(\"default.articles\", data[i:i_end], column_names=column_names)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We need to check the build status of the vector index before proceeding with the search, as it is automatically built in the background."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"articles count: 25000\n",
"index build status: Built\n"
]
}
],
"source": [
"# check count of inserted data\n",
"print(f\"articles count: {client.command('SELECT count(*) FROM default.articles')}\")\n",
"\n",
"# check the status of the vector index, make sure vector index is ready with 'Built' status\n",
"get_index_status=\"SELECT status FROM system.vector_indices WHERE name='article_content_index'\"\n",
"print(f\"index build status: {client.command(get_index_status)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search data\n",
"\n",
"Once indexed in MyScale, we can perform vector search to find similar content. First, we will use the OpenAI API to generate embeddings for our query. Then, we will perform the vector search using MyScale."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:39.379566Z",
"start_time": "2023-02-16T12:30:38.031041Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 Battle of Bannockburn\n",
"2 Wars of Scottish Independence\n",
"3 1651\n",
"4 First War of Scottish Independence\n",
"5 Robert I of Scotland\n",
"6 841\n",
"7 1716\n",
"8 1314\n",
"9 1263\n",
"10 William Wallace\n"
]
}
],
"source": [
"import openai\n",
"\n",
"query = \"Famous battles in Scottish history\"\n",
"\n",
"# creates embedding vector from user query\n",
"embed = openai.Embedding.create(\n",
" input=query,\n",
" model=\"text-embedding-3-small\",\n",
")[\"data\"][0][\"embedding\"]\n",
"\n",
"# query the database to find the top K similar content to the given query\n",
"top_k = 10\n",
"results = client.query(f\"\"\"\n",
"SELECT id, url, title, distance(content_vector, {embed}) as dist\n",
"FROM default.articles\n",
"ORDER BY dist\n",
"LIMIT {top_k}\n",
"\"\"\")\n",
"\n",
"# display results\n",
"for i, r in enumerate(results.named_results()):\n",
" print(i+1, r['title'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb
+531
View File
@@ -0,0 +1,531 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using MyScale for Embeddings Search\n",
"\n",
"This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"### What is a Vector Database\n",
"\n",
"A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
"\n",
"### Why use a Vector Database\n",
"\n",
"Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
"\n",
"\n",
"### Demo Flow\n",
"The demo flow is:\n",
"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **MyScale**\n",
" - *Setup*: Set up the MyScale Python client. For more details go [here](https://docs.myscale.com/en/python-client/)\n",
" - *Index Data*: We'll create a table and index it for __content__.\n",
" - *Search Data*: Run a few example queries with various goals in mind.\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "e2b59250",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Import the required libraries and set the embedding model that we'd like to use."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d8810f9",
"metadata": {},
"outputs": [],
"source": [
"# We'll need to install the MyScale client\n",
"!pip install clickhouse-connect\n",
"\n",
"#Install wget to pull zip file\n",
"!pip install wget"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "5be94df6",
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"from typing import List, Iterator\n",
"import pandas as pd\n",
"import numpy as np\n",
"import os\n",
"import wget\n",
"from ast import literal_eval\n",
"\n",
"# MyScale's client library for Python\n",
"import clickhouse_connect\n",
"\n",
"# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"\n",
"# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that we've prepared previous to this session."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5dff8b55",
"metadata": {},
"outputs": [],
"source": [
"embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21097972",
"metadata": {},
"outputs": [],
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "70bbd8ba",
"metadata": {},
"outputs": [],
"source": [
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "1721e45d",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "960b82af",
"metadata": {},
"outputs": [],
"source": [
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
"\n",
"# Set vector_id to be a string\n",
"article_df['vector_id'] = article_df['vector_id'].apply(str)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a334ab8b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 25000 entries, 0 to 24999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 25000 non-null int64 \n",
" 1 url 25000 non-null object\n",
" 2 title 25000 non-null object\n",
" 3 text 25000 non-null object\n",
" 4 title_vector 25000 non-null object\n",
" 5 content_vector 25000 non-null object\n",
" 6 vector_id 25000 non-null object\n",
"dtypes: int64(1), object(6)\n",
"memory usage: 1.3+ MB\n"
]
}
],
"source": [
"article_df.info(show_counts=True)"
]
},
{
"cell_type": "markdown",
"id": "56a02772",
"metadata": {},
"source": [
"## MyScale\n",
"The next vector database we'll consider is [MyScale](https://myscale.com).\n",
"\n",
"[MyScale](https://myscale.com) is a database built on Clickhouse that combines vector search and SQL analytics to offer a high-performance, streamlined, and fully managed experience. It's designed to facilitate joint queries and analyses on both structured and vector data, with comprehensive SQL support for all data processing.\n",
"\n",
"Deploy and execute vector search with SQL on your cluster within two minutes by using [MyScale Console](https://console.myscale.com)."
]
},
{
"cell_type": "markdown",
"id": "d3e1f96b",
"metadata": {},
"source": [
"### Connect to MyScale\n",
"\n",
"Follow the [connections details](https://docs.myscale.com/en/cluster-management/) section to retrieve the cluster host, username, and password information from the MyScale console, and use it to create a connection to your cluster as shown below:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "024243cf",
"metadata": {},
"outputs": [],
"source": [
"# initialize client\n",
"client = clickhouse_connect.get_client(host='YOUR_CLUSTER_HOST', port=8443, username='YOUR_USERNAME', password='YOUR_CLUSTER_PASSWORD')"
]
},
{
"cell_type": "markdown",
"id": "067009db",
"metadata": {},
"source": [
"### Index data\n",
"\n",
"We will create an SQL table called `articles` in MyScale to store the embeddings data. The table will include a vector index with a cosine distance metric and a constraint for the length of the embeddings. Use the following code to create and insert data into the articles table:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "685cba13",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "378809ac23104dc08c06fa3a53f83666",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/250 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# create articles table with vector index\n",
"embedding_len=len(article_df['content_vector'][0]) # 1536\n",
"\n",
"client.command(f\"\"\"\n",
"CREATE TABLE IF NOT EXISTS default.articles\n",
"(\n",
" id UInt64,\n",
" url String,\n",
" title String,\n",
" text String,\n",
" content_vector Array(Float32),\n",
" CONSTRAINT cons_vector_len CHECK length(content_vector) = {embedding_len},\n",
" VECTOR INDEX article_content_index content_vector TYPE HNSWFLAT('metric_type=Cosine')\n",
")\n",
"ENGINE = MergeTree ORDER BY id\n",
"\"\"\")\n",
"\n",
"# insert data into the table in batches\n",
"from tqdm.auto import tqdm\n",
"\n",
"batch_size = 100\n",
"total_records = len(article_df)\n",
"\n",
"# we only need subset of columns\n",
"article_df = article_df[['id', 'url', 'title', 'text', 'content_vector']]\n",
"\n",
"# upload data in batches\n",
"data = article_df.to_records(index=False).tolist()\n",
"column_names = article_df.columns.tolist()\n",
"\n",
"for i in tqdm(range(0, total_records, batch_size)):\n",
" i_end = min(i + batch_size, total_records)\n",
" client.insert(\"default.articles\", data[i:i_end], column_names=column_names)"
]
},
{
"cell_type": "markdown",
"id": "b0f0e591",
"metadata": {},
"source": [
"We need to check the build status of the vector index before proceeding with the search, as it is automatically built in the background."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "9251bdf1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"articles count: 25000\n",
"index build status: InProgress\n"
]
}
],
"source": [
"# check count of inserted data\n",
"print(f\"articles count: {client.command('SELECT count(*) FROM default.articles')}\")\n",
"\n",
"# check the status of the vector index, make sure vector index is ready with 'Built' status\n",
"get_index_status=\"SELECT status FROM system.vector_indices WHERE name='article_content_index'\"\n",
"print(f\"index build status: {client.command(get_index_status)}\")"
]
},
{
"cell_type": "markdown",
"id": "fe55234a",
"metadata": {},
"source": [
"### Search data\n",
"\n",
"Once indexed in MyScale, we can perform vector search to find similar content. First, we will use the OpenAI API to generate embeddings for our query. Then, we will perform the vector search using MyScale."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "fd5f03c6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 Battle of Bannockburn\n",
"2 Wars of Scottish Independence\n",
"3 1651\n",
"4 First War of Scottish Independence\n",
"5 Robert I of Scotland\n",
"6 841\n",
"7 1716\n",
"8 1314\n",
"9 1263\n",
"10 William Wallace\n"
]
}
],
"source": [
"query = \"Famous battles in Scottish history\"\n",
"\n",
"# creates embedding vector from user query\n",
"embed = openai.Embedding.create(\n",
" input=query,\n",
" model=\"text-embedding-3-small\",\n",
")[\"data\"][0][\"embedding\"]\n",
"\n",
"# query the database to find the top K similar content to the given query\n",
"top_k = 10\n",
"results = client.query(f\"\"\"\n",
"SELECT id, url, title, distance(content_vector, {embed}) as dist\n",
"FROM default.articles\n",
"ORDER BY dist\n",
"LIMIT {top_k}\n",
"\"\"\")\n",
"\n",
"# display results\n",
"for i, r in enumerate(results.named_results()):\n",
" print(i+1, r['title'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0119d87a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "vector_db_split",
"language": "python",
"name": "vector_db_split"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
},
"vscode": {
"interpreter": {
"hash": "fd16a328ca3d68029457069b79cb0b38eb39a0f5ccc4fe4473d3047707df8207"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/neon/README.md
+42
View File
@@ -0,0 +1,42 @@
# What is Neon?
[Neon](https://neon.tech/) is Serverless Postgres built for the cloud. Neon separates compute and storage to offer modern developer features such as autoscaling, database branching, scale-to-zero, and more.
## Vector search
Neon supports vector search using the [pgvector](https://neon.tech/docs/extensions/pgvector) open-source PostgreSQL extension, which enables Postgres as a vector database for storing and querying embeddings.
## OpenAI cookbook notebook
Check out the notebook in this repo for working with Neon Serverless Postgres as your vector database.
### Semantic search using Neon Postgres with pgvector and OpenAI
In this notebook you will learn how to:
1. Use embeddings created by OpenAI API
2. Store embeddings in a Neon Serverless Postgres database
3. Convert a raw text query to an embedding with OpenAI API
4. Use Neon with the `pgvector` extension to perform vector similarity search
## Scaling Support
Neon enables you to scale your AI applications with the following features:
- [Autoscaling](https://neon.tech/docs/introduction/read-replicas): If your AI application experiences heavy load during certain hours of the day or at different times, Neon can automatically scale compute resources without manual intervention. During periods of inactivity, Neon is able to scale to zero.
- [Instant read replicas](https://neon.tech/docs/introduction/read-replicas): Neon supports instant read replicas, which are independent read-only compute instances designed to perform read operations on the same data as your read-write computes. With read replicas, you can offload reads from your read-write compute instance to a dedicated read-only compute instance for your AI application.
- [The Neon serverless driver](https://neon.tech/docs/serverless/serverless-driver): Neon supports a low-latency serverless PostgreSQL driver for JavaScript and TypeScript applications that allows you to query data from serverless and edge environments, making it possible to achieve sub-10ms queries.
## More Examples
- [Build an AI-powered semantic search application](https://github.com/neondatabase/yc-idea-matcher) - Submit a startup idea and get a list of similar ideas that YCombinator has invested in before
- [Build an AI-powered chatbot](https://github.com/neondatabase/ask-neon) - A Postgres Q&A chatbot that uses Postgres as a vector database
- [Vercel Postgres pgvector Starter](https://vercel.com/templates/next.js/postgres-pgvector) - Vector similarity search with Vercel Postgres (powered by Neon)
## Additional Resources
- [Building AI applications with Neon](https://neon.tech/ai)
- [Neon AI & embeddings documentation](https://neon.tech/docs/ai/ai-intro)
- [Building an AI-powered Chatbot using Vercel, OpenAI, and Postgres](neon.tech/blog/building-an-ai-powered-chatbot-using-vercel-openai-and-postgres)
- [Web-based AI SQL Playground and connecting to Postgres from the browser](https://neon.tech/blog/postgres-ai-playground)
- [pgvector GitHub repository](https://github.com/pgvector/pgvector)
examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb
+547
View File
@@ -0,0 +1,547 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Vector similarity search using Neon Postgres\n",
"\n",
"This notebook guides you through using [Neon Serverless Postgres](https://neon.tech/) as a vector database for OpenAI embeddings. It demonstrates how to:\n",
"\n",
"1. Use embeddings created by OpenAI API.\n",
"2. Store embeddings in a Neon Serverless Postgres database.\n",
"3. Convert a raw text query to an embedding with OpenAI API.\n",
"4. Use Neon with the `pgvector` extension to perform vector similarity search."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Before you begin, ensure that you have the following:\n",
"\n",
"1. A Neon Postgres database. You can create an account and set up a project with a ready-to-use `neondb` database in a few simple steps. For instructions, see [Sign up](https://neon.tech/docs/get-started-with-neon/signing-up) and [Create your first project](https://neon.tech/docs/get-started-with-neon/setting-up-a-project).\n",
"2. A connection string for your Neon database. You can copy it from the **Connection Details** widget on the Neon **Dashboard**. See [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app).\n",
"3. The `pgvector` extension. Install the extension in Neon by running `CREATE EXTENSION vector;`. For instructions, see [Enable the pgvector extension](https://neon.tech/docs/extensions/pgvector#enable-the-pgvector-extension). \n",
"4. Your [OpenAI API key](https://platform.openai.com/account/api-keys).\n",
"5. Python and `pip`."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install required modules\n",
"\n",
"This notebook requires the `openai`, `psycopg2`, `pandas`, `wget`, and `python-dotenv` packages. You can install them with `pip`:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install openai psycopg2 pandas wget python-dotenv"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare your OpenAI API key\n",
"\n",
"An OpenAI API key is required to generate vectors for documents and queries.\n",
"\n",
"If you do not have an OpenAI API key, obtain one from https://platform.openai.com/account/api-keys.\n",
"\n",
"Add the OpenAI API key as an operating system environment variable or provide it for the session when prompted. If you define an environment variable, name the variable `OPENAI_API_KEY`.\n",
"\n",
"For information about configuring your OpenAI API key as an environment variable, refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test your OpenAPI key"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test to ensure that your OpenAI API key is defined as an environment variable or provide it when prompted\n",
"# If you run this notebook locally, you may have to reload the terminal and the notebook to make the environment available\n",
"\n",
"import os\n",
"from getpass import getpass\n",
"\n",
"# Check if OPENAI_API_KEY is set as an environment variable\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print(\"Your OPENAI_API_KEY is ready\")\n",
"else:\n",
" # If not, prompt for it\n",
" api_key = getpass(\"Enter your OPENAI_API_KEY: \")\n",
" if api_key:\n",
" print(\"Your OPENAI_API_KEY is now available for this session\")\n",
" # Optionally, you can set it as an environment variable for the current session\n",
" os.environ[\"OPENAI_API_KEY\"] = api_key\n",
" else:\n",
" print(\"You did not enter your OPENAI_API_KEY\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to your Neon database\n",
"\n",
"Provide your Neon database connection string below or define it in an `.env` file using a `DATABASE_URL` variable. For information about obtaining a Neon connection string, see [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import psycopg2\n",
"from dotenv import load_dotenv\n",
"\n",
"# Load environment variables from .env file\n",
"load_dotenv()\n",
"\n",
"# The connection string can be provided directly here.\n",
"# Replace the next line with Your Neon connection string.\n",
"connection_string = \"postgres://<user>:<password>@<hostname>/<dbname>\"\n",
"\n",
"# If connection_string is not directly provided above, \n",
"# then check if DATABASE_URL is set in the environment or .env.\n",
"if not connection_string:\n",
" connection_string = os.environ.get(\"DATABASE_URL\")\n",
"\n",
" # If neither method provides a connection string, raise an error.\n",
" if not connection_string:\n",
" raise ValueError(\"Please provide a valid connection string either in the code or in the .env file as DATABASE_URL.\")\n",
"\n",
"# Connect using the connection string\n",
"connection = psycopg2.connect(connection_string)\n",
"\n",
"# Create a new cursor object\n",
"cursor = connection.cursor()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Test the connection to your database:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Your database connection was successful!\n"
]
}
],
"source": [
"# Execute this query to test the database connection\n",
"cursor.execute(\"SELECT 1;\")\n",
"result = cursor.fetchone()\n",
"\n",
"# Check the query result\n",
"if result == (1,):\n",
" print(\"Your database connection was successful!\")\n",
"else:\n",
" print(\"Your connection failed.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n",
"\n",
"Import the pre-computed embeddings zip file:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded.zip'"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB. Importing it will take several minutes.\n",
"wget.download(embeddings_url)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Extract the downloaded zip file:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n"
]
}
],
"source": [
"import zipfile\n",
"import os\n",
"import re\n",
"import tempfile\n",
"\n",
"current_directory = os.getcwd()\n",
"zip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\n",
"output_directory = os.path.join(current_directory, \"../../data\")\n",
"\n",
"with zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(output_directory)\n",
"\n",
"\n",
"# Check to see if the csv file was extracted\n",
"file_name = \"vector_database_wikipedia_articles_embedded.csv\"\n",
"data_directory = os.path.join(current_directory, \"../../data\")\n",
"file_path = os.path.join(data_directory, file_name)\n",
"\n",
"\n",
"if os.path.exists(file_path):\n",
" print(f\"The csv file {file_name} exists in the data directory.\")\n",
"else:\n",
" print(f\"The csv file {file_name} does not exist in the data directory.\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create a table and add indexes for your vector embeddings\n",
"\n",
"The vector table created in your database is called **articles**. Each object has **title** and **content** vectors. \n",
"\n",
"An index is defined on both the **title** and **content** vector columns."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"create_table_sql = '''\n",
"CREATE TABLE IF NOT EXISTS public.articles (\n",
" id INTEGER NOT NULL,\n",
" url TEXT,\n",
" title TEXT,\n",
" content TEXT,\n",
" title_vector vector(1536),\n",
" content_vector vector(1536),\n",
" vector_id INTEGER\n",
");\n",
"\n",
"ALTER TABLE public.articles ADD PRIMARY KEY (id);\n",
"'''\n",
"\n",
"# SQL statement for creating indexes\n",
"create_indexes_sql = '''\n",
"CREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n",
"\n",
"CREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n",
"'''\n",
"\n",
"# Execute the SQL statements\n",
"cursor.execute(create_table_sql)\n",
"cursor.execute(create_indexes_sql)\n",
"\n",
"# Commit the changes\n",
"connection.commit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load the data\n",
"\n",
"Load the pre-computed vector data into your `articles` table from the `.csv` file. There are 25000 records, so expect the operation to take several minutes."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"\n",
"# Path to your local CSV file\n",
"csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n",
"\n",
"# Define a generator function to process the csv file\n",
"def process_file(file_path):\n",
" with open(file_path, 'r', encoding='utf-8') as file:\n",
" for line in file:\n",
" yield line\n",
"\n",
"# Create a StringIO object to store the modified lines\n",
"modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n",
"\n",
"# Create the COPY command for copy_expert\n",
"copy_command = '''\n",
"COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\n",
"FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n",
"'''\n",
"\n",
"# Execute the COPY command using copy_expert\n",
"cursor.copy_expert(copy_command, modified_lines)\n",
"\n",
"# Commit the changes\n",
"connection.commit()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check the number of records to ensure the data has been been loaded. There should be 25000 records."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Count:25000\n"
]
}
],
"source": [
"# Check the size of the data\n",
"count_sql = \"\"\"select count(*) from public.articles;\"\"\"\n",
"cursor.execute(count_sql)\n",
"result = cursor.fetchone()\n",
"print(f\"Count:{result[0]}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search your data\n",
"\n",
"After the data is stored in your Neon database, you can query the data for nearest neighbors. \n",
"\n",
"Start by defining the `query_neon` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n",
"\n",
"A `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\"."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def query_neon(query, collection_name, vector_name=\"title_vector\", top_k=20):\n",
"\n",
" # Create an embedding vector from the user query\n",
" embedded_query = openai.Embedding.create(\n",
" input=query,\n",
" model=\"text-embedding-3-small\",\n",
" )[\"data\"][0][\"embedding\"]\n",
"\n",
" # Convert the embedded_query to PostgreSQL compatible format\n",
" embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n",
"\n",
" # Create the SQL query\n",
" query_sql = f\"\"\"\n",
" SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n",
" FROM {collection_name}\n",
" ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n",
" LIMIT {top_k};\n",
" \"\"\"\n",
" # Execute the query\n",
" cursor.execute(query_sql)\n",
" results = cursor.fetchall()\n",
"\n",
" return results"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run a similarity search based on `title_vector` embeddings:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Greek mythology (Score: 0.998)\n",
"2. Roman mythology (Score: 0.7)\n",
"3. Greek underworld (Score: 0.637)\n",
"4. Mythology (Score: 0.635)\n",
"5. Classical mythology (Score: 0.629)\n",
"6. Japanese mythology (Score: 0.615)\n",
"7. Norse mythology (Score: 0.569)\n",
"8. Greek language (Score: 0.566)\n",
"9. Zeus (Score: 0.534)\n",
"10. List of mythologies (Score: 0.531)\n",
"11. Jupiter (mythology) (Score: 0.53)\n",
"12. Greek (Score: 0.53)\n",
"13. Gaia (mythology) (Score: 0.526)\n",
"14. Titan (mythology) (Score: 0.522)\n",
"15. Mercury (mythology) (Score: 0.521)\n",
"16. Ancient Greece (Score: 0.52)\n",
"17. Greek alphabet (Score: 0.52)\n",
"18. Venus (mythology) (Score: 0.515)\n",
"19. Pluto (mythology) (Score: 0.515)\n",
"20. Athena (Score: 0.514)\n"
]
}
],
"source": [
"# Query based on `title_vector` embeddings\n",
"import openai\n",
"\n",
"query_results = query_neon(\"Greek mythology\", \"Articles\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run a similarity search based on `content_vector` embeddings:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. 222 BC (Score: 0.489)\n",
"2. Trojan War (Score: 0.458)\n",
"3. Peloponnesian War (Score: 0.456)\n",
"4. History of the Peloponnesian War (Score: 0.449)\n",
"5. 430 BC (Score: 0.441)\n",
"6. 168 BC (Score: 0.436)\n",
"7. Ancient Greece (Score: 0.429)\n",
"8. Classical Athens (Score: 0.428)\n",
"9. 499 BC (Score: 0.427)\n",
"10. Leonidas I (Score: 0.426)\n",
"11. Battle (Score: 0.421)\n",
"12. Greek War of Independence (Score: 0.421)\n",
"13. Menelaus (Score: 0.419)\n",
"14. Thebes, Greece (Score: 0.417)\n",
"15. Patroclus (Score: 0.417)\n",
"16. 427 BC (Score: 0.416)\n",
"17. 429 BC (Score: 0.413)\n",
"18. August 2 (Score: 0.412)\n",
"19. Ionia (Score: 0.411)\n",
"20. 323 (Score: 0.409)\n"
]
}
],
"source": [
"# Query based on `content_vector` embeddings\n",
"query_results = query_neon(\"Famous battles in Greek history\", \"Articles\", \"content_vector\")\n",
"for i, result in enumerate(query_results):\n",
" print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/oracle_db/README.md
+136
View File
@@ -0,0 +1,136 @@
# LangChain, OpenAI, and Oracle AI Database for Vector Search
This example shows how to build a semantic search workflow using:
- **OpenAI embeddings** for turning text into vector representations
- **LangChain's Oracle vector store integration** for working with vectors through a familiar framework abstraction
- **Oracle AI Database Vector Search** for storing embeddings and retrieving relevant content with similarity search
If you are looking for a practical starting point for **LangChain + Oracle AI Database**, **OpenAI embeddings with Oracle Vector Search**, or a developer-friendly example of semantic retrieval that integrates directly with data stored in Oracle AI Database, this notebook is designed to be easy to read, run, and adapt.
## Why this stack works well
This combination is useful because each component solves a different part of the workflow:
- **OpenAI embeddings** convert text into semantic vectors that capture meaning beyond keyword matching
- **LangChain** provides a developer-friendly abstraction for vector stores and retrieval workflows
- **Oracle AI Database** stores vectors alongside application and relational data, making it easier to build retrieval-enabled applications without introducing a separate vector database
For developers evaluating semantic search patterns, this creates a practical setup that is easy to prototype locally and extend later into larger RAG or agent-based workflows.
## Why use Oracle AI Database with LangChain
Oracle AI Database is a strong fit for LangChain-based vector workflows when you want:
- **vector search and application data in one place**
- **less infrastructure complexity**, especially for teams already using Oracle
- **native SQL access to vector operations**
- **a smoother path from notebook prototype to production architecture**
- compatibility with **LangChain retrieval patterns**, while keeping Oracle AI Database as the retrieval backbone
This is especially helpful when vector search is not a standalone demo, but one part of a broader enterprise or application stack.
## What you will learn
By working through the notebook, you will learn how to:
- connect to Oracle AI Database from Python
- configure OpenAI embeddings for semantic search
- initialize LangChain's Oracle vector store integration
- store sample documents and embeddings in Oracle AI Database
- run vector similarity search against indexed content
- inspect how LangChain retrieval maps to Oracle AI Database Vector Search
- use the same pattern as a baseline for larger retrieval or RAG workflows
## Architecture at a glance
The example follows a simple semantic retrieval flow:
1. Load configuration and environment variables
2. Initialize OpenAI embeddings
3. Connect to Oracle AI Database
4. Initialize the LangChain Oracle vector store
5. Insert sample text documents
6. Convert the query into an embedding
7. Run vector similarity search
8. Return the most relevant matching content
In a larger application, this same pattern can become the retrieval layer behind a RAG pipeline, internal search assistant, or AI agent workflow.
## Prerequisites
You will need:
- **Oracle AI Database** with Vector Search enabled
- **OpenAI API key**
- **Python 3.10+**
- Notebook dependencies installed for this example
## Environment variables
Create a local `.env` file and provide values similar to the following:
```env
OPENAI_API_KEY=
ORACLE_USER=PDBADMIN
ORACLE_PASSWORD=YourPassword
ORACLE_DSN=localhost:1521/FREEPDB1
```
## Python dependencies
Install the required packages using pip:
```bash
pip install langchain
pip install langchain-openai
pip install langchain-oracledb
pip install oracledb
pip install python-dotenv
```
## Files in this example
- `oracle_vector_search_langchain.ipynb` - the main notebook demonstrating semantic search with LangChain, OpenAI, and Oracle AI Database
- `.env` - local configuration file for OpenAI and Oracle connection settings
## How to run
1. Set your `OPENAI_API_KEY` in the local `.env` file
2. Make sure your Oracle AI Database instance is running and the connection values are correct
3. Open `oracle_vector_search_langchain.ipynb`
4. Run the notebook cells in order
The notebook walks through dependency setup, embedding initialization, Oracle database connection, vector store creation, sample document ingestion, and semantic similarity search.
## Expected outcome
By the end of this notebook, you will have a working semantic search example where:
- text is converted into embeddings using OpenAI
- vectors are stored in Oracle AI Database
- LangChain interacts with Oracle AI Database through its vector store integration
- similarity search returns the most relevant documents for a given query
This can serve as a baseline for larger retrieval, RAG, or agent-based applications.
## Next steps
Once this example is working, you can extend it by:
- replacing the sample documents with your own data
- adding metadata and document chunking
- using the same retrieval flow in a larger RAG pipeline
- integrating the notebook logic into an API or application
## Documentation
For additional information, please refer to the documentation:
[![LangChain Oracle Vector Store](https://img.shields.io/badge/LangChain-Oracle%20Vector%20Store-blue)](https://docs.langchain.com/oss/python/integrations/vectorstores/oracle)
[![LangChain OpenAI Embeddings](https://img.shields.io/badge/LangChain-OpenAI%20Embeddings-green)](https://docs.langchain.com/oss/python/integrations/text_embedding/openai)
[![Oracle Database Vector Search](https://img.shields.io/badge/Oracle-Database%20Vector%20Search-red)](https://docs.oracle.com/en/database/oracle/oracle-database/26/vecse/)
examples/vector_databases/oracle_db/oracle_vector_search_langchain.ipynb
+592
View File
@@ -0,0 +1,592 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "intro",
"metadata": {},
"source": [
"# Building Semantic Search with OpenAI Embeddings, LangChain, and Oracle AI Database\n",
"\n",
"This cookbook shows how to build a semantic search workflow using:\n",
"\n",
"- **OpenAI embeddings** to turn text into vector representations\n",
"- **LangChain's Oracle vector store integration** to write and query vectors through a familiar Python interface\n",
"- **Oracle AI Database Vector Search** to store embeddings alongside relational/application data\n",
"\n",
"OpenAI is used for embedding generation. LangChain provides the vector store abstraction, and Oracle AI Database provides vector storage and similarity search. This keeps the OpenAI role explicit without implying that OpenAI provides a separate managed vector search feature.\n",
"\n",
"This pattern is useful for retrieval-augmented generation (RAG), internal semantic search, and applications where source data already lives in Oracle. It lets you add semantic retrieval without introducing a separate vector database, while still keeping the LangChain retrieval interface available for larger application workflows.\n",
"\n",
"## Architecture at a glance\n",
"\n",
"1. Text documents are embedded with `text-embedding-3-small`.\n",
"2. LangChain writes the text, metadata, and vectors into an Oracle AI Database table.\n",
"3. A natural-language query is embedded with the same OpenAI model.\n",
"4. Oracle AI Database Vector Search returns the nearest stored documents.\n",
"5. The retrieved text can be passed to a generation model in a larger RAG application.\n",
"\n",
"## What this notebook demonstrates\n",
"\n",
"1. Load credentials from environment variables or a local `.env` file.\n",
"2. Connect to Oracle AI Database with the Python `oracledb` driver.\n",
"3. Initialize OpenAI embeddings and LangChain's Oracle vector store.\n",
"4. Insert a small rerunnable sample dataset without accumulating duplicates.\n",
"5. Run semantic similarity search with `similarity_search(..., k=3)`.\n",
"\n",
"## Requirements\n",
"\n",
"- Python 3.10+\n",
"- OpenAI API key with embeddings access\n",
"- Oracle AI Database with Vector Search enabled\n",
"- Python packages: `langchain`, `langchain-openai`, `langchain-oracledb`, `oracledb`, `python-dotenv`, `numpy`\n",
" "
]
},
{
"cell_type": "markdown",
"id": "install-deps-md",
"metadata": {},
"source": [
"### Install dependencies\n",
"\n",
"If these packages are already installed in your environment, this cell will not make any changes.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "install-deps",
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"%pip install -q langchain langchain-oracledb langchain-openai oracledb python-dotenv numpy\n",
" "
]
},
{
"cell_type": "markdown",
"id": "config-md",
"metadata": {},
"source": [
"### Configure credentials and Oracle connectivity\n",
"\n",
"The notebook reads configuration from environment variables. You can set them in your shell, in your notebook environment, or in a local `.env` file next to this notebook:\n",
"\n",
"```env\n",
"OPENAI_API_KEY=your-openai-api-key\n",
"ORACLE_USER=your-oracle-user\n",
"ORACLE_PASSWORD=your-oracle-password\n",
"ORACLE_DSN=host:port/service_name\n",
"```\n",
"\n",
"For local experiments, one common option is an Oracle Database Free container with port `1521` exposed and a pluggable database service such as `FREEPDB1`, making `ORACLE_DSN` look like `localhost:1521/FREEPDB1`. You can also use an Oracle Autonomous Database or another Oracle AI Database instance that has Vector Search enabled.\n",
"\n",
"Oracle's Python driver can run in **Thin mode**, which needs no separate Oracle client libraries. If Oracle Instant Client is installed, the same driver can optionally use **Thick mode** for compatibility with some advanced client features. The setup cell below tries to enable Thick mode when available and otherwise continues in Thin mode.\n",
" "
]
},
{
"cell_type": "markdown",
"id": "oracle-setup-options",
"metadata": {},
"source": [
"### Oracle setup options\n",
"\n",
"This notebook assumes you already have access to an Oracle AI Database endpoint. Any of these setup paths can work:\n",
"\n",
"- **Local development:** Oracle Database Free in a container or local install, with a service name such as `FREEPDB1`.\n",
"- **Managed cloud:** Oracle Autonomous Database with Vector Search enabled.\n",
"- **Existing environment:** an internal Oracle AI Database instance provided by your team.\n",
"\n",
"The notebook only needs a SQL connection string through `ORACLE_DSN`. It does not create a database instance for you.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "config-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Oracle Client initialized in Thick mode\n",
"Configuration loaded\n"
]
}
],
"source": [
"import os\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
"warnings.filterwarnings(\n",
" \"ignore\",\n",
" message=\"Core Pydantic V1 functionality isn't compatible with Python 3.14 or greater.*\",\n",
" category=UserWarning,\n",
")\n",
"\n",
"import numpy as np\n",
"import oracledb\n",
"from dotenv import load_dotenv\n",
"from langchain_openai import OpenAIEmbeddings\n",
"from langchain_oracledb.vectorstores import OracleVS\n",
"from langchain_oracledb.vectorstores.oraclevs import DistanceStrategy\n",
"\n",
"load_dotenv()\n",
"\n",
"# Optional: uncomment and fill these in if you are not using environment variables\n",
"# or a local .env file.\n",
"# os.environ[\"OPENAI_API_KEY\"] = \"your-openai-api-key\"\n",
"# os.environ[\"ORACLE_USER\"] = \"your-oracle-user\"\n",
"# os.environ[\"ORACLE_PASSWORD\"] = \"your-oracle-password\"\n",
"# os.environ[\"ORACLE_DSN\"] = \"host:port/service_name\"\n",
"\n",
"required_env_vars = [\n",
" \"OPENAI_API_KEY\",\n",
" \"ORACLE_USER\",\n",
" \"ORACLE_PASSWORD\",\n",
" \"ORACLE_DSN\",\n",
"]\n",
"\n",
"missing = [name for name in required_env_vars if not os.getenv(name)]\n",
"if missing:\n",
" raise EnvironmentError(\n",
" \"Missing required environment variables: \"\n",
" f\"{missing}. Set them in your environment or in a local .env file.\"\n",
" )\n",
"\n",
"try:\n",
" oracledb.init_oracle_client()\n",
" print(\"Oracle Client initialized in Thick mode\")\n",
"except Exception:\n",
" print(\"Oracle Client running in Thin mode\")\n",
"\n",
"print(\"Configuration loaded\")\n",
" "
]
},
{
"cell_type": "markdown",
"id": "connect-md",
"metadata": {},
"source": [
"### Connect to Oracle AI Database\n",
"\n",
"This connection is used by LangChain's Oracle vector store to create or reuse the demo table, insert embeddings, and run similarity search.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "connect-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Connected to Oracle AI Database\n"
]
}
],
"source": [
"conn = oracledb.connect(\n",
" user=os.getenv(\"ORACLE_USER\"),\n",
" password=os.getenv(\"ORACLE_PASSWORD\"),\n",
" dsn=os.getenv(\"ORACLE_DSN\"),\n",
")\n",
"\n",
"print(\"Connected to Oracle AI Database\")\n",
" "
]
},
{
"cell_type": "markdown",
"id": "table-distance-md",
"metadata": {},
"source": [
"### Choose a demo table and distance metric\n",
"\n",
"The demo uses one table, `LANGCHAIN_DEMO_VECTORS`, and cosine distance. Cosine distance is a common default for text embeddings because it compares vector direction rather than raw magnitude.\n",
" "
]
},
{
"cell_type": "markdown",
"id": "vector-store-md",
"metadata": {},
"source": [
"### Initialize embeddings and the Oracle vector store\n",
"\n",
"We initialize the OpenAI embedding model close to the vector store setup, because LangChain uses the embedding function when documents are inserted and when natural language queries are searched.\n",
"\n",
"If an incompatible demo table already exists from an earlier version of the notebook, it is dropped so LangChain can recreate it with the columns it expects: `ID`, `TEXT`, `METADATA`, and `EMBEDDING`.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "vector-store-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Oracle Vector Store initialized\n"
]
}
],
"source": [
"table_name = \"LANGCHAIN_DEMO_VECTORS\"\n",
"distance_strategy = DistanceStrategy.COSINE\n",
"\n",
"embeddings = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n",
"\n",
"required_columns = {\"ID\", \"TEXT\", \"METADATA\", \"EMBEDDING\"}\n",
"\n",
"with conn.cursor() as cursor:\n",
" cursor.execute(\n",
" \"\"\"\n",
" SELECT column_name\n",
" FROM user_tab_columns\n",
" WHERE table_name = :table_name\n",
" \"\"\",\n",
" table_name=table_name.upper(),\n",
" )\n",
" existing_columns = {row[0] for row in cursor.fetchall()}\n",
"\n",
"if existing_columns and not required_columns.issubset(existing_columns):\n",
" with conn.cursor() as cursor:\n",
" cursor.execute(f\"DROP TABLE {table_name} PURGE\")\n",
" conn.commit()\n",
" print(f\"Dropped incompatible existing demo table: {table_name}\")\n",
"\n",
"try:\n",
" oracle_vs = OracleVS(\n",
" client=conn,\n",
" embedding_function=embeddings,\n",
" table_name=table_name,\n",
" distance_strategy=distance_strategy,\n",
" )\n",
" print(\"Oracle Vector Store initialized\")\n",
"except Exception as e:\n",
" oracle_vs = None\n",
" print(\"Oracle Vector Store not initialized.\")\n",
" print(\"This can happen if embedding generation is unavailable or configuration is incomplete.\")\n",
" print(e)\n",
" "
]
},
{
"cell_type": "markdown",
"id": "sample-data-md",
"metadata": {},
"source": [
"### Insert rerunnable sample data\n",
"\n",
"The sample data is intentionally small so the retrieval behavior is easy to inspect. Before inserting, the cell clears the demo table. This keeps the notebook idempotent: repeated runs do not accumulate duplicate rows or change retrieval results.\n",
" "
]
},
{
"cell_type": "markdown",
"id": "idempotency-note",
"metadata": {},
"source": [
"The `DELETE FROM` in the next cell is intentional. It resets only the demo table's sample rows before insertion, making repeated notebook runs deterministic. Without that reset, rerunning the notebook would append the same documents again and could change the top-k retrieval results.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "sample-data-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cleared existing sample documents\n",
"Sample documents embedded and stored\n"
]
}
],
"source": [
"texts = [\n",
" \"Oracle AI Database provides native vector search.\",\n",
" \"LangChain integrates Oracle AI Database using the langchain-oracledb plugin.\",\n",
" \"Vector search enables semantic similarity over unstructured text.\",\n",
" \"Relational AI databases can now store and search embeddings efficiently.\",\n",
"]\n",
"\n",
"if oracle_vs:\n",
" try:\n",
" with conn.cursor() as cursor:\n",
" cursor.execute(f\"DELETE FROM {table_name}\")\n",
" conn.commit()\n",
" print(\"Cleared existing sample documents\")\n",
"\n",
" oracle_vs.add_texts(texts)\n",
" print(\"Sample documents embedded and stored\")\n",
" except Exception as e:\n",
" oracle_vs = None\n",
" print(\"Skipping document embedding because vector store population failed.\")\n",
" print(e)\n",
"else:\n",
" print(\"Skipping document embedding (vector store not available)\")\n",
" "
]
},
{
"cell_type": "markdown",
"id": "inspect-md",
"metadata": {},
"source": [
"### Inspect embedding and table details\n",
"\n",
"These checks are optional and are guarded so the notebook can continue gracefully if embedding quota is unavailable. The table inspection does not call the OpenAI API.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "inspect-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Embedding statistics:\n",
"- Embedding dimension: 1536\n",
"- Number of documents embedded: 4\n",
"- Oracle embedding table: LANGCHAIN_DEMO_VECTORS\n",
"Storage size estimation:\n",
"- Approx. bytes per embedding: 12288\n",
"- Approx. total bytes stored: 49152\n",
"Vector table LANGCHAIN_DEMO_VECTORS columns: ['ID', 'TEXT', 'METADATA', 'EMBEDDING']\n"
]
}
],
"source": [
"sample_embedding = None\n",
"embedding_dim = None\n",
"\n",
"if oracle_vs:\n",
" try:\n",
" sample_embedding = embeddings.embed_query(\"test\")\n",
" embedding_dim = len(sample_embedding)\n",
"\n",
" print(\"Embedding statistics:\")\n",
" print(f\"- Embedding dimension: {embedding_dim}\")\n",
" print(f\"- Number of documents embedded: {len(texts)}\")\n",
" print(f\"- Oracle embedding table: {table_name}\")\n",
"\n",
" embedding_bytes = np.array(sample_embedding).nbytes\n",
" print(\"Storage size estimation:\")\n",
" print(f\"- Approx. bytes per embedding: {embedding_bytes}\")\n",
" print(f\"- Approx. total bytes stored: {embedding_bytes * len(texts)}\")\n",
" except Exception as e:\n",
" print(\"Skipping embedding statistics because embedding generation is unavailable.\")\n",
" print(e)\n",
"else:\n",
" print(\"Skipping embedding statistics (vector store not available)\")\n",
"\n",
"with conn.cursor() as cursor:\n",
" cursor.execute(\n",
" \"\"\"\n",
" SELECT column_name\n",
" FROM user_tab_columns\n",
" WHERE table_name = :table_name\n",
" ORDER BY column_id\n",
" \"\"\",\n",
" table_name=table_name.upper(),\n",
" )\n",
" columns = [row[0] for row in cursor.fetchall()]\n",
"\n",
"print(f\"Vector table {table_name} columns: {columns}\")\n",
" "
]
},
{
"cell_type": "markdown",
"id": "search-md",
"metadata": {},
"source": [
"### Run semantic similarity search\n",
"\n",
"LangChain embeds the natural language query and searches the Oracle vector store for the closest stored documents. The example uses `similarity_search(..., k=3)`, which returns the top matching documents.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "search-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"LangChain similarity search returned 3 results\n",
"1. Oracle AI Database provides native vector search.\n",
"2. Vector search enables semantic similarity over unstructured text.\n",
"3. Relational AI databases can now store and search embeddings efficiently.\n"
]
}
],
"source": [
"query = \"How does Oracle support vector search?\"\n",
"\n",
"if oracle_vs:\n",
" try:\n",
" results = oracle_vs.similarity_search(query, k=3)\n",
" print(f\"LangChain similarity search returned {len(results)} results\")\n",
"\n",
" for i, doc in enumerate(results, start=1):\n",
" print(f\"{i}. {doc.page_content}\")\n",
" except Exception as e:\n",
" print(\"Skipping similarity search because query embedding or retrieval failed.\")\n",
" print(e)\n",
"else:\n",
" print(\"Skipping similarity search (vector store not available)\")\n",
" "
]
},
{
"cell_type": "markdown",
"id": "inspect-results-md",
"metadata": {},
"source": [
"### Inspect returned documents\n",
"\n",
"LangChain returns `Document` objects. In this simple example the important field is `page_content`, but in larger workflows metadata can be used to track source IDs, URLs, document sections, or other application-specific attributes.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "inspect-results-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result 1\n",
"- content: Oracle AI Database provides native vector search.\n",
"- metadata: {}\n",
"Result 2\n",
"- content: Vector search enables semantic similarity over unstructured text.\n",
"- metadata: {}\n",
"Result 3\n",
"- content: Relational AI databases can now store and search embeddings efficiently.\n",
"- metadata: {}\n"
]
}
],
"source": [
"if oracle_vs and \"results\" in globals():\n",
" for i, doc in enumerate(results, start=1):\n",
" print(f\"Result {i}\")\n",
" print(f\"- content: {doc.page_content}\")\n",
" print(f\"- metadata: {doc.metadata}\")\n",
"else:\n",
" print(\"Skipping document inspection because no search results are available.\")\n",
" "
]
},
{
"cell_type": "markdown",
"id": "second-query-md",
"metadata": {},
"source": [
"### Try another query\n",
"\n",
"The same vector store can be reused for additional questions without reinitializing the embedding model.\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "second-query-code",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Oracle AI Database vector search results:\n",
"Number of results: 3\n",
"1. Oracle AI Database provides native vector search.\n",
"2. LangChain integrates Oracle AI Database using the langchain-oracledb plugin.\n",
"3. Vector search enables semantic similarity over unstructured text.\n"
]
}
],
"source": [
"if oracle_vs:\n",
" try:\n",
" print(\"Oracle AI Database vector search results:\")\n",
" results = oracle_vs.similarity_search(\"What is Oracle Search?\", k=3)\n",
"\n",
" print(f\"Number of results: {len(results)}\")\n",
" for i, doc in enumerate(results, start=1):\n",
" print(f\"{i}. {doc.page_content}\")\n",
" except Exception as e:\n",
" print(\"Skipping final vector search because query embedding or retrieval failed.\")\n",
" print(e)\n",
"else:\n",
" print(\"Skipping final vector search because the Oracle vector store was not initialized.\")\n",
" "
]
},
{
"cell_type": "markdown",
"id": "conclusion",
"metadata": {},
"source": [
"### Conclusion\n",
"\n",
"This notebook implemented semantic vector search with OpenAI embeddings, LangChain, and Oracle AI Database. OpenAI converts text and queries into embeddings, LangChain provides the vector store API, and Oracle AI Database stores and searches the vectors alongside application data.\n",
"\n",
"The same pattern can be extended into a RAG workflow by passing retrieved documents to a generation model, or into an application search layer where Oracle remains the source of truth for both relational and vector data.\n",
" "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.14.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb
+1936
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/pinecone/Gen_QA.ipynb
+1017
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/pinecone/README.md
+15
View File
@@ -0,0 +1,15 @@
# Pinecone Vector Database
[Vector search](https://www.pinecone.io/learn/vector-search-basics/) is an innovative technology that enables developers and engineers to efficiently store, search, and recommend information by representing complex data as mathematical vectors. By comparing the similarities between these vectors, you can quickly retrieve relevant information in a seamless and intuitive manner.
[Pinecone](https://pinecone.io/) is a [vector database](https://www.pinecone.io/learn/vector-database/) designed with developers and engineers in mind. As a managed service, it alleviates the burden of maintenance and engineering, allowing you to focus on extracting valuable insights from your data. The free tier supports up to 5 million vectors, making it an accessible and cost-effective way to experiment with vector search capabilities. With Pinecone, you'll experience impressive speed, accuracy, and scalability, as well as access to advanced features like single-stage metadata filtering and the cutting-edge sparse-dense index.
## Examples
This folder contains examples of using Pinecone and OpenAI together. More will be added over time so check back for updates!
| Name | Description | Google Colab |
| --- | --- | --- |
| [GPT-4 Retrieval Augmentation](./GPT4_Retrieval_Augmentation.ipynb) | How to supercharge GPT-4 with retrieval augmentation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/master/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb) |
| [Generative Question-Answering](./Gen_QA.ipynb) | A simple walkthrough demonstrating the use of Generative Question-Answering | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/master/examples/vector_databases/pinecone/Gen_QA.ipynb) |
| [Semantic Search](./Semantic_Search.ipynb) | A guide to building a simple semantic search process | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/master/examples/vector_databases/pinecone/Semantic_Search.ipynb) |
examples/vector_databases/pinecone/Semantic_Search.ipynb
+2581
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb
+668
View File
@@ -0,0 +1,668 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Pinecone for Embeddings Search\n",
"\n",
"This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"### What is a Vector Database\n",
"\n",
"A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
"\n",
"### Why use a Vector Database\n",
"\n",
"Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
"\n",
"\n",
"### Demo Flow\n",
"The demo flow is:\n",
"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **Pinecone**\n",
" - *Setup*: Here we'll set up the Python client for Pinecone. For more details go [here](https://docs.pinecone.io/docs/quickstart)\n",
" - *Index Data*: We'll create an index with namespaces for __titles__ and __content__\n",
" - *Search Data*: We'll test out both namespaces with search queries to confirm it works\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "e2b59250",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Import the required libraries and set the embedding model that we'd like to use."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "8d8810f9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: pinecone-client in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (2.2.2)\n",
"Requirement already satisfied: requests>=2.19.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.31.0)\n",
"Requirement already satisfied: pyyaml>=5.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (6.0)\n",
"Requirement already satisfied: loguru>=0.5.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (0.7.0)\n",
"Requirement already satisfied: typing-extensions>=3.7.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.5.0)\n",
"Requirement already satisfied: dnspython>=2.0.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.3.0)\n",
"Requirement already satisfied: python-dateutil>=2.5.3 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.8.2)\n",
"Requirement already satisfied: urllib3>=1.21.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.26.16)\n",
"Requirement already satisfied: tqdm>=4.64.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.65.0)\n",
"Requirement already satisfied: numpy>=1.22.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.25.0)\n",
"Requirement already satisfied: six>=1.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from python-dateutil>=2.5.3->pinecone-client) (1.16.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.1.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (2023.5.7)\n",
"Requirement already satisfied: wget in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (3.2)\n"
]
}
],
"source": [
"# We'll need to install the Pinecone client\n",
"!pip install pinecone-client\n",
"\n",
"#Install wget to pull zip file\n",
"!pip install wget"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5be94df6",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
" from tqdm.autonotebook import tqdm\n"
]
}
],
"source": [
"import openai\n",
"\n",
"from typing import List, Iterator\n",
"import pandas as pd\n",
"import numpy as np\n",
"import os\n",
"import wget\n",
"from ast import literal_eval\n",
"\n",
"# Pinecone's client library for Python\n",
"import pinecone\n",
"\n",
"# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"\n",
"# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that we've prepared [in this article](../../Embedding_Wikipedia_articles_for_search.ipynb)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5dff8b55",
"metadata": {},
"outputs": [],
"source": [
"embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21097972",
"metadata": {},
"outputs": [],
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "70bbd8ba",
"metadata": {},
"outputs": [],
"source": [
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1721e45d",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "960b82af",
"metadata": {},
"outputs": [],
"source": [
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
"\n",
"# Set vector_id to be a string\n",
"article_df['vector_id'] = article_df['vector_id'].apply(str)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a334ab8b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 25000 entries, 0 to 24999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 25000 non-null int64 \n",
" 1 url 25000 non-null object\n",
" 2 title 25000 non-null object\n",
" 3 text 25000 non-null object\n",
" 4 title_vector 25000 non-null object\n",
" 5 content_vector 25000 non-null object\n",
" 6 vector_id 25000 non-null object\n",
"dtypes: int64(1), object(6)\n",
"memory usage: 1.3+ MB\n"
]
}
],
"source": [
"article_df.info(show_counts=True)"
]
},
{
"cell_type": "markdown",
"id": "ed32fc87",
"metadata": {},
"source": [
"## Pinecone\n",
"\n",
"The next option we'll look at is **Pinecone**, a managed vector database which offers a cloud-native option.\n",
"\n",
"Before you proceed with this step you'll need to navigate to [Pinecone](pinecone.io), sign up and then save your API key as an environment variable titled ```PINECONE_API_KEY```.\n",
"\n",
"For section we will:\n",
"- Create an index with multiple namespaces for article titles and content\n",
"- Store our data in the index with separate searchable \"namespaces\" for article **titles** and **content**\n",
"- Fire some similarity search queries to verify our setup is working"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "92e6152a",
"metadata": {},
"outputs": [],
"source": [
"api_key = os.getenv(\"PINECONE_API_KEY\")\n",
"pinecone.init(api_key=api_key)"
]
},
{
"cell_type": "markdown",
"id": "63b28543",
"metadata": {},
"source": [
"### Create Index\n",
"\n",
"First we will need to create an index, which we'll call `wikipedia-articles`. Once we have an index, we can create multiple namespaces, which can make a single index searchable for various use cases. For more details, consult [Pinecone documentation](https://docs.pinecone.io/docs/namespaces#:~:text=Pinecone%20allows%20you%20to%20partition,different%20subsets%20of%20your%20index.).\n",
"\n",
"If you want to batch insert to your index in parallel to increase insertion speed then there is a great guide in the Pinecone documentation on [batch inserts in parallel](https://docs.pinecone.io/docs/insert-data#sending-upserts-in-parallel)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "0a71c575",
"metadata": {},
"outputs": [],
"source": [
"# Models a simple batch generator that make chunks out of an input DataFrame\n",
"class BatchGenerator:\n",
" \n",
" \n",
" def __init__(self, batch_size: int = 10) -> None:\n",
" self.batch_size = batch_size\n",
" \n",
" # Makes chunks out of an input DataFrame\n",
" def to_batches(self, df: pd.DataFrame) -> Iterator[pd.DataFrame]:\n",
" splits = self.splits_num(df.shape[0])\n",
" if splits <= 1:\n",
" yield df\n",
" else:\n",
" for chunk in np.array_split(df, splits):\n",
" yield chunk\n",
"\n",
" # Determines how many chunks DataFrame contains\n",
" def splits_num(self, elements: int) -> int:\n",
" return round(elements / self.batch_size)\n",
" \n",
" __call__ = to_batches\n",
"\n",
"df_batcher = BatchGenerator(300)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "7ea9ad46",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['podcasts', 'wikipedia-articles']"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Pick a name for the new index\n",
"index_name = 'wikipedia-articles'\n",
"\n",
"# Check whether the index with the same name already exists - if so, delete it\n",
"if index_name in pinecone.list_indexes():\n",
" pinecone.delete_index(index_name)\n",
" \n",
"# Creates new index\n",
"pinecone.create_index(name=index_name, dimension=len(article_df['content_vector'][0]))\n",
"index = pinecone.Index(index_name=index_name)\n",
"\n",
"# Confirm our index was created\n",
"pinecone.list_indexes()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "5daeba00",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Uploading vectors to content namespace..\n"
]
}
],
"source": [
"# Upsert content vectors in content namespace - this can take a few minutes\n",
"print(\"Uploading vectors to content namespace..\")\n",
"for batch_df in df_batcher(article_df):\n",
" index.upsert(vectors=zip(batch_df.vector_id, batch_df.content_vector), namespace='content')"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "5fc1b083",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Uploading vectors to title namespace..\n"
]
}
],
"source": [
"# Upsert title vectors in title namespace - this can also take a few minutes\n",
"print(\"Uploading vectors to title namespace..\")\n",
"for batch_df in df_batcher(article_df):\n",
" index.upsert(vectors=zip(batch_df.vector_id, batch_df.title_vector), namespace='title')"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "f90c7fba",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'dimension': 1536,\n",
" 'index_fullness': 0.1,\n",
" 'namespaces': {'content': {'vector_count': 25000},\n",
" 'title': {'vector_count': 25000}},\n",
" 'total_vector_count': 50000}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Check index size for each namespace to confirm all of our docs have loaded\n",
"index.describe_index_stats()"
]
},
{
"cell_type": "markdown",
"id": "2da40a69",
"metadata": {},
"source": [
"### Search data\n",
"\n",
"Now we'll enter some dummy searches and check we get decent results back"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "c8280363",
"metadata": {},
"outputs": [],
"source": [
"# First we'll create dictionaries mapping vector IDs to their outputs so we can retrieve the text for our search results\n",
"titles_mapped = dict(zip(article_df.vector_id,article_df.title))\n",
"content_mapped = dict(zip(article_df.vector_id,article_df.text))"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "3c8c2aa1",
"metadata": {},
"outputs": [],
"source": [
"def query_article(query, namespace, top_k=5):\n",
" '''Queries an article using its title in the specified\n",
" namespace and prints results.'''\n",
"\n",
" # Create vector embeddings based on the title column\n",
" embedded_query = openai.Embedding.create(\n",
" input=query,\n",
" model=EMBEDDING_MODEL,\n",
" )[\"data\"][0]['embedding']\n",
"\n",
" # Query namespace passed as parameter using title vector\n",
" query_result = index.query(embedded_query, \n",
" namespace=namespace, \n",
" top_k=top_k)\n",
"\n",
" # Print query results \n",
" print(f'\\nMost similar results to {query} in \"{namespace}\" namespace:\\n')\n",
" if not query_result.matches:\n",
" print('no query result')\n",
" \n",
" matches = query_result.matches\n",
" ids = [res.id for res in matches]\n",
" scores = [res.score for res in matches]\n",
" df = pd.DataFrame({'id':ids, \n",
" 'score':scores,\n",
" 'title': [titles_mapped[_id] for _id in ids],\n",
" 'content': [content_mapped[_id] for _id in ids],\n",
" })\n",
" \n",
" counter = 0\n",
" for k,v in df.iterrows():\n",
" counter += 1\n",
" print(f'{v.title} (score = {v.score})')\n",
" \n",
" print('\\n')\n",
"\n",
" return df"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "3402b1b1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Most similar results to modern art in Europe in \"title\" namespace:\n",
"\n",
"Museum of Modern Art (score = 0.875177085)\n",
"Western Europe (score = 0.867441177)\n",
"Renaissance art (score = 0.864156306)\n",
"Pop art (score = 0.860346854)\n",
"Northern Europe (score = 0.854658186)\n",
"\n",
"\n"
]
}
],
"source": [
"query_output = query_article('modern art in Europe','title')"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "64a3f90a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Most similar results to Famous battles in Scottish history in \"content\" namespace:\n",
"\n",
"Battle of Bannockburn (score = 0.869336188)\n",
"Wars of Scottish Independence (score = 0.861470938)\n",
"1651 (score = 0.852588475)\n",
"First War of Scottish Independence (score = 0.84962213)\n",
"Robert I of Scotland (score = 0.846214116)\n",
"\n",
"\n"
]
}
],
"source": [
"content_query_output = query_article(\"Famous battles in Scottish history\",'content')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0119d87a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "vector_db_split",
"language": "python",
"name": "vector_db_split"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
},
"vscode": {
"interpreter": {
"hash": "fd16a328ca3d68029457069b79cb0b38eb39a0f5ccc4fe4473d3047707df8207"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/pinecone/Using_vision_modality_for_RAG_with_Pinecone.ipynb
+1127
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/pinecone/images/.gitignore
View File
examples/vector_databases/qdrant/Getting_started_with_Qdrant_and_OpenAI.ipynb
+727
View File
@@ -0,0 +1,727 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using Qdrant as a vector database for OpenAI embeddings\n",
"\n",
"This notebook guides you step by step on using **`Qdrant`** as a vector database for OpenAI embeddings. [Qdrant](https://qdrant.tech) is a high-performant vector search database written in Rust. It offers RESTful and gRPC APIs to manage your embeddings. There is an official Python [qdrant-client](https://github.com/qdrant/qdrant_client) that eases the integration with your apps.\n",
"\n",
"This notebook presents an end-to-end process of:\n",
"1. Using precomputed embeddings created by OpenAI API.\n",
"2. Storing the embeddings in a local instance of Qdrant.\n",
"3. Converting raw text query to an embedding with OpenAI API.\n",
"4. Using Qdrant to perform the nearest neighbour search in the created collection.\n",
"\n",
"### What is Qdrant\n",
"\n",
"[Qdrant](https://qdrant.tech) is an Open Source vector database that allows storing neural embeddings along with the metadata, a.k.a [payload](https://qdrant.tech/documentation/payload/). Payloads are not only available for keeping some additional attributes of a particular point, but might be also used for filtering. [Qdrant](https://qdrant.tech) offers a unique filtering mechanism which is built-in into the vector search phase, what makes it really efficient.\n",
"\n",
"### Deployment options\n",
"\n",
"[Qdrant](https://qdrant.tech) might be launched in various ways, depending on the target load on the application it might be hosted:\n",
"\n",
"- Locally or on premise, with Docker containers\n",
"- On Kubernetes cluster, with the [Helm chart](https://github.com/qdrant/qdrant-helm)\n",
"- Using [Qdrant Cloud](https://cloud.qdrant.io/)\n",
"\n",
"### Integration\n",
"\n",
"[Qdrant](https://qdrant.tech) provides both RESTful and gRPC APIs which makes integration easy, no matter the programming language you use. However, there are some official clients for the most popular languages available, and if you use Python then the [Python Qdrant client library](https://github.com/qdrant/qdrant_client) might be the best choice."
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"For the purposes of this exercise we need to prepare a couple of things:\n",
"\n",
"1. Qdrant server instance. In our case a local Docker container.\n",
"2. The [qdrant-client](https://github.com/qdrant/qdrant_client) library to interact with the vector database.\n",
"3. An [OpenAI API key](https://platform.openai.com/settings/organization/api-keys).\n",
"\n",
"### Start Qdrant server\n",
"\n",
"We're going to use a local Qdrant instance running in a Docker container. The easiest way to launch it is to use the attached [docker-compose.yaml] file and run the following command:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:04:28.198036Z",
"start_time": "2023-02-16T12:04:26.950014Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/0\n",
" \u001b[32m✔\u001b[0m Container qdrant-qdrant-1 \u001b[32mRunning\u001b[0m \u001b[34m0.0s \u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"! docker compose up -d"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We might validate if the server was launched successfully by running a simple curl command:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:04:29.301651Z",
"start_time": "2023-02-16T12:04:29.173153Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"title\":\"qdrant - vector search engine\",\"version\":\"1.3.0\"}"
]
}
],
"source": [
"! curl http://localhost:6333"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install requirements\n",
"\n",
"This notebook obviously requires the `openai` and `qdrant-client` packages, but there are also some other additional libraries we will use. The following command installs them all:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.718972Z",
"start_time": "2023-02-16T12:04:30.434820Z"
}
},
"outputs": [],
"source": [
"! pip install openai qdrant-client pandas wget"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare your OpenAI API key\n",
"\n",
"The OpenAI API key is used for vectorization of the documents and queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by running following command:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"! export OPENAI_API_KEY=\"your API key\""
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.730338Z",
"start_time": "2023-02-16T12:05:05.723351Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print(\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print(\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to Qdrant\n",
"\n",
"Connecting to a running instance of Qdrant server is easy with the official Python library:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.827143Z",
"start_time": "2023-02-16T12:05:05.733771Z"
}
},
"outputs": [],
"source": [
"import qdrant_client\n",
"\n",
"client = qdrant_client.QdrantClient(\n",
" host=\"localhost\",\n",
" prefer_grpc=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can test the connection by running any available method:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.848488Z",
"start_time": "2023-02-16T12:05:06.832612Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CollectionsResponse(collections=[])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.get_collections()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:37.371951Z",
"start_time": "2023-02-16T12:05:06.851634Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100% [......................................................................] 698933052 / 698933052"
]
},
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded (9).zip'"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The downloaded file has to be then extracted:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:06:01.538851Z",
"start_time": "2023-02-16T12:05:37.376042Z"
}
},
"outputs": [],
"source": [
"import zipfile\n",
"\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we can finally load it from the provided CSV file:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:17:35.483972Z",
"start_time": "2023-02-16T12:06:01.540172Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"from ast import literal_eval\n",
"\n",
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n",
"# Read vectors from strings back into a list\n",
"article_df[\"title_vector\"] = article_df.title_vector.apply(literal_eval)\n",
"article_df[\"content_vector\"] = article_df.content_vector.apply(literal_eval)\n",
"article_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index data\n",
"\n",
"Qdrant stores data in __collections__ where each object is described by at least one vector and may contain an additional metadata called __payload__. Our collection will be called **Articles** and each object will be described by both **title** and **content** vectors. Qdrant does not require you to set up any kind of schema beforehand, so you can freely put points to the collection with a simple setup only.\n",
"\n",
"We will start with creating a collection, and then we will fill it with our precomputed embeddings."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:17:36.366066Z",
"start_time": "2023-02-16T12:17:35.486872Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from qdrant_client.http import models as rest\n",
"\n",
"vector_size = len(article_df[\"content_vector\"][0])\n",
"\n",
"client.create_collection(\n",
" collection_name=\"Articles\",\n",
" vectors_config={\n",
" \"title\": rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" \"content\": rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" }\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:37.518210Z",
"start_time": "2023-02-16T12:17:36.368564Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"UpdateResult(operation_id=0, status=<UpdateStatus.COMPLETED: 'completed'>)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.upsert(\n",
" collection_name=\"Articles\",\n",
" points=[\n",
" rest.PointStruct(\n",
" id=k,\n",
" vector={\n",
" \"title\": v[\"title_vector\"],\n",
" \"content\": v[\"content_vector\"],\n",
" },\n",
" payload=v.to_dict(),\n",
" )\n",
" for k, v in article_df.iterrows()\n",
" ],\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.675202Z",
"start_time": "2023-02-16T12:30:40.655654Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CountResult(count=25000)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Check the collection size to make sure all the points have been stored\n",
"client.count(collection_name=\"Articles\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search data\n",
"\n",
"Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-ada-002` OpenAI model we also have to use it during search.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:38.024370Z",
"start_time": "2023-02-16T12:30:37.712816Z"
}
},
"outputs": [],
"source": [
"from openai import OpenAI\n",
"\n",
"openai_client = OpenAI()\n",
"\n",
"def query_qdrant(query, collection_name, vector_name=\"title\", top_k=20):\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai_client.embeddings.create(\n",
" input=query,\n",
" model=\"text-embedding-ada-002\",\n",
" ).data[0].embedding\n",
"\n",
" query_results = client.search(\n",
" collection_name=collection_name,\n",
" query_vector=(\n",
" vector_name, embedded_query\n",
" ),\n",
" limit=top_k,\n",
" )\n",
"\n",
" return query_results"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:39.379566Z",
"start_time": "2023-02-16T12:30:38.031041Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art (Score: 0.875)\n",
"2. Western Europe (Score: 0.867)\n",
"3. Renaissance art (Score: 0.864)\n",
"4. Pop art (Score: 0.86)\n",
"5. Northern Europe (Score: 0.855)\n",
"6. Hellenistic art (Score: 0.853)\n",
"7. Modernist literature (Score: 0.847)\n",
"8. Art film (Score: 0.843)\n",
"9. Central Europe (Score: 0.843)\n",
"10. European (Score: 0.841)\n",
"11. Art (Score: 0.841)\n",
"12. Byzantine art (Score: 0.841)\n",
"13. Postmodernism (Score: 0.84)\n",
"14. Eastern Europe (Score: 0.839)\n",
"15. Cubism (Score: 0.839)\n",
"16. Europe (Score: 0.839)\n",
"17. Impressionism (Score: 0.838)\n",
"18. Bauhaus (Score: 0.838)\n",
"19. Surrealism (Score: 0.837)\n",
"20. Expressionism (Score: 0.837)\n"
]
}
],
"source": [
"query_results = query_qdrant(\"modern art in Europe\", \"Articles\")\n",
"for i, article in enumerate(query_results):\n",
" print(f\"{i + 1}. {article.payload['title']} (Score: {round(article.score, 3)})\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.652676Z",
"start_time": "2023-02-16T12:30:39.382555Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn (Score: 0.869)\n",
"2. Wars of Scottish Independence (Score: 0.861)\n",
"3. 1651 (Score: 0.852)\n",
"4. First War of Scottish Independence (Score: 0.85)\n",
"5. Robert I of Scotland (Score: 0.846)\n",
"6. 841 (Score: 0.844)\n",
"7. 1716 (Score: 0.844)\n",
"8. 1314 (Score: 0.837)\n",
"9. 1263 (Score: 0.836)\n",
"10. William Wallace (Score: 0.835)\n",
"11. Stirling (Score: 0.831)\n",
"12. 1306 (Score: 0.831)\n",
"13. 1746 (Score: 0.83)\n",
"14. 1040s (Score: 0.828)\n",
"15. 1106 (Score: 0.827)\n",
"16. 1304 (Score: 0.826)\n",
"17. David II of Scotland (Score: 0.825)\n",
"18. Braveheart (Score: 0.824)\n",
"19. 1124 (Score: 0.824)\n",
"20. Second War of Scottish Independence (Score: 0.823)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_results = query_qdrant(\"Famous battles in Scottish history\", \"Articles\", \"content\")\n",
"for i, article in enumerate(query_results):\n",
" print(f\"{i + 1}. {article.payload['title']} (Score: {round(article.score, 3)})\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb
+561
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb
+790
View File
@@ -0,0 +1,790 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Qdrant for Embeddings Search\n",
"\n",
"This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"### What is a Vector Database\n",
"\n",
"A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
"\n",
"### Why use a Vector Database\n",
"\n",
"Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
"\n",
"\n",
"### Demo Flow\n",
"The demo flow is:\n",
"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **Qdrant**\n",
" - *Setup*: Here we'll set up the Python client for Qdrant. For more details go [here](https://github.com/qdrant/qdrant_client)\n",
" - *Index Data*: We'll create a collection with vectors for __titles__ and __content__\n",
" - *Search Data*: We'll run a few searches to confirm it works\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "e2b59250",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Import the required libraries and set the embedding model that we'd like to use."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d8810f9",
"metadata": {},
"outputs": [],
"source": [
"# We'll need to install Qdrant client\n",
"!pip install qdrant-client"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "5be94df6",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:49:06.926613Z",
"start_time": "2024-05-21T23:49:06.923221Z"
}
},
"outputs": [],
"source": [
"import openai\n",
"import pandas as pd\n",
"from ast import literal_eval\n",
"import qdrant_client # Qdrant's client library for Python\n",
"\n",
"# This can be changed to the embedding model of your choice. Make sure its the same model that is used for generating embeddings\n",
"EMBEDDING_MODEL = \"text-embedding-ada-002\"\n",
"\n",
"# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that we've prepared previous to this session."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "5dff8b55",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:49:54.889503Z",
"start_time": "2024-05-21T23:49:41.132888Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100% [......................................................................] 698933052 / 698933052"
]
},
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded (10).zip'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "21097972",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:50:56.268540Z",
"start_time": "2024-05-21T23:50:53.171125Z"
}
},
"outputs": [],
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "70bbd8ba",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:51:08.388674Z",
"start_time": "2024-05-21T23:50:57.592940Z"
}
},
"outputs": [],
"source": [
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "1721e45d",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:51:13.706819Z",
"start_time": "2024-05-21T23:51:13.700231Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "960b82af",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:20.588010Z",
"start_time": "2024-05-21T23:51:16.274336Z"
}
},
"outputs": [],
"source": [
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
"\n",
"# Set vector_id to be a string\n",
"article_df['vector_id'] = article_df['vector_id'].apply(str)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "a334ab8b",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:36.075327Z",
"start_time": "2024-05-21T23:55:36.038710Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 25000 entries, 0 to 24999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 25000 non-null int64 \n",
" 1 url 25000 non-null object\n",
" 2 title 25000 non-null object\n",
" 3 text 25000 non-null object\n",
" 4 title_vector 25000 non-null object\n",
" 5 content_vector 25000 non-null object\n",
" 6 vector_id 25000 non-null object\n",
"dtypes: int64(1), object(6)\n",
"memory usage: 1.3+ MB\n"
]
}
],
"source": [
"article_df.info(show_counts=True)"
]
},
{
"cell_type": "markdown",
"id": "9cfaed9d",
"metadata": {},
"source": [
"## Qdrant\n",
"\n",
"**[Qdrant](https://qdrant.tech/)**. is a high-performant vector search database written in Rust. It offers both on-premise and cloud version, but for the purposes of that example we're going to use the local deployment mode.\n",
"\n",
"Setting everything up will require:\n",
"- Spinning up a local instance of Qdrant\n",
"- Configuring the collection and storing the data in it\n",
"- Trying out with some queries"
]
},
{
"cell_type": "markdown",
"id": "38774565",
"metadata": {},
"source": [
"### Setup\n",
"\n",
"For the local deployment, we are going to use Docker, according to the Qdrant documentation: https://qdrant.tech/documentation/quick_start/. Qdrant requires just a single container, but an example of the docker-compose.yaml file is available at `./qdrant/docker-compose.yaml` in this repo.\n",
"\n",
"You can start Qdrant instance locally by navigating to this directory and running `docker-compose up -d `\n",
"\n",
"> You might need to increase the memory limit for Docker to 8GB or more. Or Qdrant might fail to execute with an error message like `7 Killed`.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "fc9a0203-30c2-4a7c-95cb-c5bacc908a4d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/0\n",
" \u001b[32m✔\u001b[0m Container qdrant-qdrant-1 \u001b[32mRunning\u001b[0m \u001b[34m0.0s \u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"! docker compose up -d"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "76d697e9",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:56.550765Z",
"start_time": "2024-05-21T23:55:56.517724Z"
}
},
"outputs": [],
"source": [
"qdrant = qdrant_client.QdrantClient(host=\"localhost\", port=6333)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "1deeb539",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:57.340006Z",
"start_time": "2024-05-21T23:55:57.312830Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CollectionsResponse(collections=[CollectionDescription(name='Articles')])"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"qdrant.get_collections()"
]
},
{
"cell_type": "markdown",
"id": "bc006b6f",
"metadata": {},
"source": [
"### Index data\n",
"\n",
"Qdrant stores data in __collections__ where each object is described by at least one vector and may contain an additional metadata called __payload__. Our collection will be called **Articles** and each object will be described by both **title** and **content** vectors.\n",
"\n",
"We'll be using an official [qdrant-client](https://github.com/qdrant/qdrant_client) package that has all the utility methods already built-in."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "1a84ee1d",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:56:04.066640Z",
"start_time": "2024-05-21T23:56:04.064878Z"
}
},
"outputs": [],
"source": [
"from qdrant_client.http import models as rest"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "00876f92",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:56:05.462165Z",
"start_time": "2024-05-21T23:56:05.247948Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get the vector size from the first row to set up the collection\n",
"vector_size = len(article_df['content_vector'][0])\n",
"\n",
"# Set up the collection with the vector configuration. You need to declare the vector size and distance metric for the collection. Distance metric enables vector database to index and search vectors efficiently.\n",
"qdrant.recreate_collection(\n",
" collection_name='Articles',\n",
" vectors_config={\n",
" 'title': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" 'content': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" }\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "9f39a8c395554ca3",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:56:21.577594Z",
"start_time": "2024-05-21T23:56:21.460740Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vector_size = len(article_df['content_vector'][0])\n",
"\n",
"qdrant.recreate_collection(\n",
" collection_name='Articles',\n",
" vectors_config={\n",
" 'title': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" 'content': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"id": "e95be6e0c9af4c21",
"metadata": {},
"source": [
"In addition to the vector configuration defined under `vector`, we can also define the `payload` configuration. Payload is an optional field that allows you to store additional metadata alongside the vectors. In our case, we'll store the `id`, `title`, and `url` of the articles. As we return the title of nearest articles in the search results from payload, we can also provide the user with the URL to the article (which is part of the meta-data)."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "f24e76ab",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:25.183855Z",
"start_time": "2024-05-21T23:56:50.664145Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Upserting articles: 100%|█████████████████████████████████████████████████████████████████████████████████████| 25000/25000 [02:52<00:00, 144.82it/s]\n"
]
}
],
"source": [
"from qdrant_client.models import PointStruct # Import the PointStruct to store the vector and payload\n",
"from tqdm import tqdm # Library to show the progress bar \n",
"\n",
"# Populate collection with vectors using tqdm to show progress\n",
"for k, v in tqdm(article_df.iterrows(), desc=\"Upserting articles\", total=len(article_df)):\n",
" try:\n",
" qdrant.upsert(\n",
" collection_name='Articles',\n",
" points=[\n",
" PointStruct(\n",
" id=k,\n",
" vector={'title': v['title_vector'], \n",
" 'content': v['content_vector']},\n",
" payload={\n",
" 'id': v['id'],\n",
" 'title': v['title'],\n",
" 'url': v['url']\n",
" }\n",
" )\n",
" ]\n",
" )\n",
" except Exception as e:\n",
" print(f\"Failed to upsert row {k}: {v}\")\n",
" print(f\"Exception: {e}\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "d1188a12",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:27.558407Z",
"start_time": "2024-05-21T23:58:27.549740Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CountResult(count=25000)"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Check the collection size to make sure all the points have been stored\n",
"qdrant.count(collection_name='Articles')"
]
},
{
"cell_type": "markdown",
"id": "06ed119b",
"metadata": {},
"source": [
"### Search Data\n",
"\n",
"Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Ensure you use the text-embedding-ada-002 model as the original embeddings in file were created with this model."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "f1bac4ef",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:35.492725Z",
"start_time": "2024-05-21T23:58:35.488963Z"
}
},
"outputs": [],
"source": [
"def query_qdrant(query, collection_name, vector_name='title', top_k=20):\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.embeddings.create(\n",
" input=query,\n",
" model=EMBEDDING_MODEL,\n",
" ).data[0].embedding # We take the first embedding from the list\n",
" \n",
" query_results = qdrant.search(\n",
" collection_name=collection_name,\n",
" query_vector=(\n",
" vector_name, embedded_query\n",
" ),\n",
" limit=top_k, \n",
" query_filter=None\n",
" )\n",
" \n",
" return query_results"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "aa92f3d3",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:37.183718Z",
"start_time": "2024-05-21T23:58:36.949491Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art, URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art (Score: 0.875)\n",
"2. Western Europe, URL: https://simple.wikipedia.org/wiki/Western%20Europe (Score: 0.867)\n",
"3. Renaissance art, URL: https://simple.wikipedia.org/wiki/Renaissance%20art (Score: 0.864)\n",
"4. Pop art, URL: https://simple.wikipedia.org/wiki/Pop%20art (Score: 0.86)\n",
"5. Northern Europe, URL: https://simple.wikipedia.org/wiki/Northern%20Europe (Score: 0.855)\n",
"6. Hellenistic art, URL: https://simple.wikipedia.org/wiki/Hellenistic%20art (Score: 0.853)\n",
"7. Modernist literature, URL: https://simple.wikipedia.org/wiki/Modernist%20literature (Score: 0.847)\n",
"8. Art film, URL: https://simple.wikipedia.org/wiki/Art%20film (Score: 0.843)\n",
"9. Central Europe, URL: https://simple.wikipedia.org/wiki/Central%20Europe (Score: 0.843)\n",
"10. European, URL: https://simple.wikipedia.org/wiki/European (Score: 0.841)\n",
"11. Art, URL: https://simple.wikipedia.org/wiki/Art (Score: 0.841)\n",
"12. Byzantine art, URL: https://simple.wikipedia.org/wiki/Byzantine%20art (Score: 0.841)\n",
"13. Postmodernism, URL: https://simple.wikipedia.org/wiki/Postmodernism (Score: 0.84)\n",
"14. Eastern Europe, URL: https://simple.wikipedia.org/wiki/Eastern%20Europe (Score: 0.839)\n",
"15. Cubism, URL: https://simple.wikipedia.org/wiki/Cubism (Score: 0.839)\n",
"16. Europe, URL: https://simple.wikipedia.org/wiki/Europe (Score: 0.839)\n",
"17. Impressionism, URL: https://simple.wikipedia.org/wiki/Impressionism (Score: 0.838)\n",
"18. Bauhaus, URL: https://simple.wikipedia.org/wiki/Bauhaus (Score: 0.838)\n",
"19. Surrealism, URL: https://simple.wikipedia.org/wiki/Surrealism (Score: 0.837)\n",
"20. Expressionism, URL: https://simple.wikipedia.org/wiki/Expressionism (Score: 0.837)\n"
]
}
],
"source": [
"query_results = query_qdrant('modern art in Europe', 'Articles', 'title')\n",
"for i, article in enumerate(query_results):\n",
" print(f'{i + 1}. {article.payload[\"title\"]}, URL: {article.payload[\"url\"]} (Score: {round(article.score, 3)})')"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "7ed116b8",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:53.144123Z",
"start_time": "2024-05-21T23:58:52.924091Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn, URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn (Score: 0.869)\n",
"2. Wars of Scottish Independence, URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence (Score: 0.861)\n",
"3. 1651, URL: https://simple.wikipedia.org/wiki/1651 (Score: 0.852)\n",
"4. First War of Scottish Independence, URL: https://simple.wikipedia.org/wiki/First%20War%20of%20Scottish%20Independence (Score: 0.85)\n",
"5. Robert I of Scotland, URL: https://simple.wikipedia.org/wiki/Robert%20I%20of%20Scotland (Score: 0.846)\n",
"6. 841, URL: https://simple.wikipedia.org/wiki/841 (Score: 0.844)\n",
"7. 1716, URL: https://simple.wikipedia.org/wiki/1716 (Score: 0.844)\n",
"8. 1314, URL: https://simple.wikipedia.org/wiki/1314 (Score: 0.837)\n",
"9. 1263, URL: https://simple.wikipedia.org/wiki/1263 (Score: 0.836)\n",
"10. William Wallace, URL: https://simple.wikipedia.org/wiki/William%20Wallace (Score: 0.835)\n",
"11. Stirling, URL: https://simple.wikipedia.org/wiki/Stirling (Score: 0.831)\n",
"12. 1306, URL: https://simple.wikipedia.org/wiki/1306 (Score: 0.831)\n",
"13. 1746, URL: https://simple.wikipedia.org/wiki/1746 (Score: 0.83)\n",
"14. 1040s, URL: https://simple.wikipedia.org/wiki/1040s (Score: 0.828)\n",
"15. 1106, URL: https://simple.wikipedia.org/wiki/1106 (Score: 0.827)\n",
"16. 1304, URL: https://simple.wikipedia.org/wiki/1304 (Score: 0.826)\n",
"17. David II of Scotland, URL: https://simple.wikipedia.org/wiki/David%20II%20of%20Scotland (Score: 0.825)\n",
"18. Braveheart, URL: https://simple.wikipedia.org/wiki/Braveheart (Score: 0.824)\n",
"19. 1124, URL: https://simple.wikipedia.org/wiki/1124 (Score: 0.824)\n",
"20. July 27, URL: https://simple.wikipedia.org/wiki/July%2027 (Score: 0.823)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_results = query_qdrant('Famous battles in Scottish history', 'Articles', 'content')\n",
"for i, article in enumerate(query_results):\n",
" print(f'{i + 1}. {article.payload[\"title\"]}, URL: {article.payload[\"url\"]} (Score: {round(article.score, 3)})')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
},
"vscode": {
"interpreter": {
"hash": "fd16a328ca3d68029457069b79cb0b38eb39a0f5ccc4fe4473d3047707df8207"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/qdrant/docker-compose.yaml
+8
View File
@@ -0,0 +1,8 @@
version: '3.4'
services:
qdrant:
image: qdrant/qdrant:v1.3.0
restart: on-failure
ports:
- "6333:6333"
- "6334:6334"
examples/vector_databases/redis/README.md
+126
View File
@@ -0,0 +1,126 @@
# Redis
### What is Redis?
Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.
In addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new capabilities, commands and data types. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).
### Deployment options
There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.
For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.
Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.
### What is RediSearch?
RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/).
### Features
RediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:
* Full-Text indexing of multiple fields in Redis hashes
* Incremental indexing without performance loss
* Vector similarity search
* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)
* Field weighting
* Complex boolean queries with AND, OR, and NOT operators
* Prefix matching, fuzzy matching, and exact-phrase queries
* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)
* Auto-complete suggestions (with fuzzy prefix suggestions)
* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))
* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))
* Numeric filters and ranges
* Geospatial searches using [Redis geospatial indexing](/commands/georadius)
* A powerful aggregations engine
* Supports for all utf-8 encoded text
* Retrieve full documents, selected fields, or only the document IDs
* Sorting results (for example, by creation date)
* JSON support through RedisJSON
### Clients
Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).
| Project | Language | License | Author | Stars |
|----------|---------|--------|---------|-------|
| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |
| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |
| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |
| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |
[redis-url]: https://redis.com
[redis-py-url]: https://github.com/redis/redis-py
[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000
[redis-py-package]: https://pypi.python.org/pypi/redis
[jedis-url]: https://github.com/redis/jedis
[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis
[nredisstack-url]: https://github.com/redis/nredisstack
[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000
[nredisstack-package]: https://www.nuget.org/packages/nredisstack/
[node-redis-url]: https://github.com/redis/node-redis
[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[node-redis-package]: https://www.npmjs.com/package/redis
[redis-om-python-url]: https://github.com/redis/redis-om-python
[redis-om-python-author]: https://redis.com
[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000
[redisearch-go-url]: https://github.com/RediSearch/redisearch-go
[redisearch-go-author]: https://redis.com
[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000
[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs
[redisearch-api-rs-author]: https://redis.com
[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000
### Deployment Options
There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment such as
- [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/)
- Cloud marketplaces: [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)
- On-premise: [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/)
- Kubernetes: [Redis Enterprise Software on Kubernetes](https://docs.redis.com/latest/kubernetes/)
- [Docker (RediSearch)](https://hub.docker.com/r/redislabs/redisearch)
- [Docker (Redis Stack)](https://hub.docker.com/r/redis/redis-stack)
### Cluster support
RediSearch has a distributed cluster version that scales to billions of documents across hundreds of servers. At the moment, distributed RediSearch is available as part of [Redis Enterprise Cloud](https://redis.com/redis-enterprise-cloud/overview/) and [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/).
See [RediSearch on Redis Enterprise](https://redis.com/modules/redisearch/) for more information.
### Examples
- [Product Search](https://github.com/RedisVentures/redis-product-search) - eCommerce product search (with image and text)
- [Product Recommendations with DocArray / Jina](https://github.com/jina-ai/product-recommendation-redis-docarray) - Content-based product recommendations example with Redis and DocArray.
- [Redis VSS in RecSys](https://github.com/RedisVentures/Redis-Recsys) - 3 end-to-end Redis & NVIDIA Merlin Recommendation System Architectures.
- [Azure OpenAI Embeddings Q&A](https://github.com/ruoccofabrizio/azure-open-ai-embeddings-qna) - OpenAI and Redis as a Q&A service on Azure.
- [ArXiv Paper Search](https://github.com/RedisVentures/redis-arXiv-search) - Semantic search over arXiv scholarly papers
### More Resources
For more information on how to use Redis as a vector database, check out the following resources:
- [Redis Vector Similarity Docs](https://redis.io/docs/stack/search/reference/vectors/) - Redis official docs for Vector Search.
- [Redis-py Search Docs](https://redis.readthedocs.io/en/latest/redismodules.html#redisearch-commands) - Redis-py client library docs for RediSearch.
- [Vector Similarity Search: From Basics to Production](https://mlops.community/vector-similarity-search-from-basics-to-production/) - Introductory blog post to VSS and Redis as a VectorDB.
- [AI-Powered Document Search](https://datasciencedojo.com/blog/ai-powered-document-search/) - Blog post covering AI Powered Document Search Use Cases & Architectures.
- [Vector Database Benchmarks](https://jina.ai/news/benchmark-vector-search-databases-with-one-million-data/) - Jina AI VectorDB benchmarks comparing Redis against others.
examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb
+799
View File
@@ -0,0 +1,799 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Redis for Embeddings Search\n",
"\n",
"This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"### What is a Vector Database\n",
"\n",
"A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
"\n",
"### Why use a Vector Database\n",
"\n",
"Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
"\n",
"\n",
"### Demo Flow\n",
"The demo flow is:\n",
"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **Redis**\n",
" - *Setup*: Set up the Redis-Py client. For more details go [here](https://github.com/redis/redis-py)\n",
" - *Index Data*: Create the search index for vector search and hybrid search (vector + full-text search) on all available fields.\n",
" - *Search Data*: Run a few example queries with various goals in mind.\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "e2b59250",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Import the required libraries and set the embedding model that we'd like to use."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d8810f9",
"metadata": {},
"outputs": [],
"source": [
"# We'll need to install the Redis client\n",
"!pip install redis\n",
"\n",
"#Install wget to pull zip file\n",
"!pip install wget"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5be94df6",
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"from typing import List, Iterator\n",
"import pandas as pd\n",
"import numpy as np\n",
"import os\n",
"import wget\n",
"from ast import literal_eval\n",
"\n",
"# Redis client library for Python\n",
"import redis\n",
"\n",
"# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"\n",
"# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that we've prepared previous to this session."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5dff8b55",
"metadata": {},
"outputs": [],
"source": [
"embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21097972",
"metadata": {},
"outputs": [],
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "70bbd8ba",
"metadata": {},
"outputs": [],
"source": [
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1721e45d",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "960b82af",
"metadata": {},
"outputs": [],
"source": [
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
"\n",
"# Set vector_id to be a string\n",
"article_df['vector_id'] = article_df['vector_id'].apply(str)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a334ab8b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 25000 entries, 0 to 24999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 25000 non-null int64 \n",
" 1 url 25000 non-null object\n",
" 2 title 25000 non-null object\n",
" 3 text 25000 non-null object\n",
" 4 title_vector 25000 non-null object\n",
" 5 content_vector 25000 non-null object\n",
" 6 vector_id 25000 non-null object\n",
"dtypes: int64(1), object(6)\n",
"memory usage: 1.3+ MB\n"
]
}
],
"source": [
"article_df.info(show_counts=True)"
]
},
{
"cell_type": "markdown",
"id": "43bffd04",
"metadata": {},
"source": [
"# Redis\n",
"\n",
"The next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.\n",
"\n",
"Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n",
"\n",
"| Project | Language | License | Author | Stars |\n",
"|----------|---------|--------|---------|-------|\n",
"| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n",
"| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n",
"| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n",
"| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n",
"| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |\n",
"| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [![redisearch-api-rs-stars]][redisearch-api-rs-url] |\n",
"\n",
"[redis-url]: https://redis.com\n",
"\n",
"[redis-py-url]: https://github.com/redis/redis-py\n",
"[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
"[redis-py-package]: https://pypi.python.org/pypi/redis\n",
"\n",
"[jedis-url]: https://github.com/redis/jedis\n",
"[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
"[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis\n",
"\n",
"[nredisstack-url]: https://github.com/redis/nredisstack\n",
"[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
"[nredisstack-package]: https://www.nuget.org/packages/nredisstack/\n",
"\n",
"[node-redis-url]: https://github.com/redis/node-redis\n",
"[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
"[node-redis-package]: https://www.npmjs.com/package/redis\n",
"\n",
"[redis-om-python-url]: https://github.com/redis/redis-om-python\n",
"[redis-om-python-author]: https://redis.com\n",
"[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
"\n",
"[redisearch-go-url]: https://github.com/RediSearch/redisearch-go\n",
"[redisearch-go-author]: https://redis.com\n",
"[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
"\n",
"[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs\n",
"[redisearch-api-rs-author]: https://redis.com\n",
"[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
"\n",
"\n",
"In the below cells, we will walk you through using Redis as a vector database. Since many of you are likely already used to the Redis API, this should be familiar to most."
]
},
{
"cell_type": "markdown",
"id": "698e24f6",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment. For other deployment options, see the [redis directory](./redis) in this repo.\n",
"\n",
"For this tutorial, we will use Redis Stack on Docker.\n",
"\n",
"Start a version of Redis with RediSearch (Redis Stack) by running the following docker command\n",
"\n",
"```bash\n",
"$ cd redis\n",
"$ docker compose up -d\n",
"```\n",
"This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n",
"\n",
"You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d2ce669a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import redis\n",
"from redis.commands.search.indexDefinition import (\n",
" IndexDefinition,\n",
" IndexType\n",
")\n",
"from redis.commands.search.query import Query\n",
"from redis.commands.search.field import (\n",
" TextField,\n",
" VectorField\n",
")\n",
"\n",
"REDIS_HOST = \"localhost\"\n",
"REDIS_PORT = 6379\n",
"REDIS_PASSWORD = \"\" # default for passwordless Redis\n",
"\n",
"# Connect to Redis\n",
"redis_client = redis.Redis(\n",
" host=REDIS_HOST,\n",
" port=REDIS_PORT,\n",
" password=REDIS_PASSWORD\n",
")\n",
"redis_client.ping()"
]
},
{
"cell_type": "markdown",
"id": "3f6f0af9",
"metadata": {},
"source": [
"## Creating a Search Index\n",
"\n",
"The below cells will show how to specify and create a search index in Redis. We will\n",
"\n",
"1. Set some constants for defining our index like the distance metric and the index name\n",
"2. Define the index schema with RediSearch fields\n",
"3. Create the index\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a7c64cb9",
"metadata": {},
"outputs": [],
"source": [
"# Constants\n",
"VECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors\n",
"VECTOR_NUMBER = len(article_df) # initial number of vectors\n",
"INDEX_NAME = \"embeddings-index\" # name of the search index\n",
"PREFIX = \"doc\" # prefix for the document keys\n",
"DISTANCE_METRIC = \"COSINE\" # distance metric for the vectors (ex. COSINE, IP, L2)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "d95fcd06",
"metadata": {},
"outputs": [],
"source": [
"# Define RediSearch fields for each of the columns in the dataset\n",
"title = TextField(name=\"title\")\n",
"url = TextField(name=\"url\")\n",
"text = TextField(name=\"text\")\n",
"title_embedding = VectorField(\"title_vector\",\n",
" \"FLAT\", {\n",
" \"TYPE\": \"FLOAT32\",\n",
" \"DIM\": VECTOR_DIM,\n",
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
" \"INITIAL_CAP\": VECTOR_NUMBER,\n",
" }\n",
")\n",
"text_embedding = VectorField(\"content_vector\",\n",
" \"FLAT\", {\n",
" \"TYPE\": \"FLOAT32\",\n",
" \"DIM\": VECTOR_DIM,\n",
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
" \"INITIAL_CAP\": VECTOR_NUMBER,\n",
" }\n",
")\n",
"fields = [title, url, text, title_embedding, text_embedding]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "7418480d",
"metadata": {},
"outputs": [],
"source": [
"# Check if index exists\n",
"try:\n",
" redis_client.ft(INDEX_NAME).info()\n",
" print(\"Index already exists\")\n",
"except:\n",
" # Create RediSearch Index\n",
" redis_client.ft(INDEX_NAME).create_index(\n",
" fields = fields,\n",
" definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "f3563eec",
"metadata": {},
"source": [
"## Load Documents into the Index\n",
"\n",
"Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "e98d63ad",
"metadata": {},
"outputs": [],
"source": [
"def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n",
" records = documents.to_dict(\"records\")\n",
" for doc in records:\n",
" key = f\"{prefix}:{str(doc['id'])}\"\n",
"\n",
" # create byte vectors for title and content\n",
" title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n",
" content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n",
"\n",
" # replace list of floats with byte vectors\n",
" doc[\"title_vector\"] = title_embedding\n",
" doc[\"content_vector\"] = content_embedding\n",
"\n",
" client.hset(key, mapping = doc)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "098d3c5a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 25000 documents in Redis search index with name: embeddings-index\n"
]
}
],
"source": [
"index_documents(redis_client, PREFIX, article_df)\n",
"print(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")"
]
},
{
"cell_type": "markdown",
"id": "f646bff4",
"metadata": {},
"source": [
"## Running Search Queries\n",
"\n",
"Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.\n",
"\n",
"1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.\n",
"2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "508d1f89",
"metadata": {},
"outputs": [],
"source": [
"def search_redis(\n",
" redis_client: redis.Redis,\n",
" user_query: str,\n",
" index_name: str = \"embeddings-index\",\n",
" vector_field: str = \"title_vector\",\n",
" return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n",
" hybrid_fields = \"*\",\n",
" k: int = 20,\n",
") -> List[dict]:\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.Embedding.create(input=user_query,\n",
" model=EMBEDDING_MODEL,\n",
" )[\"data\"][0]['embedding']\n",
"\n",
" # Prepare the Query\n",
" base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n",
" query = (\n",
" Query(base_query)\n",
" .return_fields(*return_fields)\n",
" .sort_by(\"vector_score\")\n",
" .paging(0, k)\n",
" .dialect(2)\n",
" )\n",
" params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n",
"\n",
" # perform vector search\n",
" results = redis_client.ft(index_name).search(query, params_dict)\n",
" for i, article in enumerate(results.docs):\n",
" score = 1 - float(article.vector_score)\n",
" print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n",
" return results.docs"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "1f0eef07",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Museum of Modern Art (Score: 0.875)\n",
"1. Western Europe (Score: 0.867)\n",
"2. Renaissance art (Score: 0.864)\n",
"3. Pop art (Score: 0.86)\n",
"4. Northern Europe (Score: 0.855)\n",
"5. Hellenistic art (Score: 0.853)\n",
"6. Modernist literature (Score: 0.847)\n",
"7. Art film (Score: 0.843)\n",
"8. Central Europe (Score: 0.843)\n",
"9. European (Score: 0.841)\n"
]
}
],
"source": [
"# For using OpenAI to generate query embedding\n",
"openai.api_key = os.getenv(\"OPENAI_API_KEY\", \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\n",
"results = search_redis(redis_client, 'modern art in Europe', k=10)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "7b805a81",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Battle of Bannockburn (Score: 0.869)\n",
"1. Wars of Scottish Independence (Score: 0.861)\n",
"2. 1651 (Score: 0.853)\n",
"3. First War of Scottish Independence (Score: 0.85)\n",
"4. Robert I of Scotland (Score: 0.846)\n",
"5. 841 (Score: 0.844)\n",
"6. 1716 (Score: 0.844)\n",
"7. 1314 (Score: 0.837)\n",
"8. 1263 (Score: 0.836)\n",
"9. William Wallace (Score: 0.835)\n"
]
}
],
"source": [
"results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)"
]
},
{
"cell_type": "markdown",
"id": "0ed0b34e",
"metadata": {},
"source": [
"## Hybrid Queries with Redis\n",
"\n",
"The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "c94d5cce",
"metadata": {},
"outputs": [],
"source": [
"def create_hybrid_field(field_name: str, value: str) -> str:\n",
" return f'@{field_name}:\"{value}\"'"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "bfcd31c2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. First War of Scottish Independence (Score: 0.892)\n",
"1. Wars of Scottish Independence (Score: 0.889)\n",
"2. Second War of Scottish Independence (Score: 0.879)\n",
"3. List of Scottish monarchs (Score: 0.873)\n",
"4. Scottish Borders (Score: 0.863)\n"
]
}
],
"source": [
"# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\n",
"results = search_redis(redis_client,\n",
" \"Famous battles in Scottish history\",\n",
" vector_field=\"title_vector\",\n",
" k=5,\n",
" hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "28ab1e30",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Art (Score: 1.0)\n",
"1. Paint (Score: 0.896)\n",
"2. Renaissance art (Score: 0.88)\n",
"3. Painting (Score: 0.874)\n",
"4. Renaissance (Score: 0.846)\n"
]
},
{
"data": {
"text/plain": [
"'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\n",
"results = search_redis(redis_client,\n",
" \"Art\",\n",
" vector_field=\"title_vector\",\n",
" k=5,\n",
" hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n",
" )\n",
"\n",
"# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\n",
"mention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\n",
"mention"
]
},
{
"cell_type": "markdown",
"id": "f94b5be2",
"metadata": {},
"source": [
"For more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0119d87a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "vector_db_split",
"language": "python",
"name": "vector_db_split"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
},
"vscode": {
"interpreter": {
"hash": "fd16a328ca3d68029457069b79cb0b38eb39a0f5ccc4fe4473d3047707df8207"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/redis/docker-compose.yml
+22
View File
@@ -0,0 +1,22 @@
version: '3.7'
services:
vector-db:
image: redis/redis-stack:latest
ports:
- 6379:6379
- 8001:8001
environment:
- REDISEARCH_ARGS=CONCURRENT_WRITE_MODE
volumes:
- vector-db:/var/lib/redis
- ./redis.conf:/usr/local/etc/redis/redis.conf
healthcheck:
test: ["CMD", "redis-cli", "-h", "localhost", "-p", "6379", "ping"]
interval: 2s
timeout: 1m30s
retries: 5
start_period: 5s
volumes:
vector-db:
examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb
+877
View File
@@ -0,0 +1,877 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Redis as a Vector Database with OpenAI\n",
"\n",
"This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The RediSearch module allows you to index and search for vectors in Redis. This notebook will show you how to use the RediSearch module to index and search for vectors created by using the OpenAI API and stored in Redis.\n",
"\n",
"### What is Redis?\n",
"\n",
"Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.\n",
"\n",
"In addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new data types and commands. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).\n",
"\n",
"### What is RediSearch?\n",
"\n",
"RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [README](./README.md) or the [RediSearch documentation](https://redis.io/docs/stack/search/).\n",
"\n",
"### Deployment options\n",
"\n",
"There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n",
"\n",
"For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n",
"\n",
"Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f1a618c5",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Before we start this project, we need to set up the following:\n",
"\n",
"* start a Redis database with RediSearch (redis-stack)\n",
"* install libraries\n",
" * [Redis-py](https://github.com/redis/redis-py)\n",
"* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n",
"\n",
"===========================================================\n",
"\n",
"### Start Redis\n",
"\n",
"To keep this example simple, we will use the Redis Stack docker container which we can start as follows\n",
"\n",
"```bash\n",
"$ docker-compose up -d\n",
"```\n",
"\n",
"This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n",
"\n",
"You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b9babafe",
"metadata": {},
"source": [
"## Install Requirements\n",
"\n",
"Redis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2b04113f",
"metadata": {},
"outputs": [],
"source": [
"! pip install redis wget pandas openai"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "36fe86f4",
"metadata": {},
"source": [
"===========================================================\n",
"## Prepare your OpenAI API key\n",
"\n",
"The `OpenAI API key` is used for vectorization of query data.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "be28faa6",
"metadata": {},
"outputs": [],
"source": [
"! export OPENAI_API_KEY=\"your API key\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "88be138c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"import openai\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
" print (\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print (\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "97fefe4c",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "9fbebe0d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"File Downloaded\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import sys\n",
"import numpy as np\n",
"import pandas as pd\n",
"from typing import List\n",
"\n",
"# use helper function in nbutils.py to download and read the data\n",
"# this should take from 5-10 min to run\n",
"if os.getcwd() not in sys.path:\n",
" sys.path.append(os.getcwd())\n",
"import nbutils\n",
"\n",
"nbutils.download_wikipedia_data()\n",
"data = nbutils.read_wikipedia_data()\n",
"\n",
"data.head()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "91df4d5b",
"metadata": {},
"source": [
"## Connect to Redis\n",
"\n",
"Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cc662c1b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import redis\n",
"from redis.commands.search.indexDefinition import (\n",
" IndexDefinition,\n",
" IndexType\n",
")\n",
"from redis.commands.search.query import Query\n",
"from redis.commands.search.field import (\n",
" TextField,\n",
" VectorField\n",
")\n",
"\n",
"REDIS_HOST = \"localhost\"\n",
"REDIS_PORT = 6379\n",
"REDIS_PASSWORD = \"\" # default for passwordless Redis\n",
"\n",
"# Connect to Redis\n",
"redis_client = redis.Redis(\n",
" host=REDIS_HOST,\n",
" port=REDIS_PORT,\n",
" password=REDIS_PASSWORD\n",
")\n",
"redis_client.ping()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7d3dac3c",
"metadata": {},
"source": [
"## Creating a Search Index in Redis\n",
"\n",
"The below cells will show how to specify and create a search index in Redis. We will:\n",
"\n",
"1. Set some constants for defining our index like the distance metric and the index name\n",
"2. Define the index schema with RediSearch fields\n",
"3. Create the index"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f894b911",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Constants\n",
"VECTOR_DIM = len(data['title_vector'][0]) # length of the vectors\n",
"VECTOR_NUMBER = len(data) # initial number of vectors\n",
"INDEX_NAME = \"embeddings-index\" # name of the search index\n",
"PREFIX = \"doc\" # prefix for the document keys\n",
"DISTANCE_METRIC = \"COSINE\" # distance metric for the vectors (ex. COSINE, IP, L2)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "15db8380",
"metadata": {},
"outputs": [],
"source": [
"# Define RediSearch fields for each of the columns in the dataset\n",
"title = TextField(name=\"title\")\n",
"url = TextField(name=\"url\")\n",
"text = TextField(name=\"text\")\n",
"title_embedding = VectorField(\"title_vector\",\n",
" \"FLAT\", {\n",
" \"TYPE\": \"FLOAT32\",\n",
" \"DIM\": VECTOR_DIM,\n",
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
" \"INITIAL_CAP\": VECTOR_NUMBER,\n",
" }\n",
")\n",
"text_embedding = VectorField(\"content_vector\",\n",
" \"FLAT\", {\n",
" \"TYPE\": \"FLOAT32\",\n",
" \"DIM\": VECTOR_DIM,\n",
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
" \"INITIAL_CAP\": VECTOR_NUMBER,\n",
" }\n",
")\n",
"fields = [title, url, text, title_embedding, text_embedding]"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "3658693c",
"metadata": {},
"outputs": [],
"source": [
"# Check if index exists\n",
"try:\n",
" redis_client.ft(INDEX_NAME).info()\n",
" print(\"Index already exists\")\n",
"except:\n",
" # Create RediSearch Index\n",
" redis_client.ft(INDEX_NAME).create_index(\n",
" fields = fields,\n",
" definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "775c15b4",
"metadata": {},
"source": [
"## Load Documents into the Index\n",
"\n",
"Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "0d791186",
"metadata": {},
"outputs": [],
"source": [
"def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n",
" records = documents.to_dict(\"records\")\n",
" for doc in records:\n",
" key = f\"{prefix}:{str(doc['id'])}\"\n",
"\n",
" # create byte vectors for title and content\n",
" title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n",
" content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n",
"\n",
" # replace list of floats with byte vectors\n",
" doc[\"title_vector\"] = title_embedding\n",
" doc[\"content_vector\"] = content_embedding\n",
"\n",
" client.hset(key, mapping = doc)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "5bfaeafa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 25000 documents in Redis search index with name: embeddings-index\n"
]
}
],
"source": [
"index_documents(redis_client, PREFIX, data)\n",
"print(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "46050ca9",
"metadata": {},
"source": [
"## Simple Vector Search Queries with OpenAI Query Embeddings\n",
"\n",
"Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "b044aa93",
"metadata": {},
"outputs": [],
"source": [
"def search_redis(\n",
" redis_client: redis.Redis,\n",
" user_query: str,\n",
" index_name: str = \"embeddings-index\",\n",
" vector_field: str = \"title_vector\",\n",
" return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n",
" hybrid_fields = \"*\",\n",
" k: int = 20,\n",
" print_results: bool = True,\n",
") -> List[dict]:\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.Embedding.create(input=user_query,\n",
" model=\"text-embedding-3-small\",\n",
" )[\"data\"][0]['embedding']\n",
"\n",
" # Prepare the Query\n",
" base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n",
" query = (\n",
" Query(base_query)\n",
" .return_fields(*return_fields)\n",
" .sort_by(\"vector_score\")\n",
" .paging(0, k)\n",
" .dialect(2)\n",
" )\n",
" params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n",
"\n",
" # perform vector search\n",
" results = redis_client.ft(index_name).search(query, params_dict)\n",
" if print_results:\n",
" for i, article in enumerate(results.docs):\n",
" score = 1 - float(article.vector_score)\n",
" print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n",
" return results.docs"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "7e2025f6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Museum of Modern Art (Score: 0.875)\n",
"1. Western Europe (Score: 0.868)\n",
"2. Renaissance art (Score: 0.864)\n",
"3. Pop art (Score: 0.86)\n",
"4. Northern Europe (Score: 0.855)\n",
"5. Hellenistic art (Score: 0.853)\n",
"6. Modernist literature (Score: 0.847)\n",
"7. Art film (Score: 0.843)\n",
"8. Central Europe (Score: 0.843)\n",
"9. European (Score: 0.841)\n"
]
}
],
"source": [
"# For using OpenAI to generate query embedding\n",
"results = search_redis(redis_client, 'modern art in Europe', k=10)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "93c4a696",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Battle of Bannockburn (Score: 0.869)\n",
"1. Wars of Scottish Independence (Score: 0.861)\n",
"2. 1651 (Score: 0.853)\n",
"3. First War of Scottish Independence (Score: 0.85)\n",
"4. Robert I of Scotland (Score: 0.846)\n",
"5. 841 (Score: 0.844)\n",
"6. 1716 (Score: 0.844)\n",
"7. 1314 (Score: 0.837)\n",
"8. 1263 (Score: 0.836)\n",
"9. William Wallace (Score: 0.835)\n"
]
}
],
"source": [
"results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2007be48",
"metadata": {},
"source": [
"## Hybrid Queries with Redis\n",
"\n",
"The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "6c25ee8d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. First War of Scottish Independence (Score: 0.892)\n",
"1. Wars of Scottish Independence (Score: 0.889)\n",
"2. Second War of Scottish Independence (Score: 0.879)\n",
"3. List of Scottish monarchs (Score: 0.873)\n",
"4. Scottish Borders (Score: 0.863)\n"
]
}
],
"source": [
"def create_hybrid_field(field_name: str, value: str) -> str:\n",
" return f'@{field_name}:\"{value}\"'\n",
"\n",
"# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\n",
"results = search_redis(redis_client,\n",
" \"Famous battles in Scottish history\",\n",
" vector_field=\"title_vector\",\n",
" k=5,\n",
" hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "2c0d11d8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Art (Score: 1.0)\n",
"1. Paint (Score: 0.896)\n",
"2. Renaissance art (Score: 0.88)\n",
"3. Painting (Score: 0.874)\n",
"4. Renaissance (Score: 0.846)\n"
]
},
{
"data": {
"text/plain": [
"'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\n",
"results = search_redis(redis_client,\n",
" \"Art\",\n",
" vector_field=\"title_vector\",\n",
" k=5,\n",
" hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n",
" )\n",
"\n",
"# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\n",
"mention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\n",
"mention"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f8aebbe3",
"metadata": {},
"source": [
"## HNSW Index\n",
"\n",
"Up until now, we've been using the ``FLAT`` or \"brute-force\" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.\n",
"\n",
"``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.\n",
"\n",
"The following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "865c30f3",
"metadata": {},
"outputs": [],
"source": [
"# re-define RediSearch vector fields to use HNSW index\n",
"title_embedding = VectorField(\"title_vector\",\n",
" \"HNSW\", {\n",
" \"TYPE\": \"FLOAT32\",\n",
" \"DIM\": VECTOR_DIM,\n",
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
" \"INITIAL_CAP\": VECTOR_NUMBER\n",
" }\n",
")\n",
"text_embedding = VectorField(\"content_vector\",\n",
" \"HNSW\", {\n",
" \"TYPE\": \"FLOAT32\",\n",
" \"DIM\": VECTOR_DIM,\n",
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
" \"INITIAL_CAP\": VECTOR_NUMBER\n",
" }\n",
")\n",
"fields = [title, url, text, title_embedding, text_embedding]"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "347e1e70",
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"# Check if index exists\n",
"HNSW_INDEX_NAME = INDEX_NAME+ \"_HNSW\"\n",
"\n",
"try:\n",
" redis_client.ft(HNSW_INDEX_NAME).info()\n",
" print(\"Index already exists\")\n",
"except:\n",
" # Create RediSearch Index\n",
" redis_client.ft(HNSW_INDEX_NAME).create_index(\n",
" fields = fields,\n",
" definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n",
" )\n",
"\n",
"# since RediSearch creates the index in the background for existing documents, we will wait until\n",
"# indexing is complete before running our queries. Although this is not necessary for the first query,\n",
"# some queries may take longer to run if the index is not fully built. In general, Redis will perform\n",
"# best when adding new documents to existing indices rather than new indices on existing documents.\n",
"while redis_client.ft(HNSW_INDEX_NAME).info()[\"indexing\"] == \"1\":\n",
" time.sleep(5)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "8e474447",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Western Europe (Score: 0.868)\n",
"1. Northern Europe (Score: 0.855)\n",
"2. Central Europe (Score: 0.843)\n",
"3. European (Score: 0.841)\n",
"4. Eastern Europe (Score: 0.839)\n",
"5. Europe (Score: 0.839)\n",
"6. Western European Union (Score: 0.837)\n",
"7. Southern Europe (Score: 0.831)\n",
"8. Western civilization (Score: 0.83)\n",
"9. Council of Europe (Score: 0.827)\n"
]
}
],
"source": [
"results = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "cb799e69",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" ----- Flat Index ----- \n",
"0. Museum of Modern Art (Score: 0.875)\n",
"1. Western Europe (Score: 0.867)\n",
"2. Renaissance art (Score: 0.864)\n",
"3. Pop art (Score: 0.861)\n",
"4. Northern Europe (Score: 0.855)\n",
"5. Hellenistic art (Score: 0.853)\n",
"6. Modernist literature (Score: 0.847)\n",
"7. Art film (Score: 0.843)\n",
"8. Central Europe (Score: 0.843)\n",
"9. Art (Score: 0.842)\n",
"Flat index query time: 0.263 seconds\n",
"\n",
" ----- HNSW Index ------ \n",
"0. Western Europe (Score: 0.867)\n",
"1. Northern Europe (Score: 0.855)\n",
"2. Central Europe (Score: 0.843)\n",
"3. European (Score: 0.841)\n",
"4. Eastern Europe (Score: 0.839)\n",
"5. Europe (Score: 0.839)\n",
"6. Western European Union (Score: 0.837)\n",
"7. Southern Europe (Score: 0.831)\n",
"8. Western civilization (Score: 0.83)\n",
"9. Council of Europe (Score: 0.827)\n",
"HNSW index query time: 0.129 seconds\n",
" ------------------------ \n"
]
}
],
"source": [
"# compare the results of the HNSW index to the FLAT index and time both queries\n",
"def time_queries(iterations: int = 10):\n",
" print(\" ----- Flat Index ----- \")\n",
" t0 = time.time()\n",
" for i in range(iterations):\n",
" results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=False)\n",
" t0 = (time.time() - t0) / iterations\n",
" results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=True)\n",
" print(f\"Flat index query time: {round(t0, 3)} seconds\\n\")\n",
" time.sleep(1)\n",
" print(\" ----- HNSW Index ------ \")\n",
" t1 = time.time()\n",
" for i in range(iterations):\n",
" results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=False)\n",
" t1 = (time.time() - t1) / iterations\n",
" results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=True)\n",
" print(f\"HNSW index query time: {round(t1, 3)} seconds\")\n",
" print(\" ------------------------ \")\n",
"time_queries()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "69aa7a09",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "redisvl2",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"vscode": {
"interpreter": {
"hash": "9b1e6e9c2967143209c2f955cb869d1d3234f92dc4787f49f155f3abbdfb1316"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/redis/nbutils.py
+46
View File
@@ -0,0 +1,46 @@
import os
import wget
import zipfile
import numpy as np
import pandas as pd
from ast import literal_eval
def download_wikipedia_data(
data_path: str = '../../data/',
download_path: str = "./",
file_name: str = "vector_database_wikipedia_articles_embedded") -> pd.DataFrame:
data_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'
csv_file_path = os.path.join(data_path, file_name + ".csv")
zip_file_path = os.path.join(download_path, file_name + ".zip")
if os.path.isfile(csv_file_path):
print("File Downloaded")
else:
if os.path.isfile(zip_file_path):
print("Zip downloaded but not unzipped, unzipping now...")
else:
print("File not found, downloading now...")
# Download the data
wget.download(data_url, out=download_path)
# Unzip the data
with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
zip_ref.extractall(data_path)
# Remove the zip file
os.remove('vector_database_wikipedia_articles_embedded.zip')
print(f"File downloaded to {data_path}")
def read_wikipedia_data(data_path: str = '../../data/', file_name: str = "vector_database_wikipedia_articles_embedded") -> pd.DataFrame:
csv_file_path = os.path.join(data_path, file_name + ".csv")
data = pd.read_csv(csv_file_path)
# Read vectors from strings back into a list
data['title_vector'] = data.title_vector.apply(literal_eval)
data['content_vector'] = data.content_vector.apply(literal_eval)
# Set vector_id to be a string
data['vector_id'] = data['vector_id'].apply(str)
return data
examples/vector_databases/redis/redis-hybrid-query-examples.ipynb
+914
View File
@@ -0,0 +1,914 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Running Hybrid VSS Queries with Redis and OpenAI\n",
"\n",
"This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Query and Search capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The Redis Query and Search capability allows you to index and search for vectors in Redis. This notebook will show you how to use the Redis Query and Search to index and search for vectors created by using the OpenAI API and stored in Redis.\n",
"\n",
"Hybrid queries combine vector similarity with traditional Redis Query and Search filtering capabilities on GEO, NUMERIC, TAG or TEXT data simplifying application code. A common example of a hybrid query in an e-commerce use case is to find items visually similar to a given query image limited to items available in a GEO location and within a price range."
]
},
{
"cell_type": "markdown",
"id": "f1a618c5",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Before we start this project, we need to set up the following:\n",
"\n",
"* start a Redis database with RediSearch (redis-stack)\n",
"* install libraries\n",
" * [Redis-py](https://github.com/redis/redis-py)\n",
"* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n",
"\n",
"===========================================================\n",
"\n",
"### Start Redis\n",
"\n",
"To keep this example simple, we will use the Redis Stack docker container which we can start as follows\n",
"\n",
"```bash\n",
"$ docker-compose up -d\n",
"```\n",
"\n",
"This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n",
"\n",
"You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created."
]
},
{
"cell_type": "markdown",
"id": "b9babafe",
"metadata": {},
"source": [
"## Install Requirements\n",
"\n",
"Redis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "2b04113f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Defaulting to user installation because normal site-packages is not writeable\n",
"Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)\n",
"Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)\n",
"Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)\n",
"Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n",
"Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n",
"Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.4)\n",
"Requirement already satisfied: requests>=2.20 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (2.28.1)\n",
"Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)\n",
"Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)\n",
"Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
"Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)\n",
"Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n",
"Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n"
]
}
],
"source": [
"! pip install redis pandas openai\n"
]
},
{
"cell_type": "markdown",
"id": "36fe86f4",
"metadata": {},
"source": [
"===========================================================\n",
"## Prepare your OpenAI API key\n",
"\n",
"The `OpenAI API key` is used for vectorization of query data.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "88be138c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"import openai\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = '<YOUR_OPENAI_API_KEY>'\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
" print (\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print (\"OPENAI_API_KEY environment variable not found\")\n"
]
},
{
"cell_type": "markdown",
"id": "97fefe4c",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "9fbebe0d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"Index: 1978 entries, 0 to 1998\n",
"Data columns (total 10 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 1978 non-null int64 \n",
" 1 gender 1978 non-null object\n",
" 2 masterCategory 1978 non-null object\n",
" 3 subCategory 1978 non-null object\n",
" 4 articleType 1978 non-null object\n",
" 5 baseColour 1978 non-null object\n",
" 6 season 1978 non-null object\n",
" 7 year 1978 non-null int64 \n",
" 8 usage 1978 non-null object\n",
" 9 productDisplayName 1978 non-null object\n",
"dtypes: int64(2), object(8)\n",
"memory usage: 170.0+ KB\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>gender</th>\n",
" <th>masterCategory</th>\n",
" <th>subCategory</th>\n",
" <th>articleType</th>\n",
" <th>baseColour</th>\n",
" <th>season</th>\n",
" <th>year</th>\n",
" <th>usage</th>\n",
" <th>productDisplayName</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>15970</td>\n",
" <td>Men</td>\n",
" <td>Apparel</td>\n",
" <td>Topwear</td>\n",
" <td>Shirts</td>\n",
" <td>Navy Blue</td>\n",
" <td>Fall</td>\n",
" <td>2011</td>\n",
" <td>Casual</td>\n",
" <td>Turtle Check Men Navy Blue Shirt</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>39386</td>\n",
" <td>Men</td>\n",
" <td>Apparel</td>\n",
" <td>Bottomwear</td>\n",
" <td>Jeans</td>\n",
" <td>Blue</td>\n",
" <td>Summer</td>\n",
" <td>2012</td>\n",
" <td>Casual</td>\n",
" <td>Peter England Men Party Blue Jeans</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>59263</td>\n",
" <td>Women</td>\n",
" <td>Accessories</td>\n",
" <td>Watches</td>\n",
" <td>Watches</td>\n",
" <td>Silver</td>\n",
" <td>Winter</td>\n",
" <td>2016</td>\n",
" <td>Casual</td>\n",
" <td>Titan Women Silver Watch</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>21379</td>\n",
" <td>Men</td>\n",
" <td>Apparel</td>\n",
" <td>Bottomwear</td>\n",
" <td>Track Pants</td>\n",
" <td>Black</td>\n",
" <td>Fall</td>\n",
" <td>2011</td>\n",
" <td>Casual</td>\n",
" <td>Manchester United Men Solid Black Track Pants</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>53759</td>\n",
" <td>Men</td>\n",
" <td>Apparel</td>\n",
" <td>Topwear</td>\n",
" <td>Tshirts</td>\n",
" <td>Grey</td>\n",
" <td>Summer</td>\n",
" <td>2012</td>\n",
" <td>Casual</td>\n",
" <td>Puma Men Grey T-shirt</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id gender masterCategory subCategory articleType baseColour season \n",
"0 15970 Men Apparel Topwear Shirts Navy Blue Fall \\\n",
"1 39386 Men Apparel Bottomwear Jeans Blue Summer \n",
"2 59263 Women Accessories Watches Watches Silver Winter \n",
"3 21379 Men Apparel Bottomwear Track Pants Black Fall \n",
"4 53759 Men Apparel Topwear Tshirts Grey Summer \n",
"\n",
" year usage productDisplayName \n",
"0 2011 Casual Turtle Check Men Navy Blue Shirt \n",
"1 2012 Casual Peter England Men Party Blue Jeans \n",
"2 2016 Casual Titan Women Silver Watch \n",
"3 2011 Casual Manchester United Men Solid Black Track Pants \n",
"4 2012 Casual Puma Men Grey T-shirt "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"from typing import List\n",
"\n",
"from utils.embeddings_utils import (\n",
" get_embeddings,\n",
" distances_from_embeddings,\n",
" tsne_components_from_embeddings,\n",
" chart_from_components,\n",
" indices_of_nearest_neighbors_from_distances,\n",
")\n",
"\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"\n",
"# load in data and clean data types and drop null rows\n",
"df = pd.read_csv(\"../../data/styles_2k.csv\", on_bad_lines='skip')\n",
"df.dropna(inplace=True)\n",
"df[\"year\"] = df[\"year\"].astype(int)\n",
"df.info()\n",
"\n",
"# print dataframe\n",
"n_examples = 5\n",
"df.head(n_examples)\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "3ce1ec50",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"Index: 1978 entries, 0 to 1998\n",
"Data columns (total 11 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 product_id 1978 non-null int64 \n",
" 1 gender 1978 non-null object\n",
" 2 masterCategory 1978 non-null object\n",
" 3 subCategory 1978 non-null object\n",
" 4 articleType 1978 non-null object\n",
" 5 baseColour 1978 non-null object\n",
" 6 season 1978 non-null object\n",
" 7 year 1978 non-null int64 \n",
" 8 usage 1978 non-null object\n",
" 9 productDisplayName 1978 non-null object\n",
" 10 product_text 1978 non-null object\n",
"dtypes: int64(2), object(9)\n",
"memory usage: 185.4+ KB\n"
]
}
],
"source": [
"df[\"product_text\"] = df.apply(lambda row: f\"name {row['productDisplayName']} category {row['masterCategory']} subcategory {row['subCategory']} color {row['baseColour']} gender {row['gender']}\".lower(), axis=1)\n",
"df.rename({\"id\":\"product_id\"}, inplace=True, axis=1)\n",
"\n",
"df.info()\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "13859ab5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# check out one of the texts we will use to create semantic embeddings\n",
"df[\"product_text\"][0]\n"
]
},
{
"cell_type": "markdown",
"id": "91df4d5b",
"metadata": {},
"source": [
"## Connect to Redis\n",
"\n",
"Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "cc662c1b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import redis\n",
"from redis.commands.search.indexDefinition import (\n",
" IndexDefinition,\n",
" IndexType\n",
")\n",
"from redis.commands.search.query import Query\n",
"from redis.commands.search.field import (\n",
" TagField,\n",
" NumericField,\n",
" TextField,\n",
" VectorField\n",
")\n",
"\n",
"REDIS_HOST = \"localhost\"\n",
"REDIS_PORT = 6379\n",
"REDIS_PASSWORD = \"\" # default for passwordless Redis\n",
"\n",
"# Connect to Redis\n",
"redis_client = redis.Redis(\n",
" host=REDIS_HOST,\n",
" port=REDIS_PORT,\n",
" password=REDIS_PASSWORD\n",
")\n",
"redis_client.ping()\n"
]
},
{
"cell_type": "markdown",
"id": "7d3dac3c",
"metadata": {},
"source": [
"## Creating a Search Index in Redis\n",
"\n",
"The below cells will show how to specify and create a search index in Redis. We will:\n",
"\n",
"1. Set some constants for defining our index like the distance metric and the index name\n",
"2. Define the index schema with RediSearch fields\n",
"3. Create the index"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "f894b911",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Constants\n",
"INDEX_NAME = \"product_embeddings\" # name of the search index\n",
"PREFIX = \"doc\" # prefix for the document keys\n",
"DISTANCE_METRIC = \"L2\" # distance metric for the vectors (ex. COSINE, IP, L2)\n",
"NUMBER_OF_VECTORS = len(df)\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "15db8380",
"metadata": {},
"outputs": [],
"source": [
"# Define RediSearch fields for each of the columns in the dataset\n",
"name = TextField(name=\"productDisplayName\")\n",
"category = TagField(name=\"masterCategory\")\n",
"articleType = TagField(name=\"articleType\")\n",
"gender = TagField(name=\"gender\")\n",
"season = TagField(name=\"season\")\n",
"year = NumericField(name=\"year\")\n",
"text_embedding = VectorField(\"product_vector\",\n",
" \"FLAT\", {\n",
" \"TYPE\": \"FLOAT32\",\n",
" \"DIM\": 1536,\n",
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
" \"INITIAL_CAP\": NUMBER_OF_VECTORS,\n",
" }\n",
")\n",
"fields = [name, category, articleType, gender, season, year, text_embedding]\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "3658693c",
"metadata": {},
"outputs": [],
"source": [
"# Check if index exists\n",
"try:\n",
" redis_client.ft(INDEX_NAME).info()\n",
" print(\"Index already exists\")\n",
"except:\n",
" # Create RediSearch Index\n",
" redis_client.ft(INDEX_NAME).create_index(\n",
" fields = fields,\n",
" definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "775c15b4",
"metadata": {},
"source": [
"## Generate OpenAI Embeddings and Load Documents into the Index\n",
"\n",
"Now that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "852cff45",
"metadata": {},
"outputs": [],
"source": [
"# Use OpenAI get_embeddings batch requests to speed up embedding creation\n",
"def embeddings_batch_request(documents: pd.DataFrame):\n",
" records = documents.to_dict(\"records\")\n",
" print(\"Records to process: \", len(records))\n",
" product_vectors = []\n",
" docs = []\n",
" batchsize = 1000\n",
"\n",
" for idx,doc in enumerate(records,start=1):\n",
" # create byte vectors\n",
" docs.append(doc[\"product_text\"])\n",
" if idx % batchsize == 0:\n",
" product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n",
" docs.clear()\n",
" print(\"Vectors processed \", len(product_vectors), end='\\r')\n",
" product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n",
" print(\"Vectors processed \", len(product_vectors), end='\\r')\n",
" return product_vectors\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "0d791186",
"metadata": {},
"outputs": [],
"source": [
"def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n",
" product_vectors = embeddings_batch_request(documents)\n",
" records = documents.to_dict(\"records\")\n",
" batchsize = 500\n",
"\n",
" # Use Redis pipelines to batch calls and save on round trip network communication\n",
" pipe = client.pipeline()\n",
" for idx,doc in enumerate(records,start=1):\n",
" key = f\"{prefix}:{str(doc['product_id'])}\"\n",
"\n",
" # create byte vectors\n",
" text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n",
"\n",
" # replace list of floats with byte vectors\n",
" doc[\"product_vector\"] = text_embedding\n",
"\n",
" pipe.hset(key, mapping = doc)\n",
" if idx % batchsize == 0:\n",
" pipe.execute()\n",
" pipe.execute()\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "5bfaeafa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Records to process: 1978\n",
"Loaded 1978 documents in Redis search index with name: product_embeddings\n",
"CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms\n",
"Wall time: 3.34 s\n"
]
}
],
"source": [
"%%time\n",
"index_documents(redis_client, PREFIX, df)\n",
"print(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n"
]
},
{
"cell_type": "markdown",
"id": "46050ca9",
"metadata": {},
"source": [
"## Simple Vector Search Queries with OpenAI Query Embeddings\n",
"\n",
"Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "b044aa93",
"metadata": {},
"outputs": [],
"source": [
"def search_redis(\n",
" redis_client: redis.Redis,\n",
" user_query: str,\n",
" index_name: str = \"product_embeddings\",\n",
" vector_field: str = \"product_vector\",\n",
" return_fields: list = [\"productDisplayName\", \"masterCategory\", \"gender\", \"season\", \"year\", \"vector_score\"],\n",
" hybrid_fields = \"*\",\n",
" k: int = 20,\n",
" print_results: bool = True,\n",
") -> List[dict]:\n",
"\n",
" # Use OpenAI to create embedding vector from user query\n",
" embedded_query = openai.Embedding.create(input=user_query,\n",
" model=\"text-embedding-3-small\",\n",
" )[\"data\"][0]['embedding']\n",
"\n",
" # Prepare the Query\n",
" base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n",
" query = (\n",
" Query(base_query)\n",
" .return_fields(*return_fields)\n",
" .sort_by(\"vector_score\")\n",
" .paging(0, k)\n",
" .dialect(2)\n",
" )\n",
" params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n",
"\n",
" # perform vector search\n",
" results = redis_client.ft(index_name).search(query, params_dict)\n",
" if print_results:\n",
" for i, product in enumerate(results.docs):\n",
" score = 1 - float(product.vector_score)\n",
" print(f\"{i}. {product.productDisplayName} (Score: {round(score ,3) })\")\n",
" return results.docs\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "7e2025f6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. John Players Men Blue Jeans (Score: 0.791)\n",
"1. Lee Men Tino Blue Jeans (Score: 0.775)\n",
"2. Peter England Men Party Blue Jeans (Score: 0.763)\n",
"3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n",
"4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n",
"5. French Connection Men Blue Jeans (Score: 0.74)\n",
"6. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
"7. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
"8. Do U Speak Green Men Blue Shorts (Score: 0.736)\n",
"9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n"
]
}
],
"source": [
"# Execute a simple vector search in Redis\n",
"results = search_redis(redis_client, 'man blue jeans', k=10)\n"
]
},
{
"cell_type": "markdown",
"id": "2007be48",
"metadata": {},
"source": [
"## Hybrid Queries with Redis\n",
"\n",
"The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "2c81fbb7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. John Players Men Blue Jeans (Score: 0.791)\n",
"1. Lee Men Tino Blue Jeans (Score: 0.775)\n",
"2. Peter England Men Party Blue Jeans (Score: 0.763)\n",
"3. French Connection Men Blue Jeans (Score: 0.74)\n",
"4. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
"5. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
"6. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n",
"7. Denizen Women Blue Jeans (Score: 0.725)\n",
"8. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n",
"9. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n"
]
}
],
"source": [
"# improve search quality by adding hybrid query for \"man blue jeans\" in the product vector combined with a phrase search for \"blue jeans\"\n",
"results = search_redis(redis_client,\n",
" \"man blue jeans\",\n",
" vector_field=\"product_vector\",\n",
" k=10,\n",
" hybrid_fields='@productDisplayName:\"blue jeans\"'\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "8a56633b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)\n",
"1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)\n",
"2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n",
"3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n",
"4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n",
"5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n",
"6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)\n",
"7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)\n"
]
}
],
"source": [
"# hybrid query for shirt in the product vector and only include results with the phrase \"slim fit\" in the title\n",
"results = search_redis(redis_client,\n",
" \"shirt\",\n",
" vector_field=\"product_vector\",\n",
" k=10,\n",
" hybrid_fields='@productDisplayName:\"slim fit\"'\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "6c25ee8d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Titan Women Gold Watch (Score: 0.544)\n",
"1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)\n",
"2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n",
"3. Titan Men Black Watch (Score: 0.543)\n",
"4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n",
"5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n",
"6. Titan Women Silver Watch (Score: 0.542)\n",
"7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)\n",
"8. Titan Raga Women Gold Watch (Score: 0.539)\n",
"9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)\n"
]
}
],
"source": [
"# hybrid query for watch in the product vector and only include results with the tag \"Accessories\" in the masterCategory field\n",
"results = search_redis(redis_client,\n",
" \"watch\",\n",
" vector_field=\"product_vector\",\n",
" k=10,\n",
" hybrid_fields='@masterCategory:{Accessories}'\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "2c0d11d8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Enroute Teens Orange Sandals (Score: 0.701)\n",
"1. Fila Men Camper Brown Sandals (Score: 0.692)\n",
"2. Clarks Men Black Leather Closed Sandals (Score: 0.691)\n",
"3. Coolers Men Black Sandals (Score: 0.69)\n",
"4. Coolers Men Black Sandals (Score: 0.69)\n",
"5. Enroute Teens Brown Sandals (Score: 0.69)\n",
"6. Crocs Dora Boots Pink Sandals (Score: 0.69)\n",
"7. Enroute Men Leather Black Sandals (Score: 0.685)\n",
"8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)\n",
"9. Coolers Men Black Sports Sandals (Score: 0.684)\n"
]
}
],
"source": [
"# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range\n",
"results = search_redis(redis_client,\n",
" \"sandals\",\n",
" vector_field=\"product_vector\",\n",
" k=10,\n",
" hybrid_fields='@year:[2011 2012]'\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "7caad384",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. ADIDAS Men Navy Blue Benton Sandals (Score: 0.691)\n",
"1. Enroute Teens Brown Sandals (Score: 0.681)\n",
"2. ADIDAS Women's Adi Groove Blue Flip Flop (Score: 0.672)\n",
"3. Enroute Women Turquoise Blue Flats (Score: 0.671)\n",
"4. Red Tape Men Black Sandals (Score: 0.67)\n",
"5. Enroute Teens Orange Sandals (Score: 0.661)\n",
"6. Vans Men Blue Era Scilla Plaid Shoes (Score: 0.658)\n",
"7. FILA Men Aruba Navy Blue Sandal (Score: 0.657)\n",
"8. Quiksilver Men Blue Flip Flops (Score: 0.656)\n",
"9. Reebok Men Navy Twist Sandals (Score: 0.656)\n"
]
}
],
"source": [
"# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range from the summer season\n",
"results = search_redis(redis_client,\n",
" \"blue sandals\",\n",
" vector_field=\"product_vector\",\n",
" k=10,\n",
" hybrid_fields='(@year:[2011 2012] @season:{Summer})'\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "f1232d3c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Wrangler Men Leather Brown Belt (Score: 0.67)\n",
"1. Wrangler Women Black Belt (Score: 0.639)\n",
"2. Wrangler Men Green Striped Shirt (Score: 0.575)\n",
"3. Wrangler Men Purple Striped Shirt (Score: 0.549)\n",
"4. Wrangler Men Griffith White Shirt (Score: 0.543)\n",
"5. Wrangler Women Stella Green Shirt (Score: 0.542)\n"
]
}
],
"source": [
"# hybrid query for a brown belt filtering results by a year (NUMERIC) with a specific article types (TAG) and with a brand name (TEXT)\n",
"results = search_redis(redis_client,\n",
" \"brown belt\",\n",
" vector_field=\"product_vector\",\n",
" k=10,\n",
" hybrid_fields='(@year:[2012 2012] @articleType:{Shirts | Belts} @productDisplayName:\"Wrangler\")'\n",
" )\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.6"
},
"vscode": {
"interpreter": {
"hash": "9b1e6e9c2967143209c2f955cb869d1d3234f92dc4787f49f155f3abbdfb1316"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/redis/redis.conf
+5
View File
@@ -0,0 +1,5 @@
port 6379
appendonly no
save ""
protected-mode no
io-threads 2
examples/vector_databases/redis/redisjson/docker-compose.yml
+5
View File
@@ -0,0 +1,5 @@
services:
redis:
image: redis/redis-stack-server:latest
ports:
- 6379:6379
examples/vector_databases/redis/redisjson/redisjson.ipynb
+413
View File
@@ -0,0 +1,413 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Redis Vectors as JSON with OpenAI\n",
"This notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.\n",
"[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)\n",
"\n",
"## Prerequisites\n",
"* Redis instance with the Redis Search and Redis JSON modules\n",
"* Redis-py client lib\n",
"* OpenAI API key\n",
"\n",
"## Installation\n",
"Install Python modules necessary for the examples."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "shellscript"
}
},
"outputs": [],
"source": [
"! pip install redis openai python-dotenv openai[datalib]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## OpenAI API Key\n",
"Create a .env file and add your OpenAI key to it"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "shellscript"
}
},
"outputs": [],
"source": [
"OPENAI_API_KEY=your_key"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Text Vectors\n",
"Create embeddings (array of floats) of the news excerpts below."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"import os\n",
"from dotenv import load_dotenv\n",
"\n",
"load_dotenv()\n",
"openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
"\n",
"def get_vector(text, model=\"text-embedding-3-small\"):\n",
" text = text.replace(\"\\n\", \" \")\n",
" return openai.Embedding.create(input = [text], model = model)['data'][0]['embedding']\n",
"\n",
"text_1 = \"\"\"Japan narrowly escapes recession\n",
"\n",
"Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n",
"\n",
"Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\n",
"The government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. \"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.\n",
"\"\"\"\n",
"\n",
"text_2 = \"\"\"Dibaba breaks 5,000m world record\n",
"\n",
"Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n",
"\n",
"Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. \"I didn't want to sit back and get out-kicked,\" said Cragg. \"So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine.\" Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. \n",
"\"\"\"\n",
"\n",
"\n",
"text_3 = \"\"\"Google's toolbar sparks concern\n",
"\n",
"Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.\n",
"\n",
"The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, \"adds useful links\". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.\n",
"\n",
"AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.\n",
"\n",
"If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a \"bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth\". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: \"The user can choose never to click on the AutoLink button, and web pages she views will never be modified. \"In addition, the user can choose to disable the AutoLink feature entirely at any time.\"\n",
"\n",
"The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any \"click through\" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. \"Of course Google should be allowed to direct people to whatever proxies it chooses. \"But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'.\" Mr Doctorow said the only objection would be if users were forced into using AutoLink or \"tricked into using the service\".\n",
"\"\"\"\n",
"\n",
"doc_1 = {\"content\": text_1, \"vector\": get_vector(text_1)}\n",
"doc_2 = {\"content\": text_2, \"vector\": get_vector(text_2)}\n",
"doc_3 = {\"content\": text_3, \"vector\": get_vector(text_3)}\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Start the Redis Stack Docker container"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"vscode": {
"languageId": "shellscript"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 0/0\n",
" ⠿ Container redisjson-redis-1 Starting \u001b[34m0.1s \u001b[0m\n",
"\u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n",
" ⠿ Container redisjson-redis-1 Starting \u001b[34m0.2s \u001b[0m\n",
"\u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n",
" ⠿ Container redisjson-redis-1 Starting \u001b[34m0.3s \u001b[0m\n",
"\u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n",
" ⠿ Container redisjson-redis-1 Starting \u001b[34m0.4s \u001b[0m\n",
"\u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 1/1\u001b[0m\n",
" \u001b[32m✔\u001b[0m Container redisjson-redis-1 \u001b[32mStarted\u001b[0m \u001b[34m0.4s \u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"! docker compose up -d"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect Redis client"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from redis import from_url\n",
"\n",
"REDIS_URL = 'redis://localhost:6379'\n",
"client = from_url(REDIS_URL)\n",
"client.ping()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Index\n",
"[FT.CREATE](https://redis.io/commands/ft.create/)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"b'OK'"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from redis.commands.search.field import TextField, VectorField\n",
"from redis.commands.search.indexDefinition import IndexDefinition, IndexType\n",
"\n",
"schema = [ VectorField('$.vector', \n",
" \"FLAT\", \n",
" { \"TYPE\": 'FLOAT32', \n",
" \"DIM\": len(doc_1['vector']), \n",
" \"DISTANCE_METRIC\": \"COSINE\"\n",
" }, as_name='vector' ),\n",
" TextField('$.content', as_name='content')\n",
" ]\n",
"idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\n",
"try: \n",
" client.ft('idx').dropindex()\n",
"except:\n",
" pass\n",
"client.ft('idx').create_index(schema, definition=idx_def)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Data into Redis as JSON objects\n",
"[Redis JSON](https://redis.io/docs/stack/json/)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.json().set('doc:1', '$', doc_1)\n",
"client.json().set('doc:2', '$', doc_2)\n",
"client.json().set('doc:3', '$', doc_3)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Semantic Search\n",
"Given a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles. \n",
"[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"distance:0.188 content:Dibaba breaks 5,000m world record\n",
"\n",
"Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n",
"\n",
"Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. \"I didn't want to sit back and get out-kicked,\" said Cragg. \"So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine.\" Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. \n",
"\n",
"\n",
"distance:0.268 content:Japan narrowly escapes recession\n",
"\n",
"Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n",
"\n",
"Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\n",
"The government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. \"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.\n",
"\n",
"\n",
"distance:0.287 content:Google's toolbar sparks concern\n",
"\n",
"Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.\n",
"\n",
"The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, \"adds useful links\". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.\n",
"\n",
"AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.\n",
"\n",
"If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a \"bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth\". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: \"The user can choose never to click on the AutoLink button, and web pages she views will never be modified. \"In addition, the user can choose to disable the AutoLink feature entirely at any time.\"\n",
"\n",
"The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any \"click through\" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. \"Of course Google should be allowed to direct people to whatever proxies it chooses. \"But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'.\" Mr Doctorow said the only objection would be if users were forced into using AutoLink or \"tricked into using the service\".\n",
"\n",
"\n"
]
}
],
"source": [
"from redis.commands.search.query import Query\n",
"import numpy as np\n",
"\n",
"text_4 = \"\"\"Radcliffe yet to answer GB call\n",
"\n",
"Paula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.\n",
"\n",
"The 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. \"There is no question that Paula would be a huge asset to the GB team,\" said Zara Hyde Peters of UK Athletics. \"But she is working out whether she can accommodate the worlds without too much compromise in her marathon training.\" Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. \"It would be fantastic to have Paula in the team,\" said the European cross-country champion. \"But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. \"She is not messing us around. We all understand the problem.\" Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.\n",
"\"\"\"\n",
"\n",
"vec = np.array(get_vector(text_4), dtype=np.float32).tobytes()\n",
"q = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\\\n",
" .sort_by('vector_score')\\\n",
" .return_fields('vector_score', 'content')\\\n",
" .dialect(2) \n",
"params = {\"query_vec\": vec}\n",
"\n",
"results = client.ft('idx').search(q, query_params=params)\n",
"for doc in results.docs:\n",
" print(f\"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\\n\")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Hybrid Search\n",
"Use a combination of full text search and VSS to find a matching article. For this scenario, we filter on a full text search of the term 'recession' and then find the KNN articles. In this case, business-related. Reminder document #1 was about a recession in Japan.\n",
"[Hybrid Queries](https://redis.io/docs/stack/search/reference/vectors/#hybrid-queries)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"distance:0.241 content:Japan narrowly escapes recession\n",
"\n",
"Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n",
"\n",
"Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\n",
"The government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. \"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.\n",
"\n",
"\n"
]
}
],
"source": [
"text_5 = \"\"\"Ethiopia's crop production up 24%\n",
"\n",
"Ethiopia produced 14.27 million tonnes of crops in 2004, 24% higher than in 2003 and 21% more than the average of the past five years, a report says.\n",
"\n",
"In 2003, crop production totalled 11.49 million tonnes, the joint report from the Food and Agriculture Organisation and the World Food Programme said. Good rains, increased use of fertilizers and improved seeds contributed to the rise in production. Nevertheless, 2.2 million Ethiopians will still need emergency assistance.\n",
"\n",
"The report calculated emergency food requirements for 2005 to be 387,500 tonnes. On top of that, 89,000 tonnes of fortified blended food and vegetable oil for \"targeted supplementary food distributions for a survival programme for children under five and pregnant and lactating women\" will be needed.\n",
"\n",
"In eastern and southern Ethiopia, a prolonged drought has killed crops and drained wells. Last year, a total of 965,000 tonnes of food assistance was needed to help seven million Ethiopians. The Food and Agriculture Organisation (FAO) recommend that the food assistance is bought locally. \"Local purchase of cereals for food assistance programmes is recommended as far as possible, so as to assist domestic markets and farmers,\" said Henri Josserand, chief of FAO's Global Information and Early Warning System. Agriculture is the main economic activity in Ethiopia, representing 45% of gross domestic product. About 80% of Ethiopians depend directly or indirectly on agriculture.\n",
"\"\"\"\n",
"\n",
"vec = np.array(get_vector(text_5), dtype=np.float32).tobytes()\n",
"q = Query('@content:recession => [KNN 3 @vector $query_vec AS vector_score]')\\\n",
" .sort_by('vector_score')\\\n",
" .return_fields('vector_score', 'content')\\\n",
" .dialect(2) \n",
"params = {\"query_vec\": vec}\n",
"\n",
"results = client.ft('idx').search(q, query_params=params)\n",
"for doc in results.docs:\n",
" print(f\"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\\n\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/redis/redisqna/assets/001.txt
+11
View File
@@ -0,0 +1,11 @@
Ad sales boost Time Warner profit
Quarterly profits at US media giant TimeWarner jumped 76% to $1.13bn (£600m) for the three months to December, from $639m year-earlier.
The firm, which is now one of the biggest investors in Google, benefited from sales of high-speed internet connections and higher advert sales. TimeWarner said fourth quarter sales rose 2% to $11.1bn from $10.9bn. Its profits were buoyed by one-off gains which offset a profit dip at Warner Bros, and less users for AOL.
Time Warner said on Friday that it now owns 8% of search-engine Google. But its own internet business, AOL, had has mixed fortunes. It lost 464,000 subscribers in the fourth quarter profits were lower than in the preceding three quarters. However, the company said AOL's underlying profit before exceptional items rose 8% on the back of stronger internet advertising revenues. It hopes to increase subscribers by offering the online service free to TimeWarner internet customers and will try to sign up AOL's existing customers for high-speed broadband. TimeWarner also has to restate 2000 and 2003 results following a probe by the US Securities Exchange Commission (SEC), which is close to concluding.
Time Warner's fourth quarter profits were slightly better than analysts' expectations. But its film division saw profits slump 27% to $284m, helped by box-office flops Alexander and Catwoman, a sharp contrast to year-earlier, when the third and final film in the Lord of the Rings trilogy boosted results. For the full-year, TimeWarner posted a profit of $3.36bn, up 27% from its 2003 performance, while revenues grew 6.4% to $42.09bn. "Our financial performance was strong, meeting or exceeding all of our full-year objectives and greatly enhancing our flexibility," chairman and chief executive Richard Parsons said. For 2005, TimeWarner is projecting operating earnings growth of around 5%, and also expects higher revenue and wider profit margins.
TimeWarner is to restate its accounts as part of efforts to resolve an inquiry into AOL by US market regulators. It has already offered to pay $300m to settle charges, in a deal that is under review by the SEC. The company said it was unable to estimate the amount it needed to set aside for legal reserves, which it previously set at $500m. It intends to adjust the way it accounts for a deal with German music publisher Bertelsmann's purchase of a stake in AOL Europe, which it had reported as advertising revenue. It will now book the sale of its stake in AOL Europe as a loss on the value of that stake.
examples/vector_databases/redis/redisqna/assets/002.txt
+7
View File
@@ -0,0 +1,7 @@
Dollar gains on Greenspan speech
The dollar has hit its highest level against the euro in almost three months after the Federal Reserve head said the US trade deficit is set to stabilise.
And Alan Greenspan highlighted the US government's willingness to curb spending and rising household savings as factors which may help to reduce it. In late trading in New York, the dollar reached $1.2871 against the euro, from $1.2974 on Thursday. Market concerns about the deficit has hit the greenback in recent months. On Friday, Federal Reserve chairman Mr Greenspan's speech in London ahead of the meeting of G7 finance ministers sent the dollar higher after it had earlier tumbled on the back of worse-than-expected US jobs data. "I think the chairman's taking a much more sanguine view on the current account deficit than he's taken for some time," said Robert Sinche, head of currency strategy at Bank of America in New York. "He's taking a longer-term view, laying out a set of conditions under which the current account deficit can improve this year and next."
Worries about the deficit concerns about China do, however, remain. China's currency remains pegged to the dollar and the US currency's sharp falls in recent months have therefore made Chinese export prices highly competitive. But calls for a shift in Beijing's policy have fallen on deaf ears, despite recent comments in a major Chinese newspaper that the "time is ripe" for a loosening of the peg. The G7 meeting is thought unlikely to produce any meaningful movement in Chinese policy. In the meantime, the US Federal Reserve's decision on 2 February to boost interest rates by a quarter of a point - the sixth such move in as many months - has opened up a differential with European rates. The half-point window, some believe, could be enough to keep US assets looking more attractive, and could help prop up the dollar. The recent falls have partly been the result of big budget deficits, as well as the US's yawning current account gap, both of which need to be funded by the buying of US bonds and assets by foreign firms and governments. The White House will announce its budget on Monday, and many commentators believe the deficit will remain at close to half a trillion dollars.
examples/vector_databases/redis/redisqna/assets/003.txt
+7
View File
@@ -0,0 +1,7 @@
Yukos unit buyer faces loan claim
The owners of embattled Russian oil giant Yukos are to ask the buyer of its former production unit to pay back a $900m (£479m) loan.
State-owned Rosneft bought the Yugansk unit for $9.3bn in a sale forced by Russia to part settle a $27.5bn tax claim against Yukos. Yukos' owner Menatep Group says it will ask Rosneft to repay a loan that Yugansk had secured on its assets. Rosneft already faces a similar $540m repayment demand from foreign banks. Legal experts said Rosneft's purchase of Yugansk would include such obligations. "The pledged assets are with Rosneft, so it will have to pay real money to the creditors to avoid seizure of Yugansk assets," said Moscow-based US lawyer Jamie Firestone, who is not connected to the case. Menatep Group's managing director Tim Osborne told the Reuters news agency: "If they default, we will fight them where the rule of law exists under the international arbitration clauses of the credit."
Rosneft officials were unavailable for comment. But the company has said it intends to take action against Menatep to recover some of the tax claims and debts owed by Yugansk. Yukos had filed for bankruptcy protection in a US court in an attempt to prevent the forced sale of its main production arm. The sale went ahead in December and Yugansk was sold to a little-known shell company which in turn was bought by Rosneft. Yukos claims its downfall was punishment for the political ambitions of its founder Mikhail Khodorkovsky and has vowed to sue any participant in the sale.
examples/vector_databases/redis/redisqna/assets/004.txt
+11
View File
@@ -0,0 +1,11 @@
High fuel prices hit BA's profits
British Airways has blamed high fuel prices for a 40% drop in profits.
Reporting its results for the three months to 31 December 2004, the airline made a pre-tax profit of £75m ($141m) compared with £125m a year earlier. Rod Eddington, BA's chief executive, said the results were "respectable" in a third quarter when fuel costs rose by £106m or 47.3%. BA's profits were still better than market expectation of £59m, and it expects a rise in full-year revenues.
To help offset the increased price of aviation fuel, BA last year introduced a fuel surcharge for passengers.
In October, it increased this from £6 to £10 one-way for all long-haul flights, while the short-haul surcharge was raised from £2.50 to £4 a leg. Yet aviation analyst Mike Powell of Dresdner Kleinwort Wasserstein says BA's estimated annual surcharge revenues - £160m - will still be way short of its additional fuel costs - a predicted extra £250m. Turnover for the quarter was up 4.3% to £1.97bn, further benefiting from a rise in cargo revenue. Looking ahead to its full year results to March 2005, BA warned that yields - average revenues per passenger - were expected to decline as it continues to lower prices in the face of competition from low-cost carriers. However, it said sales would be better than previously forecast. "For the year to March 2005, the total revenue outlook is slightly better than previous guidance with a 3% to 3.5% improvement anticipated," BA chairman Martin Broughton said. BA had previously forecast a 2% to 3% rise in full-year revenue.
It also reported on Friday that passenger numbers rose 8.1% in January. Aviation analyst Nick Van den Brul of BNP Paribas described BA's latest quarterly results as "pretty modest". "It is quite good on the revenue side and it shows the impact of fuel surcharges and a positive cargo development, however, operating margins down and cost impact of fuel are very strong," he said. Since the 11 September 2001 attacks in the United States, BA has cut 13,000 jobs as part of a major cost-cutting drive. "Our focus remains on reducing controllable costs and debt whilst continuing to invest in our products," Mr Eddington said. "For example, we have taken delivery of six Airbus A321 aircraft and next month we will start further improvements to our Club World flat beds." BA's shares closed up four pence at 274.5 pence.
examples/vector_databases/redis/redisqna/assets/005.txt
+7
View File
@@ -0,0 +1,7 @@
Pernod takeover talk lifts Domecq
Shares in UK drinks and food firm Allied Domecq have risen on speculation that it could be the target of a takeover by France's Pernod Ricard.
Reports in the Wall Street Journal and the Financial Times suggested that the French spirits firm is considering a bid, but has yet to contact its target. Allied Domecq shares in London rose 4% by 1200 GMT, while Pernod shares in Paris slipped 1.2%. Pernod said it was seeking acquisitions but refused to comment on specifics.
Pernod's last major purchase was a third of US giant Seagram in 2000, the move which propelled it into the global top three of drinks firms. The other two-thirds of Seagram was bought by market leader Diageo. In terms of market value, Pernod - at 7.5bn euros ($9.7bn) - is about 9% smaller than Allied Domecq, which has a capitalisation of £5.7bn ($10.7bn; 8.2bn euros). Last year Pernod tried to buy Glenmorangie, one of Scotland's premier whisky firms, but lost out to luxury goods firm LVMH. Pernod is home to brands including Chivas Regal Scotch whisky, Havana Club rum and Jacob's Creek wine. Allied Domecq's big names include Malibu rum, Courvoisier brandy, Stolichnaya vodka and Ballantine's whisky - as well as snack food chains such as Dunkin' Donuts and Baskin-Robbins ice cream. The WSJ said that the two were ripe for consolidation, having each dealt with problematic parts of their portfolio. Pernod has reduced the debt it took on to fund the Seagram purchase to just 1.8bn euros, while Allied has improved the performance of its fast-food chains.
examples/vector_databases/redis/redisqna/assets/006.txt
+7
View File
@@ -0,0 +1,7 @@
Japan narrowly escapes recession
Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.
Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. "It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.
examples/vector_databases/redis/redisqna/assets/007.txt
+9
View File
@@ -0,0 +1,9 @@
Jobs growth still slow in the US
The US created fewer jobs than expected in January, but a fall in jobseekers pushed the unemployment rate to its lowest level in three years.
According to Labor Department figures, US firms added only 146,000 jobs in January. The gain in non-farm payrolls was below market expectations of 190,000 new jobs. Nevertheless it was enough to push down the unemployment rate to 5.2%, its lowest level since September 2001. The job gains mean that President Bush can celebrate - albeit by a very fine margin - a net growth in jobs in the US economy in his first term in office. He presided over a net fall in jobs up to last November's Presidential election - the first President to do so since Herbert Hoover. As a result, job creation became a key issue in last year's election. However, when adding December and January's figures, the administration's first term jobs record ended in positive territory.
The Labor Department also said it had revised down the jobs gains in December 2004, from 157,000 to 133,000.
Analysts said the growth in new jobs was not as strong as could be expected given the favourable economic conditions. "It suggests that employment is continuing to expand at a moderate pace," said Rick Egelton, deputy chief economist at BMO Financial Group. "We are not getting the boost to employment that we would have got given the low value of the dollar and the still relatively low interest rate environment." "The economy is producing a moderate but not a satisfying amount of job growth," said Ken Mayland, president of ClearView Economics. "That means there are a limited number of new opportunities for workers."
examples/vector_databases/redis/redisqna/assets/008.txt
+7
View File
@@ -0,0 +1,7 @@
India calls for fair trade rules
India, which attends the G7 meeting of seven leading industrialised nations on Friday, is unlikely to be cowed by its newcomer status.
In London on Thursday ahead of the meeting, India's finance minister, lashed out at the restrictive trade policies of the G7 nations. He objected to subsidies on agriculture that make it hard for developing nations like India to compete. He also called for reform of the United Nations, the World Bank and the IMF.
Palaniappan Chidambaram, India's finance minister, argued that these organisations need to take into account the changing world order, given India and China's integration into the global economy. He said the issue is not globalisation but "the terms of engagement in globalisation." Mr Chidambaram is attending the G7 meeting as part of the G20 group of nations, which account for two thirds of the world's population. At a conference on developing enterprise hosted by UK finance minister Gordon Brown on Friday, he said that he was in favour of floating exchange rates because they help countries cope with economic shocks. "A flexible exchange rate is one more channel for absorbing both positive and negative shocks," he told the conference. India, along with China, Brazil, South Africa and Russia, has been invited to take part in the G7 meeting taking place in London on Friday and Saturday. China is expected to face renewed pressure to abandon its fixed exchange rate, which G7 nations, in particular the US, have blamed for a surge in cheap Chinese exports. "Some countries have tried to use fixed exchange rates. I do not wish to make any judgements," Mr Chidambaram said. Separately, the IMF warned on Thursday that India's budget deficit was too large and would hamper the country's economic growth, which it forecast to be around 6.5% in the year to March 2005. In the year to March 2004, the Indian economy grew by 8.5%.
examples/vector_databases/redis/redisqna/assets/009.txt
+9
View File
@@ -0,0 +1,9 @@
Ethiopia's crop production up 24%
Ethiopia produced 14.27 million tonnes of crops in 2004, 24% higher than in 2003 and 21% more than the average of the past five years, a report says.
In 2003, crop production totalled 11.49 million tonnes, the joint report from the Food and Agriculture Organisation and the World Food Programme said. Good rains, increased use of fertilizers and improved seeds contributed to the rise in production. Nevertheless, 2.2 million Ethiopians will still need emergency assistance.
The report calculated emergency food requirements for 2005 to be 387,500 tonnes. On top of that, 89,000 tonnes of fortified blended food and vegetable oil for "targeted supplementary food distributions for a survival programme for children under five and pregnant and lactating women" will be needed.
In eastern and southern Ethiopia, a prolonged drought has killed crops and drained wells. Last year, a total of 965,000 tonnes of food assistance was needed to help seven million Ethiopians. The Food and Agriculture Organisation (FAO) recommend that the food assistance is bought locally. "Local purchase of cereals for food assistance programmes is recommended as far as possible, so as to assist domestic markets and farmers," said Henri Josserand, chief of FAO's Global Information and Early Warning System. Agriculture is the main economic activity in Ethiopia, representing 45% of gross domestic product. About 80% of Ethiopians depend directly or indirectly on agriculture.
examples/vector_databases/redis/redisqna/assets/010.txt
+7
View File
@@ -0,0 +1,7 @@
Court rejects $280bn tobacco case
A US government claim accusing the country's biggest tobacco companies of covering up the effects of smoking has been thrown out by an appeal court.
The demand for $280bn (£155bn) - filed by the Clinton administration in 1999 - was rejected in a 2-1 decision. The court in Washington found that the case could not be brought under federal anti-racketeering laws. Among the accused were Altria Group, RJ Reynolds Tobacco, Lorillard Tobacco, Liggett Group and Brown and Williamson. In its case, the government claimed tobacco firms manipulated nicotine levels to increase addiction, targeted teenagers with multi-billion dollar advertising campaigns, lied about the dangers of smoking and ignored research to the contrary.
Prosecutors wanted the cigarette firms to surrender $280bn in profits accumulated over the past 50 years and impose tougher rules on marketing their products. But the Court of Appeals for the District of Columbia ruled that the US government could not sue the firms under legislation drawn up to counteract Mafia infiltration of business. The tobacco companies deny that they illegally conspired to promote smoking and defraud the public. They also say they have already met many of the government's demands in a landmark $206bn settlement reached with 46 states in 1998. Shares of tobacco companies closed higher after the ruling, with Altria rising 5% and Reynolds showing gains of 4.5%.
examples/vector_databases/redis/redisqna/assets/011.txt
+60
View File
@@ -0,0 +1,60 @@
Embattled Crypto Exchange FTX Files for Bankruptcy
Nov. 11, 2022
On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: “FTX is fine,” he wrote. “Assets are fine.”
On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.
In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.
The speed of FTX’s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.
“Here we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,” said Jared Ellias, a bankruptcy professor at Harvard Law School. “The velocity of this failure is just unbelievable.”
Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn’t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.
FTX’s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX’s collapse.
Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.
The company’s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.
“I’m really sorry, again, that we ended up here,” Mr. Bankman-Fried said on Twitter on Friday. “Hopefully this can bring some amount of transparency, trust, and governance.”
The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.
The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.
The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.
Mr. Bankman-Fried’s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.
“We told them additional employees added too quickly were net negative,” Mr. Bankman-Fried said on Twitter in June. “They could take it or leave it.”
Unusually for a major start-up, none of FTX’s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.
FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.
In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX’s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.
As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade — in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies — that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.
Mr. Bankman-Fried’s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.
On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company’s financial documents, Binance’s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.
In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill.
FTX’s bankruptcy is the latest — and by far the biggest — in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager’s remaining assets.
As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.
The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried’s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.
Mr. Bankman-Fried’s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.
“We were shocked and immensely saddened to learn of the recent events at FTX,” they wrote in a statement. “We have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.”
Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.
“I’m disappointed,” Mr. Scaramucci said in an interview on CNBC on Friday. “Duped, I guess, is the right word.”
examples/vector_databases/redis/redisqna/assets/012.txt
+53
View File
@@ -0,0 +1,53 @@
FTX founder Sam Bankman-Fried hit with four new criminal charges
PUBLISHED THU, FEB 23 202310:11 AM ESTUPDATED THU, FEB 23 20232:06 PM EST
FTX co-founder Sam Bankman-Fried was hit Thursday with four new criminal charges, including ones related to commodities fraud and making unlawful political contributions, in a superseding indictment filed in New York federal court.
A source familiar with the new counts said that SBF, as he is popularly known, could face an additional 40 years in prison if convicted in the case, where he is accused of “multiple schemes to defraud.”
The new charging document lays out in greater detail Bankman-Fried’s allegedly fraudulent conduct related to his cryptocurrency exchange FTX and an associated hedge fund, Alameda Research, both of which went bust in late 2022.
The 12-count indictment also provides new details of hundreds of political donations that Bankman-Fried allegedly directed in violation of federal campaign finance laws.
Bankman-Fried is accused of stealing FTX customer deposits and using billions of dollars of those stolen funds to support FTX’s and Alameda’s operations and investments, to fund speculative investments, to make charitable contributions, and to enrich himself, the indictment notes.
He also tried “to purchase influence over cryptocurrency regulation in Washington, D.C., by steering tens of millions of dollars in illegal campaign contributions to both Democrats and Republicans,” according to the new indictment, which was was unsealed in U.S. District Court in Manhattan.
Before the criminal case, SBF was known as a major donor to Democrats.
Bankman-Fried, who remains free on a $250 million personal recognizance bond after being first charged in late 2022, has pleaded not guilty in the case.
The new indictment adds yet more legal pressure on SBF, whose close associates, FTX co-founder Gary Wang and ex-Alameda CEO Caroline Ellison, pleaded guilty in December to multiple fraud and other charges. Both Wang and Ellison are cooperating with the U.S. attorney’s office in Manhattan against Bankman-Fried.
The new indictment accuses him of securities fraud, wire fraud, and multiple conspiracy counts related to wire fraud on FTX customers and Alameda’s lenders; illegal campaign contributions; money laundering; operating an unlicensed money transmitting business; and bank fraud.
Manhattan U.S. Attorney Damian Williams, in a statement on the new indictment said, “We are hard at work and will remain so until justice is done.”
The charging document lays out how Bankman-Fried allegedly operated an illegal straw donor scheme as he moved to use customers funds to run a multimillion-dollar political influence campaign.
Bankman-Fried and fellow FTX executives combined to contribute more than $70 million toward the 2022 midterm elections, according to campaign finance watchdog OpenSecrets.
The indictment claims that Bankman-Fried and his co-conspirators “made over 300 political contributions, totaling tens of millions of dollars, that were unlawful because they were made in the name of a straw donor or paid for with corporate funds.”
“To avoid certain contributions being publicly reported in his name, Bankman-Fried conspired to and did have certain political contributions made in the names of two other FTX executives,” the new filing claims.
The document refers to one such example, in 2022, when Bankman-Fried and “others agreed that he and his co-conspirators should contribute at least a million dollars to a super PAC that was supporting a candidate running for a United States Congressional seat and appeared to be affiliated with pro-LGBTQ issues.”
The group of conspirators, according to the document, selected an individual only identified in the document as “CC-1” or co-conspirator 1, to be the donor.
However, in 2022, then-FTX Director of Engineering Nishad Singh contributed $1.1 million to the LGBTQ Victory Fund Federal PAC, according to Federal Election Commission filings.
Singh, who did not immediately respond to a request for comment, has not been charged with any wrongdoing. Albert Fujii, a spokesman for the PAC, told CNBC “we have set aside funds and will take appropriate action once we receive guidance from authorities.”
SBF’s alleged campaign finance scheme included efforts to keep his contributions to Republicans “dark,” according to the new indictment.
And, the alleged straw donor scheme was coordinated, at least in part, “through an encrypted, auto-deleting Signal chat called ‘Donation Processing,’” according to the indictment.
The document says another unnamed co-conspirator “who publicly aligned himself with conservatives, made contributions to Republican candidates that were directed by Bankman-Fried and funded by Alameda,” the crypto tycoon’s hedge fund.
Again, the document does do not name the alleged second FTX co-conspirator who contributed to Republican candidates.
Ryan Salame, the co-CEO of FTX Digital Markets, a subsidiary of FTX, donated more than $20 million to Republicans during the 2022 election cycle, according to OpenSecrets. Salame has not been charged with any wrongdoing.
Salame could not be reached for comment. A spokeswoman for Salame did not return a request for comment.
examples/vector_databases/redis/redisqna/assets/013.txt
+72
View File
@@ -0,0 +1,72 @@
FTX Trading Ltd., commonly known as FTX (short for "Futures Exchange"),[5] is a bankrupt company that formerly operated a cryptocurrency exchange and crypto hedge fund.[6][7] The exchange was founded in 2019 by Sam Bankman-Fried and Gary Wang and, at its peak in July 2021, had over one million users and was the third-largest cryptocurrency exchange by volume.[8][9] FTX is incorporated in Antigua and Barbuda and headquartered in the Bahamas.[10] FTX is closely associated with FTX.US, a separate exchange available to US residents.[11]
Since November 11, 2022, FTX has been in Chapter 11 bankruptcy proceedings in the US court system.[12][13][14][15] Public concern began when a November 2022 CoinDesk article stated that FTX's partner firm Alameda Research held a significant portion of its assets in FTX's native token (FTT).[16][17] Following this revelation, rival exchange Binance's CEO Changpeng Zhao announced that Binance would sell its holdings of the token, which was quickly followed by a spike in customer withdrawals from FTX.[18] FTX was unable to meet the demand for customer withdrawals.[19] Binance signed a letter of intent to acquire the firm, with due diligence to follow, to ensure that customers could recover their assets from FTX in a timely manner, but Binance withdrew its offer the next day, citing reports of mishandled customer funds and U.S. agency investigations.[20] On December 12, 2022, founder Sam Bankman-Fried was arrested by the Bahamian authorities for financial offences, at the request of the US government.[21]
The current CEO of FTX is John J. Ray III, who specializes in recovering funds from failed corporations. Speaking of its previous management, Ray stated: "Never in my career have I seen such a complete failure of corporate controls and such a complete absence of trustworthy financial information as occurred here." He added that "this situation is unprecedented."
Sam Bankman-Fried and Zixiao "Gary" Wang[22] founded FTX in May 2019.[23] FTX began within Alameda Research, a trading firm founded by Bankman-Fried, Caroline Ellison, and other former employees of Jane Street in 2017, in Berkeley, California.[5][24][25] FTX is an abbreviation of "Futures Exchange".[5] Changpeng Zhao of Binance purchased a 20% stake in FTX for approximately $100 million, six months after Bankman-Fried and Wang started the firm.[26]
In August 2020, FTX acquired Blockfolio, a cryptocurrency portfolio tracking app, for $150 million.[27] In July 2021, the venture raised $900 million at an $18 billion valuation from over 60 investors, including Softbank, Sequoia Capital, and other firms.[28][29] Bankman-Fried bought out Zhao's stake for approximately $2 billion.[26] In September of that year, FTX moved its headquarters from Hong Kong to The Bahamas.[30]
On January 14, 2022, FTX announced a $2 billion venture fund named FTX Ventures,[31] raising $400 million in Series C funding at a $32 billion valuation that month.[32] The FTX Ventures website went offline in November 2022.[33] On February 11, 2022, FTX.US announced that the company would soon begin offering stock trading to its US customers.[34]
In February 2022, it was reported that FTX was creating a gaming division that would help developers add cryptocurrency, NFTs, and other blockchain-related assets into video games.[35]
In July 2022, FTX finalized a deal giving it the option to buy BlockFi for about $240 million. The deal included a $400 million credit facility for BlockFi.[36][37]
In August 2022, the Federal Deposit Insurance Corporation (FDIC) issued a cease-and-desist order to FTX for making "false and misleading representations" about deposits being covered by FDIC insurance following FTX president Brett Harrison's tweet implying otherwise.[38] Following the regulatory action, Harrison deleted the tweet and Bankman-Fried clarified in another tweet that FTX deposits are not insured by the FDIC.[39]
On September 26, 2022, FTX.US won its bid at auction for the digital assets of bankrupt crypto brokerage Voyager Digital. The value of the deal was approximately $1.42 billion, including $1.31 billion in Voyager-held cryptocurrency and $111 million in additional consideration. The deal was subject to approval by bankruptcy courts and Voyager's creditors.[40] Following the FTX bankruptcy, in December 2022, the US subsidiary of Binance won the bid to buy the assets of Voyager for approximately $1 billion.[41]
On September 27, 2022, FTX.US President Brett Harrison announced he would be stepping down from an active role at the exchange but would stay on in an advisory capacity. The company did not immediately announce a replacement for Harrison, who had been FTX.US president since May 2021.[42]
In October 2022, it was reported that FTX was under investigation in Texas for allegedly selling unregistered securities.[43]
In September 2022, Bloomberg reported on the close relationship between Alameda Research and FTX. Bloomberg noted that Alameda had functioned as a market maker for FTX early in the exchange's history, and that the trading firm remained, in June and July 2022, the biggest known depositor of stable coins on FTX.[44] Bloomberg further stated that the regulatory oversight which applies to companies operating in traditional equities markets would have prohibited the relationship between the two firms were it applicable.[44] Alameda's trading on FTX meant the trading firm was potentially in a position to gain financially when others lost money on the exchange.[45] Bankman-Fried, at points, defended Alameda's use of FTX as a liquidity provider.[45]
According to John J. Ray III, Alameda had a "secret exemption" from FTX's auto-liquidation protocol.[46] Later, the existence of such an undisclosed beneficial relationship was described by Ray, the new CEO of FTX, as a "complete failure of corporate controls"[47] and indicated gross mismanagement.[46] Between early 2021 and March 2022, Alameda Research amassed crypto tokens ahead of FTX announcing the decision to list them for trading.[48]
According to anonymous sources cited by The Wall Street Journal, FTX had lent $10 billion of its customers' assets to Alameda Research in 2022.[49] Alameda CEO Caroline Ellison disclosed to other Alameda employees that she, Sam Bankman-Fried, Gary Wang, and Nishad Singh knew about that decision.[50] An anonymous source cited by the New York Times said the same.[51] According to the sources cited by The Wall Street Journal, Ellison said the funds were used in part to pay back loans Alameda had taken to make investments.[50] Ray said that FTX used software to conceal the misuse of customer funds.[46][47][52]
Several months after Bloomberg's initial report on the relationship between the two firms, on November 2, 2022, CoinDesk reported that a significant portion of Alameda Research's assets were held in FTT, the exchange token issued by FTX. It said that there were $5.1 billion worth of FTT tokens in circulation, and that Alameda's balance sheet held $3.66 billion of "unlocked FTT", $2.16 billion of "FTT collateral", and $292 million of "locked FTT".[16] In the weeks immediately preceding the publication of the story by CoinDesk, Bankman-Fried was characterized by anonymous sources cited by Bloomberg as "desperately" attempting to raise money for FTX.[53] Additionally, Bankman-Fried had been publicly "dueling" with Changpeng Zhao on Twitter in the months preceding the CoinDesk article, in part due to disagreements stemming from their differing views on regulation of cryptocurrency.[54]
Crisis begins: Binance FTT sale, sell-off, and withdrawn rescue bid
Several days after the publication of the CoinDesk article, on November 6, Binance CEO Changpeng Zhao said on Twitter that his firm intended to sell all its holdings of FTT.[55] Binance had received FTT from FTX in 2021 during a transaction in which FTX bought back Binance's equity stake in FTX.[56] Zhao cited "recent revelations that came to light" as the motivation for selling FTT.[56] Bloomberg and TechCrunch reported that any sale by Binance would likely have an outsized impact on FTT's price, given the token's low trading volume.[57][58] The announcement by Zhao of the pending sale and disputes between Zhao and Bankman-Fried on Twitter led to a decline in the price of FTT and other cryptocurrencies,[59] resulting in a three-day depositor selloff, like a bank run, of an estimated $6 billion that sent FTX into crisis.[60] On November 8, Zhao announced Binance had entered into a non-binding agreement to purchase FTX due to what he referred to as a "liquidity crisis" at FTX.[61][62] The deal did not include the sale of FTX.US.[61] Zhao announced on Twitter that the company would complete due diligence soon, adding that all cryptocurrency exchanges should avoid using FTT tokens as collateral.[63][64] He also wrote that he expected FTT to be "highly volatile in the coming days as things develop". On the day of that announcement, FTT price dropped by 80 percent, erasing $2 billion in value.[65]
On November 9, Bloomberg called the acquisition of FTX by Binance "unlikely" due to the poor state of FTX's finances.[66] Bloomberg also reported that the United States Securities and Exchange Commission and Commodity Futures Trading Commission were investigating the nature of FTX's connections to Bankman-Fried's other holdings and its handling of client funds.[67] Later that day, the Wall Street Journal reported that Binance would not move forward with the deal to acquire FTX.[68] Binance cited FTX's reported mishandling of customer funds and pending investigations of FTX as the reasons for not pursuing the deal.[69] Bankman-Fried said in a Slack message that FTX had learned through the press about Binance’s concern and decision.[68]
On November 9, FTX's website said that it was not processing withdrawals at that time.[60] Bankman-Fried said that although the firm's assets were worth more than its clients' deposits, it would need funds from outside to meet demand for withdrawals due to a lack of liquidity.[70][71] Bankman-Fried stated on November 9 that FTX.US, as a separate company, was "not currently impacted" by the crisis.[72]
Bankruptcy and unauthorized transactions
On November 10, Axios cited anonymous sources who said that FTX approached Kraken for a potential rescue deal.[73] Bankman-Fried made several statements on November 10, taking responsibility for FTX's failure and indicating that FTX was still seeking capital to remain solvent.[74] Bankman-Fried also announced that Alameda Research would cease trading and end operations.[75] FTX's in-house legal and compliance teams had, for the most part, resigned by November 10.[76][77] Anonymous sources cited by the Wall Street Journal on November 10 said that Alameda Research owed FTX some $10 billion, as FTX had lent funds placed on the exchange for trading to Alameda so that Alameda could make investments with the money.[49] On November 12, anonymous sources cited by the Wall Street Journal said Alameda CEO Caroline Ellison disclosed to other Alameda employees that she, Sam Bankman-Fried, Gary Wang, and Nishad Singh knew that client deposits were transferred from FTX to Alameda.[50] An anonymous source cited by the New York Times on November 14 said the same.[51] According to the sources cited by The Wall Street Journal, Ellison said the funds were used in part to pay back loans Alameda had taken to make investments.[50]
Though Bankman-Fried, on November 10, wrote on Twitter that FTX's US customers did not have reason to worry, employees began attempting to sell assets belonging to the firm on the same day.[78] These assets include stock-clearing company Embed Financial Technologies and the naming rights to FTX Arena.[78]
On November 10, the Securities Commission of the Bahamas froze the assets of one of FTX's subsidiaries, FTX Digital Markets Ltd, "and related parties", and provisionally appointed an attorney as liquidator.[79][80] Japan's Financial Services Agency ordered FTX Japan to suspend some operations.[81][82] The company's Australian subsidiary was placed under administration.[81]
On November 10, the team running the FTX Future Fund, an otensibly charitable group bankrolled by Bankman-Fried, announced that they had resigned earlier that day.[12] Future Fund had committed $160 million in charitable grants and investments by September 1 of that year.[83] Crypto lender BlockFi, which was affiliated with FTX, announced on November 10 that it was suspending operations as a result of FTX's collapse.[12]
On November 11, FTX, FTX.US, Alameda Research, and more than 100 affiliates filed for bankruptcy in Delaware.[14][12][15] Anonymous sources cited by the New York Times said that the exchange owed as much as $8 billion.[12] Bankman-Fried resigned as CEO and was replaced by John J. Ray III, a corporate restructuring specialist who'd previously overseen the liquidation of Enron.[14][15][84]
Late on November 11, over $473 million in funds were siphoned from FTX through what Ryne Miller, FTX US's general counsel, characterized as "unauthorized transactions".[85] Miller announced that FTX and FTX US intended to move remaining funds denominated in cryptocurrency to offline "cold storage" for security.[85] The funds taken from FTX were mostly stablecoins such as Tether, and were quickly exchanged for Ether, a method used by cryptocurrency thieves to thwart attempts to retrieve stolen funds.[86] A person speaking on behalf of FTX referred to the "unauthorized transactions" as a "hack" and encouraged users to delete FTX mobile apps as they were compromised.[87] Kraken has since offered to assist in identifying the perpetrator.[88]
As of November 12, Bankman-Fried told Reuters that he was still in the Bahamas,[89] though other high-ranking FTX employees had begun leaving for Hong Kong, the location of the company's former headquarters, or other locations.[87] Authorities in the Bahamas, including the Royal Bahamas Police Force, questioned Bankman-Fried on November 12.[90] Despite FTX's bankruptcy, Bankman-Fried continued to attempt to raise money for the firm during the weekend of November 12 and 13.[91]
On November 14, Kraken's chief security officer said on Twitter that the firm knew "the identity" of a user who paid transaction fees associated with moving the stolen money through their Kraken account.[92] In an interview with Kelsey Piper published November 16 by Vox, Bankman-Fried blamed an "ex-employee" or malware on a device owned by an ex-employee for the theft.[93]
According to anonymous sources cited by Reuters, between $1 billion and $2 billion in customer funds could not be accounted for as of November 12.[94][95] The Financial Times reported that FTX's balance sheet shortly before the bankruptcy showed $9 billion in liabilities, with $900 million in liquid assets, $5 billion in "less liquid" assets, and $3.2 billion in illiquid private equity investments.[96]
Bankman-Fried began publishing "cryptic" messages in sequence on Twitter on November 14.[97] As of November 15, the messages all read "What HAPPENED".[97]
On November 15, FTX sought to raise $10 billion in liquidity from investors.[98]
On November 16, the Bahamas unit of FTX, FTX Digital Markets, officially filed for Chapter 15 bankruptcy protection in the United States.[99]
On November 17, John J. Ray III, the CEO brought in as a liquidator, stated in a sworn declaration submitted in bankruptcy court that, according to FTX's records, its subsidiary Alameda Research had on September 30 lent $1 billion to Bankman-Fried and more than $500 million to FTX co-founder Nishad Singh.[100] Ray, having been involved in the bankruptcies of Enron, Residential Capital, Nortel and Overseas Shipholding, stated, "Never in my career have I seen such a complete failure of corporate controls and such a complete absence of trustworthy financial information as occurred here. From compromised systems integrity and faulty regulatory oversight abroad, to the concentration of control in the hands of a very small group of inexperienced, unsophisticated and potentially compromised individuals, this situation is unprecedented."[101][102] Speaking to the House Committee on Financial Services, he testified that "literally, there’s no record-keeping whatsoever" and that the company used for its accounting needs QuickBooks, a small-business accounting tool, despite handling "billions of dollars."[103]
Widening impact and contagion fears
The exchange token of Crypto.com, Cronos, lost approximately $1 billion in value in November,[104] a decline attributed in part due to the collapse of FTX and in part due to reporting that Crypto.com had accidentally sent $400 million of Ether to another exchange.[49][50] On November 14, Crypto.com's CEO assured users that the exchange was functioning as normal.[104] Commenters and customers remained fearful that Crypto.com could experience a collapse similar to FTX.[105]
BlockFi, a cryptocurrency lender, was reportedly taking steps to file for bankruptcy as of November 15.[106] The firm had earlier begun preventing withdrawals.[106] The company disclosed "significant exposure" to FTX on November 14.[106] Another cryptocurrency lender, Genesis, a subsidiary of Digital Currency Group, halted withdrawals on November 16.[107] This halt caused Gemini, an exchange owned by the Winklevoss twins, to cease allowing redemptions for clients using a service provided through a partnership with Genesis.[108] Another Digital Currency Group subsidiary, Grayscale, saw the value of its flagship offering, the publicly traded Grayscale Bitcoin Trust, decline by 20% over the two weeks preceding November 17.[109] Grayscale Bitcoin Trust was trading at a discounted price, 42% below the value of its Bitcoin, as of November 14.[110]
Concerns have also been raised about Silvergate Bank, as FTX was a depositor and could have also been a source of credit exposure. Silvergate has said that it has ample liquidity and no loan exposure to FTX. These concerns have been magnified due to Silvergate's key role as a gateway between its cryptocurrency clients and the wider financial world.[111][112]
examples/vector_databases/redis/redisqna/docker-compose.yml
+5
View File
@@ -0,0 +1,5 @@
services:
redis:
image: redis/redis-stack-server:latest
ports:
- 6379:6379
examples/vector_databases/redis/redisqna/redisqna.ipynb
+384
View File
@@ -0,0 +1,384 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Redis as a Context Store with OpenAI Chat\n",
"This notebook demonstrates how to use Redis as high-speed context memory with ChatGPT.\n",
"\n",
"## Prerequisites\n",
"* Redis instance with the Redis Search and Redis JSON modules\n",
"* Redis-py client lib\n",
"* OpenAI Python client lib\n",
"* OpenAI API key\n",
"\n",
"## Installation\n",
"Install Python modules necessary for the examples."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"vscode": {
"languageId": "shellscript"
}
},
"outputs": [],
"source": [
"! pip install -q redis openai python-dotenv 'openai[datalib]'"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## OpenAI API Key\n",
"Create a .env file and add your OpenAI key to it"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"vscode": {
"languageId": "shellscript"
}
},
"outputs": [],
"source": [
"OPENAI_API_KEY=your_key"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## OpenAI Setup\n",
"Key load + helper function for chat completion"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from openai import OpenAI\n",
"import os\n",
"from dotenv import load_dotenv\n",
"\n",
"load_dotenv()\n",
"oai_client = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n",
"\n",
"def get_completion(prompt, model=\"gpt-3.5-turbo\"):\n",
" messages = [{\"role\": \"user\", \"content\": prompt}]\n",
" response = oai_client.chat.completions.create(\n",
" model=model,\n",
" messages=messages,\n",
" temperature=0,\n",
" )\n",
" return response.choices[0].message.content"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experiment - Chat Completion on a Topic outside of the Model's Knowledge Cutoff Date\n",
"Gpt-3.5-turbo was trained on data up to Sep 2021. Let's ask it a question about something that is beyond that date. In this case, the FTX/Sam Bankman-Fried scandal. We are using an old model here for demonstration. Newer models such as got-4o has later knowledge cutoffs (late 2023) and will work here as well."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Yes, FTX is generally considered a well-managed company. Sam Bankman-Fried, the founder and CEO of FTX, has a strong track record in the cryptocurrency industry and has successfully grown the company into one of the leading cryptocurrency exchanges in the world. FTX has also received positive reviews for its user-friendly platform, innovative products, and strong customer service. Additionally, FTX has been proactive in regulatory compliance and has taken steps to ensure the security of its users' funds. Overall, FTX is seen as a well-managed company in the cryptocurrency space.\n"
]
}
],
"source": [
"prompt = \"Is Sam Bankman-Fried's company, FTX, considered a well-managed company?\"\n",
"response = get_completion(prompt)\n",
"print(response)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Incomplete Information\n",
"An unfortunate behavior of these AI systems is the system will provide a confident-sounding response - even when the system is not confident with its result. One way to mitigate this is prompt re-engineering, as seen below."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"FTX is generally considered a well-managed company. Sam Bankman-Fried, the founder and CEO, has a strong reputation in the cryptocurrency industry for his leadership and strategic vision. FTX has also experienced significant growth and success since its founding in 2017. However, without specific insider knowledge or data, it is ultimately unknown whether FTX is definitively considered a well-managed company.\n"
]
}
],
"source": [
"prompt =\"Is Sam Bankman-Fried's company, FTX, considered a well-managed company? If you don't know for certain, say unknown.\"\n",
"response = get_completion(prompt)\n",
"print(response)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Additional Context\n",
"Another way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing. We'll use Redis as the source for that additional context. We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Start the Redis Stack Docker container"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! docker compose up -d"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect Redis client"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from redis import from_url\n",
"\n",
"REDIS_URL = 'redis://localhost:6379'\n",
"client = from_url(REDIS_URL)\n",
"client.ping()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Index\n",
"[FT.CREATE](https://redis.io/commands/ft.create/)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"b'OK'"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from redis.commands.search.field import TextField, VectorField\n",
"from redis.commands.search.indexDefinition import IndexDefinition, IndexType\n",
"\n",
"schema = [ VectorField('$.vector', \n",
" \"FLAT\", \n",
" { \"TYPE\": 'FLOAT32', \n",
" \"DIM\": 1536, \n",
" \"DISTANCE_METRIC\": \"COSINE\"\n",
" }, as_name='vector' ),\n",
" TextField('$.content', as_name='content')\n",
" ]\n",
"idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\n",
"try: \n",
" client.ft('idx').dropindex()\n",
"except:\n",
" pass\n",
"client.ft('idx').create_index(schema, definition=idx_def)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Data Files into Redis as JSON Objects with Text and Vector Fields\n",
"[Redis JSON](https://redis.io/docs/stack/json/)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"directory = './assets/'\n",
"model = 'text-embedding-3-small'\n",
"i = 1\n",
"\n",
"for file in os.listdir(directory):\n",
" with open(os.path.join(directory, file), 'r') as f:\n",
" content = f.read()\n",
" # Create the embedding using the new client-based method\n",
" response = oai_client.embeddings.create(\n",
" model=model,\n",
" input=[content]\n",
" )\n",
" # Access the embedding from the response object\n",
" vector = response.data[0].embedding\n",
" \n",
" # Store the content and vector using your JSON client\n",
" client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})\n",
" i += 1"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Embed the Question and Perform VSS to find the most relevant document\n",
"[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from redis.commands.search.query import Query\n",
"import numpy as np\n",
"\n",
"response = oai_client.embeddings.create(\n",
" input=[prompt],\n",
" model=model\n",
")\n",
"# Extract the embedding vector from the response\n",
"embedding_vector = response.data[0].embedding\n",
"\n",
"# Convert the embedding to a numpy array of type float32 and then to bytes\n",
"vec = np.array(embedding_vector, dtype=np.float32).tobytes()\n",
"\n",
"# Build and execute the Redis query\n",
"q = Query('*=>[KNN 1 @vector $query_vec AS vector_score]') \\\n",
" .sort_by('vector_score') \\\n",
" .return_fields('content') \\\n",
" .dialect(2)\n",
"params = {\"query_vec\": vec}\n",
"\n",
"context = client.ft('idx').search(q, query_params=params).docs[0].content\n",
"print(context)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Repeat the Question to OpenAI with context\n",
"Now that we have relevant context, add that to the prompt to OpenAI and get a very different response."
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Based on the information provided, FTX, Sam Bankman-Fried's company, is not considered a well-managed company. The company has faced bankruptcy proceedings, mishandling of customer funds, unauthorized transactions, freezing of assets by regulatory authorities, and a lack of trustworthy financial information. The new CEO, John J. Ray III, described the situation as a \"complete failure of corporate controls\" and indicated gross mismanagement. Additionally, the company's financial situation, lack of record-keeping, and use of inadequate accounting tools despite handling billions of dollars have raised serious concerns about its management practices.\n"
]
}
],
"source": [
"prompt = f\"\"\"\n",
"Using the information delimited by triple backticks, answer this question: Is Sam Bankman-Fried's company, FTX, considered a well-managed company?\n",
"\n",
"Context: ```{context}```\n",
"\"\"\"\n",
"\n",
"response = get_completion(prompt)\n",
"print(response)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
examples/vector_databases/supabase/README.md
+32
View File
@@ -0,0 +1,32 @@
# Supabase Vector Database
[Supabase](https://supabase.com/docs) is an open-source Firebase alternative built on top of [Postgres](https://en.wikipedia.org/wiki/PostgreSQL), a production-grade SQL database.
[Supabase Vector](https://supabase.com/docs/guides/ai) is a vector toolkit built on [pgvector](https://github.com/pgvector/pgvector), a Postgres extension that allows you to store your embeddings inside the same database that holds the rest of your application data. When combined with pgvector's indexing algorithms, vector search remains [fast at large scales](https://supabase.com/blog/increase-performance-pgvector-hnsw).
Supabase adds an ecosystem of services and tools on top of Postgres that makes app development as quick as possible, including:
- [Auto-generated REST APIs](https://supabase.com/docs/guides/api)
- [Auto-generated GraphQL APIs](https://supabase.com/docs/guides/graphql)
- [Realtime APIs](https://supabase.com/docs/guides/realtime)
- [Authentication](https://supabase.com/docs/guides/auth)
- [File storage](https://supabase.com/docs/guides/storage)
- [Edge functions](https://supabase.com/docs/guides/functions)
We can use these services alongside pgvector to store and query embeddings within Postgres.
## OpenAI Cookbook Examples
Below are guides and resources that walk you through how to use OpenAI embedding models with Supabase Vector.
| Guide | Description |
| ---------------------------------------- | ---------------------------------------------------------- |
| [Semantic search](./semantic-search.mdx) | Store, index, and query embeddings at scale using pgvector |
## Additional resources
- [Vector columns](https://supabase.com/docs/guides/ai/vector-columns)
- [Vector indexes](https://supabase.com/docs/guides/ai/vector-indexes)
- [RAG with permissions](https://supabase.com/docs/guides/ai/rag-with-permissions)
- [Going to production](https://supabase.com/docs/guides/ai/going-to-prod)
- [Deciding on compute](https://supabase.com/docs/guides/ai/choosing-compute-addon)
examples/vector_databases/supabase/semantic-search.mdx
+276
View File
@@ -0,0 +1,276 @@
# Semantic search using Supabase Vector
The purpose of this guide is to demonstrate how to store OpenAI embeddings in [Supabase Vector](https://supabase.com/docs/guides/ai) (Postgres + pgvector) for the purposes of semantic search.
[Supabase](https://supabase.com/docs) is an open-source Firebase alternative built on top of [Postgres](https://en.wikipedia.org/wiki/PostgreSQL), a production-grade SQL database. Since Supabase Vector is built on [pgvector](https://github.com/pgvector/pgvector), you can store your embeddings within the same database that holds the rest of your application data. When combined with pgvector's indexing algorithms, vector search remains [fast at large scales](https://supabase.com/blog/increase-performance-pgvector-hnsw).
Supabase adds an ecosystem of services and tools to make app development as quick as possible (such as an [auto-generated REST API](https://postgrest.org/)). We'll use these services to store and query embeddings within Postgres.
This guide covers:
1. [Setting up your database](#setup-database)
2. [Creating a SQL table](#create-a-vector-table) that can store vector data
3. [Generating OpenAI embeddings](#generate-openai-embeddings) using OpenAI's JavaScript client
4. [Storing the embeddings](#store-embeddings-in-database) in your SQL table using the Supabase JavaScript client
5. [Performing semantic search](#semantic-search) over the embeddings using a Postgres function and the Supabase JavaScript client
## Setup database
First head over to https://database.new to provision your Supabase database. This will create a Postgres database on the Supabase cloud platform. Alternatively, you can follow the [local development](https://supabase.com/docs/guides/cli/getting-started) options if you prefer to run your database locally using Docker.
In the studio, jump to the [SQL editor](https://supabase.com/dashboard/project/_/sql/new) and execute the following SQL to enable pgvector:
```sql
-- Enable the pgvector extension
create extension if not exists vector;
```
> In a production application, the best practice is to use [database migrations](https://supabase.com/docs/guides/cli/local-development#database-migrations) so that all SQL operations are managed within source control. To keep things simple in this guide, we'll execute queries directly in the SQL Editor. If you are building a production app, feel free to move these into a database migration.
## Create a vector table
Next we'll create a table to store documents and embeddings. In the SQL Editor, run:
```sql
create table documents (
id bigint primary key generated always as identity,
content text not null,
embedding vector (1536) not null
);
```
Since Supabase is built on Postgres, we're just using regular SQL here. You can modify this table however you like to better fit your application. If you have existing database tables, you can simply add a new `vector` column to the appropriate table.
The important piece to understand is the `vector` data type, which is a new data type that became available when we enabled the pgvector extension earlier. The size of the vector (1536 here) represents the number of dimensions in the embedding. Since we're using OpenAI's `text-embedding-3-small` model in this example, we set the vector size to 1536.
Let's go ahead and create a vector index on this table so that future queries remain performant as the table grows:
```sql
create index on documents using hnsw (embedding vector_ip_ops);
```
This index uses the [HNSW](https://supabase.com/docs/guides/ai/vector-indexes/hnsw-indexes) algorithm to index vectors stored in the `embedding` column, and specifically when using the inner product operator (`<#>`). We'll explain more about this operator later when we implement our match function.
Let's also follow security best practices by enabling row level security on the table:
```sql
alter table documents enable row level security;
```
This will prevent unauthorized access to this table through the auto-generated REST API (more on this shortly).
## Generate OpenAI embeddings
This guide uses JavaScript to generate embeddings, but you can easily modify it to use any [language supported by OpenAI](https://platform.openai.com/docs/libraries).
If you are using JavaScript, feel free to use whichever server-side JavaScript runtime that you prefer (Node.js, Deno, Supabase Edge Functions).
If you're using Node.js, first install `openai` as a dependency:
```shell
npm install openai
```
then import it:
```js
import OpenAI from "openai";
```
If you're using Deno or Supabase Edge Functions, you can import `openai` directly from a URL:
```js
import OpenAI from "https://esm.sh/openai@4";
```
> In this example we import from https://esm.sh which is a CDN that automatically fetches the respective NPM module for you and serves it over HTTP.
Next we'll generate an OpenAI embedding using [`text-embedding-3-small`](https://platform.openai.com/docs/guides/embeddings/embedding-models):
```js
const openai = new OpenAI();
const input = "The cat chases the mouse";
const result = await openai.embeddings.create({
input,
model: "text-embedding-3-small",
});
const [{ embedding }] = result.data;
```
Remember that you will need an [OpenAI API key](https://platform.openai.com/api-keys) to interact with the OpenAI API. You can pass this as an environment variable called `OPENAI_API_KEY`, or manually set it when you instantiate your OpenAI client:
```js
const openai = new OpenAI({
apiKey: "<openai-api-key>",
});
```
_**Remember:** Never hard-code API keys in your code. Best practice is to either store it in a `.env` file and load it using a library like [`dotenv`](https://github.com/motdotla/dotenv) or load it from an external key management system._
## Store embeddings in database
Supabase comes with an [auto-generated REST API](https://postgrest.org/) that dynamically builds REST endpoints for each of your tables. This means you don't need to establish a direct Postgres connection to your database - instead you can interact with it simply using by the REST API. This is especially useful in serverless environments that run short-lived processes where re-establishing a database connection every time can be expensive.
Supabase comes with a number of [client libraries](https://supabase.com/docs#client-libraries) to simplify interaction with the REST API. In this guide we'll use the [JavaScript client library](https://supabase.com/docs/reference/javascript), but feel free to adjust this to your preferred language.
If you're using Node.js, install `@supabase/supabase-js` as a dependency:
```shell
npm install @supabase/supabase-js
```
then import it:
```js
import { createClient } from "@supabase/supabase-js";
```
If you're using Deno or Supabase Edge Functions, you can import `@supabase/supabase-js` directly from a URL:
```js
import { createClient } from "https://esm.sh/@supabase/supabase-js@2";
```
Next we'll instantiate our Supabase client and configure it so that it points to your Supabase project. In this guide we'll store a reference to your Supabase URL and key in a `.env` file, but feel free to modify this based on how your application handles configuration.
If you are using Node.js or Deno, add your Supabase URL and service role key to a `.env` file. If you are using the cloud platform, you can find these from your Supabase dashboard [settings page](https://supabase.com/dashboard/project/_/settings/api). If you're running Supabase locally, you can find these by running `npx supabase status` in a terminal.
_.env_
```
SUPABASE_URL=<supabase-url>
SUPABASE_SERVICE_ROLE_KEY=<supabase-service-role-key>
```
If you are using Supabase Edge Functions, these environment variables are automatically injected into your function for you so you can skip the above step.
Next we'll pull these environment variables into our app.
In Node.js, install the `dotenv` dependency:
```shell
npm install dotenv
```
And retrieve the environment variables from `process.env`:
```js
import { config } from "dotenv";
// Load .env file
config();
const supabaseUrl = process.env["SUPABASE_URL"];
const supabaseServiceRoleKey = process.env["SUPABASE_SERVICE_ROLE_KEY"];
```
In Deno, load the `.env` file using the `dotenv` standard library:
```js
import { load } from "https://deno.land/std@0.208.0/dotenv/mod.ts";
// Load .env file
const env = await load();
const supabaseUrl = env["SUPABASE_URL"];
const supabaseServiceRoleKey = env["SUPABASE_SERVICE_ROLE_KEY"];
```
In Supabase Edge Functions, simply load the injected environment variables directly:
```js
const supabaseUrl = Deno.env.get("SUPABASE_URL");
const supabaseServiceRoleKey = Deno.env.get("SUPABASE_SERVICE_ROLE_KEY");
```
Next let's instantiate our `supabase` client:
```js
const supabase = createClient(supabaseUrl, supabaseServiceRoleKey, {
auth: { persistSession: false },
});
```
From here we use the `supabase` client to insert our text and embedding (generated earlier) into the database:
```js
const { error } = await supabase.from("documents").insert({
content: input,
embedding,
});
```
> In production, best practice would be to check the response `error` to see if there were any problems inserting the data and handle it accordingly.
## Semantic search
Finally let's perform semantic search over the embeddings in our database. At this point we'll assume your `documents` table has been filled with multiple records that we can search over.
Let's create a match function in Postgres that performs the semantic search query. Execute the following in the [SQL Editor](https://supabase.com/dashboard/project/_/sql/new):
```sql
create function match_documents (
query_embedding vector (1536),
match_threshold float,
)
returns setof documents
language plpgsql
as $$
begin
return query
select *
from documents
where documents.embedding <#> query_embedding < -match_threshold
order by documents.embedding <#> query_embedding;
end;
$$;
```
This function accepts a `query_embedding` which represents the embedding generated from the search query text (more on this shortly). It also accepts a `match_threshold` which specifies how similar the document embeddings have to be in order for `query_embedding` to count as a match.
Inside the function we implement the query which does two things:
- Filters the documents to only include those who's embeddings match within the above `match_threshold`. Since the `<#>` operator performs the negative inner product (versus positive inner product), we negate the similarity threshold before comparing. This means a `match_threshold` of 1 is most similar, and -1 is most dissimilar.
- Orders the documents by negative inner product (`<#>`) ascending. This allows us to retrieve documents that match closest first.
> Since OpenAI embeddings are normalized, we opted to use inner product (`<#>`) because it is slightly more performant than other operators like cosine distance (`<=>`). It is important to note though this only works because the embeddings are normalized - if they weren't, cosine distance should be used.
Now we can call this function from our application using the `supabase.rpc()` method:
```js
const query = "What does the cat chase?";
// First create an embedding on the query itself
const result = await openai.embeddings.create({
input: query,
model: "text-embedding-3-small",
});
const [{ embedding }] = result.data;
// Then use this embedding to search for matches
const { data: documents, error: matchError } = await supabase
.rpc("match_documents", {
query_embedding: embedding,
match_threshold: 0.8,
})
.select("content")
.limit(5);
```
In this example, we set a match threshold to 0.8. Adjust this threshold based on what works best with your data.
Note that since `match_documents` returns a set of `documents`, we can treat this `rpc()` like a regular table query. Specifically this means we can chain additional commands to this query, like `select()` and `limit()`. Here we select just the columns we care about from the `documents` table (`content`), and we limit the number of documents returned (max 5 in this example).
At this point you have a list of documents that matched the query based on semantic relationship, ordered by most similar first.
## Next steps
You can use this example as the foundation for other semantic search techniques, like retrieval augmented generation (RAG).
For more information on OpenAI embeddings, read the [Embedding](https://platform.openai.com/docs/guides/embeddings) docs.
For more information on Supabase Vector, read the [AI & Vector](https://supabase.com/docs/guides/ai) docs.
examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb
+546
View File
@@ -0,0 +1,546 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using Tair as a vector database for OpenAI embeddings\n",
"\n",
"This notebook guides you step by step on using Tair as a vector database for OpenAI embeddings.\n",
"\n",
"This notebook presents an end-to-end process of:\n",
"1. Using precomputed embeddings created by OpenAI API.\n",
"2. Storing the embeddings in a cloud instance of Tair.\n",
"3. Converting raw text query to an embedding with OpenAI API.\n",
"4. Using Tair to perform the nearest neighbour search in the created collection.\n",
"\n",
"### What is Tair\n",
"\n",
"[Tair](https://www.alibabacloud.com/help/en/tair/latest/what-is-tair) is a cloud native in-memory database service that is developed by Alibaba Cloud. Tair is compatible with open source Redis and provides a variety of data models and enterprise-class capabilities to support your real-time online scenarios. Tair also introduces persistent memory-optimized instances that are based on the new non-volatile memory (NVM) storage medium. These instances can reduce costs by 30%, ensure data persistence, and provide almost the same performance as in-memory databases. Tair has been widely used in areas such as government affairs, finance, manufacturing, healthcare, and pan-Internet to meet their high-speed query and computing requirements.\n",
"\n",
"[Tairvector](https://www.alibabacloud.com/help/en/tair/latest/tairvector) is an in-house data structure that provides high-performance real-time storage and retrieval of vectors. TairVector provides two indexing algorithms: Hierarchical Navigable Small World (HNSW) and Flat Search. Additionally, TairVector supports multiple distance functions, such as Euclidean distance, inner product, and Jaccard distance. Compared with traditional vector retrieval services, TairVector has the following advantages:\n",
"- Stores all data in memory and supports real-time index updates to reduce latency of read and write operations.\n",
"- Uses an optimized data structure in memory to better utilize storage capacity.\n",
"- Functions as an out-of-the-box data structure in a simple and efficient architecture without complex modules or dependencies.\n",
"\n",
"### Deployment options\n",
"\n",
"- Using [Tair Cloud Vector Database](https://www.alibabacloud.com/help/en/tair/latest/getting-started-overview). [Click here](https://www.alibabacloud.com/product/tair) to fast deploy it.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"For the purposes of this exercise we need to prepare a couple of things:\n",
"\n",
"1. Tair cloud server instance.\n",
"2. The 'tair' library to interact with the tair database.\n",
"3. An [OpenAI API key](https://beta.openai.com/account/api-keys).\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install requirements\n",
"\n",
"This notebook obviously requires the `openai` and `tair` packages, but there are also some other additional libraries we will use. The following command installs them all:\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.718972Z",
"start_time": "2023-02-16T12:04:30.434820Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: http://sg.mirrors.cloud.aliyuncs.com/pypi/simple/\n",
"Requirement already satisfied: openai in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.28.0)\n",
"Requirement already satisfied: redis in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (5.0.0)\n",
"Requirement already satisfied: tair in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (1.3.6)\n",
"Requirement already satisfied: pandas in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (2.1.0)\n",
"Requirement already satisfied: wget in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (3.2)\n",
"Requirement already satisfied: requests>=2.20 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (2.31.0)\n",
"Requirement already satisfied: tqdm in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (4.66.1)\n",
"Requirement already satisfied: aiohttp in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (3.8.5)\n",
"Requirement already satisfied: async-timeout>=4.0.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from redis) (4.0.3)\n",
"Requirement already satisfied: numpy>=1.22.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (1.25.2)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2023.3.post1)\n",
"Requirement already satisfied: tzdata>=2022.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2023.3)\n",
"Requirement already satisfied: six>=1.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.2.0)\n",
"Requirement already satisfied: idna<4,>=2.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.4)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2.0.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2023.7.22)\n",
"Requirement already satisfied: attrs>=17.3.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (22.1.0)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (6.0.4)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.9.2)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.4.0)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.3.1)\n",
"\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"! pip install openai redis tair pandas wget"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare your OpenAI API key\n",
"\n",
"The OpenAI API key is used for vectorization of the documents and queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it by getpass."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.730338Z",
"start_time": "2023-02-16T12:05:05.723351Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input your OpenAI API key:········\n"
]
}
],
"source": [
"import getpass\n",
"import openai\n",
"\n",
"openai.api_key = getpass.getpass(\"Input your OpenAI API key:\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to Tair\n",
"First add it to your environment variables.\n",
"\n",
"Connecting to a running instance of Tair server is easy with the official Python library."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input your tair url:········\n"
]
}
],
"source": [
"# The format of url: redis://[[username]:[password]]@localhost:6379/0\n",
"TAIR_URL = getpass.getpass(\"Input your tair url:\")"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"from tair import Tair as TairClient\n",
"\n",
"# connect to tair from url and create a client\n",
"\n",
"url = TAIR_URL\n",
"client = TairClient.from_url(url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can test the connection by ping:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.848488Z",
"start_time": "2023-02-16T12:05:06.832612Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.ping()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:37.371951Z",
"start_time": "2023-02-16T12:05:06.851634Z"
},
"pycharm": {
"is_executing": true
},
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100% [......................................................................] 698933052 / 698933052"
]
},
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded (1).zip'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The downloaded file has to then be extracted:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:06:01.538851Z",
"start_time": "2023-02-16T12:05:37.376042Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n"
]
}
],
"source": [
"import zipfile\n",
"import os\n",
"import re\n",
"import tempfile\n",
"\n",
"current_directory = os.getcwd()\n",
"zip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\n",
"output_directory = os.path.join(current_directory, \"../../data\")\n",
"\n",
"with zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n",
" zip_ref.extractall(output_directory)\n",
"\n",
"\n",
"# check the csv file exist\n",
"file_name = \"vector_database_wikipedia_articles_embedded.csv\"\n",
"data_directory = os.path.join(current_directory, \"../../data\")\n",
"file_path = os.path.join(data_directory, file_name)\n",
"\n",
"\n",
"if os.path.exists(file_path):\n",
" print(f\"The file {file_name} exists in the data directory.\")\n",
"else:\n",
" print(f\"The file {file_name} does not exist in the data directory.\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Index\n",
"\n",
"Tair stores data in indexes where each object is described by one key. Each key contains a vector and multiple attribute_keys.\n",
"\n",
"We will start with creating two indexes, one for **title_vector** and one for **content_vector**, and then we will fill it with our precomputed embeddings."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Index already exists\n",
"Index already exists\n"
]
}
],
"source": [
"# set index parameters\n",
"index = \"openai_test\"\n",
"embedding_dim = 1536\n",
"distance_type = \"L2\"\n",
"index_type = \"HNSW\"\n",
"data_type = \"FLOAT32\"\n",
"\n",
"# Create two indexes, one for title_vector and one for content_vector, skip if already exists\n",
"index_names = [index + \"_title_vector\", index+\"_content_vector\"]\n",
"for index_name in index_names:\n",
" index_connection = client.tvs_get_index(index_name)\n",
" if index_connection is not None:\n",
" print(\"Index already exists\")\n",
" else:\n",
" client.tvs_create_index(name=index_name, dim=embedding_dim, distance_type=distance_type,\n",
" index_type=index_type, data_type=data_type)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"from ast import literal_eval\n",
"# Path to your local CSV file\n",
"csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n",
"article_df = pd.read_csv(csv_file_path)\n",
"\n",
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval).values\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval).values\n",
"\n",
"# add/update data to indexes\n",
"for i in range(len(article_df)):\n",
" # add data to index with title_vector\n",
" client.tvs_hset(index=index_names[0], key=article_df.id[i].item(), vector=article_df.title_vector[i], is_binary=False,\n",
" **{\"url\": article_df.url[i], \"title\": article_df.title[i], \"text\": article_df.text[i]})\n",
" # add data to index with content_vector\n",
" client.tvs_hset(index=index_names[1], key=article_df.id[i].item(), vector=article_df.content_vector[i], is_binary=False,\n",
" **{\"url\": article_df.url[i], \"title\": article_df.title[i], \"text\": article_df.text[i]})"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.675202Z",
"start_time": "2023-02-16T12:30:40.655654Z"
},
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Count in openai_test_title_vector:25000\n",
"Count in openai_test_content_vector:25000\n"
]
}
],
"source": [
"# Check the data count to make sure all the points have been stored\n",
"for index_name in index_names:\n",
" stats = client.tvs_get_index(index_name)\n",
" count = int(stats[\"current_record_count\"]) - int(stats[\"delete_record_count\"])\n",
" print(f\"Count in {index_name}:{count}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search data\n",
"\n",
"Once the data is put into Tair we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model, we also have to use it during search.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:38.024370Z",
"start_time": "2023-02-16T12:30:37.712816Z"
}
},
"outputs": [],
"source": [
"def query_tair(client, query, vector_name=\"title_vector\", top_k=5):\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.Embedding.create(\n",
" input= query,\n",
" model=\"text-embedding-3-small\",\n",
" )[\"data\"][0]['embedding']\n",
" embedded_query = np.array(embedded_query)\n",
"\n",
" # search for the top k approximate nearest neighbors of vector in an index\n",
" query_result = client.tvs_knnsearch(index=index+\"_\"+vector_name, k=top_k, vector=embedded_query)\n",
"\n",
" return query_result"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:39.379566Z",
"start_time": "2023-02-16T12:30:38.031041Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art (Distance: 0.125)\n",
"2. Western Europe (Distance: 0.133)\n",
"3. Renaissance art (Distance: 0.136)\n",
"4. Pop art (Distance: 0.14)\n",
"5. Northern Europe (Distance: 0.145)\n"
]
}
],
"source": [
"import openai\n",
"import numpy as np\n",
"\n",
"query_result = query_tair(client=client, query=\"modern art in Europe\", vector_name=\"title_vector\")\n",
"for i in range(len(query_result)):\n",
" title = client.tvs_hmget(index+\"_\"+\"content_vector\", query_result[i][0].decode('utf-8'), \"title\")\n",
" print(f\"{i + 1}. {title[0].decode('utf-8')} (Distance: {round(query_result[i][1],3)})\")"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.652676Z",
"start_time": "2023-02-16T12:30:39.382555Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn (Distance: 0.131)\n",
"2. Wars of Scottish Independence (Distance: 0.139)\n",
"3. 1651 (Distance: 0.147)\n",
"4. First War of Scottish Independence (Distance: 0.15)\n",
"5. Robert I of Scotland (Distance: 0.154)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_result = query_tair(client=client, query=\"Famous battles in Scottish history\", vector_name=\"content_vector\")\n",
"for i in range(len(query_result)):\n",
" title = client.tvs_hmget(index+\"_\"+\"content_vector\", query_result[i][0].decode('utf-8'), \"title\")\n",
" print(f\"{i + 1}. {title[0].decode('utf-8')} (Distance: {round(query_result[i][1],3)})\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [conda env:notebook] *",
"language": "python",
"name": "conda-env-notebook-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb
+496
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/typesense/.gitignore
+1
View File
@@ -0,0 +1 @@
typesense-data
examples/vector_databases/typesense/README.md
+31
View File
@@ -0,0 +1,31 @@
# Typesense
Typesense is an open source, in-memory search engine, that you can either [self-host](https://typesense.org/docs/guide/install-typesense.html#option-2-local-machine-self-hosting) or run on [Typesense Cloud](https://cloud.typesense.org/).
## Why Typesense?
Typesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults.
It also lets you combine attribute-based filtering together with vector queries, to fetch the most relevant documents.
### Other features
Besides vector storage and search, Typesense also offers the following features:
- Typo Tolerance: Handles typographical errors elegantly, out-of-the-box.
- Tunable Ranking: Easy to tailor your search results to perfection.
- Sorting: Dynamically sort results based on a particular field at query time (helpful for features like "Sort by Price (asc)").
- Faceting & Filtering: Drill down and refine results.
- Grouping & Distinct: Group similar results together to show more variety.
- Federated Search: Search across multiple collections (indices) in a single HTTP request.
- Scoped API Keys: Generate API keys that only allow access to certain records, for multi-tenant applications.
- Synonyms: Define words as equivalents of each other, so searching for a word will also return results for the synonyms defined.
- Curation & Merchandizing: Boost particular records to a fixed position in the search results, to feature them.
- Raft-based Clustering: Set up a distributed cluster that is highly available.
- Seamless Version Upgrades: As new versions of Typesense come out, upgrading is as simple as swapping out the binary and restarting Typesense.
- No Runtime Dependencies: Typesense is a single binary that you can run locally or in production with a single command.
## How To
- To learn more about how to use Typesense with OpenAI embeddings, see the notebook here for an example: [examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb](/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb)
- To learn more about Typesense's vector search feature, read the docs here: [https://typesense.org/docs/0.24.1/api/vector-search.html](https://typesense.org/docs/0.24.1/api/vector-search.html).
examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb
+879
View File
@@ -0,0 +1,879 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Typesense for Embeddings Search\n",
"\n",
"This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"### What is a Vector Database\n",
"\n",
"A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
"\n",
"### Why use a Vector Database\n",
"\n",
"Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
"\n",
"\n",
"### Demo Flow\n",
"The demo flow is:\n",
"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **Typesense**\n",
" - *Setup*: Set up the Typesense Python client. For more details go [here](https://typesense.org/docs/0.24.0/api/)\n",
" - *Index Data*: We'll create a collection and index it for both __titles__ and __content__.\n",
" - *Search Data*: Run a few example queries with various goals in mind.\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "e2b59250",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Import the required libraries and set the embedding model that we'd like to use."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d8810f9",
"metadata": {},
"outputs": [],
"source": [
"# We'll need to install the Typesense client\n",
"!pip install typesense\n",
"\n",
"#Install wget to pull zip file\n",
"!pip install wget"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5be94df6",
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"from typing import List, Iterator\n",
"import pandas as pd\n",
"import numpy as np\n",
"import os\n",
"import wget\n",
"from ast import literal_eval\n",
"\n",
"# Typesense's client library for Python\n",
"import typesense\n",
"\n",
"# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"\n",
"# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that we've prepared previous to this session."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5dff8b55",
"metadata": {},
"outputs": [],
"source": [
"embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21097972",
"metadata": {},
"outputs": [],
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "70bbd8ba",
"metadata": {},
"outputs": [],
"source": [
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1721e45d",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "960b82af",
"metadata": {},
"outputs": [],
"source": [
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
"\n",
"# Set vector_id to be a string\n",
"article_df['vector_id'] = article_df['vector_id'].apply(str)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a334ab8b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 25000 entries, 0 to 24999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 25000 non-null int64 \n",
" 1 url 25000 non-null object\n",
" 2 title 25000 non-null object\n",
" 3 text 25000 non-null object\n",
" 4 title_vector 25000 non-null object\n",
" 5 content_vector 25000 non-null object\n",
" 6 vector_id 25000 non-null object\n",
"dtypes: int64(1), object(6)\n",
"memory usage: 1.3+ MB\n"
]
}
],
"source": [
"article_df.info(show_counts=True)"
]
},
{
"cell_type": "markdown",
"id": "bb09e0ec",
"metadata": {},
"source": [
"## Typesense\n",
"\n",
"The next vector store we'll look at is [Typesense](https://typesense.org/), which is an open source, in-memory search engine, that you can either self-host or run on [Typesense Cloud](https://cloud.typesense.org).\n",
"\n",
"Typesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. It also lets you combine attribute-based filtering together with vector queries.\n",
"\n",
"For this example, we will set up a local docker-based Typesense server, index our vectors in Typesense and then do some nearest-neighbor search queries. If you use Typesense Cloud, you can skip the docker setup part and just obtain the hostname and API keys from your cluster dashboard."
]
},
{
"cell_type": "markdown",
"id": "bd629f7d",
"metadata": {},
"source": [
"### Setup\n",
"\n",
"To run Typesense locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Typesense documentation [here](https://typesense.org/docs/guide/install-typesense.html#docker-compose), we created an example docker-compose.yml file in this repo saved at [./typesense/docker-compose.yml](./typesense/docker-compose.yml).\n",
"\n",
"After starting Docker, you can start Typesense locally by navigating to the `examples/vector_databases/typesense/` directory and running `docker-compose up -d`.\n",
"\n",
"The default API key is set to `xyz` in the Docker compose file, and the default Typesense port to `8108`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "2bee46b1",
"metadata": {},
"outputs": [],
"source": [
"import typesense\n",
"\n",
"typesense_client = \\\n",
" typesense.Client({\n",
" \"nodes\": [{\n",
" \"host\": \"localhost\", # For Typesense Cloud use xxx.a1.typesense.net\n",
" \"port\": \"8108\", # For Typesense Cloud use 443\n",
" \"protocol\": \"http\" # For Typesense Cloud use https\n",
" }],\n",
" \"api_key\": \"xyz\",\n",
" \"connection_timeout_seconds\": 60\n",
" })"
]
},
{
"cell_type": "markdown",
"id": "11910afb",
"metadata": {},
"source": [
"### Index data\n",
"\n",
"To index vectors in Typesense, we'll first create a Collection (which is a collection of Documents) and turn on vector indexing for a particular field. You can even store multiple vector fields in a single document."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "dd055c80",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'created_at': 1687165065, 'default_sorting_field': '', 'enable_nested_fields': False, 'fields': [{'facet': False, 'index': True, 'infix': False, 'locale': '', 'name': 'content_vector', 'num_dim': 1536, 'optional': False, 'sort': False, 'type': 'float[]'}, {'facet': False, 'index': True, 'infix': False, 'locale': '', 'name': 'title_vector', 'num_dim': 1536, 'optional': False, 'sort': False, 'type': 'float[]'}], 'name': 'wikipedia_articles', 'num_documents': 0, 'symbols_to_index': [], 'token_separators': []}\n",
"Created new collection wikipedia-articles\n"
]
}
],
"source": [
"# Delete existing collections if they already exist\n",
"try:\n",
" typesense_client.collections['wikipedia_articles'].delete()\n",
"except Exception as e:\n",
" pass\n",
"\n",
"# Create a new collection\n",
"\n",
"schema = {\n",
" \"name\": \"wikipedia_articles\",\n",
" \"fields\": [\n",
" {\n",
" \"name\": \"content_vector\",\n",
" \"type\": \"float[]\",\n",
" \"num_dim\": len(article_df['content_vector'][0])\n",
" },\n",
" {\n",
" \"name\": \"title_vector\",\n",
" \"type\": \"float[]\",\n",
" \"num_dim\": len(article_df['title_vector'][0])\n",
" }\n",
" ]\n",
"}\n",
"\n",
"create_response = typesense_client.collections.create(schema)\n",
"print(create_response)\n",
"\n",
"print(\"Created new collection wikipedia-articles\")"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "94bbbb11",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Indexing vectors in Typesense...\n",
"Processed 100 / 25000 \n",
"Processed 200 / 25000 \n",
"Processed 300 / 25000 \n",
"Processed 400 / 25000 \n",
"Processed 500 / 25000 \n",
"Processed 600 / 25000 \n",
"Processed 700 / 25000 \n",
"Processed 800 / 25000 \n",
"Processed 900 / 25000 \n",
"Processed 1000 / 25000 \n",
"Processed 1100 / 25000 \n",
"Processed 1200 / 25000 \n",
"Processed 1300 / 25000 \n",
"Processed 1400 / 25000 \n",
"Processed 1500 / 25000 \n",
"Processed 1600 / 25000 \n",
"Processed 1700 / 25000 \n",
"Processed 1800 / 25000 \n",
"Processed 1900 / 25000 \n",
"Processed 2000 / 25000 \n",
"Processed 2100 / 25000 \n",
"Processed 2200 / 25000 \n",
"Processed 2300 / 25000 \n",
"Processed 2400 / 25000 \n",
"Processed 2500 / 25000 \n",
"Processed 2600 / 25000 \n",
"Processed 2700 / 25000 \n",
"Processed 2800 / 25000 \n",
"Processed 2900 / 25000 \n",
"Processed 3000 / 25000 \n",
"Processed 3100 / 25000 \n",
"Processed 3200 / 25000 \n",
"Processed 3300 / 25000 \n",
"Processed 3400 / 25000 \n",
"Processed 3500 / 25000 \n",
"Processed 3600 / 25000 \n",
"Processed 3700 / 25000 \n",
"Processed 3800 / 25000 \n",
"Processed 3900 / 25000 \n",
"Processed 4000 / 25000 \n",
"Processed 4100 / 25000 \n",
"Processed 4200 / 25000 \n",
"Processed 4300 / 25000 \n",
"Processed 4400 / 25000 \n",
"Processed 4500 / 25000 \n",
"Processed 4600 / 25000 \n",
"Processed 4700 / 25000 \n",
"Processed 4800 / 25000 \n",
"Processed 4900 / 25000 \n",
"Processed 5000 / 25000 \n",
"Processed 5100 / 25000 \n",
"Processed 5200 / 25000 \n",
"Processed 5300 / 25000 \n",
"Processed 5400 / 25000 \n",
"Processed 5500 / 25000 \n",
"Processed 5600 / 25000 \n",
"Processed 5700 / 25000 \n",
"Processed 5800 / 25000 \n",
"Processed 5900 / 25000 \n",
"Processed 6000 / 25000 \n",
"Processed 6100 / 25000 \n",
"Processed 6200 / 25000 \n",
"Processed 6300 / 25000 \n",
"Processed 6400 / 25000 \n",
"Processed 6500 / 25000 \n",
"Processed 6600 / 25000 \n",
"Processed 6700 / 25000 \n",
"Processed 6800 / 25000 \n",
"Processed 6900 / 25000 \n",
"Processed 7000 / 25000 \n",
"Processed 7100 / 25000 \n",
"Processed 7200 / 25000 \n",
"Processed 7300 / 25000 \n",
"Processed 7400 / 25000 \n",
"Processed 7500 / 25000 \n",
"Processed 7600 / 25000 \n",
"Processed 7700 / 25000 \n",
"Processed 7800 / 25000 \n",
"Processed 7900 / 25000 \n",
"Processed 8000 / 25000 \n",
"Processed 8100 / 25000 \n",
"Processed 8200 / 25000 \n",
"Processed 8300 / 25000 \n",
"Processed 8400 / 25000 \n",
"Processed 8500 / 25000 \n",
"Processed 8600 / 25000 \n",
"Processed 8700 / 25000 \n",
"Processed 8800 / 25000 \n",
"Processed 8900 / 25000 \n",
"Processed 9000 / 25000 \n",
"Processed 9100 / 25000 \n",
"Processed 9200 / 25000 \n",
"Processed 9300 / 25000 \n",
"Processed 9400 / 25000 \n",
"Processed 9500 / 25000 \n",
"Processed 9600 / 25000 \n",
"Processed 9700 / 25000 \n",
"Processed 9800 / 25000 \n",
"Processed 9900 / 25000 \n",
"Processed 10000 / 25000 \n",
"Processed 10100 / 25000 \n",
"Processed 10200 / 25000 \n",
"Processed 10300 / 25000 \n",
"Processed 10400 / 25000 \n",
"Processed 10500 / 25000 \n",
"Processed 10600 / 25000 \n",
"Processed 10700 / 25000 \n",
"Processed 10800 / 25000 \n",
"Processed 10900 / 25000 \n",
"Processed 11000 / 25000 \n",
"Processed 11100 / 25000 \n",
"Processed 11200 / 25000 \n",
"Processed 11300 / 25000 \n",
"Processed 11400 / 25000 \n",
"Processed 11500 / 25000 \n",
"Processed 11600 / 25000 \n",
"Processed 11700 / 25000 \n",
"Processed 11800 / 25000 \n",
"Processed 11900 / 25000 \n",
"Processed 12000 / 25000 \n",
"Processed 12100 / 25000 \n",
"Processed 12200 / 25000 \n",
"Processed 12300 / 25000 \n",
"Processed 12400 / 25000 \n",
"Processed 12500 / 25000 \n",
"Processed 12600 / 25000 \n",
"Processed 12700 / 25000 \n",
"Processed 12800 / 25000 \n",
"Processed 12900 / 25000 \n",
"Processed 13000 / 25000 \n",
"Processed 13100 / 25000 \n",
"Processed 13200 / 25000 \n",
"Processed 13300 / 25000 \n",
"Processed 13400 / 25000 \n",
"Processed 13500 / 25000 \n",
"Processed 13600 / 25000 \n",
"Processed 13700 / 25000 \n",
"Processed 13800 / 25000 \n",
"Processed 13900 / 25000 \n",
"Processed 14000 / 25000 \n",
"Processed 14100 / 25000 \n",
"Processed 14200 / 25000 \n",
"Processed 14300 / 25000 \n",
"Processed 14400 / 25000 \n",
"Processed 14500 / 25000 \n",
"Processed 14600 / 25000 \n",
"Processed 14700 / 25000 \n",
"Processed 14800 / 25000 \n",
"Processed 14900 / 25000 \n",
"Processed 15000 / 25000 \n",
"Processed 15100 / 25000 \n",
"Processed 15200 / 25000 \n",
"Processed 15300 / 25000 \n",
"Processed 15400 / 25000 \n",
"Processed 15500 / 25000 \n",
"Processed 15600 / 25000 \n",
"Processed 15700 / 25000 \n",
"Processed 15800 / 25000 \n",
"Processed 15900 / 25000 \n",
"Processed 16000 / 25000 \n",
"Processed 16100 / 25000 \n",
"Processed 16200 / 25000 \n",
"Processed 16300 / 25000 \n",
"Processed 16400 / 25000 \n",
"Processed 16500 / 25000 \n",
"Processed 16600 / 25000 \n",
"Processed 16700 / 25000 \n",
"Processed 16800 / 25000 \n",
"Processed 16900 / 25000 \n",
"Processed 17000 / 25000 \n",
"Processed 17100 / 25000 \n",
"Processed 17200 / 25000 \n",
"Processed 17300 / 25000 \n",
"Processed 17400 / 25000 \n",
"Processed 17500 / 25000 \n",
"Processed 17600 / 25000 \n",
"Processed 17700 / 25000 \n",
"Processed 17800 / 25000 \n",
"Processed 17900 / 25000 \n",
"Processed 18000 / 25000 \n",
"Processed 18100 / 25000 \n",
"Processed 18200 / 25000 \n",
"Processed 18300 / 25000 \n",
"Processed 18400 / 25000 \n",
"Processed 18500 / 25000 \n",
"Processed 18600 / 25000 \n",
"Processed 18700 / 25000 \n",
"Processed 18800 / 25000 \n",
"Processed 18900 / 25000 \n",
"Processed 19000 / 25000 \n",
"Processed 19100 / 25000 \n",
"Processed 19200 / 25000 \n",
"Processed 19300 / 25000 \n",
"Processed 19400 / 25000 \n",
"Processed 19500 / 25000 \n",
"Processed 19600 / 25000 \n",
"Processed 19700 / 25000 \n",
"Processed 19800 / 25000 \n",
"Processed 19900 / 25000 \n",
"Processed 20000 / 25000 \n",
"Processed 20100 / 25000 \n",
"Processed 20200 / 25000 \n",
"Processed 20300 / 25000 \n",
"Processed 20400 / 25000 \n",
"Processed 20500 / 25000 \n",
"Processed 20600 / 25000 \n",
"Processed 20700 / 25000 \n",
"Processed 20800 / 25000 \n",
"Processed 20900 / 25000 \n",
"Processed 21000 / 25000 \n",
"Processed 21100 / 25000 \n",
"Processed 21200 / 25000 \n",
"Processed 21300 / 25000 \n",
"Processed 21400 / 25000 \n",
"Processed 21500 / 25000 \n",
"Processed 21600 / 25000 \n",
"Processed 21700 / 25000 \n",
"Processed 21800 / 25000 \n",
"Processed 21900 / 25000 \n",
"Processed 22000 / 25000 \n",
"Processed 22100 / 25000 \n",
"Processed 22200 / 25000 \n",
"Processed 22300 / 25000 \n",
"Processed 22400 / 25000 \n",
"Processed 22500 / 25000 \n",
"Processed 22600 / 25000 \n",
"Processed 22700 / 25000 \n",
"Processed 22800 / 25000 \n",
"Processed 22900 / 25000 \n",
"Processed 23000 / 25000 \n",
"Processed 23100 / 25000 \n",
"Processed 23200 / 25000 \n",
"Processed 23300 / 25000 \n",
"Processed 23400 / 25000 \n",
"Processed 23500 / 25000 \n",
"Processed 23600 / 25000 \n",
"Processed 23700 / 25000 \n",
"Processed 23800 / 25000 \n",
"Processed 23900 / 25000 \n",
"Processed 24000 / 25000 \n",
"Processed 24100 / 25000 \n",
"Processed 24200 / 25000 \n",
"Processed 24300 / 25000 \n",
"Processed 24400 / 25000 \n",
"Processed 24500 / 25000 \n",
"Processed 24600 / 25000 \n",
"Processed 24700 / 25000 \n",
"Processed 24800 / 25000 \n",
"Processed 24900 / 25000 \n",
"Processed 25000 / 25000 \n",
"Imported (25000) articles.\n"
]
}
],
"source": [
"# Upsert the vector data into the collection we just created\n",
"#\n",
"# Note: This can take a few minutes, especially if your on an M1 and running docker in an emulated mode\n",
"\n",
"print(\"Indexing vectors in Typesense...\")\n",
"\n",
"document_counter = 0\n",
"documents_batch = []\n",
"\n",
"for k,v in article_df.iterrows():\n",
" # Create a document with the vector data\n",
"\n",
" # Notice how you can add any fields that you haven't added to the schema to the document.\n",
" # These will be stored on disk and returned when the document is a hit.\n",
" # This is useful to store attributes required for display purposes.\n",
"\n",
" document = {\n",
" \"title_vector\": v[\"title_vector\"],\n",
" \"content_vector\": v[\"content_vector\"],\n",
" \"title\": v[\"title\"],\n",
" \"content\": v[\"text\"],\n",
" }\n",
" documents_batch.append(document)\n",
" document_counter = document_counter + 1\n",
"\n",
" # Upsert a batch of 100 documents\n",
" if document_counter % 100 == 0 or document_counter == len(article_df):\n",
" response = typesense_client.collections['wikipedia_articles'].documents.import_(documents_batch)\n",
" # print(response)\n",
"\n",
" documents_batch = []\n",
" print(f\"Processed {document_counter} / {len(article_df)} \")\n",
"\n",
"print(f\"Imported ({len(article_df)}) articles.\")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f774ecb2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Collection has 25000 documents\n"
]
}
],
"source": [
"# Check the number of documents imported\n",
"\n",
"collection = typesense_client.collections['wikipedia_articles'].retrieve()\n",
"print(f'Collection has {collection[\"num_documents\"]} documents')"
]
},
{
"cell_type": "markdown",
"id": "fbc6f5c5",
"metadata": {},
"source": [
"### Search Data\n",
"\n",
"Now that we've imported the vectors into Typesense, we can do a nearest neighbor search on the `title_vector` or `content_vector` field."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "d9a3f0dc",
"metadata": {},
"outputs": [],
"source": [
"def query_typesense(query, field='title', top_k=20):\n",
"\n",
" # Creates embedding vector from user query\n",
" openai.api_key = os.getenv(\"OPENAI_API_KEY\", \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\n",
" embedded_query = openai.Embedding.create(\n",
" input=query,\n",
" model=EMBEDDING_MODEL,\n",
" )['data'][0]['embedding']\n",
"\n",
" typesense_results = typesense_client.multi_search.perform({\n",
" \"searches\": [{\n",
" \"q\": \"*\",\n",
" \"collection\": \"wikipedia_articles\",\n",
" \"vector_query\": f\"{field}_vector:([{','.join(str(v) for v in embedded_query)}], k:{top_k})\"\n",
" }]\n",
" }, {})\n",
"\n",
" return typesense_results"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "24183c36",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art (Distance: 0.12482291460037231)\n",
"2. Western Europe (Distance: 0.13255876302719116)\n",
"3. Renaissance art (Distance: 0.13584274053573608)\n",
"4. Pop art (Distance: 0.1396539807319641)\n",
"5. Northern Europe (Distance: 0.14534103870391846)\n",
"6. Hellenistic art (Distance: 0.1472070813179016)\n",
"7. Modernist literature (Distance: 0.15296930074691772)\n",
"8. Art film (Distance: 0.1567266583442688)\n",
"9. Central Europe (Distance: 0.15741699934005737)\n",
"10. European (Distance: 0.1585891842842102)\n"
]
}
],
"source": [
"query_results = query_typesense('modern art in Europe', 'title')\n",
"\n",
"for i, hit in enumerate(query_results['results'][0]['hits']):\n",
" document = hit[\"document\"]\n",
" vector_distance = hit[\"vector_distance\"]\n",
" print(f'{i + 1}. {document[\"title\"]} (Distance: {vector_distance})')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "a64e3c80",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn (Distance: 0.1306111216545105)\n",
"2. Wars of Scottish Independence (Distance: 0.1384994387626648)\n",
"3. 1651 (Distance: 0.14744246006011963)\n",
"4. First War of Scottish Independence (Distance: 0.15033596754074097)\n",
"5. Robert I of Scotland (Distance: 0.15376019477844238)\n",
"6. 841 (Distance: 0.15609073638916016)\n",
"7. 1716 (Distance: 0.15615153312683105)\n",
"8. 1314 (Distance: 0.16280347108840942)\n",
"9. 1263 (Distance: 0.16361045837402344)\n",
"10. William Wallace (Distance: 0.16464537382125854)\n"
]
}
],
"source": [
"query_results = query_typesense('Famous battles in Scottish history', 'content')\n",
"\n",
"for i, hit in enumerate(query_results['results'][0]['hits']):\n",
" document = hit[\"document\"]\n",
" vector_distance = hit[\"vector_distance\"]\n",
" print(f'{i + 1}. {document[\"title\"]} (Distance: {vector_distance})')"
]
},
{
"cell_type": "markdown",
"id": "55afccbf",
"metadata": {},
"source": [
"Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0119d87a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "vector_db_split",
"language": "python",
"name": "vector_db_split"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
},
"vscode": {
"interpreter": {
"hash": "fd16a328ca3d68029457069b79cb0b38eb39a0f5ccc4fe4473d3047707df8207"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/typesense/docker-compose.yml
+10
View File
@@ -0,0 +1,10 @@
version: '3.4'
services:
typesense:
image: typesense/typesense:0.24.0
restart: on-failure
ports:
- "8108:8108"
volumes:
- ./typesense-data:/data
command: '--data-dir /data --api-key=xyz --enable-cors'
examples/vector_databases/weaviate/README.md
+20
View File
@@ -0,0 +1,20 @@
# Weaviate <> OpenAI
[​Weaviate](https://weaviate.io) is an open-source vector search engine ([docs](https://weaviate.io/developers/weaviate) - [Github](https://github.com/weaviate/weaviate)) that can store and search through OpenAI embeddings _and_ data objects. The database allows you to do similarity search, hybrid search (the combining of multiple search techniques, such as keyword-based and vector search), and generative search (like Q&A). Weaviate also supports a wide variety of OpenAI-based modules (e.g., [`text2vec-openai`](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), [`qna-openai`](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai)), allowing you to vectorize and query data fast and efficiently.
You can run Weaviate (including the OpenAI modules if desired) in three ways:
1. Open source inside a Docker-container ([example](./docker-compose.yml))
2. Using the Weaviate Cloud Service ([get started](https://weaviate.io/developers/weaviate/quickstart/installation#weaviate-cloud-service))
3. In a Kubernetes cluster ([learn more](https://weaviate.io/developers/weaviate/installation/kubernetes))
### Examples
This folder contains a variety of Weaviate and OpenAI examples.
| Name | Description | language | Google Colab |
| --- | --- | --- | --- |
| [Getting Started with Weaviate and OpenAI](./getting-started-with-weaviate-and-openai.ipynb) | A simple getting started for *semantic vector search* using the OpenAI vectorization module in Weaviate (`text2vec-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1RxpDE_ruCnoBB3TfwAZqdjYgHJhtdwhK) |
| [Hybrid Search with Weaviate and OpenAI](./hybrid-search-with-weaviate-and-openai.ipynb) | A simple getting started for *hybrid search* using the OpenAI vectorization module in Weaviate (`text2vec-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1E75BALWoKrOjvUhaznJKQO0A-B1QUPZ4) |
| [Question Answering with Weaviate and OpenAI](./question-answering-with-weaviate-and-openai.ipynb) | A simple getting started for *question answering (Q&A)* using the OpenAI Q&A module in Weaviate (`qna-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1pUerUZrJaknEboDxDxsuf3giCK0MJJgm) |
| [Docker-compose example](./docker-compose.yml) | A Docker-compose file with all OpenAI modules enabled | Docker |
examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb
+1173
View File
File diff suppressed because it is too large Load Diff
examples/vector_databases/weaviate/docker-compose.yml
+33
View File
@@ -0,0 +1,33 @@
#################
#
# This is an example Docker file for Weaviate with all OpenAI modules enabled
# You can, but don't have to set `OPENAI_APIKEY` because it can also be set at runtime
#
# Find the latest version here: https://weaviate.io/developers/weaviate/installation/docker-compose
#
#################
---
version: '3.4'
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: semitechnologies/weaviate:1.17.2
ports:
- 8080:8080
restart: on-failure:0
environment:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
ENABLE_MODULES: 'text2vec-openai,qna-openai'
CLUSTER_HOSTNAME: 'openai-weaviate-cluster'
# The following parameter (`OPENAI_APIKEY`) is optional, as you can also provide it at insert/query time
# OPENAI_APIKEY: sk-foobar
...
examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb
+275
View File
@@ -0,0 +1,275 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Weaviate with Generative OpenAI module for Generative Search\n",
"\n",
"This notebook is prepared for a scenario where:\n",
"* Your data is already in Weaviate\n",
"* You want to use Weaviate with the Generative OpenAI module ([generative-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai)).\n",
"\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f1a618c5",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"This cookbook only coveres Generative Search examples, however, it doesn't cover the configuration and data imports.\n",
"\n",
"In order to make the most of this cookbook, please complete the [Getting Started cookbook](./getting-started-with-weaviate-and-openai.ipynb) first, where you will learn the essentials of working with Weaviate and import the demo data.\n",
"\n",
"Checklist:\n",
"* completed [Getting Started cookbook](./getting-started-with-weaviate-and-openai.ipynb),\n",
"* crated a `Weaviate` instance,\n",
"* imported data into your `Weaviate` instance,\n",
"* you have an [OpenAI API key](https://beta.openai.com/account/api-keys)"
]
},
{
"cell_type": "markdown",
"id": "36fe86f4",
"metadata": {},
"source": [
"===========================================================\n",
"## Prepare your OpenAI API key\n",
"\n",
"The `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43395339",
"metadata": {},
"outputs": [],
"source": [
"# Export OpenAI API Key\n",
"!export OPENAI_API_KEY=\"your key\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88be138c",
"metadata": {},
"outputs": [],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = 'your-key-goes-here'\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print (\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print (\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"id": "91df4d5b",
"metadata": {},
"source": [
"## Connect to your Weaviate instance\n",
"\n",
"In this section, we will:\n",
"\n",
"1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n",
"2. connect to your Weaviate with your `OpenAI API Key`\n",
"3. and test the client connection\n",
"\n",
"### The client \n",
"\n",
"After this step, the `client` object will be used to perform all Weaviate-related operations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc662c1b",
"metadata": {},
"outputs": [],
"source": [
"import weaviate\n",
"from datasets import load_dataset\n",
"import os\n",
"\n",
"# Connect to your Weaviate instance\n",
"client = weaviate.Client(\n",
" url=\"https://your-wcs-instance-name.weaviate.network/\",\n",
" # url=\"http://localhost:8080/\",\n",
" auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n",
" additional_headers={\n",
" \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n",
" }\n",
")\n",
"\n",
"# Check if your instance is live and ready\n",
"# This should return `True`\n",
"client.is_ready()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ceb14da9",
"metadata": {},
"source": [
"## Generative Search\n",
"Weaviate offers a [Generative Search OpenAI](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai) module, which generates responses based on the data stored in your Weaviate instance.\n",
"\n",
"The way you construct a generative search query is very similar to a standard semantic search query in Weaviate. \n",
"\n",
"For example:\n",
"* search in \"Articles\", \n",
"* return \"title\", \"content\", \"url\"\n",
"* look for objects related to \"football clubs\"\n",
"* limit results to 5 objects\n",
"\n",
"```\n",
" result = (\n",
" client.query\n",
" .get(\"Articles\", [\"title\", \"content\", \"url\"])\n",
" .with_near_text(\"concepts\": \"football clubs\")\n",
" .with_limit(5)\n",
" # generative query will go here\n",
" .do()\n",
" )\n",
"```\n",
"\n",
"Now, you can add `with_generate()` function to apply generative transformation. `with_generate` takes either:\n",
"- `single_prompt` - to generate a response for each returned object,\n",
"- `grouped_task` – to generate a single response from all returned objects.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "51559251",
"metadata": {},
"outputs": [],
"source": [
"def generative_search_per_item(query, collection_name):\n",
" prompt = \"Summarize in a short tweet the following content: {content}\"\n",
"\n",
" result = (\n",
" client.query\n",
" .get(collection_name, [\"title\", \"content\", \"url\"])\n",
" .with_near_text({ \"concepts\": [query], \"distance\": 0.7 })\n",
" .with_limit(5)\n",
" .with_generate(single_prompt=prompt)\n",
" .do()\n",
" )\n",
" \n",
" # Check for errors\n",
" if (\"errors\" in result):\n",
" print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.\")\n",
" raise Exception(result[\"errors\"][0]['message'])\n",
" \n",
" return result[\"data\"][\"Get\"][collection_name]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a4604726",
"metadata": {},
"outputs": [],
"source": [
"query_result = generative_search_per_item(\"football clubs\", \"Article\")\n",
"\n",
"for i, article in enumerate(query_result):\n",
" print(f\"{i+1}. { article['title']}\")\n",
" print(article['_additional']['generate']['singleResult']) # print generated response\n",
" print(\"-----------------------\")"
]
},
{
"cell_type": "code",
"execution_count": 79,
"id": "a45ea160",
"metadata": {},
"outputs": [],
"source": [
"def generative_search_group(query, collection_name):\n",
" generateTask = \"Explain what these have in common\"\n",
"\n",
" result = (\n",
" client.query\n",
" .get(collection_name, [\"title\", \"content\", \"url\"])\n",
" .with_near_text({ \"concepts\": [query], \"distance\": 0.7 })\n",
" .with_generate(grouped_task=generateTask)\n",
" .with_limit(5)\n",
" .do()\n",
" )\n",
" \n",
" # Check for errors\n",
" if (\"errors\" in result):\n",
" print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.\")\n",
" raise Exception(result[\"errors\"][0]['message'])\n",
" \n",
" return result[\"data\"][\"Get\"][collection_name]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11e0dad2",
"metadata": {},
"outputs": [],
"source": [
"query_result = generative_search_group(\"football clubs\", \"Article\")\n",
"\n",
"print (query_result[0]['_additional']['generate']['groupedResult'])"
]
},
{
"cell_type": "markdown",
"id": "2007be48",
"metadata": {},
"source": [
"Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"vscode": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb
+573
View File
@@ -0,0 +1,573 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Weaviate with OpenAI vectorize module for Embeddings Search\n",
"\n",
"This notebook is prepared for a scenario where:\n",
"* Your data is not vectorized\n",
"* You want to run Vector Search on your data\n",
"* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.\n",
"\n",
"This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run semantic search.\n",
"\n",
"This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"## What is Weaviate\n",
"\n",
"Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n",
"\n",
"Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n",
"\n",
"Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n",
"\n",
"### Deployment options\n",
"\n",
"Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n",
"* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.\n",
"* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n",
"* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service.\n",
"\n",
"### Programming languages\n",
"\n",
"Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n",
"* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n",
"* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n",
"* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n",
"* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n",
"\n",
"Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests."
]
},
{
"cell_type": "markdown",
"id": "45956173",
"metadata": {},
"source": [
"## Demo Flow\n",
"The demo flow is:\n",
"- **Prerequisites Setup**: Create a Weaviate instance and install the required libraries\n",
"- **Connect**: Connect to your Weaviate instance \n",
"- **Schema Configuration**: Configure the schema of your data\n",
" - *Note*: Here we can define which OpenAI Embedding Model to use\n",
" - *Note*: Here we can configure which properties to index\n",
"- **Import data**: Load a demo dataset and import it into Weaviate\n",
" - *Note*: The import process will automatically index your data - based on the configuration in the schema\n",
" - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n",
"- **Run Queries**: Query \n",
" - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "2a4a145e",
"metadata": {},
"source": [
"## OpenAI Module in Weaviate\n",
"All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.\n",
"\n",
"This module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.\n",
"\n",
"### No need to manually vectorize data\n",
"This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.\n",
"\n",
"All you need to do is:\n",
"1. provide your OpenAI API Key – when you connected to the Weaviate Client\n",
"2. define which OpenAI vectorizer to use in your Schema"
]
},
{
"cell_type": "markdown",
"id": "f1a618c5",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Before we start this project, we need setup the following:\n",
"\n",
"* create a `Weaviate` instance\n",
"* install libraries\n",
" * `weaviate-client`\n",
" * `datasets`\n",
" * `apache-beam`\n",
"* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n",
"\n",
"===========================================================\n",
"### Create a Weaviate instance\n",
"\n",
"To create a Weaviate instance we have 2 options:\n",
"\n",
"1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n",
"2. Install and run Weaviate locally with Docker.\n",
"\n",
"#### Option 1 – WCS Installation Steps\n",
"\n",
"Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n",
"1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n",
"2. create a `Weaviate Cluster` with the following settings:\n",
" * Sandbox: `Sandbox Free`\n",
" * Weaviate Version: Use default (latest)\n",
" * OIDC Authentication: `Disabled`\n",
"3. your instance should be ready in a minute or two\n",
"4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` \n",
"\n",
"#### Option 2 – local Weaviate instance with Docker\n",
"\n",
"Install and run Weaviate locally with Docker.\n",
"1. Download the [./docker-compose.yml](./docker-compose.yml) file\n",
"2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n",
"3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n",
"\n",
"Note. To shut down your docker instance you can call: `docker-compose down`\n",
"\n",
"##### Learn more\n",
"To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose)."
]
},
{
"cell_type": "markdown",
"id": "b9babafe",
"metadata": {},
"source": [
"=========================================================== \n",
"## Install required libraries\n",
"\n",
"Before running this project make sure to have the following libraries:\n",
"\n",
"### Weaviate Python client\n",
"\n",
"The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n",
"\n",
"### datasets & apache-beam\n",
"\n",
"To load sample data, you need the `datasets` library and its dependency `apache-beam`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2b04113f",
"metadata": {},
"outputs": [],
"source": [
"# Install the Weaviate client for Python\n",
"!pip install weaviate-client>=3.11.0\n",
"\n",
"# Install datasets and apache-beam to load the sample datasets\n",
"!pip install datasets apache-beam"
]
},
{
"cell_type": "markdown",
"id": "36fe86f4",
"metadata": {},
"source": [
"===========================================================\n",
"## Prepare your OpenAI API key\n",
"\n",
"The `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43395339",
"metadata": {},
"outputs": [],
"source": [
"# Export OpenAI API Key\n",
"!export OPENAI_API_KEY=\"your key\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88be138c",
"metadata": {},
"outputs": [],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = 'your-key-goes-here'\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print (\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print (\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"id": "91df4d5b",
"metadata": {},
"source": [
"## Connect to your Weaviate instance\n",
"\n",
"In this section, we will:\n",
"\n",
"1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n",
"2. connect to your Weaviate with your `OpenAI API Key`\n",
"3. and test the client connection\n",
"\n",
"### The client \n",
"\n",
"After this step, the `client` object will be used to perform all Weaviate-related operations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc662c1b",
"metadata": {},
"outputs": [],
"source": [
"import weaviate\n",
"from datasets import load_dataset\n",
"import os\n",
"\n",
"# Connect to your Weaviate instance\n",
"client = weaviate.Client(\n",
" url=\"https://your-wcs-instance-name.weaviate.network/\",\n",
" # url=\"http://localhost:8080/\",\n",
" auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n",
" additional_headers={\n",
" \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n",
" }\n",
")\n",
"\n",
"# Check if your instance is live and ready\n",
"# This should return `True`\n",
"client.is_ready()"
]
},
{
"cell_type": "markdown",
"id": "7d3dac3c",
"metadata": {},
"source": [
"# Schema\n",
"\n",
"In this section, we will:\n",
"1. configure the data schema for your data\n",
"2. select OpenAI module\n",
"\n",
"> This is the second and final step, which requires OpenAI specific configuration.\n",
"> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n",
"\n",
"\n",
"## What is a schema\n",
"\n",
"In Weaviate you create __schemas__ to capture each of the entities you will be searching.\n",
"\n",
"A schema is how you tell Weaviate:\n",
"* what embedding model should be used to vectorize the data\n",
"* what your data is made of (property names and types)\n",
"* which properties should be vectorized and indexed\n",
"\n",
"In this cookbook we will use a dataset for `Articles`, which contains:\n",
"* `title`\n",
"* `content`\n",
"* `url`\n",
"\n",
"We want to vectorize `title` and `content`, but not the `url`.\n",
"\n",
"To vectorize and query the data, we will use `text-embedding-3-small`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f894b911",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Clear up the schema, so that we can recreate it\n",
"client.schema.delete_all()\n",
"client.schema.get()\n",
"\n",
"# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\n",
"article_schema = {\n",
" \"class\": \"Article\",\n",
" \"description\": \"A collection of articles\",\n",
" \"vectorizer\": \"text2vec-openai\",\n",
" \"moduleConfig\": {\n",
" \"text2vec-openai\": {\n",
" \"model\": \"ada\",\n",
" \"modelVersion\": \"002\",\n",
" \"type\": \"text\"\n",
" }\n",
" },\n",
" \"properties\": [{\n",
" \"name\": \"title\",\n",
" \"description\": \"Title of the article\",\n",
" \"dataType\": [\"string\"]\n",
" },\n",
" {\n",
" \"name\": \"content\",\n",
" \"description\": \"Contents of the article\",\n",
" \"dataType\": [\"text\"]\n",
" },\n",
" {\n",
" \"name\": \"url\",\n",
" \"description\": \"URL to the article\",\n",
" \"dataType\": [\"string\"],\n",
" \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n",
" }]\n",
"}\n",
"\n",
"# add the Article schema\n",
"client.schema.create_class(article_schema)\n",
"\n",
"# get the schema to make sure it worked\n",
"client.schema.get()"
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Import data\n",
"\n",
"In this section we will:\n",
"1. load the Simple Wikipedia dataset\n",
"2. configure Weaviate Batch import (to make the import more efficient)\n",
"3. import the data into Weaviate\n",
"\n",
"> Note: <br/>\n",
"> Like mentioned before. We don't need to manually vectorize the data.<br/>\n",
"> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc3efadd",
"metadata": {},
"outputs": [],
"source": [
"### STEP 1 - load the dataset\n",
"\n",
"from datasets import load_dataset\n",
"from typing import List, Iterator\n",
"\n",
"# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\n",
"dataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n",
"\n",
"# For testing, limited to 2.5k articles for demo purposes\n",
"dataset = dataset[:2_500]\n",
"\n",
"# Limited to 25k articles for larger demo purposes\n",
"# dataset = dataset[:25_000]\n",
"\n",
"# for free OpenAI acounts, you can use 50 objects\n",
"# dataset = dataset[:50]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5044da96",
"metadata": {},
"outputs": [],
"source": [
"### Step 2 - configure Weaviate Batch, with\n",
"# - starting batch size of 100\n",
"# - dynamically increase/decrease based on performance\n",
"# - add timeout retries if something goes wrong\n",
"\n",
"client.batch.configure(\n",
" batch_size=10, \n",
" dynamic=True,\n",
" timeout_retries=3,\n",
"# callback=None,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15db8380",
"metadata": {},
"outputs": [],
"source": [
"### Step 3 - import data\n",
"\n",
"print(\"Importing Articles\")\n",
"\n",
"counter=0\n",
"\n",
"with client.batch as batch:\n",
" for article in dataset:\n",
" if (counter %10 == 0):\n",
" print(f\"Import {counter} / {len(dataset)} \")\n",
"\n",
" properties = {\n",
" \"title\": article[\"title\"],\n",
" \"content\": article[\"text\"],\n",
" \"url\": article[\"url\"]\n",
" }\n",
" \n",
" batch.add_data_object(properties, \"Article\")\n",
" counter = counter+1\n",
"\n",
"print(\"Importing Articles complete\") "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3658693c",
"metadata": {},
"outputs": [],
"source": [
"# Test that all data has loaded – get object count\n",
"result = (\n",
" client.query.aggregate(\"Article\")\n",
" .with_fields(\"meta { count }\")\n",
" .do()\n",
")\n",
"print(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0d791186",
"metadata": {},
"outputs": [],
"source": [
"# Test one article has worked by checking one object\n",
"test_article = (\n",
" client.query\n",
" .get(\"Article\", [\"title\", \"url\", \"content\"])\n",
" .with_limit(1)\n",
" .do()\n",
")[\"data\"][\"Get\"][\"Article\"][0]\n",
"\n",
"print(test_article['title'])\n",
"print(test_article['url'])\n",
"print(test_article['content'])"
]
},
{
"cell_type": "markdown",
"id": "46050ca9",
"metadata": {},
"source": [
"### Search Data\n",
"\n",
"As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b044aa93",
"metadata": {},
"outputs": [],
"source": [
"def query_weaviate(query, collection_name):\n",
" \n",
" nearText = {\n",
" \"concepts\": [query],\n",
" \"distance\": 0.7,\n",
" }\n",
"\n",
" properties = [\n",
" \"title\", \"content\", \"url\",\n",
" \"_additional {certainty distance}\"\n",
" ]\n",
"\n",
" result = (\n",
" client.query\n",
" .get(collection_name, properties)\n",
" .with_near_text(nearText)\n",
" .with_limit(10)\n",
" .do()\n",
" )\n",
" \n",
" # Check for errors\n",
" if (\"errors\" in result):\n",
" print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.\")\n",
" raise Exception(result[\"errors\"][0]['message'])\n",
" \n",
" return result[\"data\"][\"Get\"][collection_name]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e2025f6",
"metadata": {},
"outputs": [],
"source": [
"query_result = query_weaviate(\"modern art in Europe\", \"Article\")\n",
"\n",
"for i, article in enumerate(query_result):\n",
" print(f\"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "93c4a696",
"metadata": {},
"outputs": [],
"source": [
"query_result = query_weaviate(\"Famous battles in Scottish history\", \"Article\")\n",
"\n",
"for i, article in enumerate(query_result):\n",
" print(f\"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })\")"
]
},
{
"cell_type": "markdown",
"id": "2007be48",
"metadata": {},
"source": [
"Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"vscode": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb
+575
View File
@@ -0,0 +1,575 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Weaviate with OpenAI vectorize module for Hybrid Search\n",
"\n",
"This notebook is prepared for a scenario where:\n",
"* Your data is not vectorized\n",
"* You want to run Hybrid Search ([learn more](https://weaviate.io/blog/hybrid-search-explained)) on your data\n",
"* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.\n",
"\n",
"This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run hybrid search (mixing of vector and BM25 search).\n",
"\n",
"This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"## What is Weaviate\n",
"\n",
"Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n",
"\n",
"Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n",
"\n",
"Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n",
"\n",
"### Deployment options\n",
"\n",
"Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n",
"* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.\n",
"* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n",
"* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service \n",
"\n",
"### Programming languages\n",
"\n",
"Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n",
"* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n",
"* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n",
"* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n",
"* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n",
"\n",
"Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests."
]
},
{
"cell_type": "markdown",
"id": "45956173",
"metadata": {},
"source": [
"## Demo Flow\n",
"The demo flow is:\n",
"- **Prerequisites Setup**: Create a Weaviate instance and install required libraries\n",
"- **Connect**: Connect to your Weaviate instance \n",
"- **Schema Configuration**: Configure the schema of your data\n",
" - *Note*: Here we can define which OpenAI Embedding Model to use\n",
" - *Note*: Here we can configure which properties to index\n",
"- **Import data**: Load a demo dataset and import it into Weaviate\n",
" - *Note*: The import process will automatically index your data - based on the configuration in the schema\n",
" - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n",
"- **Run Queries**: Query \n",
" - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "2a4a145e",
"metadata": {},
"source": [
"## OpenAI Module in Weaviate\n",
"All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.\n",
"\n",
"This module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.\n",
"\n",
"### No need to manually vectorize data\n",
"This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.\n",
"\n",
"All you need to do is:\n",
"1. provide your OpenAI API Key – when you connected to the Weaviate Client\n",
"2. define which OpenAI vectorizer to use in your Schema"
]
},
{
"cell_type": "markdown",
"id": "f1a618c5",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Before we start this project, we need setup the following:\n",
"\n",
"* create a `Weaviate` instance\n",
"* install libraries\n",
" * `weaviate-client`\n",
" * `datasets`\n",
" * `apache-beam`\n",
"* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n",
"\n",
"===========================================================\n",
"### Create a Weaviate instance\n",
"\n",
"To create a Weaviate instance we have 2 options:\n",
"\n",
"1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n",
"2. Install and run Weaviate locally with Docker.\n",
"\n",
"#### Option 1 – WCS Installation Steps\n",
"\n",
"Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n",
"1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n",
"2. create a `Weaviate Cluster` with the following settings:\n",
" * Sandbox: `Sandbox Free`\n",
" * Weaviate Version: Use default (latest)\n",
" * OIDC Authentication: `Disabled`\n",
"3. your instance should be ready in a minute or two\n",
"4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` \n",
"\n",
"#### Option 2 – local Weaviate instance with Docker\n",
"\n",
"Install and run Weaviate locally with Docker.\n",
"1. Download the [./docker-compose.yml](./docker-compose.yml) file\n",
"2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n",
"3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n",
"\n",
"Note. To shut down your docker instance you can call: `docker-compose down`\n",
"\n",
"##### Learn more\n",
"To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose)."
]
},
{
"cell_type": "markdown",
"id": "b9babafe",
"metadata": {},
"source": [
"=========================================================== \n",
"## Install required libraries\n",
"\n",
"Before running this project make sure to have the following libraries:\n",
"\n",
"### Weaviate Python client\n",
"\n",
"The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n",
"\n",
"### datasets & apache-beam\n",
"\n",
"To load sample data, you need the `datasets` library and its' dependency `apache-beam`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2b04113f",
"metadata": {},
"outputs": [],
"source": [
"# Install the Weaviate client for Python\n",
"!pip install weaviate-client>3.11.0\n",
"\n",
"# Install datasets and apache-beam to load the sample datasets\n",
"!pip install datasets apache-beam"
]
},
{
"cell_type": "markdown",
"id": "36fe86f4",
"metadata": {},
"source": [
"===========================================================\n",
"## Prepare your OpenAI API key\n",
"\n",
"The `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "09fefff0",
"metadata": {},
"outputs": [],
"source": [
"# Export OpenAI API Key\n",
"!export OPENAI_API_KEY=\"your key\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88be138c",
"metadata": {},
"outputs": [],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print (\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print (\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"id": "91df4d5b",
"metadata": {},
"source": [
"## Connect to your Weaviate instance\n",
"\n",
"In this section, we will:\n",
"\n",
"1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n",
"2. connect to your Weaviate your `OpenAI API Key`\n",
"3. and test the client connection\n",
"\n",
"### The client \n",
"\n",
"After this step, the `client` object will be used to perform all Weaviate-related operations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc662c1b",
"metadata": {},
"outputs": [],
"source": [
"import weaviate\n",
"from datasets import load_dataset\n",
"import os\n",
"\n",
"# Connect to your Weaviate instance\n",
"client = weaviate.Client(\n",
" url=\"https://your-wcs-instance-name.weaviate.network/\",\n",
"# url=\"http://localhost:8080/\",\n",
" auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n",
" additional_headers={\n",
" \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n",
" }\n",
")\n",
"\n",
"# Check if your instance is live and ready\n",
"# This should return `True`\n",
"client.is_ready()"
]
},
{
"cell_type": "markdown",
"id": "7d3dac3c",
"metadata": {},
"source": [
"# Schema\n",
"\n",
"In this section, we will:\n",
"1. configure the data schema for your data\n",
"2. select OpenAI module\n",
"\n",
"> This is the second and final step, which requires OpenAI specific configuration.\n",
"> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n",
"\n",
"\n",
"## What is a schema\n",
"\n",
"In Weaviate you create __schemas__ to capture each of the entities you will be searching.\n",
"\n",
"A schema is how you tell Weaviate:\n",
"* what embedding model should be used to vectorize the data\n",
"* what your data is made of (property names and types)\n",
"* which properties should be vectorized and indexed\n",
"\n",
"In this cookbook we will use a dataset for `Articles`, which contains:\n",
"* `title`\n",
"* `content`\n",
"* `url`\n",
"\n",
"We want to vectorize `title` and `content`, but not the `url`.\n",
"\n",
"To vectorize and query the data, we will use `text-embedding-3-small`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f894b911",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Clear up the schema, so that we can recreate it\n",
"client.schema.delete_all()\n",
"client.schema.get()\n",
"\n",
"# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\n",
"article_schema = {\n",
" \"class\": \"Article\",\n",
" \"description\": \"A collection of articles\",\n",
" \"vectorizer\": \"text2vec-openai\",\n",
" \"moduleConfig\": {\n",
" \"text2vec-openai\": {\n",
" \"model\": \"ada\",\n",
" \"modelVersion\": \"002\",\n",
" \"type\": \"text\"\n",
" }\n",
" },\n",
" \"properties\": [{\n",
" \"name\": \"title\",\n",
" \"description\": \"Title of the article\",\n",
" \"dataType\": [\"string\"]\n",
" },\n",
" {\n",
" \"name\": \"content\",\n",
" \"description\": \"Contents of the article\",\n",
" \"dataType\": [\"text\"]\n",
" },\n",
" {\n",
" \"name\": \"url\",\n",
" \"description\": \"URL to the article\",\n",
" \"dataType\": [\"string\"],\n",
" \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n",
" }]\n",
"}\n",
"\n",
"# add the Article schema\n",
"client.schema.create_class(article_schema)\n",
"\n",
"# get the schema to make sure it worked\n",
"client.schema.get()"
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Import data\n",
"\n",
"In this section we will:\n",
"1. load the Simple Wikipedia dataset\n",
"2. configure Weaviate Batch import (to make the import more efficient)\n",
"3. import the data into Weaviate\n",
"\n",
"> Note: <br/>\n",
"> Like mentioned before. We don't need to manually vectorize the data.<br/>\n",
"> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc3efadd",
"metadata": {},
"outputs": [],
"source": [
"### STEP 1 - load the dataset\n",
"\n",
"from datasets import load_dataset\n",
"from typing import List, Iterator\n",
"\n",
"# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\n",
"dataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n",
"\n",
"# For testing, limited to 2.5k articles for demo purposes\n",
"dataset = dataset[:2_500]\n",
"\n",
"# Limited to 25k articles for larger demo purposes\n",
"# dataset = dataset[:25_000]\n",
"\n",
"# for free OpenAI acounts, you can use 50 objects\n",
"# dataset = dataset[:50]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5044da96",
"metadata": {},
"outputs": [],
"source": [
"### Step 2 - configure Weaviate Batch, with\n",
"# - starting batch size of 100\n",
"# - dynamically increase/decrease based on performance\n",
"# - add timeout retries if something goes wrong\n",
"\n",
"client.batch.configure(\n",
" batch_size=10, \n",
" dynamic=True,\n",
" timeout_retries=3,\n",
"# callback=None,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15db8380",
"metadata": {},
"outputs": [],
"source": [
"### Step 3 - import data\n",
"\n",
"print(\"Importing Articles\")\n",
"\n",
"counter=0\n",
"\n",
"with client.batch as batch:\n",
" for article in dataset:\n",
" if (counter %10 == 0):\n",
" print(f\"Import {counter} / {len(dataset)} \")\n",
"\n",
" properties = {\n",
" \"title\": article[\"title\"],\n",
" \"content\": article[\"text\"],\n",
" \"url\": article[\"url\"]\n",
" }\n",
" \n",
" batch.add_data_object(properties, \"Article\")\n",
" counter = counter+1\n",
"\n",
"print(\"Importing Articles complete\") "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3658693c",
"metadata": {},
"outputs": [],
"source": [
"# Test that all data has loaded – get object count\n",
"result = (\n",
" client.query.aggregate(\"Article\")\n",
" .with_fields(\"meta { count }\")\n",
" .do()\n",
")\n",
"print(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0d791186",
"metadata": {},
"outputs": [],
"source": [
"# Test one article has worked by checking one object\n",
"test_article = (\n",
" client.query\n",
" .get(\"Article\", [\"title\", \"url\", \"content\"])\n",
" .with_limit(1)\n",
" .do()\n",
")[\"data\"][\"Get\"][\"Article\"][0]\n",
"\n",
"print(test_article['title'])\n",
"print(test_article['url'])\n",
"print(test_article['content'])"
]
},
{
"cell_type": "markdown",
"id": "46050ca9",
"metadata": {},
"source": [
"### Search Data\n",
"\n",
"As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n",
"\n",
"Learn more about the `alpha` setting [here](https://weaviate.io/developers/weaviate/api/graphql/vector-search-parameters#hybrid)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b044aa93",
"metadata": {},
"outputs": [],
"source": [
"def hybrid_query_weaviate(query, collection_name, alpha_val):\n",
" \n",
" nearText = {\n",
" \"concepts\": [query],\n",
" \"distance\": 0.7,\n",
" }\n",
"\n",
" properties = [\n",
" \"title\", \"content\", \"url\",\n",
" \"_additional { score }\"\n",
" ]\n",
"\n",
" result = (\n",
" client.query\n",
" .get(collection_name, properties)\n",
" .with_hybrid(nearText, alpha=alpha_val)\n",
" .with_limit(10)\n",
" .do()\n",
" )\n",
" \n",
" # Check for errors\n",
" if (\"errors\" in result):\n",
" print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.\")\n",
" raise Exception(result[\"errors\"][0]['message'])\n",
" \n",
" return result[\"data\"][\"Get\"][collection_name]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e2025f6",
"metadata": {},
"outputs": [],
"source": [
"query_result = hybrid_query_weaviate(\"modern art in Europe\", \"Article\", 0.5)\n",
"\n",
"for i, article in enumerate(query_result):\n",
" print(f\"{i+1}. { article['title']} (Score: {article['_additional']['score']})\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "93c4a696",
"metadata": {},
"outputs": [],
"source": [
"query_result = hybrid_query_weaviate(\"Famous battles in Scottish history\", \"Article\", 0.5)\n",
"\n",
"for i, article in enumerate(query_result):\n",
" print(f\"{i+1}. { article['title']} (Score: {article['_additional']['score']})\")"
]
},
{
"cell_type": "markdown",
"id": "2007be48",
"metadata": {},
"source": [
"Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"vscode": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb
+583
View File
@@ -0,0 +1,583 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Question Answering in Weaviate with OpenAI Q&A module\n",
"\n",
"This notebook is prepared for a scenario where:\n",
"* Your data is not vectorized\n",
"* You want to run Q&A ([learn more](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai)) on your data based on the [OpenAI completions](https://beta.openai.com/docs/api-reference/completions) endpoint.\n",
"* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.\n",
"\n",
"This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run question answering.\n",
"\n",
"## What is Weaviate\n",
"\n",
"Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n",
"\n",
"Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n",
"\n",
"Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n",
"\n",
"### Deployment options\n",
"\n",
"Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n",
"* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.\n",
"* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n",
"* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service \n",
"\n",
"### Programming languages\n",
"\n",
"Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n",
"* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n",
"* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n",
"* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n",
"* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n",
"\n",
"Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests."
]
},
{
"cell_type": "markdown",
"id": "45956173",
"metadata": {},
"source": [
"## Demo Flow\n",
"The demo flow is:\n",
"- **Prerequisites Setup**: Create a Weaviate instance and install required libraries\n",
"- **Connect**: Connect to your Weaviate instance \n",
"- **Schema Configuration**: Configure the schema of your data\n",
" - *Note*: Here we can define which OpenAI Embedding Model to use\n",
" - *Note*: Here we can configure which properties to index\n",
"- **Import data**: Load a demo dataset and import it into Weaviate\n",
" - *Note*: The import process will automatically index your data - based on the configuration in the schema\n",
" - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n",
"- **Run Queries**: Query \n",
" - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n",
" - *Note*: The `qna-openai` module automatically communicates with the OpenAI completions endpoint\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases for question answering."
]
},
{
"cell_type": "markdown",
"id": "2a4a145e",
"metadata": {},
"source": [
"## OpenAI Module in Weaviate\n",
"All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) and the [qna-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai) modules.\n",
"\n",
"The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.\n",
"\n",
"### No need to manually vectorize data\n",
"This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.\n",
"\n",
"All you need to do is:\n",
"1. provide your OpenAI API Key – when you connected to the Weaviate Client\n",
"2. define which OpenAI vectorizer to use in your Schema"
]
},
{
"cell_type": "markdown",
"id": "f1a618c5",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Before we start this project, we need setup the following:\n",
"\n",
"* create a `Weaviate` instance\n",
"* install libraries\n",
" * `weaviate-client`\n",
" * `datasets`\n",
" * `apache-beam`\n",
"* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n",
"\n",
"===========================================================\n",
"### Create a Weaviate instance\n",
"\n",
"To create a Weaviate instance we have 2 options:\n",
"\n",
"1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n",
"2. Install and run Weaviate locally with Docker.\n",
"\n",
"#### Option 1 – WCS Installation Steps\n",
"\n",
"Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n",
"1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n",
"2. create a `Weaviate Cluster` with the following settings:\n",
" * Sandbox: `Sandbox Free`\n",
" * Weaviate Version: Use default (latest)\n",
" * OIDC Authentication: `Disabled`\n",
"3. your instance should be ready in a minute or two\n",
"4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` \n",
"\n",
"#### Option 2 – local Weaviate instance with Docker\n",
"\n",
"Install and run Weaviate locally with Docker.\n",
"1. Download the [./docker-compose.yml](./docker-compose.yml) file\n",
"2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n",
"3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n",
"\n",
"Note. To shut down your docker instance you can call: `docker-compose down`\n",
"\n",
"##### Learn more\n",
"To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose)."
]
},
{
"cell_type": "markdown",
"id": "b9babafe",
"metadata": {},
"source": [
"=========================================================== \n",
"## Install required libraries\n",
"\n",
"Before running this project make sure to have the following libraries:\n",
"\n",
"### Weaviate Python client\n",
"\n",
"The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n",
"\n",
"### datasets & apache-beam\n",
"\n",
"To load sample data, you need the `datasets` library and its' dependency `apache-beam`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2b04113f",
"metadata": {},
"outputs": [],
"source": [
"# Install the Weaviate client for Python\n",
"!pip install weaviate-client>3.11.0\n",
"\n",
"# Install datasets and apache-beam to load the sample datasets\n",
"!pip install datasets apache-beam"
]
},
{
"cell_type": "markdown",
"id": "36fe86f4",
"metadata": {},
"source": [
"===========================================================\n",
"## Prepare your OpenAI API key\n",
"\n",
"The `OpenAI API key` is used for vectorization of your data at import, and for queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5a2ded4b",
"metadata": {},
"outputs": [],
"source": [
"# Export OpenAI API Key\n",
"!export OPENAI_API_KEY=\"your key\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "88be138c",
"metadata": {},
"outputs": [],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print (\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print (\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"id": "91df4d5b",
"metadata": {},
"source": [
"## Connect to your Weaviate instance\n",
"\n",
"In this section, we will:\n",
"\n",
"1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n",
"2. connect to your Weaviate your `OpenAI API Key`\n",
"3. and test the client connection\n",
"\n",
"### The client \n",
"\n",
"After this step, the `client` object will be used to perform all Weaviate-related operations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cc662c1b",
"metadata": {},
"outputs": [],
"source": [
"import weaviate\n",
"from datasets import load_dataset\n",
"import os\n",
"\n",
"# Connect to your Weaviate instance\n",
"client = weaviate.Client(\n",
" url=\"https://your-wcs-instance-name.weaviate.network/\",\n",
"# url=\"http://localhost:8080/\",\n",
" auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n",
" additional_headers={\n",
" \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n",
" }\n",
")\n",
"\n",
"# Check if your instance is live and ready\n",
"# This should return `True`\n",
"client.is_ready()"
]
},
{
"cell_type": "markdown",
"id": "7d3dac3c",
"metadata": {},
"source": [
"# Schema\n",
"\n",
"In this section, we will:\n",
"1. configure the data schema for your data\n",
"2. select OpenAI module\n",
"\n",
"> This is the second and final step, which requires OpenAI specific configuration.\n",
"> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n",
"\n",
"\n",
"## What is a schema\n",
"\n",
"In Weaviate you create __schemas__ to capture each of the entities you will be searching.\n",
"\n",
"A schema is how you tell Weaviate:\n",
"* what embedding model should be used to vectorize the data\n",
"* what your data is made of (property names and types)\n",
"* which properties should be vectorized and indexed\n",
"\n",
"In this cookbook we will use a dataset for `Articles`, which contains:\n",
"* `title`\n",
"* `content`\n",
"* `url`\n",
"\n",
"We want to vectorize `title` and `content`, but not the `url`.\n",
"\n",
"To vectorize and query the data, we will use `text-embedding-3-small`. For Q&A we will use `gpt-3.5-turbo-instruct`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f894b911",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Clear up the schema, so that we can recreate it\n",
"client.schema.delete_all()\n",
"client.schema.get()\n",
"\n",
"# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\n",
"article_schema = {\n",
" \"class\": \"Article\",\n",
" \"description\": \"A collection of articles\",\n",
" \"vectorizer\": \"text2vec-openai\",\n",
" \"moduleConfig\": {\n",
" \"text2vec-openai\": {\n",
" \"model\": \"ada\",\n",
" \"modelVersion\": \"002\",\n",
" \"type\": \"text\"\n",
" }, \n",
" \"qna-openai\": {\n",
" \"model\": \"gpt-3.5-turbo-instruct\",\n",
" \"maxTokens\": 16,\n",
" \"temperature\": 0.0,\n",
" \"topP\": 1,\n",
" \"frequencyPenalty\": 0.0,\n",
" \"presencePenalty\": 0.0\n",
" }\n",
" },\n",
" \"properties\": [{\n",
" \"name\": \"title\",\n",
" \"description\": \"Title of the article\",\n",
" \"dataType\": [\"string\"]\n",
" },\n",
" {\n",
" \"name\": \"content\",\n",
" \"description\": \"Contents of the article\",\n",
" \"dataType\": [\"text\"]\n",
" },\n",
" {\n",
" \"name\": \"url\",\n",
" \"description\": \"URL to the article\",\n",
" \"dataType\": [\"string\"],\n",
" \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n",
" }]\n",
"}\n",
"\n",
"# add the Article schema\n",
"client.schema.create_class(article_schema)\n",
"\n",
"# get the schema to make sure it worked\n",
"client.schema.get()"
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Import data\n",
"\n",
"In this section we will:\n",
"1. load the Simple Wikipedia dataset\n",
"2. configure Weaviate Batch import (to make the import more efficient)\n",
"3. import the data into Weaviate\n",
"\n",
"> Note: <br/>\n",
"> Like mentioned before. We don't need to manually vectorize the data.<br/>\n",
"> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc3efadd",
"metadata": {},
"outputs": [],
"source": [
"### STEP 1 - load the dataset\n",
"\n",
"from datasets import load_dataset\n",
"from typing import List, Iterator\n",
"\n",
"# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\n",
"dataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n",
"\n",
"# For testing, limited to 2.5k articles for demo purposes\n",
"dataset = dataset[:2_500]\n",
"\n",
"# Limited to 25k articles for larger demo purposes\n",
"# dataset = dataset[:25_000]\n",
"\n",
"# for free OpenAI acounts, you can use 50 objects\n",
"# dataset = dataset[:50]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5044da96",
"metadata": {},
"outputs": [],
"source": [
"### Step 2 - configure Weaviate Batch, with\n",
"# - starting batch size of 100\n",
"# - dynamically increase/decrease based on performance\n",
"# - add timeout retries if something goes wrong\n",
"\n",
"client.batch.configure(\n",
" batch_size=10, \n",
" dynamic=True,\n",
" timeout_retries=3,\n",
"# callback=None,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15db8380",
"metadata": {},
"outputs": [],
"source": [
"### Step 3 - import data\n",
"\n",
"print(\"Importing Articles\")\n",
"\n",
"counter=0\n",
"\n",
"with client.batch as batch:\n",
" for article in dataset:\n",
" if (counter %10 == 0):\n",
" print(f\"Import {counter} / {len(dataset)} \")\n",
"\n",
" properties = {\n",
" \"title\": article[\"title\"],\n",
" \"content\": article[\"text\"],\n",
" \"url\": article[\"url\"]\n",
" }\n",
" \n",
" batch.add_data_object(properties, \"Article\")\n",
" counter = counter+1\n",
"\n",
"print(\"Importing Articles complete\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3658693c",
"metadata": {},
"outputs": [],
"source": [
"# Test that all data has loaded – get object count\n",
"result = (\n",
" client.query.aggregate(\"Article\")\n",
" .with_fields(\"meta { count }\")\n",
" .do()\n",
")\n",
"print(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0d791186",
"metadata": {},
"outputs": [],
"source": [
"# Test one article has worked by checking one object\n",
"test_article = (\n",
" client.query\n",
" .get(\"Article\", [\"title\", \"url\", \"content\"])\n",
" .with_limit(1)\n",
" .do()\n",
")[\"data\"][\"Get\"][\"Article\"][0]\n",
"\n",
"print(test_article['title'])\n",
"print(test_article['url'])\n",
"print(test_article['content'])"
]
},
{
"cell_type": "markdown",
"id": "46050ca9",
"metadata": {},
"source": [
"### Question Answering on the Data\n",
"\n",
"As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b044aa93",
"metadata": {},
"outputs": [],
"source": [
"def qna(query, collection_name):\n",
" \n",
" properties = [\n",
" \"title\", \"content\", \"url\",\n",
" \"_additional { answer { hasAnswer property result startPosition endPosition } distance }\"\n",
" ]\n",
"\n",
" ask = {\n",
" \"question\": query,\n",
" \"properties\": [\"content\"]\n",
" }\n",
"\n",
" result = (\n",
" client.query\n",
" .get(collection_name, properties)\n",
" .with_ask(ask)\n",
" .with_limit(1)\n",
" .do()\n",
" )\n",
" \n",
" # Check for errors\n",
" if (\"errors\" in result):\n",
" print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.\")\n",
" raise Exception(result[\"errors\"][0]['message'])\n",
" \n",
" return result[\"data\"][\"Get\"][collection_name]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e2025f6",
"metadata": {},
"outputs": [],
"source": [
"query_result = qna(\"Did Alanis Morissette win a Grammy?\", \"Article\")\n",
"\n",
"for i, article in enumerate(query_result):\n",
" print(f\"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "93c4a696",
"metadata": {},
"outputs": [],
"source": [
"query_result = qna(\"What is the capital of China?\", \"Article\")\n",
"\n",
"for i, article in enumerate(query_result):\n",
" if article['_additional']['answer']['hasAnswer'] == False:\n",
" print('No answer found')\n",
" else:\n",
" print(f\"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })\")"
]
},
{
"cell_type": "markdown",
"id": "2007be48",
"metadata": {},
"source": [
"Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"vscode": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb
+327
View File
@@ -0,0 +1,327 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Filtered Search with Zilliz and OpenAI\n",
"### Finding your next movie\n",
"\n",
"In this notebook we will be going over generating embeddings of movie descriptions with OpenAI and using those embeddings within Zilliz to find relevant movies. To narrow our search results and try something new, we are going to be using filtering to do metadata searches. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 8 thousand movie entries.\n",
"\n",
"Lets begin by first downloading the required libraries for this notebook:\n",
"- `openai` is used for communicating with the OpenAI embedding service\n",
"- `pymilvus` is used for communicating with the Zilliz server\n",
"- `datasets` is used for downloading the dataset\n",
"- `tqdm` is used for the progress bars\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pip install openai pymilvus datasets tqdm"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"To get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:\n",
"- URI: The URI your database is running on\n",
"- USER: Your database username\n",
"- PASSWORD: Your database password\n",
"- COLLECTION_NAME: What to name the collection within Zilliz\n",
"- DIMENSION: The dimension of the embeddings\n",
"- OPENAI_ENGINE: Which embedding model to use\n",
"- openai.api_key: Your OpenAI account key\n",
"- INDEX_PARAM: The index settings to use for the collection\n",
"- QUERY_PARAM: The search parameters to use\n",
"- BATCH_SIZE: How many texts to embed and insert at once"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"URI = 'your_uri'\n",
"TOKEN = 'your_token' # TOKEN == user:password or api_key\n",
"COLLECTION_NAME = 'book_search'\n",
"DIMENSION = 1536\n",
"OPENAI_ENGINE = 'text-embedding-3-small'\n",
"openai.api_key = 'sk-your_key'\n",
"\n",
"INDEX_PARAM = {\n",
" 'metric_type':'L2',\n",
" 'index_type':\"AUTOINDEX\",\n",
" 'params':{}\n",
"}\n",
"\n",
"QUERY_PARAM = {\n",
" \"metric_type\": \"L2\",\n",
" \"params\": {},\n",
"}\n",
"\n",
"BATCH_SIZE = 1000"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n",
"\n",
"# Connect to Zilliz Database\n",
"connections.connect(uri=URI, token=TOKEN)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Remove collection if it already exists\n",
"if utility.has_collection(COLLECTION_NAME):\n",
" utility.drop_collection(COLLECTION_NAME)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# Create collection which includes the id, title, and embedding.\n",
"fields = [\n",
" FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n",
" FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='type', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='release_year', dtype=DataType.INT64),\n",
" FieldSchema(name='rating', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n",
"]\n",
"schema = CollectionSchema(fields=fields)\n",
"collection = Collection(name=COLLECTION_NAME, schema=schema)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# Create the index on the collection and load it.\n",
"collection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\n",
"collection.load()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dataset\n",
"With Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Zilliz along with its title, type, release_year and rating."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"Found cached dataset csv (/Users/filiphaltmayer/.cache/huggingface/datasets/hugginglearners___csv/hugginglearners--netflix-shows-03475319fc65a05a/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317)\n"
]
}
],
"source": [
"import datasets\n",
"\n",
"# Download the dataset \n",
"dataset = datasets.load_dataset('hugginglearners/netflix-shows', split='train')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Insert the Data\n",
"Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format. "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# Simple function that converts the texts to embeddings\n",
"def embed(texts):\n",
" embeddings = openai.Embedding.create(\n",
" input=texts,\n",
" engine=OPENAI_ENGINE\n",
" )\n",
" return [x['embedding'] for x in embeddings['data']]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaning batch if it exists. "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 8807/8807 [00:54<00:00, 162.59it/s]\n"
]
}
],
"source": [
"from tqdm import tqdm\n",
"\n",
"data = [\n",
" [], # title\n",
" [], # type\n",
" [], # release_year\n",
" [], # rating\n",
" [], # description\n",
"]\n",
"\n",
"# Embed and insert in batches\n",
"for i in tqdm(range(0, len(dataset))):\n",
" data[0].append(dataset[i]['title'] or '')\n",
" data[1].append(dataset[i]['type'] or '')\n",
" data[2].append(dataset[i]['release_year'] or -1)\n",
" data[3].append(dataset[i]['rating'] or '')\n",
" data[4].append(dataset[i]['description'] or '')\n",
" if len(data[0]) % BATCH_SIZE == 0:\n",
" data.append(embed(data[4]))\n",
" collection.insert(data)\n",
" data = [[],[],[],[],[]]\n",
"\n",
"# Embed and insert the remainder \n",
"if len(data[0]) != 0:\n",
" data.append(embed(data[4]))\n",
" collection.insert(data)\n",
" data = [[],[],[],[],[]]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Query the Database\n",
"With our data safely inserted into Zilliz, we can now perform a query. The query takes in a tuple of the movie description you are searching for and the filter to use. More info about the filter can be found [here](https://milvus.io/docs/boolean.md). The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating and description of the result movies. "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Description: movie about a fluffly animal Expression: release_year < 2019 and rating like \"PG%\"\n",
"Results:\n",
"\tRank: 1 Score: 0.30085673928260803 Title: The Lamb\n",
"\t\tType: Movie Release Year: 2017 Rating: PG\n",
"A big-dreaming donkey escapes his menial existence and befriends some free-spirited\n",
"animal pals in this imaginative retelling of the Nativity Story.\n",
"\n",
"\tRank: 2 Score: 0.3352621793746948 Title: Puss in Boots\n",
"\t\tType: Movie Release Year: 2011 Rating: PG\n",
"The fabled feline heads to the Land of Giants with friends Humpty Dumpty and Kitty\n",
"Softpaws on a quest to nab its greatest treasure: the Golden Goose.\n",
"\n",
"\tRank: 3 Score: 0.3415083587169647 Title: Show Dogs\n",
"\t\tType: Movie Release Year: 2018 Rating: PG\n",
"A rough and tough police dog must go undercover with an FBI agent as a prim and proper\n",
"pet at a dog show to save a baby panda from an illegal sale.\n",
"\n",
"\tRank: 4 Score: 0.3428957462310791 Title: Open Season 2\n",
"\t\tType: Movie Release Year: 2008 Rating: PG\n",
"Elliot the buck and his forest-dwelling cohorts must rescue their dachshund pal from\n",
"some spoiled pets bent on returning him to domesticity.\n",
"\n",
"\tRank: 5 Score: 0.34376364946365356 Title: Stuart Little 2\n",
"\t\tType: Movie Release Year: 2002 Rating: PG\n",
"Zany misadventures are in store as lovable city mouse Stuart and his human brother,\n",
"George, raise the roof in this sequel to the 1999 blockbuster.\n",
"\n"
]
}
],
"source": [
"import textwrap\n",
"\n",
"def query(query, top_k = 5):\n",
" text, expr = query\n",
" res = collection.search(embed(text), anns_field='embedding', expr = expr, param=QUERY_PARAM, limit = top_k, output_fields=['title', 'type', 'release_year', 'rating', 'description'])\n",
" for i, hit in enumerate(res):\n",
" print('Description:', text, 'Expression:', expr)\n",
" print('Results:')\n",
" for ii, hits in enumerate(hit):\n",
" print('\\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))\n",
" print('\\t\\t' + 'Type:', hits.entity.get('type'), 'Release Year:', hits.entity.get('release_year'), 'Rating:', hits.entity.get('rating'))\n",
" print(textwrap.fill(hits.entity.get('description'), 88))\n",
" print()\n",
"\n",
"my_query = ('movie about a fluffly animal', 'release_year < 2019 and rating like \\\"PG%\\\"')\n",
"\n",
"query(my_query)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "haystack",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb
+422
View File
@@ -0,0 +1,422 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Getting Started with Zilliz and OpenAI\n",
"### Finding your next book\n",
"\n",
"In this notebook we will be going over generating embeddings of book descriptions with OpenAI and using those embeddings within Zilliz to find relevant books. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 1 million title-description pairs.\n",
"\n",
"Lets begin by first downloading the required libraries for this notebook:\n",
"- `openai` is used for communicating with the OpenAI embedding service\n",
"- `pymilvus` is used for communicating with the Zilliz instance\n",
"- `datasets` is used for downloading the dataset\n",
"- `tqdm` is used for the progress bars\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
"Requirement already satisfied: openai in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (0.27.2)\n",
"Requirement already satisfied: pymilvus in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (2.2.2)\n",
"Requirement already satisfied: datasets in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (2.10.1)\n",
"Requirement already satisfied: tqdm in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (4.64.1)\n",
"Requirement already satisfied: requests>=2.20 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from openai) (2.28.2)\n",
"Requirement already satisfied: aiohttp in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from openai) (3.8.4)\n",
"Requirement already satisfied: ujson<=5.4.0,>=2.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (5.1.0)\n",
"Requirement already satisfied: grpcio-tools<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)\n",
"Requirement already satisfied: grpcio<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)\n",
"Requirement already satisfied: mmh3<=3.0.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (3.0.0)\n",
"Requirement already satisfied: pandas>=1.2.4 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.5.3)\n",
"Requirement already satisfied: numpy>=1.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (1.23.5)\n",
"Requirement already satisfied: xxhash in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (3.2.0)\n",
"Requirement already satisfied: responses<0.19 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.18.0)\n",
"Requirement already satisfied: dill<0.3.7,>=0.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.3.6)\n",
"Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.12.1)\n",
"Requirement already satisfied: pyarrow>=6.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (10.0.1)\n",
"Requirement already satisfied: multiprocess in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.70.14)\n",
"Requirement already satisfied: pyyaml>=5.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (5.4.1)\n",
"Requirement already satisfied: fsspec[http]>=2021.11.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (2023.1.0)\n",
"Requirement already satisfied: packaging in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (23.0)\n",
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.3)\n",
"Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (4.0.2)\n",
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.1)\n",
"Requirement already satisfied: attrs>=17.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (22.2.0)\n",
"Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (3.0.1)\n",
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.8.2)\n",
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (6.0.4)\n",
"Requirement already satisfied: six>=1.5.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio<=1.48.0,>=1.47.0->pymilvus) (1.16.0)\n",
"Requirement already satisfied: protobuf<4.0dev,>=3.12.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (3.20.1)\n",
"Requirement already satisfied: setuptools in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (65.6.3)\n",
"Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (4.5.0)\n",
"Requirement already satisfied: filelock in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.9.0)\n",
"Requirement already satisfied: python-dateutil>=2.8.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2.8.2)\n",
"Requirement already satisfied: pytz>=2020.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2022.7.1)\n",
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (1.26.14)\n",
"Requirement already satisfied: idna<4,>=2.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (3.4)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.7)\n"
]
}
],
"source": [
"! pip install openai pymilvus datasets tqdm"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"To get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:\n",
"- URI: The URI your database is running on\n",
"- USER: Your database username\n",
"- PASSWORD: Your database password\n",
"- COLLECTION_NAME: What to name the collection within Zilliz\n",
"- DIMENSION: The dimension of the embeddings\n",
"- OPENAI_ENGINE: Which embedding model to use\n",
"- openai.api_key: Your OpenAI account key\n",
"- INDEX_PARAM: The index settings to use for the collection\n",
"- QUERY_PARAM: The search parameters to use\n",
"- BATCH_SIZE: How many texts to embed and insert at once"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import openai\n",
"\n",
"URI = 'your_uri'\n",
"TOKEN = 'your_token' # TOKEN == user:password or api_key\n",
"COLLECTION_NAME = 'book_search'\n",
"DIMENSION = 1536\n",
"OPENAI_ENGINE = 'text-embedding-3-small'\n",
"openai.api_key = 'sk-your-key'\n",
"\n",
"INDEX_PARAM = {\n",
" 'metric_type':'L2',\n",
" 'index_type':\"AUTOINDEX\",\n",
" 'params':{}\n",
"}\n",
"\n",
"QUERY_PARAM = {\n",
" \"metric_type\": \"L2\",\n",
" \"params\": {},\n",
"}\n",
"\n",
"BATCH_SIZE = 1000"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Zilliz\n",
"This segment deals with Zilliz and setting up the database for this use case. Within Zilliz we need to setup a collection and index it."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n",
"\n",
"# Connect to Zilliz Database\n",
"connections.connect(uri=URI, token=TOKEN)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Remove collection if it already exists\n",
"if utility.has_collection(COLLECTION_NAME):\n",
" utility.drop_collection(COLLECTION_NAME)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# Create collection which includes the id, title, and embedding.\n",
"fields = [\n",
" FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n",
" FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n",
" FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n",
"]\n",
"schema = CollectionSchema(fields=fields)\n",
"collection = Collection(name=COLLECTION_NAME, schema=schema)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# Create the index on the collection and load it.\n",
"collection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\n",
"collection.load()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dataset\n",
"With Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using Skelebor's book dataset. This dataset contains title-description pairs for over 1 million books. We are going to embed each description and store it within Zilliz along with its title."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"Found cached dataset parquet (/Users/filiphaltmayer/.cache/huggingface/datasets/Skelebor___parquet/Skelebor--book_titles_and_descriptions_en_clean-3596935b1d8a7747/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
]
}
],
"source": [
"import datasets\n",
"\n",
"# Download the dataset and only use the `train` portion (file is around 800Mb)\n",
"dataset = datasets.load_dataset('Skelebor/book_titles_and_descriptions_en_clean', split='train')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Insert the Data\n",
"Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# Simple function that converts the texts to embeddings\n",
"def embed(texts):\n",
" embeddings = openai.Embedding.create(\n",
" input=texts,\n",
" engine=OPENAI_ENGINE\n",
" )\n",
" return [x['embedding'] for x in embeddings['data']]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This next step does the actual inserting. Due to having so many datapoints, if you want to immediately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" 0%| | 2999/1032335 [00:19<1:49:30, 156.66it/s]\n"
]
},
{
"ename": "KeyboardInterrupt",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[10], line 14\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mlen\u001b[39m(data[\u001b[39m0\u001b[39m]) \u001b[39m%\u001b[39m BATCH_SIZE \u001b[39m==\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m 13\u001b[0m data\u001b[39m.\u001b[39mappend(embed(data[\u001b[39m1\u001b[39m]))\n\u001b[0;32m---> 14\u001b[0m collection\u001b[39m.\u001b[39;49minsert(data)\n\u001b[1;32m 15\u001b[0m data \u001b[39m=\u001b[39m [[],[]]\n\u001b[1;32m 17\u001b[0m \u001b[39m# Embed and insert the remainder \u001b[39;00m\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/orm/collection.py:430\u001b[0m, in \u001b[0;36mCollection.insert\u001b[0;34m(self, data, partition_name, timeout, **kwargs)\u001b[0m\n\u001b[1;32m 427\u001b[0m entities \u001b[39m=\u001b[39m Prepare\u001b[39m.\u001b[39mprepare_insert_data(data, \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_schema)\n\u001b[1;32m 429\u001b[0m conn \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_get_connection()\n\u001b[0;32m--> 430\u001b[0m res \u001b[39m=\u001b[39m conn\u001b[39m.\u001b[39;49mbatch_insert(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_name, entities, partition_name,\n\u001b[1;32m 431\u001b[0m timeout\u001b[39m=\u001b[39;49mtimeout, schema\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_schema_dict, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 433\u001b[0m \u001b[39mif\u001b[39;00m kwargs\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39m_async\u001b[39m\u001b[39m\"\u001b[39m, \u001b[39mFalse\u001b[39;00m):\n\u001b[1;32m 434\u001b[0m \u001b[39mreturn\u001b[39;00m MutationFuture(res)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:105\u001b[0m, in \u001b[0;36merror_handler.<locals>.wrapper.<locals>.handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 103\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m 104\u001b[0m record_dict[\u001b[39m\"\u001b[39m\u001b[39mRPC start\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(datetime\u001b[39m.\u001b[39mdatetime\u001b[39m.\u001b[39mnow())\n\u001b[0;32m--> 105\u001b[0m \u001b[39mreturn\u001b[39;00m func(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 106\u001b[0m \u001b[39mexcept\u001b[39;00m MilvusException \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 107\u001b[0m record_dict[\u001b[39m\"\u001b[39m\u001b[39mRPC error\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mstr\u001b[39m(datetime\u001b[39m.\u001b[39mdatetime\u001b[39m.\u001b[39mnow())\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:136\u001b[0m, in \u001b[0;36mtracing_request.<locals>.wrapper.<locals>.handler\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 134\u001b[0m \u001b[39mif\u001b[39;00m req_id:\n\u001b[1;32m 135\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mset_onetime_request_id(req_id)\n\u001b[0;32m--> 136\u001b[0m ret \u001b[39m=\u001b[39m func(\u001b[39mself\u001b[39;49m, \u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 137\u001b[0m \u001b[39mreturn\u001b[39;00m ret\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:50\u001b[0m, in \u001b[0;36mretry_on_rpc_failure.<locals>.wrapper.<locals>.handler\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 48\u001b[0m \u001b[39mwhile\u001b[39;00m \u001b[39mTrue\u001b[39;00m:\n\u001b[1;32m 49\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m---> 50\u001b[0m \u001b[39mreturn\u001b[39;00m func(\u001b[39mself\u001b[39;49m, \u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m 51\u001b[0m \u001b[39mexcept\u001b[39;00m grpc\u001b[39m.\u001b[39mRpcError \u001b[39mas\u001b[39;00m e:\n\u001b[1;32m 52\u001b[0m \u001b[39m# DEADLINE_EXCEEDED means that the task wat not completed\u001b[39;00m\n\u001b[1;32m 53\u001b[0m \u001b[39m# UNAVAILABLE means that the service is not reachable currently\u001b[39;00m\n\u001b[1;32m 54\u001b[0m \u001b[39m# Reference: https://grpc.github.io/grpc/python/grpc.html#grpc-status-code\u001b[39;00m\n\u001b[1;32m 55\u001b[0m \u001b[39mif\u001b[39;00m e\u001b[39m.\u001b[39mcode() \u001b[39m!=\u001b[39m grpc\u001b[39m.\u001b[39mStatusCode\u001b[39m.\u001b[39mDEADLINE_EXCEEDED \u001b[39mand\u001b[39;00m e\u001b[39m.\u001b[39mcode() \u001b[39m!=\u001b[39m grpc\u001b[39m.\u001b[39mStatusCode\u001b[39m.\u001b[39mUNAVAILABLE:\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:378\u001b[0m, in \u001b[0;36mGrpcHandler.batch_insert\u001b[0;34m(self, collection_name, entities, partition_name, timeout, **kwargs)\u001b[0m\n\u001b[1;32m 375\u001b[0m f\u001b[39m.\u001b[39madd_callback(ts_utils\u001b[39m.\u001b[39mupdate_ts_on_mutation(collection_name))\n\u001b[1;32m 376\u001b[0m \u001b[39mreturn\u001b[39;00m f\n\u001b[0;32m--> 378\u001b[0m response \u001b[39m=\u001b[39m rf\u001b[39m.\u001b[39;49mresult()\n\u001b[1;32m 379\u001b[0m \u001b[39mif\u001b[39;00m response\u001b[39m.\u001b[39mstatus\u001b[39m.\u001b[39merror_code \u001b[39m==\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m 380\u001b[0m m \u001b[39m=\u001b[39m MutationResult(response)\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_channel.py:733\u001b[0m, in \u001b[0;36m_MultiThreadedRendezvous.result\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m 728\u001b[0m \u001b[39m\u001b[39m\u001b[39m\"\"\"Returns the result of the computation or raises its exception.\u001b[39;00m\n\u001b[1;32m 729\u001b[0m \n\u001b[1;32m 730\u001b[0m \u001b[39mSee grpc.Future.result for the full API contract.\u001b[39;00m\n\u001b[1;32m 731\u001b[0m \u001b[39m\"\"\"\u001b[39;00m\n\u001b[1;32m 732\u001b[0m \u001b[39mwith\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_state\u001b[39m.\u001b[39mcondition:\n\u001b[0;32m--> 733\u001b[0m timed_out \u001b[39m=\u001b[39m _common\u001b[39m.\u001b[39;49mwait(\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_state\u001b[39m.\u001b[39;49mcondition\u001b[39m.\u001b[39;49mwait,\n\u001b[1;32m 734\u001b[0m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_is_complete,\n\u001b[1;32m 735\u001b[0m timeout\u001b[39m=\u001b[39;49mtimeout)\n\u001b[1;32m 736\u001b[0m \u001b[39mif\u001b[39;00m timed_out:\n\u001b[1;32m 737\u001b[0m \u001b[39mraise\u001b[39;00m grpc\u001b[39m.\u001b[39mFutureTimeoutError()\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:141\u001b[0m, in \u001b[0;36mwait\u001b[0;34m(wait_fn, wait_complete_fn, timeout, spin_cb)\u001b[0m\n\u001b[1;32m 139\u001b[0m \u001b[39mif\u001b[39;00m timeout \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[1;32m 140\u001b[0m \u001b[39mwhile\u001b[39;00m \u001b[39mnot\u001b[39;00m wait_complete_fn():\n\u001b[0;32m--> 141\u001b[0m _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)\n\u001b[1;32m 142\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 143\u001b[0m end \u001b[39m=\u001b[39m time\u001b[39m.\u001b[39mtime() \u001b[39m+\u001b[39m timeout\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:106\u001b[0m, in \u001b[0;36m_wait_once\u001b[0;34m(wait_fn, timeout, spin_cb)\u001b[0m\n\u001b[1;32m 105\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m_wait_once\u001b[39m(wait_fn, timeout, spin_cb):\n\u001b[0;32m--> 106\u001b[0m wait_fn(timeout\u001b[39m=\u001b[39;49mtimeout)\n\u001b[1;32m 107\u001b[0m \u001b[39mif\u001b[39;00m spin_cb \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[1;32m 108\u001b[0m spin_cb()\n",
"File \u001b[0;32m~/miniconda3/envs/haystack/lib/python3.9/threading.py:316\u001b[0m, in \u001b[0;36mCondition.wait\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m 314\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 315\u001b[0m \u001b[39mif\u001b[39;00m timeout \u001b[39m>\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[0;32m--> 316\u001b[0m gotit \u001b[39m=\u001b[39m waiter\u001b[39m.\u001b[39;49macquire(\u001b[39mTrue\u001b[39;49;00m, timeout)\n\u001b[1;32m 317\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m 318\u001b[0m gotit \u001b[39m=\u001b[39m waiter\u001b[39m.\u001b[39macquire(\u001b[39mFalse\u001b[39;00m)\n",
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
]
}
],
"source": [
"from tqdm import tqdm\n",
"\n",
"data = [\n",
" [], # title\n",
" [], # description\n",
"]\n",
"\n",
"# Embed and insert in batches\n",
"for i in tqdm(range(0, len(dataset))):\n",
" data[0].append(dataset[i]['title'])\n",
" data[1].append(dataset[i]['description'])\n",
" if len(data[0]) % BATCH_SIZE == 0:\n",
" data.append(embed(data[1]))\n",
" collection.insert(data)\n",
" data = [[],[]]\n",
"\n",
"# Embed and insert the remainder \n",
"if len(data[0]) != 0:\n",
" data.append(embed(data[1]))\n",
" collection.insert(data)\n",
" data = [[],[]]\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Query the Database\n",
"With our data safely inserted in Zilliz, we can now perform a query. The query takes in a string or a list of strings and searches them. The results print out your provided description and the results that include the result score, the result title, and the result book description."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"import textwrap\n",
"\n",
"def query(queries, top_k = 5):\n",
" if type(queries) != list:\n",
" queries = [queries]\n",
" res = collection.search(embed(queries), anns_field='embedding', param=QUERY_PARAM, limit = top_k, output_fields=['title', 'description'])\n",
" for i, hit in enumerate(res):\n",
" print('Description:', queries[i])\n",
" print('Results:')\n",
" for ii, hits in enumerate(hit):\n",
" print('\\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))\n",
" print(textwrap.fill(hits.entity.get('description'), 88))\n",
" print()"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Description: Book about a k-9 from europe\n",
"Results:\n",
"\tRank: 1 Score: 0.3047754764556885 Title: Bark M For Murder\n",
"Who let the dogs out? Evildoers beware! Four of mystery fiction's top storytellers are\n",
"setting the hounds on your trail -- in an incomparable quartet of crime stories with a\n",
"canine edge. Man's (and woman's) best friends take the lead in this phenomenal\n",
"collection of tales tense and surprising, humorous and thrilling: New York\n",
"Timesbestselling author J.A. Jance's spellbinding saga of a scam-busting septuagenarian\n",
"and her two golden retrievers; Anthony Award winner Virginia Lanier's pureblood thriller\n",
"featuring bloodhounds and bloody murder; Chassie West's suspenseful stunner about a\n",
"life-saving German shepherd and a ghastly forgotten crime; rising star Lee Charles\n",
"Kelley's edge-of-your-seat yarn that pits an ex-cop/kennel owner and a yappy toy poodle\n",
"against a craven killer.\n",
"\n",
"\tRank: 2 Score: 0.3283390402793884 Title: Texas K-9 Unit Christmas: Holiday Hero\\Rescuing Christmas\n",
"CHRISTMAS COMES WRAPPED IN DANGER Holiday Hero by Shirlee McCoy Emma Fairchild never\n",
"expected to find trouble in sleepy Sagebrush, Texas. But when she's attacked and left\n",
"for dead in her own diner, her childhood friend turned K-9 cop Lucas Harwood offers a\n",
"chance at justice--and love. Rescuing Christmas by Terri Reed She escaped a kidnapper,\n",
"but now a killer has set his sights on K-9 dog trainer Lily Anderson. When fellow\n",
"officer Jarrod Evans appoints himself her bodyguard, Lily knows more than her life is at\n",
"risk--so is her heart. Texas K-9 Unit: These lawmen solve the toughest cases with the\n",
"help of their brave canine partners\n",
"\n",
"\tRank: 3 Score: 0.33899369835853577 Title: Dogs on Duty: Soldiers' Best Friends on the Battlefield and Beyond\n",
"When the news of the raid on Osama Bin Laden's compound broke, the SEAL team member that\n",
"stole the show was a highly trained canine companion. Throughout history, dogs have been\n",
"key contributors to military units. Dorothy Hinshaw Patent follows man's best friend\n",
"onto the battlefield, showing readers why dogs are uniquely qualified for the job at\n",
"hand, how they are trained, how they contribute to missions, and what happens when they\n",
"retire. With full-color photographs throughout and sidebars featuring heroic canines\n",
"throughout history, Dogs on Duty provides a fascinating look at these exceptional\n",
"soldiers and companions.\n",
"\n",
"\tRank: 4 Score: 0.34207457304000854 Title: Toute Allure: Falling in Love in Rural France\n",
"After saying goodbye to life as a successful fashion editor in London, Karen Wheeler is\n",
"now happy in her small village house in rural France. Her idyll is complete when she\n",
"meets the love of her life - he has shaggy hair, four paws and a wet nose!\n",
"\n",
"\tRank: 5 Score: 0.343595951795578 Title: Otherwise Alone (Evan Arden, #1)\n",
"Librarian's note: This is an alternate cover edition for ASIN: B00AP5NNWC. Lieutenant\n",
"Evan Arden sits in a shack in the middle of nowhere, waiting for orders that will send\n",
"him back home - if he ever gets them. Other than his loyal Great Pyrenees, there's no\n",
"one around to break up the monotony. The tedium is excruciating, but it is suddenly\n",
"interrupted when a young woman stumbles up his path. \"It's only 50-something pages, but\n",
"in that short amount of time, the author's awesome writing packs in a whole lotta\n",
"character detail. And sets the stage for the series, perfectly.\" -Maryse.net, 4.5 Stars\n",
"He has two choices - pick her off from a distance with his trusty sniper-rifle, or dare\n",
"let her approach his cabin and enter his life. Why not? It's been ages, and he is\n",
"otherwise alone...\n",
"\n"
]
}
],
"source": [
"query('Book about a k-9 from europe')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "haystack",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}