mirrors/openai--openai-cookbook
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Rebuild Cookbook Website / deploy (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:41:49 +08:00
commit 327604cc89
3065 changed files with 860207 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/vector_databases/qdrant/Getting_started_with_Qdrant_and_OpenAI.ipynb
+727
View File
@@ -0,0 +1,727 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Using Qdrant as a vector database for OpenAI embeddings\n",
"\n",
"This notebook guides you step by step on using **`Qdrant`** as a vector database for OpenAI embeddings. [Qdrant](https://qdrant.tech) is a high-performant vector search database written in Rust. It offers RESTful and gRPC APIs to manage your embeddings. There is an official Python [qdrant-client](https://github.com/qdrant/qdrant_client) that eases the integration with your apps.\n",
"\n",
"This notebook presents an end-to-end process of:\n",
"1. Using precomputed embeddings created by OpenAI API.\n",
"2. Storing the embeddings in a local instance of Qdrant.\n",
"3. Converting raw text query to an embedding with OpenAI API.\n",
"4. Using Qdrant to perform the nearest neighbour search in the created collection.\n",
"\n",
"### What is Qdrant\n",
"\n",
"[Qdrant](https://qdrant.tech) is an Open Source vector database that allows storing neural embeddings along with the metadata, a.k.a [payload](https://qdrant.tech/documentation/payload/). Payloads are not only available for keeping some additional attributes of a particular point, but might be also used for filtering. [Qdrant](https://qdrant.tech) offers a unique filtering mechanism which is built-in into the vector search phase, what makes it really efficient.\n",
"\n",
"### Deployment options\n",
"\n",
"[Qdrant](https://qdrant.tech) might be launched in various ways, depending on the target load on the application it might be hosted:\n",
"\n",
"- Locally or on premise, with Docker containers\n",
"- On Kubernetes cluster, with the [Helm chart](https://github.com/qdrant/qdrant-helm)\n",
"- Using [Qdrant Cloud](https://cloud.qdrant.io/)\n",
"\n",
"### Integration\n",
"\n",
"[Qdrant](https://qdrant.tech) provides both RESTful and gRPC APIs which makes integration easy, no matter the programming language you use. However, there are some official clients for the most popular languages available, and if you use Python then the [Python Qdrant client library](https://github.com/qdrant/qdrant_client) might be the best choice."
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"For the purposes of this exercise we need to prepare a couple of things:\n",
"\n",
"1. Qdrant server instance. In our case a local Docker container.\n",
"2. The [qdrant-client](https://github.com/qdrant/qdrant_client) library to interact with the vector database.\n",
"3. An [OpenAI API key](https://platform.openai.com/settings/organization/api-keys).\n",
"\n",
"### Start Qdrant server\n",
"\n",
"We're going to use a local Qdrant instance running in a Docker container. The easiest way to launch it is to use the attached [docker-compose.yaml] file and run the following command:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:04:28.198036Z",
"start_time": "2023-02-16T12:04:26.950014Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/0\n",
" \u001b[32m✔\u001b[0m Container qdrant-qdrant-1 \u001b[32mRunning\u001b[0m \u001b[34m0.0s \u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"! docker compose up -d"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We might validate if the server was launched successfully by running a simple curl command:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:04:29.301651Z",
"start_time": "2023-02-16T12:04:29.173153Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"title\":\"qdrant - vector search engine\",\"version\":\"1.3.0\"}"
]
}
],
"source": [
"! curl http://localhost:6333"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install requirements\n",
"\n",
"This notebook obviously requires the `openai` and `qdrant-client` packages, but there are also some other additional libraries we will use. The following command installs them all:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.718972Z",
"start_time": "2023-02-16T12:04:30.434820Z"
}
},
"outputs": [],
"source": [
"! pip install openai qdrant-client pandas wget"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare your OpenAI API key\n",
"\n",
"The OpenAI API key is used for vectorization of the documents and queries.\n",
"\n",
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
"\n",
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by running following command:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"! export OPENAI_API_KEY=\"your API key\""
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:05.730338Z",
"start_time": "2023-02-16T12:05:05.723351Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"OPENAI_API_KEY is ready\n"
]
}
],
"source": [
"# Test that your OpenAI API key is correctly set as an environment variable\n",
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
"import os\n",
"\n",
"# Note. alternatively you can set a temporary env variable like this:\n",
"# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n",
"\n",
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
" print(\"OPENAI_API_KEY is ready\")\n",
"else:\n",
" print(\"OPENAI_API_KEY environment variable not found\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Connect to Qdrant\n",
"\n",
"Connecting to a running instance of Qdrant server is easy with the official Python library:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.827143Z",
"start_time": "2023-02-16T12:05:05.733771Z"
}
},
"outputs": [],
"source": [
"import qdrant_client\n",
"\n",
"client = qdrant_client.QdrantClient(\n",
" host=\"localhost\",\n",
" prefer_grpc=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can test the connection by running any available method:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:06.848488Z",
"start_time": "2023-02-16T12:05:06.832612Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CollectionsResponse(collections=[])"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.get_collections()\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:05:37.371951Z",
"start_time": "2023-02-16T12:05:06.851634Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100% [......................................................................] 698933052 / 698933052"
]
},
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded (9).zip'"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The downloaded file has to be then extracted:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:06:01.538851Z",
"start_time": "2023-02-16T12:05:37.376042Z"
}
},
"outputs": [],
"source": [
"import zipfile\n",
"\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we can finally load it from the provided CSV file:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:17:35.483972Z",
"start_time": "2023-02-16T12:06:01.540172Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"from ast import literal_eval\n",
"\n",
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n",
"# Read vectors from strings back into a list\n",
"article_df[\"title_vector\"] = article_df.title_vector.apply(literal_eval)\n",
"article_df[\"content_vector\"] = article_df.content_vector.apply(literal_eval)\n",
"article_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Index data\n",
"\n",
"Qdrant stores data in __collections__ where each object is described by at least one vector and may contain an additional metadata called __payload__. Our collection will be called **Articles** and each object will be described by both **title** and **content** vectors. Qdrant does not require you to set up any kind of schema beforehand, so you can freely put points to the collection with a simple setup only.\n",
"\n",
"We will start with creating a collection, and then we will fill it with our precomputed embeddings."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:17:36.366066Z",
"start_time": "2023-02-16T12:17:35.486872Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from qdrant_client.http import models as rest\n",
"\n",
"vector_size = len(article_df[\"content_vector\"][0])\n",
"\n",
"client.create_collection(\n",
" collection_name=\"Articles\",\n",
" vectors_config={\n",
" \"title\": rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" \"content\": rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" }\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:37.518210Z",
"start_time": "2023-02-16T12:17:36.368564Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"UpdateResult(operation_id=0, status=<UpdateStatus.COMPLETED: 'completed'>)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.upsert(\n",
" collection_name=\"Articles\",\n",
" points=[\n",
" rest.PointStruct(\n",
" id=k,\n",
" vector={\n",
" \"title\": v[\"title_vector\"],\n",
" \"content\": v[\"content_vector\"],\n",
" },\n",
" payload=v.to_dict(),\n",
" )\n",
" for k, v in article_df.iterrows()\n",
" ],\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.675202Z",
"start_time": "2023-02-16T12:30:40.655654Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CountResult(count=25000)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Check the collection size to make sure all the points have been stored\n",
"client.count(collection_name=\"Articles\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Search data\n",
"\n",
"Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-ada-002` OpenAI model we also have to use it during search.\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:38.024370Z",
"start_time": "2023-02-16T12:30:37.712816Z"
}
},
"outputs": [],
"source": [
"from openai import OpenAI\n",
"\n",
"openai_client = OpenAI()\n",
"\n",
"def query_qdrant(query, collection_name, vector_name=\"title\", top_k=20):\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai_client.embeddings.create(\n",
" input=query,\n",
" model=\"text-embedding-ada-002\",\n",
" ).data[0].embedding\n",
"\n",
" query_results = client.search(\n",
" collection_name=collection_name,\n",
" query_vector=(\n",
" vector_name, embedded_query\n",
" ),\n",
" limit=top_k,\n",
" )\n",
"\n",
" return query_results"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:39.379566Z",
"start_time": "2023-02-16T12:30:38.031041Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art (Score: 0.875)\n",
"2. Western Europe (Score: 0.867)\n",
"3. Renaissance art (Score: 0.864)\n",
"4. Pop art (Score: 0.86)\n",
"5. Northern Europe (Score: 0.855)\n",
"6. Hellenistic art (Score: 0.853)\n",
"7. Modernist literature (Score: 0.847)\n",
"8. Art film (Score: 0.843)\n",
"9. Central Europe (Score: 0.843)\n",
"10. European (Score: 0.841)\n",
"11. Art (Score: 0.841)\n",
"12. Byzantine art (Score: 0.841)\n",
"13. Postmodernism (Score: 0.84)\n",
"14. Eastern Europe (Score: 0.839)\n",
"15. Cubism (Score: 0.839)\n",
"16. Europe (Score: 0.839)\n",
"17. Impressionism (Score: 0.838)\n",
"18. Bauhaus (Score: 0.838)\n",
"19. Surrealism (Score: 0.837)\n",
"20. Expressionism (Score: 0.837)\n"
]
}
],
"source": [
"query_results = query_qdrant(\"modern art in Europe\", \"Articles\")\n",
"for i, article in enumerate(query_results):\n",
" print(f\"{i + 1}. {article.payload['title']} (Score: {round(article.score, 3)})\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-16T12:30:40.652676Z",
"start_time": "2023-02-16T12:30:39.382555Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn (Score: 0.869)\n",
"2. Wars of Scottish Independence (Score: 0.861)\n",
"3. 1651 (Score: 0.852)\n",
"4. First War of Scottish Independence (Score: 0.85)\n",
"5. Robert I of Scotland (Score: 0.846)\n",
"6. 841 (Score: 0.844)\n",
"7. 1716 (Score: 0.844)\n",
"8. 1314 (Score: 0.837)\n",
"9. 1263 (Score: 0.836)\n",
"10. William Wallace (Score: 0.835)\n",
"11. Stirling (Score: 0.831)\n",
"12. 1306 (Score: 0.831)\n",
"13. 1746 (Score: 0.83)\n",
"14. 1040s (Score: 0.828)\n",
"15. 1106 (Score: 0.827)\n",
"16. 1304 (Score: 0.826)\n",
"17. David II of Scotland (Score: 0.825)\n",
"18. Braveheart (Score: 0.824)\n",
"19. 1124 (Score: 0.824)\n",
"20. Second War of Scottish Independence (Score: 0.823)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_results = query_qdrant(\"Famous battles in Scottish history\", \"Articles\", \"content\")\n",
"for i, article in enumerate(query_results):\n",
" print(f\"{i + 1}. {article.payload['title']} (Score: {round(article.score, 3)})\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb
+561
View File
File diff suppressed because one or more lines are too long
examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb
+790
View File
@@ -0,0 +1,790 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "cb1537e6",
"metadata": {},
"source": [
"# Using Qdrant for Embeddings Search\n",
"\n",
"This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
"\n",
"### What is a Vector Database\n",
"\n",
"A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
"\n",
"### Why use a Vector Database\n",
"\n",
"Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
"\n",
"\n",
"### Demo Flow\n",
"The demo flow is:\n",
"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **Qdrant**\n",
" - *Setup*: Here we'll set up the Python client for Qdrant. For more details go [here](https://github.com/qdrant/qdrant_client)\n",
" - *Index Data*: We'll create a collection with vectors for __titles__ and __content__\n",
" - *Search Data*: We'll run a few searches to confirm it works\n",
"\n",
"Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
]
},
{
"cell_type": "markdown",
"id": "e2b59250",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Import the required libraries and set the embedding model that we'd like to use."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d8810f9",
"metadata": {},
"outputs": [],
"source": [
"# We'll need to install Qdrant client\n",
"!pip install qdrant-client"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "5be94df6",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:49:06.926613Z",
"start_time": "2024-05-21T23:49:06.923221Z"
}
},
"outputs": [],
"source": [
"import openai\n",
"import pandas as pd\n",
"from ast import literal_eval\n",
"import qdrant_client # Qdrant's client library for Python\n",
"\n",
"# This can be changed to the embedding model of your choice. Make sure its the same model that is used for generating embeddings\n",
"EMBEDDING_MODEL = \"text-embedding-ada-002\"\n",
"\n",
"# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
"import warnings\n",
"\n",
"warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
"warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
]
},
{
"cell_type": "markdown",
"id": "e5d9d2e1",
"metadata": {},
"source": [
"## Load data\n",
"\n",
"In this section we'll load embedded data that we've prepared previous to this session."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "5dff8b55",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:49:54.889503Z",
"start_time": "2024-05-21T23:49:41.132888Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100% [......................................................................] 698933052 / 698933052"
]
},
{
"data": {
"text/plain": [
"'vector_database_wikipedia_articles_embedded (10).zip'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import wget\n",
"\n",
"embeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n",
"\n",
"# The file is ~700 MB so this will take some time\n",
"wget.download(embeddings_url)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "21097972",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:50:56.268540Z",
"start_time": "2024-05-21T23:50:53.171125Z"
}
},
"outputs": [],
"source": [
"import zipfile\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
" zip_ref.extractall(\"../data\")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "70bbd8ba",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:51:08.388674Z",
"start_time": "2024-05-21T23:50:57.592940Z"
}
},
"outputs": [],
"source": [
"article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "1721e45d",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:51:13.706819Z",
"start_time": "2024-05-21T23:51:13.700231Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>url</th>\n",
" <th>title</th>\n",
" <th>text</th>\n",
" <th>title_vector</th>\n",
" <th>content_vector</th>\n",
" <th>vector_id</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>https://simple.wikipedia.org/wiki/April</td>\n",
" <td>April</td>\n",
" <td>April is the fourth month of the year in the J...</td>\n",
" <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
" <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>https://simple.wikipedia.org/wiki/August</td>\n",
" <td>August</td>\n",
" <td>August (Aug.) is the eighth month of the year ...</td>\n",
" <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
" <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>6</td>\n",
" <td>https://simple.wikipedia.org/wiki/Art</td>\n",
" <td>Art</td>\n",
" <td>Art is a creative activity that expresses imag...</td>\n",
" <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
" <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
" <td>2</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>8</td>\n",
" <td>https://simple.wikipedia.org/wiki/A</td>\n",
" <td>A</td>\n",
" <td>A or a is the first letter of the English alph...</td>\n",
" <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
" <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>9</td>\n",
" <td>https://simple.wikipedia.org/wiki/Air</td>\n",
" <td>Air</td>\n",
" <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
" <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
" <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id url title \\\n",
"0 1 https://simple.wikipedia.org/wiki/April April \n",
"1 2 https://simple.wikipedia.org/wiki/August August \n",
"2 6 https://simple.wikipedia.org/wiki/Art Art \n",
"3 8 https://simple.wikipedia.org/wiki/A A \n",
"4 9 https://simple.wikipedia.org/wiki/Air Air \n",
"\n",
" text \\\n",
"0 April is the fourth month of the year in the J... \n",
"1 August (Aug.) is the eighth month of the year ... \n",
"2 Art is a creative activity that expresses imag... \n",
"3 A or a is the first letter of the English alph... \n",
"4 Air refers to the Earth's atmosphere. Air is a... \n",
"\n",
" title_vector \\\n",
"0 [0.001009464613161981, -0.020700545981526375, ... \n",
"1 [0.0009286514250561595, 0.000820168002974242, ... \n",
"2 [0.003393713850528002, 0.0061537534929811954, ... \n",
"3 [0.0153952119871974, -0.013759135268628597, 0.... \n",
"4 [0.02224554680287838, -0.02044147066771984, -0... \n",
"\n",
" content_vector vector_id \n",
"0 [-0.011253940872848034, -0.013491976074874401,... 0 \n",
"1 [0.0003609954728744924, 0.007262262050062418, ... 1 \n",
"2 [-0.004959689453244209, 0.015772193670272827, ... 2 \n",
"3 [0.024894846603274345, -0.022186409682035446, ... 3 \n",
"4 [0.021524671465158463, 0.018522677943110466, -... 4 "
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"article_df.head()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "960b82af",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:20.588010Z",
"start_time": "2024-05-21T23:51:16.274336Z"
}
},
"outputs": [],
"source": [
"# Read vectors from strings back into a list\n",
"article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
"article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
"\n",
"# Set vector_id to be a string\n",
"article_df['vector_id'] = article_df['vector_id'].apply(str)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "a334ab8b",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:36.075327Z",
"start_time": "2024-05-21T23:55:36.038710Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 25000 entries, 0 to 24999\n",
"Data columns (total 7 columns):\n",
" # Column Non-Null Count Dtype \n",
"--- ------ -------------- ----- \n",
" 0 id 25000 non-null int64 \n",
" 1 url 25000 non-null object\n",
" 2 title 25000 non-null object\n",
" 3 text 25000 non-null object\n",
" 4 title_vector 25000 non-null object\n",
" 5 content_vector 25000 non-null object\n",
" 6 vector_id 25000 non-null object\n",
"dtypes: int64(1), object(6)\n",
"memory usage: 1.3+ MB\n"
]
}
],
"source": [
"article_df.info(show_counts=True)"
]
},
{
"cell_type": "markdown",
"id": "9cfaed9d",
"metadata": {},
"source": [
"## Qdrant\n",
"\n",
"**[Qdrant](https://qdrant.tech/)**. is a high-performant vector search database written in Rust. It offers both on-premise and cloud version, but for the purposes of that example we're going to use the local deployment mode.\n",
"\n",
"Setting everything up will require:\n",
"- Spinning up a local instance of Qdrant\n",
"- Configuring the collection and storing the data in it\n",
"- Trying out with some queries"
]
},
{
"cell_type": "markdown",
"id": "38774565",
"metadata": {},
"source": [
"### Setup\n",
"\n",
"For the local deployment, we are going to use Docker, according to the Qdrant documentation: https://qdrant.tech/documentation/quick_start/. Qdrant requires just a single container, but an example of the docker-compose.yaml file is available at `./qdrant/docker-compose.yaml` in this repo.\n",
"\n",
"You can start Qdrant instance locally by navigating to this directory and running `docker-compose up -d `\n",
"\n",
"> You might need to increase the memory limit for Docker to 8GB or more. Or Qdrant might fail to execute with an error message like `7 Killed`.\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "fc9a0203-30c2-4a7c-95cb-c5bacc908a4d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 1/0\n",
" \u001b[32m✔\u001b[0m Container qdrant-qdrant-1 \u001b[32mRunning\u001b[0m \u001b[34m0.0s \u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"! docker compose up -d"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "76d697e9",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:56.550765Z",
"start_time": "2024-05-21T23:55:56.517724Z"
}
},
"outputs": [],
"source": [
"qdrant = qdrant_client.QdrantClient(host=\"localhost\", port=6333)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "1deeb539",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:55:57.340006Z",
"start_time": "2024-05-21T23:55:57.312830Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CollectionsResponse(collections=[CollectionDescription(name='Articles')])"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"qdrant.get_collections()"
]
},
{
"cell_type": "markdown",
"id": "bc006b6f",
"metadata": {},
"source": [
"### Index data\n",
"\n",
"Qdrant stores data in __collections__ where each object is described by at least one vector and may contain an additional metadata called __payload__. Our collection will be called **Articles** and each object will be described by both **title** and **content** vectors.\n",
"\n",
"We'll be using an official [qdrant-client](https://github.com/qdrant/qdrant_client) package that has all the utility methods already built-in."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "1a84ee1d",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:56:04.066640Z",
"start_time": "2024-05-21T23:56:04.064878Z"
}
},
"outputs": [],
"source": [
"from qdrant_client.http import models as rest"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "00876f92",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:56:05.462165Z",
"start_time": "2024-05-21T23:56:05.247948Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get the vector size from the first row to set up the collection\n",
"vector_size = len(article_df['content_vector'][0])\n",
"\n",
"# Set up the collection with the vector configuration. You need to declare the vector size and distance metric for the collection. Distance metric enables vector database to index and search vectors efficiently.\n",
"qdrant.recreate_collection(\n",
" collection_name='Articles',\n",
" vectors_config={\n",
" 'title': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" 'content': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" }\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "9f39a8c395554ca3",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:56:21.577594Z",
"start_time": "2024-05-21T23:56:21.460740Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vector_size = len(article_df['content_vector'][0])\n",
"\n",
"qdrant.recreate_collection(\n",
" collection_name='Articles',\n",
" vectors_config={\n",
" 'title': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" 'content': rest.VectorParams(\n",
" distance=rest.Distance.COSINE,\n",
" size=vector_size,\n",
" ),\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"id": "e95be6e0c9af4c21",
"metadata": {},
"source": [
"In addition to the vector configuration defined under `vector`, we can also define the `payload` configuration. Payload is an optional field that allows you to store additional metadata alongside the vectors. In our case, we'll store the `id`, `title`, and `url` of the articles. As we return the title of nearest articles in the search results from payload, we can also provide the user with the URL to the article (which is part of the meta-data)."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "f24e76ab",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:25.183855Z",
"start_time": "2024-05-21T23:56:50.664145Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Upserting articles: 100%|█████████████████████████████████████████████████████████████████████████████████████| 25000/25000 [02:52<00:00, 144.82it/s]\n"
]
}
],
"source": [
"from qdrant_client.models import PointStruct # Import the PointStruct to store the vector and payload\n",
"from tqdm import tqdm # Library to show the progress bar \n",
"\n",
"# Populate collection with vectors using tqdm to show progress\n",
"for k, v in tqdm(article_df.iterrows(), desc=\"Upserting articles\", total=len(article_df)):\n",
" try:\n",
" qdrant.upsert(\n",
" collection_name='Articles',\n",
" points=[\n",
" PointStruct(\n",
" id=k,\n",
" vector={'title': v['title_vector'], \n",
" 'content': v['content_vector']},\n",
" payload={\n",
" 'id': v['id'],\n",
" 'title': v['title'],\n",
" 'url': v['url']\n",
" }\n",
" )\n",
" ]\n",
" )\n",
" except Exception as e:\n",
" print(f\"Failed to upsert row {k}: {v}\")\n",
" print(f\"Exception: {e}\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "d1188a12",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:27.558407Z",
"start_time": "2024-05-21T23:58:27.549740Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"CountResult(count=25000)"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Check the collection size to make sure all the points have been stored\n",
"qdrant.count(collection_name='Articles')"
]
},
{
"cell_type": "markdown",
"id": "06ed119b",
"metadata": {},
"source": [
"### Search Data\n",
"\n",
"Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Ensure you use the text-embedding-ada-002 model as the original embeddings in file were created with this model."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "f1bac4ef",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:35.492725Z",
"start_time": "2024-05-21T23:58:35.488963Z"
}
},
"outputs": [],
"source": [
"def query_qdrant(query, collection_name, vector_name='title', top_k=20):\n",
"\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.embeddings.create(\n",
" input=query,\n",
" model=EMBEDDING_MODEL,\n",
" ).data[0].embedding # We take the first embedding from the list\n",
" \n",
" query_results = qdrant.search(\n",
" collection_name=collection_name,\n",
" query_vector=(\n",
" vector_name, embedded_query\n",
" ),\n",
" limit=top_k, \n",
" query_filter=None\n",
" )\n",
" \n",
" return query_results"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "aa92f3d3",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:37.183718Z",
"start_time": "2024-05-21T23:58:36.949491Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Museum of Modern Art, URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art (Score: 0.875)\n",
"2. Western Europe, URL: https://simple.wikipedia.org/wiki/Western%20Europe (Score: 0.867)\n",
"3. Renaissance art, URL: https://simple.wikipedia.org/wiki/Renaissance%20art (Score: 0.864)\n",
"4. Pop art, URL: https://simple.wikipedia.org/wiki/Pop%20art (Score: 0.86)\n",
"5. Northern Europe, URL: https://simple.wikipedia.org/wiki/Northern%20Europe (Score: 0.855)\n",
"6. Hellenistic art, URL: https://simple.wikipedia.org/wiki/Hellenistic%20art (Score: 0.853)\n",
"7. Modernist literature, URL: https://simple.wikipedia.org/wiki/Modernist%20literature (Score: 0.847)\n",
"8. Art film, URL: https://simple.wikipedia.org/wiki/Art%20film (Score: 0.843)\n",
"9. Central Europe, URL: https://simple.wikipedia.org/wiki/Central%20Europe (Score: 0.843)\n",
"10. European, URL: https://simple.wikipedia.org/wiki/European (Score: 0.841)\n",
"11. Art, URL: https://simple.wikipedia.org/wiki/Art (Score: 0.841)\n",
"12. Byzantine art, URL: https://simple.wikipedia.org/wiki/Byzantine%20art (Score: 0.841)\n",
"13. Postmodernism, URL: https://simple.wikipedia.org/wiki/Postmodernism (Score: 0.84)\n",
"14. Eastern Europe, URL: https://simple.wikipedia.org/wiki/Eastern%20Europe (Score: 0.839)\n",
"15. Cubism, URL: https://simple.wikipedia.org/wiki/Cubism (Score: 0.839)\n",
"16. Europe, URL: https://simple.wikipedia.org/wiki/Europe (Score: 0.839)\n",
"17. Impressionism, URL: https://simple.wikipedia.org/wiki/Impressionism (Score: 0.838)\n",
"18. Bauhaus, URL: https://simple.wikipedia.org/wiki/Bauhaus (Score: 0.838)\n",
"19. Surrealism, URL: https://simple.wikipedia.org/wiki/Surrealism (Score: 0.837)\n",
"20. Expressionism, URL: https://simple.wikipedia.org/wiki/Expressionism (Score: 0.837)\n"
]
}
],
"source": [
"query_results = query_qdrant('modern art in Europe', 'Articles', 'title')\n",
"for i, article in enumerate(query_results):\n",
" print(f'{i + 1}. {article.payload[\"title\"]}, URL: {article.payload[\"url\"]} (Score: {round(article.score, 3)})')"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "7ed116b8",
"metadata": {
"ExecuteTime": {
"end_time": "2024-05-21T23:58:53.144123Z",
"start_time": "2024-05-21T23:58:52.924091Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. Battle of Bannockburn, URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn (Score: 0.869)\n",
"2. Wars of Scottish Independence, URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence (Score: 0.861)\n",
"3. 1651, URL: https://simple.wikipedia.org/wiki/1651 (Score: 0.852)\n",
"4. First War of Scottish Independence, URL: https://simple.wikipedia.org/wiki/First%20War%20of%20Scottish%20Independence (Score: 0.85)\n",
"5. Robert I of Scotland, URL: https://simple.wikipedia.org/wiki/Robert%20I%20of%20Scotland (Score: 0.846)\n",
"6. 841, URL: https://simple.wikipedia.org/wiki/841 (Score: 0.844)\n",
"7. 1716, URL: https://simple.wikipedia.org/wiki/1716 (Score: 0.844)\n",
"8. 1314, URL: https://simple.wikipedia.org/wiki/1314 (Score: 0.837)\n",
"9. 1263, URL: https://simple.wikipedia.org/wiki/1263 (Score: 0.836)\n",
"10. William Wallace, URL: https://simple.wikipedia.org/wiki/William%20Wallace (Score: 0.835)\n",
"11. Stirling, URL: https://simple.wikipedia.org/wiki/Stirling (Score: 0.831)\n",
"12. 1306, URL: https://simple.wikipedia.org/wiki/1306 (Score: 0.831)\n",
"13. 1746, URL: https://simple.wikipedia.org/wiki/1746 (Score: 0.83)\n",
"14. 1040s, URL: https://simple.wikipedia.org/wiki/1040s (Score: 0.828)\n",
"15. 1106, URL: https://simple.wikipedia.org/wiki/1106 (Score: 0.827)\n",
"16. 1304, URL: https://simple.wikipedia.org/wiki/1304 (Score: 0.826)\n",
"17. David II of Scotland, URL: https://simple.wikipedia.org/wiki/David%20II%20of%20Scotland (Score: 0.825)\n",
"18. Braveheart, URL: https://simple.wikipedia.org/wiki/Braveheart (Score: 0.824)\n",
"19. 1124, URL: https://simple.wikipedia.org/wiki/1124 (Score: 0.824)\n",
"20. July 27, URL: https://simple.wikipedia.org/wiki/July%2027 (Score: 0.823)\n"
]
}
],
"source": [
"# This time we'll query using content vector\n",
"query_results = query_qdrant('Famous battles in Scottish history', 'Articles', 'content')\n",
"for i, article in enumerate(query_results):\n",
" print(f'{i + 1}. {article.payload[\"title\"]}, URL: {article.payload[\"url\"]} (Score: {round(article.score, 3)})')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
},
"vscode": {
"interpreter": {
"hash": "fd16a328ca3d68029457069b79cb0b38eb39a0f5ccc4fe4473d3047707df8207"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
examples/vector_databases/qdrant/docker-compose.yaml
+8
View File
@@ -0,0 +1,8 @@
version: '3.4'
services:
qdrant:
image: qdrant/qdrant:v1.3.0
restart: on-failure
ports:
- "6333:6333"
- "6334:6334"