wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
756 lines
26 KiB
Plaintext
756 lines
26 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "016b5598",
|
||
"metadata": {},
|
||
"source": [
|
||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/vector_stores/DragonflyIndexDemo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "0b692c73",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Dragonfly and Vector Store"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "1e7787c2",
|
||
"metadata": {},
|
||
"source": [
|
||
"In this notebook we are going to show a quick demo of using the Dragonfly with Vector Store."
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "c479ce87",
|
||
"metadata": {},
|
||
"source": [
|
||
"If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "1730d643",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install -U llama-index llama-index-vector-stores-redis llama-index-embeddings-cohere llama-index-embeddings-openai"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "47264e32",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import os\n",
|
||
"import getpass\n",
|
||
"import sys\n",
|
||
"import logging\n",
|
||
"import textwrap\n",
|
||
"import warnings\n",
|
||
"\n",
|
||
"warnings.filterwarnings(\"ignore\")\n",
|
||
"\n",
|
||
"logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
|
||
"\n",
|
||
"from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n",
|
||
"from llama_index.vector_stores.redis import RedisVectorStore"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "3c692310",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Start Dragonfly\n",
|
||
"\n",
|
||
"The easiest way to start Dragonfly is using the Dragonfly docker image or\n",
|
||
"quickly signing up for a [Dragonfly Cloud](https://www.dragonflydb.io/cloud) demo instance.\n",
|
||
"\n",
|
||
"To follow every step of this tutorial, launch the image as follows:\n",
|
||
"\n",
|
||
"```bash\n",
|
||
"docker run -d -p 6379:6379 --name dragonfly docker.dragonflydb.io/dragonflydb/dragonfly\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "f9b97a89",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Setup OpenAI\n",
|
||
"Lets first begin by adding the openai api key. This will allow us to access openai for embeddings and to use chatgpt."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "0c9f4d21-145a-401e-95ff-ccb259e8ef84",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"oai_api_key = getpass.getpass(\"OpenAI API Key:\")\n",
|
||
"os.environ[\"OPENAI_API_KEY\"] = oai_api_key"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "103ff054",
|
||
"metadata": {},
|
||
"source": [
|
||
"Download Data"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "304ad9d8",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"--2025-06-30 14:41:20-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
|
||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.110.133, ...\n",
|
||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.\n",
|
||
"HTTP request sent, awaiting response... 200 OK\n",
|
||
"Length: 75042 (73K) [text/plain]\n",
|
||
"Saving to: ‘data/paul_graham/paul_graham_essay.txt’\n",
|
||
"\n",
|
||
"data/paul_graham/pa 100%[===================>] 73.28K --.-KB/s in 0.04s \n",
|
||
"\n",
|
||
"2025-06-30 14:41:20 (2.00 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!mkdir -p 'data/paul_graham/'\n",
|
||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "59ff935d",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Read in a dataset\n",
|
||
"Here we will use a set of Paul Graham essays to provide the text to turn into embeddings, store in a vector store and query to find context for our LLM QnA loop."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "68cbd239-880e-41a3-98d8-dbb3fab55431",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Document ID: a5cae17c-27eb-411e-8967-fb6ef98bcdcf Document Filename: paul_graham_essay.txt\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# load documents\n",
|
||
"documents = SimpleDirectoryReader(\"./data/paul_graham\").load_data()\n",
|
||
"print(\n",
|
||
" \"Document ID:\",\n",
|
||
" documents[0].id_,\n",
|
||
" \"Document Filename:\",\n",
|
||
" documents[0].metadata[\"file_name\"],\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "dd270925",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Initialize the default vector store\n",
|
||
"\n",
|
||
"Now we have our documents prepared, we can initialize the vector store with **default** settings. This will allow us to store our vectors in Dragonfly and create an index for real-time search."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "ba1558b3",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:41:29 llama_index.vector_stores.redis.base INFO Using default RedisVectorStore schema.\n",
|
||
"14:41:31 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
|
||
"14:41:31 llama_index.vector_stores.redis.base INFO Added 22 documents to index llama_index\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from llama_index.core import StorageContext\n",
|
||
"from redis import Redis\n",
|
||
"\n",
|
||
"# create a client connection\n",
|
||
"redis_client = Redis.from_url(\"redis://localhost:6379\")\n",
|
||
"\n",
|
||
"# create the vector store wrapper\n",
|
||
"vector_store = RedisVectorStore(redis_client=redis_client, overwrite=True)\n",
|
||
"\n",
|
||
"# load storage context\n",
|
||
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
||
"\n",
|
||
"# build and load index from documents and storage context\n",
|
||
"index = VectorStoreIndex.from_documents(\n",
|
||
" documents, storage_context=storage_context\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "dc00b3fb",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Query the default vector store\n",
|
||
"\n",
|
||
"Now that we have our data stored in the index, we can ask questions against the index.\n",
|
||
"\n",
|
||
"The index will use the data as the knowledge base for an LLM. The default setting for as_query_engine() utilizes OpenAI embeddings and GPT as the language model. Therefore, an OpenAI key is required unless you opt for a customized or local language model.\n",
|
||
"\n",
|
||
"Below we will test searches against out index and then full RAG with an LLM."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "c50a593f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"query_engine = index.as_query_engine()\n",
|
||
"retriever = index.as_retriever()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "e3f0daf7",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:41:40 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
|
||
"14:41:40 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2\n",
|
||
"14:41:40 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185', 'llama_index/vector_a67c3af9-14cc-45fd-a2dd-142753a61d79']\n",
|
||
"Node ID: f12d31cc-d154-4ae2-9511-81a1e0b2c185\n",
|
||
"Text: What I Worked On February 2021 Before college the two main\n",
|
||
"things I worked on, outside of school, were writing and programming. I\n",
|
||
"didn't write essays. I wrote what beginning writers were supposed to\n",
|
||
"write then, and probably still are: short stories. My stories were\n",
|
||
"awful. They had hardly any plot, just characters with strong feelings,\n",
|
||
"which I ...\n",
|
||
"Score: 0.819\n",
|
||
"\n",
|
||
"Node ID: a67c3af9-14cc-45fd-a2dd-142753a61d79\n",
|
||
"Text: In the summer of 2016 we moved to England. We wanted our kids to\n",
|
||
"see what it was like living in another country, and since I was a\n",
|
||
"British citizen by birth, that seemed the obvious choice. We only\n",
|
||
"meant to stay for a year, but we liked it so much that we still live\n",
|
||
"there. So most of Bel was written in England. In the fall of 2019,\n",
|
||
"Bel was final...\n",
|
||
"Score: 0.815\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"result_nodes = retriever.retrieve(\"What did the author learn?\")\n",
|
||
"for node in result_nodes:\n",
|
||
" print(node)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "e13d7726",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:41:44 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
|
||
"14:41:44 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2\n",
|
||
"14:41:44 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185', 'llama_index/vector_a67c3af9-14cc-45fd-a2dd-142753a61d79']\n",
|
||
"14:41:45 httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
|
||
"The author learned that philosophy courses in college were boring to him, leading him to switch his\n",
|
||
"focus to studying AI.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"response = query_engine.query(\"What did the author learn?\")\n",
|
||
"print(textwrap.fill(str(response), 100))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "4b99b79b",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:41:47 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
|
||
"14:41:47 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2\n",
|
||
"14:41:47 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_8c02f420-3cfc-4da6-859b-97469872ef46', 'llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185']\n",
|
||
"Node ID: 8c02f420-3cfc-4da6-859b-97469872ef46\n",
|
||
"Text: HN was no doubt good for YC, but it was also by far the biggest\n",
|
||
"source of stress for me. If all I'd had to do was select and help\n",
|
||
"founders, life would have been so easy. And that implies that HN was a\n",
|
||
"mistake. Surely the biggest source of stress in one's work should at\n",
|
||
"least be something close to the core of the work. Whereas I was like\n",
|
||
"someone ...\n",
|
||
"Score: 0.804\n",
|
||
"\n",
|
||
"Node ID: f12d31cc-d154-4ae2-9511-81a1e0b2c185\n",
|
||
"Text: What I Worked On February 2021 Before college the two main\n",
|
||
"things I worked on, outside of school, were writing and programming. I\n",
|
||
"didn't write essays. I wrote what beginning writers were supposed to\n",
|
||
"write then, and probably still are: short stories. My stories were\n",
|
||
"awful. They had hardly any plot, just characters with strong feelings,\n",
|
||
"which I ...\n",
|
||
"Score: 0.802\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"result_nodes = retriever.retrieve(\"What was a hard moment for the author?\")\n",
|
||
"for node in result_nodes:\n",
|
||
" print(node)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "c0838ee1",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:41:51 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
|
||
"14:41:51 llama_index.vector_stores.redis.base INFO Querying index llama_index with query *=>[KNN 2 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 2\n",
|
||
"14:41:51 llama_index.vector_stores.redis.base INFO Found 2 results for query with id ['llama_index/vector_8c02f420-3cfc-4da6-859b-97469872ef46', 'llama_index/vector_f12d31cc-d154-4ae2-9511-81a1e0b2c185']\n",
|
||
"14:41:52 httpx INFO HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
|
||
"Dealing with urgent problems related to Hacker News (HN) was a significant source of stress for the\n",
|
||
"author.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"response = query_engine.query(\"What was a hard moment for the author?\")\n",
|
||
"print(textwrap.fill(str(response), 100))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "ba33eb01",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:41:55 llama_index.vector_stores.redis.base INFO Deleting index llama_index\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"index.vector_store.delete_index()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "831452c8",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Use a custom index schema\n",
|
||
"\n",
|
||
"In most use cases, you need the ability to customize the underling index configuration\n",
|
||
"and specification. For example, this is handy in order to define specific metadata filters you wish to enable.\n",
|
||
"\n",
|
||
"With Dragonfly, this is as simple as defining an index schema object\n",
|
||
"(from file or dict) and passing it through to the vector store client wrapper.\n",
|
||
"\n",
|
||
"For this example, we will:\n",
|
||
"1. switch the embedding model to [Cohere](https://cohere.com/)\n",
|
||
"2. add an additional metadata field for the document `updated_at` timestamp\n",
|
||
"3. index the existing `file_name` metadata field"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2022e92a",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.core.settings import Settings\n",
|
||
"from llama_index.embeddings.cohere import CohereEmbedding\n",
|
||
"\n",
|
||
"# set up Cohere Key\n",
|
||
"co_api_key = getpass.getpass(\"Cohere API Key:\")\n",
|
||
"\n",
|
||
"Settings.embed_model = CohereEmbedding(api_key=co_api_key)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "c07e9747",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from redisvl.schema import IndexSchema\n",
|
||
"\n",
|
||
"\n",
|
||
"custom_schema = IndexSchema.from_dict(\n",
|
||
" {\n",
|
||
" # customize basic index specs\n",
|
||
" \"index\": {\n",
|
||
" \"name\": \"paul_graham\",\n",
|
||
" \"prefix\": \"essay\",\n",
|
||
" \"key_separator\": \":\",\n",
|
||
" },\n",
|
||
" # customize fields that are indexed\n",
|
||
" \"fields\": [\n",
|
||
" # required fields for llamaindex\n",
|
||
" {\"type\": \"tag\", \"name\": \"id\"},\n",
|
||
" {\"type\": \"tag\", \"name\": \"doc_id\"},\n",
|
||
" {\"type\": \"text\", \"name\": \"text\"},\n",
|
||
" # custom metadata fields\n",
|
||
" {\"type\": \"numeric\", \"name\": \"updated_at\"},\n",
|
||
" {\"type\": \"tag\", \"name\": \"file_name\"},\n",
|
||
" # custom vector field definition for cohere embeddings\n",
|
||
" {\n",
|
||
" \"type\": \"vector\",\n",
|
||
" \"name\": \"vector\",\n",
|
||
" \"attrs\": {\n",
|
||
" \"dims\": 1024,\n",
|
||
" \"algorithm\": \"hnsw\",\n",
|
||
" \"distance_metric\": \"cosine\",\n",
|
||
" },\n",
|
||
" },\n",
|
||
" ],\n",
|
||
" }\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "22184dd0",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"IndexInfo(name='paul_graham', prefix='essay', key_separator=':', storage_type=<StorageType.HASH: 'hash'>)"
|
||
]
|
||
},
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"custom_schema.index"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2bf50ab5",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"{'id': TagField(name='id', type=<FieldTypes.TAG: 'tag'>, path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),\n",
|
||
" 'doc_id': TagField(name='doc_id', type=<FieldTypes.TAG: 'tag'>, path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),\n",
|
||
" 'text': TextField(name='text', type=<FieldTypes.TEXT: 'text'>, path=None, attrs=TextFieldAttributes(sortable=False, weight=1, no_stem=False, withsuffixtrie=False, phonetic_matcher=None)),\n",
|
||
" 'updated_at': NumericField(name='updated_at', type=<FieldTypes.NUMERIC: 'numeric'>, path=None, attrs=NumericFieldAttributes(sortable=False)),\n",
|
||
" 'file_name': TagField(name='file_name', type=<FieldTypes.TAG: 'tag'>, path=None, attrs=TagFieldAttributes(sortable=False, separator=',', case_sensitive=False, withsuffixtrie=False)),\n",
|
||
" 'vector': HNSWVectorField(name='vector', type='vector', path=None, attrs=HNSWVectorFieldAttributes(dims=1024, algorithm=<VectorIndexAlgorithm.HNSW: 'HNSW'>, datatype=<VectorDataType.FLOAT32: 'FLOAT32'>, distance_metric=<VectorDistanceMetric.COSINE: 'COSINE'>, initial_cap=None, m=16, ef_construction=200, ef_runtime=10, epsilon=0.01))}"
|
||
]
|
||
},
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"custom_schema.fields"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "61b01276",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from datetime import datetime\n",
|
||
"\n",
|
||
"\n",
|
||
"def date_to_timestamp(date_string: str) -> int:\n",
|
||
" date_format: str = \"%Y-%m-%d\"\n",
|
||
" return int(datetime.strptime(date_string, date_format).timestamp())\n",
|
||
"\n",
|
||
"\n",
|
||
"# iterate through documents and add new field\n",
|
||
"for document in documents:\n",
|
||
" document.metadata[\"updated_at\"] = date_to_timestamp(\n",
|
||
" document.metadata[\"last_modified_date\"]\n",
|
||
" )"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "e871823e",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:42:26 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed \"HTTP/1.1 200 OK\"\n",
|
||
"14:42:26 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed \"HTTP/1.1 200 OK\"\n",
|
||
"14:42:27 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed \"HTTP/1.1 200 OK\"\n",
|
||
"14:42:27 llama_index.vector_stores.redis.base INFO Added 22 documents to index paul_graham\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"vector_store = RedisVectorStore(\n",
|
||
" schema=custom_schema, # provide customized schema\n",
|
||
" redis_client=redis_client,\n",
|
||
" overwrite=True,\n",
|
||
")\n",
|
||
"\n",
|
||
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
||
"\n",
|
||
"# build and load index from documents and storage context\n",
|
||
"index = VectorStoreIndex.from_documents(\n",
|
||
" documents, storage_context=storage_context\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3791a32c",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Query the vector store and filter on metadata\n",
|
||
"Now that we have additional metadata indexed in Dragonfly, let's try some queries with filters."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "bb2c21ad",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.core.vector_stores import (\n",
|
||
" MetadataFilters,\n",
|
||
" MetadataFilter,\n",
|
||
" ExactMatchFilter,\n",
|
||
")\n",
|
||
"\n",
|
||
"retriever = index.as_retriever(\n",
|
||
" similarity_top_k=3,\n",
|
||
" filters=MetadataFilters(\n",
|
||
" filters=[\n",
|
||
" ExactMatchFilter(key=\"file_name\", value=\"paul_graham_essay.txt\"),\n",
|
||
" MetadataFilter(\n",
|
||
" key=\"updated_at\",\n",
|
||
" value=date_to_timestamp(\"2023-01-01\"),\n",
|
||
" operator=\">=\",\n",
|
||
" ),\n",
|
||
" MetadataFilter(\n",
|
||
" key=\"text\",\n",
|
||
" value=\"learn\",\n",
|
||
" operator=\"text_match\",\n",
|
||
" ),\n",
|
||
" ],\n",
|
||
" condition=\"and\",\n",
|
||
" ),\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "d136cfb3",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"14:42:37 httpx INFO HTTP Request: POST https://api.cohere.com/v2/embed \"HTTP/1.1 200 OK\"\n",
|
||
"14:42:37 llama_index.vector_stores.redis.base INFO Querying index paul_graham with query ((@file_name:{paul_graham_essay\\.txt} @updated_at:[1672524000 +inf]) @text:(learn))=>[KNN 3 @vector $vector AS vector_distance] RETURN 5 id doc_id text _node_content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 3\n",
|
||
"14:42:37 llama_index.vector_stores.redis.base INFO Found 3 results for query with id ['essay:30148f62-13c6-4edb-b09f-1cf3054c5c98', 'essay:054f9488-83c7-4bf6-a408-9ef17eea0446', 'essay:608adb71-a995-489d-81dc-0deab7bbe656']\n",
|
||
"Node ID: 30148f62-13c6-4edb-b09f-1cf3054c5c98\n",
|
||
"Text: If he even knew about the strange classes I was taking, he never\n",
|
||
"said anything. So now I was in a PhD program in computer science, yet\n",
|
||
"planning to be an artist, yet also genuinely in love with Lisp hacking\n",
|
||
"and working away at On Lisp. In other words, like many a grad student,\n",
|
||
"I was working energetically on multiple projects that were not my\n",
|
||
"the...\n",
|
||
"Score: 0.404\n",
|
||
"\n",
|
||
"Node ID: 054f9488-83c7-4bf6-a408-9ef17eea0446\n",
|
||
"Text: I wanted to go back to RISD, but I was now broke and RISD was\n",
|
||
"very expensive, so I decided to get a job for a year and then return\n",
|
||
"to RISD the next fall. I got one at a company called Interleaf, which\n",
|
||
"made software for creating documents. You mean like Microsoft Word?\n",
|
||
"Exactly. That was how I learned that low end software tends to eat\n",
|
||
"high end so...\n",
|
||
"Score: 0.396\n",
|
||
"\n",
|
||
"Node ID: 608adb71-a995-489d-81dc-0deab7bbe656\n",
|
||
"Text: All that seemed left for philosophy were edge cases that people\n",
|
||
"in other fields felt could safely be ignored. I couldn't have put\n",
|
||
"this into words when I was 18. All I knew at the time was that I kept\n",
|
||
"taking philosophy courses and they kept being boring. So I decided to\n",
|
||
"switch to AI. AI was in the air in the mid 1980s, but there were two\n",
|
||
"things...\n",
|
||
"Score: 0.394\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"result_nodes = retriever.retrieve(\"What did the author learn?\")\n",
|
||
"\n",
|
||
"for node in result_nodes:\n",
|
||
" print(node)"
|
||
]
|
||
},
|
||
{
|
||
"attachments": {},
|
||
"cell_type": "markdown",
|
||
"id": "52b975a7",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Deleting documents or index completely\n",
|
||
"\n",
|
||
"Sometimes it may be useful to delete documents or the entire index. This can be done using the `delete` and `delete_index` methods."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "6fe322f7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"document_id = documents[0].doc_id\n",
|
||
"document_id"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "0ce45788",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"Number of documents before deleting\", redis_client.dbsize())\n",
|
||
"vector_store.delete(document_id)\n",
|
||
"print(\"Number of documents after deleting\", redis_client.dbsize())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "442e8acf",
|
||
"metadata": {},
|
||
"source": [
|
||
"However, the index still exists (with no associated documents)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "12eda458",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"vector_store.index_exists()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "c380605a",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# now lets delete the index entirely\n",
|
||
"# this will delete all the documents and the index\n",
|
||
"vector_store.delete_index()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "474ad4ee",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(\"Number of documents after deleting\", redis_client.dbsize())"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|