wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
531 lines
19 KiB
Plaintext
531 lines
19 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "1496f9de",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/vector_stores/MilvusIndexDemo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "0b692c73",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Milvus Vector Store\n",
|
|
"\n",
|
|
"This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LlamaIndex and Milvus.\n",
|
|
"\n",
|
|
"The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using a vector similarity search engine like Milvus, and then uses a generative model to generate new text based on the retrieved documents.\n",
|
|
"\n",
|
|
"[Milvus](https://milvus.io/) is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.\n",
|
|
"\n",
|
|
"In this notebook we are going to show a quick demo of using the MilvusVectorStore."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "f81e2c81",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Before you begin\n",
|
|
"\n",
|
|
"### Install dependencies"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0d0e46d8",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "3e0c18ca",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-vector-stores-milvus"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6b80700a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eab0d1a3",
|
|
"metadata": {},
|
|
"source": [
|
|
"This notebook will use [Milvus Lite](https://milvus.io/docs/milvus_lite.md) requiring a higher version of pymilvus:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7661b098",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install pymilvus>=2.4.2"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "70cc8c56",
|
|
"metadata": {},
|
|
"source": [
|
|
"> If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (click on the \"Runtime\" menu at the top of the screen, and select \"Restart session\" from the dropdown menu)."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "f9b97a89",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Setup OpenAI\n",
|
|
"\n",
|
|
"Lets first begin by adding the openai api key. This will allow us to access chatgpt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0c9f4d21-145a-401e-95ff-ccb259e8ef84",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import openai\n",
|
|
"\n",
|
|
"openai.api_key = \"sk-***********\""
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "a3d4e638",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Prepare data\n",
|
|
"\n",
|
|
"You can download sample data with the following commands:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2a2e24d1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"! mkdir -p \"data/\"\n",
|
|
"! wget \"https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt\" -O \"data/paul_graham_essay.txt\"\n",
|
|
"! wget \"https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/10k/uber_2021.pdf\" -O \"data/uber_2021.pdf\""
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "59ff935d",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Getting Started\n",
|
|
"\n",
|
|
"### Generate our data\n",
|
|
"As a first example, lets generate a document from the file `paul_graham_essay.txt`. It is a single essay from Paul Graham titled `What I Worked On`. To generate the documents we will use the SimpleDirectoryReader."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "68cbd239-880e-41a3-98d8-dbb3fab55431",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Document ID: 95f25e4d-f270-4650-87ce-006d69d82033\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from llama_index.core import SimpleDirectoryReader\n",
|
|
"\n",
|
|
"# load documents\n",
|
|
"documents = SimpleDirectoryReader(\n",
|
|
" input_files=[\"./data/paul_graham_essay.txt\"]\n",
|
|
").load_data()\n",
|
|
"\n",
|
|
"print(\"Document ID:\", documents[0].doc_id)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "dd270925",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Create an index across the data\n",
|
|
"\n",
|
|
"Now that we have a document, we can can create an index and insert the document. For the index we will use a MilvusVectorStore. MilvusVectorStore takes in a few arguments:\n",
|
|
"\n",
|
|
"**basic args**\n",
|
|
"- `uri (str, optional)`: The URI to connect to, comes in the form of \"https://address:port\" for Milvus or Zilliz Cloud service, or \"path/to/local/milvus.db\" for the lite local Milvus. Defaults to \"./milvus_llamaindex.db\".\n",
|
|
"- `token (str, optional)`: The token for log in. Empty if not using rbac, if using rbac it will most likely be \"username:password\".\n",
|
|
"- `collection_name (str, optional)`: The name of the collection where data will be stored. Defaults to \"llamalection\".\n",
|
|
"- `overwrite (bool, optional)`: Whether to overwrite existing collection with the same name. Defaults to False.\n",
|
|
"\n",
|
|
"**scalar fields including doc id & text**\n",
|
|
"- `doc_id_field (str, optional)`: The name of the doc_id field for the collection. Defaults to DEFAULT_DOC_ID_KEY.\n",
|
|
"- `text_key (str, optional)`: What key text is stored in in the passed collection. Used when bringing your own collection. Defaults to DEFAULT_TEXT_KEY.\n",
|
|
"- `scalar_field_names (list, optional)`: The names of the extra scalar fields to be included in the collection schema.\n",
|
|
"- `scalar_field_types (list, optional)`: The types of the extra scalar fields.\n",
|
|
"\n",
|
|
"**dense field**\n",
|
|
"- `enable_dense (bool)`: A boolean flag to enable or disable dense embedding. Defaults to True.\n",
|
|
"- `dim (int, optional)`: The dimension of the embedding vectors for the collection. Required when creating a new collection with enable_sparse is False.\n",
|
|
"- `embedding_field (str, optional)`: The name of the dense embedding field for the collection, defaults to DEFAULT_EMBEDDING_KEY.\n",
|
|
"- `index_config (dict, optional)`: The configuration used for building the dense embedding index. Defaults to None.\n",
|
|
"- `search_config (dict, optional)`: The configuration used for searching the Milvus dense index. Note that this must be compatible with the index type specified by `index_config`. Defaults to None.\n",
|
|
"- `similarity_metric (str, optional)`: The similarity metric to use for dense embedding, currently supports IP, COSINE and L2.\n",
|
|
"\n",
|
|
"**sparse field**\n",
|
|
"- `enable_sparse (bool)`: A boolean flag to enable or disable sparse embedding. Defaults to False.\n",
|
|
"- `sparse_embedding_field (str)`: The name of sparse embedding field, defaults to DEFAULT_SPARSE_EMBEDDING_KEY.\n",
|
|
"- `sparse_embedding_function (Union[BaseSparseEmbeddingFunction, BaseMilvusBuiltInFunction], optional)`: If enable_sparse is True, this object should be provided to convert text to a sparse embedding. If None, the default sparse embedding function (BM25BuiltInFunction) will be used.\n",
|
|
"- `sparse_index_config (dict, optional)`: The configuration used to build the sparse embedding index. Defaults to None.\n",
|
|
"\n",
|
|
"**hybrid ranker**\n",
|
|
"- `hybrid_ranker (str)`: Specifies the type of ranker used in hybrid search queries. Currently only supports [\"RRFRanker\", \"WeightedRanker\"]. Defaults to \"RRFRanker\".\n",
|
|
"- `hybrid_ranker_params (dict, optional)`: Configuration parameters for the hybrid ranker. The structure of this dictionary depends on the specific ranker being used:\n",
|
|
" - For \"RRFRanker\", it should include:\n",
|
|
" - \"k\" (int): A parameter used in Reciprocal Rank Fusion (RRF). This value is used to calculate the rank scores as part of the RRF algorithm, which combines multiple ranking strategies into a single score to improve search relevance.\n",
|
|
" - For \"WeightedRanker\", it expects:\n",
|
|
" - \"weights\" (list of float): A list of exactly two weights:\n",
|
|
" 1. The weight for the dense embedding component.\n",
|
|
" 2. The weight for the sparse embedding component.\n",
|
|
" These weights are used to adjust the importance of the dense and sparse components of the embeddings in the hybrid retrieval process.\n",
|
|
" Defaults to an empty dictionary, implying that the ranker will operate with its predefined default settings.\n",
|
|
"\n",
|
|
"**others**\n",
|
|
"- `collection_properties (dict, optional)`: The collection properties such as TTL (Time-To-Live) and MMAP (memory mapping). Defaults to None. It could include:\n",
|
|
" - \"collection.ttl.seconds\" (int): Once this property is set, data in the current collection expires in the specified time. Expired data in the collection will be cleaned up and will not be involved in searches or queries.\n",
|
|
" - \"mmap.enabled\" (bool): Whether to enable memory-mapped storage at the collection level.\n",
|
|
"- `index_management (IndexManagement)`: Specifies the index management strategy to use. Defaults to \"create_if_not_exists\".\n",
|
|
"- `batch_size (int)`: Configures the number of documents processed in one batch when inserting data into Milvus. Defaults to DEFAULT_BATCH_SIZE.\n",
|
|
"- `consistency_level (str, optional)`: Which consistency level to use for a newly created collection. Defaults to \"Session\".\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ba1558b3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create an index over the documents\n",
|
|
"from llama_index.core import VectorStoreIndex, StorageContext\n",
|
|
"from llama_index.vector_stores.milvus import MilvusVectorStore\n",
|
|
"\n",
|
|
"\n",
|
|
"vector_store = MilvusVectorStore(\n",
|
|
" uri=\"./milvus_demo.db\", dim=1536, overwrite=True\n",
|
|
")\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" documents, storage_context=storage_context\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a75a5773",
|
|
"metadata": {},
|
|
"source": [
|
|
"> For the parameters of `MilvusVectorStore`:\n",
|
|
"> - Setting the `uri` as a local file, e.g.`./milvus.db`, is the most convenient method, as it automatically utilizes [Milvus Lite](https://milvus.io/docs/milvus_lite.md) to store all data in this file.\n",
|
|
"> - If you have large scale of data, you can set up a more performant Milvus server on [docker or kubernetes](https://milvus.io/docs/quickstart.md). In this setup, please use the server uri, e.g.`http://localhost:19530`, as your `uri`.\n",
|
|
"> - If you want to use [Zilliz Cloud](https://zilliz.com/cloud), the fully managed cloud service for Milvus, adjust the `uri` and `token`, which correspond to the [Public Endpoint and Api key](https://docs.zilliz.com/docs/on-zilliz-cloud-console#free-cluster-details) in Zilliz Cloud."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "04304299-fc3e-40a0-8600-f50c3292767e",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Query the data\n",
|
|
"Now that we have our document stored in the index, we can ask questions against the index. The index will use the data stored in itself as the knowledge base for chatgpt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "35369eda",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The author learned that philosophy courses in college were boring to him, leading him to switch his focus to studying AI.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"query_engine = index.as_query_engine()\n",
|
|
"res = query_engine.query(\"What did the author learn?\")\n",
|
|
"print(res)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "99212d33",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The disease posed challenges for the author as it affected his mother's health, leading to a stroke caused by colon cancer. This resulted in her losing her balance and needing to be placed in a nursing home. The author and his sister were determined to help their mother get out of the nursing home and back to her house.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"res = query_engine.query(\n",
|
|
" \"What challenges did the disease pose for the author?\"\n",
|
|
")\n",
|
|
"print(res)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "64cc925b",
|
|
"metadata": {},
|
|
"source": [
|
|
"This next test shows that overwriting removes the previous data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8d641e24",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The author is the individual who created the context information.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from llama_index.core import Document\n",
|
|
"\n",
|
|
"\n",
|
|
"vector_store = MilvusVectorStore(\n",
|
|
" uri=\"./milvus_demo.db\", dim=1536, overwrite=True\n",
|
|
")\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" [Document(text=\"The number that is being searched for is ten.\")],\n",
|
|
" storage_context,\n",
|
|
")\n",
|
|
"query_engine = index.as_query_engine()\n",
|
|
"res = query_engine.query(\"Who is the author?\")\n",
|
|
"print(res)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "d8123529",
|
|
"metadata": {},
|
|
"source": [
|
|
"The next test shows adding additional data to an already existing index."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a5c429a4",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The number is ten.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"del index, vector_store, storage_context, query_engine\n",
|
|
"\n",
|
|
"vector_store = MilvusVectorStore(uri=\"./milvus_demo.db\", overwrite=False)\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" documents, storage_context=storage_context\n",
|
|
")\n",
|
|
"query_engine = index.as_query_engine()\n",
|
|
"res = query_engine.query(\"What is the number?\")\n",
|
|
"print(res)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "56ac3375-371b-4e5f-bac9-8124b6871429",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Paul Graham\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"res = query_engine.query(\"Who is the author?\")\n",
|
|
"print(res)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "154ec1b3",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Metadata filtering\n",
|
|
"\n",
|
|
"We can generate results by filtering specific sources. The following example illustrates loading all documents from the directory and subsequently filtering them based on metadata."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2a845c5d-f10b-4fba-9cd2-e62871f836f3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.vector_stores import ExactMatchFilter, MetadataFilters\n",
|
|
"\n",
|
|
"# Load all the two documents loaded before\n",
|
|
"documents_all = SimpleDirectoryReader(\"./data/\").load_data()\n",
|
|
"\n",
|
|
"vector_store = MilvusVectorStore(\n",
|
|
" uri=\"./milvus_demo.db\", dim=1536, overwrite=True\n",
|
|
")\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
|
"index = VectorStoreIndex.from_documents(documents_all, storage_context)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8de85343",
|
|
"metadata": {},
|
|
"source": [
|
|
"We want to only retrieve documents from the file `uber_2021.pdf`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d9f9bcb5-43de-4983-b754-a822ac7b5278",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The disease posed challenges related to the adverse impact on the business and operations, including reduced demand for Mobility offerings globally, affecting travel behavior and demand. Additionally, the pandemic led to driver supply constraints, impacted by concerns regarding COVID-19, with uncertainties about when supply levels would return to normal. The rise of the Omicron variant further affected travel, resulting in advisories and restrictions that could adversely impact both driver supply and consumer demand for Mobility offerings.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"filters = MetadataFilters(\n",
|
|
" filters=[ExactMatchFilter(key=\"file_name\", value=\"uber_2021.pdf\")]\n",
|
|
")\n",
|
|
"query_engine = index.as_query_engine(filters=filters)\n",
|
|
"res = query_engine.query(\n",
|
|
" \"What challenges did the disease pose for the author?\"\n",
|
|
")\n",
|
|
"\n",
|
|
"print(res)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "92e95db9",
|
|
"metadata": {},
|
|
"source": [
|
|
"We get a different result this time when retrieve from the file `paul_graham_essay.txt`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0f134a35-dbd3-49d8-b7d8-48bdd2349701",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The disease posed challenges for the author as it affected his mother's health, leading to a stroke caused by colon cancer. This resulted in his mother losing her balance and needing to be placed in a nursing home. The author and his sister were determined to help their mother get out of the nursing home and back to her house.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"filters = MetadataFilters(\n",
|
|
" filters=[ExactMatchFilter(key=\"file_name\", value=\"paul_graham_essay.txt\")]\n",
|
|
")\n",
|
|
"query_engine = index.as_query_engine(filters=filters)\n",
|
|
"res = query_engine.query(\n",
|
|
" \"What challenges did the disease pose for the author?\"\n",
|
|
")\n",
|
|
"\n",
|
|
"print(res)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|