wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
437 lines
13 KiB
Plaintext
437 lines
13 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/vector_stores/MilvusOperatorFunctionDemo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Milvus Vector Store - Metadata Filter\n",
|
|
"\n",
|
|
"This notebook illustrates the use of the Milvus vector store in LlamaIndex, focusing on metadata filtering capabilities. You will learn how to index documents with metadata, perform vector searches with LlamaIndex's built-in metadata filters, and apply Milvus's native filtering expressions to the vector store.\n",
|
|
"\n",
|
|
"By the end of this notebook, you will understand how to utilize Milvus's filtering features to narrow down search results based on document metadata."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prerequisites\n",
|
|
"\n",
|
|
"**Install dependencies**\n",
|
|
"\n",
|
|
"Before getting started, make sure you have the following dependencies installed:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"! pip install llama-index-vector-stores-milvus llama-index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> If you're using Google Colab, you may need to **restart the runtime** (Navigate to the \"Runtime\" menu at the top of the interface, and select \"Restart session\" from the dropdown menu.)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Set up accounts**\n",
|
|
"\n",
|
|
"This tutorial uses OpenAI for text embeddings and answer generation. You need to prepare the [OpenAI API key](https://platform.openai.com/api-keys). "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import openai\n",
|
|
"\n",
|
|
"openai.api_key = \"sk-\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"To use the Milvus vector store, specify your Milvus server `URI` (and optionally with the `TOKEN`). To start a Milvus server, you can set up a Milvus server by following the [Milvus installation guide](https://milvus.io/docs/install-overview.md) or simply trying [Zilliz Cloud](https://docs.zilliz.com/docs/register-with-zilliz-cloud) for free."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"URI = \"./milvus_filter_demo.db\" # Use Milvus-Lite for demo purpose\n",
|
|
"# TOKEN = \"\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Prepare data**\n",
|
|
"\n",
|
|
"For this example, we'll use a few books with similar or identical titles but different metadata (author, genre, and publication year) as the sample data. This will help demonstrate how Milvus can filter and retrieve documents based on both vector similarity and metadata attributes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.schema import TextNode\n",
|
|
"\n",
|
|
"nodes = [\n",
|
|
" TextNode(\n",
|
|
" text=\"Life: A User's Manual\",\n",
|
|
" metadata={\n",
|
|
" \"author\": \"Georges Perec\",\n",
|
|
" \"genre\": \"Postmodern Fiction\",\n",
|
|
" \"year\": 1978,\n",
|
|
" },\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" text=\"Life and Fate\",\n",
|
|
" metadata={\n",
|
|
" \"author\": \"Vasily Grossman\",\n",
|
|
" \"genre\": \"Historical Fiction\",\n",
|
|
" \"year\": 1980,\n",
|
|
" },\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" text=\"Life\",\n",
|
|
" metadata={\n",
|
|
" \"author\": \"Keith Richards\",\n",
|
|
" \"genre\": \"Memoir\",\n",
|
|
" \"year\": 2010,\n",
|
|
" },\n",
|
|
" ),\n",
|
|
" TextNode(\n",
|
|
" text=\"The Life\",\n",
|
|
" metadata={\n",
|
|
" \"author\": \"Malcolm Knox\",\n",
|
|
" \"genre\": \"Literary Fiction\",\n",
|
|
" \"year\": 2011,\n",
|
|
" },\n",
|
|
" ),\n",
|
|
"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Build Index\n",
|
|
"\n",
|
|
"In this section, we will store sample data in Milvus using the default embedding model (OpenAI's `text-embedding-ada-002`). Titles will be converted into text embeddings and stored in a dense embedding field, while all metadata will be stored in scalar fields."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2025-04-22 08:31:09,871 [DEBUG][_create_connection]: Created new connection using: 19675caa8f894772b3db175b65d0063a (async_milvus_client.py:547)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from llama_index.vector_stores.milvus import MilvusVectorStore\n",
|
|
"from llama_index.core import StorageContext, VectorStoreIndex\n",
|
|
"\n",
|
|
"\n",
|
|
"vector_store = MilvusVectorStore(\n",
|
|
" uri=URI,\n",
|
|
" # token=TOKEN,\n",
|
|
" collection_name=\"test_filter_collection\", # Change collection name here\n",
|
|
" dim=1536, # Vector dimension depends on the embedding model\n",
|
|
" overwrite=True, # Drop collection if exists\n",
|
|
")\n",
|
|
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
|
"index = VectorStoreIndex(nodes, storage_context=storage_context)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Metadata Filters\n",
|
|
"\n",
|
|
"In this section, we will apply LlamaIndex's built-in metadata filters and conditions to Milvus search."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Define metadata filters**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.vector_stores import (\n",
|
|
" MetadataFilter,\n",
|
|
" MetadataFilters,\n",
|
|
" FilterOperator,\n",
|
|
")\n",
|
|
"\n",
|
|
"filters = MetadataFilters(\n",
|
|
" filters=[\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"year\", value=2000, operator=FilterOperator.GT\n",
|
|
" ) # year > 2000\n",
|
|
" ]\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Retrieve from vector store with filters**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The Life\n",
|
|
"{'author': 'Malcolm Knox', 'genre': 'Literary Fiction', 'year': 2011}\n",
|
|
"\n",
|
|
"\n",
|
|
"Life\n",
|
|
"{'author': 'Keith Richards', 'genre': 'Memoir', 'year': 2010}\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"retriever = index.as_retriever(filters=filters, similarity_top_k=5)\n",
|
|
"result_nodes = retriever.retrieve(\"Books about life\")\n",
|
|
"for node in result_nodes:\n",
|
|
" print(node.text)\n",
|
|
" print(node.metadata)\n",
|
|
" print(\"\\n\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Multiple Metdata Filters\n",
|
|
"\n",
|
|
"You can also combine multiple metadata filters to create more complex queries. LlamaIndex supports both `AND` and `OR` conditions to combine filters. This allows for more precise and flexible retrieval of documents based on their metadata attributes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Condition `AND`**\n",
|
|
"\n",
|
|
"Try an example filtering for books published between 1979 and 2010 (specifically, where 1979 < year ≤ 2010):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Life and Fate\n",
|
|
"{'author': 'Vasily Grossman', 'genre': 'Historical Fiction', 'year': 1980}\n",
|
|
"\n",
|
|
"\n",
|
|
"Life\n",
|
|
"{'author': 'Keith Richards', 'genre': 'Memoir', 'year': 2010}\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from llama_index.core.vector_stores import FilterCondition\n",
|
|
"\n",
|
|
"filters = MetadataFilters(\n",
|
|
" filters=[\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"year\", value=1979, operator=FilterOperator.GT\n",
|
|
" ), # year > 1979\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"year\", value=2010, operator=FilterOperator.LTE\n",
|
|
" ), # year <= 2010\n",
|
|
" ],\n",
|
|
" condition=FilterCondition.AND,\n",
|
|
")\n",
|
|
"\n",
|
|
"retriever = index.as_retriever(filters=filters, similarity_top_k=5)\n",
|
|
"result_nodes = retriever.retrieve(\"Books about life\")\n",
|
|
"for node in result_nodes:\n",
|
|
" print(node.text)\n",
|
|
" print(node.metadata)\n",
|
|
" print(\"\\n\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Condition `OR`**\n",
|
|
"\n",
|
|
"Try another example that filters books written by either Georges Perec or Keith Richards:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Life\n",
|
|
"{'author': 'Keith Richards', 'genre': 'Memoir', 'year': 2010}\n",
|
|
"\n",
|
|
"\n",
|
|
"Life: A User's Manual\n",
|
|
"{'author': 'Georges Perec', 'genre': 'Postmodern Fiction', 'year': 1978}\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"filters = MetadataFilters(\n",
|
|
" filters=[\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"author\", value=\"Georges Perec\", operator=FilterOperator.EQ\n",
|
|
" ), # author is Georges Perec\n",
|
|
" MetadataFilter(\n",
|
|
" key=\"author\", value=\"Keith Richards\", operator=FilterOperator.EQ\n",
|
|
" ), # author is Keith Richards\n",
|
|
" ],\n",
|
|
" condition=FilterCondition.OR,\n",
|
|
")\n",
|
|
"\n",
|
|
"retriever = index.as_retriever(filters=filters, similarity_top_k=5)\n",
|
|
"result_nodes = retriever.retrieve(\"Books about life\")\n",
|
|
"for node in result_nodes:\n",
|
|
" print(node.text)\n",
|
|
" print(node.metadata)\n",
|
|
" print(\"\\n\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Use Milvus's Keyword Arguments\n",
|
|
"\n",
|
|
"In addition to the built-in filtering capabilities, you can use Milvus's native filtering expressions by the `string_expr` keyword argument. This allows you to pass specific filter expressions directly to Milvus during search operations, extending beyond the standard metadata filtering to access Milvus's advanced filtering capabilities.\n",
|
|
"\n",
|
|
"Milvus provides powerful and flexible filtering options that enable precise querying of your vector data:\n",
|
|
"\n",
|
|
"- Basic Operators: Comparison operators, range filters, arithmetic operators, and logical operators\n",
|
|
"- Filter Expression Templates: Predefined patterns for common filtering scenarios\n",
|
|
"- Specialized Operators: Data type-specific operators for JSON or array fields\n",
|
|
"\n",
|
|
"For comprehensive documentation and examples of Milvus filtering expressions, refer to the official documentation of [Milvus Filtering](https://milvus.io/docs/boolean.md)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The Life\n",
|
|
"{'author': 'Malcolm Knox', 'genre': 'Literary Fiction', 'year': 2011}\n",
|
|
"\n",
|
|
"\n",
|
|
"Life and Fate\n",
|
|
"{'author': 'Vasily Grossman', 'genre': 'Historical Fiction', 'year': 1980}\n",
|
|
"\n",
|
|
"\n",
|
|
"Life: A User's Manual\n",
|
|
"{'author': 'Georges Perec', 'genre': 'Postmodern Fiction', 'year': 1978}\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"retriever = index.as_retriever(\n",
|
|
" vector_store_kwargs={\n",
|
|
" \"string_expr\": \"genre like '%Fiction'\",\n",
|
|
" },\n",
|
|
" similarity_top_k=5,\n",
|
|
")\n",
|
|
"result_nodes = retriever.retrieve(\"Books about life\")\n",
|
|
"for node in result_nodes:\n",
|
|
" print(node.text)\n",
|
|
" print(node.metadata)\n",
|
|
" print(\"\\n\")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|