run-llama--llama_index/docs/examples/vector_stores/MilvusFullTextSearchDemo.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1496f9de",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/vector_stores/MilvusFullTextSearchDemo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0b692c73",
   "metadata": {},
   "source": [
    "# Milvus Vector Store with Full-Text Search\n",
    "\n",
    "**Full-text search** uses exact keyword matching, often leveraging algorithms like BM25 to rank documents by relevance. In **Retrieval-Augmented Generation (RAG)** systems, this method retrieves pertinent text to enhance AI-generated responses.\n",
    "\n",
    "Meanwhile, **semantic search** interprets contextual meaning to provide broader results. Combining both approaches creates a **hybrid search** that improves information retrieval—especially in cases where a single method falls short.\n",
    "\n",
    "With [Milvus 2.5](https://milvus.io/blog/introduce-milvus-2-5-full-text-search-powerful-metadata-filtering-and-more.md)'s Sparse-BM25 approach, raw text is automatically converted into sparse vectors. This eliminates the need for manual sparse embedding generation and enables a hybrid search strategy that balances semantic understanding with keyword relevance.\n",
    "\n",
    "In this tutorial, you'll learn how to use LlamaIndex and Milvus to build a RAG system using full-text search and hybrid search. We'll start by implementing full-text search alone and then enhance it by integrating semantic search for more comprehensive results.\n",
    "\n",
    "> Before proceeding with this tutorial, ensure you are familiar with [full-text search](https://milvus.io/docs/full-text-search.md#Full-Text-Search) and the [basics of using Milvus in LlamaIndex](https://milvus.io/docs/integrate_with_llamaindex.md)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f81e2c81",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "**Install dependencies**\n",
    "\n",
    "Before getting started, make sure you have the following dependencies installed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e0c18ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-vector-stores-milvus\n",
    "%pip install llama-index-embeddings-openai\n",
    "%pip install llama-index-llms-openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f04f0ae",
   "metadata": {},
   "source": [
    "> If you're using Google Colab, you may need to **restart the runtime** (Navigate to the \"Runtime\" menu at the top of the interface, and select \"Restart session\" from the dropdown menu.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08d58733",
   "metadata": {},
   "source": [
    "**Set up accounts**\n",
    "\n",
    "This tutorial uses OpenAI for text embeddings and answer generation. You need to prepare the [OpenAI API key](https://platform.openai.com/api-keys). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b2d300",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "openai.api_key = \"sk-\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c4f445b",
   "metadata": {},
   "source": [
    "To use the Milvus vector store, specify your Milvus server `URI` (and optionally with the `TOKEN`). To start a Milvus server, you can set up a Milvus server by following the [Milvus installation guide](https://milvus.io/docs/install-overview.md) or simply trying [Zilliz Cloud](https://docs.zilliz.com/docs/register-with-zilliz-cloud) for free.\n",
    "\n",
    "> Full-text search is currently supported in Milvus Standalone, Milvus Distributed, and Zilliz Cloud, but not yet in Milvus Lite (planned for future implementation). Reach out support@zilliz.com for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cb18406",
   "metadata": {},
   "outputs": [],
   "source": [
    "URI = \"http://localhost:19530\"\n",
    "# TOKEN = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f5889ef",
   "metadata": {},
   "source": [
    "**Download example data**\n",
    "\n",
    "Run the following commands to download sample documents into the \"data/paul_graham\" directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c76c9a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2025-03-27 07:49:01--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 75042 (73K) [text/plain]\n",
      "Saving to: ‘data/paul_graham/paul_graham_essay.txt’\n",
      "\n",
      "data/paul_graham/pa 100%[===================>]  73.28K  --.-KB/s    in 0.07s   \n",
      "\n",
      "2025-03-27 07:49:01 (1.01 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%mkdir -p 'data/paul_graham/'\n",
    "%wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "630ed34a",
   "metadata": {},
   "source": [
    "## RAG with Full-Text Search\n",
    "\n",
    "Integrating full-text search into a RAG system balances semantic search with precise and predictable keyword-based retrieval. You can also choose to only use full text search though it's recommended to combine full text search with semantic search for better search results. Here for demonstration purpose we will show full text search alone and hybrid search."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92cdd451",
   "metadata": {},
   "source": [
    "To get started, use `SimpleDirectoryReaderLoad` to load the essay \"What I Worked On\" by Paul Graham:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfcc6479",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example document:\n",
      " Doc ID: 16b7942f-bf1a-4197-85e1-f31d51ea25a9\n",
      "Text: What I Worked On  February 2021  Before college the two main\n",
      "things I worked on, outside of school, were writing and programming. I\n",
      "didn't write essays. I wrote what beginning writers were supposed to\n",
      "write then, and probably still are: short stories. My stories were\n",
      "awful. They had hardly any plot, just characters with strong feelings,\n",
      "which I ...\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core import SimpleDirectoryReader\n",
    "\n",
    "documents = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()\n",
    "\n",
    "# Let's take a look at the first document\n",
    "print(\"Example document:\\n\", documents[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "856a4043",
   "metadata": {},
   "source": [
    "### Full-Text Search with BM25\n",
    "\n",
    "LlamaIndex's `MilvusVectorStore` supports full-text search, enabling efficient keyword-based retrieval. By using a built-in function as the `sparse_embedding_function`, it applies BM25 scoring to rank search results.\n",
    "\n",
    "In this section, we’ll demonstrate how to implement a RAG system using BM25 for full-text search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb19b205",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Embeddings have been explicitly disabled. Using MockEmbedding.\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core import VectorStoreIndex, StorageContext\n",
    "from llama_index.vector_stores.milvus import MilvusVectorStore\n",
    "from llama_index.vector_stores.milvus.utils import BM25BuiltInFunction\n",
    "from llama_index.core import Settings\n",
    "\n",
    "# Skip dense embedding model\n",
    "Settings.embed_model = None\n",
    "\n",
    "# Build Milvus vector store creating a new collection\n",
    "vector_store = MilvusVectorStore(\n",
    "    uri=URI,\n",
    "    # token=TOKEN,\n",
    "    enable_dense=False,\n",
    "    enable_sparse=True,  # Only enable sparse to demo full text search\n",
    "    sparse_embedding_function=BM25BuiltInFunction(),\n",
    "    overwrite=True,\n",
    ")\n",
    "\n",
    "# Store documents in Milvus\n",
    "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
    "index = VectorStoreIndex.from_documents(\n",
    "    documents, storage_context=storage_context\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47b9072c",
   "metadata": {},
   "source": [
    "The above code inserts example documents into Milvus and builds an index to enable BM25 ranking for full-text search. It disables dense embedding and utilizes `BM25BuiltInFunction` with default parameters.\n",
    "\n",
    "You can specify the input and output fields in the `BM25BuiltInFunction` parameters:\n",
    "\n",
    "- `input_field_names (str)`: The input text field (default: \"text\"). It indicates which text field the BM25 algorithm applied to. Change this if using your own collection with a different text field name.\n",
    "- `output_field_names (str)`: The field where outputs of this BM25 function are stored (default: \"sparse_embedding\").\n",
    "\n",
    "Once the vector store is set up, you can perform full-text search queries using Milvus with query mode \"sparse\" or \"text_search\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "929d26ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author learned several important lessons at Viaweb. They learned about the importance of growth\n",
      "rate as the ultimate test of a startup, the value of building stores for users to understand retail\n",
      "and software usability, and the significance of being the \"entry level\" option in a market.\n",
      "Additionally, they discovered the accidental success of making Viaweb inexpensive, the challenges of\n",
      "hiring too many people, and the relief felt when the company was acquired by Yahoo.\n"
     ]
    }
   ],
   "source": [
    "import textwrap\n",
    "\n",
    "query_engine = index.as_query_engine(\n",
    "    vector_store_query_mode=\"sparse\", similarity_top_k=5\n",
    ")\n",
    "answer = query_engine.query(\"What did the author learn at Viaweb?\")\n",
    "print(textwrap.fill(str(answer), 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92373153",
   "metadata": {},
   "source": [
    "#### Customize text analyzer\n",
    "\n",
    "Analyzers play a vital role in full-text search by breaking sentences into tokens and performing lexical processing, such as stemming and stop-word removal. They are typically language-specific. For more details, refer to [Milvus Analyzer Guide](https://milvus.io/docs/analyzer-overview.md#Analyzer-Overview).\n",
    "\n",
    "Milvus supports two types of analyzers: **Built-in Analyzers** and **Custom Analyzers**. By default, the `BM25BuiltInFunction` uses the standard built-in analyzer, which tokenizes text based on punctuation.\n",
    "\n",
    "To use a different analyzer or customize the existing one, you can pass value to the `analyzer_params` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac3e9828",
   "metadata": {},
   "outputs": [],
   "source": [
    "bm25_function = BM25BuiltInFunction(\n",
    "    analyzer_params={\n",
    "        \"tokenizer\": \"standard\",\n",
    "        \"filter\": [\n",
    "            \"lowercase\",  # Built-in filter\n",
    "            {\"type\": \"length\", \"max\": 40},  # Custom cap size of a single token\n",
    "            {\"type\": \"stop\", \"stop_words\": [\"of\", \"to\"]},  # Custom stopwords\n",
    "        ],\n",
    "    },\n",
    "    enable_match=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6897a455",
   "metadata": {},
   "source": [
    "### Hybrid Search with Reranker\n",
    "\n",
    "A hybrid search system combines semantic search and full-text search, optimizing retrieval performance in a RAG system.\n",
    "\n",
    "The following example uses OpenAI embedding for semantic search and BM25 for full-text search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba1558b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create index over the documnts\n",
    "vector_store = MilvusVectorStore(\n",
    "    uri=URI,\n",
    "    # token=TOKEN,\n",
    "    # enable_dense=True,  # enable_dense defaults to True\n",
    "    dim=1536,\n",
    "    enable_sparse=True,\n",
    "    sparse_embedding_function=BM25BuiltInFunction(),\n",
    "    overwrite=True,\n",
    "    # hybrid_ranker=\"RRFRanker\",  # hybrid_ranker defaults to \"RRFRanker\"\n",
    "    # hybrid_ranker_params={},  # hybrid_ranker_params defaults to {}\n",
    ")\n",
    "\n",
    "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
    "index = VectorStoreIndex.from_documents(\n",
    "    documents,\n",
    "    storage_context=storage_context,\n",
    "    embed_model=\"default\",  # \"default\" will use OpenAI embedding\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e94cab2",
   "metadata": {},
   "source": [
    "**How it works**\n",
    "\n",
    "This approach stores documents in a Milvus collection with both vector fields:\n",
    "\n",
    "- `embedding`: Dense embeddings generated by OpenAI embedding model for semantic search.\n",
    "- `sparse_embedding`: Sparse embeddings computed using BM25BuiltInFunction for full-text search.\n",
    "\n",
    "In addition, we have applied a reranking strategy using \"RRFRanker\" with its default parameters. To customize reranker, you are able to configure `hybrid_ranker` and `hybrid_ranker_params` following the [Milvus Reranking Guide](https://milvus.io/docs/reranking.md).\n",
    "\n",
    "Now, let's test the RAG system with a sample query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e960b24",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The author learned several important lessons at Viaweb. These included the importance of\n",
      "understanding growth rate as the ultimate test of a startup, the impact of hiring too many people,\n",
      "the challenges of being at the mercy of investors, and the relief experienced when Yahoo bought the\n",
      "company. Additionally, the author learned about the significance of user feedback, the value of\n",
      "building stores for users, and the realization that growth rate is crucial for the long-term success\n",
      "of a startup.\n"
     ]
    }
   ],
   "source": [
    "# Query\n",
    "query_engine = index.as_query_engine(\n",
    "    vector_store_query_mode=\"hybrid\", similarity_top_k=5\n",
    ")\n",
    "answer = query_engine.query(\"What did the author learn at Viaweb?\")\n",
    "print(textwrap.fill(str(answer), 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c85d0b6a",
   "metadata": {},
   "source": [
    "This hybrid approach ensures more accurate, context-aware responses in a RAG system by leveraging both semantic and keyword-based retrieval."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}