{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "2f7b6543", "metadata": {}, "source": [ "\"Open" ] }, { "attachments": {}, "cell_type": "markdown", "id": "08293776-0084-4d01-a054-32a2d6a6b5c2", "metadata": {}, "source": [ "# DocArray InMemory Vector Store\n", "\n", "[DocArrayInMemoryVectorStore](https://docs.docarray.org/user_guide/storing/index_in_memory/) is a document index provided by [Docarray](https://github.com/docarray/docarray) that stores documents in memory. It is a great starting point for small datasets, where you may not want to launch a database server.\n", "\n", "\n" ] }, { "attachments": {}, "cell_type": "markdown", "id": "218b7c5a", "metadata": {}, "source": [ "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙." ] }, { "cell_type": "code", "execution_count": null, "id": "15a94bdd", "metadata": {}, "outputs": [], "source": [ "%pip install llama-index-vector-stores-docarray" ] }, { "cell_type": "code", "execution_count": null, "id": "f495caf4", "metadata": {}, "outputs": [], "source": [ "!pip install llama-index" ] }, { "cell_type": "code", "execution_count": null, "id": "7e46f643-8dd9-4224-87f9-08b4f0edaebd", "metadata": {}, "outputs": [], "source": [ "import os\n", "import sys\n", "import logging\n", "import textwrap\n", "\n", "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "# stop huggingface warnings\n", "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n", "\n", "# Uncomment to see debug logs\n", "# logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n", "# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n", "\n", "from llama_index.core import (\n", " GPTVectorStoreIndex,\n", " SimpleDirectoryReader,\n", " Document,\n", ")\n", "from llama_index.vector_stores.docarray import DocArrayInMemoryVectorStore\n", "from IPython.display import Markdown, display" ] }, { "cell_type": "code", "execution_count": null, "id": "779f2eaa-c097-47e5-90cb-b40ce278922f", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "os.environ[\"OPENAI_API_KEY\"] = \"\"" ] }, { "attachments": {}, "cell_type": "markdown", "id": "b34c1928", "metadata": {}, "source": [ "Download Data" ] }, { "cell_type": "code", "execution_count": null, "id": "6171c840", "metadata": {}, "outputs": [], "source": [ "!mkdir -p 'data/paul_graham/'\n", "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'" ] }, { "cell_type": "code", "execution_count": null, "id": "d86c31cd-21a6-4f1d-95ff-04b6e67d4901", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Document ID: 1c21062a-50a3-4133-a0b1-75f837a953e5 Document Hash: 77ae91ab542f3abb308c4d7c77c9bc4c9ad0ccd63144802b7cbe7e1bb3a4094e\n" ] } ], "source": [ "# load documents\n", "documents = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()\n", "print(\n", " \"Document ID:\",\n", " documents[0].doc_id,\n", " \"Document Hash:\",\n", " documents[0].doc_hash,\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "id": "51ceb667-ae88-4069-9f6a-f37448a122b0", "metadata": {}, "source": [ "## Initialization and indexing" ] }, { "cell_type": "code", "execution_count": null, "id": "a3e4ed7c-7409-41dc-8a60-e079df28a717", "metadata": {}, "outputs": [], "source": [ "from llama_index.core import StorageContext\n", "\n", "\n", "vector_store = DocArrayInMemoryVectorStore()\n", "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n", "index = GPTVectorStoreIndex.from_documents(\n", " documents, storage_context=storage_context\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "id": "a71d905d-6674-41dc-b90a-1c0303e3107e", "metadata": {}, "source": [ "## Querying" ] }, { "cell_type": "code", "execution_count": null, "id": "294d10ec-8f49-4bef-8d08-a3e707178199", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Token indices sequence length is longer than the specified maximum sequence length for this model (1830 > 1024). Running this sequence through the model will result in indexing errors\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " Growing up, the author wrote short stories, programmed on an IBM 1401, and nagged his father to buy\n", "him a TRS-80 microcomputer. He wrote simple games, a program to predict how high his model rockets\n", "would fly, and a word processor. He also studied philosophy in college, but switched to AI after\n", "becoming bored with it. He then took art classes at Harvard and applied to art schools, eventually\n", "attending RISD.\n" ] } ], "source": [ "# set Logging to DEBUG for more detailed outputs\n", "query_engine = index.as_query_engine()\n", "response = query_engine.query(\"What did the author do growing up?\")\n", "print(textwrap.fill(str(response), 100))" ] }, { "cell_type": "code", "execution_count": null, "id": "5d78975c-f1ab-4243-a172-0353b768a666", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " A hard moment for the author was when he realized that the AI programs of the time were a hoax and\n", "that there was an unbridgeable gap between what they could do and actually understanding natural\n", "language. He had invested a lot of time and energy into learning about AI and was disappointed to\n", "find out that it was not going to get him the results he had hoped for.\n" ] } ], "source": [ "response = query_engine.query(\"What was a hard moment for the author?\")\n", "print(textwrap.fill(str(response), 100))" ] }, { "attachments": {}, "cell_type": "markdown", "id": "ba1aac19-6885-4774-ba94-ea5f1112d98f", "metadata": {}, "source": [ "## Querying with filters" ] }, { "cell_type": "code", "execution_count": null, "id": "5f69e833-4549-4a76-91a6-494476186e1c", "metadata": {}, "outputs": [], "source": [ "from llama_index.core.schema import TextNode\n", "\n", "nodes = [\n", " TextNode(\n", " text=\"The Shawshank Redemption\",\n", " metadata={\n", " \"author\": \"Stephen King\",\n", " \"theme\": \"Friendship\",\n", " },\n", " ),\n", " TextNode(\n", " text=\"The Godfather\",\n", " metadata={\n", " \"director\": \"Francis Ford Coppola\",\n", " \"theme\": \"Mafia\",\n", " },\n", " ),\n", " TextNode(\n", " text=\"Inception\",\n", " metadata={\n", " \"director\": \"Christopher Nolan\",\n", " },\n", " ),\n", "]" ] }, { "cell_type": "code", "execution_count": null, "id": "048e9cd6-9d85-4f01-932d-84b1b3bbbecf", "metadata": {}, "outputs": [], "source": [ "from llama_index.core import StorageContext\n", "\n", "\n", "vector_store = DocArrayInMemoryVectorStore()\n", "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n", "\n", "index = GPTVectorStoreIndex(nodes, storage_context=storage_context)" ] }, { "cell_type": "code", "execution_count": null, "id": "571af38c-ba4f-48b0-8498-5001ee1c1559", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[NodeWithScore(node=Node(text='director: Francis Ford Coppola\\ntheme: Mafia\\n\\nThe Godfather', doc_id='41c99963-b200-4ce6-a9c4-d06ffeabdbc5', embedding=None, doc_hash='b770e43e6a94854a22dc01421d3d9ef6a94931c2b8dbbadf4fdb6eb6fbe41010', extra_info=None, node_info=None, relationships={: 'None'}), score=0.7681788983417586)]" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from llama_index.core.vector_stores import ExactMatchFilter, MetadataFilters\n", "\n", "\n", "filters = MetadataFilters(\n", " filters=[ExactMatchFilter(key=\"theme\", value=\"Mafia\")]\n", ")\n", "\n", "retriever = index.as_retriever(filters=filters)\n", "retriever.retrieve(\"What is inception about?\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3" } }, "nbformat": 4, "nbformat_minor": 5 }