run-llama--llama_index/docs/examples/docstore/DocstoreDemo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "92f78f93",
   "metadata": {},
   "source": [
    "# Docstore Demo\n",
    "\n",
    "This guide shows you how to directly use our `DocumentStore` abstraction. By putting nodes in the docstore, this allows you to define multiple indices over the same underlying docstore, instead of duplicating data across indices.\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/docstore/DocstoreDemo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c4aaf98",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7e65113",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b337f20a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a54d1c43-4b7f-4917-939f-a964f6f3dafc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa67fa07-1395-4aab-a356-72bdb302f6b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d12d766-3ca8-4012-9da2-248be80bb6ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import SimpleDirectoryReader\n",
    "from llama_index.core import VectorStoreIndex, SimpleKeywordTableIndex\n",
    "from llama_index.core import SummaryIndex\n",
    "from llama_index.core import ComposableGraph\n",
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.core import Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7fe8558",
   "metadata": {},
   "source": [
    "#### Download Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aac4108",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p 'data/paul_graham/'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f6dd9d5f-a601-4097-894e-fe98a0c35a5b",
   "metadata": {},
   "source": [
    "#### Load Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7cdaf9d-cfbd-4ced-8d4e-6eef8508224d",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = SimpleDirectoryReader(\"./data/paul_graham/\")\n",
    "documents = reader.load_data()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bae82b55-5c9f-432a-9e06-1fccb6f9fc7f",
   "metadata": {},
   "source": [
    "#### Parse into Nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f97e558a-c29f-44ec-ab33-1f481da1a6ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "\n",
    "nodes = SentenceSplitter().get_nodes_from_documents(documents)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "aff4c8e1-b2ba-4ea6-a8df-978c2788fedc",
   "metadata": {},
   "source": [
    "#### Add to Docstore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ba8b0da-67a8-4653-8cdb-09e39583a2d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.storage.docstore import SimpleDocumentStore\n",
    "\n",
    "docstore = SimpleDocumentStore()\n",
    "docstore.add_documents(nodes)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "528149c1-5bde-4eba-b75a-e8fa1da17d7c",
   "metadata": {},
   "source": [
    "#### Define Multiple Indexes\n",
    "\n",
    "Each index uses the same underlying Node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "316fb6ac-2031-4d17-9999-ffdb827f46d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import StorageContext\n",
    "\n",
    "\n",
    "storage_context = StorageContext.from_defaults(docstore=docstore)\n",
    "summary_index = SummaryIndex(nodes, storage_context=storage_context)\n",
    "vector_index = VectorStoreIndex(nodes, storage_context=storage_context)\n",
    "keyword_table_index = SimpleKeywordTableIndex(\n",
    "    nodes, storage_context=storage_context\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c6b2141-fc77-4dec-891b-d4dad0633b35",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NOTE: the docstore still has the same nodes\n",
    "len(storage_context.docstore.docs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d3bf6aaf-3375-4212-8323-777969a918f7",
   "metadata": {},
   "source": [
    "#### Test out some Queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bba68f3-2743-437e-93b6-ce9ba92e40c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:llama_index.llm_predictor.base:Unknown max input size for gpt-3.5-turbo, using defaults.\n",
      "Unknown max input size for gpt-3.5-turbo, using defaults.\n"
     ]
    }
   ],
   "source": [
    "llm = OpenAI(temperature=0, model=\"gpt-3.5-turbo\")\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.chunk_size = 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "544c0565-72a0-434b-98e5-83138ebdaa2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = summary_index.as_query_engine()\n",
    "response = query_engine.query(\"What is a summary of this document?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "036077b7-108e-4026-9628-44c694343460",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = vector_index.as_query_engine()\n",
    "response = query_engine.query(\"What did the author do growing up?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecd7719c-f663-4edb-a239-d2a8f0a5c091",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = keyword_table_index.as_query_engine()\n",
    "response = query_engine.query(\"What did the author do after his time at YC?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37524641-2632-4a76-8ae6-00f1285256d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "After his time at YC, the author decided to take a break and focus on painting. He spent most of 2014 painting and then, in November, he ran out of steam and stopped. He then moved to Florence, Italy to attend the Accademia di Belle Arti di Firenze, where he studied painting and drawing. He also started painting still lives in his bedroom at night. In March 2015, he started working on Lisp again and wrote a new Lisp, called Bel, in itself in Arc. He wrote essays through 2020, but also started to think about other things he could work on. He wrote an essay for himself to answer the question of how he should choose what to do next and then wrote a more detailed version for others to read. He also created the Y Combinator logo, which was an inside joke referencing the Viaweb logo, a white V on a red circle, so he made the YC logo a white Y on an orange square. He also created a fund for YC for a couple of years, but after Heroku got bought, he had enough money to go back to being self-funded. He also disliked the term \"deal flow\" because it implies that the number of new startups at any given time\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama_index",
   "language": "python",
   "name": "llama_index"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}