{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "68da6d2b", "metadata": {}, "source": [ "\"Open" ] }, { "cell_type": "markdown", "id": "307804a3-c02b-4a57-ac0d-172c30ddc851", "metadata": {}, "source": [ "# Awadb Vector Store" ] }, { "attachments": {}, "cell_type": "markdown", "id": "295deb84", "metadata": {}, "source": [ "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙." ] }, { "cell_type": "code", "execution_count": null, "id": "65d6e7e8", "metadata": {}, "outputs": [], "source": [ "%pip install llama-index-embeddings-huggingface\n", "%pip install llama-index-vector-stores-awadb" ] }, { "cell_type": "code", "execution_count": null, "id": "0f9014eb", "metadata": {}, "outputs": [], "source": [ "!pip install llama-index" ] }, { "cell_type": "markdown", "id": "4334feda", "metadata": {}, "source": [ "## Creating an Awadb index" ] }, { "cell_type": "code", "execution_count": null, "id": "a1b5e530", "metadata": {}, "outputs": [], "source": [ "import logging\n", "import sys\n", "\n", "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n", "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))" ] }, { "cell_type": "markdown", "id": "8ee4473a-094f-4d0a-a825-e1213db07240", "metadata": {}, "source": [ "#### Load documents, build the VectorStoreIndex" ] }, { "cell_type": "code", "execution_count": null, "id": "0a2bcc07", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:numexpr.utils:Note: NumExpr detected 12 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 8.\n", "Note: NumExpr detected 12 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 8.\n", "INFO:numexpr.utils:NumExpr defaulting to 8 threads.\n", "NumExpr defaulting to 8 threads.\n" ] } ], "source": [ "from llama_index.core import (\n", " SimpleDirectoryReader,\n", " VectorStoreIndex,\n", " StorageContext,\n", ")\n", "from IPython.display import Markdown, display\n", "import openai\n", "\n", "openai.api_key = \"\"" ] }, { "attachments": {}, "cell_type": "markdown", "id": "bfa5ac36", "metadata": {}, "source": [ "#### Download Data" ] }, { "cell_type": "code", "execution_count": null, "id": "5ae97358", "metadata": {}, "outputs": [], "source": [ "!mkdir -p 'data/paul_graham/'\n", "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'" ] }, { "attachments": {}, "cell_type": "markdown", "id": "f5060ac6", "metadata": {}, "source": [ "#### Load Data" ] }, { "cell_type": "code", "execution_count": null, "id": "68cbd239-880e-41a3-98d8-dbb3fab55431", "metadata": {}, "outputs": [], "source": [ "# load documents\n", "documents = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()" ] }, { "cell_type": "code", "execution_count": null, "id": "ba1558b3", "metadata": {}, "outputs": [], "source": [ "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n", "from llama_index.vector_stores.awadb import AwaDBVectorStore\n", "\n", "embed_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\")\n", "\n", "vector_store = AwaDBVectorStore()\n", "storage_context = StorageContext.from_defaults(vector_store=vector_store)\n", "\n", "index = VectorStoreIndex.from_documents(\n", " documents, storage_context=storage_context, embed_model=embed_model\n", ")" ] }, { "cell_type": "markdown", "id": "04304299-fc3e-40a0-8600-f50c3292767e", "metadata": {}, "source": [ "#### Query Index" ] }, { "cell_type": "code", "execution_count": null, "id": "35369eda", "metadata": {}, "outputs": [], "source": [ "# set Logging to DEBUG for more detailed outputs\n", "query_engine = index.as_query_engine()\n", "response = query_engine.query(\"What did the author do growing up?\")" ] }, { "cell_type": "code", "execution_count": null, "id": "bedbb693-725f-478f-be26-fa7180ea38b2", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "\n", "Growing up, the author wrote short stories, experimented with programming on an IBM 1401, nagged his father to buy a TRS-80 computer, wrote simple games, a program to predict how high his model rockets would fly, and a word processor. He also studied philosophy in college, switched to AI, and worked on building the infrastructure of the web. He wrote essays and published them online, had dinners for a group of friends every Thursday night, painted, and bought a building in Cambridge." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(Markdown(f\"{response}\"))" ] }, { "cell_type": "code", "execution_count": null, "id": "99212d33", "metadata": {}, "outputs": [], "source": [ "# set Logging to DEBUG for more detailed outputs\n", "query_engine = index.as_query_engine()\n", "response = query_engine.query(\n", " \"What did the author do after his time at Y Combinator?\"\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "1a720ad6", "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "\n", "After his time at Y Combinator, the author wrote essays, worked on Lisp, and painted. He also visited his mother in Oregon and helped her get out of a nursing home." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(Markdown(f\"{response}\"))" ] } ], "metadata": { "kernelspec": { "display_name": "new_pytorch", "language": "python", "name": "new_pytorch" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3" } }, "nbformat": 4, "nbformat_minor": 5 }