run-llama--llama_index/docs/examples/data_connectors/DeepLakeReader.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/data_connectors/DeepLakeReader.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DeepLake Reader"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-readers-deeplake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "import random\n",
    "import textwrap\n",
    "\n",
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.readers.deeplake import DeepLakeReader\n",
    "\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"open ai api key: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/adilkhansarsen/Documents/work/LlamaIndex/llama_index/GPTIndex/lib/python3.9/site-packages/deeplake/util/warnings.py:7: UserWarning: Checking out dataset in read only mode as another machine has locked this version for writing.\n",
      "  warnings.warn(*args, **kwargs)\n",
      "\n",
      "-"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/activeloop/paul_graham_essay\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\\"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hub://activeloop/paul_graham_essay loaded successfully.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " \r"
     ]
    }
   ],
   "source": [
    "reader = DeepLakeReader()\n",
    "query_vector = [random.random() for _ in range(1536)]\n",
    "documents = reader.load_data(\n",
    "    query_vector=query_vector,\n",
    "    dataset_path=\"hub://activeloop/paul_graham_essay\",\n",
    "    limit=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens\n",
      "INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 14220 tokens\n",
      "INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 3975 tokens\n",
      "INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 9 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " A hard moment for the author was when he realized that the AI programs of the time were not going\n",
      "to be able to understand natural language and bridge the gap between what they could do and actually\n",
      "understanding natural language. He had expected college to help him understand the ultimate truths,\n",
      "but instead he found that the other fields took up so much of the space of ideas that there wasn't\n",
      "much left for these supposed ultimate truths. He also found himself in a situation where the\n",
      "students and faculty had an arrangement that didn't require either to learn or teach anything, and\n",
      "he was the only one painting the nude model. He was also painting still lives in his bedroom at\n",
      "night on scraps of canvas due to his financial situation.\n"
     ]
    }
   ],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)\n",
    "query_engine = index.as_query_engine()\n",
    "response = query_engine.query(\"What was a hard moment for the author?\")\n",
    "print(textwrap.fill(str(response), 100))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.6 ('GPTIndex': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  },
  "vscode": {
   "interpreter": {
    "hash": "6e44765f40d39e4c6e3d7a9b35e5b42b8711c1c0fb3c237b84fa62e4b3e35e04"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}