wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
852 lines
184 KiB
Plaintext
852 lines
184 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Activeloop Deep Memory\n",
|
|
"\n",
|
|
"**How do we get +15% RAG hit_rate improvement for question answering on documentation?**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Retrieval-Augmented Generators (RAGs) have recently gained significant attention. As advanced RAG techniques and agents emerge, they expand the potential of what RAGs can accomplish. However, several challenges may limit the integration of RAGs into production. The primary factors to consider when implementing RAGs in production settings are accuracy (recall), cost, and latency. For basic use cases, OpenAI's Ada model paired with a naive similarity search can produce satisfactory results. Yet, for higher accuracy or recall during searches, one might need to employ advanced retrieval techniques. These methods might involve varying data chunk sizes, rewriting queries multiple times, and more, potentially increasing latency and costs. [Activeloop's](https://activeloop.ai/) [Deep Memory](https://www.activeloop.ai/resources/use-deep-memory-to-boost-rag-apps-accuracy-by-up-to-22/) a feature available to Activeloop Deep Lake users, addresses these issuea by introducing a tiny neural network layer trained to match user queries with relevant data from a corpus. While this addition incurs minimal latency during search, it can boost retrieval accuracy by up to 27\n",
|
|
"% and remains cost-effective and simple to use, without requiring any additional advanced rag techniques."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-vector-stores-deeplake\n",
|
|
"%pip install llama-index-llms-openai"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import nest_asyncio\n",
|
|
"import os\n",
|
|
"import getpass\n",
|
|
"\n",
|
|
"nest_asyncio.apply()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install deeplake beautifulsoup4 html2text tiktoken openai llama-index python-dotenv"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"For this tutorial we will parse deeplake documentation, and create a RAG system that could answer the question from the docs. \n",
|
|
"\n",
|
|
"The tutorial can be divided into several parts:\n",
|
|
"1. [Dataset creation and uploading](#1-dataset-creation-and-ingestion)\n",
|
|
"2. [Generating synthetic queries and training deep_memory](#2-training-deep-memory)\n",
|
|
"3. [Evaluating deep memory performance](#3-deepmemory-evaluation)\n",
|
|
"4. [Deep Memory inference](#4-deep-memory-inference)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a name=\"training\"></a>\n",
|
|
"## 1. Dataset Creation and ingestion"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let me parse all of the links using BeautifulSoup and convert them into LlamaIndex documents:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import requests\n",
|
|
"from bs4 import BeautifulSoup\n",
|
|
"from urllib.parse import urljoin\n",
|
|
"\n",
|
|
"\n",
|
|
"def get_all_links(url):\n",
|
|
" response = requests.get(url)\n",
|
|
" if response.status_code != 200:\n",
|
|
" print(f\"Failed to retrieve the page: {url}\")\n",
|
|
" return []\n",
|
|
"\n",
|
|
" soup = BeautifulSoup(response.content, \"html.parser\")\n",
|
|
"\n",
|
|
" # Finding all 'a' tags which typically contain href attribute for links\n",
|
|
" links = [\n",
|
|
" urljoin(url, a[\"href\"])\n",
|
|
" for a in soup.find_all(\"a\", href=True)\n",
|
|
" if a[\"href\"]\n",
|
|
" ]\n",
|
|
"\n",
|
|
" return links"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Fetching pages: 100%|##########| 120/120 [00:13<00:00, 8.70it/s]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain.document_loaders import AsyncHtmlLoader\n",
|
|
"from langchain.document_transformers import Html2TextTransformer\n",
|
|
"from llama_index.core import Document\n",
|
|
"\n",
|
|
"\n",
|
|
"def load_documents(url):\n",
|
|
" all_links = get_all_links(url)\n",
|
|
" loader = AsyncHtmlLoader(all_links)\n",
|
|
" docs = loader.load()\n",
|
|
"\n",
|
|
" html2text = Html2TextTransformer()\n",
|
|
" docs_transformed = html2text.transform_documents(docs)\n",
|
|
" docs = [Document.from_langchain_format(doc) for doc in docs_transformed]\n",
|
|
" return docs\n",
|
|
"\n",
|
|
"\n",
|
|
"docs = load_documents(\"https://docs.deeplake.ai/en/latest/\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"120"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"len(docs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Deep Lake Dataset in hub://activeloop-test/deeplake_docs_deepmemory2 already exists, loading from the storage\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from llama_index.core.evaluation import generate_question_context_pairs\n",
|
|
"from llama_index.core import (\n",
|
|
" VectorStoreIndex,\n",
|
|
" SimpleDirectoryReader,\n",
|
|
" StorageContext,\n",
|
|
")\n",
|
|
"from llama_index.vector_stores.deeplake import DeepLakeVectorStore\n",
|
|
"from llama_index.core.node_parser import SimpleNodeParser\n",
|
|
"from llama_index.llms.openai import OpenAI\n",
|
|
"\n",
|
|
"\n",
|
|
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter your OpenAI API token: \")\n",
|
|
"# # activeloop token is needed if you are not signed in using CLI: `activeloop login -u <USERNAME> -p <PASSWORD>`\n",
|
|
"os.environ[\"ACTIVELOOP_TOKEN\"] = getpass.getpass(\n",
|
|
" \"Enter your ActiveLoop API token: \"\n",
|
|
") # Get your API token from https://app.activeloop.ai, click on your profile picture in the top right corner, and select \"API Tokens\"\n",
|
|
"\n",
|
|
"token = os.getenv(\"ACTIVELOOP_TOKEN\")\n",
|
|
"\n",
|
|
"vector_store = DeepLakeVectorStore(\n",
|
|
" dataset_path=\"hub://activeloop-test/deeplake_docs_deepmemory2\",\n",
|
|
" overwrite=False, # set to True to overwrite the existing dataset\n",
|
|
" runtime={\"tensor_db\": True},\n",
|
|
" token=token,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def create_modules(vector_store, docs=[], populate_vector_store=True):\n",
|
|
" if populate_vector_store:\n",
|
|
" node_parser = SimpleNodeParser.from_defaults(chunk_size=512)\n",
|
|
" nodes = node_parser.get_nodes_from_documents(docs)\n",
|
|
" else:\n",
|
|
" nodes = []\n",
|
|
"\n",
|
|
" # by default, the node ids are set to random uuids. To ensure same id's per run, we manually set them.\n",
|
|
" for idx, node in enumerate(nodes):\n",
|
|
" node.id_ = f\"node_{idx}\"\n",
|
|
"\n",
|
|
" llm = OpenAI(model=\"gpt-4\")\n",
|
|
" storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
|
" return storage_context, nodes, llm"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"(\n",
|
|
" storage_context,\n",
|
|
" nodes,\n",
|
|
" llm,\n",
|
|
") = create_modules(\n",
|
|
" docs=docs,\n",
|
|
" vector_store=vector_store,\n",
|
|
" # populate_vector_store=False, # uncomment this line to skip populating the vector store\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"vector_index = VectorStoreIndex(nodes, storage_context=storage_context)\n",
|
|
"deep_memory_retriever = vector_index.as_retriever(\n",
|
|
" similarity_top_k=4, deep_memory=True\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a name=\"training\"></a>\n",
|
|
"## 2. Training Deep Memory"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here above, we showed the overall schema of how deep_memory works. So as you can see, in order to train it, you need relevance, queries together with corpus data (data that we want to query). The corpus data was already populated in the previous section; here, we will be generating questions and relevance.\n",
|
|
"\n",
|
|
"1. `questions` - is a text of strings, where each string represents a query.\n",
|
|
"2. `relevance` - contains links to the ground truth for each question. There might be several docs that contain an answer to the given question. Because of this, relevance is List[List[tuple[str, float]]], where the outer list represents queries and the inner list relevant documents. The tuple contains a str, float pair where the string represents the id of the source doc (corresponds to the id tensor in the dataset), while the float corresponds to how much the current document is related to the question."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.evaluation import (\n",
|
|
" generate_question_context_pairs,\n",
|
|
" EmbeddingQAFinetuneDataset,\n",
|
|
")\n",
|
|
"import random\n",
|
|
"\n",
|
|
"\n",
|
|
"def create_train_test_datasets(\n",
|
|
" number_of_samples=600, llm=None, nodes=None, save=False\n",
|
|
"):\n",
|
|
" random_indices = random.sample(range(len(nodes)), number_of_samples)\n",
|
|
"\n",
|
|
" ratio = int(len(random_indices) * 0.8)\n",
|
|
"\n",
|
|
" train_indices = random_indices[:ratio]\n",
|
|
" test_indices = random_indices[ratio:]\n",
|
|
"\n",
|
|
" train_nodes = [nodes[i] for i in train_indices]\n",
|
|
" test_nodes = [nodes[i] for i in test_indices]\n",
|
|
"\n",
|
|
" train_qa_dataset = generate_question_context_pairs(\n",
|
|
" train_nodes, llm=llm, num_questions_per_chunk=1\n",
|
|
" )\n",
|
|
"\n",
|
|
" test_qa_dataset = generate_question_context_pairs(\n",
|
|
" test_nodes, llm=llm, num_questions_per_chunk=1\n",
|
|
" )\n",
|
|
"\n",
|
|
" # [optional] save\n",
|
|
" if save:\n",
|
|
" train_qa_dataset.save_json(\n",
|
|
" f\"deeplake_docs_{number_of_samples}_train.json\"\n",
|
|
" )\n",
|
|
" test_qa_dataset.save_json(\n",
|
|
" f\"deeplake_docs_{number_of_samples}_test.json\"\n",
|
|
" )\n",
|
|
" return train_qa_dataset, test_qa_dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" 4%|▍ | 19/480 [02:25<1:04:00, 8.33s/it]"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"train_qa_dataset, test_qa_dataset = create_train_test_datasets(\n",
|
|
" number_of_samples=600, llm=llm, nodes=nodes, save=True\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"train_qa_dataset = EmbeddingQAFinetuneDataset.from_json(\n",
|
|
" \"deeplake_docs_600_train.json\"\n",
|
|
")\n",
|
|
"test_qa_dataset = EmbeddingQAFinetuneDataset.from_json(\n",
|
|
" \"deeplake_docs_600_test.json\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def create_query_relevance(qa_dataset):\n",
|
|
" \"\"\"Function for converting llama-index dataset to correct format for deep memory training\"\"\"\n",
|
|
" queries = [text for _, text in qa_dataset.queries.items()]\n",
|
|
" relevant_docs = qa_dataset.relevant_docs\n",
|
|
" relevance = []\n",
|
|
" for doc in relevant_docs:\n",
|
|
" relevance.append([(relevant_docs[doc][0], 1)])\n",
|
|
" return queries, relevance"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"train_queries, train_relevance = create_query_relevance(train_qa_dataset)\n",
|
|
"test_queries, test_relevance = create_query_relevance(test_qa_dataset)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['In the context of creating a bounding box tensor in a dataset, explain the significance of the \"coords\" argument and its keys \"type\" and \"mode\". What does the \"type\" key specify about the bounding box coordinates?',\n",
|
|
" 'Explain the process of creating an intrinsics tensor and appending intrinsics matrices in the context of computer vision. What are the dimensions of the intrinsics parameters and what do they represent? Also, describe the concept of a Segmentation Mask Htype and its role in image processing.',\n",
|
|
" 'In the context of querying for images in the MNIST Train Dataset using `ds.query`, what does the command \"select * where labels == 0\" signify and what is the expected output?']"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"train_queries[:3]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[[('node_788', 1)], [('node_861', 1)], [('node_82', 1)]]"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"train_relevance[:3]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['What are the steps to update the information of keypoints and connections in a tensor, and what types of data can be appended to keypoints?',\n",
|
|
" 'What is the command to create a mesh tensor in DeepLake and what are the supported compressions? Also, explain how to append a ply file containing mesh data to this tensor.',\n",
|
|
" 'What is a Sequence htype in the context of tensors and how does it function as a wrapper for other htypes? Provide examples.']"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"test_queries[:3]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[[('node_933', 1)], [('node_671', 1)], [('node_471', 1)]]"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"test_relevance[:3]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Starting DeepMemory training job\n",
|
|
"Your Deep Lake dataset has been successfully created!\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": []
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Preparing training data for deepmemory:\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Creating 483 embeddings in 1 batches of size 483:: 100%|██████████| 1/1 [00:03<00:00, 3.67s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"DeepMemory training job started. Job ID: 65421a5003888c9ca36c72e8\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
|
|
"\n",
|
|
"embeddings = OpenAIEmbeddings()\n",
|
|
"\n",
|
|
"\n",
|
|
"job_id = vector_store.vectorstore.deep_memory.train(\n",
|
|
" queries=train_queries,\n",
|
|
" relevance=train_relevance,\n",
|
|
" embedding_function=embeddings.embed_documents,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/adilkhan/deeplake_docs_deepmemory2\n",
|
|
"--------------------------------------------------------------\n",
|
|
"| 65421a5003888c9ca36c72e8 |\n",
|
|
"--------------------------------------------------------------\n",
|
|
"| status | completed |\n",
|
|
"--------------------------------------------------------------\n",
|
|
"| progress | eta: 12.2 seconds |\n",
|
|
"| | recall@10: 67.01% (+18.56%) |\n",
|
|
"--------------------------------------------------------------\n",
|
|
"| results | recall@10: 67.01% (+18.56%) |\n",
|
|
"--------------------------------------------------------------\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"vector_store.vectorstore.deep_memory.status(job_id)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a name=\"training\"></a>\n",
|
|
"## 3. DeepMemory Evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Fantastic! The training has led to some remarkable improvements! Now, let's assess its performance on a test set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[37minfo \u001b[35mWed Nov 1 09:32:44 2023 GMT \u001b[0m Added distance metric `deepmemory_distance`.\n",
|
|
"Embedding queries took 0.95 seconds\n",
|
|
"---- Evaluating without Deep Memory ---- \n",
|
|
"Recall@1:\t 12.5%\n",
|
|
"Recall@3:\t 23.3%\n",
|
|
"Recall@5:\t 30.8%\n",
|
|
"Recall@10:\t 50.8%\n",
|
|
"Recall@50:\t 94.2%\n",
|
|
"Recall@100:\t 95.8%\n",
|
|
"---- Evaluating with Deep Memory ---- \n",
|
|
"Recall@1:\t 11.7%\n",
|
|
"Recall@3:\t 27.5%\n",
|
|
"Recall@5:\t 40.8%\n",
|
|
"Recall@10:\t 65.0%\n",
|
|
"Recall@50:\t 96.7%\n",
|
|
"Recall@100:\t 98.3%\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"recalls = vector_store.vectorstore.deep_memory.evaluate(\n",
|
|
" queries=test_queries,\n",
|
|
" relevance=test_relevance,\n",
|
|
" embedding_function=embeddings.embed_documents,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Impressive! We've observed a 15% increase in recall on the test set. Next, let's employ the RetrieverEvaluator to examine the MRR (Mean Reciprocal Rank) and hit rates."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"\n",
|
|
"\n",
|
|
"def display_results(eval_results):\n",
|
|
" \"\"\"Display results from evaluate.\"\"\"\n",
|
|
" hit_rates = []\n",
|
|
" mrrs = []\n",
|
|
" names = []\n",
|
|
" for name, eval_result in eval_results.items():\n",
|
|
" metric_dicts = []\n",
|
|
" for er in eval_result:\n",
|
|
" metric_dict = er.metric_vals_dict\n",
|
|
" metric_dicts.append(metric_dict)\n",
|
|
"\n",
|
|
" full_df = pd.DataFrame(metric_dicts)\n",
|
|
"\n",
|
|
" hit_rate = full_df[\"hit_rate\"].mean()\n",
|
|
" mrr = full_df[\"mrr\"].mean()\n",
|
|
"\n",
|
|
" hit_rates.append(hit_rate)\n",
|
|
" mrrs.append(mrr)\n",
|
|
" names.append(name)\n",
|
|
"\n",
|
|
" metric_df = pd.DataFrame(\n",
|
|
" [\n",
|
|
" {\"retrievers\": names[i], \"hit_rate\": hit_rates[i], \"mrr\": mrrs[i]}\n",
|
|
" for i in range(2)\n",
|
|
" ],\n",
|
|
" )\n",
|
|
"\n",
|
|
" return metric_df"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Evaluating performance of retrieval with deep memory:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.evaluation import RetrieverEvaluator\n",
|
|
"\n",
|
|
"deep_memory_retriever = vector_index.as_retriever(\n",
|
|
" similarity_top_k=10, vector_store_kwargs={\"deep_memory\": True}\n",
|
|
")\n",
|
|
"dm_retriever_evaluator = RetrieverEvaluator.from_metric_names(\n",
|
|
" [\"mrr\", \"hit_rate\"], retriever=deep_memory_retriever\n",
|
|
")\n",
|
|
"\n",
|
|
"dm_eval_results = await dm_retriever_evaluator.aevaluate_dataset(\n",
|
|
" test_qa_dataset, retriever=dm_retriever_evaluator\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core.evaluation import RetrieverEvaluator\n",
|
|
"\n",
|
|
"naive_retriever = vector_index.as_retriever(similarity_top_k=10)\n",
|
|
"naive_retriever_evaluator = RetrieverEvaluator.from_metric_names(\n",
|
|
" [\"mrr\", \"hit_rate\"], retriever=naive_retriever\n",
|
|
")\n",
|
|
"\n",
|
|
"naive_eval_results = await naive_retriever_evaluator.aevaluate_dataset(\n",
|
|
" test_qa_dataset, retriever=naive_retriever\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>retrievers</th>\n",
|
|
" <th>hit_rate</th>\n",
|
|
" <th>mrr</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>with with Deep Memory top-10 eval</td>\n",
|
|
" <td>0.650000</td>\n",
|
|
" <td>0.244775</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>without with Deep Memory top-10 eval</td>\n",
|
|
" <td>0.508333</td>\n",
|
|
" <td>0.215129</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>"
|
|
],
|
|
"text/plain": [
|
|
" retrievers hit_rate mrr\n",
|
|
"0 with with Deep Memory top-10 eval 0.650000 0.244775\n",
|
|
"1 without with Deep Memory top-10 eval 0.508333 0.215129"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"eval_results = {\n",
|
|
" f\"{mode} with Deep Memory top-10 eval\": eval_result\n",
|
|
" for mode, eval_result in zip(\n",
|
|
" [\"with\", \"without\"], [dm_eval_results, naive_eval_results]\n",
|
|
" )\n",
|
|
"}\n",
|
|
"\n",
|
|
"display_results(eval_results)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Not only hit_rate has increased but also MRR"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a name=\"training\"></a>\n",
|
|
"## 4. Deep Memory Inference"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\u001b[37minfo \u001b[35mWed Nov 1 11:37:33 2023 GMT \u001b[0m Can't find any metric in the dataset.\n",
|
|
"You can connect your own storage to deeplake by using the `connect()` function in the deeplake API.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"query_engine = vector_index.as_query_engine(\n",
|
|
" vector_store_kwargs={\"deep_memory\": True}, llm=llm\n",
|
|
")\n",
|
|
"response = query_engine.query(\n",
|
|
" \"How can you connect your own storage to the deeplake?\"\n",
|
|
")\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"The context does not provide information on how to connect your own storage to Deep Lake.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"query_engine = vector_index.as_query_engine(\n",
|
|
" vector_store_kwargs={\"deep_memory\": False}, llm=llm\n",
|
|
")\n",
|
|
"response = query_engine.query(\n",
|
|
" \"How can you connect your own storage to the deeplake?\"\n",
|
|
")\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"From our observations, without \"deep memory\", our model tends to produce inaccuracies because it retrieves the wrong context."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|