{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "41aecfc5",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/metadata_extraction/MetadataExtraction_LLMSurvey.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b07531d9-7473-480d-bee6-c1ee4cbc207c",
   "metadata": {},
   "source": [
    "# Automated Metadata Extraction for Better Retrieval + Synthesis\n",
    "\n",
    "In this tutorial, we show you how to perform automated metadata extraction for better retrieval results.\n",
    "We use two extractors: a QuestionAnsweredExtractor which generates question/answer pairs from a piece of text, and also a SummaryExtractor which extracts summaries, not only within the current text, but also within adjacent texts.\n",
    "\n",
    "We show that this allows for \"chunk dreaming\" - each individual chunk can have more \"holistic\" details, leading to higher answer quality given retrieved results.\n",
    "\n",
    "Our data source is taken from Eugene Yan's popular article on LLM Patterns: https://eugeneyan.com/writing/llm-patterns/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a4873de-eaa9-4854-8aeb-050704bd894f",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8068876f",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf94b39c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-openai\n",
    "%pip install llama-index-readers-web"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42c5e11d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40d399c4-c93c-41bf-9a47-48aefabb75e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "import os\n",
    "import openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c80a958-e94f-4260-81fc-8791019dae87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# OPTIONAL: setup W&B callback handling for tracing\n",
    "from llama_index.core import set_global_handler\n",
    "\n",
    "set_global_handler(\"wandb\", run_args={\"project\": \"llamaindex\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89cb51f7-1764-40d1-bd25-d7551aa57207",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-...\"\n",
    "openai.api_key = os.environ[\"OPENAI_API_KEY\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a6ef3fc-0d04-43a2-b0d6-4d8f3c90ef3d",
   "metadata": {},
   "source": [
    "## Define Metadata Extractors\n",
    "\n",
    "Here we define metadata extractors. We define two variants:\n",
    "- metadata_extractor_1 only contains the QuestionsAnsweredExtractor\n",
    "- metadata_extractor_2 contains both the QuestionsAnsweredExtractor as well as the SummaryExtractor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0adb8e4a-6728-4073-8256-8b3be4ab1e64",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.core.schema import MetadataMode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0231dff-7443-46bf-9b9d-759198d3408e",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = OpenAI(temperature=0.1, model=\"gpt-3.5-turbo\", max_tokens=512)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2db2cf90-f295-4a3d-a47c-4b2b1dd2d7c5",
   "metadata": {},
   "source": [
    "We also show how to instantiate the `SummaryExtractor` and `QuestionsAnsweredExtractor`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bda151d-6fb8-427e-82fc-0f3bb469d705",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.node_parser import TokenTextSplitter\n",
    "from llama_index.core.extractors import (\n",
    "    SummaryExtractor,\n",
    "    QuestionsAnsweredExtractor,\n",
    ")\n",
    "\n",
    "node_parser = TokenTextSplitter(\n",
    "    separator=\" \", chunk_size=256, chunk_overlap=128\n",
    ")\n",
    "\n",
    "\n",
    "extractors_1 = [\n",
    "    QuestionsAnsweredExtractor(\n",
    "        questions=3, llm=llm, metadata_mode=MetadataMode.EMBED\n",
    "    ),\n",
    "]\n",
    "\n",
    "extractors_2 = [\n",
    "    SummaryExtractor(summaries=[\"prev\", \"self\", \"next\"], llm=llm),\n",
    "    QuestionsAnsweredExtractor(\n",
    "        questions=3, llm=llm, metadata_mode=MetadataMode.EMBED\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4e54937-e9e7-48ed-8600-72cd2f3c529b",
   "metadata": {},
   "source": [
    "## Load in Data, Run Extractors\n",
    "\n",
    "We load in Eugene's essay (https://eugeneyan.com/writing/llm-patterns/) using our LlamaHub SimpleWebPageReader.\n",
    "\n",
    "We then run our extractors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c72c45a9-dcad-4925-b2f7-d25fe5d80c2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import SimpleDirectoryReader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4044f6bd-be50-4958-8d25-4edd6552f103",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load in blog\n",
    "\n",
    "from llama_index.readers.web import SimpleWebPageReader\n",
    "\n",
    "reader = SimpleWebPageReader(html_to_text=True)\n",
    "docs = reader.load_data(urls=[\"https://eugeneyan.com/writing/llm-patterns/\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2701c7e-b67e-4c24-98df-73f96e3756a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(docs[0].get_content())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "269f8ecc-489d-435f-9d81-a9c64fd4d400",
   "metadata": {},
   "outputs": [],
   "source": [
    "orig_nodes = node_parser.get_nodes_from_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d63b7df2-0e5a-4e98-85ea-88ddcf37c99e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# take just the first 8 nodes for testing\n",
    "nodes = orig_nodes[20:28]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb2a96d9-03fe-4d60-9829-4fa80f7ff571",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "is to measure the distance that words would\n",
      "have to move to convert one sequence to another.\n",
      "\n",
      "However, there are several pitfalls to using these conventional benchmarks and\n",
      "metrics.\n",
      "\n",
      "First, there’s **poor correlation between these metrics and human judgments.**\n",
      "BLEU, ROUGE, and others have had [negative correlation with how humans\n",
      "evaluate fluency](https://arxiv.org/abs/2008.12009). They also showed moderate\n",
      "to less correlation with human adequacy scores. In particular, BLEU and ROUGE\n",
      "have [low correlation with tasks that require creativity and\n",
      "diversity](https://arxiv.org/abs/2303.16634).\n",
      "\n",
      "Second, these metrics often have **poor adaptability to a wider variety of\n",
      "tasks**. Adopting a metric proposed for one task to another is not always\n",
      "prudent. For example, exact match metrics such as BLEU and ROUGE are a poor\n",
      "fit for tasks like abstractive summarization or dialogue. Since they’re based\n",
      "on n-gram overlap between output and reference, they don’t make sense for a\n",
      "dialogue task where a wide variety\n"
     ]
    }
   ],
   "source": [
    "print(nodes[3].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "970f4dd0-d5e2-4bef-abc1-494020a9a2b5",
   "metadata": {},
   "source": [
    "### Run metadata extractors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17fd3537-c8e0-4160-b39d-815dbb6504c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e828522a65bb4304bdaeae041a2eee31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing documents into nodes:   0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f8d5a4483c4e49f5b26afc869b3f39dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting questions:   0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from llama_index.core.ingestion import IngestionPipeline\n",
    "\n",
    "# process nodes with metadata extractors\n",
    "pipeline = IngestionPipeline(transformations=[node_parser, *extractors_1])\n",
    "\n",
    "nodes_1 = pipeline.run(nodes=nodes, in_place=False, show_progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4d37c8d-f36e-483d-a3d8-77f100b9e1c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Excerpt from document]\n",
      "questions_this_excerpt_can_answer: 1. What is the correlation between conventional metrics like BLEU and ROUGE and human judgments in evaluating fluency and adequacy in natural language processing tasks?\n",
      "2. How do conventional metrics like BLEU and ROUGE perform in tasks that require creativity and diversity?\n",
      "3. Why are exact match metrics like BLEU and ROUGE not suitable for tasks like abstractive summarization or dialogue in natural language processing?\n",
      "Excerpt:\n",
      "-----\n",
      "is to measure the distance that words would\n",
      "have to move to convert one sequence to another.\n",
      "\n",
      "However, there are several pitfalls to using these conventional benchmarks and\n",
      "metrics.\n",
      "\n",
      "First, there’s **poor correlation between these metrics and human judgments.**\n",
      "BLEU, ROUGE, and others have had [negative correlation with how humans\n",
      "evaluate fluency](https://arxiv.org/abs/2008.12009). They also showed moderate\n",
      "to less correlation with human adequacy scores. In particular, BLEU and ROUGE\n",
      "have [low correlation with tasks that require creativity and\n",
      "diversity](https://arxiv.org/abs/2303.16634).\n",
      "\n",
      "Second, these metrics often have **poor adaptability to a wider variety of\n",
      "tasks**. Adopting a metric proposed for one task to another is not always\n",
      "prudent. For example, exact match metrics such as BLEU and ROUGE are a poor\n",
      "fit for tasks like abstractive summarization or dialogue. Since they’re based\n",
      "on n-gram overlap between output and reference, they don’t make sense for a\n",
      "dialogue task where a wide variety\n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "print(nodes_1[3].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4f86b9c-4c9d-48e2-915a-328d5887a1a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f74cdd7524064d42bb4c250db190b0f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing documents into nodes:   0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec9a4e2ac9424fa4b425b08facf6adfe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting summaries:   0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09dcebe7f8ee437d88792ddf1eecce14",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting questions:   0%|          | 0/8 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 2nd pass: run summaries, and then metadata extractor\n",
    "\n",
    "# process nodes with metadata extractor\n",
    "pipeline = IngestionPipeline(transformations=[node_parser, *extractors_2])\n",
    "\n",
    "nodes_2 = pipeline.run(nodes=nodes, in_place=False, show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d52daa-b57a-4f9a-9f67-70cddd5304a4",
   "metadata": {},
   "source": [
    "### Visualize some sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7822c90d-d048-41d1-b64e-8fbcd8f6e65b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Excerpt from document]\n",
      "prev_section_summary: The section discusses the comparison between BERTScore and MoverScore, two metrics used to evaluate the quality of text generation models. MoverScore is described as a metric that measures the effort required to transform one text sequence into another by mapping semantically related words. The section also highlights the limitations of conventional benchmarks and metrics, such as poor correlation with human judgments and low correlation with tasks requiring creativity.\n",
      "next_section_summary: The section discusses the limitations of current evaluation metrics in natural language processing tasks. It highlights three main issues: lack of creativity and diversity in metrics, poor adaptability to different tasks, and poor reproducibility. The section mentions specific metrics like BLEU and ROUGE, and also references studies that have reported high variance in metric scores.\n",
      "section_summary: The section discusses the limitations of conventional benchmarks and metrics used to measure the distance between word sequences. It highlights two main issues: the poor correlation between these metrics and human judgments, and their limited adaptability to different tasks. The section mentions specific metrics like BLEU and ROUGE, which have been found to have low correlation with human evaluations of fluency, adequacy, creativity, and diversity. It also points out that metrics based on n-gram overlap, such as BLEU and ROUGE, are not suitable for tasks like abstractive summarization or dialogue.\n",
      "questions_this_excerpt_can_answer: 1. What are the limitations of conventional benchmarks and metrics in measuring the distance between word sequences?\n",
      "2. How do metrics like BLEU and ROUGE correlate with human judgments in terms of fluency, adequacy, creativity, and diversity?\n",
      "3. Why are metrics based on n-gram overlap, such as BLEU and ROUGE, not suitable for tasks like abstractive summarization or dialogue?\n",
      "Excerpt:\n",
      "-----\n",
      "is to measure the distance that words would\n",
      "have to move to convert one sequence to another.\n",
      "\n",
      "However, there are several pitfalls to using these conventional benchmarks and\n",
      "metrics.\n",
      "\n",
      "First, there’s **poor correlation between these metrics and human judgments.**\n",
      "BLEU, ROUGE, and others have had [negative correlation with how humans\n",
      "evaluate fluency](https://arxiv.org/abs/2008.12009). They also showed moderate\n",
      "to less correlation with human adequacy scores. In particular, BLEU and ROUGE\n",
      "have [low correlation with tasks that require creativity and\n",
      "diversity](https://arxiv.org/abs/2303.16634).\n",
      "\n",
      "Second, these metrics often have **poor adaptability to a wider variety of\n",
      "tasks**. Adopting a metric proposed for one task to another is not always\n",
      "prudent. For example, exact match metrics such as BLEU and ROUGE are a poor\n",
      "fit for tasks like abstractive summarization or dialogue. Since they’re based\n",
      "on n-gram overlap between output and reference, they don’t make sense for a\n",
      "dialogue task where a wide variety\n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "print(nodes_2[3].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbf5ee6a-daf8-4862-a80a-15c03f0db4b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Excerpt from document]\n",
      "prev_section_summary: The section discusses the F_{BERT} formula used in BERTScore and highlights the advantages of BERTScore over simpler metrics like BLEU and ROUGE. It also introduces MoverScore, another metric that uses contextualized embeddings but allows for many-to-one matching. The key topics are BERTScore, MoverScore, and the differences between them.\n",
      "next_section_summary: The section discusses the comparison between BERTScore and MoverScore, two metrics used to evaluate the quality of text generation models. MoverScore is described as a metric that measures the effort required to transform one text sequence into another by mapping semantically related words. The section also highlights the limitations of conventional benchmarks and metrics, such as poor correlation with human judgments and low correlation with tasks requiring creativity.\n",
      "section_summary: The key topics of this section are BERTScore and MoverScore, which are methods used to compute the similarity between generated output and reference in tasks like image captioning and machine translation. BERTScore uses one-to-one matching of tokens, while MoverScore allows for many-to-one matching. MoverScore solves an optimization problem to measure the distance that words would have to move to convert one sequence to another.\n",
      "questions_this_excerpt_can_answer: 1. What is the main difference between BERTScore and MoverScore?\n",
      "2. How does MoverScore allow for many-to-one matching of tokens?\n",
      "3. What problem does MoverScore solve to measure the distance between two sequences?\n",
      "Excerpt:\n",
      "-----\n",
      "to have better correlation for tasks\n",
      "such as image captioning and machine translation.\n",
      "\n",
      "**[MoverScore](https://arxiv.org/abs/1909.02622)** also uses contextualized\n",
      "embeddings to compute the distance between tokens in the generated output and\n",
      "reference. But unlike BERTScore, which is based on one-to-one matching (or\n",
      "“hard alignment”) of tokens, MoverScore allows for many-to-one matching (or\n",
      "“soft alignment”).\n",
      "\n",
      "![BERTScore \\(left\\) vs. MoverScore \\(right\\)](/assets/mover-score.jpg)\n",
      "\n",
      "BERTScore (left) vs. MoverScore (right;\n",
      "[source](https://arxiv.org/abs/1909.02622))\n",
      "\n",
      "MoverScore enables the mapping of semantically related words in one sequence\n",
      "to their counterparts in another sequence. It does this by solving a\n",
      "constrained optimization problem that finds the minimum effort to transform\n",
      "one text into another. The idea is to measure the distance that words would\n",
      "have to move to convert one sequence to another.\n",
      "\n",
      "However, there\n",
      "-----\n"
     ]
    }
   ],
   "source": [
    "print(nodes_2[1].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "641ae280-68e0-4003-b5de-d9f7ecdc6423",
   "metadata": {},
   "source": [
    "## Setup RAG Query Engines, Compare Results! \n",
    "\n",
    "We setup 3 indexes/query engines on top of the three node variants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e430f74-dd95-4aef-acc1-07aa1f3cbd7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.core.response.notebook_utils import (\n",
    "    display_source_node,\n",
    "    display_response,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd729fb8-1e00-4cd0-9505-a86a7daa89d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# try out different query engines\n",
    "\n",
    "# index0 = VectorStoreIndex(orig_nodes)\n",
    "# index1 = VectorStoreIndex(nodes_1 + orig_nodes[8:])\n",
    "# index2 = VectorStoreIndex(nodes_2 + orig_nodes[8:])\n",
    "\n",
    "index0 = VectorStoreIndex(orig_nodes)\n",
    "index1 = VectorStoreIndex(orig_nodes[:20] + nodes_1 + orig_nodes[28:])\n",
    "index2 = VectorStoreIndex(orig_nodes[:20] + nodes_2 + orig_nodes[28:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "061b1de8-628f-4dbe-852b-3c467ae86aba",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine0 = index0.as_query_engine(similarity_top_k=1)\n",
    "query_engine1 = index1.as_query_engine(similarity_top_k=1)\n",
    "query_engine2 = index2.as_query_engine(similarity_top_k=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79e264bb-ad46-4ed3-a461-6258ce80f944",
   "metadata": {},
   "source": [
    "### Try out some questions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef80afab-e30e-4b57-8128-9d9e174147cb",
   "metadata": {},
   "source": [
    "In this question, we see that the naive response `response0` only mentions BLEU and ROUGE, and lacks context about other metrics.\n",
    "\n",
    "`response2` on the other hand has all metrics within its context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e1e448d-632c-42a0-ad60-4a315491945f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# query_str = \"In the original RAG paper, can you describe the two main approaches for generation and compare them?\"\n",
    "query_str = (\n",
    "    \"Can you describe metrics for evaluating text generation quality, compare\"\n",
    "    \" them, and tell me about their downsides\"\n",
    ")\n",
    "\n",
    "response0 = query_engine0.query(query_str)\n",
    "response1 = query_engine1.query(query_str)\n",
    "response2 = query_engine2.query(query_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5672f974-1a4d-435f-904a-77222eaaed16",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_response(\n",
    "    response0, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "549d1b6a-423e-472e-931c-a37ec23f2e97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "require creativity and\n",
      "diversity](https://arxiv.org/abs/2303.16634).\n",
      "\n",
      "Second, these metrics often have **poor adaptability to a wider variety of\n",
      "tasks**. Adopting a metric proposed for one task to another is not always\n",
      "prudent. For example, exact match metrics such as BLEU and ROUGE are a poor\n",
      "fit for tasks like abstractive summarization or dialogue. Since they’re based\n",
      "on n-gram overlap between output and reference, they don’t make sense for a\n",
      "dialogue task where a wide variety of responses are possible. An output can\n",
      "have zero n-gram overlap with the reference but yet be a good response.\n",
      "\n",
      "Third, these metrics have **poor reproducibility**. Even for the same metric,\n",
      "[high variance is reported across different\n",
      "studies](https://arxiv.org/abs/2008.12009), possibly due to variations in\n",
      "human judgment collection or metric parameter settings. Another study of\n",
      "[ROUGE scores](https://aclanthology.org/2023.acl-long.107/) across 2,000\n",
      "studies found that scores were hard\n"
     ]
    }
   ],
   "source": [
    "print(response0.source_nodes[0].node.get_content())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7093adf9-5fdd-47e3-9114-fc2d264df81c",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_response(\n",
    "    response1, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b188a0dc-cb46-4370-ad28-ce3ca9d9fa4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_response(\n",
    "    response2, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bed27382-21b2-414d-96e4-31c29d7f95f2",
   "metadata": {},
   "source": [
    "In this next question, we ask about BERTScore/MoverScore. \n",
    "\n",
    "The responses are similar. But `response2` gives slightly more detail than `response0` since it has more information about MoverScore contained in the Metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5afe4f69-b676-43fd-bc15-39e18e94801f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# query_str = \"What are some reproducibility issues with the ROUGE metric? Give some details related to benchmarks and also describe other ROUGE issues. \"\n",
    "query_str = (\n",
    "    \"Can you give a high-level overview of BERTScore/MoverScore + formulas if\"\n",
    "    \" available?\"\n",
    ")\n",
    "\n",
    "response0 = query_engine0.query(query_str)\n",
    "response1 = query_engine1.query(query_str)\n",
    "response2 = query_engine2.query(query_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4db36ec6-b64b-45da-aff1-ed51b79c571f",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_response(\n",
    "    response0, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddef5ae7-1030-4257-ba16-8340e46bd548",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_response(\n",
    "    response1, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43c607eb-6133-43f9-83c7-f6e4e03d78df",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_response(\n",
    "    response2, source_length=1000, show_source=True, show_source_metadata=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66805b9a-ee86-4799-8309-b7895f518fee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'questions_this_excerpt_can_answer': '1. What is the advantage of using BERTScore over simpler metrics like BLEU and ROUGE?\\n2. How does MoverScore differ from BERTScore in terms of token matching?\\n3. What tasks have shown better correlation with BERTScore, such as image captioning and machine translation?'}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response1.source_nodes[0].node.metadata"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama-index",
   "language": "python",
   "name": "llama-index"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}