run-llama--llama_index/docs/examples/evaluation/HotpotQADistractor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/evaluation/HotpotQADistractor.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HotpotQADistractor Demo\n",
    "\n",
    "This notebook walks through evaluating a query engine using the HotpotQA dataset. In this task, the LLM must answer a question given a pre-configured context. The answer usually has to be concise, and accuracy is measured by calculating the overlap (measured by F1) and exact match."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parsing documents into nodes: 100%|██████████| 1/1 [00:00<00:00, 129.13it/s]\n",
      "Generating embeddings: 100%|██████████| 1/1 [00:00<00:00, 36.62it/s]\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.evaluation.benchmarks import HotpotQAEvaluator\n",
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.core import Document\n",
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.core.embeddings import resolve_embed_model\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\")\n",
    "embed_model = resolve_embed_model(\n",
    "    \"local:sentence-transformers/all-MiniLM-L6-v2\"\n",
    ")\n",
    "\n",
    "index = VectorStoreIndex.from_documents(\n",
    "    [Document.example()], embed_model=embed_model, show_progress=True\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we try with a very simple engine. In this particular benchmark, the retriever and hence index is actually ignored, as the documents retrieved for each query is provided in the dataset. This is known as the \"distractor\" setting in HotpotQA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset: hotpot_dev_distractor downloaded at: /Users/loganmarkewich/Library/Caches/llama_index/datasets/HotpotQA\n",
      "Evaluating on dataset: hotpot_dev_distractor\n",
      "-------------------------------------\n",
      "Loading 5 queries out of 7405 (fraction: 0.00068)\n",
      "Question:  Were Scott Derrickson and Ed Wood of the same nationality?\n",
      "Response: No.\n",
      "Correct answer:  yes\n",
      "EM: 0 F1: 0\n",
      "-------------------------------------\n",
      "Question:  What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell?\n",
      "Response: Unknown\n",
      "Correct answer:  Chief of Protocol\n",
      "EM: 0 F1: 0\n",
      "-------------------------------------\n",
      "Question:  What science fantasy young adult series, told in first person, has a set of companion books narrating the stories of enslaved worlds and alien species?\n",
      "Response: Animorphs\n",
      "Correct answer:  Animorphs\n",
      "EM: 1 F1: 1.0\n",
      "-------------------------------------\n",
      "Question:  Are the Laleli Mosque and Esma Sultan Mansion located in the same neighborhood?\n",
      "Response: Yes.\n",
      "Correct answer:  no\n",
      "EM: 0 F1: 0\n",
      "-------------------------------------\n",
      "Question:  The director of the romantic comedy \"Big Stone Gap\" is based in what New York city?\n",
      "Response: Greenwich Village\n",
      "Correct answer:  Greenwich Village, New York City\n",
      "EM: 0 F1: 0.5714285714285715\n",
      "-------------------------------------\n",
      "Scores:  {'exact_match': 0.2, 'f1': 0.31428571428571433}\n"
     ]
    }
   ],
   "source": [
    "engine = index.as_query_engine(llm=llm)\n",
    "\n",
    "HotpotQAEvaluator().run(engine, queries=5, show_result=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we try with a sentence transformer reranker, which selects 3 out of the 10 nodes proposed by the retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset: hotpot_dev_distractor downloaded at: /Users/loganmarkewich/Library/Caches/llama_index/datasets/HotpotQA\n",
      "Evaluating on dataset: hotpot_dev_distractor\n",
      "-------------------------------------\n",
      "Loading 5 queries out of 7405 (fraction: 0.00068)\n",
      "Question:  Were Scott Derrickson and Ed Wood of the same nationality?\n",
      "Response: No.\n",
      "Correct answer:  yes\n",
      "EM: 0 F1: 0\n",
      "-------------------------------------\n",
      "Question:  What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell?\n",
      "Response: No government position.\n",
      "Correct answer:  Chief of Protocol\n",
      "EM: 0 F1: 0\n",
      "-------------------------------------\n",
      "Question:  What science fantasy young adult series, told in first person, has a set of companion books narrating the stories of enslaved worlds and alien species?\n",
      "Response: Animorphs\n",
      "Correct answer:  Animorphs\n",
      "EM: 1 F1: 1.0\n",
      "-------------------------------------\n",
      "Question:  Are the Laleli Mosque and Esma Sultan Mansion located in the same neighborhood?\n",
      "Response: No.\n",
      "Correct answer:  no\n",
      "EM: 1 F1: 1.0\n",
      "-------------------------------------\n",
      "Question:  The director of the romantic comedy \"Big Stone Gap\" is based in what New York city?\n",
      "Response: New York City.\n",
      "Correct answer:  Greenwich Village, New York City\n",
      "EM: 0 F1: 0.7499999999999999\n",
      "-------------------------------------\n",
      "Scores:  {'exact_match': 0.4, 'f1': 0.55}\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.postprocessor import SentenceTransformerRerank\n",
    "\n",
    "rerank = SentenceTransformerRerank(top_n=3)\n",
    "\n",
    "engine = index.as_query_engine(\n",
    "    llm=llm,\n",
    "    node_postprocessors=[rerank],\n",
    ")\n",
    "\n",
    "HotpotQAEvaluator().run(engine, queries=5, show_result=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The F1 and exact match scores appear to improve slightly.\n",
    "\n",
    "Note that the benchmark optimizes for producing short factoid answers without explanations, although it is known that CoT prompting can sometimes help in output quality. \n",
    "\n",
    "The scores used are also not a perfect measure of correctness, but can be a quick way to identify how changes in your query engine change the output."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}