wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
1087 lines
45 KiB
Plaintext
1087 lines
45 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e5df0f51",
|
||
"metadata": {},
|
||
"source": [
|
||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/low_level/response_synthesis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "942d3df2-9bf6-450b-8cb4-ff5f2378e26b",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Building Response Synthesis from Scratch\n",
|
||
"\n",
|
||
"In this tutorial, we show you how to build the \"LLM synthesis\" component of a RAG pipeline from scratch. Given a set of retrieved Nodes, we'll show you how to synthesize a response even if the retrieved context overflows the context window.\n",
|
||
"\n",
|
||
"We'll walk through some synthesis strategies:\n",
|
||
"- Create and Refine\n",
|
||
"- Tree Summarization\n",
|
||
"\n",
|
||
"We're essentially unpacking our \"Response Synthesis\" module and exposing that for the user.\n",
|
||
"\n",
|
||
"We use OpenAI as a default LLM but you're free to plug in any LLM you wish."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b30f3c3e-712e-4ea2-b7f5-b6fb32fdc576",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Setup\n",
|
||
"\n",
|
||
"We build an empty Pinecone Index, and define the necessary LlamaIndex wrappers/abstractions so that we can load/index data and get back a vector retriever."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "82a40564",
|
||
"metadata": {},
|
||
"source": [
|
||
"If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "9590a49f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"%pip install llama-index-readers-file pymupdf\n",
|
||
"%pip install llama-index-vector-stores-pinecone\n",
|
||
"%pip install llama-index-llms-openai"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "09df046f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install llama-index"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e7de2c0d-4eae-47b1-b7fc-24efcd7ee83c",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Load Data"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2372579b-aa84-4ed4-beb5-e9e44831ed27",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"!mkdir data\n",
|
||
"!wget --user-agent \"Mozilla\" \"https://arxiv.org/pdf/2307.09288.pdf\" -O \"data/llama2.pdf\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "17a6e0d2-e256-44de-b078-a4c5b2ef40ff",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from pathlib import Path\n",
|
||
"from llama_index.readers.file import PyMuPDFReader"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "da2dc652-028b-488e-908f-1a6fc4c380ba",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"loader = PyMuPDFReader()\n",
|
||
"documents = loader.load(file_path=\"./data/llama2.pdf\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f9cf9d64-2287-44c9-baad-73ce1ba88ca6",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Build Pinecone Index, Get Retriever\n",
|
||
"\n",
|
||
"We use our high-level LlamaIndex abstractions to 1) ingest data into Pinecone, and then 2) get a vector retriever.\n",
|
||
"\n",
|
||
"Note that we set chunk sizes to 1024."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "75b980c9-f769-4c42-b15a-c9cc6dca458e",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"/Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
|
||
" from tqdm.autonotebook import tqdm\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import pinecone\n",
|
||
"import os\n",
|
||
"\n",
|
||
"api_key = os.environ[\"PINECONE_API_KEY\"]\n",
|
||
"pinecone.init(api_key=api_key, environment=\"us-west1-gcp\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "20a17eb0-5da5-44fc-a068-1527fce942c1",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# dimensions are for text-embedding-ada-002\n",
|
||
"pinecone.create_index(\n",
|
||
" \"quickstart\", dimension=1536, metric=\"euclidean\", pod_type=\"p1\"\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "cf6b3e1b-efa6-4a50-a70a-40fb184de49b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"pinecone_index = pinecone.Index(\"quickstart\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "271ad03f-99fa-481d-ab84-8da0354ae996",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"{}"
|
||
]
|
||
},
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# [Optional] drop contents in index\n",
|
||
"pinecone_index.delete(deleteAll=True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "3a7dc957-d147-47a4-8019-e0ad7f199403",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.vector_stores.pinecone import PineconeVectorStore\n",
|
||
"from llama_index.core import VectorStoreIndex\n",
|
||
"from llama_index.core.node_parser import SentenceSplitter\n",
|
||
"from llama_index.core import StorageContext"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "527bc36c-711b-4939-8b60-78ee48ae98de",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"vector_store = PineconeVectorStore(pinecone_index=pinecone_index)\n",
|
||
"# NOTE: set chunk size of 1024\n",
|
||
"splitter = SentenceSplitter(chunk_size=1024)\n",
|
||
"storage_context = StorageContext.from_defaults(vector_store=vector_store)\n",
|
||
"index = VectorStoreIndex.from_documents(\n",
|
||
" documents, transformations=[splitter], storage_context=storage_context\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "f3255a92-32c3-4c5d-b859-2be4366d0484",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"retriever = index.as_retriever()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7d394373-f025-446e-9306-66fba2686bf2",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### Given an example question, get a retrieved set of nodes.\n",
|
||
"\n",
|
||
"We use the retriever to get a set of relevant nodes given a user query. These nodes will then be passed to the response synthesis modules below."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "621f274c-632a-4ee3-9144-a02db39292b8",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"query_str = (\n",
|
||
" \"Can you tell me about results from RLHF using both model-based and\"\n",
|
||
" \" human-based evaluation?\"\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2dadf393-48aa-40dc-84ba-ef0e5c08a4fc",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"retrieved_nodes = retriever.retrieve(query_str)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5d05b0c9-e8c4-43ec-8d45-0a7cacc80d63",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Building Response Synthesis with LLMs\n",
|
||
"\n",
|
||
"In this section we'll show how to use LLMs + Prompts to build a response synthesis module.\n",
|
||
"\n",
|
||
"We'll start from simple strategies (simply stuffing context into a prompt), to more advanced strategies that can handle context overflows."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "58ee8bff-46a4-4bd4-a7ea-883f06e89d4d",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 1. Try a Simple Prompt\n",
|
||
"\n",
|
||
"We first try to synthesize the response using a single input prompt + LLM call."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "f2dd50a9-4f79-409b-8554-120e8f4805e7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.llms.openai import OpenAI\n",
|
||
"from llama_index.core import PromptTemplate\n",
|
||
"\n",
|
||
"llm = OpenAI(model=\"text-davinci-003\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7c5fb0da-e2e6-4a55-be5f-db98df9fc916",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"qa_prompt = PromptTemplate(\n",
|
||
" \"\"\"\\\n",
|
||
"Context information is below.\n",
|
||
"---------------------\n",
|
||
"{context_str}\n",
|
||
"---------------------\n",
|
||
"Given the context information and not prior knowledge, answer the query.\n",
|
||
"Query: {query_str}\n",
|
||
"Answer: \\\n",
|
||
"\"\"\"\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4bd336a5-092f-4181-b8ab-4ca8c995a9c4",
|
||
"metadata": {},
|
||
"source": [
|
||
"Given an example question, retrieve the set of relevant nodes and try to put it all in the prompt, separated by newlines."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7ba8b04e-375c-42d8-923c-dceb2c5d5315",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"query_str = (\n",
|
||
" \"Can you tell me about results from RLHF using both model-based and\"\n",
|
||
" \" human-based evaluation?\"\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "13adede0-0690-4b54-8717-63e1e70f5c28",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"retrieved_nodes = retriever.retrieve(query_str)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "e4bcbce2-b0b4-425d-842f-07d48a4527c7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def generate_response(retrieved_nodes, query_str, qa_prompt, llm):\n",
|
||
" context_str = \"\\n\\n\".join([r.get_content() for r in retrieved_nodes])\n",
|
||
" fmt_qa_prompt = qa_prompt.format(\n",
|
||
" context_str=context_str, query_str=query_str\n",
|
||
" )\n",
|
||
" response = llm.complete(fmt_qa_prompt)\n",
|
||
" return str(response), fmt_qa_prompt"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "66828367-a725-44ea-ae2e-a1352c2bbfb2",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"response, fmt_qa_prompt = generate_response(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, llm\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "b9072c1c-aded-4b2d-98df-fdc096bb9e75",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"*****Response******:\n",
|
||
"\n",
|
||
"RLHF used both model-based and human-based evaluation to select the best-performing models among several ablations. Model-based evaluation was used to measure the robustness of the reward model by collecting a test set of prompts for both helpfulness and safety, and asking three annotators to judge the quality of the answers based on a 7-point Likert scale. Human evaluation was used to validate major model versions. Additionally, a more general reward was trained to ensure the measure wouldn't diverge from the human preferences. Results showed that the reward models were well calibrated with the human preference annotations.\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(f\"*****Response******:\\n{response}\\n\\n\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "bfa5705d-2a66-45aa-85dd-8e37c26574b7",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"*****Formatted Prompt*****:\n",
|
||
"Context information is below.\n",
|
||
"---------------------\n",
|
||
"3.4\n",
|
||
"RLHF Results\n",
|
||
"3.4.1\n",
|
||
"Model-Based Evaluation\n",
|
||
"Evaluating LLMs is a challenging open-research problem. Human evaluation, while a gold standard, can\n",
|
||
"be complicated by various HCI considerations (Clark et al., 2021; Gehrmann et al., 2023), and is not always\n",
|
||
"scalable. Thus, to select the best-performing models among several ablations at each iteration from RLHF-V1\n",
|
||
"to V5, we first observed the improvement of the rewards from the latest reward models, to save costs and\n",
|
||
"increase iteration speed. We later validated major model versions with human evaluations.\n",
|
||
"How Far Can Model-Based Evaluation Go?\n",
|
||
"To measure the robustness of our reward model, we collected\n",
|
||
"a test set of prompts for both helpfulness and safety, and asked three annotators to judge the quality of the\n",
|
||
"answers based on a 7-point Likert scale (the higher the better). We observe that our reward models overall\n",
|
||
"are well calibrated with our human preference annotations, as illustrated in Figure 29 in the appendix. This\n",
|
||
"confirms the relevance of using our reward as a point-wise metric, despite being trained with a Pairwise\n",
|
||
"Ranking Loss.\n",
|
||
"Still, as Goodhart’s Law states, when a measure becomes a target, it ceases to be a good measure. To ensure\n",
|
||
"our measure won’t diverge from the human preferences, we additionally used a more general reward, trained\n",
|
||
"17\n",
|
||
"\n",
|
||
"5\n",
|
||
"Discussion\n",
|
||
"Here, we discuss the interesting properties we have observed with RLHF (Section 5.1). We then discuss the\n",
|
||
"limitations of Llama 2-Chat (Section 5.2). Lastly, we present our strategy for responsibly releasing these\n",
|
||
"models (Section 5.3).\n",
|
||
"5.1\n",
|
||
"Learnings and Observations\n",
|
||
"Our tuning process revealed several interesting results, such as Llama 2-Chat’s abilities to temporally\n",
|
||
"organize its knowledge, or to call APIs for external tools.\n",
|
||
"SFT (Mix)\n",
|
||
"SFT (Annotation)\n",
|
||
"RLHF (V1)\n",
|
||
"0.0\n",
|
||
"0.2\n",
|
||
"0.4\n",
|
||
"0.6\n",
|
||
"0.8\n",
|
||
"1.0\n",
|
||
"Reward Model Score\n",
|
||
"RLHF (V2)\n",
|
||
"Figure 20: Distribution shift for progressive versions of Llama 2-Chat, from SFT models towards RLHF.\n",
|
||
"Beyond Human Supervision.\n",
|
||
"At the outset of the project, many among us expressed a preference for\n",
|
||
"supervised annotation, attracted by its denser signal. Meanwhile reinforcement learning, known for its insta-\n",
|
||
"bility, seemed a somewhat shadowy field for those in the NLP research community. However, reinforcement\n",
|
||
"learning proved highly effective, particularly given its cost and time effectiveness. Our findings underscore\n",
|
||
"that the crucial determinant of RLHF’s success lies in the synergy it fosters between humans and LLMs\n",
|
||
"throughout the annotation process.\n",
|
||
"Even with proficient annotators, each individual writes with significant variation. A model fine-tuned on\n",
|
||
"SFT annotation learns this diversity, including, unfortunately, the tail-end of poorly executed annotation. Fur-\n",
|
||
"thermore, the model’s performance is capped by the writing abilities of the most skilled annotators. Human\n",
|
||
"annotators are arguably less subject to discrepancy when comparing two outputs’ preference annotation\n",
|
||
"for RLHF. Consequently, the reward mechanism swiftly learns to assign low scores to undesirable tail-end\n",
|
||
"distribution and aligns towards the human preference. This phenomena is illustrated in Figure 20, where we\n",
|
||
"can see that the worst answers are progressively removed, shifting the distribution to the right.\n",
|
||
"In addition, during annotation, the model has the potential to venture into writing trajectories that even the\n",
|
||
"best annotators may not chart. Nonetheless, humans can still provide valuable feedback when comparing two\n",
|
||
"answers, beyond their own writing competencies. Drawing a parallel, while we may not all be accomplished\n",
|
||
"artists, our ability to appreciate and critique art remains intact. We posit that the superior writing abilities of\n",
|
||
"LLMs, as manifested in surpassing human annotators in certain tasks, are fundamentally driven by RLHF, as\n",
|
||
"documented in Gilardi et al. (2023) and Huang et al. (2023). Supervised data may no longer be the gold\n",
|
||
"standard, and this evolving circumstance compels a re-evaluation of the concept of “supervision.”\n",
|
||
"In-Context Temperature Rescaling.\n",
|
||
"We have observed an intriguing phenomenon related to RLHF, a feature\n",
|
||
"not previously reported to the best of our knowledge: the dynamic re-scaling of temperature contingent upon\n",
|
||
"the context. As indicated in Figure 8, the temperature appears to be influenced by RLHF. Yet, intriguingly,\n",
|
||
"our findings also revealed that the shifts are not uniformly applied across all prompts, as shown in Figure 21.\n",
|
||
"For instance, when it comes to prompts associated with creativity, such as “Write a poem,” an increase in\n",
|
||
"temperature continues to generate diversity across our various RLHF iterations. This can be observed in the\n",
|
||
"Self-BLEU slope, which mirrors a pattern comparable to that of the SFT model.\n",
|
||
"On the other hand, for prompts based on factual information, such as “What is the capital of ?” the Self-BLEU\n",
|
||
"slope diminishes over time. This pattern suggests that despite the rising temperature, the model learns to\n",
|
||
"consistently provide the same response to factual prompts.\n",
|
||
"32\n",
|
||
"---------------------\n",
|
||
"Given the context information and not prior knowledge, answer the query.\n",
|
||
"Query: Can you tell me about results from RLHF using both model-based and human-based evaluation?\n",
|
||
"Answer: \n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(f\"*****Formatted Prompt*****:\\n{fmt_qa_prompt}\\n\\n\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7544d146-76f5-46b2-a3e6-4fbc6b53c138",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Problem**: What if we set the top-k retriever to a higher value? The context would overflow!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7fe998c5-5bad-4ca6-bffb-1104ae1daee4",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"retriever = index.as_retriever(similarity_top_k=6)\n",
|
||
"retrieved_nodes = retriever.retrieve(query_str)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "36185ec0-6aa9-4710-a007-357f87d0ce3a",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"ename": "ValueError",
|
||
"evalue": "The prompt is too long for the model. Please use a prompt that is less than 4097 tokens.",
|
||
"output_type": "error",
|
||
"traceback": [
|
||
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
||
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
|
||
"Cell \u001b[0;32mIn[34], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m response, fmt_qa_prompt \u001b[38;5;241m=\u001b[39m \u001b[43mgenerate_response\u001b[49m\u001b[43m(\u001b[49m\u001b[43mretrieved_nodes\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mquery_str\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mqa_prompt\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mllm\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mResponse (k=5): \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mresponse\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m)\n",
|
||
"Cell \u001b[0;32mIn[16], line 4\u001b[0m, in \u001b[0;36mgenerate_response\u001b[0;34m(retrieved_nodes, query_str, qa_prompt, llm)\u001b[0m\n\u001b[1;32m 2\u001b[0m context_str \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mjoin([r\u001b[38;5;241m.\u001b[39mget_content() \u001b[38;5;28;01mfor\u001b[39;00m r \u001b[38;5;129;01min\u001b[39;00m retrieved_nodes])\n\u001b[1;32m 3\u001b[0m fmt_qa_prompt \u001b[38;5;241m=\u001b[39m qa_prompt\u001b[38;5;241m.\u001b[39mformat(context_str\u001b[38;5;241m=\u001b[39mcontext_str, query_str\u001b[38;5;241m=\u001b[39mquery_str)\n\u001b[0;32m----> 4\u001b[0m response \u001b[38;5;241m=\u001b[39m \u001b[43mllm\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcomplete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfmt_qa_prompt\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mstr\u001b[39m(response), fmt_qa_prompt\n",
|
||
"File \u001b[0;32m~/Programming/gpt_index/llama_index/llms/base.py:277\u001b[0m, in \u001b[0;36mllm_completion_callback.<locals>.wrap.<locals>.wrapped_llm_predict\u001b[0;34m(_self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 267\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m wrapper_logic(_self) \u001b[38;5;28;01mas\u001b[39;00m callback_manager:\n\u001b[1;32m 268\u001b[0m event_id \u001b[38;5;241m=\u001b[39m callback_manager\u001b[38;5;241m.\u001b[39mon_event_start(\n\u001b[1;32m 269\u001b[0m CBEventType\u001b[38;5;241m.\u001b[39mLLM,\n\u001b[1;32m 270\u001b[0m payload\u001b[38;5;241m=\u001b[39m{\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 274\u001b[0m },\n\u001b[1;32m 275\u001b[0m )\n\u001b[0;32m--> 277\u001b[0m f_return_val \u001b[38;5;241m=\u001b[39m \u001b[43mf\u001b[49m\u001b[43m(\u001b[49m\u001b[43m_self\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 278\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(f_return_val, Generator):\n\u001b[1;32m 279\u001b[0m \u001b[38;5;66;03m# intercept the generator and add a callback to the end\u001b[39;00m\n\u001b[1;32m 280\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mwrapped_gen\u001b[39m() \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m CompletionResponseGen:\n",
|
||
"File \u001b[0;32m~/Programming/gpt_index/llama_index/llms/openai.py:144\u001b[0m, in \u001b[0;36mOpenAI.complete\u001b[0;34m(self, prompt, **kwargs)\u001b[0m\n\u001b[1;32m 142\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 143\u001b[0m complete_fn \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_complete\n\u001b[0;32m--> 144\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mcomplete_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[43mprompt\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
|
||
"File \u001b[0;32m~/Programming/gpt_index/llama_index/llms/openai.py:281\u001b[0m, in \u001b[0;36mOpenAI._complete\u001b[0;34m(self, prompt, **kwargs)\u001b[0m\n\u001b[1;32m 278\u001b[0m all_kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_get_all_kwargs(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m 279\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmax_tokens \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 280\u001b[0m \u001b[38;5;66;03m# NOTE: non-chat completion endpoint requires max_tokens to be set\u001b[39;00m\n\u001b[0;32m--> 281\u001b[0m max_tokens \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_get_max_token_for_prompt\u001b[49m\u001b[43m(\u001b[49m\u001b[43mprompt\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 282\u001b[0m all_kwargs[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmax_tokens\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m max_tokens\n\u001b[1;32m 284\u001b[0m response \u001b[38;5;241m=\u001b[39m completion_with_retry(\n\u001b[1;32m 285\u001b[0m is_chat_model\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_is_chat_model,\n\u001b[1;32m 286\u001b[0m max_retries\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmax_retries,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 289\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mall_kwargs,\n\u001b[1;32m 290\u001b[0m )\n",
|
||
"File \u001b[0;32m~/Programming/gpt_index/llama_index/llms/openai.py:343\u001b[0m, in \u001b[0;36mOpenAI._get_max_token_for_prompt\u001b[0;34m(self, prompt)\u001b[0m\n\u001b[1;32m 341\u001b[0m max_token \u001b[38;5;241m=\u001b[39m context_window \u001b[38;5;241m-\u001b[39m \u001b[38;5;28mlen\u001b[39m(tokens)\n\u001b[1;32m 342\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m max_token \u001b[38;5;241m<\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[0;32m--> 343\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 344\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThe prompt is too long for the model. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 345\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPlease use a prompt that is less than \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mcontext_window\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m tokens.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 346\u001b[0m )\n\u001b[1;32m 347\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m max_token\n",
|
||
"\u001b[0;31mValueError\u001b[0m: The prompt is too long for the model. Please use a prompt that is less than 4097 tokens."
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"response, fmt_qa_prompt = generate_response(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, llm\n",
|
||
")\n",
|
||
"print(f\"Response (k=5): {response}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "235c18a5-e401-4564-a02a-56b4a50541af",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2. Try a \"Create and Refine\" strategy\n",
|
||
"\n",
|
||
"To deal with context overflows, we can try a strategy where we synthesize a response sequentially through all nodes. Start with the first node and generate an initial response. Then for subsequent nodes, refine the answer using additional context.\n",
|
||
"\n",
|
||
"This requires us to define a \"refine\" prompt as well."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "287e777c-1e40-4b31-92a1-31a402cf3e20",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"refine_prompt = PromptTemplate(\n",
|
||
" \"\"\"\\\n",
|
||
"The original query is as follows: {query_str}\n",
|
||
"We have provided an existing answer: {existing_answer}\n",
|
||
"We have the opportunity to refine the existing answer \\\n",
|
||
"(only if needed) with some more context below.\n",
|
||
"------------\n",
|
||
"{context_str}\n",
|
||
"------------\n",
|
||
"Given the new context, refine the original answer to better answer the query. \\\n",
|
||
"If the context isn't useful, return the original answer.\n",
|
||
"Refined Answer: \\\n",
|
||
"\"\"\"\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "bdfeef5e-1e39-4ec5-abf8-7d120ad18914",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.core.response.notebook_utils import display_source_node\n",
|
||
"\n",
|
||
"\n",
|
||
"def generate_response_cr(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, refine_prompt, llm\n",
|
||
"):\n",
|
||
" \"\"\"Generate a response using create and refine strategy.\n",
|
||
"\n",
|
||
" The first node uses the 'QA' prompt.\n",
|
||
" All subsequent nodes use the 'refine' prompt.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" cur_response = None\n",
|
||
" fmt_prompts = []\n",
|
||
" for idx, node in enumerate(retrieved_nodes):\n",
|
||
" print(f\"[Node {idx}]\")\n",
|
||
" display_source_node(node, source_length=2000)\n",
|
||
" context_str = node.get_content()\n",
|
||
" if idx == 0:\n",
|
||
" fmt_prompt = qa_prompt.format(\n",
|
||
" context_str=context_str, query_str=query_str\n",
|
||
" )\n",
|
||
" else:\n",
|
||
" fmt_prompt = refine_prompt.format(\n",
|
||
" context_str=context_str,\n",
|
||
" query_str=query_str,\n",
|
||
" existing_answer=str(cur_response),\n",
|
||
" )\n",
|
||
"\n",
|
||
" cur_response = llm.complete(fmt_prompt)\n",
|
||
" fmt_prompts.append(fmt_prompt)\n",
|
||
"\n",
|
||
" return str(cur_response), fmt_prompts"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "60d7baea-1a29-4c51-abb7-9b5c9cebae2b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"response, fmt_prompts = generate_response_cr(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, refine_prompt, llm\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "c9ff1b2e-1fa0-4493-9e6d-1cb17e010879",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"print(str(response))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "09f469f9-7265-48cc-b240-a2a82119ea0c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# view a sample qa prompt\n",
|
||
"print(fmt_prompts[0])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "9cf2caf4-6751-4de1-87ec-1d4d6035fa62",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# view a sample refine prompt\n",
|
||
"print(fmt_prompts[1])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5a098682-5686-45b2-b0ef-2a99c9a09159",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Observation**: This is an initial step, but obviously there are inefficiencies. One is the fact that it's quite slow - we make sequential calls. The second piece is that each LLM call is inefficient - we are only inserting a single node, but not \"stuffing\" the prompt with as much context as necessary."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "82dd0f2d-82fa-4005-8182-ffe8fbede127",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 3. Try a Hierarchical Summarization Strategy\n",
|
||
"\n",
|
||
"Another approach is to try a hierarchical summarization strategy. We generate an answer for each node independently, and then hierarchically combine the answers. This \"combine\" step could happen once, or for maximum generality can happen recursively until there is one \"root\" node. That \"root\" node is then returned as the answer.\n",
|
||
"\n",
|
||
"We implement this approach below. We have a fixed number of children of 5, so we hierarchically combine 5 children at a time.\n",
|
||
"\n",
|
||
"**NOTE**: In LlamaIndex this is referred to as \"tree_summarize\", in LangChain this is referred to as map-reduce."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "54253c22-0a16-496f-a138-88cf034fba56",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"def combine_results(\n",
|
||
" texts,\n",
|
||
" query_str,\n",
|
||
" qa_prompt,\n",
|
||
" llm,\n",
|
||
" cur_prompt_list,\n",
|
||
" num_children=10,\n",
|
||
"):\n",
|
||
" new_texts = []\n",
|
||
" for idx in range(0, len(texts), num_children):\n",
|
||
" text_batch = texts[idx : idx + num_children]\n",
|
||
" context_str = \"\\n\\n\".join([t for t in text_batch])\n",
|
||
" fmt_qa_prompt = qa_prompt.format(\n",
|
||
" context_str=context_str, query_str=query_str\n",
|
||
" )\n",
|
||
" combined_response = llm.complete(fmt_qa_prompt)\n",
|
||
" new_texts.append(str(combined_response))\n",
|
||
" cur_prompt_list.append(fmt_qa_prompt)\n",
|
||
"\n",
|
||
" if len(new_texts) == 1:\n",
|
||
" return new_texts[0]\n",
|
||
" else:\n",
|
||
" return combine_results(\n",
|
||
" new_texts, query_str, qa_prompt, llm, num_children=num_children\n",
|
||
" )\n",
|
||
"\n",
|
||
"\n",
|
||
"def generate_response_hs(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, llm, num_children=10\n",
|
||
"):\n",
|
||
" \"\"\"Generate a response using hierarchical summarization strategy.\n",
|
||
"\n",
|
||
" Combine num_children nodes hierarchically until we get one root node.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" fmt_prompts = []\n",
|
||
" node_responses = []\n",
|
||
" for node in retrieved_nodes:\n",
|
||
" context_str = node.get_content()\n",
|
||
" fmt_qa_prompt = qa_prompt.format(\n",
|
||
" context_str=context_str, query_str=query_str\n",
|
||
" )\n",
|
||
" node_response = llm.complete(fmt_qa_prompt)\n",
|
||
" node_responses.append(node_response)\n",
|
||
" fmt_prompts.append(fmt_qa_prompt)\n",
|
||
"\n",
|
||
" response_txt = combine_results(\n",
|
||
" [str(r) for r in node_responses],\n",
|
||
" query_str,\n",
|
||
" qa_prompt,\n",
|
||
" llm,\n",
|
||
" fmt_prompts,\n",
|
||
" num_children=num_children,\n",
|
||
" )\n",
|
||
"\n",
|
||
" return response_txt, fmt_prompts"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "ea40a9e5-f83d-4082-a722-6c25321dbbbf",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"response, fmt_prompts = generate_response_hs(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, llm\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "f80c031c-3f17-42e5-955c-80d14558a12e",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"The results from RLHF using both model-based and human-based evaluation showed that Llama 2-Chat models outperformed open-source models by a significant margin on both single turn and multi-turn prompts. For human-based evaluation, we compared Llama 2-Chat models to open-source models and closed-source models on over 4,000 single and multi-turn prompts. The results showed that Llama 2-Chat models outperformed the other models by a significant margin on both single turn and multi-turn prompts. The human preference annotation agreement rate was also higher on more distinct responses than similar pairs. The largest RLHF model was competitive with ChatGPT, with a win rate of 36% and a tie rate of 31.5% relative to ChatGPT. RLHF 70B model also outperformed PaLM-bison chat model by a large percentage on the prompt set.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(str(response))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f1dbe8f4-8ae6-4079-9a00-36e72d9b5c33",
|
||
"metadata": {},
|
||
"source": [
|
||
"**Observation**: Note that the answer is much more concise than the create-and-refine approach. This is a well-known phemonenon - the reason is because hierarchical summarization tends to compress information at each stage, whereas create and refine encourages adding on more information with each node.\n",
|
||
"\n",
|
||
"**Observation**: Similar to the above section, there are inefficiencies. We are still generating an answer for each node independently that we can try to optimize away.\n",
|
||
"\n",
|
||
"Our `ResponseSynthesizer` module handles this!"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1e44d731-1841-47fc-baae-a280ad1ca10e",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### 4. [Optional] Let's create an async version of hierarchical summarization!\n",
|
||
"\n",
|
||
"A pro of the hierarchical summarization approach is that the LLM calls can be parallelized, leading to big speedups in response synthesis.\n",
|
||
"\n",
|
||
"We implement an async version below. We use asyncio.gather to execute coroutines (LLM calls) for each Node concurrently."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "eb0f7a23-2234-4540-b4f1-010b45f9d6f7",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import nest_asyncio\n",
|
||
"import asyncio\n",
|
||
"\n",
|
||
"nest_asyncio.apply()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "63f6719b-ea05-442e-88c5-35f3d0f60501",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"async def acombine_results(\n",
|
||
" texts,\n",
|
||
" query_str,\n",
|
||
" qa_prompt,\n",
|
||
" llm,\n",
|
||
" cur_prompt_list,\n",
|
||
" num_children=10,\n",
|
||
"):\n",
|
||
" fmt_prompts = []\n",
|
||
" for idx in range(0, len(texts), num_children):\n",
|
||
" text_batch = texts[idx : idx + num_children]\n",
|
||
" context_str = \"\\n\\n\".join([t for t in text_batch])\n",
|
||
" fmt_qa_prompt = qa_prompt.format(\n",
|
||
" context_str=context_str, query_str=query_str\n",
|
||
" )\n",
|
||
" fmt_prompts.append(fmt_qa_prompt)\n",
|
||
" cur_prompt_list.append(fmt_qa_prompt)\n",
|
||
"\n",
|
||
" tasks = [llm.acomplete(p) for p in fmt_prompts]\n",
|
||
" combined_responses = await asyncio.gather(*tasks)\n",
|
||
" new_texts = [str(r) for r in combined_responses]\n",
|
||
"\n",
|
||
" if len(new_texts) == 1:\n",
|
||
" return new_texts[0]\n",
|
||
" else:\n",
|
||
" return await acombine_results(\n",
|
||
" new_texts, query_str, qa_prompt, llm, num_children=num_children\n",
|
||
" )\n",
|
||
"\n",
|
||
"\n",
|
||
"async def agenerate_response_hs(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, llm, num_children=10\n",
|
||
"):\n",
|
||
" \"\"\"Generate a response using hierarchical summarization strategy.\n",
|
||
"\n",
|
||
" Combine num_children nodes hierarchically until we get one root node.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" fmt_prompts = []\n",
|
||
" node_responses = []\n",
|
||
" for node in retrieved_nodes:\n",
|
||
" context_str = node.get_content()\n",
|
||
" fmt_qa_prompt = qa_prompt.format(\n",
|
||
" context_str=context_str, query_str=query_str\n",
|
||
" )\n",
|
||
" fmt_prompts.append(fmt_qa_prompt)\n",
|
||
"\n",
|
||
" tasks = [llm.acomplete(p) for p in fmt_prompts]\n",
|
||
" node_responses = await asyncio.gather(*tasks)\n",
|
||
"\n",
|
||
" response_txt = combine_results(\n",
|
||
" [str(r) for r in node_responses],\n",
|
||
" query_str,\n",
|
||
" qa_prompt,\n",
|
||
" llm,\n",
|
||
" fmt_prompts,\n",
|
||
" num_children=num_children,\n",
|
||
" )\n",
|
||
"\n",
|
||
" return response_txt, fmt_prompts"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "2331a426-45b6-4302-afe4-1f4ba6999701",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"response, fmt_prompts = await agenerate_response_hs(\n",
|
||
" retrieved_nodes, query_str, qa_prompt, llm\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "3d0965c7-e164-4a65-a0a7-4907abdbbd81",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
" Results from RLHF using both model-based and human-based evaluation show that larger models generally obtain higher performance for a similar volume of data. Additionally, the accuracy on more distinct responses matters the most to improve Llama 2-Chat performance. The human preference annotation agreement rate is also higher on more distinct responses than similar pairs. Furthermore, two main algorithms were explored for RLHF fine-tuning: Proximal Policy Optimization (PPO) and Rejection Sampling fine-tuning. The largest Llama 2-Chat model was found to be competitive with ChatGPT, with a win rate of 36% and a tie rate of 31.5% relative to ChatGPT. Additionally, Llama 2-Chat 70B model outperformed PaLM-bison chat model by a large percentage on our prompt set. Inter-Rater Reliability (IRR) was measured using Gwet’s AC1/2 statistic, with scores varying between 0.37 and 0.55 depending on the specific model comparison.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(str(response))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "90db6801-ff13-4fe7-84cc-3c49701788e2",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Let's put it all together!\n",
|
||
"\n",
|
||
"Let's define a simple query engine that can be initialized with a retriever, prompt, llm etc. And have it implement a simple `query` function. We also implement an async version, can be used if you completed part 4 above! \n",
|
||
"\n",
|
||
"**NOTE**: We skip subclassing our own `QueryEngine` abstractions. This is a big TODO to make it more easily sub-classable! "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "0f260550-51e5-43f7-aef7-1375546e303c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from llama_index.core.retrievers import BaseRetriever\n",
|
||
"from llama_index.core.llms import LLM\n",
|
||
"from dataclasses import dataclass\n",
|
||
"from typing import Optional, List\n",
|
||
"\n",
|
||
"\n",
|
||
"@dataclass\n",
|
||
"class Response:\n",
|
||
" response: str\n",
|
||
" source_nodes: Optional[List] = None\n",
|
||
"\n",
|
||
" def __str__(self):\n",
|
||
" return self.response\n",
|
||
"\n",
|
||
"\n",
|
||
"class MyQueryEngine:\n",
|
||
" \"\"\"My query engine.\n",
|
||
"\n",
|
||
" Uses the tree summarize response synthesis module by default.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" def __init__(\n",
|
||
" self,\n",
|
||
" retriever: BaseRetriever,\n",
|
||
" qa_prompt: PromptTemplate,\n",
|
||
" llm: LLM,\n",
|
||
" num_children=10,\n",
|
||
" ) -> None:\n",
|
||
" self._retriever = retriever\n",
|
||
" self._qa_prompt = qa_prompt\n",
|
||
" self._llm = llm\n",
|
||
" self._num_children = num_children\n",
|
||
"\n",
|
||
" def query(self, query_str: str):\n",
|
||
" retrieved_nodes = self._retriever.retrieve(query_str)\n",
|
||
" response_txt, _ = generate_response_hs(\n",
|
||
" retrieved_nodes,\n",
|
||
" query_str,\n",
|
||
" self._qa_prompt,\n",
|
||
" self._llm,\n",
|
||
" num_children=self._num_children,\n",
|
||
" )\n",
|
||
" response = Response(response_txt, source_nodes=retrieved_nodes)\n",
|
||
" return response\n",
|
||
"\n",
|
||
" async def aquery(self, query_str: str):\n",
|
||
" retrieved_nodes = await self._retriever.aretrieve(query_str)\n",
|
||
" response_txt, _ = await agenerate_response_hs(\n",
|
||
" retrieved_nodes,\n",
|
||
" query_str,\n",
|
||
" self._qa_prompt,\n",
|
||
" self._llm,\n",
|
||
" num_children=self._num_children,\n",
|
||
" )\n",
|
||
" response = Response(response_txt, source_nodes=retrieved_nodes)\n",
|
||
" return response"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "b84e67a4-0f86-41ec-9bbe-f0c9e43617d6",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"query_engine = MyQueryEngine(retriever, qa_prompt, llm, num_children=10)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "86d5b88c-b6e1-4be2-a3ee-6f51334a48f3",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"response = query_engine.query(query_str)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "60f609b0-f8c4-42f6-b287-eb68ad8ebef4",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"The results from RLHF using both model-based and human-based evaluation showed that larger models generally obtained higher performance for a similar volume of data. The accuracy on more distinct responses was higher than on similar pairs, indicating that learning to model human preferences becomes challenging when deciding between two similar model responses. Additionally, the largest Llama 2-Chat model was found to be competitive with ChatGPT, with a win rate of 36% and a tie rate of 31.5% relative to ChatGPT. Llama 2-Chat 70B model was also found to outperform PaLM-bison chat model by a large percentage on the prompt set. Inter-Rater Reliability (IRR) was measured using Gwet’s AC1/2 statistic, with scores varying between 0.37 and 0.55 depending on the specific model comparison.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(str(response))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "865557d4-dc19-45c1-8499-fdcb88a7bfc1",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"response = await query_engine.aquery(query_str)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "1acd73d7-0d63-4094-ab3b-988f02492a8d",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"The results from RLHF using both model-based and human-based evaluation showed that larger models generally obtained higher performance for a similar volume of data. The accuracy on more distinct responses was higher than on similar pairs, indicating that learning to model human preferences becomes challenging when deciding between two similar model responses. Additionally, the largest Llama 2-Chat model was found to be competitive with ChatGPT, with a win rate of 36% and a tie rate of 31.5%. Human evaluations were conducted using a 7-point Likert scale helpfulness task, with Gwet’s AC2 score varying between 0.37 and 0.55 depending on the specific model comparison.\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"print(str(response))"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "llama_index_v2",
|
||
"language": "python",
|
||
"name": "llama_index_v2"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|