wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
739 lines
28 KiB
Plaintext
739 lines
28 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/observability/UpTrainCallback.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# UpTrain Callback Handler\n",
|
|
"\n",
|
|
"UpTrain ([github](https://github.com/uptrain-ai/uptrain) || [website](https://github.com/uptrain-ai/uptrain/) || [docs](https://docs.uptrain.ai/)) is an open-source platform to evaluate and improve GenAI applications. It provides grades for 20+ preconfigured checks (covering language, code, embedding use cases), performs root cause analysis on failure cases and gives insights on how to resolve them. \n",
|
|
"\n",
|
|
"This notebook showcases how to use UpTrain Callback Handler to evaluate different components of your RAG pipelines.\n",
|
|
"\n",
|
|
"## 1. **RAG Query Engine Evaluations**:\n",
|
|
"The RAG query engine plays a crucial role in retrieving context and generating responses. To ensure its performance and response quality, we conduct the following evaluations:\n",
|
|
"\n",
|
|
"- **[Context Relevance](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-relevance)**: Determines if the retrieved context has sufficient information to answer the user query or not.\n",
|
|
"- **[Factual Accuracy](https://docs.uptrain.ai/predefined-evaluations/context-awareness/factual-accuracy)**: Assesses if the LLM's response can be verified via the retrieved context.\n",
|
|
"- **[Response Completeness](https://docs.uptrain.ai/predefined-evaluations/response-quality/response-completeness)**: Checks if the response contains all the information required to answer the user query comprehensively.\n",
|
|
"\n",
|
|
"## 2. **Sub-Question Query Generation Evaluation**:\n",
|
|
"The SubQuestionQueryGeneration operator decomposes a question into sub-questions, generating responses for each using an RAG query engine. To measure it's accuracy, we use:\n",
|
|
"\n",
|
|
"- **[Sub Query Completeness](https://docs.uptrain.ai/predefined-evaluations/query-quality/sub-query-completeness)**: Assures that the sub-questions accurately and comprehensively cover the original query.\n",
|
|
"\n",
|
|
"## 3. **Re-Ranking Evaluations**:\n",
|
|
"Re-ranking involves reordering nodes based on relevance to the query and choosing the top nodes. Different evaluations are performed based on the number of nodes returned after re-ranking.\n",
|
|
"\n",
|
|
"a. Same Number of Nodes\n",
|
|
"- **[Context Reranking](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-reranking)**: Checks if the order of re-ranked nodes is more relevant to the query than the original order.\n",
|
|
"\n",
|
|
"b. Different Number of Nodes:\n",
|
|
"- **[Context Conciseness](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-conciseness)**: Examines whether the reduced number of nodes still provides all the required information.\n",
|
|
"\n",
|
|
"These evaluations collectively ensure the robustness and effectiveness of the RAG query engine, SubQuestionQueryGeneration operator, and the re-ranking process in the LlamaIndex pipeline."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### **Note:** \n",
|
|
"- We have performed evaluations using basic RAG query engine, the same evaluations can be performed using the advanced RAG query engine as well.\n",
|
|
"- Same is true for Re-Ranking evaluations, we have performed evaluations using SentenceTransformerRerank, the same evaluations can be performed using other re-rankers as well."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Install Dependencies and Import Libraries\n",
|
|
"\n",
|
|
"Install notebook dependencies."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index-readers-web\n",
|
|
"%pip install llama-index-callbacks-uptrain\n",
|
|
"%pip install -q html2text llama-index pandas tqdm uptrain torch sentence-transformers"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Import libraries.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from getpass import getpass\n",
|
|
"\n",
|
|
"from llama_index.core import Settings, VectorStoreIndex\n",
|
|
"from llama_index.core.node_parser import SentenceSplitter\n",
|
|
"from llama_index.readers.web import SimpleWebPageReader\n",
|
|
"from llama_index.core.callbacks import CallbackManager\n",
|
|
"from llama_index.callbacks.uptrain.base import UpTrainCallbackHandler\n",
|
|
"from llama_index.core.query_engine import SubQuestionQueryEngine\n",
|
|
"from llama_index.core.tools import QueryEngineTool, ToolMetadata\n",
|
|
"from llama_index.core.postprocessor import SentenceTransformerRerank\n",
|
|
"\n",
|
|
"import os"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setup\n",
|
|
"\n",
|
|
"UpTrain provides you with:\n",
|
|
"1. Dashboards with advanced drill-down and filtering options\n",
|
|
"1. Insights and common topics among failing cases\n",
|
|
"1. Observability and real-time monitoring of production data\n",
|
|
"1. Regression testing via seamless integration with your CI/CD pipelines\n",
|
|
"\n",
|
|
"You can choose between the following options for evaluating using UpTrain:\n",
|
|
"### 1. **UpTrain's Open-Source Software (OSS)**: \n",
|
|
"You can use the open-source evaluation service to evaluate your model. In this case, you will need to provide an OpenAI API key. You can get yours [here](https://platform.openai.com/account/api-keys).\n",
|
|
"\n",
|
|
"In order to view your evaluations in the UpTrain dashboard, you will need to set it up by running the following commands in your terminal:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"git clone https://github.com/uptrain-ai/uptrain\n",
|
|
"cd uptrain\n",
|
|
"bash run_uptrain.sh\n",
|
|
"```\n",
|
|
"\n",
|
|
"This will start the UpTrain dashboard on your local machine. You can access it at `http://localhost:3000/dashboard`.\n",
|
|
"\n",
|
|
"Parameters:\n",
|
|
"- key_type=\"openai\"\n",
|
|
"- api_key=\"OPENAI_API_KEY\"\n",
|
|
"- project_name=\"PROJECT_NAME\"\n",
|
|
"\n",
|
|
"\n",
|
|
"### 2. **UpTrain Managed Service and Dashboards**:\n",
|
|
"Alternatively, you can use UpTrain's managed service to evaluate your model. You can create a free UpTrain account [here](https://uptrain.ai/) and get free trial credits. If you want more trial credits, [book a call with the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min).\n",
|
|
"\n",
|
|
"The benefits of using the managed service are:\n",
|
|
"1. No need to set up the UpTrain dashboard on your local machine.\n",
|
|
"1. Access to many LLMs without needing their API keys.\n",
|
|
"\n",
|
|
"Once you perform the evaluations, you can view them in the UpTrain dashboard at `https://dashboard.uptrain.ai/dashboard`\n",
|
|
"\n",
|
|
"Parameters:\n",
|
|
"- key_type=\"uptrain\"\n",
|
|
"- api_key=\"UPTRAIN_API_KEY\"\n",
|
|
"- project_name=\"PROJECT_NAME\"\n",
|
|
"\n",
|
|
"\n",
|
|
"**Note:** The `project_name` will be the project name under which the evaluations performed will be shown in the UpTrain dashboard."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Create the UpTrain Callback Handler"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"os.environ[\"OPENAI_API_KEY\"] = getpass()\n",
|
|
"\n",
|
|
"callback_handler = UpTrainCallbackHandler(\n",
|
|
" key_type=\"openai\",\n",
|
|
" api_key=os.environ[\"OPENAI_API_KEY\"],\n",
|
|
" project_name=\"uptrain_llamaindex\",\n",
|
|
")\n",
|
|
"\n",
|
|
"Settings.callback_manager = CallbackManager([callback_handler])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Load and Parse Documents\n",
|
|
"\n",
|
|
"Load documents from Paul Graham's essay \"What I Worked On\"."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"documents = SimpleWebPageReader().load_data(\n",
|
|
" [\n",
|
|
" \"https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt\"\n",
|
|
" ]\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Parse the document into nodes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"parser = SentenceSplitter()\n",
|
|
"nodes = parser.get_nodes_from_documents(documents)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 1. RAG Query Engine Evaluation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"UpTrain callback handler will automatically capture the query, context and response once generated and will run the following three evaluations *(Graded from 0 to 1)* on the response:\n",
|
|
"- **[Context Relevance](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-relevance)**: Determines if the retrieved context has sufficient information to answer the user query or not.\n",
|
|
"- **[Factual Accuracy](https://docs.uptrain.ai/predefined-evaluations/context-awareness/factual-accuracy)**: Assesses if the LLM's response can be verified via the retrieved context.\n",
|
|
"- **[Response Completeness](https://docs.uptrain.ai/predefined-evaluations/response-quality/response-completeness)**: Checks if the response contains all the information required to answer the user query comprehensively."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.33s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.36s/it]\n",
|
|
"100%|██████████| 1/1 [00:03<00:00, 3.50s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.32s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What did Paul Graham do growing up?\n",
|
|
"Response: Growing up, Paul Graham worked on writing short stories and programming. He started programming on an IBM 1401 in 9th grade using an early version of Fortran. Later, he got a TRS-80 computer and wrote simple games, a rocket prediction program, and a word processor. Despite his interest in programming, he initially planned to study philosophy in college before eventually switching to AI.\n",
|
|
"\n",
|
|
"Context Relevance Score: 0.0\n",
|
|
"Factual Accuracy Score: 1.0\n",
|
|
"Response Completeness Score: 1.0\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.59s/it]\n",
|
|
"100%|██████████| 1/1 [00:00<00:00, 1.01it/s]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.76s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.28s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: When and how did Paul Graham's mother die?\n",
|
|
"Response: Paul Graham's mother died when he was 18 years old, from a brain tumor.\n",
|
|
"\n",
|
|
"Context Relevance Score: 0.0\n",
|
|
"Factual Accuracy Score: 0.0\n",
|
|
"Response Completeness Score: 0.5\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.75s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.55s/it]\n",
|
|
"100%|██████████| 1/1 [00:03<00:00, 3.39s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.48s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What, in Paul Graham's opinion, is the most distinctive thing about YC?\n",
|
|
"Response: The most distinctive thing about Y Combinator, according to Paul Graham, is that instead of deciding for himself what to work on, the problems come to him. Every 6 months, a new batch of startups brings their problems, which then become the focus of YC. This engagement with a variety of startup problems and the direct involvement in solving them is what Graham finds most unique about Y Combinator.\n",
|
|
"\n",
|
|
"Context Relevance Score: 1.0\n",
|
|
"Factual Accuracy Score: 0.3333333333333333\n",
|
|
"Response Completeness Score: 1.0\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.92s/it]\n",
|
|
"100%|██████████| 1/1 [00:00<00:00, 1.20it/s]\n",
|
|
"100%|██████████| 1/1 [00:02<00:00, 2.15s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.08s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: When and how did Paul Graham meet Jessica Livingston?\n",
|
|
"Response: Paul Graham met Jessica Livingston at a big party at his house in October 2003.\n",
|
|
"\n",
|
|
"Context Relevance Score: 1.0\n",
|
|
"Factual Accuracy Score: 0.5\n",
|
|
"Response Completeness Score: 1.0\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.82s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.14s/it]\n",
|
|
"100%|██████████| 1/1 [00:03<00:00, 3.19s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.50s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What is Bel, and when and where was it written?\n",
|
|
"Response: Bel is a new Lisp that was written in Arc. It was developed over a period of 4 years, from March 26, 2015 to October 12, 2019. The majority of Bel was written in England.\n",
|
|
"\n",
|
|
"Context Relevance Score: 1.0\n",
|
|
"Factual Accuracy Score: 1.0\n",
|
|
"Response Completeness Score: 1.0\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" documents,\n",
|
|
")\n",
|
|
"query_engine = index.as_query_engine()\n",
|
|
"\n",
|
|
"max_characters_per_line = 80\n",
|
|
"queries = [\n",
|
|
" \"What did Paul Graham do growing up?\",\n",
|
|
" \"When and how did Paul Graham's mother die?\",\n",
|
|
" \"What, in Paul Graham's opinion, is the most distinctive thing about YC?\",\n",
|
|
" \"When and how did Paul Graham meet Jessica Livingston?\",\n",
|
|
" \"What is Bel, and when and where was it written?\",\n",
|
|
"]\n",
|
|
"for query in queries:\n",
|
|
" response = query_engine.query(query)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 2. Sub-Question Query Engine Evaluation\n",
|
|
"\n",
|
|
"The **sub-question query engine** is used to tackle the problem of answering a complex query using multiple data sources. It first breaks down the complex query into sub-questions for each relevant data source, then gathers all the intermediate responses and synthesizes a final response.\n",
|
|
"\n",
|
|
"UpTrain callback handler will automatically capture the sub-question and the responses for each of them once generated and will run the following three evaluations *(Graded from 0 to 1)* on the response:\n",
|
|
"- **[Context Relevance](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-relevance)**: Determines if the retrieved context has sufficient information to answer the user query or not.\n",
|
|
"- **[Factual Accuracy](https://docs.uptrain.ai/predefined-evaluations/context-awareness/factual-accuracy)**: Assesses if the LLM's response can be verified via the retrieved context.\n",
|
|
"- **[Response Completeness](https://docs.uptrain.ai/predefined-evaluations/response-quality/response-completeness)**: Checks if the response contains all the information required to answer the user query comprehensively.\n",
|
|
"\n",
|
|
"In addition to the above evaluations, the callback handler will also run the following evaluation:\n",
|
|
"- **[Sub Query Completeness](https://docs.uptrain.ai/predefined-evaluations/query-quality/sub-query-completeness)**: Assures that the sub-questions accurately and comprehensively cover the original query."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Generated 3 sub questions.\n",
|
|
"\u001b[1;3;38;2;237;90;200m[documents] Q: What did Paul Graham work on before YC?\n",
|
|
"\u001b[0m\u001b[1;3;38;2;90;149;237m[documents] Q: What did Paul Graham work on during YC?\n",
|
|
"\u001b[0m\u001b[1;3;38;2;11;159;203m[documents] Q: What did Paul Graham work on after YC?\n",
|
|
"\u001b[0m\u001b[1;3;38;2;11;159;203m[documents] A: After Y Combinator, Paul Graham decided to focus on painting as his next endeavor.\n",
|
|
"\u001b[0m\u001b[1;3;38;2;90;149;237m[documents] A: Paul Graham worked on writing essays and working on Y Combinator during YC.\n",
|
|
"\u001b[0m\u001b[1;3;38;2;237;90;200m[documents] A: Before Y Combinator, Paul Graham worked on projects with his colleagues Robert and Trevor.\n",
|
|
"\u001b[0m"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 3/3 [00:02<00:00, 1.47it/s]\n",
|
|
"100%|██████████| 3/3 [00:00<00:00, 3.28it/s]\n",
|
|
"100%|██████████| 3/3 [00:01<00:00, 1.68it/s]\n",
|
|
"100%|██████████| 3/3 [00:01<00:00, 2.28it/s]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What did Paul Graham work on after YC?\n",
|
|
"Response: After Y Combinator, Paul Graham decided to focus on painting as his next endeavor.\n",
|
|
"\n",
|
|
"Context Relevance Score: 0.0\n",
|
|
"Factual Accuracy Score: 0.0\n",
|
|
"Response Completeness Score: 0.5\n",
|
|
"\n",
|
|
"\n",
|
|
"Question: What did Paul Graham work on during YC?\n",
|
|
"Response: Paul Graham worked on writing essays and working on Y Combinator during YC.\n",
|
|
"\n",
|
|
"Context Relevance Score: 0.0\n",
|
|
"Factual Accuracy Score: 1.0\n",
|
|
"Response Completeness Score: 0.5\n",
|
|
"\n",
|
|
"\n",
|
|
"Question: What did Paul Graham work on before YC?\n",
|
|
"Response: Before Y Combinator, Paul Graham worked on projects with his colleagues Robert and Trevor.\n",
|
|
"\n",
|
|
"Context Relevance Score: 0.0\n",
|
|
"Factual Accuracy Score: 0.0\n",
|
|
"Response Completeness Score: 0.5\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.24s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: How was Paul Grahams life different before, during, and after YC?\n",
|
|
"Sub Query Completeness Score: 1.0\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# build index and query engine\n",
|
|
"vector_query_engine = VectorStoreIndex.from_documents(\n",
|
|
" documents=documents,\n",
|
|
" use_async=True,\n",
|
|
").as_query_engine()\n",
|
|
"\n",
|
|
"query_engine_tools = [\n",
|
|
" QueryEngineTool(\n",
|
|
" query_engine=vector_query_engine,\n",
|
|
" metadata=ToolMetadata(\n",
|
|
" name=\"documents\",\n",
|
|
" description=\"Paul Graham essay on What I Worked On\",\n",
|
|
" ),\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"\n",
|
|
"query_engine = SubQuestionQueryEngine.from_defaults(\n",
|
|
" query_engine_tools=query_engine_tools,\n",
|
|
" use_async=True,\n",
|
|
")\n",
|
|
"\n",
|
|
"response = query_engine.query(\n",
|
|
" \"How was Paul Grahams life different before, during, and after YC?\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 3. Re-ranking \n",
|
|
"\n",
|
|
"Re-ranking is the process of reordering the nodes based on their relevance to the query. There are multiple classes of re-ranking algorithms offered by Llamaindex. We have used LLMRerank for this example.\n",
|
|
"\n",
|
|
"The re-ranker allows you to enter the number of top n nodes that will be returned after re-ranking. If this value remains the same as the original number of nodes, the re-ranker will only re-rank the nodes and not change the number of nodes. Otherwise, it will re-rank the nodes and return the top n nodes.\n",
|
|
"\n",
|
|
"We will perform different evaluations based on the number of nodes returned after re-ranking."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 3a. Re-ranking (With same number of nodes)\n",
|
|
"\n",
|
|
"If the number of nodes returned after re-ranking is the same as the original number of nodes, the following evaluation will be performed:\n",
|
|
"\n",
|
|
"- **[Context Reranking](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-reranking)**: Checks if the order of re-ranked nodes is more relevant to the query than the original order."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.89s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What did Sam Altman do in this essay?\n",
|
|
"Context Reranking Score: 1.0\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.88s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.44s/it]\n",
|
|
"100%|██████████| 1/1 [00:02<00:00, 2.77s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.45s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What did Sam Altman do in this essay?\n",
|
|
"Response: Sam Altman was asked to become the president of Y Combinator after the original founders decided to step down and reorganize the company for long-term sustainability.\n",
|
|
"\n",
|
|
"Context Relevance Score: 1.0\n",
|
|
"Factual Accuracy Score: 1.0\n",
|
|
"Response Completeness Score: 0.5\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"callback_handler = UpTrainCallbackHandler(\n",
|
|
" key_type=\"openai\",\n",
|
|
" api_key=os.environ[\"OPENAI_API_KEY\"],\n",
|
|
" project_name=\"uptrain_llamaindex\",\n",
|
|
")\n",
|
|
"Settings.callback_manager = CallbackManager([callback_handler])\n",
|
|
"\n",
|
|
"rerank_postprocessor = SentenceTransformerRerank(\n",
|
|
" top_n=3, # number of nodes after reranking\n",
|
|
" keep_retrieval_score=True,\n",
|
|
")\n",
|
|
"\n",
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" documents=documents,\n",
|
|
")\n",
|
|
"\n",
|
|
"query_engine = index.as_query_engine(\n",
|
|
" similarity_top_k=3, # number of nodes before reranking\n",
|
|
" node_postprocessors=[rerank_postprocessor],\n",
|
|
")\n",
|
|
"\n",
|
|
"response = query_engine.query(\n",
|
|
" \"What did Sam Altman do in this essay?\",\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# 3b. Re-ranking (With different number of nodes)\n",
|
|
"\n",
|
|
"If the number of nodes returned after re-ranking is the lesser as the original number of nodes, the following evaluation will be performed:\n",
|
|
"\n",
|
|
"- **[Context Conciseness](https://docs.uptrain.ai/predefined-evaluations/context-awareness/context-conciseness)**: Examines whether the reduced number of nodes still provides all the required information."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:02<00:00, 2.22s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What did Sam Altman do in this essay?\n",
|
|
"Context Conciseness Score: 0.0\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.58s/it]\n",
|
|
"100%|██████████| 1/1 [00:00<00:00, 1.19it/s]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.62s/it]\n",
|
|
"100%|██████████| 1/1 [00:01<00:00, 1.42s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Question: What did Sam Altman do in this essay?\n",
|
|
"Response: Sam Altman offered unsolicited advice to the author during a visit to California for interviews.\n",
|
|
"\n",
|
|
"Context Relevance Score: 0.0\n",
|
|
"Factual Accuracy Score: 1.0\n",
|
|
"Response Completeness Score: 0.5\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"callback_handler = UpTrainCallbackHandler(\n",
|
|
" key_type=\"openai\",\n",
|
|
" api_key=os.environ[\"OPENAI_API_KEY\"],\n",
|
|
" project_name=\"uptrain_llamaindex\",\n",
|
|
")\n",
|
|
"Settings.callback_manager = CallbackManager([callback_handler])\n",
|
|
"\n",
|
|
"rerank_postprocessor = SentenceTransformerRerank(\n",
|
|
" top_n=2, # Number of nodes after re-ranking\n",
|
|
" keep_retrieval_score=True,\n",
|
|
")\n",
|
|
"\n",
|
|
"index = VectorStoreIndex.from_documents(\n",
|
|
" documents=documents,\n",
|
|
")\n",
|
|
"query_engine = index.as_query_engine(\n",
|
|
" similarity_top_k=5, # Number of nodes before re-ranking\n",
|
|
" node_postprocessors=[rerank_postprocessor],\n",
|
|
")\n",
|
|
"\n",
|
|
"# Use your advanced RAG\n",
|
|
"response = query_engine.query(\n",
|
|
" \"What did Sam Altman do in this essay?\",\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# UpTrain's Dashboard and Insights\n",
|
|
"\n",
|
|
"Here's a short video showcasing the dashboard and the insights:\n",
|
|
"\n",
|
|
""
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|