wehub-resource-sync
103 lines
3.8 KiB
Plaintext
103 lines
3.8 KiB
Plaintext
---
|
|
id: introduction
|
|
title: RAG Agent Evaluation Tutorial
|
|
sidebar_label: Introduction
|
|
---
|
|
import { ASSETS } from "@site/src/assets";
|
|
|
|
This tutorial walks you through the entire process of building a reliable **RAG (_Retrieval-Augmented Generation_) QA Agent**,
|
|
from initial development to iterative improvement through `deepeval`'s evaluations. We'll build this RAG QA Agent using **OpenAI**, **LangChain** and **DeepEval**.
|
|
|
|
<TechStackCards
|
|
techStack={[
|
|
{
|
|
name: "OpenAI",
|
|
logo: "https://registry.npmmirror.com/@lobehub/icons-static-png/latest/files/light/openai.png",
|
|
},
|
|
{
|
|
name: "LangChain",
|
|
logo: "https://logo.svgcdn.com/s/langchain-dark-8x.png",
|
|
},
|
|
{
|
|
name: "DeepEval",
|
|
logo: "https://pbs.twimg.com/profile_images/1888060560161574912/qbw1-_2g.png",
|
|
}
|
|
]}
|
|
/>
|
|
|
|
:::note
|
|
This tutorial focuses on building a RAG-based QA agent for an infamous company called **Theranos**. However, the concepts and practices used throughout this tutorial are applicable to any **RAG-based application**. If you are working with RAG applications, this tutorial will be helpful to you.
|
|
:::
|
|
|
|
## Overview
|
|
|
|
DeepEval is an open-source LLM evaluation framework that supports a wide-range of metrics to help evaluate and iterate on your LLM applications.
|
|
|
|
You can click on the links below and jump to any stage of this tutorial as you like:
|
|
|
|
<LinkCards
|
|
tutorials={[
|
|
{
|
|
number: 1,
|
|
icon: "Drill",
|
|
title: 'Develop Your RAG',
|
|
objectives: [
|
|
"Build a RAG with OpenAI & LangChain",
|
|
"Use OpenAI Embeddings",
|
|
"Use LangChain's vector stores",
|
|
"Create a full RAG QA Agent"
|
|
],
|
|
to: '/tutorials/rag-qa-agent/development',
|
|
},
|
|
{
|
|
number: 2,
|
|
icon: "TestTubes",
|
|
title: 'Evaluate Your Retriever & Generator',
|
|
objectives: [
|
|
"Define your evaluation criteria",
|
|
"Evaluate your retriever and generator in isolation",
|
|
"Evaluate your RAG as a whole",
|
|
"Create datasets for robust eval pipelines"
|
|
],
|
|
to: '/tutorials/rag-qa-agent/evaluation',
|
|
},
|
|
{
|
|
number: 3,
|
|
title: 'Improve your RAG using evals',
|
|
icon: "FolderUp",
|
|
objectives: [
|
|
"Define your hyperparamets",
|
|
"Test different configurations with DeepEval",
|
|
"Find the best set of hyperparameters for your RAG"
|
|
],
|
|
to: '/tutorials/rag-qa-agent/improvement',
|
|
},
|
|
{
|
|
number: 4,
|
|
title: 'Deploy and test your RAG in prod',
|
|
icon: "ShieldAlert",
|
|
objectives: [
|
|
"Trace your RAG components for each QA",
|
|
"Choose the metrics to apply in prod",
|
|
"Test your RAG for every new doc you push in your knowledge base"
|
|
],
|
|
to: '/tutorials/rag-qa-agent/evals-in-prod',
|
|
},
|
|
]}
|
|
/>
|
|
|
|
## What You Will Evaluate
|
|
|
|
**RAG (Retrieval-Augmented Generation)** agents let companies build domain-specific assistants without fine-tuning large models.
|
|
In this tutorial, you'll create a **RAG QA agent** that answers questions about **Theranos**, a blood diagnostics company. We will evaluate the agent's ability on:
|
|
|
|
- Generating relevant and accurate answers
|
|
- Providing correct citations to questions
|
|
|
|
Below is an example of what **Theranos**'s internal RAG QA agent might look like:.
|
|
|
|
|
|
<ImageDisplayer src={ASSETS.tutorialQaAgentOverview} alt="MadeUpCompany's RAG QA Agent" />
|
|
|
|
In the following sections of this tutorial, you'll learn how to build a reliable RAG QA Agent that retrieves correct data and generates an
|
|
accurate answer based on the retrieved context. |