wehub-resource-sync
98 lines
3.6 KiB
Plaintext
98 lines
3.6 KiB
Plaintext
---
|
|
id: introduction
|
|
title: Introduction to Summarizer Evaluation
|
|
sidebar_label: Introduction
|
|
---
|
|
import { ASSETS } from "@site/src/assets";
|
|
|
|
Learn how to build, evaluate, and deploy a reliable **LLM-powered meeting summarization agent** using **OpenAI** and **DeepEval**.
|
|
|
|
<TechStackCards
|
|
techStack={[
|
|
{
|
|
name: "OpenAI",
|
|
logo: "https://registry.npmmirror.com/@lobehub/icons-static-png/latest/files/light/openai.png",
|
|
},
|
|
{
|
|
name: "DeepEval",
|
|
logo: "https://pbs.twimg.com/profile_images/1888060560161574912/qbw1-_2g.png",
|
|
}
|
|
]}
|
|
/>
|
|
|
|
:::note
|
|
If you're working with LLMs for summarization, this tutorial is for you. While we'll specifically focus on evaluating a meeting summarizer, the concepts and practices here can be applied to **any LLM application tasked with summary generation**.
|
|
:::
|
|
|
|
## Get Started
|
|
|
|
DeepEval is an open-source LLM evaluation framework that supports a wide-range of metrics to help evaluate and iterate on your LLM applications.
|
|
|
|
Click on these links to jump to different stages of this tutorial:
|
|
|
|
<LinkCards
|
|
tutorials={[
|
|
{
|
|
number: 1,
|
|
icon: "Hammer",
|
|
title: 'Build your Summarizer',
|
|
objectives: [
|
|
"Use OpenAI to build a summarizer",
|
|
"Learn modular coding techniques to improve your summarizer",
|
|
"Learn parsing techniques to build production grade LLM applications"
|
|
],
|
|
to: '/tutorials/summarization-agent/development',
|
|
},
|
|
{
|
|
number: 2,
|
|
icon: "TestTubeDiagonal",
|
|
title: 'Evaluate your summarizer',
|
|
objectives: [
|
|
"Learn how to define your evaluation criteria",
|
|
"Create test cases using your summarizer",
|
|
"Run your first eval",
|
|
"Create datasets for future evaluations"
|
|
],
|
|
to: '/tutorials/summarization-agent/evaluation',
|
|
},
|
|
{
|
|
number: 3,
|
|
icon: "BookPlus",
|
|
title: 'Changing your model and prompts',
|
|
objectives: [
|
|
"Use evaluation scores to improve your summarizer",
|
|
"Iterate over different models to find the best one for your use case",
|
|
"Change your system prompts and check for regressions"
|
|
],
|
|
to: '/tutorials/summarization-agent/improvement',
|
|
},
|
|
{
|
|
number: 4,
|
|
title: 'Setup Evals in Production',
|
|
icon:"ShieldCheck",
|
|
objectives: [
|
|
"Trace your entire application workflow",
|
|
"Evaluate your summarizer during prod and choose your metrics",
|
|
"Setup CI/CD workflows to always get reliable summaries"
|
|
],
|
|
to: '/tutorials/summarization-agent/evals-in-prod',
|
|
},
|
|
]}
|
|
/>
|
|
|
|
## What You Will Evaluate
|
|
|
|
In this tutorial you will build and evaluate a **meeting summarization agent** that is used by famous tools like **Otter.ai** and **Circleback** to generate their summaries and action items from meeting transcripts. You will use `deepeval` and evalue the summarization agent's ability to generate:
|
|
|
|
- A concise summary of the discussion
|
|
- A clear list of action items
|
|
|
|
Below is an example of what a deliverable from a meeting summarization platform might look like:
|
|
|
|
<ImageDisplayer src={ASSETS.tutorialSummarizationOverview} alt="Webpage Image" />
|
|
|
|
In the next section, we'll build this summarization agent from scratch using OpenAI API.
|
|
|
|
:::tip
|
|
If you already have an LLM agent to evaluate, you can skip to [Evaluation Section](evaluation) of this tutorial.
|
|
::: |