---
id: introduction
title: Introduction to Summarizer Evaluation
sidebar_label: Introduction
---
import { ASSETS } from "@site/src/assets";

Learn how to build, evaluate, and deploy a reliable **LLM-powered meeting summarization agent** using **OpenAI** and **DeepEval**.

<TechStackCards
    techStack={[
        {
            name: "OpenAI",
            logo: "https://registry.npmmirror.com/@lobehub/icons-static-png/latest/files/light/openai.png",
        },
        {
            name: "DeepEval",
            logo: "https://pbs.twimg.com/profile_images/1888060560161574912/qbw1-_2g.png",
        }
    ]}
/>

:::note
If you're working with LLMs for summarization, this tutorial is for you. While we'll specifically focus on evaluating a meeting summarizer, the concepts and practices here can be applied to **any LLM application tasked with summary generation**.
:::

## Get Started

DeepEval is an open-source LLM evaluation framework that supports a wide-range of metrics to help evaluate and iterate on your LLM applications.

Click on these links to jump to different stages of this tutorial:

<LinkCards
    tutorials={[
        {
            number: 1,
            icon: "Hammer",
            title: 'Build your Summarizer',
            objectives: [
                "Use OpenAI to build a summarizer",
                "Learn modular coding techniques to improve your summarizer",
                "Learn parsing techniques to build production grade LLM applications"
            ],
            to: '/tutorials/summarization-agent/development',
        },
        {
            number: 2,
            icon: "TestTubeDiagonal",
            title: 'Evaluate your summarizer',
            objectives: [
                "Learn how to define your evaluation criteria",
                "Create test cases using your summarizer",
                "Run your first eval",
                "Create datasets for future evaluations"
            ],
            to: '/tutorials/summarization-agent/evaluation',
        },
        {
            number: 3,
            icon: "BookPlus",
            title: 'Changing your model and prompts',
            objectives: [
                "Use evaluation scores to improve your summarizer",
                "Iterate over different models to find the best one for your use case",
                "Change your system prompts and check for regressions"
            ],
            to: '/tutorials/summarization-agent/improvement',
        },
        {
            number: 4,
            title: 'Setup Evals in Production',
            icon:"ShieldCheck",
            objectives: [
                "Trace your entire application workflow",
                "Evaluate your summarizer during prod and choose your metrics",
                "Setup CI/CD workflows to always get reliable summaries"
            ],
            to: '/tutorials/summarization-agent/evals-in-prod',
        },
    ]}
/>

## What You Will Evaluate

In this tutorial you will build and evaluate a **meeting summarization agent** that is used by famous tools like **Otter.ai** and **Circleback** to generate their summaries and action items from meeting transcripts. You will use `deepeval` and evalue the summarization agent's ability to generate:

- A concise summary of the discussion
- A clear list of action items

Below is an example of what a deliverable from a meeting summarization platform might look like:

<ImageDisplayer src={ASSETS.tutorialSummarizationOverview} alt="Webpage Image" />

In the next section, we'll build this summarization agent from scratch using OpenAI API.

:::tip
If you already have an LLM agent to evaluate, you can skip to [Evaluation Section](evaluation) of this tutorial.
:::