wehub-resource-sync
129 lines
4.5 KiB
Plaintext
129 lines
4.5 KiB
Plaintext
---
|
||
id: tutorial-setup
|
||
title: Set Up DeepEval
|
||
sidebar_label: Set Up DeepEval
|
||
---
|
||
|
||
import { ASSETS } from "@site/src/assets";
|
||
|
||
## Installing DeepEval
|
||
|
||
**DeepEval** is a powerful LLM evaluation framework. Here's how you can easily get started by installing and running your first evaluation using DeepEval.
|
||
|
||
Start by installing DeepEval using pip:
|
||
|
||
```bash
|
||
pip install -U deepeval
|
||
```
|
||
|
||
### Write your first test
|
||
|
||
Let's evaluate the correctness of an LLM output using [`GEval`](https://deepeval.com/docs/metrics-llm-evals), a powerful metric based on LLM-as-a-judge evaluation.
|
||
|
||
:::note
|
||
When you pass a file explicitly (e.g. `deepeval test run evals/my_eval.py`), DeepEval runs it regardless of its name. The `test_` prefix (like `test_app.py`) is only needed for pytest's automatic discovery when you point it at a directory.
|
||
:::
|
||
|
||
```python title="test_app.py"
|
||
from deepeval import evaluate
|
||
from deepeval.test_case import LLMTestCase, SingleTurnParams
|
||
from deepeval.metrics import GEval
|
||
|
||
correctness_metric = GEval(
|
||
name="Correctness",
|
||
criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
|
||
evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
|
||
threshold=0.5
|
||
)
|
||
|
||
test_case = LLMTestCase(
|
||
input="I have a persistent cough and fever. Should I be worried?",
|
||
# Replace this with the actual output from your LLM application
|
||
actual_output="A persistent cough and fever could signal various illnesses, from minor infections to more serious conditions like pneumonia or COVID-19. It's advisable to seek medical attention if symptoms worsen, persist beyond a few days, or if you experience difficulty breathing, chest pain, or other concerning signs.",
|
||
expected_output="A persistent cough and fever could indicate a range of illnesses, from a mild viral infection to more serious conditions like pneumonia or COVID-19. You should seek medical attention if your symptoms worsen, persist for more than a few days, or are accompanied by difficulty breathing, chest pain, or other concerning signs."
|
||
)
|
||
|
||
evaluate([test_case], [correctness_metric])
|
||
```
|
||
|
||
To run your first evaluation, enter the following command in your terminal:
|
||
|
||
```bash
|
||
deepeval test run test_app.py
|
||
```
|
||
|
||
:::note
|
||
DeepEval's powerful **LLM-as-a-judge** metrics (like `GEval` used in this example) rely on an underlying LLM called the _Evaluation Model_ to perform evaluations. By default, DeepEval uses OpenAI's models for this purpose.
|
||
|
||
So you'll have to set your `OPENAI_API_KEY` as an environment variable as shown below.
|
||
|
||
```bash
|
||
export OPENAI_API_KEY="your_api_key"
|
||
|
||
```
|
||
|
||
To use ANY custom LLM of your choice, [Check out our docs on custom evaluation models](https://deepeval.com/guides/guides-using-custom-llms).
|
||
:::
|
||
|
||
Congratulations! You've successfully run your first LLM evaluation with DeepEval.
|
||
|
||
## Setting Up Confident AI
|
||
|
||
While DeepEval works great standalone, you can connect it to [Confident AI](https://www.confident-ai.com) — an AI quality platform with observability, evals, and monitoring that DeepEval integrates with natively for dashboards, logging, collaboration, and more. **It’s free to get started.**
|
||
|
||
You can [sign up here](https://www.confident-ai.com), or run:
|
||
|
||
```bash
|
||
deepeval login
|
||
```
|
||
|
||
Navigate to your Settings page and copy your Confident AI API Key from the Project API Key box. If you used the `deepeval login` command to log in, you'll be prompted to paste your Confident AI API Key after creating an account.
|
||
|
||
<div
|
||
style={{
|
||
display: "flex",
|
||
alignItems: "center",
|
||
justifyContent: "center",
|
||
}}
|
||
>
|
||
<ImageDisplayer src={ASSETS.tutorialSetup01} />
|
||
</div>
|
||
|
||
Alternatively, if you already have an account, you can log in directly using Python:
|
||
|
||
```python title="main.py"
|
||
deepeval.login("your-confident-api-key")
|
||
```
|
||
|
||
Or through the CLI:
|
||
|
||
```bash
|
||
deepeval login --confident-api-key "your-confident-api-key"
|
||
```
|
||
|
||
:::note[Login persistence]
|
||
`deepeval login` persists your key to a dotenv file by default (.env.local).
|
||
To change the target, use `--save`, e.g.:
|
||
|
||
```bash
|
||
# custom path
|
||
deepeval login --confident-api-key "ck_..." --save dotenv:.env.custom
|
||
```
|
||
|
||
For compatibility, the key is saved under `api_key` and `CONFIDENT_API_KEY`.
|
||
Secrets are never written to the JSON keystore.
|
||
:::
|
||
|
||
:::tip[Logging out / rotating keys]
|
||
Use deepeval logout to clear the JSON keystore and remove saved keys from your dotenv file:
|
||
|
||
```bash
|
||
# default removes from .env.local
|
||
deepeval logout
|
||
|
||
# or specify a custom target
|
||
deepeval logout --save dotenv:.myconf.env
|
||
```
|
||
:::
|
||
|
||
You're all set! You can now evaluate LLMs locally and monitor them in Confident AI. |