wehub-resource-sync
155 lines
5.8 KiB
Plaintext
155 lines
5.8 KiB
Plaintext
---
|
|
# id: guides-optimizing-hyperparameters
|
|
title: Optimizing Hyperparameters for LLM Applications
|
|
sidebar_label: Optimizing Hyperparameters
|
|
---
|
|
|
|
|
|
Apart from catching regressions and sanity checking your LLM applications, LLM evaluation and testing plays an pivotal role in picking the best hyperparameters for your LLM application.
|
|
|
|
:::info
|
|
In `deepeval`, hyperparameters refer to independent variables that affect the final `actual_output` of your LLM application, which includes the LLM used, the prompt template, temperature, etc.
|
|
:::
|
|
|
|
## Which Hyperparameters Should I Iterate On?
|
|
|
|
Here are typically the hyperparameters you should iterate on:
|
|
|
|
- **model**: the LLM to use for generation.
|
|
- **prompt template**: the variation of prompt templates to use for generation.
|
|
- **temperature**: the temperature value to use for generation.
|
|
- **max tokens**: the max token limit to set for your LLM generation.
|
|
- **top-K**: the number of retrieved nodes in your `retrieval_context` in a RAG pipeline.
|
|
- **chunk size**: the size of the retrieved nodes in your `retrieval_context` in a RAG pipeline.
|
|
- **reranking model**: the model used to rerank the retrieved nodes in your `retrieval_context` in a RAG pipeline.
|
|
|
|
:::tip
|
|
In the previous guide on [RAG Evaluation](/guides/guides-rag-evaluation), you already saw how `deepeval`'s RAG metrics can help iterate on many of the hyperparameters used within a RAG pipeline.
|
|
:::
|
|
|
|
## Finding The Best Hyperparameter Combination
|
|
|
|
To find the best hyperparameter combination, simply:
|
|
|
|
- choose a/multiple [LLM evaluation metrics](#metrics-introduction) that fits your evaluation criteria
|
|
- execute evaluations in a nested for-loop, while generating `actual_outputs` **at evaluation time** based on the current hyperparameter combination
|
|
|
|
:::note
|
|
In reality, you don't have to strictly generate `actual_outputs` at evaluation time and can evaluate with datasets of precomputed `actual_outputs`, but you ought to ensure that the `actual_outputs` in each [`LLMTestCase`](/docs/evaluation-test-cases) can be properly identified by a hyperparameter combination for this to work.
|
|
:::
|
|
|
|
Let's walkthrough a quick example hypothetical example showing how to find the best model and prompt template hyperparameter combination using the `AnswerRelevancyMetric` as a measurement. First, define a function to generate `actual_output`s for `LLMTestCase`s based on a certain hyperparameter combination:
|
|
|
|
```python
|
|
from typing import List
|
|
from deepeval.test_case import LLMTestCase
|
|
|
|
# Hypothetical helper function to construct LLMTestCases
|
|
def construct_test_cases(model: str, prompt_template: str) : List[LLMTestCase]:
|
|
# Hypothetical functions for you to implement
|
|
prompt = format_prompt_template(prompt_template)
|
|
llm = get_llm(model)
|
|
|
|
test_cases : List[LLMTestCase] = []
|
|
for input in list_of_inputs:
|
|
test_case = LLMTestCase(
|
|
input=input,
|
|
# Hypothetical function to generate actual outputs
|
|
# at evaluation time based on your hyperparameters!
|
|
actual_output=generate_actual_output(llm, prompt)
|
|
)
|
|
test_cases.append(test_case)
|
|
|
|
return test_cases
|
|
```
|
|
|
|
:::info
|
|
You **should definitely try** logging into Confident AI before continuing to the final step. Confident AI allows you to search, filter for, and view metric evaluation results on the web to pick the best hyperparameter combination for your LLM application.
|
|
|
|
Simply run `deepeval login`:
|
|
|
|
```bash
|
|
deepeval login
|
|
```
|
|
|
|
:::
|
|
|
|
Then, define the `AnswerRelevancyMetric` and use this helper function to construct `LLMTestCase`s:
|
|
|
|
```python
|
|
from deepeval import evaluate
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
...
|
|
|
|
# Define metric(s)
|
|
metric = AnswerRelevancyMetric()
|
|
|
|
# Start the nested for-loop
|
|
for model in models:
|
|
for prompt_template in prompt_templates:
|
|
evaluate(
|
|
test_cases=construct_test_cases(model, prompt_template),
|
|
metrics=[metric],
|
|
# log hyperparameters associated with this batch of test cases
|
|
hyperparameter={
|
|
"model": model,
|
|
"prompt template": prompt_template
|
|
}
|
|
)
|
|
```
|
|
|
|
:::tip
|
|
Remember, we're just using the `AnswerRelevancyMetric` as an example here and you should choose whichever [LLM evaluation metrics](https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation) based on whatever custom criteria you want to assess your LLM application on.
|
|
:::
|
|
|
|
## Keeping Track of Hyperparameters in CI/CD
|
|
|
|
You can also keep track of hyperparameters used during testing in your CI/CD pipelines. This is helpful since you will be able to pinpoint the hyperparameter combination associated with failing test runs.
|
|
|
|
To begin, login to Confident AI:
|
|
|
|
```bash
|
|
deepeval login
|
|
```
|
|
|
|
Then define your test function and log hyperparameters in your test file:
|
|
|
|
```python title="test_file.py"
|
|
import pytest
|
|
import deepeval
|
|
|
|
from deepeval import assert_test
|
|
from deepeval.test_case import LLMTestCase
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
|
|
test_cases = [...]
|
|
|
|
# Loop through test cases using Pytest
|
|
@pytest.mark.parametrize(
|
|
"test_case",
|
|
test_cases,
|
|
)
|
|
def test_customer_chatbot(test_case: LLMTestCase):
|
|
answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)
|
|
assert_test(test_case, [answer_relevancy_metric])
|
|
|
|
|
|
# You should aim to make these values dynamic
|
|
@deepeval.log_hyperparameters(model="gpt-4", prompt_template="...")
|
|
def hyperparameters():
|
|
# Return a dict to log additional hyperparameters.
|
|
# You can also return an empty dict {} if there's no additional parameters to log
|
|
return {
|
|
"temperature": 1,
|
|
"chunk size": 500
|
|
}
|
|
```
|
|
|
|
Lastly, run `deepeval test run`:
|
|
|
|
```bash
|
|
deepeval test run test_file.py
|
|
```
|
|
|
|
In the next guide, we'll show you to build your own custom LLM evaluation metrics in case you want more control over evaluation when picking for hyperparameters.
|