---
id: huggingface
title: Hugging Face
sidebar_label: Hugging Face
---

## Quick Summary

Hugging Face provides developers with a comprehensive suite of pre-trained NLP models through its `transformers` library. To recap, here is how you can use Mistral's `mistralai/Mistral-7B-v0.1` model through Hugging Face's `transformers` library:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

prompt = "My favourite condiment is"

model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
model.to(device)

generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
print(tokenizer.batch_decode(generated_ids)[0])
# "The expected output"
```

## Evals During Fine-Tuning

`deepeval` integrates with Hugging Face's `transformers.Trainer` module through the `DeepEvalHuggingFaceCallback`, enabling real-time evaluation of LLM outputs during model fine-tuning for each epoch.

:::info
In this section, we'll walkthrough an example of fine-tuning Mistral's 7B model.
:::

### Prepare Dataset for Fine-tuning

```python
from transformers import AutoTokenizer
from datasets import load_dataset

####################
### Load dataset ###
####################
training_dataset = load_dataset("text", data_files={"train": "train.txt"})

########################
### Tokenize dataset ###
########################
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
tokenized_dataset = training_dataset.map(tokenize_function, batched=True)
```

### Setup Training Arguments

```python
from transformers import TrainingArguments
...

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=5,
    per_device_train_batch_size=4,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs",
    logging_steps=10,
)
```

### Initialize LLM and Trainer for Fine-Tuning

```python
from transformers import AutoModelForCausalLM, Trainer
...

######################
### Initialize LLM ###
######################
llm = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")


##########################
### Initialize Trainer ###
##########################
trainer = Trainer(
    model=llm,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
)
```

### Define Evaluation Criteria

Use `deepeval` to define an `EvaluationDataset` and the metrics you want to evaluate your LLM on:

```python
from deepeval.test_case import SingleTurnParams
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import GEval

first_golden = Golden(input="...")
second_golden = Golden(input="...")

dataset = EvaluationDataset(goldens=[first_golden, second_golden])
coherence_metric = GEval(
    name="Coherence",
    criteria="Coherence - determine if the actual output is coherent with the input.",
    evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT],
)
```

:::info
We initialize an `EvaluationDataset` with [goldens instead of test cases](/docs/evaluation-datasets#with-goldens) since we're running inference at evaluation time.
:::

### Fine-tune and Evaluate

Then, create a `DeepEvalHuggingFaceCallback`:

```python
from deepeval.integrations.hugging_face import DeepEvalHuggingFaceCallback
...

deepeval_hugging_face_callback = DeepEvalHuggingFaceCallback(
    evaluation_dataset=dataset,
    metrics=[coherence_metric],
    trainer=trainer
)
```

The `DeepEvalHuggingFaceCallback` accepts the following arguments:

- `metrics`: the `deepeval` evaluation metrics you wish to leverage.
- `evaluation_dataset`: a `deepeval` `EvaluationDataset`.
- `aggregation_method`: a string of either 'avg', 'min', or 'max' to determine how metric scores are aggregated.
- `trainer`: a `transformers.trainer` instance.
- `tokenizer_args`: Arguments for the tokenizer.

Lastly, add `deepeval_hugging_face_callback` to your `transformers.Trainer`, and begin fine-tuning:

```python
...
#############################
### Add DeepEval Callback ###
#############################
trainer.add_callback(deepeval_hugging_face_callback)

#########################
### Start Fine-tuning ###
#########################
trainer.train()
```

With this setup, evaluations will be ran on the entirety of your `EvaluationDataset` according to the metrics you defined at the end of each `epoch`.