wehub-resource-sync
159 lines
4.5 KiB
Plaintext
159 lines
4.5 KiB
Plaintext
---
|
|
id: huggingface
|
|
title: Hugging Face
|
|
sidebar_label: Hugging Face
|
|
---
|
|
|
|
## Quick Summary
|
|
|
|
Hugging Face provides developers with a comprehensive suite of pre-trained NLP models through its `transformers` library. To recap, here is how you can use Mistral's `mistralai/Mistral-7B-v0.1` model through Hugging Face's `transformers` library:
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
device = "cuda" # the device to load the model onto
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
|
|
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
|
|
|
|
prompt = "My favourite condiment is"
|
|
|
|
model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
|
|
model.to(device)
|
|
|
|
generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
|
|
print(tokenizer.batch_decode(generated_ids)[0])
|
|
# "The expected output"
|
|
```
|
|
|
|
## Evals During Fine-Tuning
|
|
|
|
`deepeval` integrates with Hugging Face's `transformers.Trainer` module through the `DeepEvalHuggingFaceCallback`, enabling real-time evaluation of LLM outputs during model fine-tuning for each epoch.
|
|
|
|
:::info
|
|
In this section, we'll walkthrough an example of fine-tuning Mistral's 7B model.
|
|
:::
|
|
|
|
### Prepare Dataset for Fine-tuning
|
|
|
|
```python
|
|
from transformers import AutoTokenizer
|
|
from datasets import load_dataset
|
|
|
|
####################
|
|
### Load dataset ###
|
|
####################
|
|
training_dataset = load_dataset("text", data_files={"train": "train.txt"})
|
|
|
|
########################
|
|
### Tokenize dataset ###
|
|
########################
|
|
def tokenize_function(examples):
|
|
return tokenizer(examples["text"], padding="max_length", truncation=True)
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
|
|
tokenized_dataset = training_dataset.map(tokenize_function, batched=True)
|
|
```
|
|
|
|
### Setup Training Arguments
|
|
|
|
```python
|
|
from transformers import TrainingArguments
|
|
...
|
|
|
|
training_args = TrainingArguments(
|
|
output_dir="./results",
|
|
num_train_epochs=5,
|
|
per_device_train_batch_size=4,
|
|
warmup_steps=500,
|
|
weight_decay=0.01,
|
|
logging_dir="./logs",
|
|
logging_steps=10,
|
|
)
|
|
```
|
|
|
|
### Initialize LLM and Trainer for Fine-Tuning
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, Trainer
|
|
...
|
|
|
|
######################
|
|
### Initialize LLM ###
|
|
######################
|
|
llm = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
|
|
|
|
|
|
##########################
|
|
### Initialize Trainer ###
|
|
##########################
|
|
trainer = Trainer(
|
|
model=llm,
|
|
args=training_args,
|
|
train_dataset=tokenized_dataset["train"],
|
|
)
|
|
```
|
|
|
|
### Define Evaluation Criteria
|
|
|
|
Use `deepeval` to define an `EvaluationDataset` and the metrics you want to evaluate your LLM on:
|
|
|
|
```python
|
|
from deepeval.test_case import SingleTurnParams
|
|
from deepeval.dataset import EvaluationDataset, Golden
|
|
from deepeval.metrics import GEval
|
|
|
|
first_golden = Golden(input="...")
|
|
second_golden = Golden(input="...")
|
|
|
|
dataset = EvaluationDataset(goldens=[first_golden, second_golden])
|
|
coherence_metric = GEval(
|
|
name="Coherence",
|
|
criteria="Coherence - determine if the actual output is coherent with the input.",
|
|
evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT],
|
|
)
|
|
```
|
|
|
|
:::info
|
|
We initialize an `EvaluationDataset` with [goldens instead of test cases](/docs/evaluation-datasets#with-goldens) since we're running inference at evaluation time.
|
|
:::
|
|
|
|
### Fine-tune and Evaluate
|
|
|
|
Then, create a `DeepEvalHuggingFaceCallback`:
|
|
|
|
```python
|
|
from deepeval.integrations.hugging_face import DeepEvalHuggingFaceCallback
|
|
...
|
|
|
|
deepeval_hugging_face_callback = DeepEvalHuggingFaceCallback(
|
|
evaluation_dataset=dataset,
|
|
metrics=[coherence_metric],
|
|
trainer=trainer
|
|
)
|
|
```
|
|
|
|
The `DeepEvalHuggingFaceCallback` accepts the following arguments:
|
|
|
|
- `metrics`: the `deepeval` evaluation metrics you wish to leverage.
|
|
- `evaluation_dataset`: a `deepeval` `EvaluationDataset`.
|
|
- `aggregation_method`: a string of either 'avg', 'min', or 'max' to determine how metric scores are aggregated.
|
|
- `trainer`: a `transformers.trainer` instance.
|
|
- `tokenizer_args`: Arguments for the tokenizer.
|
|
|
|
Lastly, add `deepeval_hugging_face_callback` to your `transformers.Trainer`, and begin fine-tuning:
|
|
|
|
```python
|
|
...
|
|
#############################
|
|
### Add DeepEval Callback ###
|
|
#############################
|
|
trainer.add_callback(deepeval_hugging_face_callback)
|
|
|
|
#########################
|
|
### Start Fine-tuning ###
|
|
#########################
|
|
trainer.train()
|
|
```
|
|
|
|
With this setup, evaluations will be ran on the entirety of your `EvaluationDataset` according to the metrics you defined at the end of each `epoch`.
|