chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,630 @@
|
||||
# Zero to LitGPT: Getting Started with Pretraining, Finetuning, and Using LLMs
|
||||
|
||||
|
||||
|
||||
This tutorial walks you through the main features and usage patterns for ⚡️LitGPT, a library for pretraining, finetuning, and using LLMs that focuses on an efficient user experience while being developer-friendly.
|
||||
|
||||
The topics, following the installation of LitGPT, are in chronological order, reflecting the steps in an LLM lifecycle: Pretraining → Finetuning → Inference.
|
||||
|
||||
|
||||
|
||||
<img src="images/0_to_litgpt/usage.webp" width=500>
|
||||
|
||||
|
||||
|
||||
<img src="images/0_to_litgpt/commands.webp" width=300>
|
||||
|
||||
|
||||
|
||||
However, it is also possible, and even common, to use and deploy models with LitGPT without pretraining and finetuning. So, if you are not interested in pretraining and finetuning, please feel free to skip these sections.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Install LitGPT
|
||||
|
||||
LitGPT is available as a Python library from the PyPI package repository, and we recommend installing it using Python's `pip` installer module, including all required package dependencies:
|
||||
|
||||
```bash
|
||||
pip install 'litgpt[all]'
|
||||
```
|
||||
|
||||
Alternatively, if you are a researcher or developer planning to make changes to LitGPT, you can clone the GitHub repository and install it from a local folder as follows:
|
||||
|
||||
```
|
||||
git clone https://github.com/Lightning-AI/litgpt.git
|
||||
cd litgpt
|
||||
pip install -e '.[all]'
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Pretrain LLMs
|
||||
|
||||
Pretraining LLMs requires substantial compute resources and time commitment. For that reason, most researchers and practitioners prefer to skip this step and continue with the *Download pretrained model weights* section instead.
|
||||
|
||||
However, if you feel adventurous and want to pretrain your own LLM, here's how.
|
||||
|
||||
First, we have to decide which type of model architecture we want to use. We list the available architectures by using the `pretrain` command without any additional arguments:
|
||||
|
||||
```bash
|
||||
litgpt pretrain list
|
||||
```
|
||||
|
||||
This prints a list of all available model architectures in alphabetical order:
|
||||
|
||||
```
|
||||
Camel-Platypus2-13B
|
||||
Camel-Platypus2-70B
|
||||
CodeLlama-13b-Python-hf
|
||||
...
|
||||
EleutherAI/pythia-410m
|
||||
...
|
||||
vicuna-13b-v1.3
|
||||
vicuna-13b-v1.5
|
||||
vicuna-13b-v1.5-16k
|
||||
vicuna-33b-v1.3
|
||||
vicuna-7b-v1.3
|
||||
vicuna-7b-v1.5
|
||||
vicuna-7b-v1.5-16k
|
||||
```
|
||||
|
||||
Suppose we want to pretraining the 1.1B parameter small `tiny-llama-1.1b` model. Before starting finetuning, we must also choose and download a tokenizer.
|
||||
|
||||
We can download a tokenizer via the `download` command. Note that running `litgpt download list` will also print a list of all available models and tokenizers to download.
|
||||
|
||||
To filter for specific models, e.g., TinyLlama, we can use the `grep` command in our terminal:
|
||||
|
||||
```bash
|
||||
litgpt download list | grep TinyLlama
|
||||
```
|
||||
|
||||
This prints
|
||||
|
||||
```
|
||||
TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
||||
TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
```
|
||||
|
||||
Let's now download the tokenizer corresponding to `TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T` that we can then use to pretrain the TinyLlama model:
|
||||
|
||||
```
|
||||
litgpt download \
|
||||
TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T \
|
||||
--tokenizer_only true
|
||||
```
|
||||
|
||||
(when specified)
|
||||
|
||||
|
||||
|
||||
<img src="images/0_to_litgpt/pretrain.webp" width=400>
|
||||
|
||||
|
||||
|
||||
Next, we can pretrain the model on the OpenWebText dataset with the default setting as follows:
|
||||
|
||||
```bash
|
||||
litgpt pretrain tiny-llama-1.1b \
|
||||
--data OpenWebText \
|
||||
--tokenizer_dir TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
||||
```
|
||||
|
||||
If you are interested in additional settings, you can use the help command as follows:
|
||||
|
||||
```
|
||||
litgpt pretrain --help
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Above, we only covered the most basic commands for pretraining a model using LitGPT. We highly recommend checking the resources below if you are interested in pretraining a model.
|
||||
|
||||
|
||||
|
||||
**More information and additional resources**
|
||||
|
||||
- [tutorials/pretrain](./pretrain.md): General information about pretraining in LitGPT
|
||||
- [tutorials/pretrain_tinyllama](./pretrain_tinyllama.md): A tutorial for finetuning a 1.1B TinyLlama model on 3 trillion tokens
|
||||
- [config_hub/pretrain](../config_hub/pretrain): Pre-made config files for pretraining that work well out of the box
|
||||
- Project templates in reproducible environments with multi-GPU and multi-node support:
|
||||
- [Prepare the TinyLlama 1T token dataset](https://lightning.ai/lightning-ai/studios/prepare-the-tinyllama-1t-token-dataset)
|
||||
- [Pretrain LLMs - TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/pretrain-llms-tinyllama-1-1b)
|
||||
- [Continued Pretraining with TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/continued-pretraining-with-tinyllama-1-1b)
|
||||
|
||||
|
||||
|
||||
## Download pretrained model weights
|
||||
|
||||
Most practical use cases, like LLM inference (/chat) or finetuning, involve using pretrained model weights. LitGPT supports a large number of model weights, which can be listed by executing the `download` with `list` as an argument:
|
||||
|
||||
```bash
|
||||
litgpt download list
|
||||
```
|
||||
|
||||
This will print a (long) list of all supported pretrained models (abbreviated for readability below):
|
||||
|
||||
```
|
||||
..
|
||||
google/gemma-2b
|
||||
...
|
||||
meta-llama/Llama-2-7b-hf
|
||||
...
|
||||
microsoft/phi-2
|
||||
...
|
||||
mistralai/Mixtral-8x7B-Instruct-v0.1
|
||||
...
|
||||
```
|
||||
|
||||
To download the model weights, provide one of the model strings above as input argument:
|
||||
|
||||
```bash
|
||||
litgpt download microsoft/phi-2
|
||||
```
|
||||
|
||||
```
|
||||
model-00001-of-00002.safetensors: 100%|████████████████████████████████| 5.00G/5.00G [00:40<00:00, 124MB/s]
|
||||
model-00002-of-00002.safetensors: 100%|████████████████████████████████| 564M/564M [00:01<00:00, 330MB/s]
|
||||
tokenizer.json: 100%|██████████████████████████████████████████████████| 2.11M/2.11M [00:00<00:00, 54.0MB/s]
|
||||
...
|
||||
Converting checkpoint files to LitGPT format.
|
||||
Processing checkpoints/microsoft/phi-2/model-00001-of-00002.bin
|
||||
...
|
||||
Saving converted checkpoint to checkpoints/microsoft/phi-2
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Note that some models, such as Llama 2, require that you accept Meta AI's terms of service for this model, and you need to use a special access token via the `litgpt download ... --access_token ...` option. For more information, visit the respective Model Hub website, e.g., [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf). The access token can be created under your Model Hub in the `Profile > Access Tokens` menu.
|
||||
|
||||
|
||||
|
||||
|
||||
By default, the weights are going to be stored in a `./checkpoints` subdirectory:
|
||||
|
||||
```bash
|
||||
ls -lh checkpoints/microsoft/phi-2/
|
||||
```
|
||||
|
||||
```
|
||||
total 11G
|
||||
-rw-r--r-- 1 sebastian sebastian 863 Mar 19 21:14 config.json
|
||||
-rw-r--r-- 1 sebastian sebastian 124 Mar 19 21:14 generation_config.json
|
||||
-rw-r--r-- 1 sebastian sebastian 5.2G Mar 19 21:15 lit_model.pth
|
||||
-rw-r--r-- 1 sebastian sebastian 4.7G Mar 19 21:15 model-00001-of-00002.bin
|
||||
-rw-r--r-- 1 sebastian sebastian 538M Mar 19 21:15 model-00002-of-00002.bin
|
||||
-rw-r--r-- 1 sebastian sebastian 528 Mar 19 21:15 model_config.yaml
|
||||
-rw-r--r-- 1 sebastian sebastian 2.1M Mar 19 21:14 tokenizer.json
|
||||
-rw-r--r-- 1 sebastian sebastian 7.2K Mar 19 21:14 tokenizer_config.json
|
||||
```
|
||||
|
||||
The model is now ready for inference and chat, for example, using the `chat` command on the checkpoint directory:
|
||||
|
||||
```bash
|
||||
litgpt chat microsoft/phi-2
|
||||
```
|
||||
|
||||
```
|
||||
Now chatting with phi-2.
|
||||
To exit, press 'Enter' on an empty prompt.
|
||||
|
||||
Seed set to 1234
|
||||
>> Prompt: Why are LLMs so useful?
|
||||
>> Reply: When building applications or operating systems, you can use LLMs to know how a computer should respond to your commands. This can make your programs run faster and more efficiently.
|
||||
|
||||
Time for inference: 1.26 sec total, 27.81 tokens/sec, 35 tokens
|
||||
|
||||
>> Prompt:
|
||||
```
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `--multiline true` to support prompts that require multiple input lines.
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
**More information and additional resources**
|
||||
|
||||
- [tutorials/download_model_weights](download_model_weights.md): A more comprehensive download tutorial, tips for GPU memory limitations, and more
|
||||
|
||||
|
||||
|
||||
## Finetune LLMs
|
||||
|
||||
LitGPT supports several methods of supervised instruction finetuning, which allows you to finetune models to follow instructions.
|
||||
|
||||
Datasets for Instruction-finetuning are usually formatted in the following way:
|
||||
|
||||
|
||||
|
||||
<img src="images/0_to_litgpt/instruction-1.webp" width=400>
|
||||
|
||||
|
||||
|
||||
Alternatively, datasets for instruction finetuning can also contain an `'input'` field:
|
||||
|
||||
In an instruction-finetuning context, "full" finetuning means updating all model parameters as opposed to only a subset. Adapter and LoRA (short for low-rank adaptation) are methods for parameter-efficient finetuning that only require updating a small fraction of the model weights.
|
||||
|
||||
|
||||
|
||||
<img src="images/0_to_litgpt/finetune.webp" width=400>
|
||||
|
||||
|
||||
|
||||
Parameter-efficient finetuning is much more resource-efficient and cheaper than full finetuning, and it often results in the same good performance on downstream tasks.
|
||||
|
||||
In the following example, we will use LoRA for finetuning, which is one of the most popular LLM finetuning methods. (For more information on how LoRA works, please see [Code LoRA from Scratch](https://lightning.ai/lightning-ai/studios/code-lora-from-scratch).)
|
||||
|
||||
Before we start, we have to download a model as explained in the previous "Download pretrained model" section above:
|
||||
|
||||
```bash
|
||||
litgpt download microsoft/phi-2
|
||||
```
|
||||
|
||||
The LitGPT interface can be used via command line arguments and configuration files. We recommend starting with the configuration files from the [config_hub](../config_hub) and either modifying them directly or overriding specific settings via the command line. For example, we can use the following setting to train the downloaded 2.7B parameter `microsoft/phi-2` model, where we set `--max_steps 5` for a quick test run.
|
||||
|
||||
If you have downloaded or cloned the LitGPT repository, you can provide the `config` file via a relative path:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora microsoft/phi-2\
|
||||
--config config_hub/finetune/phi-2/lora.yaml \
|
||||
--train.max_steps 5
|
||||
```
|
||||
|
||||
Alternatively, you can provide a URL:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora microsoft/phi-2\
|
||||
--config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/phi-2/lora.yaml \
|
||||
--train.max_steps 5
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Note that the config file above will finetune the model on the `Alpaca2k` dataset on 1 GPU and save the resulting files in an `out/finetune/lora-phi-2` directory. All of these settings can be changed via a respective command line argument or by changing the config file.
|
||||
> To see more options, execute `litgpt finetune_lora --help`.
|
||||
|
||||
|
||||
|
||||
Running the previous finetuning command will initiate the finetuning process, which should only take about a minute on a GPU due to the `--train.max_steps 5` setting.
|
||||
|
||||
```
|
||||
{'checkpoint_dir': PosixPath('checkpoints/microsoft/phi-2'), # TODO
|
||||
'data': Alpaca2k(mask_prompt=False,
|
||||
val_split_fraction=0.03847,
|
||||
prompt_style=<litgpt.prompts.Alpaca object at 0x7f5fa2867e80>,
|
||||
ignore_index=-100,
|
||||
seed=42,
|
||||
num_workers=4,
|
||||
download_dir=PosixPath('data/alpaca2k')),
|
||||
'devices': 1,
|
||||
'eval': EvalArgs(interval=100, max_new_tokens=100, max_iters=100),
|
||||
'logger_name': 'csv',
|
||||
'lora_alpha': 16,
|
||||
'lora_dropout': 0.05,
|
||||
'lora_head': True,
|
||||
'lora_key': True,
|
||||
'lora_mlp': True,
|
||||
'lora_projection': True,
|
||||
'lora_query': True,
|
||||
'lora_r': 8,
|
||||
'lora_value': True,
|
||||
'num_nodes': 1,
|
||||
'out_dir': PosixPath('out/finetune/lora-phi-2'),
|
||||
'precision': 'bf16-true',
|
||||
'quantize': None,
|
||||
'seed': 1337,
|
||||
'train': TrainArgs(save_interval=800,
|
||||
log_interval=1,
|
||||
global_batch_size=8,
|
||||
micro_batch_size=4,
|
||||
lr_warmup_steps=10,
|
||||
epochs=1,
|
||||
max_tokens=None,
|
||||
max_steps=5,
|
||||
max_seq_length=512,
|
||||
tie_embeddings=None,
|
||||
learning_rate=0.0002,
|
||||
weight_decay=0.0,
|
||||
beta1=0.9,
|
||||
beta2=0.95,
|
||||
max_norm=None,
|
||||
min_lr=6e-05)}
|
||||
Seed set to 1337
|
||||
Number of trainable parameters: 12,226,560
|
||||
Number of non-trainable parameters: 2,779,683,840
|
||||
The longest sequence length in the train data is 512, the model's maximum sequence length is 512 and context length is 2048
|
||||
Validating ...
|
||||
Recommend a movie for me to watch during the weekend and explain the reason.
|
||||
Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
||||
|
||||
### Instruction:
|
||||
Recommend a movie for me to watch during the weekend and explain the reason.
|
||||
|
||||
### Response:
|
||||
I recommend you watch "Parasite" because it's a critically acclaimed movie that won multiple awards, including the Academy Award for Best Picture. It's a thought-provoking and suspenseful film that will keep you on the edge of your seat. The movie also tackles social and economic inequalities, making it a must-watch for anyone interested in meaningful storytelling.
|
||||
|
||||
/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/torchmetrics/utilities/prints.py:43: UserWarning: The ``compute`` method of metric MeanMetric was called before the ``update`` method which may lead to errors, as metric states have not yet been updated.
|
||||
warnings.warn(*args, **kwargs) # noqa: B028
|
||||
Missing logger folder: out/finetune/lora-phi-2/logs/csv
|
||||
Epoch 1 | iter 1 step 0 | loss train: 1.646, val: n/a | iter time: 820.31 ms
|
||||
Epoch 1 | iter 2 step 1 | loss train: 1.660, val: n/a | iter time: 548.72 ms (step)
|
||||
Epoch 1 | iter 3 step 1 | loss train: 1.687, val: n/a | iter time: 300.07 ms
|
||||
Epoch 1 | iter 4 step 2 | loss train: 1.597, val: n/a | iter time: 595.27 ms (step)
|
||||
Epoch 1 | iter 5 step 2 | loss train: 1.640, val: n/a | iter time: 260.75 ms
|
||||
Epoch 1 | iter 6 step 3 | loss train: 1.703, val: n/a | iter time: 568.22 ms (step)
|
||||
Epoch 1 | iter 7 step 3 | loss train: 1.678, val: n/a | iter time: 511.70 ms
|
||||
Epoch 1 | iter 8 step 4 | loss train: 1.741, val: n/a | iter time: 514.14 ms (step)
|
||||
Epoch 1 | iter 9 step 4 | loss train: 1.689, val: n/a | iter time: 423.59 ms
|
||||
Epoch 1 | iter 10 step 5 | loss train: 1.524, val: n/a | iter time: 603.03 ms (step)
|
||||
Training time: 11.20s
|
||||
Memory used: 13.90 GB
|
||||
Saving LoRA weights to 'out/finetune/lora-phi-2/final/lit_model.pth.lora'
|
||||
Saved merged weights to 'out/finetune/lora-phi-2/final/lit_model.pth'
|
||||
```
|
||||
|
||||
Notice that the LoRA script saves both the LoRA weights (`'out/finetune/lora-phi-2/final/lit_model.pth.lora'`) and the LoRA weight merged back into the original model (`'out/finetune/lora-phi-2/final/lit_model.pth'`) for convenience. This allows us to use the finetuned model via the `chat` function directly:
|
||||
|
||||
```bash
|
||||
litgpt chat out/finetune/lora-phi-2/final/
|
||||
```
|
||||
|
||||
```
|
||||
Now chatting with phi-2.
|
||||
To exit, press 'Enter' on an empty prompt.
|
||||
|
||||
Seed set to 1234
|
||||
>> Prompt: Why are LLMs so useful?
|
||||
>> Reply: LLMs are useful because they can be trained to perform various natural language tasks, such as language translation, text generation, and question-answering. They are also able to understand the context of the input data, which makes them particularly useful for tasks such as sentiment analysis and text summarization. Additionally, because LLMs can learn from large amounts of data, they are able to generalize well and perform well on new data.
|
||||
|
||||
Time for inference: 2.15 sec total, 39.57 tokens/sec, 85 tokens
|
||||
|
||||
>> Prompt:
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
**More information and additional resources**
|
||||
|
||||
- [tutorials/prepare_dataset](prepare_dataset.md): A summary of all out-of-the-box supported datasets in LitGPT and utilities for preparing custom datasets
|
||||
- [tutorials/finetune](finetune.md): An overview of the different finetuning methods supported in LitGPT
|
||||
- [tutorials/finetune_full](finetune_full.md): A tutorial on full-parameter finetuning
|
||||
- [tutorials/finetune_lora](finetune_lora.md): Options for parameter-efficient finetuning with LoRA and QLoRA
|
||||
- [tutorials/finetune_adapter](finetune_adapter.md): A description of the parameter-efficient Llama-Adapter methods supported in LitGPT
|
||||
- [tutorials/oom](oom.md): Tips for dealing with out-of-memory (OOM) errors
|
||||
- [config_hub/finetune](../config_hub/finetune): Pre-made config files for finetuning that work well out of the box
|
||||
|
||||
|
||||
## LLM inference
|
||||
|
||||
To use a downloaded or finetuned model for chat, you only need to provide the corresponding checkpoint directory containing the model and tokenizer files. For example, to chat with the phi-2 model from Microsoft, download it as follows, as described in the "Download pretrained model" section:
|
||||
|
||||
```bash
|
||||
litgpt download microsoft/phi-2
|
||||
```
|
||||
|
||||
```
|
||||
model-00001-of-00002.safetensors: 100%|████████████████████████████████| 5.00G/5.00G [00:40<00:00, 124MB/s]
|
||||
model-00002-of-00002.safetensors: 100%|████████████████████████████████| 564M/564M [00:01<00:00, 330MB/s]
|
||||
tokenizer.json: 100%|██████████████████████████████████████████████████| 2.11M/2.11M [00:00<00:00, 54.0MB/s]
|
||||
...
|
||||
Converting checkpoint files to LitGPT format.
|
||||
Processing checkpoints/microsoft/phi-2/model-00001-of-00002.bin
|
||||
...
|
||||
Saving converted checkpoint to checkpoints/microsoft/phi-2
|
||||
```
|
||||
|
||||
|
||||
|
||||
Then, chat with the model using the following command:
|
||||
|
||||
```bash
|
||||
litgpt chat microsoft/phi-2
|
||||
```
|
||||
|
||||
```
|
||||
Now chatting with phi-2.
|
||||
To exit, press 'Enter' on an empty prompt.
|
||||
|
||||
Seed set to 1234
|
||||
>> Prompt: What is the main difference between a large language model and a traditional search engine?
|
||||
>> Reply: A large language model uses deep learning algorithms to analyze and generate natural language, while a traditional search engine uses algorithms to retrieve information from web pages.
|
||||
|
||||
Time for inference: 1.14 sec total, 26.26 tokens/sec, 30 tokens
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> Most model weights are already represented in an efficient bfloat16 format. However, if the model currently exceeds your GPU memory, you can try to pass the `--precision bf16-true` option. In addition, you can check the quantization documentation for further optimization, which is linked below.
|
||||
|
||||
|
||||
|
||||
**More information and additional resources**
|
||||
|
||||
- [tutorials/inference](inference.md): Chat and inference tutorial
|
||||
- [tutorials/quantize](quantize.md): Quantizing models to reduce GPU memory requirements
|
||||
|
||||
|
||||
|
||||
## Using the LitGPT Python API for Inference
|
||||
|
||||
The previous section explained how to use the `litgpt chat` command line interface for inference. Alternatively, LitGPT also offers a Python API approach to generate text using an LLM:
|
||||
|
||||
```python
|
||||
from litgpt import LLM
|
||||
|
||||
llm = LLM.load("microsoft/phi-2")
|
||||
text = llm.generate("What do Llamas eat?", top_k=1, max_new_tokens=30)
|
||||
print(text)
|
||||
```
|
||||
|
||||
Note that the if you pass a supported model name to `LLM.load()`, as shown above, it will download the model from the HF hub if it doesn't exist locally, yet (use `litgpt download list` on the command line to get a list of all currently supported models.)
|
||||
|
||||
Alternatively, to load model from a local path, just provide the corresponding path as input to the `load` method:
|
||||
|
||||
```python
|
||||
llm = LLM.load("path/to/my/local/checkpoint")
|
||||
```
|
||||
|
||||
|
||||
**More information and additional resources**
|
||||
|
||||
- [tutorials/python-api](python-api.md): The LitGPT Python API documentation
|
||||
|
||||
|
||||
|
||||
|
||||
## Evaluating models
|
||||
|
||||
LitGPT comes with a handy `litgpt evaluate` command to evaluate models with [Eleuther AI's Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness). For example, to evaluate the previously downloaded `microsoft/phi-2` model on several tasks available from the Evaluation Harness, you can use the following command:
|
||||
|
||||
|
||||
```bash
|
||||
litgpt evaluate microsoft/phi-2
|
||||
--batch_size 16 \
|
||||
--tasks "hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge"
|
||||
```
|
||||
|
||||
(A list of supported tasks can be found [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_table.md).)
|
||||
|
||||
|
||||
|
||||
## Deploy LLMs
|
||||
|
||||
You can deploy LitGPT LLMs using your tool of choice. Below is an example using LitGPT built-in serving capabilities:
|
||||
|
||||
|
||||
```bash
|
||||
# 1) Download a pretrained model (alternatively, use your own finetuned model)
|
||||
litgpt download microsoft/phi-2
|
||||
|
||||
# 2) Start the server
|
||||
litgpt serve microsoft/phi-2
|
||||
```
|
||||
|
||||
```python
|
||||
# 3) Use the server (in a separate session)
|
||||
import requests, json
|
||||
response = requests.post(
|
||||
"http://127.0.0.1:8000/predict",
|
||||
json={"prompt": "Fix typos in the following sentence: Example input"}
|
||||
)
|
||||
print(response.json()["output"])
|
||||
```
|
||||
|
||||
This prints:
|
||||
|
||||
```
|
||||
Instruct: Fix typos in the following sentence: Example input
|
||||
Output: Example input.
|
||||
```
|
||||
|
||||
|
||||
|
||||
**More information and additional resources**
|
||||
|
||||
- [tutorials/deploy](deploy.md): A full deployment tutorial and example
|
||||
|
||||
|
||||
|
||||
## Converting LitGPT model weights to `safetensors` format
|
||||
|
||||
Sometimes, it can be useful to convert LitGPT model weights for third-party and external tools. For example, we can convert a LitGPT model to the Hugging Face format and save it via `.safetensors` files, which we can do as follows:
|
||||
|
||||
```bash
|
||||
litgpt convert_from_litgpt microsoft/phi-2 out/converted_model/
|
||||
```
|
||||
|
||||
Certain tools like the `.from_pretrained` method in Hugging Face `transformers` also require the original `config.json` file that originally came with the downloaded model:
|
||||
|
||||
```bash
|
||||
cp checkpoints/microsoft/phi-2/config.json out/converted_model/config.json
|
||||
```
|
||||
|
||||
You can now load the model into a Hugging Face transformers model and safe it in a `.safetensors` format as follows:
|
||||
|
||||
```bash
|
||||
import torch
|
||||
from transformers import AutoModel
|
||||
|
||||
# Load model
|
||||
state_dict = torch.load('out/converted_model/model.pth')
|
||||
model = AutoModel.from_pretrained(
|
||||
"microsoft/phi-2", state_dict=state_dict
|
||||
)
|
||||
|
||||
# Save .safetensors files
|
||||
model.save_pretrained("out/converted_model/")
|
||||
```
|
||||
|
||||
```
|
||||
⚡ ~/litgpt ls -lh out/converted_model
|
||||
total 16G
|
||||
-rwxr--r-- 1 sebastian sebastian 891 Mar 20 17:08 config.json
|
||||
-rw-r--r-- 1 sebastian sebastian 4.7G Mar 20 17:08 model-00001-of-00003.safetensors
|
||||
-rw-r--r-- 1 sebastian sebastian 4.7G Mar 20 17:09 model-00002-of-00003.safetensors
|
||||
-rw-r--r-- 1 sebastian sebastian 601M Mar 20 17:09 model-00003-of-00003.safetensors
|
||||
-rw-r--r-- 1 sebastian sebastian 5.2G Mar 20 16:30 model.pth
|
||||
-rw-r--r-- 1 sebastian sebastian 33K Mar 20 17:09 model.safetensors.index.json
|
||||
```
|
||||
|
||||
You can then use the model with external tools, for example, Eleuther AI's [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) (see the `lm_eval` installation instructions [here](https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#install)).
|
||||
|
||||
The LM Evaluation Harness requires a tokenizer to be present in the model checkpoint folder, which we can copy from the original download checkpoint:
|
||||
|
||||
```bash
|
||||
# Copy the tokenizer needed by the Eval Harness
|
||||
cp checkpoints/microsoft/phi-2/tokenizer*
|
||||
out/converted_model
|
||||
```
|
||||
|
||||
Then, we can run the Evaluation Harness as follows:
|
||||
|
||||
```bash
|
||||
lm_eval --model hf \
|
||||
--model_args pretrained="out/converted_model" \
|
||||
--tasks "hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge" \
|
||||
--device "cuda:0" \
|
||||
--batch_size 4
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> The Evaluation Harness tasks above are those used in Open LLM Leaderboard. You can find a list all supported tasks [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_table.md).
|
||||
|
||||
|
||||
|
||||
|
||||
**More information and additional resources**
|
||||
|
||||
- [tutorials/convert_lit_models](./convert_lit_models.md): Tutorial on converting LitGPT weights
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Get involved!
|
||||
|
||||
We appreciate your feedback and contributions. If you have feature requests, questions, or want to contribute code or config files, please don't hesitate to use the [GitHub Issue](https://github.com/Lightning-AI/litgpt/issues) tracker.
|
||||
|
||||
We welcome all individual contributors, regardless of their level of experience or hardware. Your contributions are valuable, and we are excited to see what you can accomplish in this collaborative and supportive environment.
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Unsure about contributing? Check out our [How to Contribute to LitGPT](https://lightning.ai/pages/community/tutorial/how-to-contribute-to-litgpt/) guide.
|
||||
|
||||
|
||||
|
||||
If you have general questions about building with LitGPT, please [join our Discord](https://discord.gg/VptPCZkGNa).
|
||||
@@ -0,0 +1,52 @@
|
||||
# Converting Hugging Face Transformers to LitGPT weights
|
||||
|
||||
By default, the `litgpt download` command converts the downloaded HF checkpoint files into a LitGPT compatible format after downloading. For example,
|
||||
|
||||
```bash
|
||||
litgpt download EleutherAI/pythia-14m
|
||||
```
|
||||
|
||||
creates the following files:
|
||||
|
||||
```
|
||||
checkpoints/
|
||||
└── EleutherAI/
|
||||
└── pythia-14m/
|
||||
├── config.json
|
||||
├── generation_config.json
|
||||
├── model_config.yaml # LitGPT specific file
|
||||
├── lit_model.pth # LitGPT specific file
|
||||
├── pytorch_model.bin
|
||||
├── tokenizer.json
|
||||
└── tokenizer_config.json
|
||||
```
|
||||
|
||||
|
||||
|
||||
To disable the automatic conversion, which is useful for development and debugging purposes, you can run the `litgpt download` with the `--convert_checkpoint false` flag. This will only download the checkpoint files but do not convert them for use in LitGPT:
|
||||
|
||||
```bash
|
||||
rm -rf checkpoints/EleutherAI/pythia-14m
|
||||
|
||||
litgpt download EleutherAI/pythia-14m \
|
||||
--convert_checkpoint false
|
||||
|
||||
ls checkpoints/EleutherAI/pythia-14m
|
||||
```
|
||||
|
||||
```
|
||||
checkpoints/
|
||||
└── EleutherAI/
|
||||
└── pythia-14m/
|
||||
├── config.json
|
||||
├── generation_config.json
|
||||
├── pytorch_model.bin
|
||||
├── tokenizer.json
|
||||
└── tokenizer_config.json
|
||||
```
|
||||
|
||||
The required files `model_config.yaml` and `lit_model.pth` files can then be manually generated via the `litgpt/scripts/convert_hf_checkpoint.py` script:
|
||||
|
||||
```bash
|
||||
litgpt convert_to_litgpt checkpoints/EleutherAI/pythia-14m
|
||||
```
|
||||
@@ -0,0 +1,143 @@
|
||||
## Converting LitGPT weights to Hugging Face Transformers
|
||||
|
||||
LitGPT weights need to be converted to a format that Hugging Face understands with a [conversion script](../litgpt/scripts/convert_lit_checkpoint.py) before our scripts can run.
|
||||
|
||||
We provide a helpful command to convert models LitGPT models back to their equivalent Hugging Face Transformers format:
|
||||
|
||||
```bash
|
||||
litgpt convert_from_litgpt checkpoint_dir converted_dir
|
||||
```
|
||||
|
||||
These paths are just placeholders, you will need to customize them based on which finetuning or pretraining command you ran and its configuration.
|
||||
|
||||
### Loading converted LitGPT checkpoints into transformers
|
||||
|
||||
|
||||
For example,
|
||||
|
||||
```bash
|
||||
cp checkpoints/repo_id/config.json converted/config.json
|
||||
```
|
||||
|
||||
Then, you can load the checkpoint file in a Python session as follows:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModel
|
||||
|
||||
|
||||
state_dict = torch.load("output_dir/model.pth")
|
||||
model = AutoModel.from_pretrained(
|
||||
"output_dir/", local_files_only=True, state_dict=state_dict
|
||||
)
|
||||
```
|
||||
|
||||
Alternatively, you can also load the model without copying the `config.json` file as follows:
|
||||
|
||||
```python
|
||||
model = AutoModel.from_pretrained("online_repo_id", state_dict=state_dict)
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Merging LoRA weights
|
||||
|
||||
Please note that if you want to convert a model that has been finetuned using an adapter like LoRA, these weights should be [merged](../litgpt/scripts/merge_lora.py) to the checkpoint prior to converting.
|
||||
|
||||
```sh
|
||||
litgpt merge_lora path/to/lora/checkpoint_dir
|
||||
```
|
||||
|
||||
<br>
|
||||
<br>
|
||||
|
||||
# A finetuning and conversion tutorial
|
||||
|
||||
This section contains a reproducible example for finetuning a LitGPT model and converting it back into a HF `transformer` model.
|
||||
|
||||
1. Download a model of interest:
|
||||
|
||||
For convenience, we first specify an environment variable (optional) to avoid copy and pasting the whole path:
|
||||
|
||||
```bash
|
||||
export repo_id=TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
||||
```
|
||||
|
||||
Instead of using TinyLlama, you can replace the `repo_id` target with any other model repository
|
||||
specifier that is currently supported by LitGPT. You can get a list of supported repository specifier
|
||||
by running `litgpt/scripts/download.py` without any additional arguments.
|
||||
|
||||
Then, we download the model we specified via `$repo_id` above:
|
||||
|
||||
```bash
|
||||
litgpt download $repo_id
|
||||
```
|
||||
|
||||
2. Finetune the model:
|
||||
|
||||
|
||||
```bash
|
||||
export finetuned_dir=out/lit-finetuned-model
|
||||
|
||||
litgpt finetune_lora $repo_id \
|
||||
--out_dir $finetuned_dir \
|
||||
--train.epochs 1 \
|
||||
--data Alpaca
|
||||
```
|
||||
|
||||
3. Merge LoRA weights:
|
||||
|
||||
Note that this step only applies if the model was finetuned with `lora.py` above and not when `full.py` was used for finetuning.
|
||||
|
||||
```bash
|
||||
litgpt merge_lora $finetuned_dir/final
|
||||
```
|
||||
|
||||
|
||||
4. Convert the finetuning model back into a HF format:
|
||||
|
||||
```bash
|
||||
litgpt convert_from_litgpt $finetuned_dir/final/ out/hf-tinyllama/converted
|
||||
```
|
||||
|
||||
|
||||
5. Load the model into a `transformers` model:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModel
|
||||
|
||||
state_dict = torch.load('out/hf-tinyllama/converted/model.pth')
|
||||
model = AutoModel.from_pretrained("TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T", state_dict=state_dict)
|
||||
```
|
||||
|
||||
|
||||
## Using the LM Evaluation Harness
|
||||
|
||||
To evaluate LitGPT models, use the integrated evaluation utilities based on Eleuther AI's LM Evaluation Harness. For more information, please see the [evaluation](evaluation.md) documentation.
|
||||
|
||||
Alternatively, if you wish to use converted LitGPT models with the LM Evaluation Harness from [Eleuther AI's GitHub repository](https://github.com/EleutherAI/lm-evaluation-harness), you can use the following steps.
|
||||
|
||||
1. Follow the instructions above to load the model into a Hugging Face transformers model.
|
||||
|
||||
2. Create a `model.safetensor` file:
|
||||
|
||||
```python
|
||||
model.save_pretrained("out/hf-tinyllama/converted/")
|
||||
```
|
||||
|
||||
3. Copy the tokenizer files into the model-containing directory:
|
||||
|
||||
```bash
|
||||
cp checkpoints/$repo_id/tokenizer* out/hf-tinyllama/converted
|
||||
```
|
||||
|
||||
4. Run the evaluation harness, for example:
|
||||
|
||||
```bash
|
||||
lm_eval --model hf \
|
||||
--model_args pretrained=out/hf-tinyllama/converted \
|
||||
--tasks "hellaswag,gsm8k,truthfulqa_mc2,mmlu,winogrande,arc_challenge" \
|
||||
--device "cuda:0" \
|
||||
--batch_size 4
|
||||
```
|
||||
@@ -0,0 +1,144 @@
|
||||
# Serve and Deploy LLMs
|
||||
|
||||
This document shows how you can serve a LitGPT for deployment.
|
||||
|
||||
|
||||
|
||||
## Serve an LLM with LitServe
|
||||
|
||||
This section illustrates how we can set up an inference server for a phi-2 LLM using `litgpt serve` that is minimal and highly scalable.
|
||||
|
||||
|
||||
|
||||
### Step 1: Start the inference server
|
||||
|
||||
|
||||
```bash
|
||||
# 1) Download a pretrained model (alternatively, use your own finetuned model)
|
||||
litgpt download microsoft/phi-2
|
||||
|
||||
# 2) Start the server
|
||||
litgpt serve microsoft/phi-2
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt serve --help` to display additional options, including the port, devices, LLM temperature setting, and more.
|
||||
|
||||
|
||||
|
||||
### Step 2: Query the inference server
|
||||
|
||||
You can now send requests to the inference server you started in step 2. For example, in a new Python session, we can send requests to the inference server as follows:
|
||||
|
||||
|
||||
```python
|
||||
import requests, json
|
||||
|
||||
response = requests.post(
|
||||
"http://127.0.0.1:8000/predict",
|
||||
json={"prompt": "Fix typos in the following sentence: Example input"}
|
||||
)
|
||||
|
||||
print(response.json()["output"])
|
||||
```
|
||||
|
||||
Executing the code above prints the following output:
|
||||
|
||||
```
|
||||
Example input.
|
||||
```
|
||||
|
||||
|
||||
### Optional: Use the streaming mode
|
||||
|
||||
The 2-step procedure described above returns the complete response all at once. If you want to stream the response on a token-by-token basis, start the server with the streaming option enabled:
|
||||
|
||||
```bash
|
||||
litgpt serve microsoft/phi-2 --stream true
|
||||
```
|
||||
|
||||
Then, use the following updated code to query the inference server:
|
||||
|
||||
```python
|
||||
import requests, json
|
||||
|
||||
response = requests.post(
|
||||
"http://127.0.0.1:8000/predict",
|
||||
json={"prompt": "Fix typos in the following sentence: Example input"},
|
||||
stream=True
|
||||
)
|
||||
|
||||
# stream the response
|
||||
for line in response.iter_lines(decode_unicode=True):
|
||||
if line:
|
||||
print(json.loads(line)["output"], end="")
|
||||
```
|
||||
|
||||
```
|
||||
Sure, here is the corrected sentence:
|
||||
|
||||
Example input
|
||||
```
|
||||
|
||||
|
||||
## Serve an LLM with OpenAI-compatible API
|
||||
|
||||
LitGPT provides OpenAI-compatible endpoints that allow you to use the OpenAI SDK or any OpenAI-compatible client to interact with your models. This is useful for integrating LitGPT into existing applications that use the OpenAI API.
|
||||
|
||||
|
||||
### Step 1: Start the server with OpenAI specification
|
||||
|
||||
```bash
|
||||
# 1) Download a pretrained model (alternatively, use your own finetuned model)
|
||||
litgpt download HuggingFaceTB/SmolLM2-135M-Instruct
|
||||
|
||||
# 2) Start the server with OpenAI-compatible endpoints
|
||||
litgpt serve HuggingFaceTB/SmolLM2-135M-Instruct --openai_spec true
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> The `--openai_spec true` flag enables OpenAI-compatible endpoints at `/v1/chat/completions` instead of the default `/predict` endpoint.
|
||||
|
||||
|
||||
### Step 2: Query using OpenAI-compatible endpoints
|
||||
|
||||
You can now send requests to the OpenAI-compatible endpoint using curl:
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "SmolLM2-135M-Instruct",
|
||||
"messages": [{"role": "user", "content": "Hello! How are you?"}]
|
||||
}'
|
||||
```
|
||||
|
||||
Or use the OpenAI Python SDK:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
# Configure the client to use your local LitGPT server
|
||||
client = OpenAI(
|
||||
base_url="http://127.0.0.1:8000/v1",
|
||||
api_key="not-needed" # LitGPT doesn't require authentication by default
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="SmolLM2-135M-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "Hello! How are you?"}
|
||||
]
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
|
||||
## Serve an LLM UI with Chainlit
|
||||
|
||||
If you are interested in developing a simple ChatGPT-like UI prototype, see the Chainlit tutorial in the following Studio:
|
||||
|
||||
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/chatgpt-like-llm-uis-via-chainlit">
|
||||
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg" alt="Open In Studio"/>
|
||||
</a>
|
||||
@@ -0,0 +1 @@
|
||||
LitGPT developer documentation files.
|
||||
@@ -0,0 +1,262 @@
|
||||
# Adding New Models
|
||||
|
||||
This document provides an overview and explanation of how new LLM architectures and model weights can be added to LitGPT.
|
||||
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> One of the design focus areas of LitGPT is to provide efficient readable code. At the same time, LitGPT aims to support selected LLMs that are useful to the community. LitGPT aims to reuse and share as much code as possible between different LLMs to strike a balance between code readability and enabling support for various LLMs. In short, we try to minimize writing custom code for a given LLM and aim for code reuse.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## 1. Discuss the LLM to be added
|
||||
|
||||
As an open-source project, we appreciate your contributions! However, before you begin putting valuable time and work into a contribution, ideally, open an issue to discuss whether support for a certain model is within the project's scope.
|
||||
|
||||
|
||||
## 2. Set up your development environment
|
||||
|
||||
Clone the repository:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Lightning-AI/litgpt.git
|
||||
```
|
||||
|
||||
Then, install it with the "editable" mode for development:
|
||||
|
||||
```bash
|
||||
cd litgpt
|
||||
pip install litgpt -e ".[all]"
|
||||
```
|
||||
|
||||
|
||||
## 3. Update the config file
|
||||
|
||||
Update the [litgpt/config.py](../../litgpt/config.py) config file, adding the new model configuration there. It's easiest to start with the most similar model, copy the configuration, and then modify it according to the `config.json` file on the HF hub.
|
||||
|
||||
For example, suppose an entry for Llama 3 8B already exists and you want to add support for Llama 3 70B.
|
||||
|
||||
Copy the Llama 3 8B entry:
|
||||
|
||||
```python
|
||||
# https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/config.json
|
||||
dict(
|
||||
name="Llama-3-8B{}",
|
||||
hf_config=dict(org="meta-llama", name="Meta-Llama-3-8B{}"),
|
||||
vocab_size=128256,
|
||||
padding_multiple=64,
|
||||
n_layer=32,
|
||||
n_head=32,
|
||||
n_query_groups=8,
|
||||
rotary_percentage=1.0,
|
||||
parallel_residual=False,
|
||||
bias=False,
|
||||
norm_class_name="RMSNorm",
|
||||
mlp_class_name="LLaMAMLP",
|
||||
intermediate_size=14336,
|
||||
rope_base=500000,
|
||||
),
|
||||
```
|
||||
|
||||
Then create the entry for the 70B model. Here, make sure you update the values according to the `config.json` file available on the HF hub:
|
||||
|
||||
```python
|
||||
# https://huggingface.co/meta-llama/Meta-Llama-3-70B/blob/main/config.json
|
||||
dict(
|
||||
name="Llama-3-70B{}",
|
||||
hf_config=dict(org="meta-llama", name="Meta-Llama-3-70B{}"),
|
||||
vocab_size=128256,
|
||||
padding_multiple=64,
|
||||
n_layer=80,
|
||||
n_head=64,
|
||||
n_embd=8192,
|
||||
n_query_groups=8,
|
||||
rotary_percentage=1.0,
|
||||
parallel_residual=False,
|
||||
bias=False,
|
||||
norm_class_name="RMSNorm",
|
||||
mlp_class_name="LLaMAMLP",
|
||||
intermediate_size=28672,
|
||||
rope_base=500000,
|
||||
),
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> Some models may require you to implement a new MLP class analogous to `class LLaMAMLP`.
|
||||
> A more or less reliable indicator is the presence of a `modeling.py` file in the model's original repository.
|
||||
> If this file exists, it suggests that this model requires custom code.
|
||||
> This will then also require additional changes beyond simply updating
|
||||
> the configuration in LitGPT's `config.py`.
|
||||
|
||||
|
||||
## 4. Try downloading the model
|
||||
|
||||
After making the modifications above, try downloading the model:
|
||||
|
||||
```bash
|
||||
litgpt download meta-llama/Meta-Llama-3-70B --access_token ...
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> Not all models require an access token
|
||||
|
||||
|
||||
|
||||
If the conversion following the download fails, proceed with the next section.
|
||||
|
||||
|
||||
## 5. Update the checkpoint conversion script
|
||||
|
||||
If the `litgpt download ...` command from the previous section failed, you may have to adjust the checkpoint conversion script: [litgpt/scripts/convert_hf_checkpoint.py](../../litgpt/scripts/convert_hf_checkpoint.py).
|
||||
|
||||
Here, you may have to adjust or implement a new `def copy_weights_hf_...` function.
|
||||
|
||||
You can test the updated conversion code without needing to redownload the weights as follows:
|
||||
|
||||
```bash
|
||||
python litgpt/scripts/convert_hf_checkpoint.py meta-llama/Meta-Llama-3-70B
|
||||
```
|
||||
|
||||
|
||||
## 6. Add the Prompt Style
|
||||
|
||||
If you are adding a new model class, find out its prompt style. First, check [litgpt/prompts.py](../../litgpt/prompts.py) if a similar prompt style template already exists. For Llama 3, this is as follows:
|
||||
|
||||
```python
|
||||
class Llama3(PromptStyle):
|
||||
def apply(self, prompt: str, **kwargs: str) -> str:
|
||||
# https://github.com/meta-llama/llama3/blob/359887376f0aaf30e433f23e25df858d8c2a9833/llama/tokenizer.py#L202-L229
|
||||
return (
|
||||
"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n"
|
||||
"You are a helpful assistant.<|eot_id|>\n" # The system prompt is optional
|
||||
"<|start_header_id|>user<|end_header_id|>\n\n"
|
||||
f"{prompt}<|eot_id|>\n"
|
||||
"<|start_header_id|>assistant<|end_header_id|>\n\n"
|
||||
)
|
||||
|
||||
def stop_tokens(self, tokenizer: "Tokenizer") -> Tuple[List[int], ...]:
|
||||
return (
|
||||
[tokenizer.eos_id],
|
||||
[tokenizer.token_to_id("<|eot_id|>")],
|
||||
)
|
||||
```
|
||||
|
||||
If your model requires a different prompt template, create a new `PromptStyle` class.
|
||||
|
||||
Then, in the same file, update the `prompt_styles` dictionary:
|
||||
|
||||
```python
|
||||
prompt_styles: Dict[str, Type[PromptStyle]] = {
|
||||
...
|
||||
"llama3": Llama3,
|
||||
}
|
||||
```
|
||||
|
||||
Finally, also in the same file, update the `model_name_to_prompt_style` function:
|
||||
|
||||
```python
|
||||
def model_name_to_prompt_style(model_name: str) -> PromptStyle:
|
||||
...
|
||||
if re.search("Llama-3.*-Instruct", model_name):
|
||||
return Llama3()
|
||||
```
|
||||
|
||||
|
||||
## 7. Try using the model for inference
|
||||
|
||||
Next, use the model to see if inference works:
|
||||
|
||||
```bash
|
||||
litgpt generate meta-llama/Meta-Llama-3-70B
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> If you notice that the model produces non-sensible language outputs, you need to double-check the config file and find out if there are incorrect values or other problems. The next section on adding unit tests may offer additional pointers.
|
||||
|
||||
|
||||
|
||||
|
||||
## 8. Add unit tests
|
||||
|
||||
|
||||
### 8.1 Add model unit tests
|
||||
|
||||
Open the [`tests/test_model.py`](../../tests/test_model.py) file and add a new `def test_against_hf_...` function using one of the existing functions as a template. For instance,
|
||||
|
||||
```python
|
||||
def test_against_hf_llama2(ours_kwargs, device, dtype):
|
||||
...
|
||||
# test end to end
|
||||
x = torch.tensor([[9856, 23, 491, 1536, 304]], dtype=torch.int32, device=device)
|
||||
assert x.size(1) == T
|
||||
ours_y = ours_model(x)
|
||||
theirs_y = theirs_model(x)["logits"].to(dtype) # HF converts logits to float
|
||||
torch.testing.assert_close(ours_y, theirs_y)
|
||||
```
|
||||
|
||||
If the
|
||||
|
||||
```bash
|
||||
litgpt generate meta-llama/Meta-Llama-3-70B
|
||||
```
|
||||
|
||||
command from the previous section produces incoherent text, this function can be a helpful guide for debugging. For this, modify the implementation in `transformers` and `litgpt` packages (on your local installation), to inspect or print out the intermediate values at a layer. It's recommend starting with the embedding layers and then go through one layer at the time, to find out where the values differ to get pointers for debugging.
|
||||
|
||||
Test the unit test via
|
||||
|
||||
```python
|
||||
pytest tests/test_model.py::test_against_hf_...
|
||||
```
|
||||
|
||||
|
||||
### 8.2 Add prompt style unit test
|
||||
|
||||
Open the [`tests/test_model.py`](../../tests/test_model.py) file and add a test for the respective prompts you added earlier, if applicable. For example,
|
||||
|
||||
|
||||
```python
|
||||
def test_prompt_style_from_config():
|
||||
model_names = [
|
||||
...
|
||||
"Llama-3-70B-Instruct",
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
Run the unit test via
|
||||
|
||||
```python
|
||||
pytest tests/test_prompts.py
|
||||
```
|
||||
|
||||
|
||||
## 9. Try finetuning the model
|
||||
|
||||
Now, try finetuning the model:
|
||||
|
||||
```bash
|
||||
litgpt finetune meta-llama/Meta-Llama-3-70B --train.max_steps 10
|
||||
```
|
||||
|
||||
|
||||
## 10. Update the documentation
|
||||
|
||||
Finally, update the documentation files.
|
||||
|
||||
|
||||
### 10.1 Update the README file
|
||||
|
||||
Update the "All Models" table in the [README.md](../../README.md) file.
|
||||
|
||||
|
||||
### 10.2 Update the download tutorials
|
||||
|
||||
Add the new model to the model table at the top as well as to the list under `litgpt download list`.
|
||||
@@ -0,0 +1,108 @@
|
||||
# LitGPT High-level Python API
|
||||
|
||||
This is a work-in-progress draft for a high-level LitGPT Python API.
|
||||
|
||||
|
||||
## Model loading & saving
|
||||
|
||||
The `LLM.load` command loads an `llm` object, which contains both the model object (a PyTorch module) and a preprocessor.
|
||||
|
||||
```python
|
||||
from litgpt import LLM
|
||||
|
||||
llm = LLM.load(
|
||||
model="url | local_path",
|
||||
# high-level user only needs to care about those:
|
||||
memory_reduction="none | medium | strong"
|
||||
# advanced options for technical users:
|
||||
source="hf | local | other"
|
||||
quantize="bnb.nf4",
|
||||
precision="bf16-true",
|
||||
device=""auto | cuda | cpu",
|
||||
)
|
||||
```
|
||||
|
||||
Here,
|
||||
|
||||
- `llm.model` contains the PyTorch Module
|
||||
- and `llm.preprocessor.tokenizer` contains the tokenizer
|
||||
|
||||
The `llm.save` command saves the model weights, tokenizer, and configuration information.
|
||||
|
||||
|
||||
```python
|
||||
llm.save(checkpoint_dir, format="lightning | ollama | hf")
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Inference / Chat
|
||||
|
||||
```
|
||||
response = llm.generate(
|
||||
prompt="What do Llamas eat?",
|
||||
temperature=0.1,
|
||||
top_p=0.8,
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Dataset
|
||||
|
||||
The `llm.prepare_dataset` command prepares a dataset for training.
|
||||
|
||||
```
|
||||
llm.download_dataset(
|
||||
URL,
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
```
|
||||
dataset = llm.prepare_dataset(
|
||||
path,
|
||||
task="pretrain | instruction_finetune",
|
||||
test_portion=0.1,
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## Training
|
||||
|
||||
|
||||
```python
|
||||
llm.instruction_finetune(
|
||||
config=None,
|
||||
dataset=dataset,
|
||||
max_iter=10,
|
||||
method="full | lora | adapter | adapter_v2"
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
llm.pretrain(config=None, dataset=dataset, max_iter=10, ...)
|
||||
```
|
||||
|
||||
|
||||
## Serving
|
||||
|
||||
|
||||
```python
|
||||
llm.serve(port=8000)
|
||||
```
|
||||
|
||||
Then in another Python session:
|
||||
|
||||
```python
|
||||
import requests, json
|
||||
|
||||
response = requests.post(
|
||||
"http://127.0.0.1:8000/predict",
|
||||
json={"prompt": "Fix typos in the following sentence: Example input"}
|
||||
)
|
||||
|
||||
print(response.json()["output"])
|
||||
```
|
||||
@@ -0,0 +1,454 @@
|
||||
# Download Model Weights with LitGPT
|
||||
|
||||
LitGPT supports a variety of LLM architectures with publicly available weights. You can download model weights and access a list of supported models using the `litgpt download list` command.
|
||||
|
||||
|
||||
|
||||
|
||||
| Model | Model size | Author | Reference |
|
||||
|----|----|----|----|
|
||||
| CodeGemma | 7B | Google | [Google Team, Google Deepmind](https://ai.google.dev/gemma/docs/codegemma) |
|
||||
| Code Llama | 7B, 13B, 34B, 70B | Meta AI | [Rozière et al. 2023](https://arxiv.org/abs/2308.12950) |
|
||||
| Danube2 | 1.8B | H2O.ai | [H2O.ai](https://h2o.ai/platform/danube-1-8b/) |
|
||||
| Dolly | 3B, 7B, 12B | Databricks | [Conover et al. 2023](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) |
|
||||
| Falcon | 7B, 40B, 180B | TII UAE | [TII 2023](https://falconllm.tii.ae) |
|
||||
| Falcon 3 | 1B, 3B, 7B, 10B | TII UAE | [TII 2024](https://huggingface.co/blog/falcon3) |
|
||||
| FreeWilly2 (Stable Beluga 2) | 70B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stable-beluga-large-instruction-fine-tuned-models) |
|
||||
| Function Calling Llama 2 | 7B | Trelis | [Trelis et al. 2023](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-v2) |
|
||||
| Gemma | 2B, 7B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf) |
|
||||
| Gemma 2 | 2B, 9B, 27B | Google | [Google Team, Google Deepmind](https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf) |
|
||||
| Gemma 3 | 1B, 4B, 12B, 27B | Google | [Google Team, Google Deepmind](https://arxiv.org/pdf/2503.19786)
|
||||
| Llama 2 | 7B, 13B, 70B | Meta AI | [Touvron et al. 2023](https://arxiv.org/abs/2307.09288) |
|
||||
| Llama 3 | 8B, 70B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
|
||||
| Llama 3.1 | 8B, 70B, 405B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama3) |
|
||||
| Llama 3.2 | 1B, 3B | Meta AI | [Meta AI 2024](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md) |
|
||||
| Llama 3.3 | 70B | Meta AI | [Meta AI 2024](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) |
|
||||
| Llama 3.1 Nemotron | 70B | NVIDIA | [NVIDIA AI 2024](https://build.nvidia.com/nvidia/llama-3_1-nemotron-70b-instruct/modelcard) |
|
||||
| LongChat | 7B, 13B | LMSYS | [LongChat Team 2023](https://lmsys.org/blog/2023-06-29-longchat/) |
|
||||
| Mathstral | 7B | Mistral AI | [Mistral AI 2024](https://mistral.ai/news/mathstral/) |
|
||||
| MicroLlama | 300M | Ken Wang | [MicroLlama repo](https://github.com/keeeeenw/MicroLlama)
|
||||
| Mixtral MoE | 8x7B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/mixtral-of-experts/) |
|
||||
| Mistral | 7B, 123B | Mistral AI | [Mistral AI 2023](https://mistral.ai/news/announcing-mistral-7b/) |
|
||||
| Mixtral MoE | 8x22B | Mistral AI | [Mistral AI 2024](https://mistral.ai/news/mixtral-8x22b/) |
|
||||
| Nous-Hermes | 7B, 13B, 70B | NousResearch | [Org page](https://huggingface.co/NousResearch) |
|
||||
| OLMo | 1B, 7B | Allen Institute for AI (AI2) | [Groeneveld et al. 2024](https://aclanthology.org/2024.acl-long.841/) |
|
||||
| OpenLLaMA | 3B, 7B, 13B | OpenLM Research | [Geng & Liu 2023](https://github.com/openlm-research/open_llama) |
|
||||
| Phi 1.5 & 2 | 1.3B, 2.7B | Microsoft Research | [Li et al. 2023](https://arxiv.org/abs/2309.05463) |
|
||||
| Phi 3 & 3.5 | 3.8B | Microsoft Research | [Abdin et al. 2024](https://arxiv.org/abs/2404.14219)
|
||||
| Phi 4 | 14B | Microsoft Research | [Abdin et al. 2024](https://arxiv.org/abs/2412.08905) |
|
||||
| Phi 4 Mini Instruct | 3.8B | Microsoft Research | [Microsoft 2025](https://arxiv.org/abs/2503.01743) |
|
||||
| Phi 4 Mini Reasoning | 3.8B | Microsoft Research | [Xu, Peng et al. 2025](https://arxiv.org/abs/2504.21233) |
|
||||
| Phi 4 Reasoning | 3.8B | Microsoft Research | [Abdin et al. 2025](https://arxiv.org/abs/2504.21318) |
|
||||
| Phi 4 Reasoning Plus | 3.8B | Microsoft Research | [Abdin et al. 2025](https://arxiv.org/abs/2504.21318) |
|
||||
| Platypus | 7B, 13B, 70B | Lee et al. | [Lee, Hunter, and Ruiz 2023](https://arxiv.org/abs/2308.07317) |
|
||||
| Pythia | {14,31,70,160,410}M, {1,1.4,2.8,6.9,12}B | EleutherAI | [Biderman et al. 2023](https://arxiv.org/abs/2304.01373) |
|
||||
| Qwen2.5 | 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B | Alibaba Group | [Qwen Team 2024](https://qwenlm.github.io/blog/qwen2.5/) |
|
||||
| Qwen2.5 Coder | 0.5B, 1.5B, 3B, 7B, 14B, 32B | Alibaba Group | [Hui, Binyuan et al. 2024](https://arxiv.org/abs/2409.12186) |
|
||||
| Qwen2.5 1M (Long Context) | 7B, 14B | Alibaba Group | [Qwen Team 2025](https://qwenlm.github.io/blog/qwen2.5-1m/) |
|
||||
| Qwen2.5 Math | 1.5B, 7B, 72B | Alibaba Group | [An, Yang et al. 2024](https://arxiv.org/abs/2409.12122) |
|
||||
| QwQ | 32B | Alibaba Group | [Qwen Team 2025](https://qwenlm.github.io/blog/qwq-32b/) |
|
||||
| QwQ-Preview | 32B | Alibaba Group | [Qwen Team 2024](https://qwenlm.github.io/blog/qwq-32b-preview/) |
|
||||
| Qwen3 | 0.6B, 1.7B, 4B{Hybrid, Thinking-2507, Instruct-2507}, 8B, 14B, 32B | Alibaba Group | [Qwen Team 2025](https://arxiv.org/abs/2505.09388/) |
|
||||
| Qwen3 MoE | 30B{Hybrid, Thinking-2507, Instruct-2507}, 235B{Hybrid, Thinking-2507, Instruct-2507} | Alibaba Group | [Qwen Team 2025](https://arxiv.org/abs/2505.09388/) |
|
||||
| R1 Distll Llama | 8B, 70B | DeepSeek AI | [DeepSeek AI 2025](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf) |
|
||||
| RedPajama-INCITE | 3B, 7B | Together | [Together 2023](https://together.ai/blog/redpajama-models-v1) |
|
||||
| SmolLM2 | 135M, 360M, 1.7B | Hugging Face | [Hugging Face 2024](https://github.com/huggingface/smollm) |
|
||||
| StableCode | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) |
|
||||
| Salamandra | 2B, 7B | Barcelona Supercomputing Centre | [BSC-LTC 2024](https://github.com/BSC-LTC/salamandra) |
|
||||
| StableLM | 3B, 7B | Stability AI | [Stability AI 2023](https://github.com/Stability-AI/StableLM) |
|
||||
| StableLM Zephyr | 3B | Stability AI | [Stability AI 2023](https://stability.ai/blog/stablecode-llm-generative-ai-coding) |
|
||||
| TinyLlama | 1.1B | Zhang et al. | [Zhang et al. 2023](https://github.com/jzhang38/TinyLlama) |
|
||||
| Vicuna | 7B, 13B, 33B | LMSYS | [Li et al. 2023](https://lmsys.org/blog/2023-03-30-vicuna/) | |
|
||||
|
||||
|
||||
|
||||
## General Instructions
|
||||
|
||||
### 1. List Available Models
|
||||
|
||||
To see all supported models, run the following command:
|
||||
|
||||
```bash
|
||||
litgpt download list
|
||||
```
|
||||
|
||||
The output is shown below:
|
||||
|
||||
```
|
||||
allenai/OLMo-1B-hf
|
||||
allenai/OLMo-7B-hf
|
||||
allenai/OLMo-7B-Instruct-hf
|
||||
bsc-lt/salamandra-2b
|
||||
bsc-lt/salamandra-2b-instruct
|
||||
bsc-lt/salamandra-7b
|
||||
bsc-lt/salamandra-7b-instruct
|
||||
codellama/CodeLlama-13b-hf
|
||||
codellama/CodeLlama-13b-Instruct-hf
|
||||
codellama/CodeLlama-13b-Python-hf
|
||||
codellama/CodeLlama-34b-hf
|
||||
codellama/CodeLlama-34b-Instruct-hf
|
||||
codellama/CodeLlama-34b-Python-hf
|
||||
codellama/CodeLlama-70b-hf
|
||||
codellama/CodeLlama-70b-Instruct-hf
|
||||
codellama/CodeLlama-70b-Python-hf
|
||||
codellama/CodeLlama-7b-hf
|
||||
codellama/CodeLlama-7b-Instruct-hf
|
||||
codellama/CodeLlama-7b-Python-hf
|
||||
databricks/dolly-v2-12b
|
||||
databricks/dolly-v2-3b
|
||||
databricks/dolly-v2-7b
|
||||
deepseek-ai/DeepSeek-R1-Distill-Llama-8B
|
||||
deepseek-ai/DeepSeek-R1-Distill-Llama-70B
|
||||
EleutherAI/pythia-1.4b
|
||||
EleutherAI/pythia-1.4b-deduped
|
||||
EleutherAI/pythia-12b
|
||||
EleutherAI/pythia-12b-deduped
|
||||
EleutherAI/pythia-14m
|
||||
EleutherAI/pythia-160m
|
||||
EleutherAI/pythia-160m-deduped
|
||||
EleutherAI/pythia-1b
|
||||
EleutherAI/pythia-1b-deduped
|
||||
EleutherAI/pythia-2.8b
|
||||
EleutherAI/pythia-2.8b-deduped
|
||||
EleutherAI/pythia-31m
|
||||
EleutherAI/pythia-410m
|
||||
EleutherAI/pythia-410m-deduped
|
||||
EleutherAI/pythia-6.9b
|
||||
EleutherAI/pythia-6.9b-deduped
|
||||
EleutherAI/pythia-70m
|
||||
EleutherAI/pythia-70m-deduped
|
||||
garage-bAInd/Camel-Platypus2-13B
|
||||
garage-bAInd/Camel-Platypus2-70B
|
||||
garage-bAInd/Platypus-30B
|
||||
garage-bAInd/Platypus2-13B
|
||||
garage-bAInd/Platypus2-70B
|
||||
garage-bAInd/Platypus2-70B-instruct
|
||||
garage-bAInd/Platypus2-7B
|
||||
garage-bAInd/Stable-Platypus2-13B
|
||||
google/codegemma-7b-it
|
||||
google/gemma-3-27b-it
|
||||
google/gemma-3-12b-it
|
||||
google/gemma-3-4b-it
|
||||
google/gemma-3-1b-it
|
||||
google/gemma-2-27b
|
||||
google/gemma-2-27b-it
|
||||
google/gemma-2-2b
|
||||
google/gemma-2-2b-it
|
||||
google/gemma-2-9b
|
||||
google/gemma-2-9b-it
|
||||
google/gemma-2b
|
||||
google/gemma-2b-it
|
||||
google/gemma-7b
|
||||
google/gemma-7b-it
|
||||
h2oai/h2o-danube2-1.8b-chat
|
||||
HuggingFaceTB/SmolLM2-135M
|
||||
HuggingFaceTB/SmolLM2-135M-Instruct
|
||||
HuggingFaceTB/SmolLM2-360M
|
||||
HuggingFaceTB/SmolLM2-360M-Instruct
|
||||
HuggingFaceTB/SmolLM2-1.7B
|
||||
HuggingFaceTB/SmolLM2-1.7B-Instruct
|
||||
lmsys/longchat-13b-16k
|
||||
lmsys/longchat-7b-16k
|
||||
lmsys/vicuna-13b-v1.3
|
||||
lmsys/vicuna-13b-v1.5
|
||||
lmsys/vicuna-13b-v1.5-16k
|
||||
lmsys/vicuna-33b-v1.3
|
||||
lmsys/vicuna-7b-v1.3
|
||||
lmsys/vicuna-7b-v1.5
|
||||
lmsys/vicuna-7b-v1.5-16k
|
||||
meta-llama/Llama-2-13b-chat-hf
|
||||
meta-llama/Llama-2-13b-hf
|
||||
meta-llama/Llama-2-70b-chat-hf
|
||||
meta-llama/Llama-2-70b-hf
|
||||
meta-llama/Llama-2-7b-chat-hf
|
||||
meta-llama/Llama-2-7b-hf
|
||||
meta-llama/Llama-3.2-1B
|
||||
meta-llama/Llama-3.2-1B-Instruct
|
||||
meta-llama/Llama-3.2-3B
|
||||
meta-llama/Llama-3.2-3B-Instruct
|
||||
meta-llama/Llama-3.3-70B-Instruct
|
||||
meta-llama/Meta-Llama-3-70B
|
||||
meta-llama/Meta-Llama-3-70B-Instruct
|
||||
meta-llama/Meta-Llama-3-8B
|
||||
meta-llama/Meta-Llama-3-8B-Instruct
|
||||
meta-llama/Meta-Llama-3.1-405B
|
||||
meta-llama/Meta-Llama-3.1-405B-Instruct
|
||||
meta-llama/Meta-Llama-3.1-70B
|
||||
meta-llama/Meta-Llama-3.1-70B-Instruct
|
||||
meta-llama/Meta-Llama-3.1-8B
|
||||
meta-llama/Meta-Llama-3.1-8B-Instruct
|
||||
microsoft/phi-1_5
|
||||
microsoft/phi-2
|
||||
microsoft/Phi-3-mini-128k-instruct
|
||||
microsoft/Phi-3-mini-4k-instruct
|
||||
microsoft/Phi-3.5-mini-instruct
|
||||
microsoft/phi-4
|
||||
microsoft/Phi-4-mini-instruct
|
||||
mistralai/mathstral-7B-v0.1
|
||||
mistralai/Mistral-7B-Instruct-v0.1
|
||||
mistralai/Mistral-7B-Instruct-v0.2
|
||||
mistralai/Mistral-7B-Instruct-v0.3
|
||||
mistralai/Mistral-7B-v0.1
|
||||
mistralai/Mistral-7B-v0.3
|
||||
mistralai/Mistral-Large-Instruct-2407
|
||||
mistralai/Mistral-Large-Instruct-2411
|
||||
mistralai/Mixtral-8x7B-Instruct-v0.1
|
||||
mistralai/Mixtral-8x7B-v0.1
|
||||
mistralai/Mixtral-8x22B-Instruct-v0.1
|
||||
mistralai/Mixtral-8x22B-v0.1
|
||||
NousResearch/Nous-Hermes-13b
|
||||
NousResearch/Nous-Hermes-llama-2-7b
|
||||
NousResearch/Nous-Hermes-Llama2-13b
|
||||
nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
|
||||
openlm-research/open_llama_13b
|
||||
openlm-research/open_llama_3b
|
||||
openlm-research/open_llama_7b
|
||||
Qwen/Qwen2.5-0.5B
|
||||
Qwen/Qwen2.5-0.5B-Instruct
|
||||
Qwen/Qwen2.5-1.5B
|
||||
Qwen/Qwen2.5-1.5B-Instruct
|
||||
Qwen/Qwen2.5-3B
|
||||
Qwen/Qwen2.5-3B-Instruct
|
||||
Qwen/Qwen2.5-7B
|
||||
Qwen/Qwen2.5-7B-Instruct
|
||||
Qwen/Qwen2.5-7B-Instruct-1M
|
||||
Qwen/Qwen2.5-14B
|
||||
Qwen/Qwen2.5-14B-Instruct
|
||||
Qwen/Qwen2.5-14B-Instruct-1M
|
||||
Qwen/Qwen2.5-32B
|
||||
Qwen/Qwen2.5-32B-Instruct
|
||||
Qwen/Qwen2.5-72B
|
||||
Qwen/Qwen2.5-72B-Instruct
|
||||
Qwen/Qwen2.5-Coder-0.5B
|
||||
Qwen/Qwen2.5-Coder-0.5B-Instruct
|
||||
Qwen/Qwen2.5-Coder-1.5B
|
||||
Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||||
Qwen/Qwen2.5-Coder-3B
|
||||
Qwen/Qwen2.5-Coder-3B-Instruct
|
||||
Qwen/Qwen2.5-Coder-7B
|
||||
Qwen/Qwen2.5-Coder-7B-Instruct
|
||||
Qwen/Qwen2.5-Coder-14B
|
||||
Qwen/Qwen2.5-Coder-14B-Instruct
|
||||
Qwen/Qwen2.5-Coder-32B
|
||||
Qwen/Qwen2.5-Coder-32B-Instruct
|
||||
Qwen/Qwen2.5-Math-1.5B
|
||||
Qwen/Qwen2.5-Math-1.5B-Instruct
|
||||
Qwen/Qwen2.5-Math-7B
|
||||
Qwen/Qwen2.5-Math-7B-Instruct
|
||||
Qwen/Qwen2.5-Math-72B
|
||||
Qwen/Qwen2.5-Math-72B-Instruct
|
||||
Qwen/Qwen3-0.6B
|
||||
Qwen/Qwen3-0.6B-Base
|
||||
Qwen/Qwen3-1.7B
|
||||
Qwen/Qwen3-1.7B-Base
|
||||
Qwen/Qwen3-4B
|
||||
Qwen/Qwen3-4B-Base
|
||||
Qwen/Qwen3-8B
|
||||
Qwen/Qwen3-8B-Base
|
||||
Qwen/Qwen3-14B
|
||||
Qwen/Qwen3-14B-Base
|
||||
Qwen/Qwen3-32B
|
||||
Qwen/Qwen3-30B-A3B
|
||||
Qwen/Qwen3-30B-A3B-Base
|
||||
Qwen/Qwen3-235B-A22B
|
||||
Qwen/Qwen3-4B-Thinking-2507
|
||||
Qwen/Qwen3-4B-Instruct-2507
|
||||
Qwen/Qwen3-30B-A3B-Thinking-2507
|
||||
Qwen/Qwen3-30B-A3B-Instruct-2507
|
||||
Qwen/Qwen3-235B-A22B-Thinking-2507
|
||||
Qwen/Qwen3-235B-A22B-Instruct-2507
|
||||
Qwen/QwQ-32B
|
||||
Qwen/QwQ-32B-Preview
|
||||
stabilityai/FreeWilly2
|
||||
stabilityai/stable-code-3b
|
||||
stabilityai/stablecode-completion-alpha-3b
|
||||
stabilityai/stablecode-completion-alpha-3b-4k
|
||||
stabilityai/stablecode-instruct-alpha-3b
|
||||
stabilityai/stablelm-3b-4e1t
|
||||
stabilityai/stablelm-base-alpha-3b
|
||||
stabilityai/stablelm-base-alpha-7b
|
||||
stabilityai/stablelm-tuned-alpha-3b
|
||||
stabilityai/stablelm-tuned-alpha-7b
|
||||
stabilityai/stablelm-zephyr-3b
|
||||
tiiuae/falcon-180B
|
||||
tiiuae/falcon-180B-chat
|
||||
tiiuae/falcon-40b
|
||||
tiiuae/falcon-40b-instruct
|
||||
tiiuae/falcon-7b
|
||||
tiiuae/falcon-7b-instruct
|
||||
tiiuae/Falcon3-1B-Base
|
||||
tiiuae/Falcon3-1B-Instruct
|
||||
tiiuae/Falcon3-3B-Base
|
||||
tiiuae/Falcon3-3B-Instruct
|
||||
tiiuae/Falcon3-7B-Base
|
||||
tiiuae/Falcon3-7B-Instruct
|
||||
tiiuae/Falcon3-10B-Base
|
||||
tiiuae/Falcon3-10B-Instruct
|
||||
TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
||||
togethercomputer/LLaMA-2-7B-32K
|
||||
togethercomputer/RedPajama-INCITE-7B-Base
|
||||
togethercomputer/RedPajama-INCITE-7B-Chat
|
||||
togethercomputer/RedPajama-INCITE-7B-Instruct
|
||||
togethercomputer/RedPajama-INCITE-Base-3B-v1
|
||||
togethercomputer/RedPajama-INCITE-Base-7B-v0.1
|
||||
togethercomputer/RedPajama-INCITE-Chat-3B-v1
|
||||
togethercomputer/RedPajama-INCITE-Chat-7B-v0.1
|
||||
togethercomputer/RedPajama-INCITE-Instruct-3B-v1
|
||||
togethercomputer/RedPajama-INCITE-Instruct-7B-v0.1
|
||||
Trelis/Llama-2-7b-chat-hf-function-calling-v2
|
||||
unsloth/Mistral-7B-v0.2
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> To sort the list above by model name after the `/`, use `litgpt download list | sort -f -t'/' -k2`.
|
||||
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> If you want to adopt a model variant that is not listed in the table above but has a similar architecture as one of the supported models, you can use this model by by using the `--model_name` argument as shown below:
|
||||
>
|
||||
> ```bash
|
||||
> litgpt download NousResearch/Hermes-2-Pro-Mistral-7B \
|
||||
> --model_name Mistral-7B-v0.1
|
||||
> ```
|
||||
|
||||
|
||||
|
||||
### 2. Download Model Weights
|
||||
|
||||
To download the weights for a specific model provide a `<repo_id>` with the model's repository ID. For example:
|
||||
|
||||
```bash
|
||||
litgpt download <repo_id>
|
||||
```
|
||||
|
||||
This command downloads the model checkpoint into the `checkpoints/` directory.
|
||||
|
||||
|
||||
|
||||
### 3. Additional Help
|
||||
|
||||
For more options, add the `--help` flag when running the script:
|
||||
|
||||
```bash
|
||||
litgpt download --help
|
||||
```
|
||||
|
||||
|
||||
|
||||
### 4. Run the Model
|
||||
|
||||
After conversion, run the model with the given checkpoint path as input, adjusting `repo_id` accordingly:
|
||||
|
||||
```bash
|
||||
litgpt chat <repo_id>
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Tinyllama Example
|
||||
|
||||
This section shows a typical end-to-end example for downloading and using TinyLlama:
|
||||
|
||||
1. List available TinyLlama checkpoints:
|
||||
|
||||
```bash
|
||||
litgpt download list | grep Tiny
|
||||
```
|
||||
|
||||
```
|
||||
TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
||||
TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
```
|
||||
|
||||
2. Download a TinyLlama checkpoint:
|
||||
|
||||
```bash
|
||||
export repo_id=TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
||||
litgpt download $repo_id
|
||||
```
|
||||
|
||||
3. Use the TinyLlama model:
|
||||
|
||||
```bash
|
||||
litgpt chat $repo_id
|
||||
```
|
||||
|
||||
|
||||
## Specific models and access tokens
|
||||
|
||||
Note that certain models require that you've been granted access to the weights on the Hugging Face Hub.
|
||||
|
||||
For example, to get access to the Gemma 2B model, you can do so by following the steps at <https://huggingface.co/google/gemma-2b>. After access is granted, you can find your HF hub token in <https://huggingface.co/settings/tokens>.
|
||||
|
||||
Once you've been granted access and obtained the access token you need to pass the additional `--access_token`:
|
||||
|
||||
```bash
|
||||
litgpt download google/gemma-2b \
|
||||
--access_token your_hf_token
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Finetunes and Other Model Variants
|
||||
|
||||
Sometimes you want to download the weights of a finetune of one of the models listed above. To do this, you need to manually specify the `model_name` associated to the config to use. For example:
|
||||
|
||||
```bash
|
||||
litgpt download NousResearch/Hermes-2-Pro-Mistral-7B \
|
||||
--model_name Mistral-7B-v0.1
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Tips for GPU Memory Limitations
|
||||
|
||||
The `litgpt download` command will automatically convert the downloaded model checkpoint into a LitGPT-compatible format. In case this conversion fails due to GPU memory constraints, you can try to reduce the memory requirements by passing the `--dtype bf16-true` flag to convert all parameters into this smaller precision (however, note that most model weights are already in a bfloat16 format, so it may not have any effect):
|
||||
|
||||
```bash
|
||||
litgpt download <repo_id>
|
||||
--dtype bf16-true
|
||||
```
|
||||
|
||||
(If your GPU does not support the bfloat16 format, you can also try a regular 16-bit float format via `--dtype 16-true`.)
|
||||
|
||||
|
||||
|
||||
## Converting Checkpoints Manually
|
||||
|
||||
For development purposes, for example, when adding or experimenting with new model configurations, it may be beneficial to split the weight download and model conversion into two separate steps.
|
||||
|
||||
You can do this by passing the `--convert_checkpoint false` option to the download script:
|
||||
|
||||
```bash
|
||||
litgpt download <repo_id> \
|
||||
--convert_checkpoint false
|
||||
```
|
||||
|
||||
and then calling the `convert_hf_checkpoint` command:
|
||||
|
||||
```bash
|
||||
litgpt convert_to_litgpt <repo_id>
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Downloading Tokenizers Only
|
||||
|
||||
In some cases we don't need the model weight, for example, when we are pretraining a model from scratch instead of finetuning it. For cases like this, you can use the `--tokenizer_only` flag to only download a model's tokenizer, which can then be used in the pretraining scripts:
|
||||
|
||||
```bash
|
||||
litgpt download TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T \
|
||||
--tokenizer_only true
|
||||
```
|
||||
|
||||
and
|
||||
|
||||
```bash
|
||||
litgpt pretrain tiny-llama-1.1b \
|
||||
--data ... \
|
||||
--tokenizer_dir TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T/
|
||||
```
|
||||
@@ -0,0 +1,220 @@
|
||||
# LLM Evaluation
|
||||
|
||||
|
||||
|
||||
## Using lm-evaluation-harness
|
||||
|
||||
You can evaluate LitGPT using [EleutherAI's lm-eval](https://github.com/EleutherAI/lm-evaluation-harness) framework with a large number of different evaluation tasks.
|
||||
|
||||
You need to install the `lm-eval` framework first:
|
||||
|
||||
```bash
|
||||
pip install lm_eval
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Evaluating LitGPT base models
|
||||
|
||||
Suppose you downloaded a base model that we want to evaluate. Here, we use the `microsoft/phi-2` model:
|
||||
|
||||
```bash
|
||||
litgpt download microsoft/phi-2
|
||||
```
|
||||
|
||||
The download command above will save the model to the `checkpoints/microsoft/phi-2` directory, which we can
|
||||
specify in the following evaluation command:
|
||||
|
||||
|
||||
```
|
||||
litgpt evaluate microsoft/phi-2/ \
|
||||
--batch_size 4 \
|
||||
--tasks "hellaswag,truthfulqa_mc2,mmlu" \
|
||||
--out_dir evaluate_model/
|
||||
```
|
||||
|
||||
The resulting output is as follows:
|
||||
|
||||
```
|
||||
...
|
||||
|---------------------------------------|-------|------|-----:|--------|-----:|---|-----:|
|
||||
...
|
||||
|truthfulqa_mc2 | 2|none | 0|acc |0.4656|± |0.0164|
|
||||
|hellaswag | 1|none | 0|acc |0.2569|± |0.0044|
|
||||
| | |none | 0|acc_norm|0.2632|± |0.0044|
|
||||
|
||||
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
|
||||
|------------------|-------|------|-----:|------|-----:|---|-----:|
|
||||
|mmlu |N/A |none | 0|acc |0.2434|± |0.0036|
|
||||
| - humanities |N/A |none | 0|acc |0.2578|± |0.0064|
|
||||
| - other |N/A |none | 0|acc |0.2401|± |0.0077|
|
||||
| - social_sciences|N/A |none | 0|acc |0.2301|± |0.0076|
|
||||
| - stem |N/A |none | 0|acc |0.2382|± |0.0076|
|
||||
```
|
||||
|
||||
|
||||
Please note that the `litgpt evaluate` command run an internal model conversion.
|
||||
This is only necessary the first time you want to evaluate a model, and it will skip the
|
||||
conversion steps if you run the `litgpt evaluate` on the same checkpoint directory again.
|
||||
|
||||
In some cases, for example, if you modified the model in the `checkpoint_dir` since the first `litgpt evaluate`
|
||||
call, you need to use the `--force_conversion` flag to to update the files used by litgpt evaluate accordingly:
|
||||
|
||||
```
|
||||
litgpt evaluate microsoft/phi-2/ \
|
||||
--batch_size 4 \
|
||||
--out_dir evaluate_model/ \
|
||||
--tasks "hellaswag,truthfulqa_mc2,mmlu" \
|
||||
--force_conversion true
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Run `litgpt evaluate list` to print a list
|
||||
> of the supported tasks. To filter for a specific subset of tasks, e.g., MMLU, use `litgpt evaluate list | grep mmlu`.
|
||||
|
||||
> [!TIP]
|
||||
> The evaluation may take a long time, and for testing purpoes, you may want to reduce the number of tasks
|
||||
> or set a limit for the number of examples per task, for example, `--limit 10`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### Evaluating LoRA-finetuned LLMs
|
||||
|
||||
No further conversion is necessary when evaluating LoRA-finetuned models as the `finetune_lora` command already prepares the necessary merged model files:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora microsoft/phi-2 \
|
||||
--out_dir lora_model
|
||||
```
|
||||
|
||||
|
||||
|
||||
```bash
|
||||
litgpt evaluate lora_model/final \
|
||||
--batch_size 4 \
|
||||
--tasks "hellaswag,truthfulqa_mc2,mmlu" \
|
||||
--out_dir evaluate_model/ \
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
### Evaluating on a custom test set
|
||||
|
||||
There is currently no built-in function to evaluate models on custom test sets. However, this section describes a general approach that users can take to evaluate the responses of a model using another LLM.
|
||||
|
||||
Suppose you have a test dataset with the following structure:
|
||||
|
||||
```python
|
||||
test_data = [
|
||||
{
|
||||
"instruction": "Name the author of 'Pride and Prejudice'.",
|
||||
"input": "",
|
||||
"output": "Jane Austen."
|
||||
},
|
||||
{
|
||||
"instruction": "Pick out the adjective from the following list.",
|
||||
"input": "run, tall, quickly",
|
||||
"output": "The correct adjective from the list is 'tall.'"
|
||||
},
|
||||
]
|
||||
```
|
||||
|
||||
For simplicity, the dictionary above only contains two entries. In practice, it is recommended to use test datasets that contain at least 100 entries (ideally 1000 or more).
|
||||
|
||||
If your dataset is stored in JSON format, use the following code to load it:
|
||||
|
||||
```python
|
||||
with open("test_data.json", "r") as file:
|
||||
test_data = json.load(file)
|
||||
```
|
||||
|
||||
Next, it is recommended to format the dataset according to a prompt style. For example, to use the `Alpaca` prompt style, use the following code:
|
||||
|
||||
```python
|
||||
from litgpt.prompts import Alpaca
|
||||
|
||||
prompt_style = Alpaca()
|
||||
prompt_style.apply(prompt=test_data[0]["instruction"], **test_data[0])
|
||||
```
|
||||
|
||||
which returns
|
||||
|
||||
```
|
||||
"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nName the author of 'Pride and Prejudice'.\n\n### Response:\n
|
||||
```
|
||||
|
||||
Next, load the LLM you want to evaluate. For this example, we use `phi-2`:
|
||||
|
||||
```python
|
||||
from litgpt import LLM
|
||||
|
||||
llm = LLM.load("microsoft/phi-2")
|
||||
```
|
||||
|
||||
Then, using the loaded model, we add the test set responses to the dataset:
|
||||
|
||||
|
||||
```python
|
||||
from tqdm import trange
|
||||
|
||||
|
||||
for i in trange(len(test_data)):
|
||||
response = llm.generate(prompt_style.apply(prompt=test_data[i]["instruction"], **test_data[i]))
|
||||
test_data[i]["response"] = response
|
||||
```
|
||||
|
||||
Next, we use a second LLM to calculate the response quality on a scale from 0 to 100. It is recommended to use the 70B Llama 3 instruction-fintuned model for this task, or the smaller 8B Llama 3 model, which is more resource-efficient:
|
||||
|
||||
|
||||
```python
|
||||
del llm # delete previous `llm` to free up GPU memory
|
||||
scorer = LLM.load("meta-llama/Meta-Llama-3-8B-Instruct", access_token="...")
|
||||
```
|
||||
|
||||
Then, based on this LLM, we calculate the response quality with the following function:
|
||||
|
||||
```python
|
||||
from tqdm import tqdm
|
||||
|
||||
|
||||
def generate_model_scores(data_dict, model, response_field="response", target_field="output"):
|
||||
scores = []
|
||||
for entry in tqdm(data_dict, desc="Scoring entries"):
|
||||
prompt = (
|
||||
f"Given the input `{format_input(entry)}` "
|
||||
f"and correct output `{entry[target_field]}`, "
|
||||
f"score the model response `{entry[response_field]}`"
|
||||
f" on a scale from 0 to 100, where 100 is the best score. "
|
||||
f"Respond with the integer number only."
|
||||
)
|
||||
score = model.generate(prompt, max_new_tokens=50)
|
||||
try:
|
||||
scores.append(int(score))
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
return scores
|
||||
```
|
||||
|
||||
|
||||
```python
|
||||
scores = generate_model_scores(test_data, model=scorer)
|
||||
print(f"\n{llm}")
|
||||
print(f"Number of scores: {len(scores)} of {len(test_data)}")
|
||||
print(f"Average score: {sum(scores)/len(scores):.2f}\n")
|
||||
```
|
||||
|
||||
This will print out the average score on all test set entries:
|
||||
|
||||
```
|
||||
Scoring entries: 100%|██████████| 2/2 [00:00<00:00, 4.37it/s]
|
||||
|
||||
Number of scores: 2 of 2
|
||||
Average score: 47.50
|
||||
```
|
||||
@@ -0,0 +1,27 @@
|
||||
## Minimal PyTorch Lightning Trainer Example
|
||||
|
||||
|
||||
|
||||
The script in this folder provides minimal examples showing how to train a LitGPT model using LitGPT's `GPT` class with the [PyTorch Lightning](https://github.com/Lightning-AI/pytorch-lightning) Trainer.
|
||||
|
||||
You can run the scripts as follows:
|
||||
|
||||
 
|
||||
## Small 160M model:
|
||||
|
||||
```bash
|
||||
# Download the Pythia model
|
||||
litgpt download EleutherAI/pythia-160m
|
||||
|
||||
python litgpt_ptl_small.py
|
||||
```
|
||||
|
||||
 
|
||||
## Medium-sized 8B model:
|
||||
|
||||
```bash
|
||||
# Download the Llama 3.1 model
|
||||
litgpt download meta-llama/Meta-Llama-3.1-8B --access_token hf_...
|
||||
|
||||
python litgpt_ptl_medium.py
|
||||
```
|
||||
@@ -0,0 +1,58 @@
|
||||
import lightning as L
|
||||
import torch
|
||||
|
||||
import litgpt
|
||||
from litgpt.data import Alpaca2k
|
||||
from litgpt.lora import GPT, merge_lora_weights
|
||||
|
||||
|
||||
class LitLLM(L.LightningModule):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.model = GPT.from_name(
|
||||
name="Llama-3.1-8B",
|
||||
lora_r=32,
|
||||
lora_alpha=16,
|
||||
lora_dropout=0.05,
|
||||
lora_key=False,
|
||||
lora_value=True,
|
||||
)
|
||||
litgpt.lora.mark_only_lora_as_trainable(self.model)
|
||||
|
||||
def on_train_start(self):
|
||||
state_dict = torch.load("checkpoints/meta-llama/Meta-Llama-3.1-8B/lit_model.pth", mmap=True)
|
||||
self.model.load_state_dict(state_dict, strict=False)
|
||||
|
||||
def training_step(self, batch):
|
||||
input_ids, targets = batch["input_ids"], batch["labels"]
|
||||
logits = self.model(input_ids)
|
||||
loss = litgpt.utils.chunked_cross_entropy(logits[..., :-1, :], targets[..., 1:])
|
||||
self.log("train_loss", loss, prog_bar=True)
|
||||
return loss
|
||||
|
||||
def configure_optimizers(self):
|
||||
warmup_steps = 10
|
||||
optimizer = torch.optim.AdamW(self.model.parameters(), lr=0.0002, weight_decay=0.0, betas=(0.9, 0.95))
|
||||
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda step: step / warmup_steps)
|
||||
return [optimizer], [scheduler]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
data = Alpaca2k()
|
||||
tokenizer = litgpt.Tokenizer("checkpoints/meta-llama/Meta-Llama-3.1-8B")
|
||||
data.connect(tokenizer, batch_size=1, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
max_epochs=2,
|
||||
accumulate_grad_batches=8,
|
||||
precision="bf16-true",
|
||||
)
|
||||
with trainer.init_module(empty_init=True):
|
||||
model = LitLLM()
|
||||
|
||||
trainer.fit(model, data)
|
||||
|
||||
# Save final checkpoint
|
||||
merge_lora_weights(model.model)
|
||||
trainer.save_checkpoint("checkpoints/finetuned.ckpt", weights_only=True)
|
||||
@@ -0,0 +1,150 @@
|
||||
# Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file.
|
||||
|
||||
import lightning as L
|
||||
import torch
|
||||
|
||||
from litgpt import LLM
|
||||
from litgpt.data import Alpaca2k
|
||||
|
||||
|
||||
class LitLLM(L.LightningModule):
|
||||
def __init__(self, checkpoint_dir, tokenizer_dir=None, trainer_ckpt_path=None):
|
||||
super().__init__()
|
||||
|
||||
self.llm = LLM.load(checkpoint_dir, tokenizer_dir=tokenizer_dir, distribute=None)
|
||||
self.trainer_ckpt_path = trainer_ckpt_path
|
||||
|
||||
def setup(self, stage):
|
||||
self.llm.trainer_setup(trainer_ckpt=self.trainer_ckpt_path)
|
||||
|
||||
def training_step(self, batch):
|
||||
logits, loss = self.llm(input_ids=batch["input_ids"], target_ids=batch["labels"])
|
||||
self.log("train_loss", loss, prog_bar=True)
|
||||
return loss
|
||||
|
||||
def validation_step(self, batch):
|
||||
logits, loss = self.llm(input_ids=batch["input_ids"], target_ids=batch["labels"])
|
||||
self.log("validation_loss", loss, prog_bar=True)
|
||||
return loss
|
||||
|
||||
def configure_optimizers(self):
|
||||
warmup_steps = 10
|
||||
optimizer = torch.optim.AdamW(self.llm.model.parameters(), lr=0.0002, weight_decay=0.0, betas=(0.9, 0.95))
|
||||
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda step: step / warmup_steps)
|
||||
return [optimizer], [scheduler]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
batch_size = 8
|
||||
accumulate_grad_batches = 1
|
||||
|
||||
#########################################################
|
||||
# Use case 1: Pretraining from random weights
|
||||
#########################################################
|
||||
|
||||
llm = LLM.load("EleutherAI/pythia-160m", tokenizer_dir="EleutherAI/pythia-160m", init="random")
|
||||
llm.save("pythia-160m-random-weights")
|
||||
del llm
|
||||
|
||||
lit_model = LitLLM(checkpoint_dir="pythia-160m-random-weights", tokenizer_dir="EleutherAI/pythia-160m")
|
||||
data = Alpaca2k()
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
|
||||
del lit_model
|
||||
|
||||
#############################################################################
|
||||
# Use case 2: Continued pretraining / finetuning from downloaded checkpoint
|
||||
#############################################################################
|
||||
|
||||
lit_model = LitLLM(checkpoint_dir="EleutherAI/pythia-160m")
|
||||
data = Alpaca2k()
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
|
||||
del lit_model
|
||||
|
||||
#########################################################
|
||||
# Use case 3: Resume training from Trainer checkpoint
|
||||
#########################################################
|
||||
|
||||
import os
|
||||
|
||||
def find_latest_checkpoint(directory):
|
||||
latest_checkpoint = None
|
||||
latest_time = 0
|
||||
|
||||
for root, _, files in os.walk(directory):
|
||||
for file in files:
|
||||
if file.endswith(".ckpt"):
|
||||
file_path = os.path.join(root, file)
|
||||
file_time = os.path.getmtime(file_path)
|
||||
if file_time > latest_time:
|
||||
latest_time = file_time
|
||||
latest_checkpoint = file_path
|
||||
|
||||
return latest_checkpoint
|
||||
|
||||
lit_model = LitLLM(
|
||||
checkpoint_dir="EleutherAI/pythia-160m", trainer_ckpt_path=find_latest_checkpoint("lightning_logs")
|
||||
)
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
|
||||
#################################################################
|
||||
# Use case 4: Resume training after saving a checkpoint manually
|
||||
#################################################################
|
||||
|
||||
lit_model.llm.save("finetuned_checkpoint")
|
||||
del lit_model
|
||||
lit_model = LitLLM(checkpoint_dir="finetuned_checkpoint")
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
@@ -0,0 +1,77 @@
|
||||
# Finetuning
|
||||
|
||||
We provide a simple finetuning commands (`litgpt finetune_*`) that instruction-finetune a pretrained model on datasets such as [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), [Dolly](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm), and others. For more information on the supported instruction datasets and how to prepare your own custom datasets, please see the [tutorials/prepare_dataset](prepare_dataset.md) tutorials.
|
||||
|
||||
LitGPT currently supports the following finetuning methods:
|
||||
|
||||
```bash
|
||||
litgpt finetune_full
|
||||
litgpt finetune_lora
|
||||
litgpt finetune_adapter
|
||||
litgpt finetune_adapter_v2
|
||||
```
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> To install all required dependencies before finetuning, first run `pip install "litgpt[all]"`.
|
||||
|
||||
|
||||
|
||||
The following section provides more details about these methods, including links for additional resources.
|
||||
|
||||
|
||||
|
||||
## LitGPT finetuning commands
|
||||
|
||||
The section below provides additional information on the available and links to further resources.
|
||||
|
||||
|
||||
### Full finetuning
|
||||
|
||||
```bash
|
||||
litgpt finetune_full
|
||||
```
|
||||
|
||||
This method trains all model weight parameters and is the most memory-intensive finetuning technique in LitGPT.
|
||||
|
||||
**More information and resources:**
|
||||
|
||||
- the LitGPT [tutorials/finetune_full](finetune_full.md) tutorial
|
||||
|
||||
|
||||
|
||||
### LoRA and QLoRA finetuning
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora stabilityai/stablelm-base-alpha-3b
|
||||
```
|
||||
|
||||
LoRA and QLoRA are parameter-efficient finetuning technique that only require updating a small number of parameters, which makes this a more memory-efficienty alternative to full finetuning.
|
||||
|
||||
**More information and resources:**
|
||||
|
||||
- the LitGPT [tutorials/finetune_lora](finetune_lora.md) tutorial
|
||||
- the LoRA paper by ([Hu et al. 2021](https://arxiv.org/abs/2106.09685))
|
||||
- the conceptual tutorial [Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA)](https://lightning.ai/pages/community/tutorial/lora-llm/)
|
||||
|
||||
|
||||
|
||||
### Adapter finetuning
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter stabilityai/stablelm-base-alpha-3b
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter_v2 stabilityai/stablelm-base-alpha-3b
|
||||
```
|
||||
|
||||
Similar to LoRA, adapter finetuning is a parameter-efficient finetuning technique that only requires training a small subset of weight parameters, making this finetuning method more memory-efficient than full-parameter finetuning.
|
||||
|
||||
**More information and resources:**
|
||||
|
||||
- the LitGPT [tutorials/finetune_adapter](finetune_adapter.md) tutorial
|
||||
- the Llama-Adapter ([Gao et al. 2023](https://arxiv.org/abs/2304.15010)) and Llama-Adapter v2 ([Zhang et al. 2023](https://arxiv.org/abs/2303.16199)) papers that originally introduces these methods
|
||||
- the conceptual tutorial [Understanding Parameter-Efficient Finetuning of Large Language Models: From Prefix Tuning to LLaMA-Adapters](https://lightning.ai/pages/community/article/understanding-llama-adapters/)
|
||||
@@ -0,0 +1,144 @@
|
||||
# Finetuning with Adapter
|
||||
|
||||
Adapter, first introduced for the LLaMA model as [LLaMA-Adapter](https://arxiv.org/abs/2303.16199), is a form of prefix-tuning that prepends a learnable adaption-prompt to the inputs of the attention blocks in an LLM. In total, there are only ~500k parameters to update during finetuning in StableLM 3B, which significantly reduces the memory footprint and speeds up training.
|
||||
|
||||
We are able to demonstrate instruction-finetuning LitGPT StableLM 3B on the [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) dataset on a **single RTX 3060 GPU**. If using 8 GPUs, finetuning can be completed in under 1 hour.
|
||||
|
||||
If you are new to Adapter and are interested to learn more about how it works before proceeding with the finetuning guide below, you might find our article [Understanding Parameter-Efficient Finetuning of Large Language Models: From Prefix Tuning to LLaMA-Adapters](https://lightning.ai/pages/community/article/understanding-llama-adapters/) helpful.
|
||||
|
||||
LLaMA-Adapter v2 extends the original LLaMA-Adapter idea by adding trainable bias and scale parameters to each linear layer in the transformer. Furthermore, LLaMA-Adapter v2 makes the normalization layers trainable. Where the StableLM 3B model has 500k trainable parameters with GPT v1, GPT-Adapter v2 adds an additional 1.5 M trainable parameter for the bias and scale parameters and ~300k trainable parameters for the normalization layers. So, adapter v2 has ~2.3 M trainable parameters in total.
|
||||
|
||||
## Preparation
|
||||
|
||||
The steps here only need to be done once:
|
||||
|
||||
1. Follow the instructions in the [README](../README.md) to install the dependencies.
|
||||
2. Download and convert the weights following our [guide](download_model_weights.md).
|
||||
|
||||
LitGPT provides common datasets for finetuning, such as Alpaca, LIMA, Dolly, and more.
|
||||
You can optionally [prepare your own dataset](#tune-on-your-dataset).
|
||||
For more information about dataset preparation, also see the [prepare_dataset.md](./prepare_dataset.md) tutorial.
|
||||
|
||||
## Running the finetuning
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter stabilityai/stablelm-base-alpha-3b \
|
||||
--data Alpaca \
|
||||
```
|
||||
|
||||
or for Adapter V2
|
||||
|
||||
```bash
|
||||
litgpt finetune adapter_v2 stabilityai/stablelm-base-alpha-3b \
|
||||
--data Alpaca \
|
||||
```
|
||||
|
||||
The finetuning requires at least one GPU with ~12 GB memory.
|
||||
You can speed up training by passing the `devices` argument to the script to utilize more GPUs if available.
|
||||
Depending on the available GPU memory, you can also tune the `micro_batch_size` parameter to utilize the GPU efficiently.
|
||||
To fit Adapter V2 to 12GB memory set `--train.micro_batch_size 2`.
|
||||
|
||||
For example, the following settings will let you finetune the model in under 1 hour:
|
||||
|
||||
```bash
|
||||
--devices 4 --train.micro_batch_size 4
|
||||
```
|
||||
|
||||
This script will save checkpoints periodically to the `out_dir` directory. If you are finetuning different models or on your own dataset, you can specify an output directory with your preferred name:
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter stabilityai/stablelm-base-alpha-3b \
|
||||
--data Alpaca \
|
||||
--out_dir out/adapter/my-model-finetuned
|
||||
```
|
||||
|
||||
or for Adapter V2
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter_v2 stabilityai/stablelm-base-alpha-3b \
|
||||
--data Alpaca \
|
||||
--out_dir out/adapter_v2/my-model-finetuned
|
||||
```
|
||||
|
||||
If your GPU does not support `bfloat16`, you can pass the `--precision 32-true` argument.
|
||||
For instance, to fine-tune on MPS (the GPU on modern Macs), you can run
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter stabilityai/stablelm-base-alpha-3b \
|
||||
--data Alpaca \
|
||||
--out_dir out/adapter/my-model-finetuned \
|
||||
--precision 32-true
|
||||
```
|
||||
|
||||
Note that `mps` as the accelerator will be picked up automatically by Fabric when running on a modern Mac.
|
||||
|
||||
### Quantization
|
||||
|
||||
Optionally, finetuning using quantization can be enabled via the `--quantize` flag, for example using the 4-bit NormalFloat data type:
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter stabilityai/stablelm-base-alpha-3b \
|
||||
--quantize "bnb.nf4"
|
||||
```
|
||||
|
||||
or using `adapter_v2` with double-quantization:
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter_v2 stabilityai/stablelm-base-alpha-3b \
|
||||
--quantize "bnb.nf4-dq"
|
||||
```
|
||||
|
||||
For additional benchmarks and resource requirements, please see the [Resource Tables](resource-tables.md).
|
||||
|
||||
## Test the model
|
||||
|
||||
You can test the finetuned model with your own instructions by running:
|
||||
|
||||
```bash
|
||||
litgpt generate_adapter stabilityai/stablelm-base-alpha-3b \
|
||||
--prompt "Recommend a movie to watch on the weekend."
|
||||
```
|
||||
|
||||
or for Adapter V2
|
||||
|
||||
```bash
|
||||
litgpt generate_adapter_v2 stabilityai/stablelm-base-alpha-3b \
|
||||
--prompt "Recommend a movie to watch on the weekend."
|
||||
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
A good movie to watch on the weekend would be The Lion King, since it's a classic family film that everyone can enjoy...
|
||||
```
|
||||
|
||||
If your GPU supports `bfloat16`, the script will automatically use it.
|
||||
|
||||
## Tune on your dataset
|
||||
|
||||
You can easily train on your own instruction dataset saved in JSON format.
|
||||
|
||||
1. Create a JSON file in which each row holds one instruction-response pair.
|
||||
A row has an entry for 'instruction' and 'output', and optionally 'input'. Note that currently, the 'input' field is only used in the Alpaca chat template. If you are using the Alpaca template, 'input' can be the empty string if the instruction doesn't require a context.
|
||||
Below is an example json file:
|
||||
|
||||
```text
|
||||
[
|
||||
{
|
||||
"instruction": "Arrange the given numbers in ascending order.",
|
||||
"input": "2, 4, 0, 8, 3", // Optional: only used in Alpaca chat template
|
||||
"output": "0, 2, 3, 4, 8"
|
||||
},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
2. Run `litgpt adapter` or `litgpt adapter_v2` by passing in the location of your data (and optionally other parameters):
|
||||
|
||||
```bash
|
||||
litgpt finetune_adapter tiiuae/falcon-7b \
|
||||
--data JSON \
|
||||
--data.json_path data/mydata.json \
|
||||
--out_dir data/mydata-finetuned
|
||||
```
|
||||
@@ -0,0 +1,92 @@
|
||||
# Finetuning the whole model
|
||||
|
||||
If you are interested in parameter-efficient finetuning, check out [finetune_adapter.md](finetune_adapter.md). In contrast to parameter-efficient finetuning, this "full" approach finetunes all model parameters, which is substantially more expensive. It may only be recommended as a baseline for comparison studies.
|
||||
|
||||
## Preparation
|
||||
|
||||
The steps here only need to be done once:
|
||||
|
||||
1. Follow the instructions in the [README](../README.md) to install the dependencies.
|
||||
2. Download and convert the weights following our [guide](download_model_weights.md).
|
||||
|
||||
LitGPT provides common datasets for finetuning, such as Alpaca, LIMA, Dolly, and more.
|
||||
You can optionally [prepare your own dataset](#tune-on-your-dataset).
|
||||
For more information about dataset preparation, also see the [prepare_dataset.md](./prepare_dataset.md) tutorial.
|
||||
|
||||
## Running the finetuning
|
||||
|
||||
```bash
|
||||
litgpt finetune_full tiiuae/falcon-7b \
|
||||
--data Alpaca \
|
||||
```
|
||||
|
||||
Finetuning the falcon-7b model requires at least 8 GPUs with ~40 GB memory each.
|
||||
|
||||
You can speed up training by passing the `devices` argument to the script to utilize more GPUs if available.
|
||||
Depending on the available GPU memory, you can also tune the `micro_batch_size` parameter to utilize the GPU efficiently.
|
||||
|
||||
This script will save checkpoints periodically to the `out_dir` directory. If you are finetuning different models or on your own dataset, you can specify an output directory with your preferred name:
|
||||
|
||||
```bash
|
||||
litgpt finetune_full tiiuae/falcon-7b \
|
||||
--data Alpaca \
|
||||
--out_dir out/full/my-model-finetuned
|
||||
```
|
||||
|
||||
If your GPU does not support `bfloat16`, you can pass the `--precision 32-true` argument.
|
||||
For instance, to fine-tune on MPS (the GPU on modern Macs), you can run
|
||||
|
||||
```bash
|
||||
litgpt finetune_full tiiuae/falcon-7b \
|
||||
--data Alpaca \
|
||||
--out_dir out/full/my-model-finetuned \
|
||||
--precision 32-true
|
||||
```
|
||||
|
||||
Note that `mps` as the accelerator will be picked up automatically by Fabric when running on a modern Mac.
|
||||
|
||||
## Test the model
|
||||
|
||||
You can test the finetuned model with your own instructions by running:
|
||||
|
||||
```bash
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--prompt "Recommend a movie to watch on the weekend." \
|
||||
--finetuned_path out/full/my-model-finetuned/lit_model_finetuned.pth
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
A good movie to watch on the weekend would be The Lion King, since it's a classic family film that everyone can enjoy...
|
||||
```
|
||||
|
||||
If your GPU supports `bfloat16`, the script will automatically use it.
|
||||
|
||||
## Tune on your dataset
|
||||
|
||||
You can easily train on your own instruction dataset saved in JSON format.
|
||||
|
||||
1. Create a JSON file in which each row holds one instruction-response pair.
|
||||
A row has an entry for 'instruction' and 'output', and optionally 'input'. Note that currently, the 'input' field is only used in the Alpaca chat template. If you are using the Alpaca template, 'input' can be the empty string if the instruction doesn't require a context.
|
||||
Below is an example json file:
|
||||
|
||||
```text
|
||||
[
|
||||
{
|
||||
"instruction": "Arrange the given numbers in ascending order.",
|
||||
"input": "2, 4, 0, 8, 3", // Optional: only used in Alpaca chat template
|
||||
"output": "0, 2, 3, 4, 8"
|
||||
},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
2. Run `litgpt finetune` by passing in the location of your data (and optionally other parameters):
|
||||
|
||||
```bash
|
||||
litgpt finetune tiiuae/falcon-7b \
|
||||
--data JSON \
|
||||
--data.json_path data/mydata.json \
|
||||
--out_dir data/mydata-finetuned
|
||||
```
|
||||
@@ -0,0 +1,154 @@
|
||||
# Finetuning with LoRA / QLoRA
|
||||
|
||||
[Low-rank adaption (LoRA)](https://arxiv.org/abs/2106.09685) is a technique to approximate the update to the linear layers in a LLM with a low-rank matrix factorization. This significantly reduces the number of trainable parameters and speeds up training with little impact on the final performance of the model.
|
||||
We demonstrate this method by instruction-finetuning LitGPT StableLM 3B on the [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) dataset on a **single RTX 3090 (24GB) GPU** with CUDA 11.8.
|
||||
|
||||
|
||||
|
||||
## Preparation
|
||||
|
||||
The steps here only need to be done once:
|
||||
|
||||
1. Follow the instructions in the [README](../README.md) to install the dependencies.
|
||||
2. Download and convert the weights and save them in the `./checkpoints` folder.
|
||||
Weights can be downloaded following the instructions in the [download_model_weights](download_model_weights.md) documentation:
|
||||
|
||||
LitGPT provides common datasets for finetuning, such as Alpaca, LIMA, Dolly, and more.
|
||||
You can optionally [prepare your own dataset](#tune-on-your-dataset).
|
||||
For more information about dataset preparation, also see the [prepare_dataset.md](./prepare_dataset.md) tutorial.
|
||||
|
||||
|
||||
|
||||
## Running the Finetuning
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora stabilityai/stablelm-base-alpha-3b \
|
||||
--data Alpaca
|
||||
```
|
||||
|
||||
The finetuning requires at least one GPU with ~24 GB memory (RTX 3090).
|
||||
|
||||
This script will save checkpoints periodically to the folder `out/`.
|
||||
|
||||
> [!NOTE]
|
||||
> LoRA can be applied to not only `query`, `key` or `value` matrices, but also to `projection`, `mlp` and classification `head`.
|
||||
> According to [QLoRA](https://arxiv.org/abs/2305.14314) paper (section 4): "LoRA on all linear transformer block layers are required to match full finetuning performance".
|
||||
> By default LoRA is applied only to the `query` and `value` matrices. In order to apply LoRA to other weight matrices - change the arguments to `litgpt/finetune/lora.py` accordingly.
|
||||
|
||||
Optionally, finetuning using 4-bit quantization (as in QLoRA) can be enabled via the `--quantize` flag, for example using the 4-bit NormalFloat data type:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora stabilityai/stablelm-base-alpha-3b \
|
||||
--quantize "bnb.nf4"
|
||||
```
|
||||
|
||||
and optionally with double-quantization:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora stabilityai/stablelm-base-alpha-3b \
|
||||
--quantize "bnb.nf4-dq"
|
||||
```
|
||||
|
||||
The table below lists a comparison with different settings on a StableLM 3B model finetuned with LoRA on Alpaca for 1,000 iterations using a microbatch size of 1:
|
||||
|
||||
| Settings | Training Memory | Training Time | Inference Memory |
|
||||
|---------------------------------------------|-----------------|---------------|-------------------|
|
||||
| Default (bf16-mixed) | 26.92 GB | 1.34 min | 21.43 GB |
|
||||
| --precision bf16-true | 9.69 GB | 1.24 min | 7.30 GB |
|
||||
| --precision bf16-true --quantize bnb.nf4 | 6.35 GB | 1.82 min | 3.20 GB |
|
||||
| --precision bf16-true --quantize bnb.nf4-dq | 6.19 GB | 1.87 min | 3.04 GB |
|
||||
|
||||
The advantages of QLoRA-style quantization are more pronounced in larger models, such as Llama 2 7B. The table below summarizes the results for Llama 2 7B on Alpaca for 1,000 iterations using a microbatch size of 1:
|
||||
|
||||
| Settings | Training Memory | Training Time | Inference Memory |
|
||||
|---------------------------------------------|------------------|---------------|------------------|
|
||||
| Default (bf16-mixed) | OutOfMemoryError | N/A | 40.21 GB |
|
||||
| --precision bf16-true | 21.30 GB | 2.36 min | 13.52 GB |
|
||||
| --precision bf16-true --quantize bnb.nf4 | 14.14 GB | 3.68 min | 4.57 GB |
|
||||
| --precision bf16-true --quantize bnb.nf4-dq | 13.84 GB | 3.83 min | 4.26 GB |
|
||||
|
||||
For additional benchmarks and resource requirements, please see the [Resource Tables](resource-tables.md).
|
||||
|
||||
|
||||
|
||||
## Test the Model
|
||||
|
||||
You can test the finetuned model with your own instructions by running:
|
||||
|
||||
```bash
|
||||
litgpt generate "out/lora/final" \
|
||||
--prompt "Recommend a movie to watch on the weekend."
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
I would recommend the movie The Martian (2015). It is a sci-fi movie starring Matt Damon that follows the story of...
|
||||
```
|
||||
|
||||
If your GPU supports `bfloat16`, you can additionally pass `--precision "bf16-true"` to bring the memory consumption down to ~7.6 GB for StableLM-3B (versus ~15.2 GB for `--precision "32-full"`). In addition, you may use quantization methods, for example `--precision "bf16-true" --quantize "bnb.nf4"` brings the memory consumption further down to ~4.4 GB for StableLM-3B.
|
||||
|
||||
|
||||
|
||||
## Tune on Your Dataset
|
||||
|
||||
You can easily train on your own instruction dataset saved in JSON format.
|
||||
|
||||
1. Create a JSON file in which each row holds one instruction-response pair.
|
||||
A row has an entry for 'instruction' and 'output', and optionally 'input'. Note that currently, the 'input' field is only used in the Alpaca chat template. If you are using the Alpaca template, 'input' can be the empty string if the instruction doesn't require a context.
|
||||
Below is an example json file:
|
||||
|
||||
```text
|
||||
[
|
||||
{
|
||||
"instruction": "Arrange the given numbers in ascending order.",
|
||||
"input": "2, 4, 0, 8, 3", // Optional: only used in Alpaca chat template
|
||||
"output": "0, 2, 3, 4, 8"
|
||||
},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
2. Run `litgpt finetune_lora` by passing in the location of your data (and optionally other parameters):
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora checkpoints/stabilityai/stablelm-base-alpha-3b \
|
||||
--data JSON \
|
||||
--data.json_path data/mydata.json \
|
||||
--out_dir out_dir/mydata-finetuned
|
||||
```
|
||||
|
||||
3. Test and use the finetuned model:
|
||||
|
||||
```bash
|
||||
litgpt chat out_dir/mydata-finetuned/final
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```bash
|
||||
litgpt serve out_dir/mydata-finetuned/final
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Merging LoRA Weights (Optional)
|
||||
|
||||
Finetuning a model with LoRA generates a `lit_model.pth.lora` file.
|
||||
This file exclusively contains the LoRA weights, which are much smaller than the original model checkpoint to conserve storage space.
|
||||
|
||||
> [!NOTE]
|
||||
> LitGPT will automatically merge the checkpoint for you if you use it in any of the inference commands, such as `litgpt generate` or `litgpt chat`.
|
||||
> Manual merging is only necessary if you want to use the checkpoint outside LitGPT.
|
||||
|
||||
If desired, there is the option to merge these LoRA weights manually into the original model's checkpoint, which creates a full `lit_model.pth` checkpoint.
|
||||
The advantage of this merging process is to streamline inference operations, as it eliminates the need to dynamically incorporate the LoRA weights during runtime, which can improve inference speed.
|
||||
|
||||
For example, after finetuning produced a checkpoint folder `out/lora/step-002000`, merge it as follows:
|
||||
|
||||
```bash
|
||||
litgpt merge_lora "out/lora/step-002000"
|
||||
```
|
||||
The command above creates a full `lit_model.pth` checkpoint file.
|
||||
@@ -0,0 +1,122 @@
|
||||
"""
|
||||
This script is meant to be the simplest possible starting point for full finetuning a GPT model using lightning fabric with code (not CLI).
|
||||
|
||||
- no checkpoints
|
||||
- no out dir
|
||||
- no precision
|
||||
- no resume
|
||||
- no train/eval args (or any args in general)
|
||||
- no logger (only to terminal)
|
||||
- no grad accumulation
|
||||
and no other fancy stuff.
|
||||
|
||||
To add all the above stuff, you can slowly add them in yourself by looking at the code in litgpt/finetune/full.py or the docs for litgpt/fabric.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
import lightning as L
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
from litgpt.data import Alpaca
|
||||
from litgpt.model import GPT, Config
|
||||
from litgpt.tokenizer import Tokenizer
|
||||
from litgpt.utils import num_parameters
|
||||
|
||||
# training params/args
|
||||
SEED = 1337
|
||||
MODEL_NAME = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T" # try also "stabilityai/stablelm-base-alpha-3b"!
|
||||
BATCH_SIZE = 4
|
||||
LR_WARMUP_STEPS = 100
|
||||
MAX_STEPS = 601
|
||||
|
||||
|
||||
def validate(model, val_dataloader):
|
||||
model.eval()
|
||||
loss = 0
|
||||
with torch.no_grad():
|
||||
for batch in val_dataloader:
|
||||
input_ids, targets = batch["input_ids"], batch["labels"]
|
||||
logits = model(input_ids)
|
||||
logits = logits.reshape(-1, logits.size(-1))
|
||||
targets = targets.reshape(-1)
|
||||
loss += nn.functional.cross_entropy(logits[..., :-1, :], targets[..., 1:])
|
||||
fabric.print(f"Validation loss: {loss / len(val_dataloader)}")
|
||||
|
||||
|
||||
def train(fabric, model, optimizer, scheduler, train_dataloader, val_dataloader):
|
||||
for iter_num, batch in enumerate(train_dataloader):
|
||||
input_ids, targets = batch["input_ids"], batch["labels"]
|
||||
|
||||
# get model preds (logits)
|
||||
logits = model(input_ids)
|
||||
logits = logits.reshape(-1, logits.size(-1))
|
||||
|
||||
# get loss
|
||||
targets = targets.reshape(-1)
|
||||
loss = nn.functional.cross_entropy(logits[..., :-1, :], targets[..., 1:])
|
||||
|
||||
# update weights
|
||||
fabric.backward(loss)
|
||||
optimizer.step()
|
||||
optimizer.zero_grad()
|
||||
scheduler.step()
|
||||
|
||||
# print train loss every 100 steps
|
||||
if iter_num % 100 == 0 or iter_num == 0:
|
||||
fabric.print(f"Train iter {iter_num} - loss {loss}")
|
||||
|
||||
# validate every 300 steps
|
||||
if iter_num % 300 == 0 or iter_num == 0:
|
||||
validate(model, val_dataloader)
|
||||
model.train()
|
||||
iter_num += 1
|
||||
|
||||
if iter_num >= MAX_STEPS:
|
||||
break
|
||||
|
||||
|
||||
def main(fabric):
|
||||
fabric.seed_everything(SEED)
|
||||
|
||||
# setup data, make tokenizer and make dataloaders
|
||||
data = Alpaca()
|
||||
tokenizer = Tokenizer(checkpoint_dir=f"checkpoints/{MODEL_NAME}")
|
||||
data.connect(tokenizer=tokenizer, batch_size=BATCH_SIZE, max_seq_length=1024)
|
||||
data.setup()
|
||||
train_dataloader = data.train_dataloader()
|
||||
val_dataloader = data.val_dataloader()
|
||||
train_dataloader, val_dataloader = fabric.setup_dataloaders(train_dataloader, val_dataloader)
|
||||
|
||||
# print how many steps in an epoch
|
||||
fabric.print(f"Steps in an epoch: {len(train_dataloader)}")
|
||||
|
||||
# setup model
|
||||
config = Config.from_file(f"checkpoints/{MODEL_NAME}/model_config.yaml")
|
||||
model = GPT(config)
|
||||
fabric.print(f"Number of trainable parameters: {num_parameters(model, requires_grad=True):,}")
|
||||
model = fabric.setup(model)
|
||||
|
||||
# setup optimizer
|
||||
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.02, betas=(0.9, 0.95))
|
||||
optimizer = fabric.setup_optimizers(optimizer)
|
||||
|
||||
# setup lr scheduler
|
||||
scheduler1 = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda step: step / LR_WARMUP_STEPS)
|
||||
scheduler2 = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=(MAX_STEPS - LR_WARMUP_STEPS))
|
||||
scheduler = torch.optim.lr_scheduler.SequentialLR(optimizer, [scheduler1, scheduler2], milestones=[LR_WARMUP_STEPS])
|
||||
|
||||
# Start training!!!
|
||||
train(fabric, model, optimizer, scheduler, train_dataloader, val_dataloader)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# check that the model exists (downloaded to ./checkpoints/)
|
||||
if not os.path.exists(f"checkpoints/{MODEL_NAME}"):
|
||||
print(f"Model {MODEL_NAME} not found. Please download it using `litgpt download --repo {MODEL_NAME}`")
|
||||
exit()
|
||||
|
||||
### Setup and launch
|
||||
fabric = L.Fabric(devices="auto", strategy="auto")
|
||||
fabric.launch(main)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 8.1 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 5.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 90 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
@@ -0,0 +1,125 @@
|
||||
# Inference
|
||||
|
||||
We demonstrate how to run inference (next token prediction) with the GPT base model in the [`litgpt generate`](../litgpt/generate/base.py) command:
|
||||
|
||||
```bash
|
||||
litgpt generate stabilityai/stablelm-base-alpha-3b \
|
||||
--prompt "Hello, my name is"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
Hello, my name is Levi Durrer, I'm an Austrian journalist - Chairman of the Press Blair Party, with 37 years in the Press Blair International, and two years in the Spectre of Austerity for the other. I'm crossing my fingers that you will feel
|
||||
```
|
||||
|
||||
The script assumes you have downloaded and converted the weights as described [here](download_model_weights.md).
|
||||
|
||||
This will run the 3B pre-trained model and require ~7 GB of GPU memory using the `bfloat16` datatype.
|
||||
|
||||
## Run interactively
|
||||
|
||||
You can also chat with the model interactively:
|
||||
|
||||
```bash
|
||||
litgpt chat stabilityai/stablelm-tuned-alpha-3b
|
||||
```
|
||||
|
||||
This script can work with any checkpoint. For the best chat-like experience, we recommend using it with a checkpoints
|
||||
fine-tuned for chatting such as `stabilityai/stablelm-tuned-alpha-3b` or `togethercomputer/RedPajama-INCITE-Chat-3B-v1`.
|
||||
|
||||
> [!TIP]
|
||||
> Use `--multiline true` to work with inputs that span multiple lines.
|
||||
|
||||
|
||||
## Run a large model on one smaller device
|
||||
|
||||
Check out our [quantization tutorial](quantize.md).
|
||||
|
||||
## Run a large model on multiple smaller devices
|
||||
|
||||
We offer two scripts to leverage multiple devices for inference.
|
||||
|
||||
### [`litgpt generate_sequentially`](../litgpt/generate/sequentially.py)
|
||||
|
||||
Allows you to run models that wouldn't fit in a single card by partitioning the transformer blocks across all your devices and running them sequentially.
|
||||
|
||||
For instance, `meta-llama/Llama-2-70b-chat-hf` would require ~140 GB of GPU memory to load on a single device, plus the memory for activations.
|
||||
With 80 transformer layers, we could partition them across 8, 5, 4, or 2 devices.
|
||||
|
||||
```shell
|
||||
litgpt generate_sequentially meta-llama/Llama-2-70b-chat-hf \
|
||||
--max_new_tokens 256 \
|
||||
--num_samples 2
|
||||
```
|
||||
|
||||
Using A100 40GB GPUs, we need to use at least 4. You can control the number of devices by setting the `CUDA_VISIBLE_DEVICES=` environment variable.
|
||||
|
||||
| Devices | Max GPU RAM | Token/sec |
|
||||
|---------|-------------|-----------|
|
||||
| 2 | OOM | - |
|
||||
| 4 | 35.64 GB | 7.55 |
|
||||
| 5 | 28.72 GB | 7.49 |
|
||||
| 8 | 18.35 GB | 7.47 |
|
||||
|
||||
Note that the memory usage will also depend on the `max_new_tokens` value used.
|
||||
|
||||
The script also supports quantization, using 4-bit precision, we can now use 2 GPUs
|
||||
|
||||
```shell
|
||||
litgpt generate_sequentially meta-llama/Llama-2-70b-chat-hf \
|
||||
--max_new_tokens 256 \
|
||||
--num_samples 2 \
|
||||
--quantize bnb.nf4-dq
|
||||
```
|
||||
|
||||
| Devices | Max GPU RAM | Token/sec |
|
||||
|---------|-------------|-----------|
|
||||
| 2 | 20.00 GB | 8.63 |
|
||||
| 4 | 10.80 GB | 8.23 |
|
||||
| 5 | 8.96 GB | 8.10 |
|
||||
| 8 | 6.23 GB | 8.18 |
|
||||
|
||||
Smaller devices can also be used to run inference with this technique.
|
||||
|
||||
### [`litgpt generate_tp`](../litgpt/generate/tp.py)
|
||||
|
||||
Uses tensor parallelism (TP) to run models that wouldn't fit in a single card by sharding the MLP and Attention QKV linear layers across all your devices.
|
||||
|
||||
For instance, `meta-llama/Llama-2-70b-chat-hf` would require ~140 GB of GPU memory to load on a single device, plus the memory for activations.
|
||||
The requirement is that the intermediate size (for the MLP) and the QKV size (for attention) is divisible by the number of devices.
|
||||
With an intermediate size of 28672, we can use 2, 4, 7, or 8 devices. With a QKV size of 10240 we can use 2, 4, 5, or 8 devices.
|
||||
Since the script is configured to shard both, the intersection is used: we can only use 2, 4, or 8 devices.
|
||||
|
||||
```shell
|
||||
litgpt generate_tp meta-llama/Llama-2-70b-chat-hf \
|
||||
--max_new_tokens 256 \
|
||||
--num_samples 2
|
||||
```
|
||||
|
||||
Using A100 40GB GPUs, we need to use at least 4. You can control the number of devices by setting the `CUDA_VISIBLE_DEVICES=` environment variable.
|
||||
|
||||
| Devices | Max GPU RAM | Token/sec |
|
||||
|---------|-------------|-----------|
|
||||
| 2 | OOM | - |
|
||||
| 4 | 35.46 GB | 9.33 |
|
||||
| 8 | 18.19 GB | 8.61 |
|
||||
|
||||
Note that the memory usage will also depend on the `max_new_tokens` value used.
|
||||
|
||||
The script also supports quantization, using 4-bit precision, we can now use 2 GPUs
|
||||
|
||||
```shell
|
||||
litgpt generate_tp meta-llama/Llama-2-70b-chat-hf \
|
||||
--max_new_tokens 256 \
|
||||
--num_samples 2 \
|
||||
--quantize bnb.nf4-dq
|
||||
```
|
||||
|
||||
| Devices | Max GPU RAM | Token/sec |
|
||||
|---------|-------------|-----------|
|
||||
| 2 | 19.79 GB | 6.72 |
|
||||
| 4 | 10.73 GB | 6.48 |
|
||||
| 8 | 6.15 GB | 6.20 |
|
||||
|
||||
Smaller devices can also be used to run inference with this technique.
|
||||
@@ -0,0 +1,7 @@
|
||||
site_name: LitGPT Tutorials
|
||||
|
||||
plugins:
|
||||
- pagetree
|
||||
|
||||
theme:
|
||||
name: material
|
||||
@@ -0,0 +1,67 @@
|
||||
## Dealing with out-of-memory (OOM) errors
|
||||
|
||||
If you got this error while running a script
|
||||
|
||||
```bash
|
||||
OutOfMemoryError: CUDA out of memory. Tried to allocate 2.22 GiB. GPU 0 has a total capacity of 79.15 GiB of which 228.38 MiB is free. Including non-PyTorch memory, this process
|
||||
has 78.93 GiB memory in use. Of the allocated memory 76.28 GiB is allocated by PyTorch, and 2.14 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory
|
||||
is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
|
||||
```
|
||||
|
||||
it means that your GPU memory size wasn't big enough for the model and script configuration.
|
||||
|
||||
Here's a few things you can try:
|
||||
|
||||
### Reduce the micro batch size
|
||||
|
||||
Adjust the `--train.micro_batch_size` argument in the fine-tuning and pretraining scripts. This variable determines the number of samples loaded per iteration.
|
||||
|
||||
A smaller value will simply load fewer samples simultaneously. The minimum value is 1.
|
||||
|
||||
Experiment with different micro batch sizes to find a balance between memory consumption and computational efficiency. Smaller micro batch sizes consume less memory but may result in slower training convergence. Conversely, larger micro batch sizes require more memory but can accelerate training speed.
|
||||
|
||||
### Reduce the model's context length
|
||||
|
||||
The context length (`block_size` in the code) plays a significant role in running models with attention.
|
||||
|
||||
* The pretraining scripts are configured to use the full context length of the model to train.
|
||||
* The finetuning scripts are configured to use the longest sample length of the training data to avoid allocating unnecessary memory (`--train.max_seq_length` argument).
|
||||
If that's longer than the model's context length, an error is raised. If you try to run a batch that is longer than this, an error is raised.
|
||||
|
||||
However, your hardware may not support such large context lengths. Here's what you can do:
|
||||
|
||||
* For the pretraining scripts, you can simply reduce the `Config(block_size=...)` value.
|
||||
* For the finetuning scripts, you can trim the length of the samples in your dataset.
|
||||
All the finetuning scripts expose a `--data.max_seq_length=...` argument. This might also be useful in cases where
|
||||
sample lengths are highly unbalanced, as the presence of a single very long sample would incur a larger memory usage for all other
|
||||
shorter samples. For example, the median length of the samples in Alpaca is 110 tokens. Truncating the Alpaca dataset to 256 max tokens reduces the memory requirements of a Falcon 7B model from 23.52 GB to 15.73 GB. For more information about the dataset truncation, please see the *Truncating datasets* section in the [prepare_dataset.md](prepare_dataset.md) tutorial.
|
||||
|
||||
Keep in mind that reducing the context length will affect the modelling performance on text sequences longer than the limit.
|
||||
|
||||
### Use lower precision
|
||||
|
||||
Our scripts expose the `--precision` argument, this directly impacts the memory usage.
|
||||
|
||||
Using true lower precision (`16-true`, `bf16-true`) reduces the memory usage by half compared to `32-true`, however,
|
||||
the model might start producing NaNs due to the limited range of representable values.
|
||||
|
||||
Mixed precision training (`16-mixed`, `bf16-mixed`) provides better stability but offers limited memory reduction.
|
||||
|
||||
### Do sharding across multiple GPUs
|
||||
|
||||
For exceptionally large models, the aforementioned techniques might still not suffice. If you have multiple GPUs available,
|
||||
you can trade off memory for speed by changing the `--devices 1` argument in the scripts. Enabling this option enables a parallelism technique (FSDP), sharding the memory across different GPUs.
|
||||
|
||||
The default configuration already uses activation checkpointing, but you can enable CPU offloading by changing the `cpu_offload=False` argument in the scripts.
|
||||
|
||||
### Try a different optimizer
|
||||
|
||||
Our scripts use the [`AdamW` optimizer](https://pytorch.org/docs/main/generated/torch.optim.AdamW.html).
|
||||
It maintains 2 states for each trainable parameter of the model, meaning that the optimizer memory is double compared to
|
||||
an optimizer like [`SGD`](https://pytorch.org/docs/main/generated/torch.optim.SGD.html).
|
||||
|
||||
You can try replacing it with your optimizer of choice that is lighter in memory requirements. Keep in mind that different optimizers have distinct optimization behaviors, so it's essential to assess their impact on the training process and model performance.
|
||||
An example would be the recently published [Sophia](https://arxiv.org/abs/2305.14342) or [Lion](https://arxiv.org/abs/2302.06675) optimizers.
|
||||
|
||||
This suggestion is particularly relevant for pretraining, as the trainable parameters in the model represent a small
|
||||
subset of the total in the fine-tuning scripts.
|
||||
@@ -0,0 +1,447 @@
|
||||
# Preparing Datasets
|
||||
|
||||
Below is a table of all datasets that are currently supported in LitGPT:
|
||||
|
||||
| Name | Task | Size | Reference Repo | Paper / Blog | Data License |
|
||||
|--------------|-------------|---------------------|--------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| Alpaca | Finetuning | 51,759 samples | [URL](https://github.com/tatsu-lab/stanford_alpaca) | [URL](https://crfm.stanford.edu/2023/03/13/alpaca.html) | Attribution-NonCommercial 4.0 International, [URL](https://crfm.stanford.edu/2023/03/13/alpaca.html) |
|
||||
| Alpaca-2k | Finetuning | 2000 samples | [URL](https://huggingface.co/datasets/mhenrichsen/alpaca_2k_test) | See Alpaca above | See Alpaca Above |
|
||||
| Alpaca-GPT4 | Finetuning | 52,002 samples | [URL](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM) | [URL](https://arxiv.org/abs/2304.03277) | Attribution-NonCommercial 4.0 International, [URL](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM/blob/main/DATA_LICENSE) |
|
||||
| Alpaca Libre | Finetuning | 55,370 samples | [URL](https://github.com/mobarski/alpaca-libre) | - | CC0/MIT, [URL](https://github.com/mobarski/alpaca-libre) |
|
||||
| Deita | Finetuning | 9,500 samples | [URL](https://huggingface.co/datasets/HuggingFaceH4/deita-10k-v0-sft/tree/main/data) | [URL](https://arxiv.org/abs/2312.15685) | MIT [URL](https://huggingface.co/datasets/hkust-nlp/deita-10k-v0/blob/main/README.md) |
|
||||
| Dolly | Finetuning | 15,011 samples | [URL](https://github.com/databrickslabs/dolly/tree/master/data) | [URL](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) | CC-BY-SA, [URL](https://github.com/databrickslabs/dolly#model-overview) |
|
||||
| FLAN | Finetuning | 1,753,240 samples | [UR](https://huggingface.co/datasets/Muennighoff/flan) | [URL](https://blog.research.google/2023/02/the-flan-collection-advancing-open.html) | Subset dependent |
|
||||
| LongForm | Finetuning | 23,652 samples | [URL](https://github.com/akoksal/LongForm) | [URL](https://arxiv.org/abs/2304.08460) | No information provided and subset-dependent, [URL](https://github.com/akoksal/LongForm) |
|
||||
| LIMA | Finetuning | 1,084 samples | [URL](https://huggingface.co/datasets/GAIR/lima) | [URL](https://arxiv.org/abs/2305.11206) | "If the source data of LIMA has a stricter license than CC BY-NC-SA, the LIMA dataset follows the same. Otherwise, it follows the CC BY-NC-SA license", [URL](https://huggingface.co/datasets/GAIR/lima#license) |
|
||||
| OpenWeb Text | Pretraining | 8,013,769 documents | [URL](https://github.com/jcpeterson/openwebtext) | [URL](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) | Unspecified |
|
||||
| TinyLlama | Pretraining | 1 T tokens | [URL](https://github.com/jzhang38/TinyLlama) | [URL](https://arxiv.org/abs/2401.02385) | |
|
||||
| TinyStories | Pretraining | 4,967,871 stories | [URL](https://huggingface.co/datasets/roneneldan/TinyStories) | [URL](https://arxiv.org/abs/2305.07759) | CDLA-Sharing-1.0 |
|
||||
|
||||
|
||||
|
||||
## Preparation
|
||||
|
||||
The steps here only need to be done once before preparing the finetuning datasets in the following subsections:
|
||||
|
||||
1. Follow the instructions in the [README](../README.md) to install the dependencies.
|
||||
2. Download and convert the weights following our [guide](download_model_weights.md).
|
||||
|
||||
For the following examples, we will focus on finetuning with the `litgpt finetune_lora` command and use a Falcon 7B model.
|
||||
However, the same steps apply to all other models and finetuning scripts.
|
||||
Please read the [tutorials/finetune_*.md](.) documents for more information about finetuning models.
|
||||
|
||||
|
||||
|
||||
> [!IMPORTANT]
|
||||
> By default, the maximum sequence length is obtained from the model configuration file. In case you run into out-of-memory errors, especially in the cases of LIMA and Dolly,
|
||||
> you can try to lower the context length by setting the `--train.max_seq_length` parameter, for example, `litgpt finetune lora --train.max_seq_length 256`. For more information on truncating datasets, see the *Truncating datasets* section in the Alpaca section near the top of this article.
|
||||
|
||||
|
||||
|
||||
### Alpaca
|
||||
|
||||
The Alpaca dataset consists of 52,000 instructions and demonstrations produced by OpenAI's text-davinci-003 engine. This data is used in instruction-tuning, helping improve the performance of language models to follow instructions.
|
||||
|
||||
In its development, the creators leveraged the data generation methodology from the [Self-Instruct framework](https://github.com/yizhongw/self-instruct).
|
||||
|
||||
The original [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) dataset can be used as follows:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Alpaca
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help Alpaca` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
#### Truncating datasets
|
||||
|
||||
By default, the finetuning scripts will determine the size of the longest tokenized sample in the dataset to determine the block size. However, if you are willing to truncate a few examples in the training set, you can reduce the computational resource requirements significantly. For instance you can set a sequence length threshold via `--train.max_seq_length`. We can determine an appropriate maximum sequence length by considering the distribution of the data sample lengths shown in the histogram below.
|
||||
|
||||
<img src="images/prepare_dataset/alpaca.jpg" width=400px>
|
||||
|
||||
In this case, a cut-off of 256 may be a reasonable choice:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Alpaca \
|
||||
--train.max_seq_length 256
|
||||
```
|
||||
|
||||
For comparison, the Falcon 7B model requires 23.52 GB of memory for the original Alpaca dataset and 15.73 GB of memory for the truncated Alpaca dataset when finetuning with LoRA using a micro batchsize of 1 and bfloat-16 precision.
|
||||
|
||||
|
||||
|
||||
### Alpaca-2k
|
||||
|
||||
[Alpaca-2k](https://huggingface.co/datasets/mhenrichsen/alpaca_2k_test) is a smaller, 2000-sample subset of Alpaca described above.
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora "tiiuae/falcon-7b" \
|
||||
--data Alpaca2k
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt_finetune --data.help Alpaca2k` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
The Alpaca-2k dataset distribution is shown below.
|
||||
|
||||
<img src="images/prepare_dataset/alpaca-2k.jpg" width=400px>
|
||||
|
||||
|
||||
### Alpaca-GPT4
|
||||
|
||||
The Alpaca-GPT4 was built by using the prompts of the original Alpaca dataset and generate the responses via GPT 4. The
|
||||
dataset consists of 52,000 instructions and responses.
|
||||
|
||||
The original [Alpaca-GPT4](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM) dataset can be used as follows:
|
||||
|
||||
```bash
|
||||
litgpt finetune lora "tiiuae/falcon-7b" \
|
||||
--data AlpacaGPT4
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt_finetune --data.help AlpacaGPT4` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
The Alpaca-GPT4 dataset distribution is shown below.
|
||||
|
||||
<img src="images/prepare_dataset/alpacagpt4.jpg" width=400px>
|
||||
|
||||
|
||||
|
||||
### Alpaca Libre
|
||||
|
||||
[Alpaca Libre](https://github.com/mobarski/alpaca-libre) is a reimplementation or alternative to Alpaca using the same formatting.
|
||||
|
||||
To use Alpaca Libre instead of the original Alpaca dataset, use the following command:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Alpaca \
|
||||
--data.file_url "https://raw.githubusercontent.com/mobarski/alpaca-libre/main/data/output/alpaca_libre_ok_tasks_v4.json" \
|
||||
--data.file_name "alpaca_libre_data_cleaned_archive.json"
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help Alpaca` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
The Alpaca Libre dataset distribution is shown below.
|
||||
|
||||
<img src="images/prepare_dataset/alpaca_libre.jpg" width=400px>
|
||||
|
||||
You may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 256 may be a good choice:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Alpaca \
|
||||
--data.file_url "https://raw.githubusercontent.com/mobarski/alpaca-libre/main/data/output/alpaca_libre_ok_tasks_v4.json" \
|
||||
--data.file_name "alpaca_libre_data_cleaned_archive.json" \
|
||||
--train.max_seq_length 256
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Deita
|
||||
|
||||
The Deita dataset (short for Data-Efficient Instruction Tuning for Alignment) is a collection of 9500 prompts and responses, as described in the [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://arxiv.org/abs/2312.15685) paper.
|
||||
Using Falcon 7b as an example, we can use the dataset as follows:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Deita
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help Deita` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
Deita contains multiturn conversations. By default, only the first instruction-response pairs from
|
||||
each of these multiturn conversations are included. If you want to override this behavior and include the follow-up instructions
|
||||
and responses, set `--data.include_multiturn_conversations True`, which will include all multiturn conversations as regular
|
||||
prompt-response pairs. Considering the multiturn-answers, the dataset consists of 209,272 prompt-response pairs.
|
||||
|
||||
The Deita dataset distribution without including multit-turn conversations is shown below.
|
||||
|
||||
<img src="images/prepare_dataset/deita.jpg" width=400px>
|
||||
|
||||
The Deita dataset distribution including multit-turn conversations is depicted in the following histogram.
|
||||
|
||||
<img src="images/prepare_dataset/deita-multiturn.jpg" width=400px>
|
||||
|
||||
You may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 512 may be a good choice:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Deita \
|
||||
--train.max_seq_length 512
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Dolly
|
||||
|
||||
The Dolly dataset is a publicly available collection of 15k instruction-following entries created by Databricks. It spans multiple behavioral domains, as described in the [InstructGPT paper](https://arxiv.org/abs/2203.02155) paper. These include areas like brainstorming, classification, closed QA, content creation, information retrieval, open QA, and summary generation.
|
||||
|
||||
The usage is similar to the Alpaca dataset described above. Using Falcon 7b as an example, we can use the dataset as follows:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Dolly
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help Dolly` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
The Dolly dataset distribution is shown below.
|
||||
|
||||
<img src="images/prepare_dataset/dolly.jpg" width=400px>
|
||||
|
||||
You may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 512 may be a good choice:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data Dolly \
|
||||
--train.max_seq_length 256
|
||||
```
|
||||
|
||||
|
||||
|
||||
### LongForm
|
||||
|
||||
LongForm is a semi-synthetic dataset based on raw text corpora for which the instructions were generated via an LLM. For more details about the instruction-generation process, please refer to the [LongForm research paper](https://arxiv.org/abs/2304.08460) by Köksal et al. According to the research paper, a Llama 7B model trained on LongForm achieves substantially better performance than the same Llama model trained on the 2x larger Alpaca dataset.
|
||||
|
||||
LongForm consists of 23,652 training samples, 2,042 validation samples, and 2,045 test samples. (In LitGPT, the validation samples are currently not used.)
|
||||
|
||||
The more detailed dataset composition is as follows based on a table taken from the [dataset repository](https://github.com/akoksal/LongForm):
|
||||
|
||||
| **Type** | **Source** | **Number of Examples** |
|
||||
|------------------------|----------------|------------------------|
|
||||
| **Corpora** | C4 | 10,000 |
|
||||
| | Wikipedia | 5,000 |
|
||||
| **Structured Corpora** | Stack Exchange | 4,380 |
|
||||
| | WikiHow | 2,500 |
|
||||
| **Tasks** | NIv2 | 3,684 |
|
||||
| | Big Bench | 600 |
|
||||
| | BEA-GEC | 1,203 |
|
||||
| | Enron | 372 |
|
||||
| **Total** | | 27,739 |
|
||||
| | | |
|
||||
| **Train** | | 23,652 |
|
||||
| **Validation** | | 2,042 |
|
||||
| **Test** | | 2,045 |
|
||||
|
||||
License information is not provided but would depend on the individual subsets listed above.
|
||||
|
||||
The LongForm dataset distribution is shown below.
|
||||
|
||||
<img src="images/prepare_dataset/longform.jpg" width=400px>
|
||||
|
||||
You may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 1500 may be a good choice:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data LongForm \
|
||||
--train.max_seq_length 1500
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help LongForm` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### LIMA
|
||||
|
||||
The LIMA dataset is a collection of 1,000 carefully curated prompts and responses, as described in the [LIMA: Less Is More for Alignment](https://arxiv.org/abs/2305.11206) paper. The dataset is sourced from three community Q&A websites: Stack Exchange, wikiHow, and the Pushshift Reddit Dataset. In addition, it also contains prompts and answers written and collected by the authors of the LIMA paper.
|
||||
|
||||
The usage is similar to the Dolly dataset described above except that it requires an Hugging Face access token that you need to copy & paste from your Hugging Face account. Using Falcon 7b as an example, we can use the dataset as follows:
|
||||
|
||||
```bash
|
||||
export HF_TOKEN="insert_your_huggingface_token_here"
|
||||
|
||||
litgpt finetune lora \
|
||||
--data LIMA \
|
||||
--checkpoint_dir "tiiuae/falcon-7b"
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help LIMA` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
LIMA contains a handful of multiturn conversations. By default, only the first instruction-response pairs from
|
||||
each of these multiturn conversations are included. If you want to override this behavior and include the follow-up instructions
|
||||
and responses, set `--data.include_multiturn_conversations True`.
|
||||
|
||||
The LIMA dataset distribution is shown below.
|
||||
|
||||
<img src="images/prepare_dataset/lima.jpg" width=400px>
|
||||
|
||||
You may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 512 may be a good choice:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data LIMA \
|
||||
--train.max_seq_length 512
|
||||
```
|
||||
|
||||
|
||||
|
||||
### FLAN
|
||||
|
||||
FLAN is a collection of several dataset subsets by Google. In particular, the provided script in LitGPT loads the subsets from
|
||||
[here](https://huggingface.co/datasets/Muennighoff/flan).
|
||||
|
||||
By default, all subsets (1,386,050 samples) and validations sets (367,190 subsets) are combined into a single dataset:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data FLAN
|
||||
```
|
||||
|
||||
However, you can also select individual subsets via comma-separated strings as follows:
|
||||
|
||||
```bash
|
||||
litgpt finetune lora tiiuae/falcon-7b \
|
||||
--data FLAN \
|
||||
--data.subsets "aeslc_10templates,ag_news_subset_10templates,anli_r1_10templates"
|
||||
```
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help FLAN` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
You can find a list of all 66 supported subsets [here](https://huggingface.co/datasets/Muennighoff/flan).
|
||||
|
||||
|
||||
|
||||
## Preparing Custom Datasets for Instruction Finetuning
|
||||
|
||||
The models in LitGPT expect datasets for instruction finetuning in the following format:
|
||||
|
||||
```text
|
||||
[
|
||||
{
|
||||
"instruction": "Write a limerick about a
|
||||
pelican.”,
|
||||
"input": "",
|
||||
"output": "There once was a pelican so fine,
|
||||
\nHis beak was as colorful as
|
||||
sunshine,\nHe would fish all day,\nIn
|
||||
a very unique way,\nThis pelican was
|
||||
truly divine!\n\n\n"
|
||||
},
|
||||
{
|
||||
"instruction": "Identify the odd one out from
|
||||
the group.",
|
||||
"input": "Carrot, Apple, Banana, Grape",
|
||||
"output": "Carrot\n\n"
|
||||
},
|
||||
]
|
||||
```
|
||||
|
||||
(Note that depending on the task, the `"input"` text can be an empty string, as shown above.)
|
||||
|
||||
You can use your own data in LitGPT by either reading in a JSON file in the format shown above or by implementing a custom `DataModule`.
|
||||
|
||||
|
||||
|
||||
### Preparing Custom Datasets From a JSON File
|
||||
|
||||
You can prepare custom dataset using a JSON file where each row is a dictionary with these keys:
|
||||
|
||||
- `instruction`: Column which will describe the task.
|
||||
- `input`: A string holding a special input value for the instruction. This applies to some samples, and in others, this is empty (empty string).
|
||||
- `output`: The expected response
|
||||
|
||||
> If any of the fields are missing, then the script will fail to read the dataset.
|
||||
|
||||
Then simply run any of the finetuning scripts with this input:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data JSON \
|
||||
--data.json_path path/to/your/data.json \
|
||||
--data.val_split_fraction 0.1
|
||||
```
|
||||
|
||||
You can also customize how the dataset is read by using these additional parameters
|
||||
|
||||
- `val_split_fraction`: The fraction of the data to split. Defaults to `0.1`
|
||||
|
||||
- `seed`: The seed value to reproduce the same random splits for train and test data.
|
||||
|
||||
- `mask_inputs`: Whether to mask the prompt section from the label (with `ignore_index`).
|
||||
|
||||
- `ignore_index`: The index to use for labels that should be ignored. Defaults to `-100` (used when `mask_inputs` is `True`).
|
||||
|
||||
To use the settings described above, you can add the respective command line arguments when calling the finetuning scripts as shown in the example below:
|
||||
|
||||
```bash
|
||||
litgpt finetune_lora tiiuae/falcon-7b \
|
||||
--data JSON \
|
||||
--data.json_path path/to/your/data.json \
|
||||
--data.val_split_fraction 0.1 \
|
||||
--data.seed 42 \
|
||||
--data.mask_inputs False \
|
||||
--data.ignore_index -100
|
||||
```
|
||||
|
||||
You can also pass a directory containing a `train.json` and `val.json` to `--data.json_path` to define a fixed train/val split.
|
||||
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use `litgpt finetune --data.help JSON` to list additional dataset-specific command line options.
|
||||
|
||||
|
||||
|
||||
### Preparing Custom Datasets Using DataModule
|
||||
|
||||
If you don't have a JSON file following the format described in the previous section, the easiest way to prepare a new dataset is to copy and modify one of the existing data modules in LitGPT:
|
||||
|
||||
- [`litgpt/data/alpaca.py`](https://github.com/Lightning-AI/litgpt/blob/main/litgpt/data/alpaca.py) (if you plan to load a dataset from a JSON file);
|
||||
- [`litgpt/data/lima.py`](https://github.com/Lightning-AI/litgpt/blob/main/litgpt/data/lima.py) (if you plan to load a dataset using the `datasets` Python library).
|
||||
|
||||
Note that you only need to modify a small fraction of the code file, namely the portion that downloads and formats the training data (see the `prepare_data` and `setup()` methods).
|
||||
|
||||
|
||||
|
||||
## Preparing Pretraining Datasets
|
||||
|
||||
In addition to the finetuning dataset described above, LitGPT also supports several datasets for pretraining. The pretraining datasets are described in more detail in the following separate tutorial documents:
|
||||
|
||||
- [Pretrain TinyLlama on Slimpajama and Starcoder](./pretrain_tinyllama.md)
|
||||
@@ -0,0 +1,206 @@
|
||||
# Pretrain LLMs with LitGPT
|
||||
|
||||
|
||||
This document explains how to pretrain LLMs using LitGPT.
|
||||
|
||||
|
||||
## Using the `litgpt pretrain` command
|
||||
|
||||
You can pretrain models in LitGPT using the `litgpt pretrain` API starting with any of the available architectures listed by calling `litgpt pretrain list` without any additional arguments:
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> To install all required dependencies before pretraining, first run `pip install "litgpt[all]"`.
|
||||
|
||||
|
||||
```bash
|
||||
litgpt pretrain list
|
||||
```
|
||||
|
||||
Shown below is an abbreviated list:
|
||||
|
||||
```
|
||||
ValueError: Please specify --model_name <model_name>. Available values:
|
||||
Camel-Platypus2-13B
|
||||
...
|
||||
Gemma-2b
|
||||
...
|
||||
Llama-2-7b-hf
|
||||
...
|
||||
Mixtral-8x7B-v0.1
|
||||
...
|
||||
pythia-14m
|
||||
```
|
||||
|
||||
For demonstration purposes, we can pretrain a small 14 million-parameter Pythia model on the small TinyStories dataset using the [debug.yaml config file](https://github.com/Lightning-AI/litgpt/blob/main/config_hub/pretrain/debug.yaml) as follows:
|
||||
|
||||
```bash
|
||||
litgpt pretrain pythia-14m \
|
||||
--config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/pretrain/debug.yaml
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Pretrain on custom data
|
||||
|
||||
The simplest way to get started with pretraining on a small custom dataset is by using the `TextFiles` data module, which lets you pretrain a dataset from a folder containing plain text files.
|
||||
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> This approach adds a beginning-of-sequence token at the beginning of each text file. However, it otherwise assumes that you have already cleaned the text files, for example, removing any unwanted characters and inserting beginning-of-sequence and end-of-sequence tokens if applicable in case a text file conists of multiple documents.
|
||||
|
||||
|
||||
|
||||
> [!WARNING]
|
||||
> Using this approach is only recommended for small datasets. Since text data is highly compressible, it is often stored in compressed format, and often in file formats where documents can be loaded row by row without having to load entire files at once. In other words, this `TextFiles` approach is only feasible to store the data in plain text files due to the limited size.
|
||||
> For datasets that take up multiple gigabytes, we recommend preprocessing it with [LitData](https://github.com/Lightning-AI/litdata) and then reading it from a local directory or S3 connection using `--data LitData`.
|
||||
|
||||
|
||||
|
||||
For instance, assume you stored a number of text files in a `custom_pretraining_dataset` folder (we recommend avoiding small files and concatenating them to files of at least 50 Mb for efficiency):
|
||||
|
||||
```bash
|
||||
~ ls -lh custom_pretraining_data
|
||||
total 3225M
|
||||
-rw-r--r-- 1 sebastian 50M Apr 2 18:31 combined_1.txt
|
||||
-rw-r--r-- 1 sebastian 50M Apr 2 18:31 combined_2.txt
|
||||
-rw-r--r-- 1 sebastian 50M Apr 2 18:31 combined_3.txt
|
||||
-rw-r--r-- 1 sebastian 50M Apr 2 18:31 combined_4.txt
|
||||
-rw-r--r-- 1 sebastian 50M Apr 2 18:31 combined_5.txt
|
||||
...
|
||||
```
|
||||
|
||||
You can then use the `TextFiles` API to pretrain a model (here a small `pythia-14m` model for illustration purposes) from scratch as follows:
|
||||
|
||||
```bash
|
||||
litgpt download EleutherAI/pythia-14m \
|
||||
--tokenizer_only true
|
||||
|
||||
litgpt pretrain pythia-14m \
|
||||
--tokenizer_dir EleutherAI/pythia-14m \
|
||||
--data TextFiles \
|
||||
--data.train_data_path custom_pretraining_data \
|
||||
--train.lr_warmup_steps=200 \
|
||||
--optimizer AdamW \
|
||||
--optimizer.lr 0.005
|
||||
```
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use the `litgpt pretrain --data.help TextFiles` command to list additional dataset options.
|
||||
|
||||
|
||||
|
||||
|
||||
## Continued pretraining on custom data
|
||||
|
||||
Often, it makes sense to adopt an existing pretrained model and further pretrain it on our own custom data. The existing pretrained model can be either our own pretrained model or a model downloaded from a model hub.
|
||||
|
||||
The following subsections illustrate three typical scenarioes:
|
||||
|
||||
1. Starting from a downloaded base model
|
||||
2. Continuing the pretraining after interruption
|
||||
3. Further pretraining on a different dataset
|
||||
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> This approach assumes that you have already cleaned the text files, for example, removing any unwanted characters and inserting beginning-of-sequence and end-of-sequence tokens if applicable.
|
||||
|
||||
|
||||
|
||||
> [!WARNING]
|
||||
> Using this approach is only recommended for small datasets. Since text data is highly compressible, it is often stored in compressed format, and often in file formats where documents can be loaded row by row without having to load entire files at once. In other words, this `TextFiles` approach is only feasible to store the data in plain text files due to the limited size.
|
||||
> For datasets that take up multiple gigabytes, we recommend preprocessing it with [LitData](https://github.com/Lightning-AI/litdata) and then reading it from a local directory or S3 connection using `--data LitData --data.path path/to/your/data`.
|
||||
|
||||
|
||||
|
||||
### 1) Continued pretraining when starting from a downloaded base model
|
||||
|
||||
|
||||
For instance, let's assume we download a Pythia model:
|
||||
|
||||
```bash
|
||||
litgpt download EleutherAI/pythia-160m
|
||||
```
|
||||
|
||||
Next, assume we have a custom dataset stored in text files similar to the *Pretrain on custom data* above. We can further pretrain the Pythia model via the `--initial_checkpoint_dir` setting as follows:
|
||||
|
||||
```bash
|
||||
litgpt pretrain pythia-160m \
|
||||
--initial_checkpoint_dir EleutherAI/pythia-160m \
|
||||
--tokenizer_dir EleutherAI/pythia-160m \
|
||||
--out_dir ./new_pretrained_checkpoint \
|
||||
--data TextFiles \
|
||||
--data.train_data_path custom_pretraining_data \
|
||||
--train.max_tokens 1_000_000
|
||||
```
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use the `litgpt pretrain --data.help TextFiles` command to list additional dataset options.
|
||||
|
||||
|
||||
|
||||
### 2) Continued pretraining after interruption
|
||||
|
||||
In case a you interrupted a training run, you can continue it with the `--resume` option, for example:
|
||||
|
||||
```bash
|
||||
litgpt pretrain pythia-160m \
|
||||
--resume "auto" \
|
||||
--tokenizer_dir EleutherAI/pythia-160m \
|
||||
--out_dir ./new_pretrained_checkpoint \
|
||||
--data TextFiles \
|
||||
--data.train_data_path custom_pretraining_data \
|
||||
--train.max_tokens 1_000_000
|
||||
```
|
||||
|
||||
|
||||
### 3) Continued pretraining on a new dataset
|
||||
|
||||
Suppose you pretrained a model using the examples above. To further pretrain the model on a new dataset, you first need to convert the pretrained checkpoint via the following command:
|
||||
|
||||
```bash
|
||||
litgpt convert_pretrained_checkpoint ./new_pretrained_checkpoint/final ./new_pretrained_checkpoint_converted
|
||||
```
|
||||
|
||||
Then, you can pretrain the converted model on the new dataset as follows:
|
||||
|
||||
```bash
|
||||
litgpt pretrain pythia-160m \
|
||||
--initial_checkpoint_dir ./new_pretrained_checkpoint_converted \
|
||||
--tokenizer_dir EleutherAI/pythia-160m \
|
||||
--out_dir ./new_pretrained_checkpoint_2 \
|
||||
--data TextFiles \
|
||||
--data.train_data_path custom_pretraining_data_2 \
|
||||
--train.max_tokens 1_000_000
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Pretrain a 1.1B TinyLlama model
|
||||
|
||||
You can find an end-to-end LitGPT tutorial for pretraining a TinyLlama model using LitGPT [here](pretrain_tinyllama.md).
|
||||
|
||||
|
||||
|
||||
## Optimize LitGPT pretraining with Lightning Thunder
|
||||
|
||||
[Lightning Thunder](https://github.com/Lightning-AI/lightning-thunder) is a source-to-source compiler for PyTorch, which is fully compatible with LitGPT. In experiments, Thunder resulted in a 40% speed-up compared to using regular PyTorch when finetuning a 7B Llama 2 model.
|
||||
|
||||
For more information, see the [Lightning Thunder extension README](https://github.com/Lightning-AI/lightning-thunder).
|
||||
|
||||
|
||||
|
||||
## Project templates
|
||||
|
||||
The following [Lightning Studio](https://lightning.ai/lightning-ai/studios) templates provide LitGPT pretraining projects in reproducible environments with multi-GPU and multi-node support:
|
||||
|
||||
|
||||
| | |
|
||||
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| <p align="left">[Prepare the TinyLlama 1T token dataset](https://lightning.ai/lightning-ai/studios/prepare-the-tinyllama-1t-token-dataset) <br> [<img src="https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/3.webp" width="300"></p>](https://lightning.ai/lightning-ai/studios/prepare-the-tinyllama-1t-token-dataset) | [Pretrain LLMs - TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/pretrain-llms-tinyllama-1-1b) <br> <p align="left">[<img src="https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/4.webp" width="300"></p>](https://lightning.ai/lightning-ai/studios/pretrain-llms-tinyllama-1-1b) |
|
||||
| [Continued Pretraining with TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/continued-pretraining-with-tinyllama-1-1b) <br> <p align="left">[<img src="https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/1.webp" width="300"></p>](https://lightning.ai/lightning-ai/studios/continued-pretraining-with-tinyllama-1-1b) | |
|
||||
| |
|
||||
@@ -0,0 +1,207 @@
|
||||
# Pretrain TinyLlama
|
||||
|
||||
This tutorial will walk you through pretraining [TinyLlama](https://github.com/jzhang38/TinyLlama/).
|
||||
|
||||
> [!TIP]
|
||||
> To get started with zero setup, clone the [TinyLlama studio on Lightning AI](https://lightning.ai/lightning-ai/studios/llm-pretrain-tinyllama-1-1b).
|
||||
|
||||
|
||||
## What's TinyLlama?
|
||||
|
||||
[TinyLlama](https://github.com/jzhang38/TinyLlama/) is architecturally the same as Meta AI's LLama 2, but only has 1.1B parameters and is instead trained on multiple epochs on a mix of [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B) and [Starcoder](https://huggingface.co/datasets/bigcode/starcoderdata) datasets.
|
||||
|
||||
Here is a quick fact sheet:
|
||||
|
||||
| Name | Description |
|
||||
|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| Parameters | 1.1B |
|
||||
| Model Size | Layers: 22, Heads: 32, Query Groups: 4, Embedding Size: 2048, Intermediate Size: 5632 |
|
||||
| Sequence Length | 2048 |
|
||||
| Learning Rate | 4e-4 |
|
||||
| Learning Rate Schedule | Cosine with 2000 warmup steps |
|
||||
| Training Data | [SlimPajama](https://huggingface.co/datasets/cerebras/slimpajama-627b) (893 GB), [Starcoder](https://huggingface.co/datasets/bigcode/starcoderdata) (290 GB) |
|
||||
| Combined Dataset Size | Around 950B tokens |
|
||||
| Total Tokens During Training | 3 trillion (3 epochs) |
|
||||
| Time to complete training | ~ 4 weeks with 64 A100 GPUs |
|
||||
| Model FLOPs Utilization (MFU) | 52% |
|
||||
|
||||
(this table was sourced from the author's [README](https://github.com/jzhang38/TinyLlama/))
|
||||
|
||||
|
||||
## Download datasets
|
||||
|
||||
You can download the data using git lfs:
|
||||
|
||||
```bash
|
||||
# Make sure you have git-lfs installed (https://git-lfs.com):
|
||||
sudo apt install git-lfs
|
||||
```
|
||||
|
||||
```bash
|
||||
git clone https://huggingface.co/datasets/cerebras/slimpajama-627b data/slimpajama-raw
|
||||
git clone https://huggingface.co/datasets/bigcode/starcoderdata data/starcoderdata-raw
|
||||
```
|
||||
|
||||
Around 1.2 TB of disk space is required to store both datasets.
|
||||
|
||||
|
||||
## Prepare the datasets for training
|
||||
|
||||
In order to start pretraining litgpt on it, you need to read, tokenize, and write the data in binary chunks. This will leverage the `litdata` optimization pipeline and streaming dataset.
|
||||
|
||||
First, install additional dependencies for preprocessing:
|
||||
|
||||
```bash
|
||||
pip install '.[all]'
|
||||
```
|
||||
|
||||
You will need to have the tokenizer config available:
|
||||
|
||||
```bash
|
||||
litgpt download meta-llama/Llama-2-7b-hf \
|
||||
--access_token your_hf_token \
|
||||
--tokenizer_only true
|
||||
```
|
||||
|
||||
Then, run the preprocessing script for each dataset and split.
|
||||
You will require **1.1 TB** of disk space for Starcoder and **2.5** TB of space for the SlimPajama dataset.
|
||||
|
||||
**Starcoder:**
|
||||
|
||||
```bash
|
||||
python litgpt/data/prepare_starcoder.py \
|
||||
--input_dir data/starcoderdata-raw \
|
||||
--output_dir data/starcoder \
|
||||
--tokenizer_path checkpoints/meta-llama/Llama-2-7b-hf
|
||||
```
|
||||
|
||||
**SlimPajama:**
|
||||
|
||||
```bash
|
||||
python litgpt/data/prepare_slimpajama.py \
|
||||
--input_dir data/slimpajama-raw/validation \
|
||||
--output_dir data/slimpajama/val \
|
||||
--tokenizer_path checkpoints/meta-llama/Llama-2-7b-hf
|
||||
|
||||
python litgpt/data/prepare_slimpajama.py \
|
||||
--input_dir data/slimpajama-raw/test \
|
||||
--output_dir data/slimpajama/test \
|
||||
--tokenizer_path checkpoints/meta-llama/Llama-2-7b-hf
|
||||
|
||||
python litgpt/data/prepare_slimpajama.py \
|
||||
--input_dir data/slimpajama-raw/train \
|
||||
--output_dir data/slimpajama/train \
|
||||
--tokenizer_path checkpoints/meta-llama/Llama-2-7b-hf
|
||||
```
|
||||
|
||||
If you want to run on a small slice of the datasets first, pass the flag `--fast_dev_run=true` to the commands above.
|
||||
In the above we are assuming that you will be using the same tokenizer as used in LlaMA/TinyLlama, but any trained [SentencePiece](https://github.com/google/sentencepiece) tokenizer with a 32000 vocabulary size will do here.
|
||||
|
||||
|
||||
## Pretraining
|
||||
|
||||
Running the pretraining script with its default settings requires at least 8 A100 GPUs.
|
||||
|
||||
```bash
|
||||
litgpt pretrain --config config_hub/pretrain/tinyllama.yaml
|
||||
```
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use the `litgpt pretrain --data.help TinyLlama` command to list additional dataset options.
|
||||
|
||||
|
||||
|
||||
The script will save checkpoints periodically to the folder `out/`.
|
||||
By default, the `pretrain` script will pretrain the model with FSDP in
|
||||
`bfloat16` mixed precision and gradient accumulation.
|
||||
|
||||
Note that `pretrain` is not actually a model-specific training script, so feel free [try other configurations](../config_hub)
|
||||
or change the model type and size by passing a different string to the model name argument, for example:
|
||||
|
||||
```shell
|
||||
litgpt pretrain Gemma-2b
|
||||
```
|
||||
|
||||
The currently supported model names can be listed by executing `litgpt pretrain` without any additional arguments.
|
||||
|
||||
Keep in mind that training with a single machine will take weeks. To speed up the process, you'll need access to a cluster.
|
||||
Once you're in a cluster, you can follow [these instructions](https://lightning.ai/docs/fabric/stable/fundamentals/launch.html#launch-on-a-cluster)
|
||||
to launch the script across machines:
|
||||
|
||||
- [Lightning AI](https://lightning.ai/docs/fabric/stable/guide/multi_node/cloud.html)
|
||||
- [SLURM cluster](https://lightning.ai/docs/fabric/stable/guide/multi_node/slurm.html)
|
||||
- [Barebones cluster](https://lightning.ai/docs/fabric/stable/guide/multi_node/barebones.html)
|
||||
- [MPI](https://lightning.ai/docs/fabric/stable/guide/multi_node/other.html)
|
||||
|
||||
The script exposes several hyperparameters you can tweak through the command line.
|
||||
|
||||
For instance, `--train.micro_batch_size` should be adjusted so the process will use the available
|
||||
GPU memory. For more tips to avoid out-of-memory issues, please also see the more detailed
|
||||
[Dealing with out-of-memory (OOM) errors](oom.md) guide.
|
||||
|
||||
Last, logging is kept minimal in the script, but for long-running experiments we recommend switching to a proper experiment tracker.
|
||||
LitGPT supports multiple experiment trackers including:
|
||||
|
||||
- **TensorBoard** (default): Local visualization with TensorBoard
|
||||
- **CSV Logger**: Simple local logging to CSV files
|
||||
- **WandB**: Cloud-based experiment tracking with Weights & Biases
|
||||
- **MLflow**: MLflow experiment tracking
|
||||
- **[LitLogger](https://github.com/Lightning-AI/LitLogger)**: Lightning.ai's native experiment tracking (set `--logger_name=litlogger`)
|
||||
|
||||
As an example, we included WandB (set `--logger_name=wandb`) to show how you can integrate any experiment tracking framework.
|
||||
For reference, [here are the loss curves for our reproduction](https://api.wandb.ai/links/awaelchli/y7pzdpwy).
|
||||
|
||||
|
||||
## Resume training
|
||||
|
||||
The checkpoints saved during pretraining contain all the information to resume if needed.
|
||||
Simply rerun the script with the `--resume` argument added:
|
||||
|
||||
```bash
|
||||
litgpt pretrain tiny-llama\
|
||||
--config config_hub/pretrain/tinyllama.yaml \
|
||||
--resume out/pretrain/tiny-llama/step-00060500
|
||||
```
|
||||
**Important:** Each checkpoint is a directory. Point to the directory, not the 'lit_model.pth' file inside of it.
|
||||
|
||||
|
||||
> [!TIP]
|
||||
> Use the `litgpt pretrain --data.help TinyLlama` command to list additional dataset options.
|
||||
|
||||
|
||||
|
||||
|
||||
## Export checkpoints
|
||||
|
||||
After training is completed, you can convert the checkpoint to a format that can be loaded for evaluation, inference, finetuning etc.
|
||||
|
||||
```bash
|
||||
litgpt convert_pretrained_checkpoint out/pretrain/tiny-llama/step-00060500 \
|
||||
--output_dir checkpoints/tiny-llama/final
|
||||
```
|
||||
|
||||
After conversion, the output folder will contain these files:
|
||||
```
|
||||
checkpoints/tiny-llama/final
|
||||
├── model_config.yaml
|
||||
├── lit_model.pth
|
||||
├── tokenizer_config.json
|
||||
├── tokenizer.json
|
||||
└── tokenizer.model
|
||||
```
|
||||
|
||||
You can then use this checkpoint folder to run [evaluation](evaluation.md), [inference](inference.md), [finetuning](finetune_lora.md) or [process the checkpoint further](convert_lit_models.md).
|
||||
|
||||
|
||||
|
||||
## Project templates
|
||||
|
||||
The following [Lightning Studio](https://lightning.ai/lightning-ai/studios) templates provide LitGPT pretraining projects in reproducible environments with multi-GPU and multi-node support:
|
||||
|
||||
|
||||
| | |
|
||||
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| <p align="left">[Prepare the TinyLlama 1T token dataset](https://lightning.ai/lightning-ai/studios/prepare-the-tinyllama-1t-token-dataset) <br> [<img src="https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/3.webp" width="300"></p>](https://lightning.ai/lightning-ai/studios/prepare-the-tinyllama-1t-token-dataset) | [Pretrain LLMs - TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/pretrain-llms-tinyllama-1-1b) <br> <p align="left">[<img src="https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/4.webp" width="300"></p>](https://lightning.ai/lightning-ai/studios/pretrain-llms-tinyllama-1-1b) |
|
||||
| [Continued Pretraining with TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/continued-pretraining-with-tinyllama-1-1b) <br> <p align="left">[<img src="https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/1.webp" width="300"></p>](https://lightning.ai/lightning-ai/studios/continued-pretraining-with-tinyllama-1-1b) | |
|
||||
| |
|
||||
@@ -0,0 +1,441 @@
|
||||
# LitGPT Python API
|
||||
|
||||
This is a work-in-progress draft describing the current LitGPT Python API (experimental and subject to change).
|
||||
|
||||
|
||||
## Model loading
|
||||
|
||||
Use the `LLM.load` method to load a model from a LitGPT model checkpoint folder. For example, consider loading a Phi-2 model. If a given checkpoint directory `"microsoft/phi-2"` does not exist as a local checkpoint directory, the model will be downloaded automatically from the HF Hub (assuming that `"microsoft/phi-2"` is a valid repository name):
|
||||
|
||||
```python
|
||||
from litgpt import LLM
|
||||
|
||||
llm_1 = LLM.load("microsoft/phi-2")
|
||||
```
|
||||
|
||||
```
|
||||
config.json: 100%|████████████████████████████████████████████████| 735/735 [00:00<00:00, 7.75MB/s]
|
||||
generation_config.json: 100%|█████████████████████████████████████| 124/124 [00:00<00:00, 2.06MB/s]
|
||||
model-00001-of-00002.safetensors: 100%|███████████████████████████| 5.00G/5.00G [00:12<00:00, 397MB/s]
|
||||
model-00002-of-00002.safetensors: 100%|███████████████████████████| 564M/564M [00:01<00:00, 421MB/s]
|
||||
model.safetensors.index.json: 100%|███████████████████████████████| 35.7k/35.7k [00:00<00:00, 115MB/s]
|
||||
tokenizer.json: 100%|█████████████████████████████████████████████| 2.11M/2.11M [00:00<00:00, 21.5MB/s]
|
||||
tokenizer_config.json: 100%|██████████████████████████████████████| 7.34k/7.34k [00:00<00:00, 80.6MB/s]
|
||||
```
|
||||
|
||||
|
||||
> [!NOTE]
|
||||
> To get a list of all supported models, execute `litgpt download list` in the command line terminal.
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
If you attempt to load the model again, LitGPT will load this model from a local directory since it's already been downloaded:
|
||||
|
||||
```python
|
||||
llm_2 = LLM.load("microsoft/phi-2")
|
||||
```
|
||||
|
||||
|
||||
If you created a pretrained or finetuned model checkpoint via LitGPT, you can load it in a similar fashion:
|
||||
|
||||
```python
|
||||
my_llm = LLM.load("path/to/my/local/checkpoint")
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Generate/Chat
|
||||
|
||||
Generate output using the `.generate` method:
|
||||
|
||||
```python
|
||||
from litgpt import LLM
|
||||
|
||||
llm = LLM.load("microsoft/phi-2")
|
||||
|
||||
text = llm.generate("What do Llamas eat?", top_k=1, max_new_tokens=30)
|
||||
print(text)
|
||||
```
|
||||
|
||||
```
|
||||
Llamas are herbivores and primarily eat grass, leaves, and shrubs. They have a specialized digestive system that allows them to efficiently extract
|
||||
```
|
||||
|
||||
Alternatively, stream the response one token at a time:
|
||||
|
||||
```python
|
||||
result = llm.generate("hi", stream=True)
|
||||
for e in result:
|
||||
print(e, end="", flush=True)
|
||||
```
|
||||
|
||||
```
|
||||
Llamas are herbivores and primarily eat grass, leaves, and shrubs. They have a specialized digestive system that allows them to efficiently extract
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Saving models
|
||||
|
||||
After finetuning or modifying a model, you can save it to disk using the `.save()` method:
|
||||
|
||||
```python
|
||||
from litgpt import LLM
|
||||
|
||||
llm = LLM.load("microsoft/phi-2")
|
||||
# ... perform finetuning or modifications ...
|
||||
llm.save("path/to/save/directory")
|
||||
```
|
||||
|
||||
The saved checkpoint can then be loaded later:
|
||||
|
||||
```python
|
||||
llm = LLM.load("path/to/save/directory")
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Random weights
|
||||
|
||||
To start with random weights, for example, if you plan a pretraining script, initialize the model with `init="random"`. Note that this requires passing a `tokenizer_dir` that contains a valid tokenizer file.
|
||||
|
||||
```python
|
||||
from litgpt.api import LLM
|
||||
llm = LLM.load("pythia-160m", init="random", tokenizer_dir="EleutherAI/pythia-160m")
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Multi-GPU strategies
|
||||
|
||||
By default, the model is loaded onto a single GPU. Optionally, you can use the `.distribute()` method with the "sequential" or "tensor_parallel" `generate_strategy` settings.
|
||||
|
||||
### Sequential strategy
|
||||
|
||||
The `generate_strategy="sequential"` setting loads different parts of the models onto different GPUs. The goal behind this strategy is to support models that cannot fit into single-GPU memory. (Note that if you have a model that can fit onto a single GPU, this sequential strategy will be slower.)
|
||||
|
||||
```python
|
||||
from litgpt.api import LLM
|
||||
|
||||
llm = LLM.load(
|
||||
"microsoft/phi-2",
|
||||
distribute=None
|
||||
)
|
||||
|
||||
llm.distribute(
|
||||
generate_strategy="sequential",
|
||||
devices=4, # Optional setting, otherwise uses all available GPUs
|
||||
fixed_kv_cache_size=256 # Optionally use a small kv-cache to further reduce memory usage
|
||||
)
|
||||
```
|
||||
|
||||
```
|
||||
Using 4 devices
|
||||
Moving '_forward_module.transformer.h.31' to cuda:3: 100%|██████████| 32/32 [00:00<00:00, 32.71it/s]
|
||||
```
|
||||
|
||||
After initializing the model, the model can be used via the `generate` method similar to the default `generate_strategy` setting:
|
||||
|
||||
```python
|
||||
text = llm.generate("What do llamas eat?", max_new_tokens=100)
|
||||
print(text)
|
||||
```
|
||||
|
||||
```
|
||||
Llamas are herbivores and their diet consists mainly of grasses, plants, and leaves.
|
||||
```
|
||||
|
||||
|
||||
### Tensor parallel strategy
|
||||
|
||||
The sequential strategy explained in the previous subsection distributes the model sequentially across GPUs, which allows users to load models that would not fit onto a single GPU. However, due to this method's sequential nature, processing is naturally slower than parallel processing.
|
||||
|
||||
To take advantage of parallel processing via tensor parallelism, you can use the `generate_strategy="tensor_parallel" setting. However, this method has downsides: the initial setup may be slower for large models, and it cannot run in interactive processes such as Jupyter notebooks.
|
||||
|
||||
```python
|
||||
from litgpt.api import LLM
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
llm = LLM.load(
|
||||
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
distribute=None
|
||||
)
|
||||
|
||||
llm.distribute(generate_strategy="tensor_parallel", devices=4)
|
||||
|
||||
print(llm.generate(prompt="What do llamas eat?"))
|
||||
print(llm.generate(prompt="What is 1+2?", top_k=1))
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Speed and resource estimates
|
||||
|
||||
Use the `.benchmark()` method to compare the computational performance of different settings. The `.benchmark()` method takes the same arguments as the `.generate()` method. For example, we can estimate the speed and GPU memory consumption as follows (the resulting numbers were obtained on an A10G GPU):
|
||||
|
||||
```python
|
||||
from litgpt.api import LLM
|
||||
from pprint import pprint
|
||||
|
||||
llm = LLM.load(
|
||||
model="microsoft/phi-2",
|
||||
distribute=None
|
||||
)
|
||||
|
||||
llm.distribute(fixed_kv_cache_size=500)
|
||||
|
||||
text, bench_d = llm.benchmark(prompt="What do llamas eat?", top_k=1, stream=True)
|
||||
print(text)
|
||||
pprint(bench_d)
|
||||
|
||||
|
||||
# Llamas are herbivores and primarily eat grass, leaves, and shrubs. They have a specialized
|
||||
# digestive system that allows them to efficiently extract nutrients from plant material.
|
||||
|
||||
# Using 1 device(s)
|
||||
# Llamas are herbivores and primarily eat grass, leaves, and shrubs. They have a unique digestive system that allows them to efficiently extract nutrients from tough plant material.
|
||||
|
||||
# {'Inference speed in tokens/sec': [17.617540650112936],
|
||||
# 'Seconds to first token': [0.6533610639999097],
|
||||
# 'Seconds total': [1.4758019020000575],
|
||||
# 'Tokens generated': [26],
|
||||
# 'Total GPU memory allocated in GB': [5.923729408]}
|
||||
```
|
||||
|
||||
To get more reliably estimates, it's recommended to repeat the benchmark for multiple iterations via `num_iterations=10`:
|
||||
|
||||
```python
|
||||
text, bench_d = llm.benchmark(num_iterations=10, prompt="What do llamas eat?", top_k=1, stream=True)
|
||||
print(text)
|
||||
pprint(bench_d)
|
||||
|
||||
# Using 1 device(s)
|
||||
# Llamas are herbivores and primarily eat grass, leaves, and shrubs. They have a unique digestive system that allows them to efficiently extract nutrients from tough plant material.
|
||||
|
||||
# {'Inference speed in tokens/sec': [17.08638672485105,
|
||||
# 31.79908547222976,
|
||||
# 32.83646959864293,
|
||||
# 32.95994240022436,
|
||||
# 33.01563039816964,
|
||||
# 32.85263413816648,
|
||||
# 32.82712094713627,
|
||||
# 32.69216141907453,
|
||||
# 31.52431714347663,
|
||||
# 32.56752130561681],
|
||||
# 'Seconds to first token': [0.7278506560005553,
|
||||
# 0.022963577999689733,
|
||||
# 0.02399449199947412,
|
||||
# 0.022921959999621322,
|
||||
# ...
|
||||
```
|
||||
|
||||
As one can see, the first iteration may take longer due to warmup times. So, it's recommended to discard the first iteration:
|
||||
|
||||
```python
|
||||
for key in bench_d:
|
||||
bench_d[key] = bench_d[key][1:]
|
||||
```
|
||||
|
||||
For better visualization, you can use the `benchmark_dict_to_markdown_table` function
|
||||
|
||||
```python
|
||||
from litgpt.api import benchmark_dict_to_markdown_table
|
||||
|
||||
print(benchmark_dict_to_markdown_table(bench_d_list))
|
||||
```
|
||||
|
||||
| Metric | Mean | Std Dev |
|
||||
|-------------------------------------|-----------------------------|-----------------------------|
|
||||
| Seconds total | 0.80 | 0.01 |
|
||||
| Seconds to first token | 0.02 | 0.00 |
|
||||
| Tokens generated | 26.00 | 0.00 |
|
||||
| Inference speed in tokens/sec | 32.56 | 0.50 |
|
||||
| Total GPU memory allocated in GB | 5.92 | 0.00 |
|
||||
|
||||
|
||||
|
||||
# PyTorch Lightning Trainer support
|
||||
|
||||
You can use the LitGPT `LLM` class with the [PyTorch Lightning Trainer](https://lightning.ai/docs/pytorch/stable/common/trainer.html) to pretrain and finetune models.
|
||||
|
||||
The examples below show the usage via a simple 160 million parameter model for demonstration purposes to be able to quickly try it out. However, you can replace the `EleutherAI/pythia-160m` model with any model supported by LitGPT (you can find a list of supported models by executing `litgpt download list` or visiting the [model weight docs](download_model_weights.md)).
|
||||
|
||||
|
||||
## Step 1: Define a `LightningModule`
|
||||
|
||||
First, we define a `LightningModule` similar to what we would do when working with other types of neural networks in PyTorch Lightning:
|
||||
|
||||
|
||||
```python
|
||||
import torch
|
||||
import litgpt
|
||||
from litgpt import LLM
|
||||
from litgpt.data import Alpaca2k
|
||||
import lightning as L
|
||||
|
||||
|
||||
class LitLLM(L.LightningModule):
|
||||
def __init__(self, checkpoint_dir, tokenizer_dir=None, trainer_ckpt_path=None):
|
||||
super().__init__()
|
||||
|
||||
self.llm = LLM.load(checkpoint_dir, tokenizer_dir=tokenizer_dir, distribute=None)
|
||||
self.trainer_ckpt_path = trainer_ckpt_path
|
||||
|
||||
def setup(self, stage):
|
||||
self.llm.trainer_setup(trainer_ckpt=self.trainer_ckpt_path)
|
||||
|
||||
def training_step(self, batch):
|
||||
logits, loss = self.llm(input_ids=batch["input_ids"], target_ids=batch["labels"])
|
||||
self.log("train_loss", loss, prog_bar=True)
|
||||
return loss
|
||||
|
||||
def validation_step(self, batch):
|
||||
logits, loss = self.llm(input_ids=batch["input_ids"], target_ids=batch["labels"])
|
||||
self.log("validation_loss", loss, prog_bar=True)
|
||||
return loss
|
||||
|
||||
def configure_optimizers(self):
|
||||
warmup_steps = 10
|
||||
optimizer = torch.optim.AdamW(self.llm.model.parameters(), lr=0.0002, weight_decay=0.0, betas=(0.9, 0.95))
|
||||
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda step: step / warmup_steps)
|
||||
return [optimizer], [scheduler]
|
||||
```
|
||||
|
||||
In the code example above, note how we set `distribute=None` in `llm.load()` in the `__init__` method. This step is necessary because we want to let the PyTorch Lightning Trainer handle the GPU devices. We then call `self.llm.trainer_setup` in the `setup()` method, which adjusts the LitGPT settings to be compatible with the Trainer. Other than that, everything else looks like a standard `LightningModule`.
|
||||
|
||||
Next, we have a selection of different use cases, but first, let's set some general settings to specify the batch size and gradient accumulation steps:
|
||||
|
||||
```python
|
||||
batch_size = 8
|
||||
accumulate_grad_batches = 1
|
||||
```
|
||||
|
||||
For larger models, you may want to decrease the batch size and increase the number of accumulation steps. (Setting `accumulate_grad_batches = 1` effectively disables gradient accumulation, and it is only shown here for reference in case you wish to change this setting.)
|
||||
|
||||
## Step 2: Using the Trainer
|
||||
|
||||
|
||||
### Use case 1: Pretraining from random weights
|
||||
|
||||
In case you plan to train a model from scratch (not recommended over finetuning because training a model from scratch in general requires substantial time and resources), you can do it as follows:
|
||||
|
||||
```python
|
||||
# Create model with random as opposed to pretrained weights
|
||||
llm = LLM.load("EleutherAI/pythia-160m", tokenizer_dir="EleutherAI/pythia-160m", init="random")
|
||||
llm.save("pythia-160m-random-weights")
|
||||
del llm
|
||||
|
||||
lit_model = LitLLM(checkpoint_dir="pythia-160m-random-weights", tokenizer_dir="EleutherAI/pythia-160m")
|
||||
data = Alpaca2k()
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
```
|
||||
|
||||
|
||||
### Use case 2: Continued pretraining or finetuning a downloaded model
|
||||
|
||||
The continued pretraining or finetuning from a downloaded model checkpoint is similar to the example above, except that we can skip the initial steps of instantiating a model with random weights.
|
||||
|
||||
```python
|
||||
|
||||
lit_model = LitLLM(checkpoint_dir="EleutherAI/pythia-160m")
|
||||
data = Alpaca2k()
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
```
|
||||
|
||||
|
||||
### Use case 3: Resume training from Trainer checkpoint
|
||||
|
||||
Suppose you trained a model and decide to follow up with a few additional training rounds. This can be achieved as follows by loading an existing Trainer checkpoint:
|
||||
|
||||
```python
|
||||
|
||||
import os
|
||||
|
||||
def find_latest_checkpoint(directory):
|
||||
latest_checkpoint = None
|
||||
latest_time = 0
|
||||
|
||||
for root, _, files in os.walk(directory):
|
||||
for file in files:
|
||||
if file.endswith('.ckpt'):
|
||||
file_path = os.path.join(root, file)
|
||||
file_time = os.path.getmtime(file_path)
|
||||
if file_time > latest_time:
|
||||
latest_time = file_time
|
||||
latest_checkpoint = file_path
|
||||
|
||||
return latest_checkpoint
|
||||
|
||||
lit_model = LitLLM(checkpoint_dir="EleutherAI/pythia-160m", trainer_ckpt_path=find_latest_checkpoint("lightning_logs"))
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
```
|
||||
|
||||
|
||||
### Use case 4: Resume training after saving a checkpoint manually
|
||||
|
||||
This example illustrates how we can save a LitGPT checkpoint from a previous training run that we can load and use later. Note that compared to using the Trainer checkpoint in the previous section, the model saved via this approach also contains the tokenizer and other relevant files. Hence, this approach does not require the original `"EleutherAI/pythia-160m"` model checkpoint directory.
|
||||
|
||||
```python
|
||||
lit_model.llm.save("finetuned_checkpoint")
|
||||
del lit_model
|
||||
lit_model = LitLLM(checkpoint_dir="finetuned_checkpoint")
|
||||
|
||||
data.connect(lit_model.llm.tokenizer, batch_size=batch_size, max_seq_length=512)
|
||||
|
||||
trainer = L.Trainer(
|
||||
devices=1,
|
||||
accelerator="cuda",
|
||||
max_epochs=1,
|
||||
accumulate_grad_batches=accumulate_grad_batches,
|
||||
precision="bf16-true",
|
||||
)
|
||||
trainer.fit(lit_model, data)
|
||||
|
||||
lit_model.llm.model.to(lit_model.llm.preprocessor.device)
|
||||
lit_model.llm.generate("hello world")
|
||||
```
|
||||
@@ -0,0 +1,135 @@
|
||||
# Quantize the model
|
||||
|
||||
This document provides different strategies for quantizing the various models available in LitGPT to reduce GPU memory usage, which is useful for running larger models on certain GPU hardware.
|
||||
|
||||
**All the examples below were run on an A100 40GB GPU with CUDA 12.1.**
|
||||
|
||||
> [!NOTE]
|
||||
> Quantization also supports finetuning via [QLoRA](finetune_lora.md)
|
||||
|
||||
## Baseline
|
||||
|
||||
It's useful to start with a baseline to have a reference point for memory savings via the various quantization methods.
|
||||
|
||||
```bash
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--precision 32-true \
|
||||
--max_new_tokens 256
|
||||
...
|
||||
Time for inference 1: 6.93 sec total, 36.96 tokens/sec.
|
||||
Memory used: 28.95 GB
|
||||
```
|
||||
|
||||
First, using a lower precision compared to 32-bit float can result in two times reduced memory consumption. You can either try setting `--precision 16-true` for regular 16-bit precision or `--precision bf16-true` if your GPU supports brain-float 16-bit precision. ([This brief video](https://lightning.ai/courses/deep-learning-fundamentals/9.0-overview-techniques-for-speeding-up-model-training/unit-9.1-accelerated-model-training-via-mixed-precision-training/) explains the difference between regular 16-bit and bf16-bit precision.)
|
||||
|
||||
In short, when `--precision bf16-true` or `--precision 16-true` is used, the model weights will automatically be converted and consume less memory.
|
||||
However, this might not be enough for large models or when using GPUs with limited memory.
|
||||
|
||||
```bash
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--precision bf16-true \
|
||||
--max_new_tokens 256
|
||||
...
|
||||
Time for inference 1: 5.37 sec total, 47.66 tokens/sec.
|
||||
Memory used: 14.50 GB
|
||||
```
|
||||
|
||||
To reduce the memory requirements further, LitGPT supports several quantization techniques, which are shown below.
|
||||
|
||||
> [!TIP]
|
||||
> Most quantization examples below also use the `--precision bf16-true` setting explained above. If your GPU does not support `bfloat16`, you can change it to `--precision 16-true`.
|
||||
|
||||
## `bnb.nf4`
|
||||
|
||||
Enabled with [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). Check out the [paper](https://arxiv.org/abs/2305.14314v1) to learn more about how it works.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> `bitsandbytes` only supports `CUDA` devices and the `Linux` operating system.
|
||||
> Windows users should use [WSL2](https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl).
|
||||
|
||||
Uses the normalized float 4 (nf4) data type. This is recommended over "fp4" based on the paper's experimental results and theoretical analysis.
|
||||
|
||||
```bash
|
||||
pip install bitsandbytes
|
||||
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--quantize bnb.nf4 \
|
||||
--precision bf16-true \
|
||||
--max_new_tokens 256
|
||||
...
|
||||
Time for inference 1: 6.80 sec total, 37.62 tokens/sec
|
||||
Memory used: 5.72 GB
|
||||
```
|
||||
|
||||
## `bnb.nf4-dq`
|
||||
|
||||
Enabled with [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). Check out the [paper](https://arxiv.org/abs/2305.14314v1) to learn more about how it works.
|
||||
|
||||
"dq" stands for "Double Quantization" which reduces the average memory footprint by quantizing the quantization constants.
|
||||
In average, this amounts to about 0.37 bits per parameter (approximately 3 GB for a 65B model).
|
||||
|
||||
```bash
|
||||
pip install bitsandbytes
|
||||
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--quantize bnb.nf4-dq \
|
||||
--precision bf16-true \
|
||||
--max_new_tokens 256
|
||||
|
||||
...
|
||||
Time for inference 1: 8.09 sec total, 30.87 tokens/sec
|
||||
Memory used: 5.38 GB
|
||||
```
|
||||
|
||||
## `bnb.fp4`
|
||||
|
||||
Enabled with [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). Check out the [paper](https://arxiv.org/abs/2305.14314v1) to learn more about how it works.
|
||||
|
||||
Uses pure FP4 quantization.
|
||||
|
||||
```bash
|
||||
pip install bitsandbytes
|
||||
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--quantize bnb.fp4 \
|
||||
--precision bf16-true \
|
||||
--max_new_tokens 256
|
||||
...
|
||||
Time for inference 1: 6.92 sec total, 36.98 tokens/sec
|
||||
Memory used: 5.72 GB
|
||||
```
|
||||
|
||||
## `bnb.fp4-dq`
|
||||
|
||||
Enabled with [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). Check out the [paper](https://arxiv.org/abs/2305.14314v1) to learn more about how it works.
|
||||
|
||||
"dq" stands for "Double Quantization" which reduces the average memory footprint by quantizing the quantization constants.
|
||||
In average, this amounts to about 0.37 bits per parameter (approximately 3 GB for a 65B model).
|
||||
|
||||
```bash
|
||||
pip install bitsandbytes
|
||||
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--quantize bnb.fp4-dq \
|
||||
--precision bf16-true \
|
||||
--max_new_tokens 256
|
||||
...
|
||||
Time for inference 1: 10.02 sec total, 25.54 tokens/sec
|
||||
Memory used: 5.38 GB
|
||||
```
|
||||
|
||||
## `bnb.int8`
|
||||
|
||||
Enabled with [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). Check out the [paper](https://arxiv.org/abs/2110.02861) to learn more about how it works.
|
||||
|
||||
```bash
|
||||
pip install bitsandbytes
|
||||
|
||||
litgpt generate tiiuae/falcon-7b \
|
||||
--quantize bnb.int8 \
|
||||
--precision 16-true \
|
||||
--max_new_tokens 256
|
||||
...
|
||||
Time for inference 1: 20.22 sec total, 12.66 tokens/sec
|
||||
Memory used: 8.70 GB
|
||||
```
|
||||
@@ -0,0 +1,158 @@
|
||||
# Resource Tables
|
||||
|
||||
- Last updated: 10/20/2023
|
||||
- LitGPT version: commit 8641822
|
||||
- Hardware: NVIDIA A100-SXM4-40GB
|
||||
- OS: Ubuntu 22.04.3 LTS (x86_64)
|
||||
- Nvidia driver version: 525.125.06
|
||||
- Relevant libraries
|
||||
- PyTorch 2.1.0+cu121
|
||||
- Bitsandbytes 0.41.1
|
||||
|
||||
This document provides an overview and examples of hardware requirements when running models in LitGPT.
|
||||
|
||||
For additional tips on lowering the GPU memory footprint, please also see the [Dealing with out-of-memory (OOM) errors](oom.md) document.
|
||||
|
||||
All experiments were run using 16-bit brain floating point precision (`--precision bf16-true`). If your GPU does not support brain floating point precision, you can use regular 16-bit floating point precision (`--precision 16-true`).
|
||||
|
||||
All experiments were conducted using the Alpaca dataset with its default length. Note that due to different tokenizers being used by the different models, the number of tokens in the longest training example differs based on the model:
|
||||
|
||||
- phi1.5: 1044 tokens
|
||||
- StableLM Alpha: 1034 tokens
|
||||
- Llama 2: 1304 tokens
|
||||
- Falcon 1079 tokens
|
||||
|
||||
Note that the number of tokens in the training set does not affect the supported context width (block size) of the models, which is as follows:
|
||||
|
||||
- phi1.5: 2048 tokens
|
||||
- StableLM 3B Alpha: 4096 tokens
|
||||
- Llama 2: 4048 tokens
|
||||
- Falcon: 2048 tokens
|
||||
- CodeLlama 13B: 16384 tokens
|
||||
|
||||
|
||||
|
||||
## Finetuning with LoRA on 1 GPU
|
||||
|
||||
The following experiments were conducted on 1xA100 with a minibatch size of 128 using the `litgpt finetune_lora` command.
|
||||
|
||||
| Size | Model | Quantization | Microbatch size | Trainable parameters | Max GPU RAM | Time 1k iterations |
|
||||
|-------|----------------|--------------|-----------------|----------------------|-------------|--------------------|
|
||||
| 1.3 B | phi-1.5 | None | 1 | 1,572,864 | 4.82 GB | 1.62 min |
|
||||
| 1.3 B | phi-1.5 | bnb.nf4 | 1 | 1,572,864 | 3.78 GB | 1.77 min |
|
||||
| 1.3 B | phi-1.5 | bnb.nf4-dq | 1 | 1,572,864 | 3.72 GB | 1.87 min |
|
||||
| 1.3 B | phi-1.5 | None | 2 | 1,572,864 | 6.76 GB | 1.65 min |
|
||||
| 1.3 B | phi-1.5 | None | 4 | 1,572,864 | 10.68 GB | 1.70 min |
|
||||
| | | | | | | |
|
||||
| 3 B | StableLM Alpha | None | 1 | 2,097,152 | 9.69 GB | 1.24 min |
|
||||
| 3 B | StableLM Alpha | bnb.nf4 | 1 | 2,097,152 | 6.35 GB | 1.82 min |
|
||||
| 3 B | StableLM Alpha | bnb.nf4-dq | 1 | 2,097,152 | 6.19 GB | 1.87 min |
|
||||
| 3 B | StableLM Alpha | None | 2 | 2,097,152 | 12.10 GB | 1.33 min |
|
||||
| 3 B | StableLM Alpha | None | 4 | 2,097,152 | 16.92 GB | 1.50 min |
|
||||
| | | | | | | |
|
||||
| 7 B | Llama 2 | None | 1 | 4,194,304 | 21.30 GB | 2.36 min |
|
||||
| 7 B | Llama 2 | bnb.nf4 | 1 | 4,194,304 | 14.14 GB | 3.68 min |
|
||||
| 7 B | Llama 2 | bnb.nf4-dq | 1 | 4,194,304 | 13.84 GB | 3.83 min |
|
||||
| 7 B | Llama 2 | None | 2 | 4,194,304 | 29.07 GB | 2.52 min |
|
||||
| 7 B | Llama 2 | None | 4 | 4,194,304 | OOM | - |
|
||||
| | | | | | | |
|
||||
| 13 B | Llama 2 | None | 1 | 6,553,600 | 38.12 GB | 3.19 min |
|
||||
| 13 B | Llama 2 | bnb.nf4 | 1 | 6,553,600 | 23.14 GB | 6.38 min |
|
||||
| 13 B | Llama 2 | bnb.nf4-dq | 1 | 6,553,600 | 22.55 GB | 6.55 min |
|
||||
| 13 B | Llama 2 | None | 2 | 6,553,600 | OOM | - |
|
||||
| 13 B | Llama 2 | None | 4 | 6,553,600 | OOM | - |
|
||||
| | | | | | | |
|
||||
| 40 B | Falcon | None | 1 | 12,042,240 | OOM | - |
|
||||
| 40 B | Falcon | bnb.nf4 | 1 | 12,042,240 | OOM | - |
|
||||
| 40 B | Falcon | bnb.nf4-dq | 1 | 12,042,240 | OOM | - |
|
||||
|
||||
|
||||
|
||||
## Finetuning with Adapter on 1 GPU
|
||||
|
||||
The following experiments were conducted on 1xA100 with a minibatch size of 128 using the `litgpt finetune_adapter` command.
|
||||
|
||||
| Size | Model | Quantization | Microbatch size | Trainable parameters | Max GPU RAM | Time 1k iterations |
|
||||
|------|----------------|--------------|-----------------|----------------------|-------------|--------------------|
|
||||
| 3 B | StableLM Alpha | None | 1 | 573,888 | 9.10 GB | 0.74 min |
|
||||
| 3 B | StableLM Alpha | bnb.nf4 | 1 | 573,888 | 5.65 GB | 1.38 min |
|
||||
| 3 B | StableLM Alpha | bnb.nf4-dq | 1 | 573,888 | 5.48 GB | 1.46 min |
|
||||
| | | | | | | |
|
||||
| 7 B | Llama 2 | None | 1 | 1,229,760 | 19.98 GB | 1.50 min |
|
||||
| 7 B | Llama 2 | bnb.nf4 | 1 | 1,229,760 | 12.68 GB | 2.93 min |
|
||||
| 7 B | Llama 2 | bnb.nf4-dq | 1 | 1,229,760 | 12.38 GB | 3.00 min |
|
||||
|
||||
The same config, but using the `litgpt finetune_adapter_v2` command.
|
||||
|
||||
| Size | Model | Quantization | Microbatch size | Trainable parameters | Max GPU RAM | Time 1k iterations |
|
||||
|------|----------------|--------------|-----------------|----------------------|-------------|--------------------|
|
||||
| 3 B | StableLM Alpha | None | 1 | 2,125,248 | 10.71 GB | 0.87 min |
|
||||
| 3 B | StableLM Alpha | bnb.nf4 | 1 | 2,125,248 | 7.41 GB | 1.59 min |
|
||||
| 3 B | StableLM Alpha | bnb.nf4-dq | 1 | 2,125,248 | 7.25 GB | 1.62 min |
|
||||
| | | | | | | |
|
||||
| 7 B | Llama 2 | None | 1 | 4,279,744 | 25.51 GB | 1.81 min |
|
||||
| 7 B | Llama 2 | bnb.nf4 | 1 | 4,279,744 | 18.30 GB | 3.23 min |
|
||||
| 7 B | Llama 2 | bnb.nf4-dq | 1 | 4,279,744 | 17.98 GB | 3.32 min |
|
||||
|
||||
|
||||
|
||||
## Finetuning with LoRA on Multiple GPUs
|
||||
|
||||
The following experiments were conducted on multiple A100 GPUs with a minibatch size of 128 using the `litgpt finetune_lora` command.
|
||||
|
||||
| Size | Model | Quantization | Microbatch size | Trainable parameters | GPU | Max GPU RAM | Time 1k iterations |
|
||||
|-------|----------------|--------------|-----------------|----------------------|----------|-------------|--------------------|
|
||||
| 1.3 B | phi-1.5 | None | 1 | 1,572,864 | 2 x A100 | 4.86 GB | 3.81 min |
|
||||
| 1.3 B | phi-1.5 | bnb.nf4 | 1 | 1,572,864 | 2 x A100 | N/A | - |
|
||||
| 1.3 B | phi-1.5 | bnb.nf4-dq | 1 | 1,572,864 | 2 x A100 | N/A | - |
|
||||
| 1.3 B | phi-1.5 | None | 2 | 1,572,864 | 2 x A100 | 5.05 GB | 3.63 min |
|
||||
| 1.3 B | phi-1.5 | None | 4 | 1,572,864 | 2 x A100 | 5.88 GB | 3.64 min |
|
||||
| | | | | | | | |
|
||||
| 3 B | StableLM Alpha | None | 1 | 2,097,152 | 2 x A100 | 12.75 GB | 2.92 min |
|
||||
| 3 B | StableLM Alpha | None | 2 | 2,097,152 | 2 x A100 | 12.94 GB | 3.06 min |
|
||||
| 3 B | StableLM Alpha | None | 4 | 2,097,152 | 2 x A100 | 13.45 GB | 3.86 min |
|
||||
| | | | | | | | - |
|
||||
| 7 B | Llama 2 | None | 1 | 4,194,304 | 2 x A100 | 22.18 GB | 5.93 min |
|
||||
| 7 B | Llama 2 | None | 2 | 4,194,304 | 2 x A100 | 22.47 GB | 6.48 min |
|
||||
| 7 B | Llama 2 | None | 4 | 4,194,304 | 2 x A100 | 23.39 GB | 8.66 min |
|
||||
| | | | | | | | |
|
||||
| 13 B | Llama 2 | None | 1 | 6,553,600 | 2 x A100 | OOM | - |
|
||||
| 13 B | Llama 2 | bnb.nf4 | 1 | 6,553,600 | 2 x A100 | N/A | - |
|
||||
| 13 B | Llama 2 | bnb.nf4-dq | 1 | 6,553,600 | 2 x A100 | N/A | - |
|
||||
| | | | | | | | |
|
||||
| 13 B | Llama 2 | None | 1 | 6,553,600 | 4 x A100 | 35.57 GB | 10.25 min |
|
||||
| 40 B | Falcon | None | 1 | 12,042,240 | 4 x A100 | OOM | - |
|
||||
|
||||
|
||||
|
||||
## Single-GPU Inference
|
||||
|
||||
| Size | Model | Quantization | GPU | Max GPU RAM | Token/sec |
|
||||
|-------|----------------|--------------|----------|-------------------------------------------|-----------|
|
||||
| 1.3 B | phi-1.5 | None | 1 x A100 | 2.86 GB | 42.56 |
|
||||
| 1.3 B | phi-1.5 | bnb.nf4 | 1 x A100 | 1.39 GB | 22.89 |
|
||||
| 1.3 B | phi-1.5 | bnb.nf4-dq | 1 x A100 | 1.33 GB | 22.75 |
|
||||
| | | | | | |
|
||||
| 3 B | StableLM Alpha | None | 1 x A100 | 7.30 GB | 49.01 |
|
||||
| 3 B | StableLM Alpha | bnb.nf4 | 1 x A100 | 3.20 GB | 29.04 |
|
||||
| 3 B | StableLM Alpha | bnb.nf4-dq | 1 x A100 | 3.04 GB | 27.15 |
|
||||
| | | | | | |
|
||||
| 7 B | Llama 2 | None | 1 x A100 | 13.52 GB | 30.97 |
|
||||
| 7 B | Llama 2 | bnb.nf4 | 1 x A100 | 4.57 GB | 19.98 |
|
||||
| 7 B | Llama 2 | bnb.nf4-dq | 1 x A100 | 4.26 GB | 17.3 |
|
||||
| | | | | | |
|
||||
| 13 B | Llama 2 | None | 1 x A100 | 26.21 GB | 24.82 |
|
||||
| 13 B | Llama 2 | bnb.nf4 | 1 x A100 | 8.32 GB | 16.73 |
|
||||
| 13 B | Llama 2 | bnb.nf4-dq | 1 x A100 | 7.72 GB | 14.43 |
|
||||
| | | | | | |
|
||||
| 34 B | CodeLlama | None | 1 x A100 | OOM | - |
|
||||
| 34 B | CodeLlama | bnb.nf4 | 1 x A100 | 20.52 GB | 14.32 |
|
||||
| 34 B | CodeLlama | bnb.nf4-dq | 1 x A100 | 18.95 GB | 12.37 |
|
||||
| | | | | | |
|
||||
| 40 B | Falcon | None | 1 x A100 | OOM | - |
|
||||
| 40 B | Falcon | bnb.nf4 | 1 x A100 | 26.55 GB | 13.25 |
|
||||
| 40 B | Falcon | bnb.nf4-dq | 1 x A100 | 24.63 GB | 11.64 |
|
||||
| | | | | | |
|
||||
| 70 B | Llama 2 | None | 1 x A100 | OOM | - |
|
||||
| 70 B | Llama 2 | bnb.nf4 | 1 x A100 | CUDA error: CUBLAS_STATUS_NOT_INITIALIZED | - |
|
||||
| 70 B | Llama 2 | bnb.nf4-dq | 1 x A100 | 37.21 GB | 7.97 |
|
||||
Reference in New Issue
Block a user