huggingface--transformers/docs/source/en/model_doc/flan-ul2.md

This model was contributed to Hugging Face Transformers on 2023-06-20.

FLAN-UL2

Overview

Flan-UL2 is an encoder decoder model based on the T5 architecture. It uses the same configuration as the UL2 model released earlier last year. It was fine tuned using the "Flan" prompt tuning and dataset collection. Similar to Flan-T5, one can directly use FLAN-UL2 weights without finetuning the model:

According to the original blog here are the notable improvements:

• The original UL2 model was only trained with receptive field of 512, which made it non-ideal for N-shot prompting where N is large.
• The Flan-UL2 checkpoint uses a receptive field of 2048 which makes it more usable for few-shot in-context learning.
• The original UL2 model also had mode switch tokens that was rather mandatory to get good performance. However, they were a little cumbersome as this requires often some changes during inference or finetuning. In this update/change, we continue training UL2 20B for an additional 100k steps (with small batch) to forget “mode tokens” before applying Flan instruction tuning. This Flan-UL2 checkpoint does not require mode tokens anymore. Google has released the following variants:

The original checkpoints can be found here.

Running on low resource devices

The model is pretty heavy (~40GB in half precision) so if you just want to run the model, make sure you load your model in 8bit, and use device_map="auto" to make sure you don't have any OOM issue!

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, BitsAndBytesConfig


model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-ul2", quantization_config=BitsAndBytesConfig(load_in_8bit=True), device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("google/flan-ul2")

inputs = tokenizer("A step by step recipe to make bolognese pasta:", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['In a large skillet, brown the ground beef and onion over medium heat. Add the garlic']

Refer to T5's documentation page for API reference, tips, code examples and notebooks.