*This model was published in HF papers on 2026-04-13 and contributed to Hugging Face Transformers on 2026-07-06.*
# TIPSv2
## Overview
TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of
contrastive vision-language encoders proposed in [TIPSv2: Advancing Vision-Language Pretraining with Enhanced
Patch-Text Alignment]((https://huggingface.co/papers/2604.12012)) by Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis, Kaifeng Chen, Arjun Karpur, Ye Xia, Sahil Dua, Tanmaya Dabral, Guangxing Han, Bohyung Han, Joshua Ainslie, Alex Bewley, Mithun Jacob, René Wagner, Washington Ramos, Krzysztof Choromanski, Mojtaba Seyedhosseini, Howard Zhou, André Araujo.
The abstract from the paper is the following:
*Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment – surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/.*
This model was contributed by [Ternuraz](https://huggingface.co/Ternuraz) and [guarin](https://huggingface.co/guarin).
The original code can be found [here](https://github.com/google-deepmind/tips).
You can find all the original TIPSv2 checkpoints under the [TIPSv2](https://huggingface.co/collections/google/tipsv2) collection.
> [!TIP]
> See [TIPSv2 DPT](./tipsv2_dpt) for depth estimation, normal estimation, and semantic segmentation on top of the TIPSv2 vision backbone.
```python
from transformers import pipeline
image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
candidate_labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
classifier = pipeline(task="zero-shot-image-classification", model="google/tipsv2-b14", device_map="auto")
out = classifier(image, candidate_labels=candidate_labels)
print(out)
# [{'score': 0.999, 'label': 'a photo of a cat'}, {'score': 0.001, 'label': 'a photo of a dog'}, {'score': 0.0, 'label': 'a photo of a car'}]
```
```python
import torch
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image
model_id = "google/tipsv2-b14"
model = AutoModel.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
candidate_labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = processor(text=candidate_labels, images=image, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=1)
most_likely_idx = probs.argmax(dim=1).item()
most_likely_label = candidate_labels[most_likely_idx]
print(f"Most likely label: '{most_likely_label}' with probability: {probs[0][most_likely_idx].item():.3f}")
# Most likely label: 'a photo of a cat' with probability: 0.975
```
Use [`~Tipsv2Model.get_image_features`] and [`~Tipsv2Model.get_text_features`] to encode images and texts separately, which is useful for retrieval workflows where embeddings are computed independently (e.g., pre-indexing a large image database). The returned embeddings are *not* normalized, so apply L2 normalization before computing similarity.
```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image
model_id = "google/tipsv2-b14"
model = AutoModel.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
candidate_labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
image_inputs = processor(images=image, return_tensors="pt").to(model.device)
text_inputs = processor(text=candidate_labels, return_tensors="pt").to(model.device)
with torch.no_grad():
image_outputs = model.get_image_features(**image_inputs)
text_outputs = model.get_text_features(**text_inputs)
image_embeds = F.normalize(image_outputs.pooler_output, dim=-1)
text_embeds = F.normalize(text_outputs.pooler_output, dim=-1)
probs = (image_embeds @ text_embeds.T / model.temperature).softmax(dim=-1)
most_likely_idx = probs.argmax(dim=-1).item()
most_likely_label = candidate_labels[most_likely_idx]
print(f"Most likely label: '{most_likely_label}' with probability: {probs[0][most_likely_idx].item():.3f}")
# Most likely label: 'a photo of a cat' with probability: 0.975
```
Use [`AutoBackbone`] to load the vision backbone directly and get spatial feature maps, without the text model.
```python
import torch
from transformers import AutoBackbone, AutoImageProcessor
from transformers.image_utils import load_image
model_id = "google/tipsv2-b14"
backbone = AutoBackbone.from_pretrained(model_id, device_map="auto")
image_processor = AutoImageProcessor.from_pretrained(model_id)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
inputs = image_processor(images=image, return_tensors="pt").to(backbone.device)
with torch.no_grad():
outputs = backbone(**inputs)
# feature_maps is a tuple of tensors, one per requested stage
patch_features = outputs.feature_maps[-1] # (batch_size, hidden_size, height, width) tensor
```
Use [`Tipsv2VisionModel`] if you only need access to the vision features. In particular, this allows you to access the two class tokens from Tipsv2 as shown below. Note that Tipsv2 repurposed the register token from [`Dinov2WithRegistersModel`] as a secondary class token. The two tokens differ in how they were trained:
- Class token 1: Supervised by web alt-text captions
- Class token 2: Supervised by PaliGemma synthetic captions
```python
import torch
from transformers import AutoConfig, AutoImageProcessor, AutoModel
from transformers.image_utils import load_image
model_id = "google/tipsv2-b14"
config = AutoConfig.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, config=config.vision_config, device_map="auto")
image_processor = AutoImageProcessor.from_pretrained(model_id)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
inputs = image_processor(images=image, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model(**inputs)
sequence = outputs.last_hidden_state # (batch_size, 1 + config.num_register_tokens + num_patches, hidden_size)
cls_token_1 = sequence[:, 0] # (batch_size, hidden_size)
cls_token_2 = sequence[:, 1] # (batch_size, hidden_size)
```
## Notes
- [`Tipsv2Model`] returns normalized `image_embeds` and `text_embeds`. `logits_per_image` and `logits_per_text` are convenience outputs computed as cosine similarity divided by the temperature.
- Use [`~Tipsv2Model.get_image_features`] and [`~Tipsv2Model.get_text_features`] to retrieve image and text embeddings individually.
- Use [`Tipsv2VisionBackbone`] if you need access to feature maps from all layers.
- Use [`Tipsv2VisionModel`] if you need access to the two vision class tokens.
## Tipsv2Config
[[autodoc]] Tipsv2Config
## Tipsv2TextConfig
[[autodoc]] Tipsv2TextConfig
## Tipsv2VisionConfig
[[autodoc]] Tipsv2VisionConfig
## Tipsv2Tokenizer
[[autodoc]] Tipsv2Tokenizer
- __call__
## Tipsv2ImageProcessor
[[autodoc]] Tipsv2ImageProcessor
- preprocess
## Tipsv2Processor
[[autodoc]] Tipsv2Processor
- __call__
## Tipsv2Model
[[autodoc]] Tipsv2Model
- forward
- get_text_features
- get_image_features
## Tipsv2TextModel
[[autodoc]] Tipsv2TextModel
- forward
## Tipsv2VisionModel
[[autodoc]] Tipsv2VisionModel
- forward
## Tipsv2VisionBackbone
[[autodoc]] Tipsv2VisionBackbone
- forward
