huggingface--transformers/docs/source/en/model_doc/tipsv2.md

This model was published in HF papers on 2026-04-13 and contributed to Hugging Face Transformers on 2026-07-06.

TIPSv2

Overview

TIPSv2 (Text-Image Pre-training with Spatial awareness) is a family of contrastive vision-language encoders proposed in TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment by Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis, Kaifeng Chen, Arjun Karpur, Ye Xia, Sahil Dua, Tanmaya Dabral, Guangxing Han, Bohyung Han, Joshua Ainslie, Alex Bewley, Mithun Jacob, René Wagner, Washington Ramos, Krzysztof Choromanski, Mojtaba Seyedhosseini, Howard Zhou, André Araujo.

The abstract from the paper is the following:

Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment – surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/.

This model was contributed by Ternuraz and guarin. The original code can be found here.

You can find all the original TIPSv2 checkpoints under the TIPSv2 collection.

Tip

See TIPSv2 DPT for depth estimation, normal estimation, and semantic segmentation on top of the TIPSv2 vision backbone.

from transformers import pipeline


image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
candidate_labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]

classifier = pipeline(task="zero-shot-image-classification", model="google/tipsv2-b14", device_map="auto")
out = classifier(image, candidate_labels=candidate_labels)
print(out)
# [{'score': 0.999, 'label': 'a photo of a cat'}, {'score': 0.001, 'label': 'a photo of a dog'}, {'score': 0.0, 'label': 'a photo of a car'}]

import torch

from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image


model_id = "google/tipsv2-b14"
model = AutoModel.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
candidate_labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]

inputs = processor(text=candidate_labels, images=image, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

probs = outputs.logits_per_image.softmax(dim=1)
most_likely_idx = probs.argmax(dim=1).item()
most_likely_label = candidate_labels[most_likely_idx]
print(f"Most likely label: '{most_likely_label}' with probability: {probs[0][most_likely_idx].item():.3f}")
# Most likely label: 'a photo of a cat' with probability: 0.975

Use [~Tipsv2Model.get_image_features] and [~Tipsv2Model.get_text_features] to encode images and texts separately, which is useful for retrieval workflows where embeddings are computed independently (e.g., pre-indexing a large image database). The returned embeddings are not normalized, so apply L2 normalization before computing similarity.

import torch
import torch.nn.functional as F

from transformers import AutoModel, AutoProcessor
from transformers.image_utils import load_image


model_id = "google/tipsv2-b14"
model = AutoModel.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
candidate_labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]

image_inputs = processor(images=image, return_tensors="pt").to(model.device)
text_inputs = processor(text=candidate_labels, return_tensors="pt").to(model.device)

with torch.no_grad():
    image_outputs = model.get_image_features(**image_inputs)
    text_outputs = model.get_text_features(**text_inputs)

image_embeds = F.normalize(image_outputs.pooler_output, dim=-1)
text_embeds = F.normalize(text_outputs.pooler_output, dim=-1)

probs = (image_embeds @ text_embeds.T / model.temperature).softmax(dim=-1)
most_likely_idx = probs.argmax(dim=-1).item()
most_likely_label = candidate_labels[most_likely_idx]
print(f"Most likely label: '{most_likely_label}' with probability: {probs[0][most_likely_idx].item():.3f}")
# Most likely label: 'a photo of a cat' with probability: 0.975

Use [AutoBackbone] to load the vision backbone directly and get spatial feature maps, without the text model.

import torch
from transformers import AutoBackbone, AutoImageProcessor
from transformers.image_utils import load_image


model_id = "google/tipsv2-b14"
backbone = AutoBackbone.from_pretrained(model_id, device_map="auto")
image_processor = AutoImageProcessor.from_pretrained(model_id)

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
inputs = image_processor(images=image, return_tensors="pt").to(backbone.device)

with torch.no_grad():
    outputs = backbone(**inputs)

# feature_maps is a tuple of tensors, one per requested stage
patch_features = outputs.feature_maps[-1] # (batch_size, hidden_size, height, width) tensor

Use [Tipsv2VisionModel] if you only need access to the vision features. In particular, this allows you to access the two class tokens from Tipsv2 as shown below. Note that Tipsv2 repurposed the register token from [Dinov2WithRegistersModel] as a secondary class token. The two tokens differ in how they were trained:

• Class token 1: Supervised by web alt-text captions
• Class token 2: Supervised by PaliGemma synthetic captions

import torch
from transformers import AutoConfig, AutoImageProcessor, AutoModel
from transformers.image_utils import load_image


model_id = "google/tipsv2-b14"
config = AutoConfig.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, config=config.vision_config, device_map="auto")
image_processor = AutoImageProcessor.from_pretrained(model_id)

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")
inputs = image_processor(images=image, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

sequence = outputs.last_hidden_state # (batch_size, 1 + config.num_register_tokens + num_patches, hidden_size)
cls_token_1 = sequence[:, 0] # (batch_size, hidden_size)
cls_token_2 = sequence[:, 1] # (batch_size, hidden_size)

Notes

• [Tipsv2Model] returns normalized image_embeds and text_embeds. logits_per_image and logits_per_text are convenience outputs computed as cosine similarity divided by the temperature.
• Use [~Tipsv2Model.get_image_features] and [~Tipsv2Model.get_text_features] to retrieve image and text embeddings individually.
• Use [Tipsv2VisionBackbone] if you need access to feature maps from all layers.
• Use [Tipsv2VisionModel] if you need access to the two vision class tokens.

Tipsv2Config

autodoc Tipsv2Config

Tipsv2TextConfig

autodoc Tipsv2TextConfig

Tipsv2VisionConfig

autodoc Tipsv2VisionConfig

Tipsv2Tokenizer

autodoc Tipsv2Tokenizer - call

Tipsv2ImageProcessor

autodoc Tipsv2ImageProcessor - preprocess

Tipsv2Processor

autodoc Tipsv2Processor - call

Tipsv2Model

autodoc Tipsv2Model - forward - get_text_features - get_image_features

Tipsv2TextModel

autodoc Tipsv2TextModel - forward

Tipsv2VisionModel

autodoc Tipsv2VisionModel - forward

Tipsv2VisionBackbone

autodoc Tipsv2VisionBackbone - forward