wehub-resource-sync
e06fe8e8c6
Secret Leaks / trufflehog (push) Failing after 1s
Build documentation / build (push) Failing after 1s
Build documentation / build_other_lang (push) Failing after 0s
CodeQL Security Analysis / CodeQL Analysis (push) Failing after 0s
PR CI / pr-ci (push) Failing after 1s
Slow tests on important models (on Push - A10) / Get all modified files (push) Failing after 1s
Slow tests on important models (on Push - A10) / Model CI (push) Has been skipped
Self-hosted runner (benchmark) / Benchmark (aws-g5-4xlarge-cache) (push) Has been cancelled
New model PR merged notification / Notify new model (push) Has been cancelled
Update Transformers metadata / build_and_package (push) Has been cancelled
3.0 KiB
Executable File
3.0 KiB
Executable File
This model was published in HF papers on 2025-10-01 and contributed to Hugging Face Transformers on 2026-02-23.
ColModernVBert
Overview
ColModernVBert is a model for efficient visual document retrieval. It leverages ModernVBert to construct multi-vector embeddings directly from document images, following the ColPali approach.
The model was introduced in ModernVBERT: Towards Smaller Visual Document Retrievers.
import torch
from huggingface_hub import hf_hub_download
from PIL import Image
from transformers import ColModernVBertForRetrieval, ColModernVBertProcessor
processor = ColModernVBertProcessor.from_pretrained("ModernVBERT/colmodernvbert-hf")
model = ColModernVBertForRetrieval.from_pretrained("ModernVBERT/colmodernvbert-hf", device_map="auto")
# Load the test dataset
queries = [
"A paint on the wall",
"ColModernVBERT matches the performance of models nearly 10x larger on visual document benchmarks."
]
images = [
Image.open(hf_hub_download("HuggingFaceTB/SmolVLM", "example_images/rococo.jpg", repo_type="space")),
Image.open(hf_hub_download("ModernVBERT/colmodernvbert", "table.png", repo_type="model"))
]
# Preprocess the examples
batch_images = processor(images=images).to(model.device)
batch_queries = processor(text=queries).to(model.device)
# Run inference
with torch.inference_mode():
image_embeddings = model(**batch_images).embeddings
query_embeddings = model(**batch_queries).embeddings
# Compute retrieval scores
scores = processor.score_retrieval(
query_embeddings=query_embeddings,
passage_embeddings=image_embeddings,
)
scores = torch.softmax(scores, dim=-1)
print(scores) # [[0.9350, 0.0650], [0.0015, 0.9985]]
ColModernVBertConfig
autodoc ColModernVBertConfig
ColModernVBertProcessor
autodoc ColModernVBertProcessor
ColModernVBertForRetrieval
autodoc ColModernVBertForRetrieval - forward