chore: import upstream snapshot with attribution

python/sglang/srt/multimodal/evs/README.md

@@ -0,0 +1,123 @@
+# Efficient Video Sampling (EVS)
+
+Implementation of [Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference](https://arxiv.org/abs/2510.14624).
+
+## Overview
+
+> NOTE: The current implementation in sglang is cannot work with VLMs that use positional embeddings [Such as Qwen2.5VL]. Further work is warranted.
+
+Video frames often contain redundant information, as consecutive frames may be nearly identical. EVS exploits this in the latent space [=embedding space] by computing similarity between adjacent frame token embeddings and pruning tokens that are highly similar to the previous frames. This reduces the token count while preserving informative content.
+
+Key properties:
+- The first frame is always fully retained (provides complete initial context)
+- Configurable via `video_pruning_rate` in model config.json (0 = disabled, 0.7 = ~70% reduction; ~30% retained.)
+
+
+## Performance Characteristics VS. Accuracy - Example
+
+> NOTE: Actual retained accuracy post-EVS may depend on how dynamic the input videos are, how high the pruning rate is, whether or not the model was trained with EVS on or not, etc.
+> To learn more, read the paper above. It is incumbent on the user to evaluate as per their use case and benchmarks.
+
+A cursory example of a performance boost due to EVS:
+
+```bash
+export SGLANG_VLM_CACHE_SIZE_MB=0
+sglang serve --model-path nvidia/Nemotron-Nano-12B-v2-VL-BF16 --trust-remote-code --mem-fraction-static 0.8 --max-mamba-cache-size 128 --chunked-prefill-size 8192
+```
+Example Request:
+```json
+{ "model": "nvidia/Nemotron-Nano-12B-v2-VL-BF16", "stream": true, "temperature": 0.0, "max_completion_tokens": 3, "messages": [{ "role": "user", "content": [{ "type": "video_url", "video_url": { "url": "file:///tmp/01.mp4" } }]}]}
+```
+
+- `1XH100 95GiB`
+- `BS=1`
+- All 30 videos of `https://huggingface.co/datasets/lmms-lab/Video-MME/blob/main/videos_chunked_01.zip`
+- Default [for this model] pruning rate of `--json-model-override-args '{"video_pruning_rate": 0.7}'` [i.e., 30% of tokens are preserved] VS. `--json-model-override-args '{"video_pruning_rate": 0.0}'` [EVS off]
+
+| Scenario\ Metric                	| Online TTFT (Seconds) stderr: ±0.38 	| VideoMME Accuracy       	|
+|---------------------------------	|-------------------------------------	|-------------------------	|
+| EVS Off [q=0.0]                 	| 11.96 [100%]                          | Between 0.665 and 0.668 	|
+| EVS Off [q=0.4]                 	| 09.97 [ 83%]                          |                         	|
+| EVS On  [q=0.7] (default value) 	| 08.79 [ 73%]                          |                         	|
+| EVS Off [q=0.9]                 	| 08.39 [ 70%]                         	| 0.644                   	|
+
+
+## Architecture
+
+### Request Flow
+
+1. Prompt Construction (EVSProcessor)
+    * Calculates estimated tokens per frame based on pruning rate, so the emitted input_ids tensor's length will by definition match the final sequence length post pruning. This is necessary for 3.
+2. Embedding Generation (EVS)
+    * Calls original model `get_video_feature()` for full embeddings
+    * Retains top-k dissimilar tokens
+    * Returns EVSEmbeddingResult in addition to pruned token counts *per frame*
+3. Token Redistribution (mm_utils)
+    * Adjusts input_ids so each frame's placeholder tokens matches the pruned count from 2.
+
+
+## Integration Guide
+
+### Step 1: Model [See `NemotronH_Nano_VL_V2`]
+
+Make your model inherit from `EVS` and implement `create_evs_config`:
+
+```python
+from sglang.srt.multimodal.evs import EVSConfig, EVS
+
+class MyEVSVideoModel(EVS):
+    @staticmethod
+    def create_evs_config(config):
+        return EVSConfig(
+            video_pruning_rate=config.video_pruning_rate
+        )
+
+    def __init__(self, config, ...):
+        super().__init__(config)  # EVS wraps get_video_feature
+        ...
+
+    def get_video_feature(self, items):
+        # Your existing implementation
+        # Returns: (total_frames, tokens_per_frame, hidden_dim)
+        ...
+```
+
+### Step 2: Processor [See `NanoNemotronVLImageProcessor`]
+
+Create an `EVSProcessor` as a member of your VLImageProcessor:
+
+```python
+from sglang.srt.multimodal.evs import EVSProcessor
+
+class MyProcessor:
+    models = [MyEVSVideoModel, MyNonEVSModel] # You may mix evs and non evs models in a processor
+
+    def __init__(hf_config):
+        self.evs = EVSProcessor(hf_config, config_to_evs_model={MyEVSVideoModelConfig: MyEVSVideoModel})
+
+    def process_video(self, ...):
+        for video in videos:
+            tokens_per_frame = self.tokens_per_frame()
+        mm_items = create_data_items(
+            image=image_feature,
+            image_offsets=img_offsets,
+            video=video_feature,
+            video_offsets=video_offsets,
+        )
+```
+
+### Step 3: Config [See `NemotronH_Nano_VL_V2_Config`]
+
+Add `video_pruning_rate` to your model config:
+
+```python
+class MyModelConfig(PretrainedConfig):
+    def __init__(self, ..., video_pruning_rate=0.0, ...):
+        self.video_pruning_rate = video_pruning_rate
+```
+
+## Files
+
+- `evs_core.py`: Core algorithms (retention mask computation, token redistribution)
+- `evs_module.py`: EVS, configs)
+- `evs_processor.py`: EVSProcessor

python/sglang/srt/multimodal/evs/__init__.py

@@ -0,0 +1,11 @@
+"""https://arxiv.org/abs/2510.14624: Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference"""
+
+from .evs_module import EVS, EVSConfig, EVSEmbeddingResult
+from .evs_processor import EVSProcessor
+
+__all__ = [
+    "EVS",
+    "EVSConfig",
+    "EVSEmbeddingResult",
+    "EVSProcessor",
+]

python/sglang/srt/multimodal/evs/evs_core.py

@@ -0,0 +1,178 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# Copyright 2025 SGLang Team
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+# Adapted from https://github.com/vllm-project/vllm/blob/main/vllm/multimodal/evs.py
+
+import torch
+
+
+def compute_retained_tokens_count(
+    tokens_per_frame: int, num_frames: int, q: float
+) -> int:
+    """
+    Compute the number of retained tokens for a given video.
+    Method ensures that we retain all the tokens from the first frame
+    regardless of the pruning rate.
+
+    Args:
+        tokens_per_frame: The number of tokens per frame.
+        num_frames: The total number of frames.
+        q: The pruning rate.
+
+    Returns:
+        The number of retained tokens.
+    """
+    total_tokens = tokens_per_frame * num_frames
+    evs_num_tokens = int(total_tokens * (1 - q))
+    min_num_tokens = tokens_per_frame
+    return max(min_num_tokens, evs_num_tokens)
+
+
+def compute_retention_mask(
+    video_embeds: torch.Tensor,
+    video_size_thw: torch.LongTensor | tuple[int, int, int],
+    spatial_merge_size: int,
+    q: float,
+) -> torch.Tensor:
+    """
+    Computes the retention mask for input video embeddings.
+
+    Args:
+        video_embeds (`torch.Tensor`): The input video embeddings
+            of shape `(T * H * W // spatial_merge_size ^ 2, hidden_size)`
+        video_size_thw (`torch.LongTensor` of shape `(3)`):
+            The temporal, height and width of video.
+        spatial_merge_size: Size reduction for rows & cols dimensions.
+        q: (`float`): Pruning rate factor [0,1)
+
+    Returns:
+        `torch.Tensor`: The retention mask for the video embeddings of
+            `(T * H * W // spatial_merge_size ^ 2)` shape.
+    """
+    T, H, W = map(int, video_size_thw)
+
+    # Use reshape instead of einops to avoid graph breaks
+    video_embeds = video_embeds.reshape(
+        T,
+        H // spatial_merge_size,
+        W // spatial_merge_size,
+        video_embeds.size(-1),
+    )
+    tokens_per_frame = (H // spatial_merge_size) * (W // spatial_merge_size)
+    # Core EVS
+    similarity = torch.nn.functional.cosine_similarity(
+        video_embeds[1:, ...], video_embeds[:-1, ...], dim=-1
+    )
+    dissimilarity = 1 - similarity
+
+    # Always ensure we include all tokens from the first frame
+    dissimilarity = torch.cat(
+        [255 * torch.ones_like(video_embeds[:1, :, :, 0]), dissimilarity], dim=0
+    )
+
+    dissimilarity_flat = dissimilarity.view(-1)
+    order = torch.argsort(dissimilarity_flat, dim=-1, descending=True, stable=True)
+    retain_num_tokens = compute_retained_tokens_count(
+        tokens_per_frame=tokens_per_frame, num_frames=T, q=q
+    )
+    topk_indices = order[:retain_num_tokens]
+
+    retention_mask = torch.zeros_like(dissimilarity_flat, dtype=torch.bool)
+    retention_mask[topk_indices] = True
+    retention_mask = retention_mask.reshape(dissimilarity.size())
+
+    mask = retention_mask.view(-1)  # "T H W -> (T H W)"
+    return mask
+
+
+# ▲ End of VLLM code
+
+
+def tokens_per_frame(
+    *,
+    q: float,
+    num_frames: int,
+    frame_num_tokens: int,
+) -> list[int]:
+    """
+    Before EVS pruning, we want to pre-reduce input_ids to be the same length that will be retained of embeddings due to EVS pruning, so the forward batch metadata will be correct post EVS.
+    We don't know the exact number of tokens per frame after EVS pruning, but we know the *total* number of tokens that will be retained.
+    So, we create a bogus tokens_per_frame list that sums to the total number of tokens that will be retained, and use it for placeholder spans, later to replaced, see `replace_offsets_with_tokens_per_frame` below.
+    """
+    retained = compute_retained_tokens_count(
+        tokens_per_frame=frame_num_tokens, num_frames=num_frames, q=q
+    )
+    base = retained // num_frames
+    rem = retained % num_frames
+    tpf = [base] * (num_frames - 1) + [base + rem]
+    assert sum(tpf) == retained
+    return tpf
+
+
+def replace_offsets_with_tokens_per_frame(
+    *,
+    pre_chunked_input_ids: list[int],
+    num_tokens_per_frame: list[int],
+    frame_offsets_inclusive: list[tuple[int, int]],
+    filler_token_id: int,
+) -> list[int]:
+    """
+    Given a single video, after EVS pruning of redundant tokens, we have a new `num_tokens_per_frame`, therefore the existing input_ids and offsets are stale.
+    We need to replace all stale offsets with new offsets that reflect the new `num_tokens_per_frame`, respectively.
+
+    Returns:
+        Modified input_ids with offsets replaced with new offsets.
+
+    Examples:
+    >>> assert replace_offsets_with_tokens_per_frame(
+    ...     pre_chunked_input_ids=[1, 0, 0, 4, 5, 0, 0, 0, 9, 10, 0, 0, 12, 13],
+    ...     frame_offsets_inclusive=[(1, 2), (5, 7), (10, 11)],
+    ...     num_tokens_per_frame=[1, 4, 2],
+    ...     filler_token_id=0,
+    ... ) ==                      [1, 0, 4, 5, 0, 0, 0, 0, 9, 10, 0, 0, 12, 13]
+
+    >>> assert replace_offsets_with_tokens_per_frame(
+    ...     pre_chunked_input_ids=[1, 0, 0, 4, 5, 9, 10, 0, 0, 0],
+    ...     frame_offsets_inclusive=[(1, 2), (7, 9)],
+    ...     num_tokens_per_frame=[1, 4],
+    ...     filler_token_id=0,
+    ... ) ==                      [1, 0, 4, 5, 9, 10, 0, 0, 0, 0]
+
+    >>> assert replace_offsets_with_tokens_per_frame(
+    ...     pre_chunked_input_ids=[0, 0, 1, 4, 0, 0, 0, 5, 9, 10],
+    ...     frame_offsets_inclusive=[(0, 1), (4, 6)],
+    ...     num_tokens_per_frame=[1, 4],
+    ...     filler_token_id=0,
+    ... ) ==                      [0, 1, 4, 0, 0, 0, 0, 5, 9, 10]
+    """
+    assert isinstance(pre_chunked_input_ids, list)
+    ids = pre_chunked_input_ids
+
+    if len(frame_offsets_inclusive) == 1:
+        """There might be no frame separators, in which case there will be one contiguous span of tokens"""
+        final = ids[0 : frame_offsets_inclusive[0][0]]
+        frames = [filler_token_id] * sum(num_tokens_per_frame)
+        final.extend(frames)
+    else:
+        cursor = 0
+        final = []
+        for (start, end), num_tokens in zip(
+            frame_offsets_inclusive, num_tokens_per_frame, strict=True
+        ):
+            final.extend(ids[cursor:start])
+            final.extend([filler_token_id] * num_tokens)
+            cursor = end + 1
+    final.extend(ids[frame_offsets_inclusive[-1][1] + 1 :])
+    return final

python/sglang/srt/multimodal/evs/evs_module.py

@@ -0,0 +1,201 @@
+# Copyright 2025 SGLang Team
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+
+import dataclasses
+import typing
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+
+import torch
+from transformers import PretrainedConfig
+
+from sglang.srt.managers.schedule_batch import MultimodalDataItem
+from sglang.srt.mem_cache.multimodal_cache import EmbeddingResult
+from sglang.srt.multimodal.processors.base_processor import BaseMultimodalProcessor
+from sglang.utils import logger
+
+from .evs_core import compute_retention_mask, replace_offsets_with_tokens_per_frame
+
+
+@dataclasses.dataclass(kw_only=True)
+class EVSDataItem(MultimodalDataItem):
+    thw_grids: list[tuple[int, int, int]]
+
+
+@dataclasses.dataclass(kw_only=True)
+class VideoEVSDataItem(EVSDataItem):
+    pre_chunked_input_ids: torch.Tensor
+
+    def __post_init__(self):
+        assert self.is_video()
+
+
+@dataclass(kw_only=True)
+class EVSEmbeddingResult(EmbeddingResult):
+    """
+    Embedding result that includes per-frame token counts after EVS pruning.
+
+    After pruning, each frame retains a different number of tokens based on its
+    dissimilarity to the previous frame. This metadata is needed downstream to
+    adjust the input_ids placeholder spans to match the actual embedding sizes.
+
+    Attributes:
+        embedding: The pruned video embeddings tensor.
+        num_tokens_per_frame: Actual retained token count for each frame.
+            For example, [256, 180, 195, 256] means frame 0 kept all 256 tokens
+            (first frame is never pruned), while frames 1-2 were pruned.
+    """
+
+    num_tokens_per_frame: list[int]
+
+    def redistribute_pruned_frames_placeholders(
+        self,
+        input_ids: torch.Tensor,
+        offsets: list[tuple[int, int]],
+        *,
+        item: VideoEVSDataItem,
+        extend_prefix_len: int,
+        extend_seq_len: int,
+    ) -> tuple[torch.Tensor, list[tuple[int, int]]]:
+        assert len(input_ids) == extend_seq_len
+        assert isinstance(
+            item, VideoEVSDataItem
+        ), f"Expected VideoEVSDataItem, got {type(item)}"
+        pre_chunked_input_ids = item.pre_chunked_input_ids
+        filler_token_id = item.pad_value
+        input_ids_list = replace_offsets_with_tokens_per_frame(
+            pre_chunked_input_ids=pre_chunked_input_ids,
+            num_tokens_per_frame=self.num_tokens_per_frame,
+            frame_offsets_inclusive=offsets,
+            filler_token_id=filler_token_id,
+        )
+        input_ids = torch.tensor(
+            input_ids_list, dtype=input_ids.dtype, device=input_ids.device
+        )
+        offsets = BaseMultimodalProcessor.get_mm_items_offset(
+            input_ids, filler_token_id
+        )
+        input_ids = input_ids[extend_prefix_len : extend_prefix_len + extend_seq_len]
+        assert (
+            len(input_ids) == extend_seq_len
+        ), f"Input ids length changed after redistribution, got {len(input_ids)} != {extend_seq_len}"
+        return input_ids, offsets
+
+
+@dataclass(frozen=True, kw_only=True)
+class EVSConfig:
+    video_pruning_rate: float
+    spatial_merge_size: int = 1
+
+    def __post_init__(self):
+        assert (
+            self.video_pruning_rate >= 0.0 and self.video_pruning_rate < 1.0
+        ), f"Video pruning rate must be between 0.0 and 1.0, got {self.video_pruning_rate=}"
+
+
+class EVS(torch.nn.Module, ABC):
+    """
+    Base class for video models that support EVS pruning.
+
+    Subclass this alongside your model class and implement the static `create_evs_config`.
+    On initialization, if video_pruning_rate > 0, this mixin replaces the model's
+    get_video_feature() method with a wrapper that applies EVS pruning.
+
+    Example: See `NemotronH_Nano_VL_V2`
+    """
+
+    @staticmethod
+    @abstractmethod
+    def create_evs_config(config: PretrainedConfig) -> EVSConfig:
+        """Extract EVS parameters from model config. Must be implemented by subclass."""
+        raise NotImplementedError
+
+    @abstractmethod
+    def get_video_feature(self, items: list[MultimodalDataItem]) -> torch.Tensor:
+        """Extract EVS parameters from model config. Must be implemented by subclass."""
+        raise NotImplementedError
+
+    def __init__(
+        self,
+        config: PretrainedConfig,
+        *args: typing.Any,
+        **kwargs: typing.Any,
+    ) -> None:
+        super().__init__()
+        model_name = self.__class__.__name__
+        self.original_get_video_feature = self.get_video_feature
+        self.evs_config = self.create_evs_config(config)
+        self.evs_enabled = self.evs_config.video_pruning_rate > 0.0
+        if self.evs_enabled:
+            logger.info(f"[EVS] enabled for {model_name} [{self.evs_config}]")
+            self.get_video_feature = self.evs_video
+        else:
+            logger.info(
+                f"[EVS] requested on model {model_name} but is disabled for pruning_rate == 0.0."
+            )
+
+    def evs_video(self, items: list[MultimodalDataItem]) -> EVSEmbeddingResult:
+        """
+        Apply EVS pruning to video embeddings.
+
+        Args:
+            items: List containing a single VideoEVSDataItem with video features.
+
+        Returns:
+            EVSEmbeddingResult with pruned embeddings and actual token counts per frame.
+        """
+        logger.debug(
+            f"[EVS] beginning for model {self.__class__.__name__} [evs_config={self.evs_config=}]"
+        )
+        assert len(items) == 1, f"Expected 1 item, got {len(items)}"
+        item = items[0]
+        assert isinstance(
+            item, VideoEVSDataItem
+        ), f"Expected VideoEVSDataItem with modality VIDEO, got {item}"
+
+        q = self.evs_config.video_pruning_rate
+        merge = self.evs_config.spatial_merge_size
+        videos_features = self.original_get_video_feature([item])
+        if videos_features.ndim == 3:
+            videos_features = videos_features.flatten(0, 1)
+        assert videos_features.ndim == 2, videos_features.ndim
+
+        final_embeddings: list[torch.Tensor] = []
+        num_tokens_per_frame: list[int] = []
+
+        sizes = [(t * h * w // merge**2) for t, h, w in item.thw_grids]
+        for single_video, video_size_thw in zip(
+            videos_features.split(sizes),
+            item.thw_grids,
+            strict=True,
+        ):
+            retention_mask = compute_retention_mask(
+                single_video,
+                video_size_thw=video_size_thw,
+                spatial_merge_size=merge,
+                q=q,
+            )
+            preserved = single_video[retention_mask]
+            final_embeddings.append(preserved)
+            num_frames = video_size_thw[0]
+            tokens_per_frame = (
+                retention_mask.reshape(num_frames, -1).sum(dim=-1).tolist()
+            )
+            num_tokens_per_frame.extend(tokens_per_frame)
+        final_embeddings_tensor = torch.cat(final_embeddings)
+        return EVSEmbeddingResult(
+            embedding=final_embeddings_tensor,
+            num_tokens_per_frame=num_tokens_per_frame,
+        )

python/sglang/srt/multimodal/evs/evs_processor.py

@@ -0,0 +1,132 @@
+# Copyright 2025 SGLang Team
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+
+import torch
+from transformers import PretrainedConfig
+
+from sglang.srt.managers.schedule_batch import Modality, MultimodalDataItem
+from sglang.utils import logger
+
+from .evs_core import tokens_per_frame
+from .evs_module import EVS, EVSConfig, EVSDataItem, VideoEVSDataItem
+
+
+def _non_evs_data_items(
+    *,
+    image: torch.Tensor | None,
+    image_offsets: list[tuple[int, int]],
+    video: torch.Tensor | None,
+    video_offsets: list[tuple[int, int]],
+    input_ids_list: list[int],
+):
+    items: list[MultimodalDataItem] = []
+    if image is not None:
+        item = MultimodalDataItem(
+            modality=Modality.IMAGE, feature=image, offsets=image_offsets
+        )
+        items.append(item)
+    if video is not None:
+        item = MultimodalDataItem(
+            modality=Modality.VIDEO, feature=video, offsets=video_offsets
+        )
+        items.append(item)
+    return items
+
+
+class EVSProcessor:
+    """
+    This processor handles prompt construction with the correct number of
+    placeholder tokens per frame. When EVS is active, it allocates fewer
+    placeholders based on the pruning rate. When inactive, it uses the full
+    token count.
+    """
+
+    def __init__(
+        self,
+        hf_config: PretrainedConfig,
+        config_to_evs_model: dict[type[PretrainedConfig], type[EVS]],
+    ):
+        assert len(config_to_evs_model) > 0
+        assert all(issubclass(model, EVS) for model in config_to_evs_model.values())
+
+        self.evs_config: EVSConfig | None = None
+
+        config_name = hf_config.__class__.__name__
+        evs_model = config_to_evs_model.get(hf_config.__class__)
+        if evs_model is None:
+            logger.info(
+                f"[EVS] no model matches {config_name} in {config_to_evs_model}"
+            )
+            return
+        evs_config = evs_model.create_evs_config(hf_config)
+        logger.info(
+            f"""[EVS] {evs_config} {'enabled' if evs_config.video_pruning_rate > 0.0 else 'disabled'} for model={evs_model.__name__}; model_config={config_name}"""
+        )
+        if evs_config.video_pruning_rate > 0.0:
+            self.evs_config = evs_config
+
+    def static_size_data_items(
+        self, *, frames_per_video: list[int], num_images: int, rows: int, cols: int
+    ):
+        """helper function to create data items for models with static image and video tokens per frame"""
+
+        frame_num_tokens = rows * cols
+
+        if self.evs_config is None:
+            tpf = [[frame_num_tokens] * num_frames for num_frames in frames_per_video]
+            return _non_evs_data_items, tpf
+
+        def create_evs_data_items(
+            *,
+            input_ids_list: list[int],
+            image: torch.Tensor | None,
+            image_offsets: list[tuple[int, int]],
+            video: torch.Tensor | None,
+            video_offsets: list[tuple[int, int]],
+        ) -> list[MultimodalDataItem]:
+            items = []
+            if image is not None:
+                image_thw_grids = [(1, rows, cols)] * num_images
+                item = EVSDataItem(
+                    modality=Modality.IMAGE,
+                    feature=image,
+                    offsets=image_offsets,
+                    thw_grids=image_thw_grids,
+                )
+                items.append(item)
+            if video is not None:
+                video_thw_grids = [
+                    (num_frames, rows, cols) for num_frames in frames_per_video
+                ]
+                item = VideoEVSDataItem(
+                    modality=Modality.VIDEO,
+                    feature=video,
+                    offsets=video_offsets,
+                    thw_grids=video_thw_grids,
+                    pre_chunked_input_ids=input_ids_list,
+                )
+                items.append(item)
+            return items
+
+        tpf = [
+            tokens_per_frame(
+                q=self.evs_config.video_pruning_rate,
+                num_frames=num_frames,
+                frame_num_tokens=frame_num_tokens,
+            )
+            for num_frames in frames_per_video
+        ]
+
+        return create_evs_data_items, tpf