nvidia--tensorrt/plugin/multiscaleDeformableAttnPlugin/README.md

multiscaleDeformableAttn Plugin

Table Of Contents

• Description
  • Structure
• Parameters
• Additional resources
• License
• Changelog
• Known issues

Description

NOTE: Version 1 of this plugin (using IPluginV2DynamicExt interface) is deprecated since TensorRT 10.11. Version 2 (using IPluginV3 interface) is the recommended replacement.

The multiscaleDeformableAttnPlugin is used to perform attention computation over a small set of key sampling points around a reference point rather than looking over all possible spatial locations. It makes use of multiscale feature maps to effectively represent objects at different scales. It helps to achieve faster convergence and better performance on small objects.

Structure

The multiscaleDeformableAttnPlugin takes 5 inputs in the following order : value, spatial_shapes, level_start_index, sampling_locations, and atttention_weights.

value The input feature maps from different scales concatenated to provide the input feature vector. The shape of this tensor is [N, S, M, D] where N is batch size, S is the length of the feature maps, M is the number of attentions heads, D is hidden_dim/num_heads.

spatial_shapes The shape of each feature map. The shape of this tensor is [L, 2] where L is the number of feature maps.

level_start_index This tensor is used to find the sampling locations for different feature levels as the input feature tensors are flattened. The shape of this tensor is [L,].

sampling_locations This tensor acts as a pre-filter for prominent key elements out of all the feature map pixels. The shape of this tensor is [N, Lq, M, L, P, 2] where P is the number of points, Lq is the length of feature maps(encoder)/length of queries(decoder).

attention_weights This tensor consists of the attention weights whose shape is [N, Lq, M, L, P].

The multiscaleDeformableAttnPlugin generates the attention output of shape [N, Lq, M, D].

Parameters

multiscaleDeformableAttnPlugin has plugin creator class multiscaleDeformableAttnPluginCreator and plugin class multiscaleDeformableAttnPlugin.

The plugin does not require any parameters to be built and used.

Additional resources

The following resources provide a deeper understanding of the multiscaleDeformableAttnPlugin plugin:

Networks:

• Deformable DETR

License

For terms and conditions for use, reproduction, and distribution, see the TensorRT Software License Agreement documentation.

Changelog

Apr 2025 Added version 2 of the plugin that uses the IPluginV3 interface. The version 1 (using IPluginV2DynamicExt interface) is now deprecated. The version 2 mirrors version 1 in IO and attributes.

Feb 2022 This is the first release of this README.md file.

Known issues

There are no known issues in this plugin.