wehub-resource-sync
e06fe8e8c6
Secret Leaks / trufflehog (push) Failing after 1s
Build documentation / build (push) Failing after 1s
Build documentation / build_other_lang (push) Failing after 0s
CodeQL Security Analysis / CodeQL Analysis (push) Failing after 0s
PR CI / pr-ci (push) Failing after 1s
Slow tests on important models (on Push - A10) / Get all modified files (push) Failing after 1s
Slow tests on important models (on Push - A10) / Model CI (push) Has been skipped
Self-hosted runner (benchmark) / Benchmark (aws-g5-4xlarge-cache) (push) Has been cancelled
New model PR merged notification / Notify new model (push) Has been cancelled
Update Transformers metadata / build_and_package (push) Has been cancelled
65 lines
2.8 KiB
Markdown
65 lines
2.8 KiB
Markdown
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
*This model was published in HF papers on 2022-07-21 and contributed to Hugging Face Transformers on 2023-07-10.*
|
|
|
|
# MRA
|
|
|
|
|
|
## Overview
|
|
|
|
The MRA model was proposed in [Multi Resolution Analysis (MRA) for Approximate Self-Attention](https://huggingface.co/papers/2207.10284) by Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Transformers have emerged as a preferred model for many tasks in natural language processing and vision. Recent efforts on training and deploying Transformers more efficiently have identified many strategies to approximate the self-attention matrix, a key module in a Transformer architecture. Effective ideas include various prespecified sparsity patterns, low-rank basis expansions and combinations thereof. In this paper, we revisit classical Multiresolution Analysis (MRA) concepts such as Wavelets, whose potential value in this setting remains underexplored thus far. We show that simple approximations based on empirical feedback and design choices informed by modern hardware and implementation challenges, eventually yield a MRA-based approach for self-attention with an excellent performance profile across most criteria of interest. We undertake an extensive set of experiments and demonstrate that this multi-resolution scheme outperforms most efficient self-attention proposals and is favorable for both short and long sequences. Code is available at https://github.com/mlpen/mra-attention.*
|
|
|
|
This model was contributed by [novice03](https://huggingface.co/novice03).
|
|
The original code can be found [here](https://github.com/mlpen/mra-attention).
|
|
|
|
## MraConfig
|
|
|
|
[[autodoc]] MraConfig
|
|
|
|
## MraModel
|
|
|
|
[[autodoc]] MraModel
|
|
- forward
|
|
|
|
## MraForMaskedLM
|
|
|
|
[[autodoc]] MraForMaskedLM
|
|
- forward
|
|
|
|
## MraForSequenceClassification
|
|
|
|
[[autodoc]] MraForSequenceClassification
|
|
- forward
|
|
|
|
## MraForMultipleChoice
|
|
|
|
[[autodoc]] MraForMultipleChoice
|
|
- forward
|
|
|
|
## MraForTokenClassification
|
|
|
|
[[autodoc]] MraForTokenClassification
|
|
- forward
|
|
|
|
## MraForQuestionAnswering
|
|
|
|
[[autodoc]] MraForQuestionAnswering
|
|
- forward
|