wehub-resource-sync
23 lines
789 B
Markdown
23 lines
789 B
Markdown
# Linformer: Self-Attention with Linear Complexity (Wang et al., 2020)
|
|
|
|
This example contains code to train Linformer models as described in our paper
|
|
[Linformer: Self-Attention with Linear Complexity](https://arxiv.org/abs/2006.04768).
|
|
|
|
## Training a new Linformer RoBERTa model
|
|
|
|
You can mostly follow the [RoBERTa pretraining README](/examples/roberta/README.pretraining.md),
|
|
updating your training command with `--user-dir examples/linformer/linformer_src --arch linformer_roberta_base`.
|
|
|
|
## Citation
|
|
|
|
If you use our work, please cite:
|
|
|
|
```bibtex
|
|
@article{wang2020linformer,
|
|
title={Linformer: Self-Attention with Linear Complexity},
|
|
author={Wang, Sinong and Li, Belinda and Khabsa, Madian and Fang, Han and Ma, Hao},
|
|
journal={arXiv preprint arXiv:2006.04768},
|
|
year={2020}
|
|
}
|
|
```
|