wehub-resource-sync
121 lines
6.8 KiB
Markdown
121 lines
6.8 KiB
Markdown
# E5 Text Embeddings
|
|
|
|
[Multilingual E5 Text Embeddings: A Technical Report](https://arxiv.org/pdf/2402.05672).
|
|
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024
|
|
|
|
[Improving Text Embeddings with Large Language Models](https://arxiv.org/pdf/2401.00368.pdf).
|
|
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024
|
|
|
|
[Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://arxiv.org/pdf/2212.03533.pdf).
|
|
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022
|
|
|
|
## LLM based Models
|
|
|
|
| | BEIR | # of layers | embedding dimension | Huggingface |
|
|
|------------------------|------|:-----------:|:-------------------:|-----------------------------------------------------------------------------------------|
|
|
| E5-mistral-7b-instruct | 56.9 | 32 | 4096 | [intfloat/e5-mistral-7b-instruct](https://huggingface.co/intfloat/e5-mistral-7b-instruct)|
|
|
|
|
## English Pre-trained Models
|
|
|
|
| | BEIR | # of layers | embedding dimension | Huggingface |
|
|
|------------------------|------|:-----------:|:-------------------:|-----------------------------------------------------------------------------------------|
|
|
| E5-small-v2 | 49.0 | 12 | 384 | [intfloat/e5-small-v2](https://huggingface.co/intfloat/e5-small-v2) |
|
|
| E5-base-v2 | 50.3 | 12 | 768 | [intfloat/e5-base-v2](https://huggingface.co/intfloat/e5-base-v2) |
|
|
| E5-large-v2 | 50.6 | 24 | 1024 | [intfloat/e5-large-v2](https://huggingface.co/intfloat/e5-large-v2) |
|
|
| | | | | |
|
|
| E5-small | 46.0 | 12 | 384 | [intfloat/e5-small](https://huggingface.co/intfloat/e5-small) |
|
|
| E5-base | 48.8 | 12 | 768 | [intfloat/e5-base](https://huggingface.co/intfloat/e5-base) |
|
|
| E5-large | 50.0 | 24 | 1024 | [intfloat/e5-large](https://huggingface.co/intfloat/e5-large) |
|
|
| | | | | |
|
|
| E5-small-unsupervised | 40.8 | 12 | 384 | [intfloat/e5-small-unsupervised](https://huggingface.co/intfloat/e5-small-unsupervised) |
|
|
| E5-base-unsupervised | 42.9 | 12 | 768 | [intfloat/e5-base-unsupervised](https://huggingface.co/intfloat/e5-base-unsupervised) |
|
|
| E5-large-unsupervised | 44.2 | 24 | 1024 | [intfloat/e5-large-unsupervised](https://huggingface.co/intfloat/e5-large-unsupervised) |
|
|
|
|
The models with `-unsupervised` suffix only pre-trains on unlabeled datasets.
|
|
|
|
## Multilingual Pre-trained Models
|
|
|
|
| | BEIR | # of layers | embedding dimension | Huggingface |
|
|
|--------------------------------|------|:-----------:|:-------------------:|-----------------------------------------------------------------------------------------------------------|
|
|
| multilingual-e5-small | 46.6 | 12 | 384 | [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) |
|
|
| multilingual-e5-base | 48.9 | 12 | 768 | [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) |
|
|
| multilingual-e5-large | 51.4 | 24 | 1024 | [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) |
|
|
| multilingual-e5-large-instruct | 52.5 | 24 | 1024 | [intfloat/multilingual-e5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct) |
|
|
|
|
## Install Python Package Requirements
|
|
|
|
```shell
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
For `e5-mistral-7b-instruct`, it would require `transformers>=4.34` to load Mistral model.
|
|
|
|
## Evaluate on the [BEIR Benchmark](https://arxiv.org/abs/2104.08663)
|
|
|
|
After installing the required python packages,
|
|
run the following command on GPU machines:
|
|
|
|
```shell
|
|
bash scripts/eval_mteb_beir.sh intfloat/e5-small-v2
|
|
```
|
|
|
|
By default,
|
|
the evaluation script will use all the available GPUs.
|
|
|
|
Caution: it could take quite a long time (~10 hours) due to corpus encoding.
|
|
For `intfloat/e5-mistral-7b-instruct`, it could take even longer (several days).
|
|
|
|
## Evaluate on the [MTEB Benchmark](https://arxiv.org/abs/2210.07316)
|
|
|
|
Run the following command:
|
|
|
|
```shell
|
|
bash scripts/eval_mteb_except_retrieval.sh intfloat/e5-small-v2
|
|
```
|
|
|
|
For multilingual models, simply add a `--multilingual` suffix:
|
|
|
|
```shell
|
|
bash scripts/eval_mteb_except_retrieval.sh intfloat/multilingual-e5-base --multilingual
|
|
```
|
|
|
|
## Other Resources
|
|
|
|
The data for our proposed synthetic task _personalized passkey retrieval_ is available at [https://huggingface.co/datasets/intfloat/personalized_passkey_retrieval](https://huggingface.co/datasets/intfloat/personalized_passkey_retrieval).
|
|
|
|
## Troubleshooting
|
|
|
|
If you encounter OOM error, please try to reduce the batch size.
|
|
|
|
## Citation
|
|
|
|
If you find our paper or models helpful, please consider cite as follows:
|
|
|
|
```
|
|
@article{wang2024multilingual,
|
|
title={Multilingual E5 Text Embeddings: A Technical Report},
|
|
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
|
|
journal={arXiv preprint arXiv:2402.05672},
|
|
year={2024}
|
|
}
|
|
|
|
@article{wang2023improving,
|
|
title={Improving Text Embeddings with Large Language Models},
|
|
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
|
|
journal={arXiv preprint arXiv:2401.00368},
|
|
year={2023}
|
|
}
|
|
|
|
@article{wang2022text,
|
|
title={Text Embeddings by Weakly-Supervised Contrastive Pre-training},
|
|
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu},
|
|
journal={arXiv preprint arXiv:2212.03533},
|
|
year={2022}
|
|
}
|
|
```
|
|
|
|
## License
|
|
|
|
This project is licensed under the license found in the LICENSE file in the root directory of this source tree.
|
|
[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct)
|