chore: import upstream snapshot with attribution

docs/source/API/evaluation/airbench.rst

@@ -0,0 +1,42 @@
+AIR-Bench
+=========
+
+`AIR-Bench <https://github.com/AIR-Bench/AIR-Bench>`_ (Automated heterogeneous Information Retrieval Benchmark) is a dynamic (actively being updated) benchmark for information retrieval.
+Now the benchmark contains two versions. Notice that the testing data is generated by LLMs with out human intervention. 
+This helps the evaluation of new domains easier and faster to be updated. It also makes it impossible for any models to have the test data covered in their training sets.
+
+You can evaluate model's performance on AIR-Bench by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/air_bench/eval_air_bench.sh
+    ./examples/evaluation/air_bench/eval_air_bench.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.air_bench \
+    --benchmark_version AIR-Bench_24.05 \
+    --task_types qa long-doc \
+    --domains arxiv \
+    --languages en \
+    --splits dev test \
+    --output_dir ./air_bench/search_results \
+    --search_top_k 1000 \
+    --rerank_top_k 100 \
+    --cache_dir /root/.cache/huggingface/hub \
+    --overwrite False \
+    --embedder_name_or_path BAAI/bge-m3 \
+    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
+    --devices cuda:0 cuda:1 \
+    --model_cache_dir /root/.cache/huggingface/hub \
+    --reranker_max_length 1024
+
+change the embedder, reranker, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   airbench/arguments
+   airbench/runner

docs/source/API/evaluation/airbench/arguments.rst

@@ -0,0 +1,4 @@
+arguments
+=========
+
+.. autoclass:: FlagEmbedding.evaluation.air_bench.AIRBenchEvalModelArgs

docs/source/API/evaluation/airbench/runner.rst

@@ -0,0 +1,4 @@
+runner
+======
+
+.. autoclass:: FlagEmbedding.evaluation.air_bench.AIRBenchEvalRunner

docs/source/API/evaluation/beir.rst

@@ -0,0 +1,47 @@
+BEIR
+====
+
+`BEIR <https://github.com/beir-cellar/beir>`_ (Benchmarking-IR) is a heterogeneous evaluation benchmark for information retrieval. 
+It is designed for evaluating the performance of NLP-based retrieval models and widely used by research of modern embedding models.
+
+You can evaluate model's performance on the BEIR benchmark by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/beir/eval_beir.sh
+    ./examples/evaluation/beir/eval_beir.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.beir \
+    --eval_name beir \
+    --dataset_dir ./beir/data \
+    --dataset_names fiqa arguana cqadupstack \
+    --splits test dev \
+    --corpus_embd_save_dir ./beir/corpus_embd \
+    --output_dir ./beir/search_results \
+    --search_top_k 1000 \
+    --rerank_top_k 100 \
+    --cache_path /root/.cache/huggingface/hub \
+    --overwrite False \
+    --k_values 10 100 \
+    --eval_output_method markdown \
+    --eval_output_path ./beir/beir_eval_results.md \
+    --eval_metrics ndcg_at_10 recall_at_100 \
+    --ignore_identical_ids True \
+    --embedder_name_or_path BAAI/bge-large-en-v1.5 \
+    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
+    --devices cuda:0 cuda:1 \
+    --reranker_max_length 1024 \
+
+change the embedder, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   beir/arguments
+   beir/data_loader
+   beir/evaluator
+   beir/runner

docs/source/API/evaluation/beir/arguments.rst

@@ -0,0 +1,4 @@
+arguments
+=========
+
+.. autoclass:: FlagEmbedding.evaluation.beir.arguments.BEIREvalArgs

docs/source/API/evaluation/beir/data_loader.rst

@@ -0,0 +1,4 @@
+data loader
+===========
+
+.. autoclass:: FlagEmbedding.evaluation.beir.data_loader.BEIREvalDataLoader

docs/source/API/evaluation/beir/evaluator.rst

@@ -0,0 +1,4 @@
+evaluator
+=========
+
+.. autoclass:: FlagEmbedding.evaluation.beir.evaluator.BEIREvaluator

docs/source/API/evaluation/beir/runner.rst

@@ -0,0 +1,4 @@
+runner
+======
+
+.. autoclass:: FlagEmbedding.evaluation.beir.BEIREvalRunner

docs/source/API/evaluation/miracl.rst

@@ -0,0 +1,48 @@
+MIRACL
+======
+
+`MIRACL <https://project-miracl.github.io/>`_ (Multilingual Information Retrieval Across a Continuum of Languages) 
+is an WSDM 2023 Cup challenge that focuses on search across 18 different languages.
+They release a multilingual retrieval dataset containing the train and dev set for 16 "known languages" and only dev set for 2 "surprise languages".
+The topics are generated by native speakers of each language, who also label the relevance between the topics and a given document list.
+You can found the `dataset <https://huggingface.co/datasets/miracl/miracl-corpus>`_ on HuggingFace.
+
+You can evaluate model's performance on MIRACL simply by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/miracl/eval_miracl.sh
+    ./examples/evaluation/miracl/eval_miracl.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.miracl \
+    --eval_name miracl \
+    --dataset_dir ./miracl/data \
+    --dataset_names bn hi sw te th yo \
+    --splits dev \
+    --corpus_embd_save_dir ./miracl/corpus_embd \
+    --output_dir ./miracl/search_results \
+    --search_top_k 1000 \
+    --rerank_top_k 100 \
+    --cache_path /root/.cache/huggingface/hub \
+    --overwrite False \
+    --k_values 10 100 \
+    --eval_output_method markdown \
+    --eval_output_path ./miracl/miracl_eval_results.md \
+    --eval_metrics ndcg_at_10 recall_at_100 \
+    --embedder_name_or_path BAAI/bge-m3 \
+    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
+    --devices cuda:0 cuda:1 \
+    --cache_dir /root/.cache/huggingface/hub \
+    --reranker_max_length 1024
+
+change the embedder, reranker, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   miracl/data_loader
+   miracl/runner

docs/source/API/evaluation/miracl/data_loader.rst

@@ -0,0 +1,13 @@
+data_loader
+===========
+
+.. autoclass:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader
+
+Methods
+-------
+
+.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader.available_dataset_names
+.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader.available_splits
+.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader._load_remote_corpus
+.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader._load_remote_qrels
+.. automethod:: FlagEmbedding.evaluation.miracl.MIRACLEvalDataLoader._load_remote_queries

docs/source/API/evaluation/miracl/runner.rst

@@ -0,0 +1,5 @@
+runner
+======
+
+.. autoclass:: FlagEmbedding.evaluation.miracl.MIRACLEvalRunner
+    :members:

docs/source/API/evaluation/mkqa.rst

@@ -0,0 +1,89 @@
+MKQA
+====
+
+`MKQA <https://github.com/apple/ml-mkqa>`_ is an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages.
+The queries are sampled from the [Google Natural Questions Dataset](https://github.com/google-research-datasets/natural-questions). 
+
+Each example in the dataset has the following structure:
+
+.. code:: bash
+
+    {
+        'example_id': 563260143484355911,
+        'queries': {
+            'en': "who sings i hear you knocking but you can't come in",
+            'ru': "кто поет i hear you knocking but you can't come in",
+            'ja': '「 I hear you knocking」は誰が歌っていますか',
+            'zh_cn': "《i hear you knocking but you can't come in》是谁演唱的",
+            ...
+        },
+        'query': "who sings i hear you knocking but you can't come in",
+        'answers': {
+            'en': [{
+                'type': 'entity',
+                'entity': 'Q545186',
+                'text': 'Dave Edmunds',
+                'aliases': [],
+            }],
+            'ru': [{
+                'type': 'entity',
+                'entity': 'Q545186',
+                'text': 'Эдмундс, Дэйв',
+                'aliases': ['Эдмундс', 'Дэйв Эдмундс', 'Эдмундс Дэйв', 'Dave Edmunds'],
+            }],
+            'ja': [{
+                'type': 'entity',
+                'entity': 'Q545186',
+                'text': 'デイヴ・エドモンズ',
+                'aliases': ['デーブ・エドモンズ', 'デイブ・エドモンズ'],
+            }],
+            'zh_cn': [{
+                'type': 'entity', 
+                'text': '戴维·埃德蒙兹 ', 
+                'entity': 'Q545186',
+            }],
+            ...
+        },
+    }
+
+
+You can evaluate model's performance on MKQA simply by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/mkqa/eval_mkqa.sh
+    ./examples/evaluation/mkqa/eval_mkqa.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.mkqa \
+    --eval_name mkqa \
+    --dataset_dir ./mkqa/data \
+    --dataset_names en zh_cn \
+    --splits test \
+    --corpus_embd_save_dir ./mkqa/corpus_embd \
+    --output_dir ./mkqa/search_results \
+    --search_top_k 1000 \
+    --rerank_top_k 100 \
+    --cache_path /root/.cache/huggingface/hub \
+    --overwrite False \
+    --k_values 20 \
+    --eval_output_method markdown \
+    --eval_output_path ./mkqa/mkqa_eval_results.md \
+    --eval_metrics qa_recall_at_20 \
+    --embedder_name_or_path BAAI/bge-m3 \
+    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
+    --devices cuda:0 cuda:1 \
+    --cache_dir /root/.cache/huggingface/hub \
+    --reranker_max_length 1024
+
+change the embedder, reranker, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   mkqa/data_loader
+   mkqa/evaluator
+   mkqa/runner

docs/source/API/evaluation/mkqa/data_loader.rst

@@ -0,0 +1,15 @@
+data_loader
+===========
+
+.. autoclass:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader
+
+Methods
+-------
+
+.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader.available_dataset_names
+.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader.available_splits
+.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader.load_corpus
+.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_local_qrels
+.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_remote_corpus
+.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_remote_qrels
+.. automethod:: FlagEmbedding.evaluation.mkqa.MKQAEvalDataLoader._load_remote_queries

docs/source/API/evaluation/mkqa/evaluator.rst

@@ -0,0 +1,5 @@
+evaluator
+=========
+
+.. autoclass:: FlagEmbedding.evaluation.mkqa.MKQAEvaluator
+    :members:

docs/source/API/evaluation/mkqa/runner.rst

@@ -0,0 +1,4 @@
+runner
+======
+.. autoclass:: FlagEmbedding.evaluation.mkqa.MKQAEvalRunner
+    :members:

docs/source/API/evaluation/mldr.rst

@@ -0,0 +1,95 @@
+MLDR
+====
+
+`MLDR <https://huggingface.co/datasets/Shitao/MLDR>`_ is a Multilingual Long-Document Retrieval dataset built on Wikipeida, Wudao and mC4, covering 13 typologically diverse languages. 
+Specifically, we sample lengthy articles from Wikipedia, Wudao and mC4 datasets and randomly choose paragraphs from them. 
+Then we use GPT-3.5 to generate questions based on these paragraphs. 
+The generated question and the sampled article constitute a new text pair to the dataset.
+
+An example of ``train`` set looks like:
+
+.. code:: bash
+
+    {
+        'query_id': 'q-zh-<...>', 
+        'query': '...', 
+        'positive_passages': [
+            {
+                'docid': 'doc-zh-<...>',
+                'text': '...'
+            }
+        ],
+        'negative_passages': [
+            {
+                'docid': 'doc-zh-<...>',
+                'text': '...'
+            },
+            ...
+        ]
+    }
+
+An example of ``dev`` and ``test`` set looks like:
+
+.. code:: bash
+
+    {
+        'query_id': 'q-zh-<...>', 
+        'query': '...', 
+        'positive_passages': [
+            {
+                'docid': 'doc-zh-<...>',
+                'text': '...'
+            }
+        ],
+        'negative_passages': []
+    }
+
+An example of ``corpus`` looks like:
+
+.. code:: bash
+
+    {
+        'docid': 'doc-zh-<...>', 
+        'text': '...'
+    }
+
+You can evaluate model's performance on MLDR simply by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/mldr/eval_mldr.sh
+    ./examples/evaluation/mldr/eval_mldr.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.mldr \
+    --eval_name mldr \
+    --dataset_dir ./mldr/data \
+    --dataset_names hi \
+    --splits test \
+    --corpus_embd_save_dir ./mldr/corpus_embd \
+    --output_dir ./mldr/search_results \
+    --search_top_k 1000 \
+    --rerank_top_k 100 \
+    --cache_path /root/.cache/huggingface/hub \
+    --overwrite False \
+    --k_values 10 100 \
+    --eval_output_method markdown \
+    --eval_output_path ./mldr/mldr_eval_results.md \
+    --eval_metrics ndcg_at_10 \
+    --embedder_name_or_path BAAI/bge-m3 \
+    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
+    --devices cuda:0 cuda:1 \
+    --cache_dir /root/.cache/huggingface/hub \
+    --embedder_passage_max_length 8192 \
+    --reranker_max_length 8192
+
+change the args of embedder, reranker, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   mldr/data_loader
+   mldr/runner

docs/source/API/evaluation/mldr/data_loader.rst

@@ -0,0 +1,13 @@
+data_loader
+===========
+
+.. autoclass:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader
+
+Methods
+-------
+
+.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader.available_dataset_names
+.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader.available_splits
+.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader._load_remote_corpus
+.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader._load_remote_qrels
+.. automethod:: FlagEmbedding.evaluation.mldr.MLDREvalDataLoader._load_remote_queries

docs/source/API/evaluation/mldr/runner.rst

@@ -0,0 +1,5 @@
+runner
+======
+
+.. autoclass:: FlagEmbedding.evaluation.mldr.MLDREvalRunner
+    :members:

docs/source/API/evaluation/msmarco.rst

@@ -0,0 +1,46 @@
+MSMARCO
+=======
+
+`MS Marco <https://microsoft.github.io/msmarco/>`_ (Microsoft MAchine Reading Comprehension) is a large scale real-world reading comprehension dataset.
+It is widely used in information retrieval, question answering, and natural language processing research.
+
+
+You can evaluate model's performance on MS MARCO simply by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/msmarco/eval_msmarco.sh
+    ./examples/evaluation/msmarco/eval_msmarco.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.msmarco \
+    --eval_name msmarco \
+    --dataset_dir ./msmarco/data \
+    --dataset_names passage \
+    --splits dev \
+    --corpus_embd_save_dir ./msmarco/corpus_embd \
+    --output_dir ./msmarco/search_results \
+    --search_top_k 1000 \
+    --rerank_top_k 100 \
+    --cache_path /root/.cache/huggingface/hub \
+    --overwrite True \
+    --k_values 10 100 \
+    --eval_output_method markdown \
+    --eval_output_path ./msmarco/msmarco_eval_results.md \
+    --eval_metrics ndcg_at_10 recall_at_100 \
+    --embedder_name_or_path BAAI/bge-large-en-v1.5 \
+    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
+    --devices cuda:0 cuda:1 cuda:2 cuda:3 cuda:4 cuda:5 cuda:6 cuda:7 \
+    --cache_dir /root/.cache/huggingface/hub \
+    --reranker_max_length 1024
+
+change the embedder, reranker, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   msmarco/data_loader
+   msmarco/runner

docs/source/API/evaluation/msmarco/data_loader.rst

@@ -0,0 +1,13 @@
+data_loader
+===========
+
+.. autoclass:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader
+
+Methods
+-------
+
+.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader.available_dataset_names
+.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader.available_splits
+.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader._load_remote_corpus
+.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader._load_remote_qrels
+.. automethod:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalDataLoader._load_remote_queries

docs/source/API/evaluation/msmarco/runner.rst

@@ -0,0 +1,5 @@
+runner
+======
+
+.. autoclass:: FlagEmbedding.evaluation.msmarco.MSMARCOEvalRunner
+    :members:

docs/source/API/evaluation/mteb.rst

@@ -0,0 +1,37 @@
+MTEB
+====
+
+`MTEB <https://github.com/embeddings-benchmark/mteb>`_ (The Massive Text Embedding Benchmark) is a large-scale evaluation framework designed to assess the performance of text embedding models across a wide variety of NLP tasks. 
+Introduced to standardize and improve the evaluation of text embeddings, MTEB is crucial for assessing how well these models generalize across various real-world applications. 
+It contains a wide range of datasets in eight main NLP tasks and different languages, and provides an easy pipeline for evaluation.
+It also holds the well known MTEB `leaderboard <https://huggingface.co/spaces/mteb/leaderboard>`_, which contains a ranking of the latest first-class embedding models.
+
+You can evaluate model's performance on the whole MTEB benchmark by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/mteb/eval_mteb.sh
+    ./examples/evaluation/mteb/eval_mteb.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.mteb \
+    --eval_name mteb \
+    --output_dir ./mteb/search_results \
+    --languages eng \
+    --tasks NFCorpus BiorxivClusteringS2S SciDocsRR \
+    --eval_output_path ./mteb/mteb_eval_results.json \
+    --embedder_name_or_path BAAI/bge-large-en-v1.5 \
+    --devices cuda:7 \
+    --cache_dir /root/.cache/huggingface/hub
+
+change the embedder, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   mteb/arguments
+   mteb/searcher
+   mteb/runner

docs/source/API/evaluation/mteb/arguments.rst

@@ -0,0 +1,4 @@
+arguments
+=========
+
+.. autoclass:: FlagEmbedding.evaluation.mteb.arguments.MTEBEvalArgs

docs/source/API/evaluation/mteb/runner.rst

@@ -0,0 +1,4 @@
+runner
+======
+
+.. autoclass:: FlagEmbedding.evaluation.mteb.runner.MTEBEvalRunner

docs/source/API/evaluation/mteb/searcher.rst

@@ -0,0 +1,6 @@
+searcher
+========
+
+.. autoclass:: FlagEmbedding.evaluation.mteb.searcher.MTEBEvalDenseRetriever
+
+.. autoclass:: FlagEmbedding.evaluation.mteb.searcher.MTEBEvalReranker