chore: import upstream snapshot with attribution

2026-07-13 13:39:21 +08:00
commit bc34f6df14
1149 changed files with 328099 additions and 0 deletions
@@ -0,0 +1,95 @@
+MLDR
+====
+
+`MLDR <https://huggingface.co/datasets/Shitao/MLDR>`_ is a Multilingual Long-Document Retrieval dataset built on Wikipeida, Wudao and mC4, covering 13 typologically diverse languages. 
+Specifically, we sample lengthy articles from Wikipedia, Wudao and mC4 datasets and randomly choose paragraphs from them. 
+Then we use GPT-3.5 to generate questions based on these paragraphs. 
+The generated question and the sampled article constitute a new text pair to the dataset.
+
+An example of ``train`` set looks like:
+
+.. code:: bash
+
+    {
+        'query_id': 'q-zh-<...>', 
+        'query': '...', 
+        'positive_passages': [
+            {
+                'docid': 'doc-zh-<...>',
+                'text': '...'
+            }
+        ],
+        'negative_passages': [
+            {
+                'docid': 'doc-zh-<...>',
+                'text': '...'
+            },
+            ...
+        ]
+    }
+
+An example of ``dev`` and ``test`` set looks like:
+
+.. code:: bash
+
+    {
+        'query_id': 'q-zh-<...>', 
+        'query': '...', 
+        'positive_passages': [
+            {
+                'docid': 'doc-zh-<...>',
+                'text': '...'
+            }
+        ],
+        'negative_passages': []
+    }
+
+An example of ``corpus`` looks like:
+
+.. code:: bash
+
+    {
+        'docid': 'doc-zh-<...>', 
+        'text': '...'
+    }
+
+You can evaluate model's performance on MLDR simply by running our provided shell script:
+
+.. code:: bash
+
+    chmod +x /examples/evaluation/mldr/eval_mldr.sh
+    ./examples/evaluation/mldr/eval_mldr.sh
+
+Or by running:
+
+.. code:: bash
+
+    python -m FlagEmbedding.evaluation.mldr \
+    --eval_name mldr \
+    --dataset_dir ./mldr/data \
+    --dataset_names hi \
+    --splits test \
+    --corpus_embd_save_dir ./mldr/corpus_embd \
+    --output_dir ./mldr/search_results \
+    --search_top_k 1000 \
+    --rerank_top_k 100 \
+    --cache_path /root/.cache/huggingface/hub \
+    --overwrite False \
+    --k_values 10 100 \
+    --eval_output_method markdown \
+    --eval_output_path ./mldr/mldr_eval_results.md \
+    --eval_metrics ndcg_at_10 \
+    --embedder_name_or_path BAAI/bge-m3 \
+    --reranker_name_or_path BAAI/bge-reranker-v2-m3 \
+    --devices cuda:0 cuda:1 \
+    --cache_dir /root/.cache/huggingface/hub \
+    --embedder_passage_max_length 8192 \
+    --reranker_max_length 8192
+
+change the args of embedder, reranker, devices and cache directory to your preference.
+
+.. toctree::
+   :hidden:
+
+   mldr/data_loader
+   mldr/runner