wehub/flagopen--flagembedding
1
0
Fork 0
Code Issues Pull Requests Actions 2 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:21 +08:00
commit bc34f6df14
1149 changed files with 328099 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/API/abc/evaluation.rst
+9
View File
@@ -0,0 +1,9 @@
Evaluation
==========
.. toctree::
evaluation/arguments
evaluation/data_loader
evaluation/searcher
evaluation/evaluator
evaluation/runner
docs/source/API/abc/evaluation/arguments.rst
+7
View File
@@ -0,0 +1,7 @@
Arguments
=========
.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvalArgs
.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvalModelArgs
docs/source/API/abc/evaluation/data_loader.rst
+25
View File
@@ -0,0 +1,25 @@
dataset loader
==============
.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader
Methods
-------
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.available_dataset_names
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.available_splits
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.check_dataset_names
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.check_splits
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.load_corpus
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.load_qrels
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader.load_queries
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_remote_corpus
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_remote_qrels
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_remote_queries
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_local_corpus
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_local_qrels
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._load_local_queries
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._download_file
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._get_fpath_size
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._download_gz_file
.. automethod:: FlagEmbedding.abc.evaluation.AbsEvalDataLoader._download_zip_file
docs/source/API/abc/evaluation/evaluator.rst
+4
View File
@@ -0,0 +1,4 @@
Evaluator
=========
.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvaluator
docs/source/API/abc/evaluation/runner.rst
+4
View File
@@ -0,0 +1,4 @@
runner
======
.. autoclass:: FlagEmbedding.abc.evaluation.AbsEvalRunner
docs/source/API/abc/evaluation/searcher.rst
+18
View File
@@ -0,0 +1,18 @@
========
searcher
========
EvalRetriever
=============
.. autoclass:: FlagEmbedding.abc.evaluation.EvalRetriever
EvalDenseRetriever
==================
.. autoclass:: FlagEmbedding.abc.evaluation.EvalDenseRetriever
EvalReranker
============
.. autoclass:: FlagEmbedding.abc.evaluation.EvalReranker
docs/source/API/abc/finetune.rst
+6
View File
@@ -0,0 +1,6 @@
Finetune
========
.. toctree::
finetune/embedder
finetune/reranker
docs/source/API/abc/finetune/embedder.rst
+9
View File
@@ -0,0 +1,9 @@
Embedder
========
.. toctree::
embedder/AbsArguments
embedder/AbsDataset
embedder/AbsModeling
embedder/AbsTrainer
embedder/AbsRunner
docs/source/API/abc/finetune/embedder/AbsArguments.rst
+6
View File
@@ -0,0 +1,6 @@
AbsArguments
============
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModelArguments
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerDataArguments
docs/source/API/abc/finetune/embedder/AbsDataset.rst
+53
View File
@@ -0,0 +1,53 @@
==========
AbsDataset
==========
AbsEmbedderTrainDataset
=======================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainDataset
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainDataset._load_dataset
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainDataset._shuffle_text
AbsEmbedderCollator
===================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderCollator
AbsEmbedderSameDatasetTrainDataset
==================================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset.refresh_epoch
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._load_dataset
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._get_file_batch_size
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._get_train_group_size
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetTrainDataset._create_batch_data
AbsEmbedderSameDatasetCollator
==============================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderSameDatasetCollator
EmbedderTrainerCallbackForDataRefresh
=====================================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.EmbedderTrainerCallbackForDataRefresh
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.embedder.EmbedderTrainerCallbackForDataRefresh.on_epoch_end
docs/source/API/abc/finetune/embedder/AbsModeling.rst
+41
View File
@@ -0,0 +1,41 @@
===========
AbsModeling
===========
AbsEmbedderModel
================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.encode
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.compute_loss
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.compute_score
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.save
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.get_local_score
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.compute_local_score
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.forward
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel.distill_loss
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._compute_no_in_batch_neg_loss
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._compute_in_batch_neg_loss
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._compute_cross_device_neg_loss
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModel._dist_gather_tensor
EmbedderOutput
==============
.. autoclass:: FlagEmbedding.abc.finetune.embedder.EmbedderOutput
docs/source/API/abc/finetune/embedder/AbsRunner.rst
+21
View File
@@ -0,0 +1,21 @@
=========
AbsRunner
=========
AbsEmbedderTrainer
==================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_tokenizer_and_model
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_trainer
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_train_dataset
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.load_data_collator
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderRunner.run
docs/source/API/abc/finetune/embedder/AbsTrainer.rst
+13
View File
@@ -0,0 +1,13 @@
==========
AbsTrainer
==========
AbsEmbedderTrainer
==================
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainer
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderTrainer.compute_loss
docs/source/API/abc/finetune/reranker.rst
+9
View File
@@ -0,0 +1,9 @@
Reranker
========
.. toctree::
reranker/AbsArguments
reranker/AbsDataset
reranker/AbsModeling
reranker/AbsTrainer
reranker/AbsRunner
docs/source/API/abc/finetune/reranker/AbsArguments.rst
+6
View File
@@ -0,0 +1,6 @@
AbsArguments
============
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderModelArguments
.. autoclass:: FlagEmbedding.abc.finetune.embedder.AbsEmbedderDataArguments
docs/source/API/abc/finetune/reranker/AbsDataset.rst
+32
View File
@@ -0,0 +1,32 @@
==========
AbsDataset
==========
AbsRerankerTrainDataset
=======================
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset.create_one_example
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset._load_dataset
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainDataset._shuffle_text
AbsRerankerCollator
===================
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerCollator
AbsLLMRerankerTrainDataset
==========================
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsLLMRerankerTrainDataset
AbsLLMRerankerCollator
======================
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsLLMRerankerCollator
docs/source/API/abc/finetune/reranker/AbsModeling.rst
+31
View File
@@ -0,0 +1,31 @@
===========
AbsModeling
===========
AbsRerankerModel
================
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.encode
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.gradient_checkpointing_enable
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.enable_input_require_grads
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.forward
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.compute_loss
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.save
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerModel.save_pretrained
RerankerOutput
==============
.. autoclass:: FlagEmbedding.abc.finetune.reranker.RerankerOutput
docs/source/API/abc/finetune/reranker/AbsRunner.rst
+21
View File
@@ -0,0 +1,21 @@
=========
AbsRunner
=========
AbsRerankerTrainer
==================
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_tokenizer_and_model
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_trainer
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_train_dataset
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.load_data_collator
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerRunner.run
docs/source/API/abc/finetune/reranker/AbsTrainer.rst
+13
View File
@@ -0,0 +1,13 @@
==========
AbsTrainer
==========
AbsRerankerTrainer
==================
.. autoclass:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainer
Methods
-------
.. automethod:: FlagEmbedding.abc.finetune.reranker.AbsRerankerTrainer.compute_loss
docs/source/API/abc/inference.rst
+6
View File
@@ -0,0 +1,6 @@
Inference
=========
.. toctree::
inference/AbsEmbedder
inference/AbsReranker
docs/source/API/abc/inference/AbsEmbedder.rst
+29
View File
@@ -0,0 +1,29 @@
AbsEmbedder
===========
.. autoclass:: FlagEmbedding.abc.inference.AbsEmbedder
Methods
-------
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.get_target_devices
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.get_detailed_instruct
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_queries
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_corpus
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_single_device
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.start_multi_process_pool
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder._encode_multi_process_worker
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.stop_multi_process_pool
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder.encode_multi_process
.. automethod:: FlagEmbedding.abc.inference.AbsEmbedder._concatenate_results_from_multi_process
docs/source/API/abc/inference/AbsReranker.rst
+25
View File
@@ -0,0 +1,25 @@
AbsReranker
===========
.. autoclass:: FlagEmbedding.abc.inference.AbsReranker
Methods
-------
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.get_target_devices
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.get_detailed_instruct
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.get_detailed_inputs
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.compute_score
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.compute_score_single_gpu
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.start_multi_process_pool
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.encode_multi_process
.. automethod:: FlagEmbedding.abc.inference.AbsReranker._encode_multi_process_worker
.. automethod:: FlagEmbedding.abc.inference.AbsReranker.stop_multi_process_pool