wehub/huggingface--transformers
1
0
Fork 0
Code Issues Pull Requests Actions 10 Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
3386c0810530237d041cf1d0a042fcefeaceb442
huggingface--transformers/docs/source/zh/main_classes/data_collator.md
T
wehub-resource-sync e06fe8e8c6
Secret Leaks / trufflehog (push) Failing after 1s
Details
Build documentation / build (push) Failing after 1s
Details
Build documentation / build_other_lang (push) Failing after 0s
Details
CodeQL Security Analysis / CodeQL Analysis (push) Failing after 0s
Details
PR CI / pr-ci (push) Failing after 1s
Details
Slow tests on important models (on Push - A10) / Get all modified files (push) Failing after 1s
Details
Slow tests on important models (on Push - A10) / Model CI (push) Has been skipped
Details
Self-hosted runner (benchmark) / Benchmark (aws-g5-4xlarge-cache) (push) Has been cancelled
Details
New model PR merged notification / Notify new model (push) Has been cancelled
Details
Update Transformers metadata / build_and_package (push) Has been cancelled
Details
chore: import upstream snapshot with attribution
2026-07-13 11:57:37 +08:00

2.1 KiB
Raw Blame History

Data Collator

Data collators是一个对象，通过使用数据集元素列表作为输入来形成一个批次。这些元素与 train_dataset 或 eval_dataset 的元素类型相同。

为了能够构建批次，Data collators可能会应用一些预处理（比如填充）。其中一些（比如[DataCollatorForLanguageModeling]）还会在形成的批次上应用一些随机数据增强（比如随机掩码）。

在示例脚本或示例notebooks中可以找到使用的示例。

Default data collator

autodoc data.data_collator.default_data_collator

DefaultDataCollator

autodoc data.data_collator.DefaultDataCollator

DataCollatorWithPadding

autodoc data.data_collator.DataCollatorWithPadding

DataCollatorForTokenClassification

autodoc data.data_collator.DataCollatorForTokenClassification

DataCollatorForSeq2Seq

autodoc data.data_collator.DataCollatorForSeq2Seq

DataCollatorForLanguageModeling

autodoc data.data_collator.DataCollatorForLanguageModeling - numpy_mask_tokens - torch_mask_tokens

DataCollatorForWholeWordMask

autodoc data.data_collator.DataCollatorForWholeWordMask - numpy_mask_tokens - torch_mask_tokens

DataCollatorForPermutationLanguageModeling

autodoc data.data_collator.DataCollatorForPermutationLanguageModeling - numpy_mask_tokens - torch_mask_tokens

Reference in New Issue View Git Blame Copy Permalink