paddlepaddle--paddlenlp/docs/en/datasets.md

PaddleNLP Datasets API

PaddleNLP provides a quick API for loading the following datasets. When using, please add splits information as needed:

Reading Comprehension

Dataset Name	Description	Method Call
SQuAD	Stanford Question Answering Dataset, including SQuAD1.1 and SQuAD2.0	paddlenlp.datasets.load_dataset('squad')
DuReader-yesno	Qianyan Dataset: Reading comprehension, determining answer polarity	paddlenlp.datasets.load_dataset('dureader_yesno')
DuReader-robust	Qianyan Dataset: Reading comprehension, answer extraction from original text	paddlenlp.datasets.load_dataset('dureader_robust')
CMRC2018	The Second "iFLYTEK Cup" Chinese Machine Reading Comprehension Evaluation Dataset	paddlenlp.datasets.load_dataset('cmrc2018')
DRCD	Delta Reading Comprehension Dataset	paddlenlp.datasets.load_dataset('drcd')

Text Classification

Dataset Name	Description	Method Call
CoLA	Single sentence classification task, binary classification, determining sentence grammaticality	paddlenlp.datasets.load_dataset('glue','cola')
SST-2	Single sentence classification task, binary classification, determining sentence sentiment polarity	paddlenlp.datasets.load_dataset('glue','sst-2')
MRPC	Sentence pair matching task, binary classification, determining if sentence pairs have the same meaning	paddlenlp.datasets.load_dataset('glue','mrpc')
STSB	Calculating sentence pair similarity, score ranges from 1 to 5	paddlenlp.datasets.load_dataset('glue','sts-b')
QQP	Determine whether sentence pairs are equivalent, with two cases: equivalent and not equivalent, a binary classification task	paddlenlp.datasets.load_dataset('glue','qqp')
MNLI	Sentence pairs consisting of a premise and a hypothesis. The relationship between the premise and hypothesis can be one of three: entailment, contradiction, or neutral. A three-class classification problem for sentence pairs	paddlenlp.datasets.load_dataset('glue','mnli')
QNLI	Determine whether a question and a sentence entail each other, with entailment and non-entailment as the two classes, a binary classification task	paddlenlp.datasets.load_dataset('glue','qnli')
RTE	Determine whether sentence pairs entail each other, whether sentence 1 and sentence 2 are mutually entailing, a binary classification task	paddlenlp.datasets.load_dataset('glue','rte')
WNLI	Determine whether sentence pairs are related, with related and unrelated as the two classes, a binary classification task	paddlenlp.datasets.load_dataset('glue','wnli')
LCQMC	A Large-scale Chinese Question Matching Corpus for semantic matching	paddlenlp.datasets.load_dataset('lcqmc')
ChnSentiCorp	Chinese sentiment analysis corpus for reviews	paddlenlp.datasets.load_dataset('chnsenticorp')

Sequence Labeling

Dataset Name	Description	Loading Method
MSRA_NER	MSRA Named Entity Recognition dataset	paddlenlp.datasets.load_dataset('msra_ner')
People's Daily	People's Daily Named Entity Recognition dataset
paddlenlp.datasets.load_dataset('peoples_daily_ner')

Machine Translation

Dataset Name	Description	Usage
IWSLT15	IWSLT'15 English-Vietnamese data, English-Vietnamese translation dataset	paddlenlp.datasets.load_dataset('iwslt15')
WMT14ENDE	WMT14 EN-DE English-German translation dataset with BPE tokenization	paddlenlp.datasets.load_dataset('wmt14ende')

Simultaneous Interpretation

Dataset Name	Description	Usage
BSTC	Thousand Words Dataset: Simultaneous Interpretation, including transcription_translation and ASR	paddlenlp.datasets.load_dataset('bstc', 'asr')

Text Generation

Dataset Name	Description	Usage
Poetry	Classical Chinese poetry collection dataset	paddlenlp.datasets.load_dataset('poetry')
Couplet	Chinese couplet dataset	paddlenlp.datasets.load_dataset('couplet')

Corpora

Dataset Name	Description	Usage
PTB	Penn Treebank Dataset	paddlenlp.datasets.load_dataset('ptb')
Yahoo Answer 100k	Sampled 100K from Yahoo Answer	paddlenlp.datasets.load_dataset('yahoo_answer_100k')