mirrors/microsoft--unilm
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:24:13 +08:00
commit 1037506f2e
6050 changed files with 1731598 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
xtune/README.md
+98
View File
@@ -0,0 +1,98 @@
# xTune
Code for ACL2021 paper [Consistency Regularization for Cross-Lingual Fine-Tuning](https://arxiv.org/pdf/2106.08226.pdf).
## Environment
DockerFile: `dancingsoul/pytorch:xTune`
Install the fine-tuning code: `pip install --user .`
## Data & Model Preparation
### XTREME Datasets
1) Create a download folder with `mkdir -p download` in the root of this project.
2) manually download `panx_dataset` (for NER) [here][2], (note that it will download as `AmazonPhotos.zip`) to the download directory.
3) run the following command to download the remaining datasets: `bash scripts/download_data.sh`
The code of downloading dataset from XTREME is from [xtreme offical repo][1].
Note that we keep the labels in test set for easier evaluation. To prevent accidental evaluation on the test sets while running experiments, the code of [xtreme offical repo][1] removes labels of the test data during pre-processing and changes the order of the test sentences for cross-lingual sentence retrieval.
Replace `csv.writer(fout, delimiter='\t')` with `csv.writer(fout, delimiter='\t', quoting=csv.QUOTE_NONE, quotechar='')` in utils_process.py if using XTREME official repo.
### Translations
XTREME provides translations for SQuAD v1.1 (only train and dev), MLQA, PAWS-X, TyDiQA-GoldP, XNLI, and XQuAD, which can be downloaded from [here][3]. The `xtreme_translations` folder should be moved to the download directory.
The target language translations for panx and udpos are obtained with Google Translate, since they are not provided. Our processed version can be downloaded from [here][4]. It should be merged with the above `xtreme_translations` folder.
### Bi-lingual dictionaries
We obtain the bi-lingual dictionaries from the [MUSE][6] repo. For convenience, you can download them from [here][7] and move it to the download directory, i.e., `./download/dicts`.
### Models
XLM-Roberta is supported. We utilize the [huggingface][5] format, which can be downloaded with `bash scripts/download_model.sh`.
## Fine-tuning Usage
Our default settings were using Nvidia V100-32GB GPU cards. If there were out-of-memory errors, you can reduce `per_gpu_train_batch_size` while increasing `gradient_accumulation_steps`, or use multi-GPU training.
xTune consists of a two-stage training process.
- Stage 1: fine-tuning with example consistency on the English training set.
- Stage 2: fine-tuning with example consistency on the augmented training set and regularize model consistency with the model from Stage 1.
It's recommended to use both Stage 1 and Stage 2 for token-level tasks, such as sequential labeling, and question answering. For text classification, you can only use Stage 1 if the computation budget was limited.
```bash
bash ./scripts/train.sh [setting] [dataset] [model] [stage] [gpu] [data_dir] [output_dir]
```
where the options are described as follows:
- `[setting]`: `translate-train-all` (using input translation for the languages other than English) or `cross-lingual-transfer` (only using English for zero-shot cross-lingual transfer)
- `[dataset]`: dataset names in XTREME, i.e., `xnli`, `panx`, `pawsx`, `udpos`, `mlqa`, `tydiqa`, `xquad`
- `[model]`: `xlm-roberta-base`, `xlm-roberta-large`
- `[stage]`: `1` (first stage), `2` (second stage)
- `[gpu]`: used to set environment variable `CUDA_VISIBLE_DEVICES`
- `[data_dir]`: folder of training data
- `[output_dir]`: folder of fine-tuning output
## Examples: XTREME Tasks
### XNLI fine-tuning on English training set and translated training sets (`translate-train-all`)
```bash
# run stage 1 of xTune
bash ./scripts/train.sh translate-train-all xnli xlm-roberta-base 1
# run stage 2 of xTune (optional)
bash ./scripts/train.sh translate-train-all xnli xlm-roberta-base 2
```
### XNLI fine-tuning on English training set (`cross-lingual-transfer`)
```bash
# run stage 1 of xTune
bash ./scripts/train.sh cross-lingual-transfer xnli xlm-roberta-base 1
# run stage 2 of xTune (optional)
bash ./scripts/train.sh cross-lingual-transfer xnli xlm-roberta-base 2
```
## Paper
Please cite our paper `\cite{bo2021xtune}` if you found the resources in the repository useful.
```
@inproceedings{bo2021xtune,
author = {Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei},
booktitle = {Proceedings of ACL 2021},
title = {{Consistency Regularization for Cross-Lingual Fine-Tuning}},
year = {2021}
}
```
## Reference
1. https://github.com/google-research/xtreme
2. https://www.amazon.com/clouddrive/share/d3KGCRCIYwhKJF0H3eWA26hjg2ZCRhjpEQtDL70FSBN?_encoding=UTF8&%2AVersion%2A=1&%2Aentries%2A=0&mgh=1
3. https://console.cloud.google.com/storage/browser/xtreme_translations
4. https://drive.google.com/drive/folders/1Rdbc0Us_4I5MpRCwLASxBwqSW8_dlF87?usp=sharing
5. https://github.com/huggingface/transformers/
6. https://github.com/facebookresearch/MUSE
7. https://drive.google.com/drive/folders/1k9rQinwUXicglA5oyzo9xtgqiuUVDkjT?usp=sharing
xtune/scripts/cross-lingual-transfer/train_mlqa.sh
+136
View File
@@ -0,0 +1,136 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
cp -r $DATA_DIR/squad/ $DATA_DIR/mlqa/squad1.1/
TASK='mlqa'
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/SQuAD/translate-train/
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=4
MAXL=384
LANGS="en,es,de,ar,hi,vi,zh"
BSR=0.3
SA=0.3
SNBS=-1
CSR=0.3
R1_LAMBDA=5.0
R2_LAMBDA=5.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=4
GRAD_ACC=8
LR=1.5e-5
else
BATCH_SIZE=32
GRAD_ACC=1
LR=3e-5
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages es,de,ar,hi,vi,zh \
--noised_max_seq_length $MAXL
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-SS-R2_Lambda${R2_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method ss \
--max_steps 24000 \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_MODEL_PATH
fi
xtune/scripts/cross-lingual-transfer/train_panx.sh
+137
View File
@@ -0,0 +1,137 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='panx'
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAX_LENGTH=128
LANGS="ar,he,vi,id,jv,ms,tl,eu,ml,ta,te,af,nl,en,de,el,bn,hi,mr,ur,fa,fr,it,pt,es,bg,ru,ja,ka,ko,th,sw,yo,my,zh,kk,tr,et,fi,hu"
EVALUATE_STEPS=1000
BSR=0.3
SA=0.3
SNBS=-1
R1_LAMBDA=5.0
R2_LAMBDA=5.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=32
GRAD_ACC=1
LR=7e-6
else
BATCH_SIZE=32
GRAD_ACC=1
LR=1e-5
fi
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/translate_train.panx.txt
DATA_DIR=$DATA_DIR/$TASK/${TASK}_processed_maxlen${MAX_LENGTH}/
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-SS-R2_Lambda${R2_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method ss \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH \
--use_hard_labels
fi
xtune/scripts/cross-lingual-transfer/train_pawsx.sh
+124
View File
@@ -0,0 +1,124 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='pawsx'
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/PAWSX/
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAXL=256
LANGS="de,en,es,fr,ja,ko,zh"
EVALUATE_STEPS=1000
CSR=0.5
R1_LAMBDA=5.0
R2_LAMBDA=2.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=16
GRAD_ACC=2
LR=1e-5
else
BATCH_SIZE=32
GRAD_ACC=1
LR=1e-5
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages de,es,fr,ja,ko,zh
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_Lambda${R1_LAMBDA}-Aug1.0-CS-R2_Lambda${R2_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages de,es,fr,ja,ko,zh \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method cs \
--r2_lambda $R2_LAMBDA
fi
xtune/scripts/cross-lingual-transfer/train_tydiqa.sh
+135
View File
@@ -0,0 +1,135 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='tydiqa'
MODEL_PATH=$DATA_DIR/$MODEL
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/TyDiQA-GoldP/translate-train/
MAXL=384
LANGS="en,ar,bn,fi,id,ko,ru,sw,te"
BSR=0.3
SA=0.3
SNBS=-1
R1_LAMBDA=5.0
R2_LAMBDA=5.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=4
GRAD_ACC=8
LR=1.5e-5
EPOCH=10
MAX_STEPS=2500
else
BATCH_SIZE=32
GRAD_ACC=1
LR=3e-5
EPOCH=20
MAX_STEPS=5000
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-SS-R2_Lambda${R2_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method ss \
--max_steps $MAX_STEPS \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH
fi
xtune/scripts/cross-lingual-transfer/train_udpos.sh
+138
View File
@@ -0,0 +1,138 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='udpos'
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAX_LENGTH=128
LANGS="af,ar,bg,de,el,en,es,et,eu,fa,fi,fr,he,hi,hu,id,it,ja,kk,ko,mr,nl,pt,ru,ta,te,th,tl,tr,ur,vi,yo,zh"
EVALUATE_STEPS=500
BSR=0.5
SA=0.3
SNBS=-1
R1_LAMBDA=5.0
R2_LAMBDA=0.3
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=32
GRAD_ACC=1
LR=5e-6
else
BATCH_SIZE=32
GRAD_ACC=1
LR=2e-5
fi
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/translate_train.udpos.txt
DATA_DIR=$DATA_DIR/$TASK/${TASK}_processed_maxlen${MAX_LENGTH}/
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--use_pooling_strategy \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-SS-R2_Lambda${R2_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--use_pooling_strategy \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method ss \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH
fi
xtune/scripts/cross-lingual-transfer/train_xnli.sh
+124
View File
@@ -0,0 +1,124 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='xnli'
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/XNLI/
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAXL=256
LANGS="ar,bg,de,el,en,es,fr,hi,ru,sw,th,tr,ur,vi,zh"
EVALUATE_STEPS=5000
CSR=0.3
R1_LAMBDA=5.0
R2_LAMBDA=5.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=16
GRAD_ACC=2
LR=5e-6
else
BATCH_SIZE=32
GRAD_ACC=1
LR=7e-6
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages ar,bg,de,el,es,fr,hi,ru,sw,th,tr,ur,vi,zh
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_Lambda${R1_LAMBDA}-Aug1.0-CS-R2_Lambda${R2_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages ar,bg,de,el,es,fr,hi,ru,sw,th,tr,ur,vi,zh \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method cs \
--r2_lambda $R2_LAMBDA
fi
xtune/scripts/cross-lingual-transfer/train_xquad.sh
+134
View File
@@ -0,0 +1,134 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
cp -r $DATA_DIR/squad/ $DATA_DIR/xquad/squad1.1/
TASK='xquad'
MODEL_PATH=$DATA_DIR/$MODEL
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/SQuAD/translate-train/
EPOCH=4
MAXL=384
LANGS="ar,de,el,en,es,hi,ru,th,tr,vi,zh"
BSR=0.3
SA=0.3
SNBS=-1
CSR=0.3
R1_LAMBDA=5.0
R2_LAMBDA=5.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=4
GRAD_ACC=8
LR=1.5e-5
else
BATCH_SIZE=32
GRAD_ACC=1
LR=3e-5
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages ar,de,el,es,hi,ru,th,tr,vi,zh \
--noised_max_seq_length $MAXL
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-SS-R2_Lambda${R2_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method ss \
--max_steps 24000 \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH
fi
xtune/scripts/download_data.sh
+200
View File
@@ -0,0 +1,200 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
DIR=$REPO/download/
mkdir -p $DIR
# download XNLI dataset
function download_xnli {
OUTPATH=$DIR/xnli-tmp/
if [ ! -d $OUTPATH/XNLI-MT-1.0 ]; then
if [ ! -f $OUTPATH/XNLI-MT-1.0.zip ]; then
wget -c https://dl.fbaipublicfiles.com/XNLI/XNLI-MT-1.0.zip -P $OUTPATH -q --show-progress
fi
unzip -qq $OUTPATH/XNLI-MT-1.0.zip -d $OUTPATH
fi
if [ ! -d $OUTPATH/XNLI-1.0 ]; then
if [ ! -f $OUTPATH/XNLI-1.0.zip ]; then
wget -c https://dl.fbaipublicfiles.com/XNLI/XNLI-1.0.zip -P $OUTPATH -q --show-progress
fi
unzip -qq $OUTPATH/XNLI-1.0.zip -d $OUTPATH
fi
python $REPO/utils_preprocess.py \
--data_dir $OUTPATH \
--output_dir $DIR/xnli/ \
--task xnli
rm -rf $OUTPATH
echo "Successfully downloaded data at $DIR/xnli" >> $DIR/download.log
}
# download PAWS-X dataset
function download_pawsx {
cd $DIR
wget https://storage.googleapis.com/paws/pawsx/x-final.tar.gz -q --show-progress
tar xzf x-final.tar.gz -C $DIR/
python $REPO/utils_preprocess.py \
--data_dir $DIR/x-final \
--output_dir $DIR/pawsx/ \
--task pawsx
rm -rf x-final x-final.tar.gz
echo "Successfully downloaded data at $DIR/pawsx" >> $DIR/download.log
}
# download UD-POS dataset
function download_udpos {
base_dir=$DIR/udpos-tmp
out_dir=$base_dir/conll/
mkdir -p $out_dir
cd $base_dir
curl -s --remote-name-all https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-3105/ud-treebanks-v2.5.tgz
tar -xzf $base_dir/ud-treebanks-v2.5.tgz
langs=(af ar bg de el en es et eu fa fi fr he hi hu id it ja kk ko mr nl pt ru ta te th tl tr ur vi yo zh)
for x in $base_dir/ud-treebanks-v2.5/*/*.conllu; do
file="$(basename $x)"
IFS='_' read -r -a array <<< "$file"
lang=${array[0]}
if [[ " ${langs[@]} " =~ " ${lang} " ]]; then
lang_dir=$out_dir/$lang/
mkdir -p $lang_dir
y=$lang_dir/${file/conllu/conll}
if [ ! -f "$y" ]; then
echo "python $REPO/src/ud-conversion-tools/conllu_to_conll.py $x $y --lang $lang --replace_subtokens_with_fused_forms --print_fused_forms"
python $REPO/src/ud-conversion-tools/conllu_to_conll.py $x $y --lang $lang --replace_subtokens_with_fused_forms --print_fused_forms
else
echo "${y} exists"
fi
fi
done
python $REPO/utils_preprocess.py --data_dir $out_dir/ --output_dir $DIR/udpos/ --task udpos
rm -rf $out_dir ud-treebanks-v2.tgz $DIR/udpos-tmp
echo "Successfully downloaded data at $DIR/udpos" >> $DIR/download.log
}
function download_panx {
echo "Download panx NER dataset"
if [ -f $DIR/AmazonPhotos.zip ]; then
base_dir=$DIR/panx_dataset/
unzip -qq -j $DIR/AmazonPhotos.zip -d $base_dir
cd $base_dir
langs=(ar he vi id jv ms tl eu ml ta te af nl en de el bn hi mr ur fa fr it pt es bg ru ja ka ko th sw yo my zh kk tr et fi hu)
for lg in ${langs[@]}; do
tar xzf $base_dir/${lg}.tar.gz
for f in dev test train; do mv $base_dir/$f $base_dir/${lg}-${f}; done
done
cd ..
python $REPO/utils_preprocess.py \
--data_dir $base_dir \
--output_dir $DIR/panx \
--task panx
rm -rf $base_dir
echo "Successfully downloaded data at $DIR/panx" >> $DIR/download.log
else
echo "Please download the AmazonPhotos.zip file on Amazon Cloud Drive mannually and save it to $DIR/AmazonPhotos.zip"
echo "https://www.amazon.com/clouddrive/share/d3KGCRCIYwhKJF0H3eWA26hjg2ZCRhjpEQtDL70FSBN"
fi
}
function download_tatoeba {
base_dir=$DIR/tatoeba-tmp/
wget https://github.com/facebookresearch/LASER/archive/master.zip
unzip -qq -o master.zip -d $base_dir/
mv $base_dir/LASER-master/data/tatoeba/v1/* $base_dir/
python $REPO/utils_preprocess.py \
--data_dir $base_dir \
--output_dir $DIR/tatoeba \
--task tatoeba
rm -rf $base_dir master.zip
echo "Successfully downloaded data at $DIR/tatoeba" >> $DIR/download.log
}
function download_bucc18 {
base_dir=$DIR/bucc2018/
cd $DIR
for lg in zh ru de fr; do
wget https://comparable.limsi.fr/bucc2018/bucc2018-${lg}-en.training-gold.tar.bz2 -q --show-progress
tar -xjf bucc2018-${lg}-en.training-gold.tar.bz2
wget https://comparable.limsi.fr/bucc2018/bucc2018-${lg}-en.sample-gold.tar.bz2 -q --show-progress
tar -xjf bucc2018-${lg}-en.sample-gold.tar.bz2
done
mv $base_dir/*/* $base_dir/
for f in $base_dir/*training*; do mv $f ${f/training/test}; done
for f in $base_dir/*sample*; do mv $f ${f/sample/dev}; done
rm -rf $base_dir/*test.gold $DIR/bucc2018*tar.bz2 $base_dir/{zh,ru,de,fr}-en/
echo "Successfully downloaded data at $DIR/bucc2018" >> $DIR/download.log
}
function download_squad {
echo "download squad"
base_dir=$DIR/squad/
mkdir -p $base_dir && cd $base_dir
wget https://raw.githubusercontent.com/rajpurkar/SQuAD-explorer/master/dataset/train-v1.1.json -q --show-progress
wget https://raw.githubusercontent.com/rajpurkar/SQuAD-explorer/master/dataset/dev-v1.1.json -q --show-progress
echo "Successfully downloaded data at $DIR/squad" >> $DIR/download.log
}
function download_xquad {
echo "download xquad"
base_dir=$DIR/xquad/
mkdir -p $base_dir && cd $base_dir
for lang in ar de el en es hi ru th tr vi zh; do
wget https://raw.githubusercontent.com/deepmind/xquad/master/xquad.${lang}.json -q --show-progress
done
python $REPO/utils_preprocess.py --data_dir $base_dir --output_dir $base_dir --task xquad
echo "Successfully downloaded data at $DIR/xquad" >> $DIR/download.log
}
function download_mlqa {
echo "download mlqa"
base_dir=$DIR/mlqa/
mkdir -p $base_dir && cd $base_dir
zip_file=MLQA_V1.zip
wget https://dl.fbaipublicfiles.com/MLQA/${zip_file} -q --show-progress
unzip -qq ${zip_file}
rm ${zip_file}
python $REPO/utils_preprocess.py --data_dir $base_dir/MLQA_V1/test --output_dir $base_dir --task mlqa
echo "Successfully downloaded data at $DIR/mlqa" >> $DIR/download.log
}
function download_tydiqa {
echo "download tydiqa-goldp"
base_dir=$DIR/tydiqa/
mkdir -p $base_dir && cd $base_dir
tydiqa_train_file=tydiqa-goldp-v1.1-train.json
tydiqa_dev_file=tydiqa-goldp-v1.1-dev.tgz
wget https://storage.googleapis.com/tydiqa/v1.1/${tydiqa_train_file} -q --show-progress
wget https://storage.googleapis.com/tydiqa/v1.1/${tydiqa_dev_file} -q --show-progress
tar -xf ${tydiqa_dev_file}
rm ${tydiqa_dev_file}
out_dir=$base_dir/tydiqa-goldp-v1.1-train
python $REPO/utils_preprocess.py --data_dir $base_dir --output_dir $out_dir --task tydiqa
mv $base_dir/$tydiqa_train_file $out_dir/
echo "Successfully downloaded data at $DIR/tydiqa" >> $DIR/download.log
}
download_xnli
download_pawsx
download_tatoeba
download_bucc18
download_squad
download_xquad
download_mlqa
download_tydiqa
download_udpos
download_panx
cp -r $DIR/squad/ $DIR/xquad/squad1.1/
xtune/scripts/download_model.sh
+43
View File
@@ -0,0 +1,43 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
DIR=$REPO/download/
mkdir -p $DIR
# download xlm-roberta-base
function download_xlm-roberta-base {
mkdir -p $DIR/xlm-roberta-base/
cd $DIR/xlm-roberta-base/
wget https://huggingface.co/xlm-roberta-base/resolve/main/pytorch_model.bin -q --show-progress
wget https://huggingface.co/xlm-roberta-base/resolve/main/config.json -q --show-progress
wget https://huggingface.co/xlm-roberta-base/resolve/main/sentencepiece.bpe.model -q --show-progress
wget https://huggingface.co/xlm-roberta-base/resolve/main/tokenizer.json -q --show-progress
echo "Successfully downloaded xlm-roberta-base at $DIR/xlm-roberta-base" >> $DIR/download_model.log
}
# download xlm-roberta-large
function download_xlm-roberta-large {
mkdir -p $DIR/xlm-roberta-large/
cd $DIR/xlm-roberta-large/
wget https://huggingface.co/xlm-roberta-large/resolve/main/pytorch_model.bin -q --show-progress
wget https://huggingface.co/xlm-roberta-large/resolve/main/config.json -q --show-progress
wget https://huggingface.co/xlm-roberta-large/resolve/main/sentencepiece.bpe.model -q --show-progress
wget https://huggingface.co/xlm-roberta-large/resolve/main/tokenizer.json -q --show-progress
echo "Successfully downloaded xlm-roberta-large at $DIR/xlm-roberta-large" >> $DIR/download_model.log
}
download_xlm-roberta-base
download_xlm-roberta-large
xtune/scripts/preprocess_panx.sh
+44
View File
@@ -0,0 +1,44 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-bert-base-multilingual-cased}
DATA_DIR=${2:-"$REPO/download/"}
TASK='panx'
MAXL=128
LANGS="ar,he,vi,id,jv,ms,tl,eu,ml,ta,te,af,nl,en,de,el,bn,hi,mr,ur,fa,fr,it,pt,es,bg,ru,ja,ka,ko,th,sw,yo,my,zh,kk,tr,et,fi,hu"
LC=""
if [ $MODEL == "bert-base-multilingual-cased" ]; then
MODEL_TYPE="bert"
elif [ $MODEL == "xlm-mlm-100-1280" ] || [ $MODEL == "xlm-mlm-tlm-xnli15-1024" ]; then
MODEL_TYPE="xlm"
LC=" --do_lower_case"
elif [ $MODEL == "xlm-roberta-large" ] || [ $MODEL == "xlm-roberta-base" ]; then
MODEL_TYPE="xlmr"
fi
SAVE_DIR="$DATA_DIR/$TASK/${TASK}_processed_maxlen${MAXL}"
mkdir -p $SAVE_DIR
python3 $REPO/utils_preprocess.py \
--data_dir $DATA_DIR/$TASK/ \
--task panx_tokenize \
--model_name_or_path $MODEL \
--model_type $MODEL_TYPE \
--max_len $MAXL \
--output_dir $SAVE_DIR \
--languages $LANGS $LC >> $SAVE_DIR/preprocess.log
if [ ! -f $SAVE_DIR/labels.txt ]; then
cat $SAVE_DIR/*/*.${MODEL} | cut -f 2 | grep -v "^$" | sort | uniq > $SAVE_DIR/labels.txt
fi
xtune/scripts/preprocess_udpos.sh
+46
View File
@@ -0,0 +1,46 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-bert-base-multilingual-cased}
DATA_DIR=${2:-"$REPO/download/"}
TASK='udpos'
MAXL=128
LANGS='af,ar,bg,de,el,en,es,et,eu,fa,fi,fr,he,hi,hu,id,it,ja,kk,ko,mr,nl,pt,ru,ta,te,th,tl,tr,ur,vi,yo,zh'
LC=""
if [ $MODEL == "bert-base-multilingual-cased" ]; then
MODEL_TYPE="bert"
elif [ $MODEL == "xlm-mlm-100-1280" ] || [ $MODEL == "xlm-mlm-tlm-xnli15-1024" ]; then
MODEL_TYPE="xlm"
LC=" --do_lower_case"
elif [ $MODEL == "xlm-roberta-large" ] || [ $MODEL == "xlm-roberta-base" ]; then
MODEL_TYPE="xlmr"
fi
SAVE_DIR="$DATA_DIR/${TASK}/udpos_processed_maxlen${MAXL}"
mkdir -p $SAVE_DIR
python3 $REPO/utils_preprocess.py \
--data_dir $DATA_DIR/${TASK}/ \
--task udpos_tokenize \
--model_name_or_path $MODEL \
--model_type $MODEL_TYPE \
--max_len $MAXL \
--output_dir $SAVE_DIR \
--languages $LANGS $LC >> $SAVE_DIR/process.log
if [ ! -f $SAVE_DIR/labels.txt ]; then
echo "create label"
cat $SAVE_DIR/*/*.${MODEL} | cut -f 2 | grep -v "^$" | sort | uniq > $SAVE_DIR/labels.txt
fi
xtune/scripts/train.sh
+35
View File
@@ -0,0 +1,35 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
SETTING=${1:-cross-lingual-transfer}
TASK=${2:-xnli}
MODEL=${3:-"xlm-roberta-base"}
STAGE=${4:-1}
GPU=${5:-0}
DATA_DIR=${6:-"$REPO/download/"}
OUT_DIR=${7:-"$REPO/outputs/"}
SEED=${8:-1}
echo "Fine-tuning $MODEL on $TASK using GPU $GPU in STAGE $STAGE with SETTING $SETTING"
echo "Load data from $DATA_DIR, and save models to $OUT_DIR"
if [ $TASK == "udpos" ]; then
bash $REPO/scripts/preprocess_udpos.sh $MODEL $DATA_DIR
elif [ $TASK == "panx" ]; then
bash $REPO/scripts/preprocess_panx.sh $MODEL $DATA_DIR
fi
bash $REPO/scripts/$SETTING/train_${TASK}.sh $MODEL $STAGE $GPU $DATA_DIR $OUT_DIR $SEED
xtune/scripts/translate-train-all/train_mlqa.sh
+137
View File
@@ -0,0 +1,137 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
cp -r $DATA_DIR/squad/ $DATA_DIR/mlqa/squad1.1/
TASK='mlqa'
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/SQuAD/translate-train/
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=4
MAXL=384
LANGS="en,es,de,ar,hi,vi,zh"
BSR=0.3
SA=0.3
SNBS=-1
CSR=0.3
R1_LAMBDA=5.0
R2_LAMBDA=0.5
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=4
GRAD_ACC=8
LR=1.5e-5
else
BATCH_SIZE=32
GRAD_ACC=1
LR=3e-5
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages es,de,ar,hi,vi,zh \
--noised_max_seq_length $MAXL
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-MT-R2_Lambda${R2_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method mt \
--translation_path $TRANSLATION_PATH \
--max_steps 24000 \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH
fi
xtune/scripts/translate-train-all/train_panx.sh
+138
View File
@@ -0,0 +1,138 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='panx'
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAX_LENGTH=128
LANGS="ar,he,vi,id,jv,ms,tl,eu,ml,ta,te,af,nl,en,de,el,bn,hi,mr,ur,fa,fr,it,pt,es,bg,ru,ja,ka,ko,th,sw,yo,my,zh,kk,tr,et,fi,hu"
EVALUATE_STEPS=1000
BSR=0.3
SA=0.3
SNBS=-1
R1_LAMBDA=5.0
R2_LAMBDA=1.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=32
GRAD_ACC=1
LR=7e-6
else
BATCH_SIZE=32
GRAD_ACC=1
LR=1e-5
fi
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/translate_train.panx.txt
DATA_DIR=$DATA_DIR/$TASK/${TASK}_processed_maxlen${MAX_LENGTH}/
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-MT-R2_Lambda${R2_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method mt \
--translation_path $TRANSLATION_PATH \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH \
--use_hard_labels
fi
xtune/scripts/translate-train-all/train_pawsx.sh
+119
View File
@@ -0,0 +1,119 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='pawsx'
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/PAWSX/
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAXL=256
LANGS="de,en,es,fr,ja,ko,zh"
EVALUATE_STEPS=1000
R1_LAMBDA=5.0
R2_LAMBDA=1.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=16
GRAD_ACC=2
LR=1e-5
else
BATCH_SIZE=32
GRAD_ACC=1
LR=1e-5
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-Translate-R1_LAMBDA${R1_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--enable_translate_data \
--translation_path $TRANSLATION_PATH
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-Translate-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-Translate-R1_Lambda${R1_LAMBDA}-Aug1.0-MT-R2_Lambda${R2_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--enable_translate_data \
--translation_path $TRANSLATION_PATH \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method mt \
--r2_lambda $R2_LAMBDA
fi
xtune/scripts/translate-train-all/train_tydiqa.sh
+136
View File
@@ -0,0 +1,136 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='tydiqa'
MODEL_PATH=$DATA_DIR/$MODEL
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/TyDiQA-GoldP/translate-train/
MAXL=384
LANGS="en,ar,bn,fi,id,ko,ru,sw,te"
BSR=0.3
SA=0.3
SNBS=-1
R1_LAMBDA=5.0
R2_LAMBDA=0.3
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=4
GRAD_ACC=8
LR=1.5e-5
EPOCH=10
MAX_STEPS=2500
else
BATCH_SIZE=32
GRAD_ACC=1
LR=3e-5
EPOCH=20
MAX_STEPS=5000
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-MT-R2_Lambda${R2_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method mt \
--translation_path $TRANSLATION_PATH \
--max_steps $MAX_STEPS \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH
fi
xtune/scripts/translate-train-all/train_udpos.sh
+139
View File
@@ -0,0 +1,139 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='udpos'
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAX_LENGTH=128
LANGS="af,ar,bg,de,el,en,es,et,eu,fa,fi,fr,he,hi,hu,id,it,ja,kk,ko,mr,nl,pt,ru,ta,te,th,tl,tr,ur,vi,yo,zh"
EVALUATE_STEPS=500
BSR=0.5
SA=0.3
SNBS=-1
R1_LAMBDA=5.0
R2_LAMBDA=0.3
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=32
GRAD_ACC=1
LR=5e-6
else
BATCH_SIZE=32
GRAD_ACC=1
LR=2e-5
fi
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/translate_train.udpos.txt
DATA_DIR=$DATA_DIR/$TASK/${TASK}_processed_maxlen${MAX_LENGTH}/
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--use_pooling_strategy \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-MT-R2_Lambda${R2_LAMBDA}/"
python src/run_tag.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--do_predict \
--do_predict_dev \
--predict_langs $LANGS \
--train_langs en \
--data_dir $DATA_DIR \
--labels $DATA_DIR/labels.txt \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAX_LENGTH \
--noised_max_seq_length $MAX_LENGTH \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps $EVALUATE_STEPS \
--seed $SEED \
--warmup_steps -1 \
--save_only_best_checkpoint \
--eval_all_checkpoints \
--eval_patience -1 \
--fp16 --fp16_opt_level O2 \
--hidden_dropout_prob 0.1 \
--original_loss \
--use_pooling_strategy \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--use_token_label_probs \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method mt \
--translation_path $TRANSLATION_PATH \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH
fi
xtune/scripts/translate-train-all/train_xnli.sh
+117
View File
@@ -0,0 +1,117 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
TASK='xnli'
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/XNLI/
MODEL_PATH=$DATA_DIR/$MODEL
EPOCH=10
MAXL=256
LANGS="ar,bg,de,el,en,es,fr,hi,ru,sw,th,tr,ur,vi,zh"
EVALUATE_STEPS=5000
R1_LAMBDA=5.0
R2_LAMBDA=1.0
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=16
GRAD_ACC=2
LR=5e-6
else
BATCH_SIZE=32
GRAD_ACC=1
LR=7e-6
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-Translate-R1_LAMBDA${R1_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--enable_translate_data \
--translation_path $TRANSLATION_PATH
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-Translate-R1_LAMBDA${R1_LAMBDA}/checkpoint-best"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-Translate-R1_Lambda${R1_LAMBDA}-Aug1.0-MT-R2_Lambda${R2_LAMBDA}/"
mkdir -p $OUTPUT_DIR
python ./src/run_cls.py --model_type xlmr \
--model_name_or_path $MODEL_PATH \
--language $LANGS \
--train_language en \
--do_train \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 64 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--max_seq_length $MAXL \
--output_dir $OUTPUT_DIR \
--task_name $TASK \
--save_steps -1 \
--overwrite_output_dir \
--evaluate_during_training \
--evaluate_steps $EVALUATE_STEPS \
--logging_steps 50 \
--logging_steps_in_sample -1 \
--logging_each_epoch \
--gpu_id 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--enable_translate_data \
--translation_path $TRANSLATION_PATH \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method mt \
--r2_lambda $R2_LAMBDA
fi
xtune/scripts/translate-train-all/train_xquad.sh
+135
View File
@@ -0,0 +1,135 @@
#!/bin/bash
# Copyright 2020 Google and DeepMind.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
REPO=$PWD
MODEL=${1:-"xlm-roberta-base"}
STAGE=${2:-1}
GPU=${3:-0}
DATA_DIR=${4:-"$REPO/download/"}
OUT_DIR=${5:-"$REPO/outputs/"}
SEED=${6:-1}
export CUDA_VISIBLE_DEVICES=$GPU
cp -r $DATA_DIR/squad/ $DATA_DIR/xquad/squad1.1/
TASK='xquad'
MODEL_PATH=$DATA_DIR/$MODEL
TRANSLATION_PATH=$DATA_DIR/xtreme_translations/SQuAD/translate-train/
EPOCH=4
MAXL=384
LANGS="ar,de,el,en,es,hi,ru,th,tr,vi,zh"
BSR=0.3
SA=0.3
SNBS=-1
CSR=0.3
R1_LAMBDA=5.0
R2_LAMBDA=0.1
if [ $MODEL == "xlm-roberta-large" ]; then
BATCH_SIZE=4
GRAD_ACC=8
LR=1.5e-5
else
BATCH_SIZE=32
GRAD_ACC=1
LR=3e-5
fi
if [ $STAGE == 1 ]; then
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_code_switch \
--code_switch_ratio $CSR \
--dict_dir $DATA_DIR/dicts \
--dict_languages ar,de,el,es,hi,ru,th,tr,vi,zh \
--noised_max_seq_length $MAXL
elif [ $STAGE == 2 ]; then
FIRST_STAGE_MODEL_PATH="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-CS-csr${CSR}-R1_LAMBDA${R1_LAMBDA}/"
OUTPUT_DIR="${OUT_DIR}/${TASK}/${MODEL}-LR${LR}-epoch${EPOCH}-MaxLen${MAXL}-SS-bsr${BSR}-sa${SA}-snbs${SNBS}-R1_Lambda${R1_LAMBDA}-Aug1.0-MT-R2_Lambda${R2_LAMBDA}/"
python ./src/run_qa.py --model_type xlmr \
--task_name $TASK \
--model_name_or_path $MODEL_PATH \
--do_train \
--do_eval \
--language $LANGS \
--train_language en \
--data_dir $DATA_DIR/$TASK/ \
--per_gpu_train_batch_size $BATCH_SIZE \
--gradient_accumulation_steps $GRAD_ACC \
--per_gpu_eval_batch_size 128 \
--learning_rate $LR \
--num_train_epochs $EPOCH \
--save_steps 0 \
--logging_each_epoch \
--max_seq_length $MAXL \
--doc_stride 128 \
--output_dir $OUTPUT_DIR \
--overwrite_output_dir \
--evaluate_during_training \
--logging_steps 50 \
--evaluate_steps 0 \
--seed $SEED \
--fp16 --fp16_opt_level O2 \
--warmup_steps -1 \
--enable_r1_loss \
--r1_lambda $R1_LAMBDA \
--original_loss \
--overall_ratio 1.0 \
--keep_boundary_unchanged \
--enable_bpe_sampling \
--bpe_sampling_ratio $BSR \
--sampling_alpha $SA \
--sampling_nbest_size $SNBS \
--noised_max_seq_length $MAXL \
--enable_data_augmentation \
--augment_ratio 1.0 \
--augment_method mt \
--translation_path $TRANSLATION_PATH \
--max_steps 24000 \
--r2_lambda $R2_LAMBDA \
--first_stage_model_path $FIRST_STAGE_MODEL_PATH
fi
xtune/setup.py
+135
View File
@@ -0,0 +1,135 @@
"""
Simple check list from AllenNLP repo: https://github.com/allenai/allennlp/blob/master/setup.py
To create the package for pypi.
1. Change the version in __init__.py, setup.py as well as docs/source/conf.py.
2. Commit these changes with the message: "Release: VERSION"
3. Add a tag in git to mark the release: "git tag VERSION -m'Adds tag VERSION for pypi' "
Push the tag to git: git push --tags origin master
4. Build both the sources and the wheel. Do not change anything in setup.py between
creating the wheel and the source distribution (obviously).
For the wheel, run: "python setup.py bdist_wheel" in the top level directory.
(this will build a wheel for the python version you use to build it).
For the sources, run: "python setup.py sdist"
You should now have a /dist directory with both .whl and .tar.gz source versions.
5. Check that everything looks correct by uploading the package to the pypi test server:
twine upload dist/* -r pypitest
(pypi suggest using twine as other methods upload files via plaintext.)
You may have to specify the repository url, use the following command then:
twine upload dist/* -r pypitest --repository-url=https://test.pypi.org/legacy/
Check that you can install it in a virtualenv by running:
pip install -i https://testpypi.python.org/pypi transformers
6. Upload the final version to actual pypi:
twine upload dist/* -r pypi
7. Copy the release notes from RELEASE.md to the tag in github once everything is looking hunky-dory.
8. Update the documentation commit in .circleci/deploy.sh for the accurate documentation to be displayed
9. Update README.md to redirect to correct documentation.
"""
import shutil
from pathlib import Path
from setuptools import find_packages, setup
# Remove stale transformers.egg-info directory to avoid https://github.com/pypa/pip/issues/5466
stale_egg_info = Path(__file__).parent / "transformers.egg-info"
if stale_egg_info.exists():
print(
(
"Warning: {} exists.\n\n"
"If you recently updated transformers to 3.0 or later, this is expected,\n"
"but it may prevent transformers from installing in editable mode.\n\n"
"This directory is automatically generated by Python's packaging tools.\n"
"I will remove it now.\n\n"
"See https://github.com/pypa/pip/issues/5466 for details.\n"
).format(stale_egg_info)
)
shutil.rmtree(stale_egg_info)
extras = {}
extras["mecab"] = ["mecab-python3"]
extras["sklearn"] = ["scikit-learn==0.22.1"]
extras["tf"] = ["tensorflow"]
extras["tf-cpu"] = ["tensorflow-cpu"]
extras["torch"] = ["torch"]
extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
extras["all"] = extras["serving"] + ["tensorflow", "torch"]
extras["testing"] = ["pytest", "pytest-xdist"]
extras["quality"] = ["black", "isort", "flake8"]
extras["docs"] = ["recommonmark", "sphinx", "sphinx-markdown-tables", "sphinx-rtd-theme"]
extras["dev"] = extras["testing"] + extras["quality"] + ["mecab-python3", "scikit-learn", "tensorflow", "torch"]
setup(
name="transformers",
version="2.5.1",
author="Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Sam Shleifer, Google AI Language Team Authors, Open AI team Authors, Facebook AI Authors, Carnegie Mellon University Authors",
author_email="thomas@huggingface.co",
description="State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch",
long_description="",
long_description_content_type="text/markdown",
keywords="NLP deep learning transformer pytorch tensorflow BERT GPT GPT-2 google openai CMU",
license="Apache",
url="https://github.com/huggingface/transformers",
package_dir={"": "src"},
packages=find_packages("src"),
install_requires=[
"numpy",
"tokenizers == 0.5.2",
# accessing files from S3 directly
"boto3",
# filesystem locks e.g. to prevent parallel downloads
"filelock",
# for downloading models over HTTPS
"requests",
# progress bars in model download and training scripts
"tqdm >= 4.27",
# for OpenAI GPT
"regex != 2019.12.17",
# for XLNet
"sentencepiece == 0.1.91",
# for XLM
"sacremoses",
# for ndcg
"scikit-learn == 0.22",
# for tensorboard
"tensorboardX",
# for ner
"seqeval == 0.0.12",
# for torch
"torch",
# for preprocessing
"networkx == 1.11",
],
extras_require=extras,
scripts=["transformers-cli"],
python_requires=">=3.5.0",
classifiers=[
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"Intended Audience :: Education",
"Intended Audience :: Science/Research",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.5",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
],
)
xtune/src/pequod/__init__.py
View File
xtune/src/pequod/data/__init__.py
+53
View File
@@ -0,0 +1,53 @@
import logging
from transformers.data.processors.utils import InputFeatures
logger = logging.getLogger(__name__)
def convert_examples_to_features(
processor, examples, tokenizer, max_length, label_list,
pad_token=0, pad_token_segment_id=0, mask_padding_with_zero=True):
if label_list is None: label_list = processor.get_labels()
label_map = {label: i for i, label in enumerate(label_list)}
features = []
for ex_index, example in enumerate(examples):
if ex_index % 10000 == 0:
logger.info("Writing example %d" % ex_index)
inputs = tokenizer.encode_plus(
example.text_a,
example.text_b,
add_special_tokens=True,
max_length=max_length)
input_ids, token_type_ids = inputs["input_ids"], inputs["token_type_ids"]
attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)
padding_length = max_length - len(input_ids)
input_ids = input_ids + ([pad_token] * padding_length)
attention_mask = attention_mask + ([0 if mask_padding_with_zero else 1] * padding_length)
token_type_ids = token_type_ids + ([pad_token_segment_id] * padding_length)
assert len(input_ids) == max_length, "Error with input length {} vs {}".format(len(input_ids), max_length)
assert len(attention_mask) == max_length, "Error with input length {} vs {}".format(len(attention_mask), max_length)
assert len(token_type_ids) == max_length, "Error with input length {} vs {}".format(len(token_type_ids), max_length)
label = label_map[example.label]
if ex_index < 3:
logger.info("*** Example ***")
logger.info("guid: %s" % (example.guid))
logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
logger.info("attention_mask: %s" % " ".join([str(x) for x in attention_mask]))
logger.info("token_type_ids: %s" % " ".join([str(x) for x in token_type_ids]))
logger.info("label: %s (id = %d)" % (example.label, label))
features.append(InputFeatures(
input_ids=input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
label=label))
return features
xtune/src/pequod/data/dataloader.py
View File
xtune/src/pequod/data/sampler.py
+19
View File
@@ -0,0 +1,19 @@
import torch
from torch.utils.data.sampler import Sampler
class SubSampler(Sampler):
def __init__(self, data_source, num_samples):
self.data_source = data_source
self.num_samples = num_samples
def __len__(self):
return self.num_samples
def __iter__(self):
n = len(self.data_source)
if self.num_samples <= n:
return iter(torch.randperm(n).tolist()[:self.num_samples])
return iter(torch.randint(high=n, size=(self.num_samples,), dtype=torch.int64).tolist())
xtune/src/pequod/data/utils_squad.py
+996
View File
@@ -0,0 +1,996 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Load SQuAD dataset. """
from __future__ import absolute_import, division, print_function
import json
import logging
import math
import collections
from io import open
from transformers.tokenization_bert import BasicTokenizer, whitespace_tokenize
# Required by XLNet evaluation method to compute optimal threshold (see write_predictions_extended() method)
from src.pequod.data.utils_squad_evaluate import find_all_best_thresh_v2, make_qid_to_has_ans, get_raw_scores
logger = logging.getLogger(__name__)
class SquadExample(object):
"""
A single training/test example for the Squad dataset.
For examples without an answer, the start and end position are -1.
"""
def __init__(self,
qas_id,
question_text,
doc_tokens,
orig_answer_text=None,
start_position=None,
end_position=None,
is_impossible=None):
self.qas_id = qas_id
self.question_text = question_text
self.doc_tokens = doc_tokens
self.orig_answer_text = orig_answer_text
self.start_position = start_position
self.end_position = end_position
self.is_impossible = is_impossible
def __str__(self):
return self.__repr__()
def __repr__(self):
s = ""
s += "qas_id: %s" % (self.qas_id)
s += ", question_text: %s" % (
self.question_text)
s += ", doc_tokens: [%s]" % (" ".join(self.doc_tokens))
if self.start_position:
s += ", start_position: %d" % (self.start_position)
if self.end_position:
s += ", end_position: %d" % (self.end_position)
if self.is_impossible:
s += ", is_impossible: %r" % (self.is_impossible)
return s
class InputFeatures(object):
"""A single set of features of data."""
def __init__(self,
unique_id,
example_index,
doc_span_index,
tokens,
token_to_orig_map,
token_is_max_context,
input_ids,
input_mask,
segment_ids,
cls_index,
p_mask,
paragraph_len,
start_position=None,
end_position=None,
is_impossible=None):
self.unique_id = unique_id
self.example_index = example_index
self.doc_span_index = doc_span_index
self.tokens = tokens
self.token_to_orig_map = token_to_orig_map
self.token_is_max_context = token_is_max_context
self.input_ids = input_ids
self.input_mask = input_mask
self.segment_ids = segment_ids
self.cls_index = cls_index
self.p_mask = p_mask
self.paragraph_len = paragraph_len
self.start_position = start_position
self.end_position = end_position
self.is_impossible = is_impossible
def read_squad_examples(input_file, is_training, version_2_with_negative):
"""Read a SQuAD json file into a list of SquadExample."""
with open(input_file, "r", encoding='utf-8') as reader:
input_data = json.load(reader)["data"]
def is_whitespace(c):
if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F:
return True
return False
examples = []
for entry in input_data:
for paragraph in entry["paragraphs"]:
paragraph_text = paragraph["context"]
doc_tokens = []
char_to_word_offset = []
prev_is_whitespace = True
for c in paragraph_text:
if is_whitespace(c):
prev_is_whitespace = True
else:
if prev_is_whitespace:
doc_tokens.append(c)
else:
doc_tokens[-1] += c
prev_is_whitespace = False
char_to_word_offset.append(len(doc_tokens) - 1)
for qa in paragraph["qas"]:
qas_id = qa["id"]
question_text = qa["question"]
start_position = None
end_position = None
orig_answer_text = None
is_impossible = False
if is_training:
if version_2_with_negative:
is_impossible = qa["is_impossible"]
if (len(qa["answers"]) != 1) and (not is_impossible):
raise ValueError(
"For training, each question should have exactly 1 answer.")
if not is_impossible:
answer = qa["answers"][0]
orig_answer_text = answer["text"]
answer_offset = answer["answer_start"]
answer_length = len(orig_answer_text)
start_position = char_to_word_offset[answer_offset]
end_position = char_to_word_offset[answer_offset + answer_length - 1]
# Only add answers where the text can be exactly recovered from the
# document. If this CAN'T happen it's likely due to weird Unicode
# stuff so we will just skip the example.
#
# Note that this means for training mode, every example is NOT
# guaranteed to be preserved.
actual_text = " ".join(doc_tokens[start_position:(end_position + 1)])
cleaned_answer_text = " ".join(
whitespace_tokenize(orig_answer_text))
if actual_text.find(cleaned_answer_text) == -1:
logger.warning("Could not find answer: '%s' vs. '%s'",
actual_text, cleaned_answer_text)
continue
else:
start_position = -1
end_position = -1
orig_answer_text = ""
example = SquadExample(
qas_id=qas_id,
question_text=question_text,
doc_tokens=doc_tokens,
orig_answer_text=orig_answer_text,
start_position=start_position,
end_position=end_position,
is_impossible=is_impossible)
examples.append(example)
return examples
def convert_examples_to_features(examples, tokenizer, max_seq_length,
doc_stride, max_query_length, is_training,
cls_token_at_end=False,
cls_token='[CLS]', sep_token='[SEP]', pad_token=0,
sequence_a_segment_id=0, sequence_b_segment_id=1,
cls_token_segment_id=0, pad_token_segment_id=0,
mask_padding_with_zero=True):
"""Loads a data file into a list of `InputBatch`s."""
unique_id = 1000000000
# cnt_pos, cnt_neg = 0, 0
# max_N, max_M = 1024, 1024
# f = np.zeros((max_N, max_M), dtype=np.float32)
features = []
for (example_index, example) in enumerate(examples):
# if example_index % 100 == 0:
# logger.info('Converting %s/%s pos %s neg %s', example_index, len(examples), cnt_pos, cnt_neg)
query_tokens = tokenizer.tokenize(example.question_text)
if len(query_tokens) > max_query_length:
query_tokens = query_tokens[0:max_query_length]
tok_to_orig_index = []
orig_to_tok_index = []
all_doc_tokens = []
for (i, token) in enumerate(example.doc_tokens):
orig_to_tok_index.append(len(all_doc_tokens))
sub_tokens = tokenizer.tokenize(token)
for sub_token in sub_tokens:
tok_to_orig_index.append(i)
all_doc_tokens.append(sub_token)
tok_start_position = None
tok_end_position = None
if is_training and example.is_impossible:
tok_start_position = -1
tok_end_position = -1
if is_training and not example.is_impossible:
tok_start_position = orig_to_tok_index[example.start_position]
if example.end_position < len(example.doc_tokens) - 1:
tok_end_position = orig_to_tok_index[example.end_position + 1] - 1
else:
tok_end_position = len(all_doc_tokens) - 1
(tok_start_position, tok_end_position) = _improve_answer_span(
all_doc_tokens, tok_start_position, tok_end_position, tokenizer,
example.orig_answer_text)
# The -3 accounts for [CLS], [SEP] and [SEP]
max_tokens_for_doc = max_seq_length - len(query_tokens) - 3
# We can have documents that are longer than the maximum sequence length.
# To deal with this we do a sliding window approach, where we take chunks
# of the up to our max length with a stride of `doc_stride`.
_DocSpan = collections.namedtuple( # pylint: disable=invalid-name
"DocSpan", ["start", "length"])
doc_spans = []
start_offset = 0
while start_offset < len(all_doc_tokens):
length = len(all_doc_tokens) - start_offset
if length > max_tokens_for_doc:
length = max_tokens_for_doc
doc_spans.append(_DocSpan(start=start_offset, length=length))
if start_offset + length == len(all_doc_tokens):
break
start_offset += min(length, doc_stride)
for (doc_span_index, doc_span) in enumerate(doc_spans):
tokens = []
token_to_orig_map = {}
token_is_max_context = {}
segment_ids = []
# p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)
# Original TF implem also keep the classification token (set to 0) (not sure why...)
p_mask = []
# CLS token at the beginning
if not cls_token_at_end:
tokens.append(cls_token)
segment_ids.append(cls_token_segment_id)
p_mask.append(0)
cls_index = 0
# Query
for token in query_tokens:
tokens.append(token)
segment_ids.append(sequence_a_segment_id)
p_mask.append(1)
# SEP token
tokens.append(sep_token)
segment_ids.append(sequence_a_segment_id)
p_mask.append(1)
# Paragraph
for i in range(doc_span.length):
split_token_index = doc_span.start + i
token_to_orig_map[len(tokens)] = tok_to_orig_index[split_token_index]
is_max_context = _check_is_max_context(doc_spans, doc_span_index,
split_token_index)
token_is_max_context[len(tokens)] = is_max_context
tokens.append(all_doc_tokens[split_token_index])
segment_ids.append(sequence_b_segment_id)
p_mask.append(0)
paragraph_len = doc_span.length
# SEP token
tokens.append(sep_token)
segment_ids.append(sequence_b_segment_id)
p_mask.append(1)
# CLS token at the end
if cls_token_at_end:
tokens.append(cls_token)
segment_ids.append(cls_token_segment_id)
p_mask.append(0)
cls_index = len(tokens) - 1 # Index of classification token
input_ids = tokenizer.convert_tokens_to_ids(tokens)
# The mask has 1 for real tokens and 0 for padding tokens. Only real
# tokens are attended to.
input_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)
# Zero-pad up to the sequence length.
while len(input_ids) < max_seq_length:
input_ids.append(pad_token)
input_mask.append(0 if mask_padding_with_zero else 1)
segment_ids.append(pad_token_segment_id)
p_mask.append(1)
assert len(input_ids) == max_seq_length
assert len(input_mask) == max_seq_length
assert len(segment_ids) == max_seq_length
span_is_impossible = example.is_impossible
start_position = None
end_position = None
if is_training and not span_is_impossible:
# For training, if our document chunk does not contain an annotation
# we throw it out, since there is nothing to predict.
doc_start = doc_span.start
doc_end = doc_span.start + doc_span.length - 1
out_of_span = False
if not (tok_start_position >= doc_start and
tok_end_position <= doc_end):
out_of_span = True
if out_of_span:
start_position = 0
end_position = 0
span_is_impossible = True
else:
doc_offset = len(query_tokens) + 2
start_position = tok_start_position - doc_start + doc_offset
end_position = tok_end_position - doc_start + doc_offset
if is_training and span_is_impossible:
start_position = cls_index
end_position = cls_index
if example_index < 2:
logger.info("*** Example ***")
logger.info("unique_id: %s" % (unique_id))
logger.info("example_index: %s" % (example_index))
logger.info("doc_span_index: %s" % (doc_span_index))
logger.info("tokens: %s" % " ".join(tokens))
logger.info("token_to_orig_map: %s" % " ".join([
"%d:%d" % (x, y) for (x, y) in token_to_orig_map.items()]))
logger.info("token_is_max_context: %s" % " ".join([
"%d:%s" % (x, y) for (x, y) in token_is_max_context.items()
]))
logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
logger.info(
"input_mask: %s" % " ".join([str(x) for x in input_mask]))
logger.info(
"segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
if is_training and span_is_impossible:
logger.info("impossible example")
if is_training and not span_is_impossible:
answer_text = " ".join(tokens[start_position:(end_position + 1)])
logger.info("start_position: %d" % (start_position))
logger.info("end_position: %d" % (end_position))
logger.info(
"answer: %s" % (answer_text))
features.append(
InputFeatures(
unique_id=unique_id,
example_index=example_index,
doc_span_index=doc_span_index,
tokens=tokens,
token_to_orig_map=token_to_orig_map,
token_is_max_context=token_is_max_context,
input_ids=input_ids,
input_mask=input_mask,
segment_ids=segment_ids,
cls_index=cls_index,
p_mask=p_mask,
paragraph_len=paragraph_len,
start_position=start_position,
end_position=end_position,
is_impossible=span_is_impossible))
unique_id += 1
return features
def _improve_answer_span(doc_tokens, input_start, input_end, tokenizer,
orig_answer_text):
"""Returns tokenized answer spans that better match the annotated answer."""
# The SQuAD annotations are character based. We first project them to
# whitespace-tokenized words. But then after WordPiece tokenization, we can
# often find a "better match". For example:
#
# Question: What year was John Smith born?
# Context: The leader was John Smith (1895-1943).
# Answer: 1895
#
# The original whitespace-tokenized answer will be "(1895-1943).". However
# after tokenization, our tokens will be "( 1895 - 1943 ) .". So we can match
# the exact answer, 1895.
#
# However, this is not always possible. Consider the following:
#
# Question: What country is the top exporter of electornics?
# Context: The Japanese electronics industry is the lagest in the world.
# Answer: Japan
#
# In this case, the annotator chose "Japan" as a character sub-span of
# the word "Japanese". Since our WordPiece tokenizer does not split
# "Japanese", we just use "Japanese" as the annotation. This is fairly rare
# in SQuAD, but does happen.
tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text))
for new_start in range(input_start, input_end + 1):
for new_end in range(input_end, new_start - 1, -1):
text_span = " ".join(doc_tokens[new_start:(new_end + 1)])
if text_span == tok_answer_text:
return (new_start, new_end)
return (input_start, input_end)
def _check_is_max_context(doc_spans, cur_span_index, position):
"""Check if this is the 'max context' doc span for the token."""
# Because of the sliding window approach taken to scoring documents, a single
# token can appear in multiple documents. E.g.
# Doc: the man went to the store and bought a gallon of milk
# Span A: the man went to the
# Span B: to the store and bought
# Span C: and bought a gallon of
# ...
#
# Now the word 'bought' will have two scores from spans B and C. We only
# want to consider the score with "maximum context", which we define as
# the *minimum* of its left and right context (the *sum* of left and
# right context will always be the same, of course).
#
# In the example the maximum context for 'bought' would be span C since
# it has 1 left context and 3 right context, while span B has 4 left context
# and 0 right context.
best_score = None
best_span_index = None
for (span_index, doc_span) in enumerate(doc_spans):
end = doc_span.start + doc_span.length - 1
if position < doc_span.start:
continue
if position > end:
continue
num_left_context = position - doc_span.start
num_right_context = end - position
score = min(num_left_context, num_right_context) + 0.01 * doc_span.length
if best_score is None or score > best_score:
best_score = score
best_span_index = span_index
return cur_span_index == best_span_index
RawResult = collections.namedtuple("RawResult",
["unique_id", "start_logits", "end_logits"])
def write_predictions(all_examples, all_features, all_results, n_best_size,
max_answer_length, do_lower_case, output_prediction_file,
output_nbest_file, output_null_log_odds_file, verbose_logging,
version_2_with_negative, null_score_diff_threshold):
"""Write final predictions to the json file and log-odds of null if needed."""
logger.info("Writing predictions to: %s" % (output_prediction_file))
logger.info("Writing nbest to: %s" % (output_nbest_file))
example_index_to_features = collections.defaultdict(list)
for feature in all_features:
example_index_to_features[feature.example_index].append(feature)
unique_id_to_result = {}
for result in all_results:
unique_id_to_result[result.unique_id] = result
_PrelimPrediction = collections.namedtuple( # pylint: disable=invalid-name
"PrelimPrediction",
["feature_index", "start_index", "end_index", "start_logit", "end_logit"])
all_predictions = collections.OrderedDict()
all_nbest_json = collections.OrderedDict()
scores_diff_json = collections.OrderedDict()
for (example_index, example) in enumerate(all_examples):
features = example_index_to_features[example_index]
prelim_predictions = []
# keep track of the minimum score of null start+end of position 0
score_null = 1000000 # large and positive
min_null_feature_index = 0 # the paragraph slice with min null score
null_start_logit = 0 # the start logit at the slice with min null score
null_end_logit = 0 # the end logit at the slice with min null score
for (feature_index, feature) in enumerate(features):
result = unique_id_to_result[feature.unique_id]
start_indexes = _get_best_indexes(result.start_logits, n_best_size)
end_indexes = _get_best_indexes(result.end_logits, n_best_size)
# if we could have irrelevant answers, get the min score of irrelevant
if version_2_with_negative:
feature_null_score = result.start_logits[0] + result.end_logits[0]
if feature_null_score < score_null:
score_null = feature_null_score
min_null_feature_index = feature_index
null_start_logit = result.start_logits[0]
null_end_logit = result.end_logits[0]
for start_index in start_indexes:
for end_index in end_indexes:
# We could hypothetically create invalid predictions, e.g., predict
# that the start of the span is in the question. We throw out all
# invalid predictions.
if start_index >= len(feature.tokens):
continue
if end_index >= len(feature.tokens):
continue
if start_index not in feature.token_to_orig_map:
continue
if end_index not in feature.token_to_orig_map:
continue
if not feature.token_is_max_context.get(start_index, False):
continue
if end_index < start_index:
continue
length = end_index - start_index + 1
if length > max_answer_length:
continue
prelim_predictions.append(
_PrelimPrediction(
feature_index=feature_index,
start_index=start_index,
end_index=end_index,
start_logit=result.start_logits[start_index],
end_logit=result.end_logits[end_index]))
if version_2_with_negative:
prelim_predictions.append(
_PrelimPrediction(
feature_index=min_null_feature_index,
start_index=0,
end_index=0,
start_logit=null_start_logit,
end_logit=null_end_logit))
prelim_predictions = sorted(
prelim_predictions,
key=lambda x: (x.start_logit + x.end_logit),
reverse=True)
_NbestPrediction = collections.namedtuple( # pylint: disable=invalid-name
"NbestPrediction", ["text", "start_logit", "end_logit"])
seen_predictions = {}
nbest = []
for pred in prelim_predictions:
if len(nbest) >= n_best_size:
break
feature = features[pred.feature_index]
if pred.start_index > 0: # this is a non-null prediction
tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)]
orig_doc_start = feature.token_to_orig_map[pred.start_index]
orig_doc_end = feature.token_to_orig_map[pred.end_index]
orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)]
tok_text = " ".join(tok_tokens)
# De-tokenize WordPieces that have been split off.
tok_text = tok_text.replace(" ##", "")
tok_text = tok_text.replace("##", "")
# Clean whitespace
tok_text = tok_text.strip()
tok_text = " ".join(tok_text.split())
orig_text = " ".join(orig_tokens)
final_text = get_final_text(tok_text, orig_text, do_lower_case, verbose_logging)
if final_text in seen_predictions:
continue
seen_predictions[final_text] = True
else:
final_text = ""
seen_predictions[final_text] = True
nbest.append(
_NbestPrediction(
text=final_text,
start_logit=pred.start_logit,
end_logit=pred.end_logit))
# if we didn't include the empty option in the n-best, include it
if version_2_with_negative:
if "" not in seen_predictions:
nbest.append(
_NbestPrediction(
text="",
start_logit=null_start_logit,
end_logit=null_end_logit))
# In very rare edge cases we could only have single null prediction.
# So we just create a nonce prediction in this case to avoid failure.
if len(nbest)==1:
nbest.insert(0,
_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))
# In very rare edge cases we could have no valid predictions. So we
# just create a nonce prediction in this case to avoid failure.
if not nbest:
nbest.append(
_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))
assert len(nbest) >= 1
total_scores = []
best_non_null_entry = None
for entry in nbest:
total_scores.append(entry.start_logit + entry.end_logit)
if not best_non_null_entry:
if entry.text:
best_non_null_entry = entry
probs = _compute_softmax(total_scores)
nbest_json = []
for (i, entry) in enumerate(nbest):
output = collections.OrderedDict()
output["text"] = entry.text
output["probability"] = probs[i]
output["start_logit"] = entry.start_logit
output["end_logit"] = entry.end_logit
nbest_json.append(output)
assert len(nbest_json) >= 1
if not version_2_with_negative:
all_predictions[example.qas_id] = nbest_json[0]["text"]
else:
# predict "" iff the null score - the score of best non-null > threshold
score_diff = score_null - best_non_null_entry.start_logit - (
best_non_null_entry.end_logit)
scores_diff_json[example.qas_id] = score_diff
if score_diff > null_score_diff_threshold:
all_predictions[example.qas_id] = ""
else:
all_predictions[example.qas_id] = best_non_null_entry.text
all_nbest_json[example.qas_id] = nbest_json
with open(output_prediction_file, "w") as writer:
writer.write(json.dumps(all_predictions, indent=4) + "\n")
with open(output_nbest_file, "w") as writer:
writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
if version_2_with_negative:
with open(output_null_log_odds_file, "w") as writer:
writer.write(json.dumps(scores_diff_json, indent=4) + "\n")
return all_predictions
# For XLNet (and XLM which uses the same head)
RawResultExtended = collections.namedtuple("RawResultExtended",
["unique_id", "start_top_log_probs", "start_top_index",
"end_top_log_probs", "end_top_index", "cls_logits"])
def write_predictions_extended(all_examples, all_features, all_results, n_best_size,
max_answer_length, output_prediction_file,
output_nbest_file,
output_null_log_odds_file, orig_data_file,
start_n_top, end_n_top, version_2_with_negative,
tokenizer, verbose_logging):
""" XLNet write prediction logic (more complex than Bert's).
Write final predictions to the json file and log-odds of null if needed.
Requires utils_squad_evaluate.py
"""
_PrelimPrediction = collections.namedtuple( # pylint: disable=invalid-name
"PrelimPrediction",
["feature_index", "start_index", "end_index",
"start_log_prob", "end_log_prob"])
_NbestPrediction = collections.namedtuple( # pylint: disable=invalid-name
"NbestPrediction", ["text", "start_log_prob", "end_log_prob"])
logger.info("Writing predictions to: %s", output_prediction_file)
# logger.info("Writing nbest to: %s" % (output_nbest_file))
example_index_to_features = collections.defaultdict(list)
for feature in all_features:
example_index_to_features[feature.example_index].append(feature)
unique_id_to_result = {}
for result in all_results:
unique_id_to_result[result.unique_id] = result
all_predictions = collections.OrderedDict()
all_nbest_json = collections.OrderedDict()
scores_diff_json = collections.OrderedDict()
for (example_index, example) in enumerate(all_examples):
features = example_index_to_features[example_index]
prelim_predictions = []
# keep track of the minimum score of null start+end of position 0
score_null = 1000000 # large and positive
for (feature_index, feature) in enumerate(features):
result = unique_id_to_result[feature.unique_id]
cur_null_score = result.cls_logits
# if we could have irrelevant answers, get the min score of irrelevant
score_null = min(score_null, cur_null_score)
for i in range(start_n_top):
for j in range(end_n_top):
start_log_prob = result.start_top_log_probs[i]
start_index = result.start_top_index[i]
j_index = i * end_n_top + j
end_log_prob = result.end_top_log_probs[j_index]
end_index = result.end_top_index[j_index]
# We could hypothetically create invalid predictions, e.g., predict
# that the start of the span is in the question. We throw out all
# invalid predictions.
if start_index >= feature.paragraph_len - 1:
continue
if end_index >= feature.paragraph_len - 1:
continue
if not feature.token_is_max_context.get(start_index, False):
continue
if end_index < start_index:
continue
length = end_index - start_index + 1
if length > max_answer_length:
continue
prelim_predictions.append(
_PrelimPrediction(
feature_index=feature_index,
start_index=start_index,
end_index=end_index,
start_log_prob=start_log_prob,
end_log_prob=end_log_prob))
prelim_predictions = sorted(
prelim_predictions,
key=lambda x: (x.start_log_prob + x.end_log_prob),
reverse=True)
seen_predictions = {}
nbest = []
for pred in prelim_predictions:
if len(nbest) >= n_best_size:
break
feature = features[pred.feature_index]
# XLNet un-tokenizer
# Let's keep it simple for now and see if we need all this later.
#
# tok_start_to_orig_index = feature.tok_start_to_orig_index
# tok_end_to_orig_index = feature.tok_end_to_orig_index
# start_orig_pos = tok_start_to_orig_index[pred.start_index]
# end_orig_pos = tok_end_to_orig_index[pred.end_index]
# paragraph_text = example.paragraph_text
# final_text = paragraph_text[start_orig_pos: end_orig_pos + 1].strip()
# Previously used Bert untokenizer
tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)]
orig_doc_start = feature.token_to_orig_map[pred.start_index]
orig_doc_end = feature.token_to_orig_map[pred.end_index]
orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)]
tok_text = tokenizer.convert_tokens_to_string(tok_tokens)
# Clean whitespace
tok_text = tok_text.strip()
tok_text = " ".join(tok_text.split())
orig_text = " ".join(orig_tokens)
final_text = get_final_text(tok_text, orig_text, tokenizer.do_lower_case,
verbose_logging)
if final_text in seen_predictions:
continue
seen_predictions[final_text] = True
nbest.append(
_NbestPrediction(
text=final_text,
start_log_prob=pred.start_log_prob,
end_log_prob=pred.end_log_prob))
# In very rare edge cases we could have no valid predictions. So we
# just create a nonce prediction in this case to avoid failure.
if not nbest:
nbest.append(
_NbestPrediction(text="", start_log_prob=-1e6,
end_log_prob=-1e6))
total_scores = []
best_non_null_entry = None
for entry in nbest:
total_scores.append(entry.start_log_prob + entry.end_log_prob)
if not best_non_null_entry:
best_non_null_entry = entry
probs = _compute_softmax(total_scores)
nbest_json = []
for (i, entry) in enumerate(nbest):
output = collections.OrderedDict()
output["text"] = entry.text
output["probability"] = probs[i]
output["start_log_prob"] = entry.start_log_prob
output["end_log_prob"] = entry.end_log_prob
nbest_json.append(output)
assert len(nbest_json) >= 1
assert best_non_null_entry is not None
score_diff = score_null
scores_diff_json[example.qas_id] = score_diff
# note(zhiliny): always predict best_non_null_entry
# and the evaluation script will search for the best threshold
all_predictions[example.qas_id] = best_non_null_entry.text
all_nbest_json[example.qas_id] = nbest_json
with open(output_prediction_file, "w") as writer:
writer.write(json.dumps(all_predictions, indent=4) + "\n")
with open(output_nbest_file, "w") as writer:
writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
if version_2_with_negative:
with open(output_null_log_odds_file, "w") as writer:
writer.write(json.dumps(scores_diff_json, indent=4) + "\n")
with open(orig_data_file, "r", encoding='utf-8') as reader:
orig_data = json.load(reader)["data"]
qid_to_has_ans = make_qid_to_has_ans(orig_data)
has_ans_qids = [k for k, v in qid_to_has_ans.items() if v]
no_ans_qids = [k for k, v in qid_to_has_ans.items() if not v]
exact_raw, f1_raw = get_raw_scores(orig_data, all_predictions)
out_eval = {}
find_all_best_thresh_v2(out_eval, all_predictions, exact_raw, f1_raw, scores_diff_json, qid_to_has_ans)
return out_eval
def get_final_text(pred_text, orig_text, do_lower_case, verbose_logging=False):
"""Project the tokenized prediction back to the original text."""
# When we created the data, we kept track of the alignment between original
# (whitespace tokenized) tokens and our WordPiece tokenized tokens. So
# now `orig_text` contains the span of our original text corresponding to the
# span that we predicted.
#
# However, `orig_text` may contain extra characters that we don't want in
# our prediction.
#
# For example, let's say:
# pred_text = steve smith
# orig_text = Steve Smith's
#
# We don't want to return `orig_text` because it contains the extra "'s".
#
# We don't want to return `pred_text` because it's already been normalized
# (the SQuAD eval script also does punctuation stripping/lower casing but
# our tokenizer does additional normalization like stripping accent
# characters).
#
# What we really want to return is "Steve Smith".
#
# Therefore, we have to apply a semi-complicated alignment heuristic between
# `pred_text` and `orig_text` to get a character-to-character alignment. This
# can fail in certain cases in which case we just return `orig_text`.
def _strip_spaces(text):
ns_chars = []
ns_to_s_map = collections.OrderedDict()
for (i, c) in enumerate(text):
if c == " ":
continue
ns_to_s_map[len(ns_chars)] = i
ns_chars.append(c)
ns_text = "".join(ns_chars)
return (ns_text, ns_to_s_map)
# We first tokenize `orig_text`, strip whitespace from the result
# and `pred_text`, and check if they are the same length. If they are
# NOT the same length, the heuristic has failed. If they are the same
# length, we assume the characters are one-to-one aligned.
tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
tok_text = " ".join(tokenizer.tokenize(orig_text))
start_position = tok_text.find(pred_text)
if start_position == -1:
if verbose_logging:
logger.info(
"Unable to find text: '%s' in '%s'" % (pred_text, orig_text))
return orig_text
end_position = start_position + len(pred_text) - 1
(orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text)
(tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text)
if len(orig_ns_text) != len(tok_ns_text):
if verbose_logging:
logger.info("Length not equal after stripping spaces: '%s' vs '%s'",
orig_ns_text, tok_ns_text)
return orig_text
# We then project the characters in `pred_text` back to `orig_text` using
# the character-to-character alignment.
tok_s_to_ns_map = {}
for (i, tok_index) in tok_ns_to_s_map.items():
tok_s_to_ns_map[tok_index] = i
orig_start_position = None
if start_position in tok_s_to_ns_map:
ns_start_position = tok_s_to_ns_map[start_position]
if ns_start_position in orig_ns_to_s_map:
orig_start_position = orig_ns_to_s_map[ns_start_position]
if orig_start_position is None:
if verbose_logging:
logger.info("Couldn't map start position")
return orig_text
orig_end_position = None
if end_position in tok_s_to_ns_map:
ns_end_position = tok_s_to_ns_map[end_position]
if ns_end_position in orig_ns_to_s_map:
orig_end_position = orig_ns_to_s_map[ns_end_position]
if orig_end_position is None:
if verbose_logging:
logger.info("Couldn't map end position")
return orig_text
output_text = orig_text[orig_start_position:(orig_end_position + 1)]
return output_text
def _get_best_indexes(logits, n_best_size):
"""Get the n-best logits from a list."""
index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True)
best_indexes = []
for i in range(len(index_and_score)):
if i >= n_best_size:
break
best_indexes.append(index_and_score[i][0])
return best_indexes
def _compute_softmax(scores):
"""Compute softmax probability over raw logits."""
if not scores:
return []
max_score = None
for score in scores:
if max_score is None or score > max_score:
max_score = score
exp_scores = []
total_sum = 0.0
for score in scores:
x = math.exp(score - max_score)
exp_scores.append(x)
total_sum += x
probs = []
for score in exp_scores:
probs.append(score / total_sum)
return probs
xtune/src/pequod/data/utils_squad_evaluate.py
+330
View File
@@ -0,0 +1,330 @@
""" Official evaluation script for SQuAD version 2.0.
Modified by XLNet authors to update `find_best_threshold` scripts for SQuAD V2.0
In addition to basic functionality, we also compute additional statistics and
plot precision-recall curves if an additional na_prob.json file is provided.
This file is expected to map question ID's to the model's predicted probability
that a question is unanswerable.
"""
import argparse
import collections
import json
import numpy as np
import os
import re
import string
import sys
class EVAL_OPTS():
def __init__(self, data_file, pred_file, out_file="",
na_prob_file="na_prob.json", na_prob_thresh=1.0,
out_image_dir=None, verbose=False):
self.data_file = data_file
self.pred_file = pred_file
self.out_file = out_file
self.na_prob_file = na_prob_file
self.na_prob_thresh = na_prob_thresh
self.out_image_dir = out_image_dir
self.verbose = verbose
OPTS = None
def parse_args():
parser = argparse.ArgumentParser('Official evaluation script for SQuAD version 2.0.')
parser.add_argument('data_file', metavar='data.json', help='Input data JSON file.')
parser.add_argument('pred_file', metavar='pred.json', help='Model predictions.')
parser.add_argument('--out-file', '-o', metavar='eval.json',
help='Write accuracy metrics to file (default is stdout).')
parser.add_argument('--na-prob-file', '-n', metavar='na_prob.json',
help='Model estimates of probability of no answer.')
parser.add_argument('--na-prob-thresh', '-t', type=float, default=1.0,
help='Predict "" if no-answer probability exceeds this (default = 1.0).')
parser.add_argument('--out-image-dir', '-p', metavar='out_images', default=None,
help='Save precision-recall curves to directory.')
parser.add_argument('--verbose', '-v', action='store_true')
if len(sys.argv) == 1:
parser.print_help()
sys.exit(1)
return parser.parse_args()
def make_qid_to_has_ans(dataset):
qid_to_has_ans = {}
for article in dataset:
for p in article['paragraphs']:
for qa in p['qas']:
qid_to_has_ans[qa['id']] = bool(qa['answers'])
return qid_to_has_ans
def normalize_answer(s):
"""Lower text and remove punctuation, articles and extra whitespace."""
def remove_articles(text):
regex = re.compile(r'\b(a|an|the)\b', re.UNICODE)
return re.sub(regex, ' ', text)
def white_space_fix(text):
return ' '.join(text.split())
def remove_punc(text):
exclude = set(string.punctuation)
return ''.join(ch for ch in text if ch not in exclude)
def lower(text):
return text.lower()
return white_space_fix(remove_articles(remove_punc(lower(s))))
def get_tokens(s):
if not s: return []
return normalize_answer(s).split()
def compute_exact(a_gold, a_pred):
return int(normalize_answer(a_gold) == normalize_answer(a_pred))
def compute_f1(a_gold, a_pred):
gold_toks = get_tokens(a_gold)
pred_toks = get_tokens(a_pred)
common = collections.Counter(gold_toks) & collections.Counter(pred_toks)
num_same = sum(common.values())
if len(gold_toks) == 0 or len(pred_toks) == 0:
# If either is no-answer, then F1 is 1 if they agree, 0 otherwise
return int(gold_toks == pred_toks)
if num_same == 0:
return 0
precision = 1.0 * num_same / len(pred_toks)
recall = 1.0 * num_same / len(gold_toks)
f1 = (2 * precision * recall) / (precision + recall)
return f1
def get_raw_scores(dataset, preds):
exact_scores = {}
f1_scores = {}
for article in dataset:
for p in article['paragraphs']:
for qa in p['qas']:
qid = qa['id']
gold_answers = [a['text'] for a in qa['answers']
if normalize_answer(a['text'])]
if not gold_answers:
# For unanswerable questions, only correct answer is empty string
gold_answers = ['']
if qid not in preds:
print('Missing prediction for %s' % qid)
continue
a_pred = preds[qid]
# Take max over all gold answers
exact_scores[qid] = max(compute_exact(a, a_pred) for a in gold_answers)
f1_scores[qid] = max(compute_f1(a, a_pred) for a in gold_answers)
return exact_scores, f1_scores
def apply_no_ans_threshold(scores, na_probs, qid_to_has_ans, na_prob_thresh):
new_scores = {}
for qid, s in scores.items():
pred_na = na_probs[qid] > na_prob_thresh
if pred_na:
new_scores[qid] = float(not qid_to_has_ans[qid])
else:
new_scores[qid] = s
return new_scores
def make_eval_dict(exact_scores, f1_scores, qid_list=None):
if not qid_list:
total = len(exact_scores)
return collections.OrderedDict([
('exact', 100.0 * sum(exact_scores.values()) / total),
('f1', 100.0 * sum(f1_scores.values()) / total),
('total', total),
])
else:
total = len(qid_list)
return collections.OrderedDict([
('exact', 100.0 * sum(exact_scores[k] for k in qid_list) / total),
('f1', 100.0 * sum(f1_scores[k] for k in qid_list) / total),
('total', total),
])
def merge_eval(main_eval, new_eval, prefix):
for k in new_eval:
main_eval['%s_%s' % (prefix, k)] = new_eval[k]
def plot_pr_curve(precisions, recalls, out_image, title):
plt.step(recalls, precisions, color='b', alpha=0.2, where='post')
plt.fill_between(recalls, precisions, step='post', alpha=0.2, color='b')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.xlim([0.0, 1.05])
plt.ylim([0.0, 1.05])
plt.title(title)
plt.savefig(out_image)
plt.clf()
def make_precision_recall_eval(scores, na_probs, num_true_pos, qid_to_has_ans,
out_image=None, title=None):
qid_list = sorted(na_probs, key=lambda k: na_probs[k])
true_pos = 0.0
cur_p = 1.0
cur_r = 0.0
precisions = [1.0]
recalls = [0.0]
avg_prec = 0.0
for i, qid in enumerate(qid_list):
if qid_to_has_ans[qid]:
true_pos += scores[qid]
cur_p = true_pos / float(i+1)
cur_r = true_pos / float(num_true_pos)
if i == len(qid_list) - 1 or na_probs[qid] != na_probs[qid_list[i+1]]:
# i.e., if we can put a threshold after this point
avg_prec += cur_p * (cur_r - recalls[-1])
precisions.append(cur_p)
recalls.append(cur_r)
if out_image:
plot_pr_curve(precisions, recalls, out_image, title)
return {'ap': 100.0 * avg_prec}
def run_precision_recall_analysis(main_eval, exact_raw, f1_raw, na_probs,
qid_to_has_ans, out_image_dir):
if out_image_dir and not os.path.exists(out_image_dir):
os.makedirs(out_image_dir)
num_true_pos = sum(1 for v in qid_to_has_ans.values() if v)
if num_true_pos == 0:
return
pr_exact = make_precision_recall_eval(
exact_raw, na_probs, num_true_pos, qid_to_has_ans,
out_image=os.path.join(out_image_dir, 'pr_exact.png'),
title='Precision-Recall curve for Exact Match score')
pr_f1 = make_precision_recall_eval(
f1_raw, na_probs, num_true_pos, qid_to_has_ans,
out_image=os.path.join(out_image_dir, 'pr_f1.png'),
title='Precision-Recall curve for F1 score')
oracle_scores = {k: float(v) for k, v in qid_to_has_ans.items()}
pr_oracle = make_precision_recall_eval(
oracle_scores, na_probs, num_true_pos, qid_to_has_ans,
out_image=os.path.join(out_image_dir, 'pr_oracle.png'),
title='Oracle Precision-Recall curve (binary task of HasAns vs. NoAns)')
merge_eval(main_eval, pr_exact, 'pr_exact')
merge_eval(main_eval, pr_f1, 'pr_f1')
merge_eval(main_eval, pr_oracle, 'pr_oracle')
def histogram_na_prob(na_probs, qid_list, image_dir, name):
if not qid_list:
return
x = [na_probs[k] for k in qid_list]
weights = np.ones_like(x) / float(len(x))
plt.hist(x, weights=weights, bins=20, range=(0.0, 1.0))
plt.xlabel('Model probability of no-answer')
plt.ylabel('Proportion of dataset')
plt.title('Histogram of no-answer probability: %s' % name)
plt.savefig(os.path.join(image_dir, 'na_prob_hist_%s.png' % name))
plt.clf()
def find_best_thresh(preds, scores, na_probs, qid_to_has_ans):
num_no_ans = sum(1 for k in qid_to_has_ans if not qid_to_has_ans[k])
cur_score = num_no_ans
best_score = cur_score
best_thresh = 0.0
qid_list = sorted(na_probs, key=lambda k: na_probs[k])
for i, qid in enumerate(qid_list):
if qid not in scores: continue
if qid_to_has_ans[qid]:
diff = scores[qid]
else:
if preds[qid]:
diff = -1
else:
diff = 0
cur_score += diff
if cur_score > best_score:
best_score = cur_score
best_thresh = na_probs[qid]
return 100.0 * best_score / len(scores), best_thresh
def find_best_thresh_v2(preds, scores, na_probs, qid_to_has_ans):
num_no_ans = sum(1 for k in qid_to_has_ans if not qid_to_has_ans[k])
cur_score = num_no_ans
best_score = cur_score
best_thresh = 0.0
qid_list = sorted(na_probs, key=lambda k: na_probs[k])
for i, qid in enumerate(qid_list):
if qid not in scores: continue
if qid_to_has_ans[qid]:
diff = scores[qid]
else:
if preds[qid]:
diff = -1
else:
diff = 0
cur_score += diff
if cur_score > best_score:
best_score = cur_score
best_thresh = na_probs[qid]
has_ans_score, has_ans_cnt = 0, 0
for qid in qid_list:
if not qid_to_has_ans[qid]: continue
has_ans_cnt += 1
if qid not in scores: continue
has_ans_score += scores[qid]
return 100.0 * best_score / len(scores), best_thresh, 1.0 * has_ans_score / has_ans_cnt
def find_all_best_thresh(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans):
best_exact, exact_thresh = find_best_thresh(preds, exact_raw, na_probs, qid_to_has_ans)
best_f1, f1_thresh = find_best_thresh(preds, f1_raw, na_probs, qid_to_has_ans)
main_eval['best_exact'] = best_exact
main_eval['best_exact_thresh'] = exact_thresh
main_eval['best_f1'] = best_f1
main_eval['best_f1_thresh'] = f1_thresh
def find_all_best_thresh_v2(main_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans):
best_exact, exact_thresh, has_ans_exact = find_best_thresh_v2(preds, exact_raw, na_probs, qid_to_has_ans)
best_f1, f1_thresh, has_ans_f1 = find_best_thresh_v2(preds, f1_raw, na_probs, qid_to_has_ans)
main_eval['best_exact'] = best_exact
main_eval['best_exact_thresh'] = exact_thresh
main_eval['best_f1'] = best_f1
main_eval['best_f1_thresh'] = f1_thresh
main_eval['has_ans_exact'] = has_ans_exact
main_eval['has_ans_f1'] = has_ans_f1
def main(OPTS):
with open(OPTS.data_file) as f:
dataset_json = json.load(f)
dataset = dataset_json['data']
with open(OPTS.pred_file) as f:
preds = json.load(f)
if OPTS.na_prob_file:
with open(OPTS.na_prob_file) as f:
na_probs = json.load(f)
else:
na_probs = {k: 0.0 for k in preds}
qid_to_has_ans = make_qid_to_has_ans(dataset) # maps qid to True/False
has_ans_qids = [k for k, v in qid_to_has_ans.items() if v]
no_ans_qids = [k for k, v in qid_to_has_ans.items() if not v]
exact_raw, f1_raw = get_raw_scores(dataset, preds)
exact_thresh = apply_no_ans_threshold(exact_raw, na_probs, qid_to_has_ans,
OPTS.na_prob_thresh)
f1_thresh = apply_no_ans_threshold(f1_raw, na_probs, qid_to_has_ans,
OPTS.na_prob_thresh)
out_eval = make_eval_dict(exact_thresh, f1_thresh)
if has_ans_qids:
has_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=has_ans_qids)
merge_eval(out_eval, has_ans_eval, 'HasAns')
if no_ans_qids:
no_ans_eval = make_eval_dict(exact_thresh, f1_thresh, qid_list=no_ans_qids)
merge_eval(out_eval, no_ans_eval, 'NoAns')
if OPTS.na_prob_file:
find_all_best_thresh(out_eval, preds, exact_raw, f1_raw, na_probs, qid_to_has_ans)
if OPTS.na_prob_file and OPTS.out_image_dir:
run_precision_recall_analysis(out_eval, exact_raw, f1_raw, na_probs,
qid_to_has_ans, OPTS.out_image_dir)
histogram_na_prob(na_probs, has_ans_qids, OPTS.out_image_dir, 'hasAns')
histogram_na_prob(na_probs, no_ans_qids, OPTS.out_image_dir, 'noAns')
if OPTS.out_file:
with open(OPTS.out_file, 'w') as f:
json.dump(out_eval, f)
else:
print(json.dumps(out_eval, indent=2))
return out_eval
if __name__ == '__main__':
OPTS = parse_args()
if OPTS.out_image_dir:
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
main(OPTS)
xtune/src/pequod/data/wili.py
+96
View File
@@ -0,0 +1,96 @@
"""Loading examples and features for WiLI-2018 dataset"""
import logging
import os
import torch
from transformers.data.processors.utils import (DataProcessor,
InputExample, InputFeatures)
from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,
TensorDataset)
from src.data import convert_examples_to_features
from src.io import lines_gen
logger = logging.getLogger(__name__)
_alias2lang = {}
_lang2id = {}
_langs = []
def get_alias2lang(data_dir):
if len(_alias2lang) > 0: return _alias2lang, _lang2id, _langs
for line, in lines_gen(os.path.join(data_dir, "labels-new")):
value = None
for alias in line.split(";"):
alias = alias.strip()
if alias == "": continue
if value is None: value = alias
_alias2lang[alias] = value
_langs.append(value)
for i, lang in enumerate(_langs): _lang2id[lang] = i
return _alias2lang, _lang2id, _langs
def load_and_cache_examples(args, data_dir, split, run_lang2id, tokenizer, key=""):
cache_filename = os.path.join(
data_dir, "cached_%s_%s" % (split, key))
if os.path.exists(cache_filename) and not args.overwrite_cache:
logger.info("Loading features from cached file %s" % cache_filename)
features = torch.load(cache_filename)
else:
processor = WiliProcessor()
logger.info("Creating features from dataset file at %s" % data_dir)
label_list = processor.get_labels(data_dir)
examples = processor.get_examples(data_dir, split)
logger.info("%d Examples loaded" % len(examples))
features = convert_examples_to_features(
processor, examples, tokenizer, max_length=args.max_seq_length,
label_list=label_list, pad_token_segment_id=0,
pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0])
logger.info("Saving features to cache file %s" % cache_filename)
torch.save(features, cache_filename)
# Cut dataset to test langs
alias2lang, lang2id, _ = get_alias2lang(data_dir)
test_lang_ids = {lang2id[alias2lang[lang]] for lang in run_lang2id.keys()}
wili_id2run_langid = {
lang2id[alias2lang[lang]]:val for lang, val in run_lang2id.items()}
all_input_ids, all_attention_mask = [], []
all_token_type_ids, all_labels = [], []
for f in features:
if f.label not in test_lang_ids: continue
all_input_ids.append(f.input_ids)
all_attention_mask.append(f.attention_mask)
all_token_type_ids.append(f.token_type_ids)
all_labels.append(wili_id2run_langid[f.label])
all_input_ids = torch.tensor(all_input_ids, dtype=torch.long)
all_attention_mask = torch.tensor(all_attention_mask, dtype=torch.long)
all_token_type_ids = torch.tensor(all_token_type_ids, dtype=torch.long)
all_labels = torch.tensor(all_labels, dtype=torch.long)
dataset = TensorDataset(
all_input_ids, all_attention_mask, all_token_type_ids, all_labels)
return dataset
class WiliProcessor(DataProcessor):
def get_examples(self, data_dir, split):
examples = []
filename_x = os.path.join(data_dir, "x_%s.txt" % split)
filename_y = os.path.join(data_dir, "y_%s.txt" % split)
for i, (line_x, line_y) in enumerate(lines_gen(filename_x, filename_y)):
guid = "%s-%s" % (split, i)
examples.append(
InputExample(guid=guid, text_a=line_x, text_b=None, label=line_y))
return examples
def get_labels(self, data_dir):
_, _, langs = get_alias2lang(data_dir)
return langs
xtune/src/pequod/data/xdoc.py
+156
View File
@@ -0,0 +1,156 @@
"""Loading examples and features for CLS and MLDoc"""
import logging
import os
import torch
from transformers.data.processors.utils import (DataProcessor,
InputExample, InputFeatures)
from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,
TensorDataset)
logger = logging.getLogger(__name__)
def get_processor_class(dataset_name):
if dataset_name == "MLDoc": return MLDocProcessor
elif dataset_name == "CLS": return CLSProcessor
elif dataset_name == "XNLI": return XNLIProcesser
elif dataset_name == "TriXNLI": return TriXNLIProcesser
else: raise ValueError
def xdoc_convert_examples_to_features(
processor, examples, tokenizer, max_length, label_list,
pad_token=0, pad_token_segment_id=0, mask_padding_with_zero=True):
if label_list is None: label_list = processor.get_labels()
label_map = {label: i for i, label in enumerate(label_list)}
features = []
for ex_index, example in enumerate(examples):
if ex_index % 10000 == 0:
logger.info("Writing example %d" % ex_index)
inputs = tokenizer.encode_plus(
example.text_a,
example.text_b,
add_special_tokens=True,
max_length=max_length)
input_ids, token_type_ids = inputs["input_ids"], inputs["token_type_ids"]
attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)
padding_length = max_length - len(input_ids)
input_ids = input_ids + ([pad_token] * padding_length)
attention_mask = attention_mask + ([0 if mask_padding_with_zero else 1] * padding_length)
token_type_ids = token_type_ids + ([pad_token_segment_id] * padding_length)
assert len(input_ids) == max_length, "Error with input length {} vs {}".format(len(input_ids), max_length)
assert len(attention_mask) == max_length, "Error with input length {} vs {}".format(len(attention_mask), max_length)
assert len(token_type_ids) == max_length, "Error with input length {} vs {}".format(len(token_type_ids), max_length)
label = label_map[example.label]
if ex_index < 3:
logger.info("*** Example ***")
logger.info("guid: %s" % (example.guid))
logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
logger.info("attention_mask: %s" % " ".join([str(x) for x in attention_mask]))
logger.info("token_type_ids: %s" % " ".join([str(x) for x in token_type_ids]))
logger.info("label: %s (id = %d)" % (example.label, label))
features.append(InputFeatures(
input_ids=input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
label=label))
return features
def load_and_cache_examples(args, processor, split, lang, tokenizer, key=""):
cache_filename = os.path.join(
args.data_dir, "cached_%s_%s_%s" % (split, lang, key))
if os.path.exists(cache_filename) and not args.overwrite_cache:
logger.info("Loading features from cached file %s" % cache_filename)
features = torch.load(cache_filename)
else:
logger.info("Creating features from dataset file at %s" % args.data_dir)
label_list = processor.get_labels()
examples = processor.get_examples(args.data_dir, split, lang)
logger.info("%d Examples loaded" % len(examples))
features = xdoc_convert_examples_to_features(
processor, examples, tokenizer, max_length=args.max_seq_length,
label_list=label_list, pad_token_segment_id=0,
pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0])
logger.info("Saving features to cache file %s" % cache_filename)
torch.save(features, cache_filename)
all_input_ids = torch.tensor(
[f.input_ids for f in features], dtype=torch.long)
all_attention_mask = torch.tensor(
[f.attention_mask for f in features], dtype=torch.long)
all_token_type_ids = torch.tensor(
[f.token_type_ids for f in features], dtype=torch.long)
all_labels = torch.tensor([f.label for f in features], dtype=torch.long)
dataset = TensorDataset(
all_input_ids, all_attention_mask, all_token_type_ids, all_labels)
return dataset
class XDocProcessor(DataProcessor):
"""Processor for the MLDoc dataset."""
def get_example_from_tensor_dict(self, tensor_dict):
return InputExample(
tensor_dict["idx"].numpy(),
tensor_dict["sentence"].numpy().decode("utf-8"),
str(tensor_dict["label"].numpy()))
def get_examples(self, data_dir, split, lang):
filename = "%s-%s.tsv" % (split, lang)
logger.info("LOOKING AT %s" % os.path.join(data_dir, filename))
return self._create_examples(
self._read_tsv(os.path.join(data_dir, filename)), filename)
def _create_examples(self, lines, set_type):
examples = []
for i, line in enumerate(lines):
guid = "%s-%s" % (set_type, i)
try:
label, text_a = line[0], line[1]
except IndexError:
logger.warn("IndexError while decomposing line %s" % str(line))
logger.warn("Line ignored... Loop continued...")
continue
examples.append(
InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
return examples
class MLDocProcessor(XDocProcessor):
def get_labels(self): return ["ECAT", "CCAT", "GCAT", "MCAT"]
class CLSProcessor(XDocProcessor):
def get_labels(self): return ["0", "1"]
class XNLIProcesser(XDocProcessor):
"""data format: a pair: (label, text)"""
def get_labels(self): return ["neutral", "entailment", "contradiction"]
class TriXNLIProcesser(XNLIProcesser):
"""data format: a 3-tuple: (label, text-a, text-b)"""
def _create_examples(self, lines, set_type):
examples = []
for i, line in enumerate(lines):
guid = "%s-%s" % (set_type, i)
label, text_a, text_b = line[0], line[1], line[2]
examples.append(
InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
return examples
xtune/src/pequod/data/xqa.py
+62
View File
@@ -0,0 +1,62 @@
import os
import logging
import torch
from torch.utils.data import TensorDataset
from src.pequod.data.utils_squad import (read_squad_examples,
convert_examples_to_features)
logger = logging.getLogger(__name__)
def load_and_cache_examples(args, split, lang, tokenizer, key="", evaluate=False):
cache_filename = os.path.join(
args.data_dir, "cached_%s_%s_%s" % (split, lang, key))
input_file = os.path.join(args.data_dir, "%s-%s.json" % (split, lang))
if os.path.exists(cache_filename):
logger.info("Loading features from cached file %s", cache_filename)
features = torch.load(cache_filename)
if evaluate:
examples = read_squad_examples(input_file=input_file,
is_training=not evaluate,
version_2_with_negative=args.version_2_with_negative)
else: examples = None
else:
logger.info("Creating features from dataset file at %s", input_file)
examples = read_squad_examples(input_file=input_file,
is_training=not evaluate,
version_2_with_negative=args.version_2_with_negative)
features = convert_examples_to_features(examples=examples,
tokenizer=tokenizer, max_seq_length=args.max_seq_length,
doc_stride=args.doc_stride, max_query_length=args.max_query_length,
is_training=not evaluate, cls_token=tokenizer.cls_token,
sep_token=tokenizer.sep_token)
logger.info("Saving features into cached file %s", cache_filename)
torch.save(features, cache_filename)
# Convert to Tensors and build dataset
all_input_ids = torch.tensor(
[f.input_ids for f in features], dtype=torch.long)
all_input_mask = torch.tensor(
[f.input_mask for f in features], dtype=torch.long)
all_segment_ids = torch.tensor(
[f.segment_ids for f in features], dtype=torch.long)
all_cls_index = torch.tensor(
[f.cls_index for f in features], dtype=torch.long)
all_p_mask = torch.tensor(
[f.p_mask for f in features], dtype=torch.float)
if evaluate:
all_example_index = torch.arange(all_input_ids.size(0), dtype=torch.long)
dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids,
all_example_index, all_cls_index, all_p_mask)
else:
all_start_positions = torch.tensor(
[f.start_position for f in features], dtype=torch.long)
all_end_positions = torch.tensor(
[f.end_position for f in features], dtype=torch.long)
dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids,
all_start_positions, all_end_positions, all_cls_index, all_p_mask)
return dataset, examples, features
xtune/src/pequod/data/xretrieval.py
+106
View File
@@ -0,0 +1,106 @@
"""Load examples from BUCC"""
import logging
import os
import torch
from transformers.data.processors.utils import (
DataProcessor, InputExample, InputFeatures)
from torch.utils.data import (
DataLoader, RandomSampler, SequentialSampler, TensorDataset)
logger = logging.getLogger(__name__)
def load_and_cache_examples(args, langpair, lang, tokenizer, key="", prefix="tatoeba"):
cache_dir = os.path.join(args.data_dir, "pequod_cache")
os.makedirs(cache_dir, exist_ok=True)
cache_filename = os.path.join(
cache_dir, "cached_%s_%s_%s" % (langpair, lang, key))
if os.path.exists(cache_filename) and not args.overwrite_cache:
logger.info("Loading features from cached file %s" % cache_filename)
features = torch.load(cache_filename)
else:
processer = TatoebaProcesser()
logger.info("Creating features from dataset file at %s" % args.data_dir)
examples = processer.get_examples(args.data_dir, langpair, lang, prefix)
features = TatoebaProcesser.convert_examples_to_features(
examples, tokenizer, args.max_seq_length, 0,
pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],)
#logger.info("Saving features to cache file %s" % cache_filename)
#torch.save(features, cache_filename)
all_input_ids = torch.tensor(
[f.input_ids for f in features], dtype=torch.long)
all_attention_mask = torch.tensor(
[f.attention_mask for f in features], dtype=torch.long)
all_token_type_ids = torch.tensor(
[f.token_type_ids for f in features], dtype=torch.long)
dataset = TensorDataset(
all_input_ids, all_attention_mask, all_token_type_ids)
return dataset
class TatoebaProcesser(DataProcessor):
@classmethod
def convert_examples_to_features(cls, examples, tokenizer, max_length, pad_token_segment_id, pad_token, mask_padding_with_zero=True):
features = []
for ex_index, example in enumerate(examples):
inputs = tokenizer.encode_plus(
example.text_a,
None,
add_special_tokens=True,
max_length=max_length,
)
input_ids, token_type_ids = inputs["input_ids"], inputs["token_type_ids"]
attention_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)
padding_length = max_length - len(input_ids)
input_ids = input_ids + ([pad_token] * padding_length)
attention_mask = attention_mask + ([0 if mask_padding_with_zero else 1] * padding_length)
token_type_ids = token_type_ids + ([pad_token_segment_id] * padding_length)
assert len(input_ids) == max_length, "Error with input length {} vs {}".format(len(input_ids), max_length)
assert len(attention_mask) == max_length, "Error with input length {} vs {}".format(len(attention_mask), max_length)
assert len(token_type_ids) == max_length, "Error with input length {} vs {}".format(len(token_type_ids), max_length)
if ex_index < 3:
logger.info("*** Example ***")
logger.info("guid: %s" % (example.guid))
logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
logger.info("attention_mask: %s" % " ".join([str(x) for x in attention_mask]))
logger.info("token_type_ids: %s" % " ".join([str(x) for x in token_type_ids]))
features.append(InputFeatures(
input_ids=input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
label=None,
))
return features
def get_examples(self, data_dir, langpair, lang, prefix="tatoeba"):
examples = []
if prefix == "bucc":
fn = os.path.join(data_dir, "%s.%s.txt" % (langpair, lang))
else:
fn = os.path.join(data_dir, "%s.%s" % (langpair, lang))
#fn = os.path.join(data_dir, "%s.%s.%s" % (prefix, langpair, lang))
with open(fn, encoding='utf-8') as fp:
for i, line in enumerate(fp):
line = line.strip()
examples.append(InputExample(
guid="%s-%s-%d" % (langpair, lang, i),
text_a=line,
))
return examples
xtune/src/pequod/eval/__init__.py
+89
View File
@@ -0,0 +1,89 @@
import os
import numpy as np
import torch
import inspect
from src.pequod.data.utils_squad import RawResult, write_predictions
from src.pequod.data.utils_squad_evaluate import EVAL_OPTS, main as evaluate_on_squad
def to_list(tensor):
return tensor.detach().cpu().tolist()
def score_dict_to_string(score_dict):
return " ".join([("%s:%.2f" % (k, v)) for k, v in score_dict.items()])
def score_dicts_to_latex(score_dicts):
keys = [k for k in score_dicts[0]]
return "\n".join([""] + [(
" & ".join([key] + [("%.2f" % (sd[key])) for sd in score_dicts])
) for key in keys])
def eval_classification(model, batch_dict_iter):
model.eval()
preds, labels = None, None
for batch_dict in batch_dict_iter:
label_id = batch_dict["labels"].detach().cpu().numpy()
batch_dict.pop("labels")
with torch.no_grad(): logits = model(**batch_dict)[0]
pred = logits.detach().cpu().numpy()
if preds is None: preds, labels = pred, label_id
else:
preds = np.append(preds, pred, axis=0)
labels = np.append(labels, label_id)
preds = np.argmax(preds, axis=1)
result = (preds == labels).mean()
return {"acc": result*100.0}
def eval_qa(model, batch_dict_iter, prefix="", **kwargs):
features = kwargs["all_features"]
output_dir = kwargs["output_dir"]
model.eval()
all_results = []
for batch_dict, example_indices in batch_dict_iter:
with torch.no_grad(): outputs = model(**batch_dict)
for i, example_index in enumerate(example_indices):
eval_feature = features[example_index.item()]
unique_id = int(eval_feature.unique_id)
result = RawResult(unique_id = unique_id,
start_logits = to_list(outputs[0][i]),
end_logits = to_list(outputs[1][i]))
all_results.append(result)
output_prediction_file = os.path.join(
output_dir, "predictions_{}.json".format(prefix))
output_nbest_file = os.path.join(
output_dir, "nbest_predictions_{}.json".format(prefix))
if kwargs["version_2_with_negative"]:
output_null_log_odds_file = os.path.join(
output_dir, "null_odds_{}.json".format(prefix))
else: output_null_log_odds_file = None
wrt_pred_kwargs = {
"all_results": all_results,
"output_prediction_file": output_prediction_file,
"output_nbest_file": output_nbest_file,
"output_null_log_odds_file": output_null_log_odds_file}
for key in inspect.getfullargspec(write_predictions).args:
if key not in wrt_pred_kwargs:
wrt_pred_kwargs[key] = kwargs[key]
write_predictions(**wrt_pred_kwargs)
# Evaluate with the official SQuAD script
evaluate_options = EVAL_OPTS(
data_file=kwargs["predict_file"],
pred_file=output_prediction_file,
na_prob_file=output_null_log_odds_file,
out_file="/dev/null")
results = evaluate_on_squad(evaluate_options)
return results
xtune/src/pequod/eval/bretrieval.py
+172
View File
@@ -0,0 +1,172 @@
import faiss
import json
import logging
import numpy as np
import os
import torch
from src.pequod.data.xretrieval import load_and_cache_examples
from src.pequod.eval.evaluator import Evaluator
from src.pequod.eval.utils_retrieve import mine_bitext, bucc_eval
logger = logging.getLogger(__name__)
def load_embeddings(embed_file, num_sentences=None):
logger.info(' loading from {}'.format(embed_file))
embeds = np.load(embed_file)
return embeds
class BuccEvaluator(Evaluator):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.model_langs = ["share_lang", "order"]
self.proj_matrix_fast = kwargs.get("proj_matrix_fast", None)
if self.proj_matrix_fast is not None:
logger.info("proj_matrix_fast:" + str(self.proj_matrix_fast.size()))
self.proj_matrix_fast = self.proj_matrix_fast[0].float().cuda()
self.res = {}
def get_mean_emb(self, layer_outputs, pool_mask):
embs = (layer_outputs * pool_mask.unsqueeze(2).float()).sum(dim=1) / \
pool_mask.sum(dim=1).view(-1, 1).float()
return embs
def get_cxlm_emb(self, layer_outputs):
if self.proj_matrix_fast is None:
raise ValueError
ret = torch.mm(layer_outputs[:,0,:], self.proj_matrix_fast)
# ret = layer_outputs[:,0,:]
return ret
def get_cls_emb(self, layer_outputs):
return layer_outputs[:,0,:]
def bt_norm(self, x):
m = x.mean(0, keepdim=True)
v = x.var(0, unbiased=True, keepdim=True)
return (x-m) / torch.sqrt(v+1e-5)
def get_embeddings(self, batch, outputs, emb_type=None, is_bt_norm=False):
if emb_type is None:
emb_type = self.args.emb_type
last_layer_outputs, first_token_outputs, all_layer_outputs = outputs
if emb_type == "mean":
ret = self.get_mean_emb(all_layer_outputs[self.args.mean_layer_id], batch["attention_mask"])
elif emb_type == "cls":
ret = self.get_cls_emb(all_layer_outputs[-1])
elif emb_type == "cxlm":
ret = self.get_cxlm_emb(all_layer_outputs[self.args.mean_layer_id]) #TODO
else: raise ValueError
if is_bt_norm:
ret = self.bt_norm(ret)
ret = ret.cpu().numpy().astype(np.float32)
# ret = None
del last_layer_outputs, first_token_outputs, all_layer_outputs
torch.cuda.empty_cache()
return ret
def run(self):
args = self.args
self.model.eval()
best_threshold = None
SL, TL = args.src_language, args.tgt_language
for split in ['test']:
# for split in ['dev', 'test']:
prefix = f'{SL}-{TL}.{split}'
if args.extract_embeds:
for lang in [SL, TL]:
file = os.path.join(args.output_dir, f'{prefix}.{lang}.npy')
if os.path.exists(file):
continue
langpair = f'{SL}-{TL}.{split}'
dl1 = self.get_dataloader(langpair, lang)
all_emb1 = []
for batch1 in dl1:
batch1 = self._parse_batch(batch1, has_label=False)
#forward
with torch.no_grad():
outputs1 = self.model(**batch1)
all_emb1.append(self.get_embeddings(batch1, outputs1, is_bt_norm=args.bt_norm))
all_emb1 = np.concatenate(all_emb1)
file = os.path.join(args.output_dir, f'{prefix}.{lang}.npy')
logger.info('save embed {} to file {}'.format(all_emb1.shape, file))
np.save(file, all_emb1)
if args.mine_bitext:
threshold = None
cand2score_file = os.path.join(args.output_dir, 'candidates.tsv')
x = load_embeddings(os.path.join(args.output_dir, f'{prefix}.{SL}.npy'))
y = load_embeddings(os.path.join(args.output_dir, f'{prefix}.{TL}.npy'))
x_text_file = os.path.join(args.data_dir, f'{prefix}.{SL}.txt')
y_text_file = os.path.join(args.data_dir, f'{prefix}.{TL}.txt')
x_id_file = os.path.join(args.data_dir, f'{prefix}.{SL}.id')
y_id_file = os.path.join(args.data_dir, f'{prefix}.{TL}.id')
mine_bitext(x, y, x_text_file, y_text_file, cand2score_file, dist=args.dist, use_shift_embeds=args.use_shift_embeds)
gold_file = os.path.join(args.data_dir, f'{prefix}.gold')
if os.path.exists(gold_file):
predict_file = os.path.join(args.output_dir, f'test-{SL}.tsv')
results = bucc_eval(cand2score_file, gold_file, x_text_file, y_text_file, x_id_file, y_id_file, predict_file, threshold)
with open(os.path.join(args.output_dir, 'final.txt'), 'w', encoding='utf-8') as f:
f.write(json.dumps(results))
best_threshold = results['best-threshold']
logger.info('--Candidates: {}'.format(cand2score_file))
logger.info(' '.join('{}={:.4f}'.format(k,v) for k,v in results.items()))
# if args.layer_ensemble:
# threshold = None
# prefix = 'mean_l2'
# layers = args.ens_layers.split(',')
#
# cand2score_file = os.path.join(args.output_dir, 'candidates.tsv')
#
# x = load_embeddings(os.path.join(args.output_dir, f'{prefix}.{SL}.npy'))
# y = load_embeddings(os.path.join(args.output_dir, f'{prefix}.{TL}.npy'))
#
# x_text_file = os.path.join(args.data_dir, f'{prefix}.{SL}.txt')
# y_text_file = os.path.join(args.data_dir, f'{prefix}.{TL}.txt')
# x_id_file = os.path.join(args.data_dir, f'{prefix}.{SL}.id')
# y_id_file = os.path.join(args.data_dir, f'{prefix}.{TL}.id')
#
# mine_bitext(x, y, x_text_file, y_text_file, cand2score_file, dist=args.dist, use_shift_embeds=args.use_shift_embeds)
# gold_file = os.path.join(args.data_dir, f'{prefix}.gold')
# if os.path.exists(gold_file):
# predict_file = os.path.join(args.output_dir, f'test-{SL}.tsv')
# results = bucc_eval(cand2score_file, gold_file, x_text_file, y_text_file, x_id_file, y_id_file, predict_file, threshold)
#
# with open(os.path.join(args.output_dir, 'final.txt'), 'w', encoding='utf-8') as f:
# f.write(json.dumps(results))
#
# best_threshold = results['best-threshold']
# logger.info('--Candidates: {}'.format(cand2score_file))
# logger.info(' '.join('{}={:.4f}'.format(k,v) for k,v in results.items()))
# output retrieval results
# with open(os.path.join(args.output_dir, 'test-{0}.tsv'.format(lang1)), 'w', encoding='utf-8') as writer:
# for i, pred in enumerate(predictions):
# writer.write(str(pred[0]) + '\n')
def load_and_cache_examples(self, langpair, lang, **kwargs):
args = self.args
cache_key = "%s-%s" % (args.model_key, args.model_type)
return load_and_cache_examples(
args=args,
langpair=langpair,
lang=lang,
tokenizer=self.tokenizer,
key=cache_key,
prefix=args.data_prefix,
)
xtune/src/pequod/eval/evaluator.py
+45
View File
@@ -0,0 +1,45 @@
import logging
import torch
from torch.utils.data import DataLoader
from src.pequod.training.trainer import to_cuda
logger = logging.getLogger(__name__)
class Evaluator(object):
def __init__(self, args, model, tokenizer, **kwargs):
self.args = args
self.datasets = {}
self.model = model
self.tokenizer = tokenizer
def _parse_batch(self, batch, has_label=True, **kwargs):
_batch = to_cuda(batch)
# _batch = batch
ret = {"input_ids": _batch[0],
"attention_mask": _batch[1],
"token_type_ids": _batch[2] if self.args.model_type == "bert" else None,}
if has_label: ret["labels"] = _batch[3]
ret.update(**kwargs)
return ret
def run(self):
raise NotImplementedError
def get_dataset(self, *args, **kwargs):
if args in self.datasets: return self.datasets[args]
dataset = self.load_and_cache_examples(*args, **kwargs)
self.datasets[args] = dataset
return dataset
def load_and_cache_examples(self, *args, **kwargs):
raise NotImplementedError
def get_dataloader(self, *args, **kwargs):
logger.info("Getting dataloader - args: %s" % str(args))
dataset = kwargs.pop("dataset", self.get_dataset(*args, **kwargs))
dataloader = DataLoader(dataset, batch_size=self.args.eval_batch_size)
return dataloader
xtune/src/pequod/eval/utils_retrieve.py
+340
View File
@@ -0,0 +1,340 @@
# coding=utf-8
# This repository is modified based on the LASER repository.
# https://github.com/facebookresearch/LASER
# Copyright The LASER Team Authors, and The XTREME Benchmark Authors.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Utility functions for retrieval tasks."""
import os
import sys
import faiss
import tempfile
import numpy as np
def knn(x, y, k, use_gpu, dist='cosine'):
return knnGPU(x, y, k) if use_gpu else knnCPU(x, y, k, dist)
def knnGPU(x, y, k, mem=5*1024*1024*1024):
dim = x.shape[1]
batch_size = mem // (dim*4)
sim = np.zeros((x.shape[0], k), dtype=np.float32)
ind = np.zeros((x.shape[0], k), dtype=np.int64)
for xfrom in range(0, x.shape[0], batch_size):
xto = min(xfrom + batch_size, x.shape[0])
bsims, binds = [], []
for yfrom in range(0, y.shape[0], batch_size):
yto = min(yfrom + batch_size, y.shape[0])
print('{}-{} -> {}-{}'.format(xfrom, xto, yfrom, yto))
idx = faiss.IndexFlatIP(dim)
idx = faiss.index_cpu_to_all_gpus(idx)
idx.add(y[yfrom:yto])
bsim, bind = idx.search(x[xfrom:xto], min(k, yto-yfrom))
bsims.append(bsim)
binds.append(bind + yfrom)
del idx
bsims = np.concatenate(bsims, axis=1)
binds = np.concatenate(binds, axis=1)
aux = np.argsort(-bsims, axis=1)
for i in range(xfrom, xto):
for j in range(k):
sim[i, j] = bsims[i-xfrom, aux[i-xfrom, j]]
ind[i, j] = binds[i-xfrom, aux[i-xfrom, j]]
return sim, ind
def knnCPU(x, y, k, dist='cosine'):
# x: query, y: database
dim = x.shape[1]
if dist == 'cosine':
idx = faiss.IndexFlatIP(dim)
else:
idx = faiss.IndexFlatL2(dim)
idx.add(y)
sim, ind = idx.search(x, k)
if dist != 'cosine':
sim = 1 / (1 + sim)
return sim, ind
def score(x, y, fwd_mean, bwd_mean, margin, dist='cosine'):
if dist == 'cosine':
return margin(x.dot(y), (fwd_mean + bwd_mean) / 2)
else:
l2 = ((x - y) ** 2).sum()
sim = 1 / (1 + l2)
return margin(sim, (fwd_mean + bwd_mean) / 2)
def score_candidates(x, y, candidate_inds, fwd_mean, bwd_mean, margin, dist='cosine'):
print(' - scoring {:d} candidates using {}'.format(x.shape[0], dist))
scores = np.zeros(candidate_inds.shape)
for i in range(scores.shape[0]):
for j in range(scores.shape[1]):
k = candidate_inds[i, j]
scores[i, j] = score(x[i], y[k], fwd_mean[i], bwd_mean[k], margin, dist)
return scores
def text_load_unify(fname, encoding, unify=True):
print(' - loading texts {:s}: '.format(fname), end='')
fin = open(fname, encoding=encoding, errors='surrogateescape')
inds = []
sents = []
sent2ind = {}
n = 0
nu = 0
for line in fin:
new_ind = len(sent2ind)
inds.append(sent2ind.setdefault(line, new_ind))
if unify:
if inds[-1] == new_ind:
sents.append(line[:-1])
nu += 1
else:
sents.append(line[:-1])
nu += 1
n += 1
print('{:d} lines, {:d} unique'.format(n, nu))
del sent2ind
return inds, sents
def unique_embeddings(emb, ind):
aux = {j: i for i, j in enumerate(ind)}
print(' - unify embeddings: {:d} -> {:d}'.format(len(emb), len(aux)))
return emb[[aux[i] for i in range(len(aux))]]
def shift_embeddings(x, y):
print(' - shift embeddings')
delta = x.mean(axis=0) - y.mean(axis=0)
x2y = x - delta
y2x = y + delta
return x2y, y2x
def mine_bitext(x, y, src_text_file, trg_text_file, output_file, mode='mine',
retrieval='max', margin='ratio', threshold=0,
neighborhood=4, use_gpu=False, encoding='utf-8', dist='cosine', use_shift_embeds=False):
src_inds, src_sents = text_load_unify(src_text_file, encoding, True)
trg_inds, trg_sents = text_load_unify(trg_text_file, encoding, True)
x = unique_embeddings(x, src_inds)
y = unique_embeddings(y, trg_inds)
if dist == 'cosine':
faiss.normalize_L2(x)
faiss.normalize_L2(y)
if use_shift_embeds:
x2y, y2x = shift_embeddings(x, y)
# calculate knn in both directions
if retrieval is not 'bwd':
print(' - perform {:d}-nn source against target, dist={}'.format(neighborhood, dist))
if use_shift_embeds:
# project x to y space, and search k-nn ys for each x
x2y_sim, x2y_ind = knn(x2y, y, min(y.shape[0], neighborhood), use_gpu, dist)
x2y_mean = x2y_sim.mean(axis=1)
else:
x2y_sim, x2y_ind = knn(x, y, min(y.shape[0], neighborhood), use_gpu, dist)
x2y_mean = x2y_sim.mean(axis=1)
if retrieval is not 'fwd':
print(' - perform {:d}-nn target against source, dist={}'.format(neighborhood, dist))
if use_shift_embeds:
y2x_sim, y2x_ind = knn(y2x, x, min(x.shape[0], neighborhood), use_gpu, dist)
y2x_mean = y2x_sim.mean(axis=1)
else:
y2x_sim, y2x_ind = knn(y, x, min(x.shape[0], neighborhood), use_gpu, dist)
y2x_mean = y2x_sim.mean(axis=1)
# margin function
if margin == 'absolute':
margin = lambda a, b: a
elif margin == 'distance':
margin = lambda a, b: a - b
else: # margin == 'ratio':
margin = lambda a, b: a / b
fout = open(output_file, mode='w', encoding=encoding, errors='surrogateescape')
if mode == 'search':
print(' - Searching for closest sentences in target')
print(' - writing alignments to {:s}'.format(output_file))
scores = score_candidates(x, y, x2y_ind, x2y_mean, y2x_mean, margin)
best = x2y_ind[np.arange(x.shape[0]), scores.argmax(axis=1)]
nbex = x.shape[0]
ref = np.linspace(0, nbex-1, nbex).astype(int) # [0, nbex)
err = nbex - np.equal(best.reshape(nbex), ref).astype(int).sum()
print(' - errors: {:d}={:.2f}%'.format(err, 100*err/nbex))
for i in src_inds:
print(trg_sents[best[i]], file=fout)
elif mode == 'score':
for i, j in zip(src_inds, trg_inds):
s = score(x[i], y[j], x2y_mean[i], y2x_mean[j], margin)
print(s, src_sents[i], trg_sents[j], sep='\t', file=fout)
elif mode == 'mine':
print(' - mining for parallel data')
if use_shift_embeds:
fwd_scores = score_candidates(x2y, y, x2y_ind, x2y_mean, y2x_mean, margin)
bwd_scores = score_candidates(y2x, x, y2x_ind, y2x_mean, x2y_mean, margin)
else:
fwd_scores = score_candidates(x, y, x2y_ind, x2y_mean, y2x_mean, margin)
bwd_scores = score_candidates(y, x, y2x_ind, y2x_mean, x2y_mean, margin)
fwd_best = x2y_ind[np.arange(x.shape[0]), fwd_scores.argmax(axis=1)]
bwd_best = y2x_ind[np.arange(y.shape[0]), bwd_scores.argmax(axis=1)]
print(' - writing alignments to {:s}'.format(output_file))
if threshold > 0:
print(' - with threshold of {:f}'.format(threshold))
if retrieval == 'fwd':
for i, j in enumerate(fwd_best):
print(fwd_scores[i].max(), src_sents[i], trg_sents[j], sep='\t', file=fout)
if retrieval == 'bwd':
for j, i in enumerate(bwd_best):
print(bwd_scores[j].max(), src_sents[i], trg_sents[j], sep='\t', file=fout)
if retrieval == 'intersect':
for i, j in enumerate(fwd_best):
if bwd_best[j] == i:
print(fwd_scores[i].max(), src_sents[i], trg_sents[j], sep='\t', file=fout)
if retrieval == 'max':
indices = np.stack((np.concatenate((np.arange(x.shape[0]), bwd_best)),
np.concatenate((fwd_best, np.arange(y.shape[0])))), axis=1)
scores = np.concatenate((fwd_scores.max(axis=1), bwd_scores.max(axis=1)))
seen_src, seen_trg = set(), set()
for i in np.argsort(-scores):
src_ind, trg_ind = indices[i]
if not src_ind in seen_src and not trg_ind in seen_trg:
seen_src.add(src_ind)
seen_trg.add(trg_ind)
if scores[i] > threshold:
print(scores[i], src_sents[src_ind], trg_sents[trg_ind], sep='\t', file=fout)
fout.close()
def bucc_optimize(candidate2score, gold):
items = sorted(candidate2score.items(), key=lambda x: -x[1])
ngold = len(gold)
nextract = ncorrect = 0
threshold = 0
best_f1 = 0
for i in range(len(items)):
nextract += 1
if '\t'.join(items[i][0]) in gold:
ncorrect += 1
if ncorrect > 0:
precision = ncorrect / nextract
recall = ncorrect / ngold
f1 = 2 * precision * recall / (precision + recall)
if f1 > best_f1:
best_f1 = f1
threshold = (items[i][1] + items[i + 1][1]) / 2
return threshold
def bucc_extract(cand2score, th, fname):
if fname:
of = open(fname, 'w', encoding='utf-8')
bitexts = []
for (src, trg), score in cand2score.items():
if score >= th:
bitexts.append(src + '\t' + trg)
if fname:
of.write(src + '\t' + trg + '\n')
if fname:
of.close()
return bitexts
def read_sent2id(text_file, id_file, encoding='utf-8'):
repeated = set()
sent2id = {}
with open(id_file, encoding=encoding, errors='surrogateescape') as f:
ids = [l.strip() for l in f]
with open(text_file, encoding=encoding, errors='surrogateescape') as f:
sentences = [l.strip() for l in f]
for id, sent in zip(ids, sentences):
if sent in sent2id:
repeated.add(sent)
else:
sent2id[sent] = id
for sent in repeated:
del sent2id[sent]
return sent2id
def read_candidate2score(candidates_file, src_text_file, trg_text_file, src_id_file, trg_id_file, encoding='utf-8'):
print(' - reading sentences {}'.format(candidates_file))
src_sent2id = read_sent2id(src_text_file, src_id_file, encoding)
trg_sent2id = read_sent2id(trg_text_file, trg_id_file, encoding)
print(' - reading candidates {}'.format(candidates_file))
candidate2score = {}
with open(candidates_file, encoding=encoding, errors='surrogateescape') as f:
for line in f:
score, src, trg = line.split('\t')
score = float(score)
src = src.strip()
trg = trg.strip()
if src in src_sent2id and trg in trg_sent2id:
src_id = src_sent2id[src]
trg_id = trg_sent2id[trg]
score = max(score, candidate2score.get((src_id, trg_id), score))
candidate2score[(src_id, trg_id)] = score
return candidate2score
def bucc_eval(candidates_file, gold_file, src_file, trg_file, src_id_file, trg_id_file, predict_file, threshold=None, encoding='utf-8'):
candidate2score = read_candidate2score(candidates_file, src_file, trg_file, src_id_file, trg_id_file, encoding)
if threshold is not None and gold_file is None:
print(' - using threshold {}'.format(threshold))
else:
print(' - optimizing threshold on gold alignments {}'.format(gold_file))
gold = {line.strip() for line in open(gold_file)}
threshold = bucc_optimize(candidate2score, gold)
bitexts = bucc_extract(candidate2score, threshold, predict_file)
if gold_file is not None:
ncorrect = len(gold.intersection(bitexts))
if ncorrect > 0:
precision = ncorrect / len(bitexts)
recall = ncorrect / len(gold)
f1 = 2*precision*recall / (precision + recall)
else:
precision = recall = f1 = 0
print(' - best threshold={:f}: precision={:.2f}, recall={:.2f}, F1={:.2f}'
.format(threshold, 100*precision, 100*recall, 100*f1))
return {'best-threshold': threshold, 'precision': 100*precision, 'recall': 100*recall, 'F1': 100*f1}
else:
return None
def similarity_search(x, y, dim, normalize=False):
num = x.shape[0]
idx = faiss.IndexFlatL2(dim)
if normalize:
faiss.normalize_L2(x)
faiss.normalize_L2(y)
idx.add(x)
scores, prediction = idx.search(y, 1)
return prediction
xtune/src/pequod/eval/xretrieval.py
+176
View File
@@ -0,0 +1,176 @@
import faiss
import json
import logging
import numpy as np
import os
import torch
from src.pequod.data.xretrieval import load_and_cache_examples
from src.pequod.eval.evaluator import Evaluator
logger = logging.getLogger(__name__)
def similarity_search(x, y, dim, normalize=False, dist='L2'):
top_k = 10
num = x.shape[0]
if dist == 'cosine':
idx = faiss.IndexFlatIP(dim)
else:
idx = faiss.IndexFlatL2(dim)
if normalize:
faiss.normalize_L2(x)
faiss.normalize_L2(y)
idx.add(x)
scores, prediction = idx.search(y, top_k)
return prediction, scores
class TatoebaEvaluator(Evaluator):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.model_langs = ["share_lang", "order"]
self.proj_matrix_fast = kwargs.get("proj_matrix_fast", None)
if self.proj_matrix_fast is not None:
logger.info("proj_matrix_fast:" + str(self.proj_matrix_fast.size()))
self.proj_matrix_fast = self.proj_matrix_fast[0].float().cuda()
self.res = {}
def get_mean_emb(self, layer_outputs, pool_mask):
embs = (layer_outputs * pool_mask.unsqueeze(2).float()).sum(dim=1) / \
pool_mask.sum(dim=1).view(-1, 1).float()
return embs
def get_cxlm_emb(self, layer_outputs):
if self.proj_matrix_fast is None:
raise ValueError
ret = torch.mm(layer_outputs[:,0,:], self.proj_matrix_fast)
# ret = layer_outputs[:,0,:]
return ret
def get_cls_emb(self, layer_outputs):
return layer_outputs[:,0,:]
def bt_norm(self, x):
m = x.mean(0, keepdim=True)
v = x.var(0, unbiased=True, keepdim=True)
return (x-m) / torch.sqrt(v+1e-5)
def get_embeddings(self, batch, outputs, emb_type=None, is_bt_norm=False):
if emb_type is None:
emb_type = self.args.emb_type
last_layer_outputs, first_token_outputs, all_layer_outputs = outputs
if emb_type == "mean":
ret = self.get_mean_emb(all_layer_outputs[self.args.mean_layer_id], batch["attention_mask"])
elif emb_type == "cls":
ret = self.get_cls_emb(all_layer_outputs[-1])
elif emb_type == "cxlm":
ret = self.get_cxlm_emb(all_layer_outputs[self.args.mean_layer_id]) #TODO
else: raise ValueError
if is_bt_norm:
ret = self.bt_norm(ret)
ret = ret.cpu().numpy().astype(np.float32)
# ret = None
del last_layer_outputs, first_token_outputs, all_layer_outputs
torch.cuda.empty_cache()
return ret
def run(self):
args = self.args
self.model.eval()
if args.data_prefix == "tatoeba":
langs = ["ara", "bul", "deu", "ell", "spa", "fra", "hin", "rus", "swh", "tha", "tur", "urd", "vie", "cmn"]
langpairs = ["%s-eng" % lang for lang in langs]
elif args.data_prefix == "cxlm":
langpairs = "ar-en bg-en de-en el-en en-es en-fr en-hi en-ru en-sw en-th en-tr en-ur en-vi en-zh".split()
elif args.data_prefix == "debug":
langpairs = ["ar-en" ]
elif args.data_prefix == "tat15plus":
args.data_prefix = "tatoeba"
l15 = set(["ara", "bul", "deu", "ell", "spa", "fra", "hin", "rus", "swh", "tha", "tur", "urd", "vie", "cmn"])
ld = {'ara':'ar', 'heb':'he', 'vie':'vi', 'ind':'id',
'jav':'jv', 'tgl':'tl', 'eus':'eu', 'mal':'ml',
'tel':'te', 'afr':'af', 'nld':'nl', 'deu':'de',
'ell':'el', 'ben':'bn', 'hin':'hi', 'mar':'mr', 'urd':'ur',
'tam':'ta', 'fra':'fr', 'ita':'it', 'por':'pt', 'spa':'es',
'bul':'bg', 'rus':'ru', 'jpn':'ja', 'kat':'ka', 'kor':'ko',
'tha':'th', 'swh':'sw', 'cmn':'zh', 'kaz':'kk', 'tur':'tr',
'est':'et', 'fin':'fi', 'hun':'hu', 'pes':'fa'}
langs_str = 'ar he vi id jv tl eu ml ta te af nl de el bn hi mr ur fa fr it pt es bg ru ja ka ko th sw zh kk tr et fi hu'
#langs_str = 'hi mr ur fa fr it pt es bg ru ja ka ko th sw zh kk tr et fi hu'
#langs_str = 'ar he'
#langs_str = 'ara heb'
langs = langs_str.split(' ')
#for l in ld:
# if l in l15: continue
# langs.append(l)
# langs = ["afr", "jpn", "kor", "kaz", "est", "fin", "hun", "pes"]
langpairs = ["%s-en" % lang for lang in langs]
else: raise ValueError
for langpair in langpairs:
lang1, lang2 = langpair.split("-")
logger.info("Eval langpair: %s" % langpair)
dl1 = self.get_dataloader(langpair, lang1)
dl2 = self.get_dataloader(langpair, lang2)
all_emb1 = []
all_emb2 = []
for batch1, batch2 in zip(dl1, dl2):
batch1 = self._parse_batch(batch1, has_label=False)
batch2 = self._parse_batch(batch2, has_label=False)
#forward
with torch.no_grad():
outputs1 = self.model(**batch1)
all_emb1.append(self.get_embeddings(batch1, outputs1, is_bt_norm=args.bt_norm))
outputs2 = self.model(**batch2)
all_emb2.append(self.get_embeddings(batch2, outputs2, is_bt_norm=args.bt_norm))
all_emb1 = np.concatenate(all_emb1)
all_emb2 = np.concatenate(all_emb2)
emb_sz = all_emb1.shape[-1]
if args.reverse_eval:
all_emb1, all_emb2 = all_emb2, all_emb1
predictions, scores = similarity_search(
all_emb1, all_emb2, emb_sz, normalize=self.args.normalize, dist=self.args.dist)
correct = tot = 0
# output retrieval results
with open(os.path.join(args.output_dir, 'test-{0}.tsv'.format(lang1)), 'w', encoding='utf-8') as writer:
for i, pred in enumerate(predictions):
writer.write(str(pred[0]) + '\n')
with open(os.path.join(args.output_dir, 'test-{0}-scores.tsv'.format(lang1)), 'w', encoding='utf-8') as writer:
for pred, score in zip(predictions, scores):
writer.write(' '.join([str(p) for p in pred]) + '\t' + ' '.join([str(s) for s in score]) + '\n')
for i, pred in enumerate(predictions):
if i == pred[0]: correct += 1
tot += 1
logger.info("langpair:%s acc:%.2f" % (langpair, 100*correct/tot))
self.res[langpair] = 100*correct/tot
#output_fn = os.path.join(args.exp_results_dir, args.exp_name)
#if args.reverse_eval: output_fn += "-rev"
#with open(output_fn, "w") as fp:
# json.dump(self.res, fp)
def load_and_cache_examples(self, langpair, lang, **kwargs):
args = self.args
cache_key = "%s-%s" % (args.model_key, args.model_type)
return load_and_cache_examples(
args=args,
langpair=langpair,
lang=lang,
tokenizer=self.tokenizer,
key=cache_key,
prefix=args.data_prefix,
)
xtune/src/pequod/io.py
+9
View File
@@ -0,0 +1,9 @@
"""I/O"""
def _lines_gen_from_single_file(filename):
with open(filename) as fp:
for line in fp: yield line.strip()
def lines_gen(*filenames):
for ret in zip(*map(_lines_gen_from_single_file, filenames)): yield ret
xtune/src/pequod/model/__init__.py
View File
xtune/src/pequod/model/roberta.py
+53
View File
@@ -0,0 +1,53 @@
import torch
from torch import nn
from torch.nn import CrossEntropyLoss
from transformers.modeling_bert import BertPreTrainedModel, BertForQuestionAnswering
from transformers.modeling_roberta import RobertaModel
class RobertaForQuestionAnswering(BertPreTrainedModel):
base_model_prefix = "roberta"
def __init__(self, config):
BertPreTrainedModel.__init__(self, config)
self.num_labels = config.num_labels
self.roberta = RobertaModel(config)
self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)
BertPreTrainedModel.init_weights(self)
def forward(self, input_ids, attention_mask=None, token_type_ids=None, position_ids=None, head_mask=None, start_positions=None, end_positions=None, **kwargs):
outputs = self.roberta(input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
position_ids=position_ids,
head_mask=head_mask,
**kwargs)
sequence_output = outputs[0]
logits = self.qa_outputs(sequence_output)
start_logits, end_logits = logits.split(1, dim=-1)
start_logits = start_logits.squeeze(-1)
end_logits = end_logits.squeeze(-1)
outputs = (start_logits, end_logits,) + outputs[2:]
if start_positions is not None and end_positions is not None:
# If we are on multi-GPU, split add a dimension
if len(start_positions.size()) > 1:
start_positions = start_positions.squeeze(-1)
if len(end_positions.size()) > 1:
end_positions = end_positions.squeeze(-1)
# sometimes the start/end positions are outside our model inputs, we ignore these terms
ignored_index = start_logits.size(1)
start_positions.clamp_(0, ignored_index)
end_positions.clamp_(0, ignored_index)
loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
start_loss = loss_fct(start_logits, start_positions)
end_loss = loss_fct(end_logits, end_positions)
total_loss = (start_loss + end_loss) / 2
outputs = (total_loss,) + outputs
return outputs # (loss), start_logits, end_logits, (hidden_states), (attentions)
xtune/src/pequod/optim/__init__.py
View File
xtune/src/pequod/optim/la.py
+62
View File
@@ -0,0 +1,62 @@
from collections import defaultdict
import torch
from torch.optim.optimizer import Optimizer
class LookaheadWrapper(Optimizer):
r"""Implements a Lookahead wrapper around a given optimizer
"""
def __init__(self, optimizer, la_steps, la_alpha=0.5):
self.optimizer = optimizer
self._la_step = 0 # counter for inner optimizer
self.la_alpha = la_alpha
self._total_la_steps = la_steps
self.state = defaultdict(dict)
# Cache the current optimizer parameters
for group in optimizer.param_groups:
for p in group['params']:
param_state = self.state[p]
param_state['cached_params'] = torch.zeros_like(p.data)
param_state['cached_params'].copy_(p.data)
def __getstate__(self):
return self.optimizer.__getstate__()
def __setstate__(self, state):
self.optimizer.__setstate__(state)
def zero_grad(self):
self.optimizer.zero_grad()
def state_dict(self):
return self.optimizer.state_dict()
def load_state_dict(self, state_dict):
self.optimizer.load_state_dict(state_dict)
@property
def param_groups(self):
return self.optimizer.param_groups
def step(self, closure=None):
"""Performs a single Lookahead optimization step.
Arguments:
closure (callable, optional): A closure that reevaluates the model
and returns the loss.
"""
loss = self.optimizer.step(closure)
self._la_step += 1
if self._la_step >= self._total_la_steps:
self._la_step = 0
# Lookahead and cache the current optimizer parameters
for group in self.optimizer.param_groups:
for p in group['params']:
param_state = self.state[p]
p.data.mul_(self.la_alpha).add_(1 - self.la_alpha, param_state['cached_params'])
param_state['cached_params'].copy_(p.data)
return loss
xtune/src/pequod/optim/la0.py
+62
View File
@@ -0,0 +1,62 @@
from collections import defaultdict
import torch
from torch.optim.optimizer import Optimizer
class Lookahead0Wrapper(Optimizer):
r"""Implements a Lookahead wrapper around a given optimizer
"""
def __init__(self, optimizer, la_steps, la_alpha=0.5):
self.optimizer = optimizer
self._la_step = 0 # counter for inner optimizer
self.la_alpha = la_alpha
self._total_la_steps = la_steps
self.state = defaultdict(dict)
# Cache the current optimizer parameters
for group in optimizer.param_groups:
for p in group['params']:
param_state = self.state[p]
param_state['cached_params'] = torch.zeros_like(p.data)
param_state['cached_params'].copy_(p.data)
def __getstate__(self):
return self.optimizer.__getstate__()
def __setstate__(self, state):
self.optimizer.__setstate__(state)
def zero_grad(self):
self.optimizer.zero_grad()
def state_dict(self):
return self.optimizer.state_dict()
def load_state_dict(self, state_dict):
self.optimizer.load_state_dict(state_dict)
@property
def param_groups(self):
return self.optimizer.param_groups
def step(self, closure=None):
"""Performs a single Lookahead optimization step.
Arguments:
closure (callable, optional): A closure that reevaluates the model
and returns the loss.
"""
loss = self.optimizer.step(closure)
self._la_step += 1
if self._la_step >= self._total_la_steps:
self._la_step = 0
# Lookahead and cache the current optimizer parameters
for group in self.optimizer.param_groups:
for p in group['params']:
param_state = self.state[p]
p.data.mul_(self.la_alpha).add_(1 - self.la_alpha, param_state['cached_params'])
# param_state['cached_params'].copy_(p.data)
return loss
xtune/src/pequod/text/__init__.py
View File
xtune/src/pequod/text/tokenization_sentencepiece.py
+103
View File
@@ -0,0 +1,103 @@
import os
import logging
import sentencepiece as spm
from transformers.tokenization_utils import PreTrainedTokenizer
logger = logging.getLogger(__name__)
class XLMRTokenizer(PreTrainedTokenizer):
def __init__(self, bpe_file, dict_file, **kwargs):
super(XLMRTokenizer, self).__init__(
bos_token="<s>",
eos_token="</s>",
unk_token="<unk>",
pad_token="<pad>",
mask_token="<mask>",
sep_token="</s>",
cls_token="<s>",
**kwargs)
self.max_len_single_sentence = self.max_len - 2
self.max_len_sentences_pair = self.max_len - 4
self.sp = spm.SentencePieceProcessor()
self.sp.Load(bpe_file)
self.encoder = {}
self.decoder = []
for token in [self.bos_token, self.pad_token, self.eos_token, self.unk_token]:
self._add_token(token)
with open(dict_file, encoding="utf-8") as fp:
for line in fp:
# NOTE DO NOT USE .split()
tokens_cnt = line.rstrip().split(" ")
try:
assert len(tokens_cnt) >= 2, line
except AssertionError:
logger.error(
"tokenizer line %s asserterror, replaced as <unk-%d>" % (
line, len(self.decoder)))
exit(0)
self._add_token(" ".join(tokens_cnt[:-1]))
def _add_token(self, token):
idx = len(self.encoder)
self.encoder[token] = idx
self.decoder.append(token)
def _tokenize(self, text):
return self.sp.EncodeAsPieces(text)
def _convert_id_to_token(self, index):
return self.decoder[index]
def _convert_token_to_id(self, token):
return self.encoder.get(token, self.encoder.get(self.unk_token))
def convert_tokens_to_string(self, tokens):
return "".join(tokens).replace('\u2581', ' ').strip()
@classmethod
def from_pretrained(cls, model_path, **kwargs):
bpe_file = os.path.join(model_path, "sentencepiece.bpe.model")
dict_file = os.path.join(model_path, "dict.txt")
tokenizer = cls(bpe_file, dict_file)
return tokenizer
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
if token_ids_1 is None:
return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
cls = [self.cls_token_id]
sep = [self.sep_token_id]
return cls + token_ids_0 + sep + sep + token_ids_1 + sep
def get_special_tokens_mask(self, token_ids_0, token_ids_1=None, already_has_special_tokens=False):
if already_has_special_tokens:
if token_ids_1 is not None:
raise ValueError("You should not supply a second sequence if the provided sequence of ids is already formated with special tokens for the model.")
return list(map(lambda x: 1 if x in [self.sep_token_id, self.cls_token_id] else 0, token_ids_0))
if token_ids_1 is None:
return [1] + ([0] * len(token_ids_0)) + [1]
return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
def create_token_type_ids_from_sequences(self, token_ids_0, token_ids_1=None):
sep = [self.sep_token_id]
cls = [self.cls_token_id]
if token_ids_1 is None:
return len(cls + token_ids_0 + sep) * [0]
return len(cls + token_ids_0 + sep) * [0] + len(sep + token_ids_1 + sep) * [1]
if __name__ == "__main__":
tokenizer = XLMRTokenizer.from_pretrained("/home/v-zechi/data/unilm/zechi/exp/bert_data/xlmr-large")
for text in ["Hello world!", "你好，世界", "नमस्ते दुनिया", "مرحبا بالعالم", "Bonjour le monde"]:
print(tokenizer.tokenize(text))
print(tokenizer.encode_plus(text, text, add_special_tokens=True))
xtune/src/pequod/tools/__init__.py
View File
xtune/src/pequod/tools/convert.py
+227
View File
@@ -0,0 +1,227 @@
import logging
import torch
from transformers.modeling_bert import (BertConfig, BertEncoder,
BertIntermediate, BertLayer,
BertModel, BertOutput,
BertSelfAttention,
BertSelfOutput)
from transformers.modeling_roberta import (RobertaEmbeddings,
RobertaForMaskedLM,
RobertaForSequenceClassification,
RobertaModel)
logger = logging.getLogger(__name__)
def convert_cxlm_to_transformers(ckpt_path):
ckpt = torch.load(ckpt_path, map_location="cpu")
args = ckpt["args"]
config = BertConfig(
vocab_size_or_config_json_file=250002,
hidden_size=args.encoder_embed_dim,
num_hidden_layers=args.encoder_layers,
num_attention_heads=args.encoder_attention_heads,
intermediate_size=args.encoder_ffn_embed_dim,
max_position_embeddings=args.max_positions + 2,
type_vocab_size=1,
layer_norm_eps=1e-5, # PyTorch default used in fairseq
)
print("Our BERT config:", config)
stat_dict = ckpt["model"]
new_stat_dict = {}
model = RobertaForMaskedLM(config)
model.eval()
sent_enc = "model_fast.decoder.sentence_encoder"
new_stat_dict["roberta.embeddings.word_embeddings.weight"] = stat_dict[sent_enc + ".embed_tokens.weight"]
new_stat_dict["roberta.embeddings.position_embeddings.weight"] = stat_dict[sent_enc + ".embed_positions.weight"]
new_stat_dict["roberta.embeddings.token_type_embeddings.weight"] = torch.zeros_like(model.roberta.embeddings.token_type_embeddings.weight)
new_stat_dict["roberta.embeddings.LayerNorm.weight"] = stat_dict[sent_enc +".emb_layer_norm.weight"]
new_stat_dict["roberta.embeddings.LayerNorm.bias"] = stat_dict[sent_enc + ".emb_layer_norm.bias"]
for i in range(config.num_hidden_layers):
# Encoder: start of layer
# layer: BertLayer = model.roberta.encoder.layer[i]
layer = "roberta.encoder.layer.%d" % i
roberta_layer = sent_enc + (".layers.%d" % i)
### self attention
# self_attn: BertSelfAttention = layer.attention.self
self_attn = layer + ".attention.self"
assert(
stat_dict[roberta_layer+".self_attn.k_proj.weight"].data.shape == \
stat_dict[roberta_layer+".self_attn.q_proj.weight"].data.shape == \
stat_dict[roberta_layer+".self_attn.v_proj.weight"].data.shape == \
torch.Size((config.hidden_size, config.hidden_size))
)
new_stat_dict[self_attn+".query.weight"] = stat_dict[roberta_layer+".self_attn.q_proj.weight"]
new_stat_dict[self_attn+".query.bias"] = stat_dict[roberta_layer+".self_attn.q_proj.bias"]
new_stat_dict[self_attn+".key.weight"] = stat_dict[roberta_layer+".self_attn.k_proj.weight"]
new_stat_dict[self_attn+".key.bias"] = stat_dict[roberta_layer+".self_attn.k_proj.bias"]
new_stat_dict[self_attn+".value.weight"] = stat_dict[roberta_layer+".self_attn.v_proj.weight"]
new_stat_dict[self_attn+".value.bias"] = stat_dict[roberta_layer+".self_attn.v_proj.bias"]
### self-attention output
# self_output: BertSelfOutput = layer.attention.output
self_output = layer + ".attention.output"
assert(
model.roberta.encoder.layer[i].attention.output.dense.weight.shape == stat_dict[roberta_layer+".self_attn.out_proj.weight"].shape
)
new_stat_dict[self_output+".dense.weight"] = stat_dict[roberta_layer+".self_attn.out_proj.weight"]
new_stat_dict[self_output+".dense.bias"] = stat_dict[roberta_layer+".self_attn.out_proj.bias"]
new_stat_dict[self_output+".LayerNorm.weight"] = stat_dict[roberta_layer+".self_attn_layer_norm.weight"]
new_stat_dict[self_output+".LayerNorm.bias"] = stat_dict[roberta_layer+".self_attn_layer_norm.bias"]
### intermediate
# intermediate: BertIntermediate = layer.intermediate
intermediate = layer + ".intermediate"
assert(
model.roberta.encoder.layer[i].intermediate.dense.weight.shape == stat_dict[roberta_layer+".fc1.weight"].shape
)
#TODO
new_stat_dict[intermediate+".dense.weight"] = stat_dict[roberta_layer+".fc1.weight"]
new_stat_dict[intermediate+".dense.bias"] = stat_dict[roberta_layer+".fc1.bias"]
### output
# bert_output: BertOutput = layer.output
bert_output = layer + ".output"
assert(
model.roberta.encoder.layer[i].output.dense.weight.shape == stat_dict[roberta_layer+".fc2.weight"].shape
)
new_stat_dict[bert_output+".dense.weight"] = stat_dict[roberta_layer+".fc2.weight"]
new_stat_dict[bert_output+".dense.bias"] = stat_dict[roberta_layer+".fc2.bias"]
new_stat_dict[bert_output+".LayerNorm.weight"] = stat_dict[roberta_layer+".final_layer_norm.weight"]
new_stat_dict[bert_output+".LayerNorm.bias"] = stat_dict[roberta_layer+".final_layer_norm.bias"]
#### end of layer
new_stat_dict["lm_head.dense.weight"] = stat_dict["model_fast.decoder.lm_head.dense.weight"]
new_stat_dict["lm_head.dense.bias"] = stat_dict["model_fast.decoder.lm_head.dense.bias"]
new_stat_dict["lm_head.layer_norm.weight"] = stat_dict["model_fast.decoder.lm_head.layer_norm.weight"]
new_stat_dict["lm_head.layer_norm.bias"] = stat_dict["model_fast.decoder.lm_head.layer_norm.bias"]
new_stat_dict["lm_head.decoder.weight"] = stat_dict["model_fast.decoder.lm_head.weight"]
new_stat_dict["lm_head.bias"] = stat_dict["model_fast.decoder.lm_head.bias"]
new_stat_dict["roberta.pooler.dense.weight"] = model.roberta.pooler.dense.weight
new_stat_dict["roberta.pooler.dense.bias"] = model.roberta.pooler.dense.bias
if "proj_matrix_fast" in stat_dict:
new_stat_dict["proj_matrix_fast"] = stat_dict["proj_matrix_fast"]
# model.load_state_dict(new_stat_dict)
return new_stat_dict
def convert_roberta_to_transformers(ckpt_path):
ckpt = torch.load(ckpt_path, map_location="cpu")
args = ckpt["args"]
config = BertConfig(
vocab_size_or_config_json_file=250002,
hidden_size=args.encoder_embed_dim,
num_hidden_layers=args.encoder_layers,
num_attention_heads=args.encoder_attention_heads,
intermediate_size=args.encoder_ffn_embed_dim,
max_position_embeddings=args.max_positions + 2,
type_vocab_size=1,
layer_norm_eps=1e-5, # PyTorch default used in fairseq
)
print("Our BERT config:", config)
stat_dict = ckpt["model"]
new_stat_dict = {}
model = RobertaForMaskedLM(config)
model.eval()
sent_enc = "decoder.sentence_encoder"
new_stat_dict["roberta.embeddings.word_embeddings.weight"] = stat_dict[sent_enc + ".embed_tokens.weight"]
new_stat_dict["roberta.embeddings.position_embeddings.weight"] = stat_dict[sent_enc + ".embed_positions.weight"]
new_stat_dict["roberta.embeddings.token_type_embeddings.weight"] = torch.zeros_like(model.roberta.embeddings.token_type_embeddings.weight)
new_stat_dict["roberta.embeddings.LayerNorm.weight"] = stat_dict[sent_enc +".emb_layer_norm.weight"]
new_stat_dict["roberta.embeddings.LayerNorm.bias"] = stat_dict[sent_enc + ".emb_layer_norm.bias"]
for i in range(config.num_hidden_layers):
# Encoder: start of layer
# layer: BertLayer = model.roberta.encoder.layer[i]
layer = "roberta.encoder.layer.%d" % i
roberta_layer = sent_enc + (".layers.%d" % i)
### self attention
# self_attn: BertSelfAttention = layer.attention.self
self_attn = layer + ".attention.self"
assert(
stat_dict[roberta_layer+".self_attn.k_proj.weight"].data.shape == \
stat_dict[roberta_layer+".self_attn.q_proj.weight"].data.shape == \
stat_dict[roberta_layer+".self_attn.v_proj.weight"].data.shape == \
torch.Size((config.hidden_size, config.hidden_size))
)
new_stat_dict[self_attn+".query.weight"] = stat_dict[roberta_layer+".self_attn.q_proj.weight"]
new_stat_dict[self_attn+".query.bias"] = stat_dict[roberta_layer+".self_attn.q_proj.bias"]
new_stat_dict[self_attn+".key.weight"] = stat_dict[roberta_layer+".self_attn.k_proj.weight"]
new_stat_dict[self_attn+".key.bias"] = stat_dict[roberta_layer+".self_attn.k_proj.bias"]
new_stat_dict[self_attn+".value.weight"] = stat_dict[roberta_layer+".self_attn.v_proj.weight"]
new_stat_dict[self_attn+".value.bias"] = stat_dict[roberta_layer+".self_attn.v_proj.bias"]
### self-attention output
# self_output: BertSelfOutput = layer.attention.output
self_output = layer + ".attention.output"
assert(
model.roberta.encoder.layer[i].attention.output.dense.weight.shape == stat_dict[roberta_layer+".self_attn.out_proj.weight"].shape
)
new_stat_dict[self_output+".dense.weight"] = stat_dict[roberta_layer+".self_attn.out_proj.weight"]
new_stat_dict[self_output+".dense.bias"] = stat_dict[roberta_layer+".self_attn.out_proj.bias"]
new_stat_dict[self_output+".LayerNorm.weight"] = stat_dict[roberta_layer+".self_attn_layer_norm.weight"]
new_stat_dict[self_output+".LayerNorm.bias"] = stat_dict[roberta_layer+".self_attn_layer_norm.bias"]
### intermediate
# intermediate: BertIntermediate = layer.intermediate
intermediate = layer + ".intermediate"
assert(
model.roberta.encoder.layer[i].intermediate.dense.weight.shape == stat_dict[roberta_layer+".fc1.weight"].shape
)
#TODO
new_stat_dict[intermediate+".dense.weight"] = stat_dict[roberta_layer+".fc1.weight"]
new_stat_dict[intermediate+".dense.bias"] = stat_dict[roberta_layer+".fc1.bias"]
### output
# bert_output: BertOutput = layer.output
bert_output = layer + ".output"
assert(
model.roberta.encoder.layer[i].output.dense.weight.shape == stat_dict[roberta_layer+".fc2.weight"].shape
)
new_stat_dict[bert_output+".dense.weight"] = stat_dict[roberta_layer+".fc2.weight"]
new_stat_dict[bert_output+".dense.bias"] = stat_dict[roberta_layer+".fc2.bias"]
new_stat_dict[bert_output+".LayerNorm.weight"] = stat_dict[roberta_layer+".final_layer_norm.weight"]
new_stat_dict[bert_output+".LayerNorm.bias"] = stat_dict[roberta_layer+".final_layer_norm.bias"]
#### end of layer
new_stat_dict["lm_head.dense.weight"] = stat_dict["decoder.lm_head.dense.weight"]
new_stat_dict["lm_head.dense.bias"] = stat_dict["decoder.lm_head.dense.bias"]
new_stat_dict["lm_head.layer_norm.weight"] = stat_dict["decoder.lm_head.layer_norm.weight"]
new_stat_dict["lm_head.layer_norm.bias"] = stat_dict["decoder.lm_head.layer_norm.bias"]
new_stat_dict["lm_head.decoder.weight"] = stat_dict["decoder.lm_head.weight"]
new_stat_dict["lm_head.bias"] = stat_dict["decoder.lm_head.bias"]
new_stat_dict["roberta.pooler.dense.weight"] = model.roberta.pooler.dense.weight
new_stat_dict["roberta.pooler.dense.bias"] = model.roberta.pooler.dense.bias
return new_stat_dict
if __name__ == "__main__":
sd = convert_cxlm_to_transformers("/home/v-zechi/data/unilm/zechi/exp/cxlm_exp/dump-g16-lr2e-4/checkpoint_1_10000.pt")
print(sd.keys())
xtune/src/pequod/training/__init__.py
+108
View File
@@ -0,0 +1,108 @@
import re
import sys
import os
import random
import torch
import pickle
import logging
import numpy as np
# from transformers import (WEIGHTS_NAME,
# BertConfig, BertForSequenceClassification, BertTokenizer,
# RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer,
# RobertaModel, BertModel, XLMModel,
# XLMConfig, XLMForSequenceClassification, XLMTokenizer,
# XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer,
# DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer,
# BertForQuestionAnswering)
#
# from src.pequod.model.roberta import RobertaForQuestionAnswering
from transformers import XLMRobertaConfig, XLMRobertaForRetrieval, XLMRobertaTokenizer
# ALL_MODELS = sum((tuple(conf.pretrained_config_archive_map.keys()) \
# for conf in (BertConfig, XLNetConfig, XLMConfig,
# RobertaConfig, DistilBertConfig)), ())
ALL_MODELS = []
# # Model classes for classification
# MODEL_CLASSES = {
# 'bert': (BertConfig, BertForSequenceClassification, BertTokenizer),
# 'xlnet': (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer),
# 'xlm': (XLMConfig, XLMForSequenceClassification, XLMTokenizer),
# 'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer),
# 'distilbert': (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
# "xlmr": (RobertaConfig, RobertaForSequenceClassification, XLMRTokenizer)
# }
#
# QA_MODELS = {
# "bert": BertForQuestionAnswering,
# "roberta": RobertaForQuestionAnswering,
# "xlmr": RobertaForQuestionAnswering,
# }
BERT_CLASSES = {
"xlmr": (XLMRobertaConfig, XLMRobertaForRetrieval, XLMRobertaTokenizer),
}
def to_cuda(tup):
return tuple(t.cuda() for t in tup)
def set_seed(args):
random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
#TODO multi gpu support
# if args.n_gpu > 0:
# torch.cuda.manual_seed_all(args.seed)
def init_exp(args):
# dump parameters
set_dump_path(args)
pickle.dump(args, open(os.path.join(args.dump_path, 'params.pkl'), 'wb'))
# get running command
command = ["python", sys.argv[0]]
for x in sys.argv[1:]:
if x.startswith('--'):
assert '"' not in x and "'" not in x
command.append(x)
else:
assert "'" not in x
if re.match('^[a-zA-Z0-9_]+$', x):
command.append("%s" % x)
else:
command.append("'%s'" % x)
command = ' '.join(command)
args.command = command + ' --exp_id "%s"' % args.exp_id
# check experiment name
assert len(args.exp_name.strip()) > 0
logging.basicConfig(
format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
datefmt = '%m/%d/%Y %H:%M:%S',
level = logging.INFO)
logger = logging.getLogger(__name__)
logger.info("\n".join(
"%s: %s" % (k, str(v)) for k, v in sorted(dict(vars(args)).items())))
logger.info("The experiment will be stored in %s\n" % args.dump_path)
logger.info("Running command: %s" % command)
logger.info("")
def set_dump_path(args, output_dir=None, exp_name=None):
if output_dir is None: output_dir = args.output_dir
if exp_name is None: exp_name = args.exp_name
chars = 'abcdefghijklmnopqrstuvwxyz0123456789'
while True:
exp_id = ''.join(random.choice(chars) for _ in range(10))
if not os.path.isdir(os.path.join(output_dir, exp_name, exp_id)):
break
args.exp_id = exp_id
dump_path = os.path.join(output_dir, exp_name, exp_id)
os.makedirs(dump_path)
args.dump_path = dump_path
xtune/src/pequod/training/trainer.py
+540
View File
@@ -0,0 +1,540 @@
import os
import json
import logging
import random
import torch
import numpy as np
try:
from apex import amp
except ImportError:
pass
from torch.utils.data import (DataLoader,
RandomSampler, SequentialSampler, TensorDataset, SubsetRandomSampler,
Subset, ConcatDataset)
#from transformers import AdamW, ConstantLRSchedule, WarmupLinearSchedule
from transformers import AdamW, get_constant_schedule, get_linear_schedule_with_warmup
from src.pequod.training import to_cuda, set_seed
from src.pequod.eval import (eval_classification, eval_qa,
score_dict_to_string, score_dicts_to_latex)
from src.pequod.data import xdoc, xqa
from src.pequod.data.sampler import SubSampler
logger = logging.getLogger(__name__)
class Trainer(object):
def __init__(self, args, model, tokenizer):
self.args = args
self.datasets = {}
self.dataloaders = {}
self.iter_cache = {}
self.best_scores = {}
self.model = model
self.tokenizer = tokenizer
def run(self):
raise NotImplementedError
def _parse_batch(self, batch, **kwargs):
_batch = to_cuda(batch)
# _batch = batch
ret = {"input_ids": _batch[0],
"attention_mask": _batch[1],
"token_type_ids": _batch[2] if self.args.model_type == "bert" else None,
"labels": _batch[3]}
ret.update(**kwargs)
return ret
def train_epoch(self, split, lang, epoch_id):
raise NotImplementedError
def before_loop(self):
return
def train_full_epoch(self, split, lang, epoch_id):
raise NotImplementedError
def eval_epoch(self, split, lang, epoch_id):
raise NotImplementedError
def load_and_cache_examples(self, *args, **kwargs):
raise NotImplementedError
def save(self, name, epoch=0):
path = os.path.join(self.args.dump_path, "%s.pth" % name)
logger.info("Saving %s to %s ..." % (name, path))
data = {
"epoch":epoch,
"model":self.model.state_dict(),
"params": {k: v for k, v in self.args.__dict__.items()}}
torch.save(data, path)
def get_dataset_deprecated(self, *args, **kwargs):
logger.warning("cut_args is deprecated, please use train_ds_keys.")
if args in self.datasets: return self.datasets[args]
dataset = self.load_and_cache_examples(*args, **kwargs)
cut_split, cut_num = self.args.cut_args.split(",")
cut_num = int(cut_num)
if cut_num != -1 and cut_num < len(dataset) and cut_split == args[0]:
# cut_indices = random.sample(range(len(dataset)), cut_num)
cut_indices = [i for i in range(cut_num)]
# dataset = Subset(dataset, cut_indices)
dataset = TensorDataset(
*tuple(tensor[cut_indices] for tensor in dataset.tensors))
self.datasets[args] = dataset
return dataset
def get_cache_key(self):
cache_key = "%s-%s" % (self.args.model_key, self.args.model_type)
return cache_key
def get_dataset(self, *args, **kwargs):
if args in self.datasets: return self.datasets[args]
dataset = self.load_and_cache_examples(*args, **kwargs)
cut_num = int(kwargs.pop("cut", "-1"))
if cut_num != -1 and cut_num < len(dataset):
cut_indices = [i for i in range(cut_num)]
dataset = TensorDataset(
*tuple(tensor[cut_indices] for tensor in dataset.tensors))
self.datasets[args] = dataset
return dataset
def get_sampler(self, data_source, *args, **kwargs):
shuffle = kwargs.get("shuffle", args[0] == "train")
num_samples = kwargs.get("num_samples", None)
if num_samples is not None:
num_samples = int(num_samples)
sampler = SubSampler(data_source, num_samples)
else:
sampler = RandomSampler(data_source) if shuffle else SequentialSampler(data_source)
return sampler
def get_dataloader(self, *args, **kwargs):
logger.info("Getting dataloader - args: %s" % str(args))
if args in self.dataloaders: return self.dataloaders[args]
dataset = kwargs["dataset"] if "dataset" in kwargs \
else self.get_dataset(*args, **kwargs)
sampler = self.get_sampler(dataset, *args, **kwargs)
dataloader = DataLoader(dataset, sampler=sampler,
batch_size=self.args.train_batch_size)
self.dataloaders[args] = dataloader
return dataloader
def next_batch(self, *args, **kwargs):
if args not in self.iter_cache:
self.iter_cache[args] = iter(self.get_dataloader(*args, **kwargs))
try:
ret = next(self.iter_cache[args])
except StopIteration:
self.iter_cache[args] = iter(self.get_dataloader(*args, **kwargs))
ret = next(self.iter_cache[args])
return ret
def set_dataset(self, dataset, args):
self.datasets[args] = dataset
if args in self.dataloaders: self.dataloaders.pop(args)
if args in self.iter_cache: self.iter_cache.pop(args)
def copy_label(self, trg_key, src_key):
src_ds = self.get_dataset(*src_key)
trg_ds = self.get_dataset(*trg_key)
new_trg_ds = TensorDataset(*(trg_ds.tensors[:-1]) + (src_ds.tensors[-1],))
self.set_dataset(new_trg_ds, trg_key)
class SelfTrainer(Trainer):
def __init__(self, args, model=None, tokenizer=None):
super().__init__(args, model, tokenizer)
def labeling_dataset(self, model, ds_key):
logger.info("Labeling dataset %s" % str(ds_key))
model.eval()
dataset:TensorDataset = self.get_dataset(*ds_key)
# NOTE all_labels must be the last
preds = None
for batch in self.get_dataloader(*ds_key, shuffle=False):
with torch.no_grad():
batch_dict = self._parse_batch(batch, labels=None)
outputs = model(**batch_dict)
logits = outputs[0]
pred = logits.detach().cpu().numpy()
preds = pred if preds is None else np.append(preds, pred, axis=0)
new_labels = np.argmax(preds, axis=1)
new_labels = torch.tensor(new_labels, dtype=torch.long)
self.set_dataset(
TensorDataset(*(dataset.tensors[:-1] + (new_labels, ))), ds_key)
return preds
def update_concat_dataset_cache(self, ds_keys, preds_list, key_prefix="concat"):
"""
if preds_list[i] is None, then the ith dataset won't be cut by confidence.
"""
assert len(ds_keys) == len(preds_list)
assert all(ds_key[1:] == ds_keys[0][1:] for ds_key in ds_keys)
new_split = "-".join((key_prefix,) + tuple(ds_key[0] for ds_key in ds_keys))
logger.info("Concating %d dataset %s ..." % (len(ds_keys), new_split))
new_ds_key = (new_split, ) + ds_keys[0][1:]
datasets = []
for ds_key, preds in zip(ds_keys, preds_list):
dataset = self.get_dataset(*ds_key)
if preds is None:
datasets.append(dataset)
continue
new_labels = dataset.tensors[-1]
confident_indices = []
for i in range(len(new_labels)):
if preds[i,new_labels[i]] >= self.args.confidence_threshold:
confident_indices.append(i)
logger.info(
"Labeled %d confident examples out of %d examples for dataset %s" % (
len(confident_indices), len(new_labels), str(ds_key)))
if len(confident_indices) > 0:
datasets.append(Subset(dataset, confident_indices))
self.set_dataset(ConcatDataset(datasets), new_ds_key)
# self.datasets[new_ds_key] = ConcatDataset(datasets)
logger.info("Construct new dataset %s with %d examples" % (
str(new_ds_key), len(self.datasets[new_ds_key])))
return new_ds_key
class DistillTrainer(Trainer):
def __init__(self, args, model=None, tokenizer=None):
super().__init__(args, model, tokenizer)
def labeling_dataset(self, model, ds_key):
logger.info("Labeling dataset %s" % str(ds_key))
model.eval()
dataset:TensorDataset = self.get_dataset(*ds_key)
preds = None
for batch in self.get_dataloader(*ds_key, shuffle=False):
with torch.no_grad():
batch_dict = self._parse_batch(batch, labels=None)
outputs = model(**batch_dict)
logits = outputs[0]
pred = logits.detach().cpu().numpy()
preds = pred if preds is None else np.append(preds, pred, axis=0)
preds = torch.from_numpy(preds)
self.set_dataset(
TensorDataset(*(dataset.tensors[:-1] + (preds, ))), ds_key)
def update_concat_dataset_cache(self, ds_keys, key_prefix="concat"):
assert all(ds_key[1:] == ds_keys[0][1:] for ds_key in ds_keys)
new_split = "-".join((key_prefix,) + tuple(ds_key[0] for ds_key in ds_keys))
logger.info("Concating %d dataset %s ..." % (len(ds_keys), new_split))
new_ds_key = (new_split, ) + ds_keys[0][1:]
new_ds = ConcatDataset([self.get_dataset(*ds_key) for ds_key in ds_keys])
self.set_dataset(new_ds, new_ds_key)
logger.info("Construct new dataset %s with %d examples" % (
str(new_ds_key), len(new_ds)))
return new_ds_key
class XClassificationTrainer(Trainer):
def __init__(self, args, model, tokenizer):
super().__init__(args, model, tokenizer)
_, self.optimizer, self.scheduler = self.init_optimizer(
model, args.learning_rate)
self.example_feature_cache = {}
self.no_improve_cnt = 0
def init_optimizer(self, model, lr):
args = self.args
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
{"params": [p for n, p in model.named_parameters() if not any(
nd in n for nd in no_decay)], "weight_decay": args.weight_decay},
{"params": [p for n, p in model.named_parameters() if any(
nd in n for nd in no_decay)], "weight_decay": 0.0}]
# TODO calculate t_total
optimizer = AdamW(
optimizer_grouped_parameters, lr=lr, eps=args.adam_epsilon)
# scheduler = WarmupLinearSchedule(
# optimizer, warmup_steps=args.warmup_steps, t_total=t_total)
scheduler = get_constant_schedule(optimizer)
return optimizer_grouped_parameters, optimizer, scheduler
def run(self):
args = self.args
set_seed(args)
if self.optimizer is not None and args.fp16:
self.model, self.optimizer = amp.initialize(
self.model, self.optimizer, opt_level=args.fp16_opt_level)
test_langs = args.test_langs.split(",")
assert args.dev_mode in ["train_lang", "test_lang", "avg"]
if args.dev_mode == "train_lang":
assert args.train_lang in test_langs
train_lang_index = test_langs.index(args.train_lang)
logger.info("***** Running Trainer *****")
logger.info("***** Before Trainer Loop *****")
self.before_loop()
def _eval(update_no_improve_cnt=False):
score_tups = []
should_save = False
for lang in test_langs:
dev_score_dict = self.eval_epoch(
split="dev", lang=lang, epoch_id=epoch_id)
test_score_dict = self.eval_epoch(
split="test", lang=lang, epoch_id=epoch_id)
score_tup = (dev_score_dict, test_score_dict)
score_tups.append(score_tup)
logger.info("Eval epoch %d - lang - %s score - dev: %s - test: %s" % (
epoch_id, lang, score_dict_to_string(dev_score_dict),
score_dict_to_string(test_score_dict)))
dev_scores, test_scores = [], []
if args.dev_mode == "test_lang":
# select best results w.r.t. the res on test-lang dev sets.
for lang, score_tup in zip(test_langs, score_tups):
if lang not in self.best_scores:
if lang == test_langs[-1]: should_save = True
self.best_scores[lang] = score_tup
elif self.best_scores[lang][0][args.dev_criterion] < \
score_tup[0][args.dev_criterion]:
if lang == test_langs[-1]: should_save = True
self.best_scores[lang] = score_tup
dev_scores.append(self.best_scores[lang][0])
test_scores.append(self.best_scores[lang][1])
elif args.dev_mode == "train_lang":
# select best results w.r.t. the res on train-lang dev sets.
if (args.train_lang not in self.best_scores) or self.best_scores[
args.train_lang][0][args.dev_criterion] < \
score_tups[train_lang_index][0][args.dev_criterion]:
should_save = True
for lang, score_tup in zip(test_langs, score_tups):
self.best_scores[lang] = score_tup
dev_scores.append(self.best_scores[lang][0])
test_scores.append(self.best_scores[lang][1])
if update_no_improve_cnt:
self.no_improve_cnt = 0
logger.info("New best results!")
else:
for lang in test_langs:
dev_scores.append(self.best_scores[lang][0])
test_scores.append(self.best_scores[lang][1])
if update_no_improve_cnt:
self.no_improve_cnt += 1
logger.info("Results not improved, no_improve_cnt:%d" % self.no_improve_cnt)
elif args.dev_mode == "avg":
# select best results by the best sum scores
avg_key = "_avg"
sum_dev_scores = sum_test_scores = 0
for score_tup in score_tups:
sum_dev_scores += score_tup[0][args.dev_criterion]
sum_test_scores += score_tup[1][args.dev_criterion]
if (avg_key not in self.best_scores) or self.best_scores[avg_key] < sum_dev_scores:
should_save = True
self.best_scores[avg_key] = sum_dev_scores
for lang, score_tup in zip(test_langs, score_tups):
self.best_scores[lang] = score_tup
dev_scores.append(self.best_scores[lang][0])
test_scores.append(self.best_scores[lang][1])
logger.info("New best results! Dev avg: %.2f Test avg: %.2f" % (
sum_dev_scores/len(test_langs), sum_test_scores/len(test_langs),
))
if update_no_improve_cnt:
self.no_improve_cnt = 0
else:
for lang in test_langs:
dev_scores.append(self.best_scores[lang][0])
test_scores.append(self.best_scores[lang][1])
if update_no_improve_cnt:
self.no_improve_cnt += 1
logger.info("Results not improved, no_improve_cnt:%d" % self.no_improve_cnt)
logger.info("Eval epoch %d - langs %s - dev scores - %s" % (
epoch_id, " & ".join(test_langs), score_dicts_to_latex(dev_scores)))
logger.info("Eval epoch %d - langs %s - test scores - %s" % (
epoch_id, " & ".join(test_langs), score_dicts_to_latex(test_scores)))
with open(os.path.join(args.exp_results_dir, args.exp_name), "w") as fp:
json.dump(self.best_scores, fp)
fp.flush()
if should_save and args.save:
save_to = os.path.join(args.dump_path, "best-%s-%s" % (
args.dev_criterion, args.model_type))
logger.info("Epoch %d, saving best model to %s" % (
epoch_id, save_to))
torch.save(self.model.state_dict(), save_to)
logger.info("***** Start Trainer Loop *****")
for epoch_id in range(args.num_train_epochs):
self.train_full_epoch(args.train_ds_keys, epoch_id=epoch_id, algo=args.algo)
_eval(update_no_improve_cnt=args.stopping_threshold>0)
if args.stopping_threshold > 0 and self.no_improve_cnt >= args.stopping_threshold:
logger.info("***** Early stop *****")
break
if args.add_train_ds_keys != "":
logger.info("***** Additional Trainer Loop *****")
state_dict_path = os.path.join(args.dump_path, "best-%s-%s" % (
args.dev_criterion, args.model_type))
logger.info("Reloading model parameters from %s ..." % state_dict_path)
state_dict = torch.load(state_dict_path, map_location="cpu")
self.model.load_state_dict(state_dict)
self.model.cuda()
num_additional_train_epochs = getattr(
args, "num_additional_train_epochs", args.num_train_epochs)
for epoch_id in range(num_additional_train_epochs):
self.train_full_epoch(
args.add_train_ds_keys, epoch_id=epoch_id, algo=args.add_algo)
_eval()
def train_full_epoch(self, train_ds_keys, epoch_id, algo=None):
raise NotImplementedError
def train_full_epoch_deprecated(self, split, lang, epoch_id):
logger.info("***** Training epoch %d - lang: %s *****" % (epoch_id, lang))
args = self.args
model = self.model
model.train()
losses = []
n_instances = 0
for step, batch in enumerate(self.get_dataloader(split, lang)):
feed_dict = self._parse_batch(batch)
outputs = model(**feed_dict)
loss = outputs[0]
model.zero_grad()
if args.fp16:
with amp.scale_loss(loss, self.optimizer) as scaled_loss:
scaled_loss.backward()
torch.nn.utils.clip_grad_norm_(
amp.master_params(self.optimizer), args.max_grad_norm)
else:
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)
self.optimizer.step()
self.scheduler.step()
losses.append(loss)
n_instances += args.train_batch_size
if step % args.logging_steps == 0:
logger.info(
"Epoch %d - n instances %7d - loss: %.4f " % (
epoch_id, n_instances, sum(losses) / len(losses)))
losses = []
def eval_epoch(self, split, lang, epoch_id):
logger.info("***** Evaluating epoch %d - split: %s - lang: %s*****" % (
epoch_id, split, lang))
def _get_batch_iter():
for batch in self.get_dataloader(split, lang, shuffle=False):
yield self._parse_batch(batch)
return eval_classification(self.model, _get_batch_iter())
def load_and_cache_examples(self, split, lang, **kwargs):
processor = xdoc.get_processor_class(self.args.dataset_name)()
cache_key = self.get_cache_key()
return xdoc.load_and_cache_examples(
self.args, processor, split, lang, self.tokenizer, cache_key)
class XQATrainer(XClassificationTrainer):
def __init__(self, args, model, tokenizer):
super().__init__(args, model, tokenizer)
def init_optimizer(self, model, lr):
args = self.args
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
{"params": [p for n, p in model.named_parameters() if not any(
nd in n for nd in no_decay)], "weight_decay": args.weight_decay},
{"params": [p for n, p in model.named_parameters() if any(
nd in n for nd in no_decay)], "weight_decay": 0.0}]
optimizer = AdamW(
optimizer_grouped_parameters, lr=lr, eps=args.adam_epsilon)
dataloader = self.get_dataloader("train", args.train_lang)
t_total = len(dataloader) * args.num_train_epochs
scheduler = get_linear_schedule_with_warmup(
optimizer, warmup_steps=args.warmup_steps, t_total=t_total)
return optimizer_grouped_parameters, optimizer, scheduler
def _parse_batch(self, batch, training=True, **kwargs):
_batch = to_cuda(batch)
# _batch = batch
if training:
ret = {"input_ids": _batch[0],
"attention_mask": _batch[1],
"token_type_ids": _batch[2] if self.args.model_type == "bert" else None,
'start_positions': _batch[3],
'end_positions': _batch[4]}
else:
ret = {"input_ids": _batch[0],
"attention_mask": _batch[1],
"token_type_ids": _batch[2] if self.args.model_type == "bert" else None}
ret.update(**kwargs)
return ret
def eval_epoch(self, split, lang, epoch_id):
args = self.args
logger.info("***** Evaluating epoch %d - split: %s - lang: %s*****" % (
epoch_id, split, lang))
dataset, examples, features = self.get_eval_data(split, lang)
def _get_batch_iter():
for batch in self.get_dataloader(
split, lang, shuffle=False, dataset=dataset):
example_indices = batch[3]
yield self._parse_batch(batch, training=False), example_indices
return eval_qa(self.model, _get_batch_iter(), **{
"all_examples": examples,
"all_features": features,
"predict_file": os.path.join(args.data_dir, "%s-%s.json" % (split, lang)),
"output_dir": args.dump_path,
"n_best_size": args.n_best_size,
"max_answer_length": args.max_answer_length,
"do_lower_case": args.do_lower_case,
"verbose_logging": args.verbose_logging,
"version_2_with_negative": args.version_2_with_negative,
"null_score_diff_threshold": args.null_score_diff_threshold})
def load_and_cache_examples(self, split, lang, **kwargs):
evaluate = kwargs.pop("evaluate", False)
cache_key = "%s-%s" % (self.args.model_key, self.args.model_type)
dataset, _, _ = xqa.load_and_cache_examples(
self.args, split, lang, self.tokenizer, cache_key, evaluate=evaluate)
return dataset
def get_eval_data(self, split, lang):
ds_key = (split, lang)
if ds_key in self.example_feature_cache:
return self.example_feature_cache[ds_key]
cache_key = "%s-%s" % (self.args.model_key, self.args.model_type)
dataset, examples, features = xqa.load_and_cache_examples(
self.args, split, lang, self.tokenizer, cache_key, evaluate=True)
self.example_feature_cache[ds_key] = (dataset, examples, features)
return dataset, examples, features
xtune/src/pequod/training/xtrainer.py
+269
View File
@@ -0,0 +1,269 @@
import logging
import numpy as np
import os
import torch
import random
from torch.autograd import Variable
from torch.utils.data import DataLoader, TensorDataset
try:
from apex import amp
except ImportError:
pass
from src.pequod.trainer import (Trainer,
XClassificationTrainer, XQATrainer, SelfTrainer)
from transformers import AdamW, ConstantLRSchedule, WarmupLinearSchedule
logger = logging.getLogger(__name__)
class BaseTrainer(Trainer):
def __init__(self, args, model, tokenizer):
super().__init__(args, model, tokenizer)
self.optimizer = None
self.scheduler = None
self.global_steps = 0
self.all_shard_fn = {}
def init_optimizer(self, model, lr, t_total, fixed=None):
args = self.args
no_decay = ['bias', 'LayerNorm.weight']
if fixed is None: fixed = []
optimizer_grouped_parameters = [
{"params": [p for n, p in model.named_parameters() if not any(
nd in n for nd in no_decay) and not any(f in n for f in fixed)
], "weight_decay": args.weight_decay},
{"params": [p for n, p in model.named_parameters() if any(
nd in n for nd in no_decay) and not any(f in n for f in fixed)
], "weight_decay": 0.0}]
# TODO calculate t_total
optimizer = AdamW(
optimizer_grouped_parameters, lr=lr, eps=args.adam_epsilon)
if args.scheduler == "linear":
warmup_steps = t_total * args.warmup_ratio if args.warmup_steps == -1 else args.warmup_steps
logger.info("Setting scheduler, warmups=%d, lr=%.7f, total_updates=%d" % (
warmup_steps, lr, t_total))
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=warmup_steps, t_total=t_total)
elif args.scheduler == "constant":
logger.info("Setting scheduler, ConstantLRSchedule")
scheduler = ConstantLRSchedule(optimizer)
else:
raise ValueError
return optimizer_grouped_parameters, optimizer, scheduler
def optim_step(self, **kwargs):
args = self.args
# self.model.zero_grad()
if args.fp16:
# with amp.scale_loss(loss, self.optimizer) as scaled_loss:
# scaled_loss.backward()
torch.nn.utils.clip_grad_norm_(
amp.master_params(self.optimizer), args.max_grad_norm)
else:
# loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), args.max_grad_norm)
self.optimizer.step()
self.scheduler.step()
self.model.zero_grad()
def backward_step(self, loss, **kwargs):
args = self.args
if args.accumulate_steps > 1:
loss = loss / args.accumulate_steps
if args.fp16:
with amp.scale_loss(loss, self.optimizer) as scaled_loss:
scaled_loss.backward()
else:
loss.backward()
def step(self, *args, **kwargs):
algo = kwargs.pop("algo", self.args.algo)
if algo is None: algo = self.args.algo
step_func_names = ["%s_step" % s for s in algo.split(",")]
return getattr(self, random.choice(step_func_names))(*args, **kwargs)
def base_step(self, batches, is_qa=False, **kwargs):
tot_loss = 0.0
for step_batches in batches:
batch = step_batches[0]
batch_dict = self._parse_batch(batch)
loss = self.model(**batch_dict)[0]
self.backward_step(loss)
tot_loss += loss.item()
self.optim_step()
return tot_loss / len(batches)
def train_full_epoch(self, train_ds_keys, epoch_id, is_qa=False, algo=None):
if train_ds_keys == "": return
logger.info("***** Training epoch %d - train_ds_keys: %s *****" % (
epoch_id, str(train_ds_keys)))
args = self.args
n_instances = 0
data_loaders = []
if isinstance(train_ds_keys, str):
train_ds_keys = train_ds_keys.split(";")
for ds_key_str in train_ds_keys:
data_loaders.append(self.get_dataloader_from_str(ds_key_str, epoch_id))
if self.optimizer is None:
_, self.optimizer, self.scheduler = self.init_optimizer(
self.model, args.learning_rate, len(data_loaders[0]) * args.num_train_epochs // args.accumulate_steps)
if args.fp16:
self.model, self.optimizer = amp.initialize(
self.model, self.optimizer, opt_level=args.fp16_opt_level)
model = self.model
model.train()
losses = []
step = 0
step_func_dict = {"batches": [], "is_qa": is_qa, "epoch_id": epoch_id}
# for step, batches in enumerate(zip(*data_loaders)):
for batches in zip(*data_loaders):
# step_func_dict = {"batches": batches, "is_qa": is_qa, "epoch_id": epoch_id}
step_func_dict["batches"].append(batches)
if len(step_func_dict["batches"]) == args.accumulate_steps:
loss = self.step(**step_func_dict, algo=algo)
losses.append(loss)
step_func_dict["batches"] = []
else:
continue
n_instances += args.train_batch_size * args.accumulate_steps
self.global_steps += 1
step += 1
if step % args.logging_steps == 0:
cur_lr = self.scheduler.get_lr()[0]
logger.info(
"Epoch %d - step %7d - global step %d - lr %.8f - n instances %7d - loss: %.4f " % (
epoch_id, step, self.global_steps, cur_lr, n_instances, sum(losses) / len(losses)))
losses = []
def _parse_ds_key(self, ds_key_str):
assert isinstance(ds_key_str, str)
args, kwargs = [], {}
for s in ds_key_str.split(","):
if ":" in s:
k, v = s.split(":")
kwargs[k] = v
else: args.append(s)
return args, kwargs
def get_mixed_dataloader(self, *dataloaders):
iters = [iter(d) for d in dataloaders]
len_dl = len(iters)
finish = [False] * len_dl
cnt = 0
while cnt < len_dl:
idx = random.randint(0, len_dl - 1)
if finish[idx]: continue
try:
yield next(iters[idx])
except StopIteration:
finish[idx] = True
cnt += 1
def get_all_shard_fn(self, *args, cache_filename=None):
if args in self.all_shard_fn: return self.all_shard_fn[args]
all_shard_fn = []
shard_id = 0
while True:
fn = cache_filename + "." + str(shard_id)
if not os.path.exists(fn): break
all_shard_fn.append(fn)
shard_id += 1
logger.info("%d shards found." % len(all_shard_fn))
np.random.shuffle(all_shard_fn)
self.all_shard_fn[args] = all_shard_fn
return all_shard_fn
def get_sharded_dataloader(self, *args, **kwargs):
logger.info("Getting dataloader - args: %s" % str(args))
split, lang, epoch_id = args
cache_key = self.get_cache_key()
cache_filename = os.path.join(
self.args.data_dir, "cached_%s_%s_%s" % (split, lang, cache_key))
all_shard_fn = self.get_all_shard_fn(
split, lang, cache_filename=cache_filename)
fn = all_shard_fn[epoch_id % len(all_shard_fn)]
logger.info("Loading dataset from %s" % str(fn))
tensor_dict = torch.load(fn)
tensors = []
for _, t in tensor_dict.items():
tensors.append(t.long())
dataset = TensorDataset(*tensors)
sampler = self.get_sampler(dataset, *args, **kwargs)
dataloader = DataLoader(dataset, sampler=sampler,
batch_size=self.args.train_batch_size)
return dataloader
def get_dataloader_from_str(self, ds_key_str, epoch_id):
if ds_key_str.startswith("mix("):
# example: mix(train,en,cut:200|train,zh,cut:20)
assert ds_key_str[-1] == ")"
ds_key_str = ds_key_str[4:-1]
dataloaders = []
for dks in ds_key_str.split("|"):
dataloaders.append(self.get_dataloader_from_str(dks, epoch_id))
return self.get_mixed_dataloader(*dataloaders)
ds_key_args, ds_key_kwargs = self._parse_ds_key(ds_key_str)
sharded_dataloader = ds_key_kwargs.pop("sharded_dataloader", "")
if sharded_dataloader == "True":
return self.get_sharded_dataloader(*ds_key_args, epoch_id, **ds_key_kwargs)
return self.get_dataloader(*ds_key_args, **ds_key_kwargs)
def get_model_class(proto_train_class=None, is_qa=False):
class ProtoXClassificationTrainer(XClassificationTrainer, proto_train_class):
def __init__(self, args, model, tokenizer):
proto_train_class.__init__(self, args, model, tokenizer)
# _, self.optimizer, self.scheduler = self.init_optimizer(
# model, args.learning_rate)
def train_full_epoch(self, train_ds_keys, epoch_id, algo=None):
proto_train_class.train_full_epoch(
self, train_ds_keys, epoch_id, is_qa=False, algo=algo)
def before_loop(self):
# args = self.args
# if args.labeling_unlabeled_data:
# assert args.semi_split != ""
# for lang in args.test_langs.split(","):
# logger.info("Labeling lang: %s" % lang)
# self.labeling_dataset(self.model, (args.semi_split, lang))
pass
def init_optimizer(self, *args, **kwargs):
return proto_train_class.init_optimizer(self, *args, **kwargs)
class ProtoXQATrainer(XQATrainer, proto_train_class):
def __init__(self, args, model, tokenizer):
proto_train_class.__init__(self, args, model, tokenizer)
# _, self.optimizer, self.scheduler = self.init_optimizer(
# model, args.learning_rate)
self.example_feature_cache = {}
def train_full_epoch(self, train_ds_keys, epoch_id, algo=None):
proto_train_class.train_full_epoch(
self, train_ds_keys, epoch_id, is_qa=True, algo=algo)
def init_optimizer(self, *args, **kwargs):
return proto_train_class.init_optimizer(self, *args, **kwargs)
return ProtoXQATrainer if is_qa else ProtoXClassificationTrainer
xtune/src/run_cls.py
+1552
View File
File diff suppressed because it is too large Load Diff
xtune/src/run_qa.py
+1831
View File
File diff suppressed because it is too large Load Diff
xtune/src/run_tag.py
+1745
View File
File diff suppressed because it is too large Load Diff
xtune/src/tools/__init__.py
View File
xtune/src/tools/check_many2many_alignment.py
+61
View File
@@ -0,0 +1,61 @@
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--translation_path",
default=None,
type=str,
required=True,
help="",
)
drop_languages = ["en", "zh-CN", "zh", "ja", "ko", "th", "my", "ml", "ta"]
translate_languages = None
args = parser.parse_args()
src2tgt = {}
print("Reading translation from {}".format(args.translation_path))
with open(args.translation_path, encoding="utf-8") as f:
cnt = 0
for line in f:
cnt += 1
if cnt % 10000 == 0:
print("Reading lines {}".format(cnt))
items = line.split("\t")
if items == 3:
src_sent, tgt_lang, tgt_sent = line.split("\t")
alignment = None
else:
src_sent, tgt_lang, tgt_sent, alignment_str = line.split("\t")
alignment = []
for x in alignment_str.split(" "):
alignment.append((int(x.split("/")[0]), int(x.split("/")[1])))
if tgt_lang in drop_languages:
continue
if translate_languages is not None and tgt_lang not in translate_languages:
continue
cnt_src = {}
cnt_tgt = {}
for x in alignment:
if x[0] not in cnt_src:
cnt_src[x[0]] = 0
cnt_src[x[0]] += 1
if x[1] not in cnt_tgt:
cnt_tgt[x[1]] = 0
cnt_tgt[x[1]] += 1
if not (cnt_src[x[0]] <= 1 or cnt_tgt[x[1]] <= 1):
print(cnt_src, cnt_tgt)
print(alignment)
print(src_sent, tgt_sent)
assert cnt_src[x[0]] <= 1 or cnt_tgt[x[1]] <= 1
xtune/src/tools/dump_hf_state_dict.py
+167
View File
@@ -0,0 +1,167 @@
import logging
import torch
from collections import OrderedDict
from transformers.modeling_bert import (BertConfig, BertEncoder,
BertIntermediate, BertLayer,
BertModel, BertOutput,
BertSelfAttention,
BertSelfOutput)
from transformers.modeling_roberta import (RobertaEmbeddings,
RobertaForMaskedLM,
RobertaForSequenceClassification,
RobertaModel)
logger = logging.getLogger(__name__)
# NOTE transformers should be 2.5.1
def convert_cxlm_to_transformers(ckpt_path):
ckpt = torch.load(ckpt_path, map_location="cpu")
args = ckpt["args"]
config = BertConfig(
# vocab_size_or_config_json_file=250002,
vocab_size=250002,
hidden_size=args.encoder_embed_dim,
num_hidden_layers=args.encoder_layers,
num_attention_heads=args.encoder_attention_heads,
intermediate_size=args.encoder_ffn_embed_dim,
max_position_embeddings=args.max_positions + 2,
type_vocab_size=1,
layer_norm_eps=1e-5, # PyTorch default used in fairseq
)
print("Our BERT config:", config)
stat_dict = ckpt["model"]
new_stat_dict = {}
model = RobertaForMaskedLM(config)
model.eval()
sent_enc = "decoder.sentence_encoder"
new_stat_dict["roberta.embeddings.word_embeddings.weight"] = stat_dict[sent_enc + ".embed_tokens.weight"]
new_stat_dict["roberta.embeddings.position_embeddings.weight"] = stat_dict[sent_enc + ".embed_positions.weight"]
new_stat_dict["roberta.embeddings.token_type_embeddings.weight"] = torch.zeros_like(
model.roberta.embeddings.token_type_embeddings.weight)
new_stat_dict["roberta.embeddings.LayerNorm.weight"] = stat_dict[sent_enc + ".emb_layer_norm.weight"]
new_stat_dict["roberta.embeddings.LayerNorm.bias"] = stat_dict[sent_enc + ".emb_layer_norm.bias"]
for i in range(config.num_hidden_layers):
# Encoder: start of layer
# layer: BertLayer = model.roberta.encoder.layer[i]
layer = "roberta.encoder.layer.%d" % i
roberta_layer = sent_enc + (".layers.%d" % i)
### self attention
# self_attn: BertSelfAttention = layer.attention.self
self_attn = layer + ".attention.self"
assert (
stat_dict[roberta_layer + ".self_attn.k_proj.weight"].data.shape == \
stat_dict[roberta_layer + ".self_attn.q_proj.weight"].data.shape == \
stat_dict[roberta_layer + ".self_attn.v_proj.weight"].data.shape == \
torch.Size((config.hidden_size, config.hidden_size))
)
new_stat_dict[self_attn + ".query.weight"] = stat_dict[roberta_layer + ".self_attn.q_proj.weight"]
new_stat_dict[self_attn + ".query.bias"] = stat_dict[roberta_layer + ".self_attn.q_proj.bias"]
new_stat_dict[self_attn + ".key.weight"] = stat_dict[roberta_layer + ".self_attn.k_proj.weight"]
new_stat_dict[self_attn + ".key.bias"] = stat_dict[roberta_layer + ".self_attn.k_proj.bias"]
new_stat_dict[self_attn + ".value.weight"] = stat_dict[roberta_layer + ".self_attn.v_proj.weight"]
new_stat_dict[self_attn + ".value.bias"] = stat_dict[roberta_layer + ".self_attn.v_proj.bias"]
### self-attention output
# self_output: BertSelfOutput = layer.attention.output
self_output = layer + ".attention.output"
assert (
model.roberta.encoder.layer[i].attention.output.dense.weight.shape == stat_dict[
roberta_layer + ".self_attn.out_proj.weight"].shape
)
new_stat_dict[self_output + ".dense.weight"] = stat_dict[roberta_layer + ".self_attn.out_proj.weight"]
new_stat_dict[self_output + ".dense.bias"] = stat_dict[roberta_layer + ".self_attn.out_proj.bias"]
new_stat_dict[self_output + ".LayerNorm.weight"] = stat_dict[roberta_layer + ".self_attn_layer_norm.weight"]
new_stat_dict[self_output + ".LayerNorm.bias"] = stat_dict[roberta_layer + ".self_attn_layer_norm.bias"]
### intermediate
# intermediate: BertIntermediate = layer.intermediate
intermediate = layer + ".intermediate"
assert (
model.roberta.encoder.layer[i].intermediate.dense.weight.shape == stat_dict[
roberta_layer + ".fc1.weight"].shape
)
# TODO
new_stat_dict[intermediate + ".dense.weight"] = stat_dict[roberta_layer + ".fc1.weight"]
new_stat_dict[intermediate + ".dense.bias"] = stat_dict[roberta_layer + ".fc1.bias"]
### output
# bert_output: BertOutput = layer.output
bert_output = layer + ".output"
assert (
model.roberta.encoder.layer[i].output.dense.weight.shape == stat_dict[
roberta_layer + ".fc2.weight"].shape
)
new_stat_dict[bert_output + ".dense.weight"] = stat_dict[roberta_layer + ".fc2.weight"]
new_stat_dict[bert_output + ".dense.bias"] = stat_dict[roberta_layer + ".fc2.bias"]
new_stat_dict[bert_output + ".LayerNorm.weight"] = stat_dict[roberta_layer + ".final_layer_norm.weight"]
new_stat_dict[bert_output + ".LayerNorm.bias"] = stat_dict[roberta_layer + ".final_layer_norm.bias"]
#### end of layer
new_stat_dict["lm_head.dense.weight"] = stat_dict["decoder.lm_head.dense.weight"]
new_stat_dict["lm_head.dense.bias"] = stat_dict["decoder.lm_head.dense.bias"]
new_stat_dict["lm_head.layer_norm.weight"] = stat_dict["decoder.lm_head.layer_norm.weight"]
new_stat_dict["lm_head.layer_norm.bias"] = stat_dict["decoder.lm_head.layer_norm.bias"]
new_stat_dict["lm_head.decoder.weight"] = stat_dict["decoder.lm_head.weight"]
new_stat_dict["lm_head.bias"] = stat_dict["decoder.lm_head.bias"]
new_stat_dict["lm_head.decoder.bias"] = stat_dict["decoder.lm_head.bias"]
new_stat_dict["roberta.pooler.dense.weight"] = model.roberta.pooler.dense.weight
new_stat_dict["roberta.pooler.dense.bias"] = model.roberta.pooler.dense.bias
return new_stat_dict
def update_hf_sd(old_sd, xlmr_path):
x = torch.load(xlmr_path, map_location="cpu")
m = old_sd
d = OrderedDict()
for k, v in m.items():
if k == 'roberta.pooler.dense.weight':
d[k] = x[k].half().clone()
elif k not in ('proj_matrix_fast', 'lm_head.decoder.bias', 'roberta.pooler.dense.weight'):
d[k] = v.data.half().clone()
assert set(d.keys()) == set(x.keys())
for k in d.keys():
assert d[k].size() == x[k].size()
for k in d.keys():
if k != 'roberta.pooler.dense.weight':
assert (d[k].float() - m[k].float()).abs().max().item() <= 1e-4
return d
def convert_pt_to_hf(xlmr_path, inf, logger=None):
if logger:
logger.info("converting pt file at {} to hf file.".format(inf))
sd = convert_cxlm_to_transformers(inf)
return update_hf_sd(sd, xlmr_path)
if __name__ == "__main__":
import os
xlmr_path = "/home/v-zechi/data/unilm/zechi/exp/res/huggingface/hf-ckpt/xlmr-large/pytorch_model.bin"
inf = "/home/v-zechi/data/unilm/zechi/exp/cxlm_exp/dump-ifx94-large/checkpoint_2_200000.pt"
outf = "/home/v-zechi/data/unilm/zechi/usw/res/infoxlm-models/huggingface/infoxlm-large-without-meta/pytorch_model.bin"
assert not os.path.exists(outf)
sd = convert_cxlm_to_transformers(inf)
sd2 = update_hf_sd(sd, xlmr_path)
torch.save(sd2, outf)
xtune/src/tools/get_eval_results.py
+272
View File
@@ -0,0 +1,272 @@
import argparse
import json
import glob
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--log_path",
default=None,
type=str,
required=True,
help="path to evaluation log file.",
)
parser.add_argument(
"--task_name",
default=None,
type=str,
required=True,
help="task name",
)
args = parser.parse_args()
all_res_terms = []
all_best = 0.0
all_final_res = None
for path in sorted(glob.glob(args.log_path), key=lambda x: (len(x), x)):
if args.task_name != "squad" and path.find("squad") != -1:
continue
# print(path)
best = 0.0
final_res = None
lines = open(path).readlines()
for line in lines:
line = line.strip()
if args.task_name == "squad":
json_str = line.split('\t')[0]
else:
json_str = line.split('\t')[1]
# print(json_str)
res = json.loads(json_str)
if args.task_name.lower() == "xnli" or args.task_name.lower() == "pawsx":
if res["valid_avg"]["acc"] > best:
# print(res)
best = res["valid_avg"]["acc"]
final_res = res
elif args.task_name.lower() in ["panx", "udpos"]:
if "dev_avg" not in res or res["dev_avg"]["f1"] > best:
# print(res)
if "dev_avg" in res:
best = res["dev_avg"]["f1"]
final_res = res
elif args.task_name.lower() in ["xquad", "mlqa", "tydiqa", "squad"]:
final_res = res
if args.task_name in ["xnli", "pawsx", "udpos", "panx"]:
if best > all_best:
all_best = best
all_final_res = final_res
elif args.task_name == "tydiqa":
if final_res["dev_en"]["f1"] > all_best:
all_best = final_res["dev_en"]["f1"]
all_final_res = final_res
elif args.task_name == "mlqa":
if final_res["dev_avg"]["f1"] > all_best:
all_best = final_res["dev_avg"]["f1"]
all_final_res = final_res
else:
pass
res_terms = []
try:
if args.task_name == "xnli":
# order = ["en", "fr", "es", "de", "el", "bg", "ru", "tr", "ar", "vi", "th", "zh", "hi", "sw", "ur"]
order = ["en", "ar", "bg", "de", "el", "es", "fr", "hi", "ru", "sw", "th", "tr", "ur", "vi", "zh"]
acc = {}
final_output = ""
for k in final_res:
if k.find("-") == -1:
continue
mode, lang = k.split("-")
if mode == "test":
# final_output += lang + " " + str(final_res[k]["acc"]) + '\t'
acc[lang] = round(final_res[k]["acc"] * 100, 2)
gap_sum = 0.0
for lang in order:
res_terms.append(acc[lang])
gap_sum += acc["en"] - acc[lang]
res_terms.append(round(final_res["test_avg"]["acc"] * 100, 2))
res_terms.append(round(final_res["valid_avg"]["acc"] * 100, 2))
res_terms.append(round(gap_sum / (len(res_terms) - 3), 2))
for term in res_terms:
final_output += str(term) + ","
print(final_output[:-1])
elif args.task_name == "pawsx":
# order = ["de", "en", "es", "fr", "ja", "ko", "zh"]
order = ["en", "de", "es", "fr", "ja", "ko", "zh"]
acc = {}
final_output = ""
for k in final_res:
if k.find("-") == -1:
continue
mode, lang = k.split("-")
if mode == "test":
# final_output += lang + " " + str(final_res[k]["acc"]) + '\t'
acc[lang] = round(final_res[k]["acc"] * 100, 2)
for lang in order:
res_terms.append(acc[lang])
res_terms.append(round(final_res["test_avg"]["acc"] * 100, 2))
res_terms.append(round(final_res["valid_avg"]["acc"] * 100, 2))
for term in res_terms:
final_output += str(term) + ","
print(final_output[:-1])
elif args.task_name == "panx":
# order = ["ar", "he", "vi", "id", "jv", "ms", "tl", "eu", "ml", "ta", "te", "af", "nl", "en", "de", "el",
# "bn", "hi", "mr", "ur", "fa", "fr", "it", "pt", "es", "bg", "ru", "ja", "ka", "ko", "th", "sw",
# "yo", "my", "zh", "kk", "tr", "et", "fi", "hu"]
order = ["en", "af", "ar", "bg", "bn", "de", "el", "es", "et", "eu", "fa", "fi", "fr", "he", "hi", "hu",
"id", "it", "ja", "jv", "ka", "kk", "ko", "ml", "mr", "ms", "my", "nl", "pt", "ru", "sw", "ta",
"te", "th", "tl", "tr", "ur", "vi", "yo", "zh"]
f1 = {}
final_output = ""
for k in final_res:
if k.find("_") == -1:
continue
mode, lang = k.split("_")
if mode == "test":
f1[lang] = round(final_res[k]["f1"] * 100, 1)
for lang in order:
res_terms.append(f1[lang])
res_terms.append(round(final_res["test_avg"]["f1"] * 100, 1))
res_terms.append(round(final_res["dev_avg"]["f1"] * 100, 1))
for term in res_terms:
final_output += str(term) + ","
print(final_output[:-1])
elif args.task_name == "udpos":
order = ["af", "ar", "bg", "de", "el", "en", "es", "et", "eu", "fa", "fi", "fr", "he", "hi", "hu", "id",
"it", "ja", "kk", "ko", "mr", "nl", "pt", "ru", "ta", "te", "th", "tl", "tr", "ur", "vi", "yo",
"zh"]
f1 = {}
final_output = ""
for k in final_res:
if k.find("_") == -1:
continue
mode, lang = k.split("_")
if mode == "test":
f1[lang] = round(final_res[k]["f1"] * 100, 1)
gap_sum = 0.0
for lang in order:
if lang in f1:
res_terms.append(f1[lang])
gap_sum += f1["en"] - f1[lang]
res_terms.append(round(final_res["test_avg"]["f1"] * 100, 1))
res_terms.append(round(final_res["dev_avg"]["f1"] * 100, 1))
res_terms.append(round(gap_sum / (len(res_terms) - 3), 1))
for term in res_terms:
final_output += str(term) + ","
print(final_output[:-1])
elif args.task_name == "mlqa":
# order = ["en", "es", "de", "ar", "hi", "vi", "zh"]
order = ["en", "ar", "de", "es", "hi", "vi", "zh"]
res = {}
final_output = ""
for k in final_res:
if k.find("_") == -1:
continue
mode, lang = k.split("_")
if mode == "test":
res[lang] = (round(final_res[k]["f1"], 1), round(final_res[k]["exact_match"], 1))
gap_sum_f1, gap_sum_em = 0, 0
for lang in order:
res_terms.append((res[lang][0], res[lang][1]))
gap_sum_f1 += res["en"][0] - res[lang][0]
gap_sum_em += res["en"][1] - res[lang][1]
res_terms.append(
(round(final_res["test_avg"]["f1"], 1), round(final_res["test_avg"]["exact_match"], 1)))
res_terms.append(
(round(final_res["dev_avg"]["f1"], 1), round(final_res["dev_avg"]["exact_match"], 1)))
res_terms.append(
(round(gap_sum_f1 / (len(res_terms) - 3), 2), round(gap_sum_em / (len(res_terms) - 3), 2)))
for term in res_terms:
final_output += str(term[0]) + '/' + str(term[1]) + ','
print(final_output[:-1])
elif args.task_name == "xquad":
# order = ["ar", "de", "el", "en", "es", "hi", "ru", "th", "tr", "vi", "zh"]
order = ["en", "ar", "de", "el", "es", "hi", "ru", "th", "tr", "vi", "zh"]
res = {}
final_output = ""
for k in final_res:
if k.find("_") == -1:
continue
mode, lang = k.split("_")
if mode == "dev":
res[lang] = (round(final_res[k]["f1"], 2), round(final_res[k]["exact_match"], 2))
for lang in order:
res_terms.append((res[lang][0], res[lang][1]))
res_terms.append(
(round(final_res["dev_avg"]["f1"], 2), round(final_res["dev_avg"]["exact_match"], 2)))
for term in res_terms:
final_output += str(term[0]) + '/' + str(term[1]) + ','
print(final_output[:-1])
elif args.task_name == "tydiqa":
order = ["en", "ar", "bn", "fi", "id", "ko", "ru", "sw", "te"]
res = {}
final_output = ""
for k in final_res:
if k.find("_") == -1:
continue
mode, lang = k.split("_")
if mode == "dev":
res[lang] = (round(final_res[k]["f1"], 2), round(final_res[k]["exact_match"], 2))
for lang in order:
res_terms.append((res[lang][0], res[lang][1]))
res_terms.append(
(round(final_res["dev_avg"]["f1"], 2), round(final_res["dev_avg"]["exact_match"], 2)))
for term in res_terms:
final_output += str(term[0]) + '/' + str(term[1]) + ','
print(final_output[:-1])
elif args.task_name == "squad":
res_terms.append(
(round(final_res["dev_en"]["f1"], 2), round(final_res["dev_en"]["exact_match"], 2)))
final_output = ""
for term in res_terms:
final_output += str(term[0]) + '/' + str(term[1]) + ','
print(final_output[:-1])
pass
all_res_terms.append(res_terms)
except:
print()
pass
print(order)
print(all_final_res)
final_res_terms = []
for i in range(len(all_res_terms[0])):
s = 0.0 if args.task_name in ['xnli', "panx", "udpos", "pawsx"] else [0.0, 0.0]
for j in range(len(all_res_terms)):
if args.task_name in ['xnli', "panx", "udpos", "pawsx"]:
s += all_res_terms[j][i]
else:
s[0] += all_res_terms[j][i][0]
s[1] += all_res_terms[j][i][1]
final_res_terms.append(
round(s / len(all_res_terms), 2) if args.task_name in ['xnli', "panx", "udpos", "pawsx"] else (
round(s[0] / len(all_res_terms), 2), round(s[1] / len(all_res_terms), 2)))
final_output = '{}-average: '.format(len(all_res_terms))
if args.task_name in ['xnli', "panx", "udpos", "pawsx"]:
for term in final_res_terms:
final_output += str(term) + ","
print(final_output[:-1])
else:
for term in final_res_terms:
final_output += str(term[0]) + '/' + str(term[1]) + ','
print(final_output[:-1])
xtune/src/tools/sample_xnli.py
+43
View File
@@ -0,0 +1,43 @@
import argparse
import json
import random
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--input_path",
default=None,
type=str,
required=True,
help="input xnli file",
)
parser.add_argument(
"--output_path",
default=None,
type=str,
required=True,
help="output xnli file",
)
parser.add_argument(
"--sample_ratio",
default=None,
type=float,
required=True,
help="sample ratio",
)
args = parser.parse_args()
lines = open(args.input_path, "r").readlines()
head = lines[0]
lines = lines[1:]
random.seed(0)
random.shuffle(lines)
n_lines = int(len(lines) * args.sample_ratio)
fout = open(args.output_path, "w")
fout.write(head)
for i, line in enumerate(lines[:n_lines]):
fout.write(line)
xtune/src/tools/xnli_sampling_statistics.py
+159
View File
@@ -0,0 +1,159 @@
import argparse
import json
import random
from transformers import XLMRobertaTokenizer
from transformers import xglue_processors as processors
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--data_dir",
default=None,
type=str,
required=True,
help="The input data dir. Should contain the .tsv files (or other data files) for the task.",
)
parser.add_argument(
"--nbest_size", type=int, default=-1, help="nbest size in sampling subword sequence"
)
parser.add_argument(
"--alpha", type=float, default=0.2, help="alpha"
)
parser.add_argument(
"--train_language", default=None, type=str, help="Train language if is different of the evaluation language."
)
parser.add_argument(
"--language",
default=None,
type=str,
required=True,
help="Evaluation language. Also train language if `train_language` is set to None.",
)
parser.add_argument(
"--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
)
parser.add_argument(
"--model_name_or_path",
default=None,
type=str,
required=True,
help="Path to pre-trained model",
)
parser.add_argument(
"--sample_rounds",
default=1,
type=int,
required=True,
help="Path to pre-trained model",
)
args = parser.parse_args()
tokenizer = XLMRobertaTokenizer.from_pretrained(
args.model_name_or_path,
do_lower_case=args.do_lower_case,
cache_dir=None,
)
task = "xnli"
processor = processors[task](language=args.train_language, train_language=args.train_language)
examples = processor.get_train_examples(args.data_dir)
train_word_cnt_origin = {}
for example in examples:
tokens_a = tokenizer.tokenize(example.text_a, add_special_tokens=True)
tokens_b = tokenizer.tokenize(example.text_b, add_special_tokens=True)
for token in tokens_a + tokens_b:
if token not in train_word_cnt_origin:
train_word_cnt_origin[token] = 0
train_word_cnt_origin[token] += 1
all_examples = []
for i in range(args.sample_rounds):
all_examples += examples
examples = all_examples
sent_len = []
example_len = []
train_word_cnt = {}
for example in examples:
tokens_a = tokenizer.tokenize(example.text_a, add_special_tokens=True, nbest_size=args.nbest_size,
alpha=args.alpha)
tokens_b = tokenizer.tokenize(example.text_b, add_special_tokens=True, nbest_size=args.nbest_size,
alpha=args.alpha)
for token in tokens_a + tokens_b:
if token not in train_word_cnt:
train_word_cnt[token] = 0
train_word_cnt[token] += 1
sent_len += [len(tokens_a), len(tokens_b)]
example_len += [len(tokens_a) + len(tokens_b)]
print(sum(sent_len) / len(sent_len))
print(sum(example_len) / len(example_len))
total = 0
n_oov = 0
for token in train_word_cnt:
if token not in train_word_cnt_origin:
n_oov += train_word_cnt[token]
# n_oov += 1
total += train_word_cnt[token]
# total += 1
print("{} oov rate: {}".format("extra", n_oov / total))
total = 0
n_oov = 0
for token in train_word_cnt_origin:
if token not in train_word_cnt:
n_oov += train_word_cnt_origin[token]
# n_oov += 1
total += train_word_cnt_origin[token]
# total += 1
print("{} oov rate: {}".format("origin", n_oov / total))
# exit(0)
eval_datasets = []
eval_langs = args.language.split(',')
for split in ["valid"]:
for lang in eval_langs:
eval_datasets.append((split, lang))
for split, lang in eval_datasets:
processor = processors[task](language=lang, train_language=lang)
examples = processor.get_valid_examples(args.data_dir)
sent_len = []
example_len = []
valid_word_cnt = {}
for example in examples:
tokens_a = tokenizer.tokenize(example.text_a, add_special_tokens=True)
tokens_b = tokenizer.tokenize(example.text_b, add_special_tokens=True)
for token in tokens_a + tokens_b:
if token not in valid_word_cnt:
valid_word_cnt[token] = 0
valid_word_cnt[token] += 1
sent_len += [len(tokens_a), len(tokens_b)]
example_len += [len(tokens_a) + len(tokens_b)]
print(sum(sent_len) / len(sent_len))
print(sum(example_len) / len(example_len))
total = 0
n_oov = 0
for token in valid_word_cnt:
if token not in train_word_cnt:
n_oov += valid_word_cnt[token]
# n_oov += 1
total += valid_word_cnt[token]
# total += 1
print("{} oov rate: {}".format(lang, n_oov / total))
xtune/src/transformers/__init__.py
+461
View File
@@ -0,0 +1,461 @@
# flake8: noqa
# There's no way to ignore "F401 '...' imported but unused" warnings in this
# module, but to preserve other warnings. So, don't check this module at all.
__version__ = "2.5.1"
# Work around to update TensorFlow's absl.logging threshold which alters the
# default Python logging output behavior when present.
# see: https://github.com/abseil/abseil-py/issues/99
# and: https://github.com/tensorflow/tensorflow/issues/26691#issuecomment-500369493
try:
import absl.logging
except ImportError:
pass
else:
absl.logging.set_verbosity("info")
absl.logging.set_stderrthreshold("info")
absl.logging._warn_preinit_stderr = False
import logging
from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
from .configuration_auto import ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, AutoConfig
from .configuration_bart import BartConfig
from .configuration_bert import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP, BertConfig
from .configuration_camembert import CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, CamembertConfig
from .configuration_ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig
from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig
from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
from .configuration_mmbt import MMBTConfig
from .configuration_openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig
from .configuration_roberta import ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, RobertaConfig
from .configuration_t5 import T5_PRETRAINED_CONFIG_ARCHIVE_MAP, T5Config
from .configuration_transfo_xl import TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP, TransfoXLConfig
# Configurations
from .configuration_utils import PretrainedConfig
from .configuration_xlm import XLM_PRETRAINED_CONFIG_ARCHIVE_MAP, XLMConfig
from .configuration_xlm_roberta import XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, XLMRobertaConfig
from .configuration_xlnet import XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP, XLNetConfig
from .data import (
DataProcessor,
InputExample,
InputFeatures,
SingleSentenceClassificationProcessor,
SquadExample,
SquadFeatures,
SquadV1Processor,
SquadV2Processor,
xtreme_convert_examples_to_features,
xtreme_output_modes,
xtreme_processors,
xtreme_tasks_num_labels,
xglue_convert_examples_to_features,
xglue_output_modes,
xglue_processors,
xglue_tasks_num_labels,
glue_convert_examples_to_features,
glue_output_modes,
glue_processors,
glue_tasks_num_labels,
is_sklearn_available,
squad_convert_examples_to_features,
xnli_output_modes,
xnli_processors,
xnli_tasks_num_labels,
)
# Files and general utilities
from .file_utils import (
CONFIG_NAME,
MODEL_CARD_NAME,
PYTORCH_PRETRAINED_BERT_CACHE,
PYTORCH_TRANSFORMERS_CACHE,
TF2_WEIGHTS_NAME,
TF_WEIGHTS_NAME,
TRANSFORMERS_CACHE,
WEIGHTS_NAME,
add_end_docstrings,
add_start_docstrings,
cached_path,
is_tf_available,
is_torch_available,
)
# Model Cards
from .modelcard import ModelCard
# TF 2.0 <=> PyTorch conversion utilities
from .modeling_tf_pytorch_utils import (
convert_tf_weight_name_to_pt_weight_name,
load_pytorch_checkpoint_in_tf2_model,
load_pytorch_model_in_tf2_model,
load_pytorch_weights_in_tf2_model,
load_tf2_checkpoint_in_pytorch_model,
load_tf2_model_in_pytorch_model,
load_tf2_weights_in_pytorch_model,
)
# Pipelines
from .pipelines import (
CsvPipelineDataFormat,
FeatureExtractionPipeline,
FillMaskPipeline,
JsonPipelineDataFormat,
NerPipeline,
PipedPipelineDataFormat,
Pipeline,
PipelineDataFormat,
QuestionAnsweringPipeline,
TextClassificationPipeline,
TokenClassificationPipeline,
pipeline,
)
from .tokenization_albert import AlbertTokenizer
from .tokenization_auto import AutoTokenizer
from .tokenization_bart import BartTokenizer
from .tokenization_bert import BasicTokenizer, BertTokenizer, BertTokenizerFast, WordpieceTokenizer
from .tokenization_bert_japanese import BertJapaneseTokenizer, CharacterTokenizer, MecabTokenizer
from .tokenization_camembert import CamembertTokenizer
from .tokenization_ctrl import CTRLTokenizer
from .tokenization_distilbert import DistilBertTokenizer, DistilBertTokenizerFast
from .tokenization_flaubert import FlaubertTokenizer
from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
from .tokenization_openai import OpenAIGPTTokenizer, OpenAIGPTTokenizerFast
from .tokenization_roberta import RobertaTokenizer, RobertaTokenizerFast
from .tokenization_t5 import T5Tokenizer
from .tokenization_transfo_xl import TransfoXLCorpus, TransfoXLTokenizer, TransfoXLTokenizerFast
# Tokenizers
from .tokenization_utils import PreTrainedTokenizer
from .tokenization_xlm import XLMTokenizer
from .tokenization_xlm_roberta import XLMRobertaTokenizer
from .tokenization_xlnet import SPIECE_UNDERLINE, XLNetTokenizer
logger = logging.getLogger(__name__) # pylint: disable=invalid-name
if is_sklearn_available():
from .data import glue_compute_metrics, xnli_compute_metrics, xglue_compute_metrics, xtreme_compute_metrics
# Modeling
if is_torch_available():
from .modeling_utils import PreTrainedModel, prune_layer, Conv1D, top_k_top_p_filtering
from .modeling_auto import (
AutoModel,
AutoModelForPreTraining,
AutoModelForSequenceClassification,
AutoModelForQuestionAnswering,
AutoModelWithLMHead,
AutoModelForTokenClassification,
ALL_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_bert import (
BertPreTrainedModel,
BertModel,
BertForPreTraining,
BertForMaskedLM,
BertForNextSentencePrediction,
BertForMultiTaskSequenceClassification,
BertForSequenceClassification,
BertForMultipleChoice,
BertForTokenClassification,
BertForQuestionAnswering,
load_tf_weights_in_bert,
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_openai import (
OpenAIGPTPreTrainedModel,
OpenAIGPTModel,
OpenAIGPTLMHeadModel,
OpenAIGPTDoubleHeadsModel,
load_tf_weights_in_openai_gpt,
OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_transfo_xl import (
TransfoXLPreTrainedModel,
TransfoXLModel,
TransfoXLLMHeadModel,
AdaptiveEmbedding,
load_tf_weights_in_transfo_xl,
TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_gpt2 import (
GPT2PreTrainedModel,
GPT2Model,
GPT2LMHeadModel,
GPT2DoubleHeadsModel,
load_tf_weights_in_gpt2,
GPT2_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_ctrl import CTRLPreTrainedModel, CTRLModel, CTRLLMHeadModel, CTRL_PRETRAINED_MODEL_ARCHIVE_MAP
from .modeling_xlnet import (
XLNetPreTrainedModel,
XLNetModel,
XLNetLMHeadModel,
XLNetForSequenceClassification,
XLNetForTokenClassification,
XLNetForMultipleChoice,
XLNetForQuestionAnsweringSimple,
XLNetForQuestionAnswering,
load_tf_weights_in_xlnet,
XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_xlm import (
XLMPreTrainedModel,
XLMModel,
XLMWithLMHeadModel,
XLMForSequenceClassification,
XLMForQuestionAnswering,
XLMForQuestionAnsweringSimple,
XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_bart import BartForSequenceClassification, BartModel, BartForMaskedLM
from .modeling_roberta import (
RobertaForMaskedLM,
RobertaModel,
RobertaForSequenceClassification,
RobertaForMultiTaskSequenceClassification,
RobertaForMultipleChoice,
RobertaForTokenClassification,
RobertaForQuestionAnswering,
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_camembert import (
CamembertForMaskedLM,
CamembertModel,
CamembertForSequenceClassification,
CamembertForTokenClassification,
CamembertForQuestionAnswering,
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_distilbert import (
DistilBertPreTrainedModel,
DistilBertForMaskedLM,
DistilBertModel,
DistilBertForSequenceClassification,
DistilBertForQuestionAnswering,
DistilBertForTokenClassification,
DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_camembert import (
CamembertForMaskedLM,
CamembertModel,
CamembertForSequenceClassification,
CamembertForMultipleChoice,
CamembertForTokenClassification,
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_encoder_decoder import PreTrainedEncoderDecoder
from .modeling_t5 import (
T5PreTrainedModel,
T5Model,
T5WithLMHeadModel,
load_tf_weights_in_t5,
T5_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_albert import (
AlbertPreTrainedModel,
AlbertModel,
AlbertForMaskedLM,
AlbertForSequenceClassification,
AlbertForQuestionAnswering,
AlbertForTokenClassification,
load_tf_weights_in_albert,
ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_xlm_roberta import (
XLMRobertaForMaskedLM,
XLMRobertaModel,
XLMRobertaForRetrieval,
XLMRobertaForMultipleChoice,
XLMRobertaForSequenceClassification,
XLMRobertaForSequenceClassificationStable,
XLMRobertaForSequenceClassificationConsistency,
XLMRobertaForMultiTaskSequenceClassification,
XLMRobertaForTokenClassification,
XLMRobertaForTokenClassificationPoolingStable,
XLMRobertaForQuestionAnswering,
XLMRobertaForQuestionAnsweringStable,
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_mmbt import ModalEmbeddings, MMBTModel, MMBTForClassification
from .modeling_flaubert import (
FlaubertModel,
FlaubertWithLMHeadModel,
FlaubertForSequenceClassification,
FlaubertForQuestionAnswering,
FlaubertForQuestionAnsweringSimple,
FLAUBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
# Optimization
from .optimization import (
AdamW,
get_constant_schedule,
get_constant_schedule_with_warmup,
get_cosine_schedule_with_warmup,
get_cosine_with_hard_restarts_schedule_with_warmup,
get_linear_schedule_with_warmup,
)
# TensorFlow
if is_tf_available():
from .modeling_tf_utils import (
TFPreTrainedModel,
TFSharedEmbeddings,
TFSequenceSummary,
shape_list,
tf_top_k_top_p_filtering,
)
from .modeling_tf_auto import (
TFAutoModel,
TFAutoModelForPreTraining,
TFAutoModelForSequenceClassification,
TFAutoModelForQuestionAnswering,
TFAutoModelWithLMHead,
TFAutoModelForTokenClassification,
TF_ALL_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_bert import (
TFBertPreTrainedModel,
TFBertMainLayer,
TFBertEmbeddings,
TFBertModel,
TFBertForPreTraining,
TFBertForMaskedLM,
TFBertForNextSentencePrediction,
TFBertForSequenceClassification,
TFBertForMultipleChoice,
TFBertForTokenClassification,
TFBertForQuestionAnswering,
TF_BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_gpt2 import (
TFGPT2PreTrainedModel,
TFGPT2MainLayer,
TFGPT2Model,
TFGPT2LMHeadModel,
TFGPT2DoubleHeadsModel,
TF_GPT2_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_openai import (
TFOpenAIGPTPreTrainedModel,
TFOpenAIGPTMainLayer,
TFOpenAIGPTModel,
TFOpenAIGPTLMHeadModel,
TFOpenAIGPTDoubleHeadsModel,
TF_OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_transfo_xl import (
TFTransfoXLPreTrainedModel,
TFTransfoXLMainLayer,
TFTransfoXLModel,
TFTransfoXLLMHeadModel,
TF_TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_xlnet import (
TFXLNetPreTrainedModel,
TFXLNetMainLayer,
TFXLNetModel,
TFXLNetLMHeadModel,
TFXLNetForSequenceClassification,
TFXLNetForTokenClassification,
TFXLNetForQuestionAnsweringSimple,
TF_XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_xlm import (
TFXLMPreTrainedModel,
TFXLMMainLayer,
TFXLMModel,
TFXLMWithLMHeadModel,
TFXLMForSequenceClassification,
TFXLMForQuestionAnsweringSimple,
TF_XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_xlm_roberta import (
TFXLMRobertaForMaskedLM,
TFXLMRobertaModel,
TFXLMRobertaForSequenceClassification,
TFXLMRobertaForTokenClassification,
TF_XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_roberta import (
TFRobertaPreTrainedModel,
TFRobertaMainLayer,
TFRobertaModel,
TFRobertaForMaskedLM,
TFRobertaForSequenceClassification,
TFRobertaForTokenClassification,
TF_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_camembert import (
TFCamembertModel,
TFCamembertForMaskedLM,
TFCamembertForSequenceClassification,
TFCamembertForTokenClassification,
TF_CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_distilbert import (
TFDistilBertPreTrainedModel,
TFDistilBertMainLayer,
TFDistilBertModel,
TFDistilBertForMaskedLM,
TFDistilBertForSequenceClassification,
TFDistilBertForTokenClassification,
TFDistilBertForQuestionAnswering,
TF_DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_ctrl import (
TFCTRLPreTrainedModel,
TFCTRLModel,
TFCTRLLMHeadModel,
TF_CTRL_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_albert import (
TFAlbertPreTrainedModel,
TFAlbertModel,
TFAlbertForMaskedLM,
TFAlbertForSequenceClassification,
TF_ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
)
from .modeling_tf_t5 import (
TFT5PreTrainedModel,
TFT5Model,
TFT5WithLMHeadModel,
TF_T5_PRETRAINED_MODEL_ARCHIVE_MAP,
)
# Optimization
from .optimization_tf import WarmUp, create_optimizer, AdamWeightDecay, GradientAccumulator
if not is_tf_available() and not is_torch_available():
logger.warning(
"Neither PyTorch nor TensorFlow >= 2.0 have been found."
"Models won't be available and only tokenizers, configuration"
"and file/data utilities can be used."
)
xtune/src/transformers/activations.py
+51
View File
@@ -0,0 +1,51 @@
import math
import torch
import torch.nn.functional as F
def swish(x):
return x * torch.sigmoid(x)
def _gelu_python(x):
""" Original Implementation of the gelu activation function in Google Bert repo when initially created.
For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
This is now written in C in torch.nn.functional
Also see https://arxiv.org/abs/1606.08415
"""
return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
if torch.__version__ < "1.4.0":
gelu = _gelu_python
else:
gelu = F.gelu
def gelu_new(x):
""" Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
Also see https://arxiv.org/abs/1606.08415
"""
return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
ACT2FN = {
"relu": F.relu,
"swish": swish,
"gelu": gelu,
"tanh": F.tanh,
"gelu_new": gelu_new,
}
def get_activation(activation_string):
if activation_string in ACT2FN:
return ACT2FN[activation_string]
else:
raise KeyError(
"function {} not found in ACT2FN mapping {} or torch.nn.functional".format(
activation_string, list(ACT2FN.keys())
)
)
xtune/src/transformers/commands/__init__.py
+13
View File
@@ -0,0 +1,13 @@
from abc import ABC, abstractmethod
from argparse import ArgumentParser
class BaseTransformersCLICommand(ABC):
@staticmethod
@abstractmethod
def register_subcommand(parser: ArgumentParser):
raise NotImplementedError()
@abstractmethod
def run(self):
raise NotImplementedError()
xtune/src/transformers/commands/convert.py
+144
View File
@@ -0,0 +1,144 @@
from argparse import ArgumentParser, Namespace
from logging import getLogger
from transformers.commands import BaseTransformersCLICommand
def convert_command_factory(args: Namespace):
"""
Factory function used to convert a model TF 1.0 checkpoint in a PyTorch checkpoint.
:return: ServeCommand
"""
return ConvertCommand(
args.model_type, args.tf_checkpoint, args.pytorch_dump_output, args.config, args.finetuning_task_name
)
class ConvertCommand(BaseTransformersCLICommand):
@staticmethod
def register_subcommand(parser: ArgumentParser):
"""
Register this command to argparse so it's available for the transformer-cli
:param parser: Root parser to register command-specific arguments
:return:
"""
train_parser = parser.add_parser(
"convert",
help="CLI tool to run convert model from original "
"author checkpoints to Transformers PyTorch checkpoints.",
)
train_parser.add_argument("--model_type", type=str, required=True, help="Model's type.")
train_parser.add_argument(
"--tf_checkpoint", type=str, required=True, help="TensorFlow checkpoint path or folder."
)
train_parser.add_argument(
"--pytorch_dump_output", type=str, required=True, help="Path to the PyTorch savd model output."
)
train_parser.add_argument("--config", type=str, default="", help="Configuration file path or folder.")
train_parser.add_argument(
"--finetuning_task_name",
type=str,
default=None,
help="Optional fine-tuning task name if the TF model was a finetuned model.",
)
train_parser.set_defaults(func=convert_command_factory)
def __init__(
self,
model_type: str,
tf_checkpoint: str,
pytorch_dump_output: str,
config: str,
finetuning_task_name: str,
*args
):
self._logger = getLogger("transformers-cli/converting")
self._logger.info("Loading model {}".format(model_type))
self._model_type = model_type
self._tf_checkpoint = tf_checkpoint
self._pytorch_dump_output = pytorch_dump_output
self._config = config
self._finetuning_task_name = finetuning_task_name
def run(self):
if self._model_type == "bert":
try:
from transformers.convert_bert_original_tf_checkpoint_to_pytorch import (
convert_tf_checkpoint_to_pytorch,
)
except ImportError:
msg = (
"transformers can only be used from the commandline to convert TensorFlow models in PyTorch, "
"In that case, it requires TensorFlow to be installed. Please see "
"https://www.tensorflow.org/install/ for installation instructions."
)
raise ImportError(msg)
convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
elif self._model_type == "gpt":
from transformers.convert_openai_original_tf_checkpoint_to_pytorch import (
convert_openai_checkpoint_to_pytorch,
)
convert_openai_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
elif self._model_type == "transfo_xl":
try:
from transformers.convert_transfo_xl_original_tf_checkpoint_to_pytorch import (
convert_transfo_xl_checkpoint_to_pytorch,
)
except ImportError:
msg = (
"transformers can only be used from the commandline to convert TensorFlow models in PyTorch, "
"In that case, it requires TensorFlow to be installed. Please see "
"https://www.tensorflow.org/install/ for installation instructions."
)
raise ImportError(msg)
if "ckpt" in self._tf_checkpoint.lower():
TF_CHECKPOINT = self._tf_checkpoint
TF_DATASET_FILE = ""
else:
TF_DATASET_FILE = self._tf_checkpoint
TF_CHECKPOINT = ""
convert_transfo_xl_checkpoint_to_pytorch(
TF_CHECKPOINT, self._config, self._pytorch_dump_output, TF_DATASET_FILE
)
elif self._model_type == "gpt2":
try:
from transformers.convert_gpt2_original_tf_checkpoint_to_pytorch import (
convert_gpt2_checkpoint_to_pytorch,
)
except ImportError:
msg = (
"transformers can only be used from the commandline to convert TensorFlow models in PyTorch, "
"In that case, it requires TensorFlow to be installed. Please see "
"https://www.tensorflow.org/install/ for installation instructions."
)
raise ImportError(msg)
convert_gpt2_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
elif self._model_type == "xlnet":
try:
from transformers.convert_xlnet_original_tf_checkpoint_to_pytorch import (
convert_xlnet_checkpoint_to_pytorch,
)
except ImportError:
msg = (
"transformers can only be used from the commandline to convert TensorFlow models in PyTorch, "
"In that case, it requires TensorFlow to be installed. Please see "
"https://www.tensorflow.org/install/ for installation instructions."
)
raise ImportError(msg)
convert_xlnet_checkpoint_to_pytorch(
self._tf_checkpoint, self._config, self._pytorch_dump_output, self._finetuning_task_name
)
elif self._model_type == "xlm":
from transformers.convert_xlm_original_pytorch_checkpoint_to_pytorch import (
convert_xlm_checkpoint_to_pytorch,
)
convert_xlm_checkpoint_to_pytorch(self._tf_checkpoint, self._pytorch_dump_output)
else:
raise ValueError("--model_type should be selected in the list [bert, gpt, gpt2, transfo_xl, xlnet, xlm]")
xtune/src/transformers/commands/download.py
+32
View File
@@ -0,0 +1,32 @@
from argparse import ArgumentParser
from transformers.commands import BaseTransformersCLICommand
def download_command_factory(args):
return DownloadCommand(args.model, args.cache_dir, args.force)
class DownloadCommand(BaseTransformersCLICommand):
@staticmethod
def register_subcommand(parser: ArgumentParser):
download_parser = parser.add_parser("download")
download_parser.add_argument(
"--cache-dir", type=str, default=None, help="Path to location to store the models"
)
download_parser.add_argument(
"--force", action="store_true", help="Force the model to be download even if already in cache-dir"
)
download_parser.add_argument("model", type=str, help="Name of the model to download")
download_parser.set_defaults(func=download_command_factory)
def __init__(self, model: str, cache: str, force: bool):
self._model = model
self._cache = cache
self._force = force
def run(self):
from transformers import AutoModel, AutoTokenizer
AutoModel.from_pretrained(self._model, cache_dir=self._cache, force_download=self._force)
AutoTokenizer.from_pretrained(self._model, cache_dir=self._cache, force_download=self._force)
xtune/src/transformers/commands/env.py
+58
View File
@@ -0,0 +1,58 @@
import platform
from argparse import ArgumentParser
from transformers import __version__ as version
from transformers import is_tf_available, is_torch_available
from transformers.commands import BaseTransformersCLICommand
def info_command_factory(_):
return EnvironmentCommand()
class EnvironmentCommand(BaseTransformersCLICommand):
@staticmethod
def register_subcommand(parser: ArgumentParser):
download_parser = parser.add_parser("env")
download_parser.set_defaults(func=info_command_factory)
def run(self):
pt_version = "not installed"
pt_cuda_available = "NA"
if is_torch_available():
import torch
pt_version = torch.__version__
pt_cuda_available = torch.cuda.is_available()
tf_version = "not installed"
tf_cuda_available = "NA"
if is_tf_available():
import tensorflow as tf
tf_version = tf.__version__
try:
# deprecated in v2.1
tf_cuda_available = tf.test.is_gpu_available()
except AttributeError:
# returns list of devices, convert to bool
tf_cuda_available = bool(tf.config.list_physical_devices("GPU"))
info = {
"`transformers` version": version,
"Platform": platform.platform(),
"Python version": platform.python_version(),
"PyTorch version (GPU?)": "{} ({})".format(pt_version, pt_cuda_available),
"Tensorflow version (GPU?)": "{} ({})".format(tf_version, tf_cuda_available),
"Using GPU in script?": "<fill in>",
"Using distributed or parallel set-up in script?": "<fill in>",
}
print("\nCopy-and-paste the text below in your GitHub issue and FILL OUT the two last points.\n")
print(self.format_dict(info))
return info
@staticmethod
def format_dict(d):
return "\n".join(["- {}: {}".format(prop, val) for prop, val in d.items()]) + "\n"
xtune/src/transformers/commands/run.py
+96
View File
@@ -0,0 +1,96 @@
import logging
from argparse import ArgumentParser
from transformers.commands import BaseTransformersCLICommand
from transformers.pipelines import SUPPORTED_TASKS, Pipeline, PipelineDataFormat, pipeline
logger = logging.getLogger(__name__) # pylint: disable=invalid-name
def try_infer_format_from_ext(path: str):
if not path:
return "pipe"
for ext in PipelineDataFormat.SUPPORTED_FORMATS:
if path.endswith(ext):
return ext
raise Exception(
"Unable to determine file format from file extension {}. "
"Please provide the format through --format {}".format(path, PipelineDataFormat.SUPPORTED_FORMATS)
)
def run_command_factory(args):
nlp = pipeline(
task=args.task,
model=args.model if args.model else None,
config=args.config,
tokenizer=args.tokenizer,
device=args.device,
)
format = try_infer_format_from_ext(args.input) if args.format == "infer" else args.format
reader = PipelineDataFormat.from_str(
format=format,
output_path=args.output,
input_path=args.input,
column=args.column if args.column else nlp.default_input_names,
overwrite=args.overwrite,
)
return RunCommand(nlp, reader)
class RunCommand(BaseTransformersCLICommand):
def __init__(self, nlp: Pipeline, reader: PipelineDataFormat):
self._nlp = nlp
self._reader = reader
@staticmethod
def register_subcommand(parser: ArgumentParser):
run_parser = parser.add_parser("run", help="Run a pipeline through the CLI")
run_parser.add_argument("--task", choices=SUPPORTED_TASKS.keys(), help="Task to run")
run_parser.add_argument("--input", type=str, help="Path to the file to use for inference")
run_parser.add_argument("--output", type=str, help="Path to the file that will be used post to write results.")
run_parser.add_argument("--model", type=str, help="Name or path to the model to instantiate.")
run_parser.add_argument("--config", type=str, help="Name or path to the model's config to instantiate.")
run_parser.add_argument(
"--tokenizer", type=str, help="Name of the tokenizer to use. (default: same as the model name)"
)
run_parser.add_argument(
"--column",
type=str,
help="Name of the column to use as input. (For multi columns input as QA use column1,columns2)",
)
run_parser.add_argument(
"--format",
type=str,
default="infer",
choices=PipelineDataFormat.SUPPORTED_FORMATS,
help="Input format to read from",
)
run_parser.add_argument(
"--device",
type=int,
default=-1,
help="Indicate the device to run onto, -1 indicates CPU, >= 0 indicates GPU (default: -1)",
)
run_parser.add_argument("--overwrite", action="store_true", help="Allow overwriting the output file.")
run_parser.set_defaults(func=run_command_factory)
def run(self):
nlp, outputs = self._nlp, []
for entry in self._reader:
output = nlp(**entry) if self._reader.is_multi_columns else nlp(entry)
if isinstance(output, dict):
outputs.append(output)
else:
outputs += output
# Saving data
if self._nlp.binary_output:
binary_path = self._reader.save_binary(outputs)
logger.warning("Current pipeline requires output to be in binary format, saving at {}".format(binary_path))
else:
self._reader.save(outputs)
xtune/src/transformers/commands/serving.py
+214
View File
@@ -0,0 +1,214 @@
import logging
from argparse import ArgumentParser, Namespace
from typing import Any, List, Optional
from transformers import Pipeline
from transformers.commands import BaseTransformersCLICommand
from transformers.pipelines import SUPPORTED_TASKS, pipeline
try:
from uvicorn import run
from fastapi import FastAPI, HTTPException, Body
from fastapi.routing import APIRoute
from pydantic import BaseModel
from starlette.responses import JSONResponse
_serve_dependencies_installed = True
except (ImportError, AttributeError):
BaseModel = object
def Body(*x, **y):
pass
_serve_dependencies_installed = False
logger = logging.getLogger("transformers-cli/serving")
def serve_command_factory(args: Namespace):
"""
Factory function used to instantiate serving server from provided command line arguments.
:return: ServeCommand
"""
nlp = pipeline(
task=args.task,
model=args.model if args.model else None,
config=args.config,
tokenizer=args.tokenizer,
device=args.device,
)
return ServeCommand(nlp, args.host, args.port, args.workers)
class ServeModelInfoResult(BaseModel):
"""
Expose model information
"""
infos: dict
class ServeTokenizeResult(BaseModel):
"""
Tokenize result model
"""
tokens: List[str]
tokens_ids: Optional[List[int]]
class ServeDeTokenizeResult(BaseModel):
"""
DeTokenize result model
"""
text: str
class ServeForwardResult(BaseModel):
"""
Forward result model
"""
output: Any
class ServeCommand(BaseTransformersCLICommand):
@staticmethod
def register_subcommand(parser: ArgumentParser):
"""
Register this command to argparse so it's available for the transformer-cli
:param parser: Root parser to register command-specific arguments
:return:
"""
serve_parser = parser.add_parser(
"serve", help="CLI tool to run inference requests through REST and GraphQL endpoints."
)
serve_parser.add_argument(
"--task", type=str, choices=SUPPORTED_TASKS.keys(), help="The task to run the pipeline on"
)
serve_parser.add_argument("--host", type=str, default="localhost", help="Interface the server will listen on.")
serve_parser.add_argument("--port", type=int, default=8888, help="Port the serving will listen to.")
serve_parser.add_argument("--workers", type=int, default=1, help="Number of http workers")
serve_parser.add_argument("--model", type=str, help="Model's name or path to stored model.")
serve_parser.add_argument("--config", type=str, help="Model's config name or path to stored model.")
serve_parser.add_argument("--tokenizer", type=str, help="Tokenizer name to use.")
serve_parser.add_argument(
"--device",
type=int,
default=-1,
help="Indicate the device to run onto, -1 indicates CPU, >= 0 indicates GPU (default: -1)",
)
serve_parser.set_defaults(func=serve_command_factory)
def __init__(self, pipeline: Pipeline, host: str, port: int, workers: int):
self._pipeline = pipeline
self.host = host
self.port = port
self.workers = workers
if not _serve_dependencies_installed:
raise RuntimeError(
"Using serve command requires FastAPI and unicorn. "
'Please install transformers with [serving]: pip install "transformers[serving]".'
"Or install FastAPI and unicorn separately."
)
else:
logger.info("Serving model over {}:{}".format(host, port))
self._app = FastAPI(
routes=[
APIRoute(
"/",
self.model_info,
response_model=ServeModelInfoResult,
response_class=JSONResponse,
methods=["GET"],
),
APIRoute(
"/tokenize",
self.tokenize,
response_model=ServeTokenizeResult,
response_class=JSONResponse,
methods=["POST"],
),
APIRoute(
"/detokenize",
self.detokenize,
response_model=ServeDeTokenizeResult,
response_class=JSONResponse,
methods=["POST"],
),
APIRoute(
"/forward",
self.forward,
response_model=ServeForwardResult,
response_class=JSONResponse,
methods=["POST"],
),
],
timeout=600,
)
def run(self):
run(self._app, host=self.host, port=self.port, workers=self.workers)
def model_info(self):
return ServeModelInfoResult(infos=vars(self._pipeline.model.config))
def tokenize(self, text_input: str = Body(None, embed=True), return_ids: bool = Body(False, embed=True)):
"""
Tokenize the provided input and eventually returns corresponding tokens id:
- **text_input**: String to tokenize
- **return_ids**: Boolean flags indicating if the tokens have to be converted to their integer mapping.
"""
try:
tokens_txt = self._pipeline.tokenizer.tokenize(text_input)
if return_ids:
tokens_ids = self._pipeline.tokenizer.convert_tokens_to_ids(tokens_txt)
return ServeTokenizeResult(tokens=tokens_txt, tokens_ids=tokens_ids)
else:
return ServeTokenizeResult(tokens=tokens_txt)
except Exception as e:
raise HTTPException(status_code=500, detail={"model": "", "error": str(e)})
def detokenize(
self,
tokens_ids: List[int] = Body(None, embed=True),
skip_special_tokens: bool = Body(False, embed=True),
cleanup_tokenization_spaces: bool = Body(True, embed=True),
):
"""
Detokenize the provided tokens ids to readable text:
- **tokens_ids**: List of tokens ids
- **skip_special_tokens**: Flag indicating to not try to decode special tokens
- **cleanup_tokenization_spaces**: Flag indicating to remove all leading/trailing spaces and intermediate ones.
"""
try:
decoded_str = self._pipeline.tokenizer.decode(tokens_ids, skip_special_tokens, cleanup_tokenization_spaces)
return ServeDeTokenizeResult(model="", text=decoded_str)
except Exception as e:
raise HTTPException(status_code=500, detail={"model": "", "error": str(e)})
async def forward(self, inputs=Body(None, embed=True)):
"""
**inputs**:
**attention_mask**:
**tokens_type_ids**:
"""
# Check we don't have empty string
if len(inputs) == 0:
return ServeForwardResult(output=[], attention=[])
try:
# Forward through the model
output = self._pipeline(inputs)
return ServeForwardResult(output=output)
except Exception as e:
raise HTTPException(500, {"error": str(e)})
xtune/src/transformers/commands/train.py
+144
View File
@@ -0,0 +1,144 @@
import os
from argparse import ArgumentParser, Namespace
from logging import getLogger
from transformers import SingleSentenceClassificationProcessor as Processor
from transformers import TextClassificationPipeline, is_tf_available, is_torch_available
from transformers.commands import BaseTransformersCLICommand
if not is_tf_available() and not is_torch_available():
raise RuntimeError("At least one of PyTorch or TensorFlow 2.0+ should be installed to use CLI training")
# TF training parameters
USE_XLA = False
USE_AMP = False
def train_command_factory(args: Namespace):
"""
Factory function used to instantiate serving server from provided command line arguments.
:return: ServeCommand
"""
return TrainCommand(args)
class TrainCommand(BaseTransformersCLICommand):
@staticmethod
def register_subcommand(parser: ArgumentParser):
"""
Register this command to argparse so it's available for the transformer-cli
:param parser: Root parser to register command-specific arguments
:return:
"""
train_parser = parser.add_parser("train", help="CLI tool to train a model on a task.")
train_parser.add_argument(
"--train_data",
type=str,
required=True,
help="path to train (and optionally evaluation) dataset as a csv with "
"tab separated labels and sentences.",
)
train_parser.add_argument(
"--column_label", type=int, default=0, help="Column of the dataset csv file with example labels."
)
train_parser.add_argument(
"--column_text", type=int, default=1, help="Column of the dataset csv file with example texts."
)
train_parser.add_argument(
"--column_id", type=int, default=2, help="Column of the dataset csv file with example ids."
)
train_parser.add_argument(
"--skip_first_row", action="store_true", help="Skip the first row of the csv file (headers)."
)
train_parser.add_argument("--validation_data", type=str, default="", help="path to validation dataset.")
train_parser.add_argument(
"--validation_split",
type=float,
default=0.1,
help="if validation dataset is not provided, fraction of train dataset " "to use as validation dataset.",
)
train_parser.add_argument("--output", type=str, default="./", help="path to saved the trained model.")
train_parser.add_argument(
"--task", type=str, default="text_classification", help="Task to train the model on."
)
train_parser.add_argument(
"--model", type=str, default="bert-base-uncased", help="Model's name or path to stored model."
)
train_parser.add_argument("--train_batch_size", type=int, default=32, help="Batch size for training.")
train_parser.add_argument("--valid_batch_size", type=int, default=64, help="Batch size for validation.")
train_parser.add_argument("--learning_rate", type=float, default=3e-5, help="Learning rate.")
train_parser.add_argument("--adam_epsilon", type=float, default=1e-08, help="Epsilon for Adam optimizer.")
train_parser.set_defaults(func=train_command_factory)
def __init__(self, args: Namespace):
self.logger = getLogger("transformers-cli/training")
self.framework = "tf" if is_tf_available() else "torch"
os.makedirs(args.output, exist_ok=True)
assert os.path.isdir(args.output)
self.output = args.output
self.column_label = args.column_label
self.column_text = args.column_text
self.column_id = args.column_id
self.logger.info("Loading {} pipeline for {}".format(args.task, args.model))
if args.task == "text_classification":
self.pipeline = TextClassificationPipeline.from_pretrained(args.model)
elif args.task == "token_classification":
raise NotImplementedError
elif args.task == "question_answering":
raise NotImplementedError
self.logger.info("Loading dataset from {}".format(args.train_data))
self.train_dataset = Processor.create_from_csv(
args.train_data,
column_label=args.column_label,
column_text=args.column_text,
column_id=args.column_id,
skip_first_row=args.skip_first_row,
)
self.valid_dataset = None
if args.validation_data:
self.logger.info("Loading validation dataset from {}".format(args.validation_data))
self.valid_dataset = Processor.create_from_csv(
args.validation_data,
column_label=args.column_label,
column_text=args.column_text,
column_id=args.column_id,
skip_first_row=args.skip_first_row,
)
self.validation_split = args.validation_split
self.train_batch_size = args.train_batch_size
self.valid_batch_size = args.valid_batch_size
self.learning_rate = args.learning_rate
self.adam_epsilon = args.adam_epsilon
def run(self):
if self.framework == "tf":
return self.run_tf()
return self.run_torch()
def run_torch(self):
raise NotImplementedError
def run_tf(self):
self.pipeline.fit(
self.train_dataset,
validation_data=self.valid_dataset,
validation_split=self.validation_split,
learning_rate=self.learning_rate,
adam_epsilon=self.adam_epsilon,
train_batch_size=self.train_batch_size,
valid_batch_size=self.valid_batch_size,
)
# Save trained pipeline
self.pipeline.save_pretrained(self.output)
xtune/src/transformers/commands/user.py
+209
View File
@@ -0,0 +1,209 @@
import os
import sys
from argparse import ArgumentParser
from getpass import getpass
from typing import List, Union
from requests.exceptions import HTTPError
from transformers.commands import BaseTransformersCLICommand
from transformers.hf_api import HfApi, HfFolder
UPLOAD_MAX_FILES = 15
class UserCommands(BaseTransformersCLICommand):
@staticmethod
def register_subcommand(parser: ArgumentParser):
login_parser = parser.add_parser("login", help="Log in using the same credentials as on huggingface.co")
login_parser.set_defaults(func=lambda args: LoginCommand(args))
whoami_parser = parser.add_parser("whoami", help="Find out which huggingface.co account you are logged in as.")
whoami_parser.set_defaults(func=lambda args: WhoamiCommand(args))
logout_parser = parser.add_parser("logout", help="Log out")
logout_parser.set_defaults(func=lambda args: LogoutCommand(args))
# s3
s3_parser = parser.add_parser("s3", help="{ls, rm} Commands to interact with the files you upload on S3.")
s3_subparsers = s3_parser.add_subparsers(help="s3 related commands")
ls_parser = s3_subparsers.add_parser("ls")
ls_parser.set_defaults(func=lambda args: ListObjsCommand(args))
rm_parser = s3_subparsers.add_parser("rm")
rm_parser.add_argument("filename", type=str, help="individual object filename to delete from S3.")
rm_parser.set_defaults(func=lambda args: DeleteObjCommand(args))
# upload
upload_parser = parser.add_parser("upload")
upload_parser.add_argument("path", type=str, help="Local path of the folder or individual file to upload.")
upload_parser.add_argument(
"--filename", type=str, default=None, help="Optional: override individual object filename on S3."
)
upload_parser.set_defaults(func=lambda args: UploadCommand(args))
class ANSI:
"""
Helper for en.wikipedia.org/wiki/ANSI_escape_code
"""
_bold = "\u001b[1m"
_reset = "\u001b[0m"
@classmethod
def bold(cls, s):
return "{}{}{}".format(cls._bold, s, cls._reset)
class BaseUserCommand:
def __init__(self, args):
self.args = args
self._api = HfApi()
class LoginCommand(BaseUserCommand):
def run(self):
print(
"""
_| _| _| _| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _|_|_|_| _|_| _|_|_| _|_|_|_|
_| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
_|_|_|_| _| _| _| _|_| _| _|_| _| _| _| _| _| _|_| _|_|_| _|_|_|_| _| _|_|_|
_| _| _| _| _| _| _| _| _| _| _|_| _| _| _| _| _| _| _|
_| _| _|_| _|_|_| _|_|_| _|_|_| _| _| _|_|_| _| _| _| _|_|_| _|_|_|_|
"""
)
username = input("Username: ")
password = getpass()
try:
token = self._api.login(username, password)
except HTTPError as e:
# probably invalid credentials, display error message.
print(e)
exit(1)
HfFolder.save_token(token)
print("Login successful")
print("Your token:", token, "\n")
print("Your token has been saved to", HfFolder.path_token)
class WhoamiCommand(BaseUserCommand):
def run(self):
token = HfFolder.get_token()
if token is None:
print("Not logged in")
exit()
try:
user = self._api.whoami(token)
print(user)
except HTTPError as e:
print(e)
class LogoutCommand(BaseUserCommand):
def run(self):
token = HfFolder.get_token()
if token is None:
print("Not logged in")
exit()
HfFolder.delete_token()
self._api.logout(token)
print("Successfully logged out.")
class ListObjsCommand(BaseUserCommand):
def tabulate(self, rows: List[List[Union[str, int]]], headers: List[str]) -> str:
"""
Inspired by:
stackoverflow.com/a/8356620/593036
stackoverflow.com/questions/9535954/printing-lists-as-tabular-data
"""
col_widths = [max(len(str(x)) for x in col) for col in zip(*rows, headers)]
row_format = ("{{:{}}} " * len(headers)).format(*col_widths)
lines = []
lines.append(row_format.format(*headers))
lines.append(row_format.format(*["-" * w for w in col_widths]))
for row in rows:
lines.append(row_format.format(*row))
return "\n".join(lines)
def run(self):
token = HfFolder.get_token()
if token is None:
print("Not logged in")
exit(1)
try:
objs = self._api.list_objs(token)
except HTTPError as e:
print(e)
exit(1)
if len(objs) == 0:
print("No shared file yet")
exit()
rows = [[obj.filename, obj.LastModified, obj.ETag, obj.Size] for obj in objs]
print(self.tabulate(rows, headers=["Filename", "LastModified", "ETag", "Size"]))
class DeleteObjCommand(BaseUserCommand):
def run(self):
token = HfFolder.get_token()
if token is None:
print("Not logged in")
exit(1)
try:
self._api.delete_obj(token, filename=self.args.filename)
except HTTPError as e:
print(e)
exit(1)
print("Done")
class UploadCommand(BaseUserCommand):
def walk_dir(self, rel_path):
"""
Recursively list all files in a folder.
"""
entries: List[os.DirEntry] = list(os.scandir(rel_path))
files = [(os.path.join(os.getcwd(), f.path), f.path) for f in entries if f.is_file()] # (filepath, filename)
for f in entries:
if f.is_dir():
files += self.walk_dir(f.path)
return files
def run(self):
token = HfFolder.get_token()
if token is None:
print("Not logged in")
exit(1)
local_path = os.path.abspath(self.args.path)
if os.path.isdir(local_path):
if self.args.filename is not None:
raise ValueError("Cannot specify a filename override when uploading a folder.")
rel_path = os.path.basename(local_path)
files = self.walk_dir(rel_path)
elif os.path.isfile(local_path):
filename = self.args.filename if self.args.filename is not None else os.path.basename(local_path)
files = [(local_path, filename)]
else:
raise ValueError("Not a valid file or directory: {}".format(local_path))
if sys.platform == "win32":
files = [(filepath, filename.replace(os.sep, "/")) for filepath, filename in files]
if len(files) > UPLOAD_MAX_FILES:
print(
"About to upload {} files to S3. This is probably wrong. Please filter files before uploading.".format(
ANSI.bold(len(files))
)
)
exit(1)
for filepath, filename in files:
print("About to upload file {} to S3 under filename {}".format(ANSI.bold(filepath), ANSI.bold(filename)))
choice = input("Proceed? [Y/n] ").lower()
if not (choice == "" or choice == "y" or choice == "yes"):
print("Abort")
exit()
print(ANSI.bold("Uploading... This might take a while if files are large"))
for filepath, filename in files:
access_url = self._api.presign_and_upload(token=token, filename=filename, filepath=filepath)
print("Your file now lives at:")
print(access_url)
xtune/src/transformers/configuration_albert.py
+146
View File
@@ -0,0 +1,146 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" ALBERT model configuration """
from .configuration_utils import PretrainedConfig
ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"albert-base-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-base-config.json",
"albert-large-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-large-config.json",
"albert-xlarge-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-xlarge-config.json",
"albert-xxlarge-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-xxlarge-config.json",
"albert-base-v2": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-base-v2-config.json",
"albert-large-v2": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-large-v2-config.json",
"albert-xlarge-v2": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-xlarge-v2-config.json",
"albert-xxlarge-v2": "https://s3.amazonaws.com/models.huggingface.co/bert/albert-xxlarge-v2-config.json",
}
class AlbertConfig(PretrainedConfig):
r"""
This is the configuration class to store the configuration of an :class:`~transformers.AlbertModel`.
It is used to instantiate an ALBERT model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the ALBERT `xxlarge <https://huggingface.co/albert-xxlarge-v2>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 30000):
Vocabulary size of the ALBERT model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.AlbertModel`.
embedding_size (:obj:`int`, optional, defaults to 128):
Dimensionality of vocabulary embeddings.
hidden_size (:obj:`int`, optional, defaults to 4096):
Dimensionality of the encoder layers and the pooler layer.
num_hidden_layers (:obj:`int`, optional, defaults to 12):
Number of hidden layers in the Transformer encoder.
num_hidden_groups (:obj:`int`, optional, defaults to 1):
Number of groups for the hidden layers, parameters in the same group are shared.
num_attention_heads (:obj:`int`, optional, defaults to 64):
Number of attention heads for each attention layer in the Transformer encoder.
intermediate_size (:obj:`int`, optional, defaults to 16384):
The dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
inner_group_num (:obj:`int`, optional, defaults to 1):
The number of inner repetition of attention and ffn.
hidden_act (:obj:`str` or :obj:`function`, optional, defaults to "gelu_new"):
The non-linear activation function (function or string) in the encoder and pooler.
If string, "gelu", "relu", "swish" and "gelu_new" are supported.
hidden_dropout_prob (:obj:`float`, optional, defaults to 0):
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
attention_probs_dropout_prob (:obj:`float`, optional, defaults to 0):
The dropout ratio for the attention probabilities.
max_position_embeddings (:obj:`int`, optional, defaults to 512):
The maximum sequence length that this model might ever be used with. Typically set this to something
large (e.g., 512 or 1024 or 2048).
type_vocab_size (:obj:`int`, optional, defaults to 2):
The vocabulary size of the `token_type_ids` passed into :class:`~transformers.AlbertModel`.
initializer_range (:obj:`float`, optional, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
The epsilon used by the layer normalization layers.
classifier_dropout_prob (:obj:`float`, optional, defaults to 0.1):
The dropout ratio for attached classifiers.
Example::
from transformers import AlbertConfig, AlbertModel
# Initializing an ALBERT-xxlarge style configuration
albert_xxlarge_configuration = AlbertConfig()
# Initializing an ALBERT-base style configuration
albert_base_configuration = AlbertConfig(
hidden_size=768,
num_attention_heads=12,
intermediate_size=3072,
)
# Initializing a model from the ALBERT-base style configuration
model = AlbertModel(albert_xxlarge_configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "albert"
def __init__(
self,
vocab_size=30000,
embedding_size=128,
hidden_size=4096,
num_hidden_layers=12,
num_hidden_groups=1,
num_attention_heads=64,
intermediate_size=16384,
inner_group_num=1,
hidden_act="gelu_new",
hidden_dropout_prob=0,
attention_probs_dropout_prob=0,
max_position_embeddings=512,
type_vocab_size=2,
initializer_range=0.02,
layer_norm_eps=1e-12,
classifier_dropout_prob=0.1,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.embedding_size = embedding_size
self.hidden_size = hidden_size
self.num_hidden_layers = num_hidden_layers
self.num_hidden_groups = num_hidden_groups
self.num_attention_heads = num_attention_heads
self.inner_group_num = inner_group_num
self.hidden_act = hidden_act
self.intermediate_size = intermediate_size
self.hidden_dropout_prob = hidden_dropout_prob
self.attention_probs_dropout_prob = attention_probs_dropout_prob
self.max_position_embeddings = max_position_embeddings
self.type_vocab_size = type_vocab_size
self.initializer_range = initializer_range
self.layer_norm_eps = layer_norm_eps
self.classifier_dropout_prob = classifier_dropout_prob
xtune/src/transformers/configuration_auto.py
+199
View File
@@ -0,0 +1,199 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Auto Config class. """
import logging
from collections import OrderedDict
from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
from .configuration_bart import BART_PRETRAINED_CONFIG_ARCHIVE_MAP, BartConfig
from .configuration_bert import BERT_PRETRAINED_CONFIG_ARCHIVE_MAP, BertConfig
from .configuration_camembert import CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, CamembertConfig
from .configuration_ctrl import CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP, CTRLConfig
from .configuration_distilbert import DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, DistilBertConfig
from .configuration_flaubert import FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, FlaubertConfig
from .configuration_gpt2 import GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP, GPT2Config
from .configuration_openai import OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP, OpenAIGPTConfig
from .configuration_roberta import ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, RobertaConfig
from .configuration_t5 import T5_PRETRAINED_CONFIG_ARCHIVE_MAP, T5Config
from .configuration_transfo_xl import TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP, TransfoXLConfig
from .configuration_utils import PretrainedConfig
from .configuration_xlm import XLM_PRETRAINED_CONFIG_ARCHIVE_MAP, XLMConfig
from .configuration_xlm_roberta import XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, XLMRobertaConfig
from .configuration_xlnet import XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP, XLNetConfig
logger = logging.getLogger(__name__)
ALL_PRETRAINED_CONFIG_ARCHIVE_MAP = dict(
(key, value)
for pretrained_map in [
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
BART_PRETRAINED_CONFIG_ARCHIVE_MAP,
OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP,
TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP,
GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP,
CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP,
XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
XLM_PRETRAINED_CONFIG_ARCHIVE_MAP,
ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
]
for key, value, in pretrained_map.items()
)
CONFIG_MAPPING = OrderedDict(
[
("t5", T5Config,),
("distilbert", DistilBertConfig,),
("albert", AlbertConfig,),
("camembert", CamembertConfig,),
("xlm-roberta", XLMRobertaConfig,),
("bart", BartConfig,),
("roberta", RobertaConfig,),
("flaubert", FlaubertConfig,),
("bert", BertConfig,),
("openai-gpt", OpenAIGPTConfig,),
("gpt2", GPT2Config,),
("transfo-xl", TransfoXLConfig,),
("xlnet", XLNetConfig,),
("xlm", XLMConfig,),
("ctrl", CTRLConfig,),
]
)
class AutoConfig:
r"""
:class:`~transformers.AutoConfig` is a generic configuration class
that will be instantiated as one of the configuration classes of the library
when created with the :func:`~transformers.AutoConfig.from_pretrained` class method.
The :func:`~transformers.AutoConfig.from_pretrained` method takes care of returning the correct model class instance
based on the `model_type` property of the config object, or when it's missing,
falling back to using pattern matching on the `pretrained_model_name_or_path` string.
"""
def __init__(self):
raise EnvironmentError(
"AutoConfig is designed to be instantiated "
"using the `AutoConfig.from_pretrained(pretrained_model_name_or_path)` method."
)
@classmethod
def for_model(cls, model_type, *args, **kwargs):
for pattern, config_class in CONFIG_MAPPING.items():
if pattern in model_type:
return config_class(*args, **kwargs)
raise ValueError(
"Unrecognized model identifier in {}. Should contain one of {}".format(
model_type, ", ".join(CONFIG_MAPPING.keys())
)
)
@classmethod
def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
r""" Instantiates one of the configuration classes of the library
from a pre-trained model configuration.
The configuration class to instantiate is selected
based on the `model_type` property of the config object, or when it's missing,
falling back to using pattern matching on the `pretrained_model_name_or_path` string.
- contains `t5`: :class:`~transformers.T5Config` (T5 model)
- contains `distilbert`: :class:`~transformers.DistilBertConfig` (DistilBERT model)
- contains `albert`: :class:`~transformers.AlbertConfig` (ALBERT model)
- contains `camembert`: :class:`~transformers.CamembertConfig` (CamemBERT model)
- contains `xlm-roberta`: :class:`~transformers.XLMRobertaConfig` (XLM-RoBERTa model)
- contains `roberta`: :class:`~transformers.RobertaConfig` (RoBERTa model)
- contains `bert`: :class:`~transformers.BertConfig` (Bert model)
- contains `openai-gpt`: :class:`~transformers.OpenAIGPTConfig` (OpenAI GPT model)
- contains `gpt2`: :class:`~transformers.GPT2Config` (OpenAI GPT-2 model)
- contains `transfo-xl`: :class:`~transformers.TransfoXLConfig` (Transformer-XL model)
- contains `xlnet`: :class:`~transformers.XLNetConfig` (XLNet model)
- contains `xlm`: :class:`~transformers.XLMConfig` (XLM model)
- contains `ctrl` : :class:`~transformers.CTRLConfig` (CTRL model)
- contains `flaubert` : :class:`~transformers.FlaubertConfig` (Flaubert model)
Args:
pretrained_model_name_or_path (:obj:`string`):
Is either: \
- a string with the `shortcut name` of a pre-trained model configuration to load from cache or download, e.g.: ``bert-base-uncased``.
- a string with the `identifier name` of a pre-trained model configuration that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
- a path to a `directory` containing a configuration file saved using the :func:`~transformers.PretrainedConfig.save_pretrained` method, e.g.: ``./my_model_directory/``.
- a path or url to a saved configuration JSON `file`, e.g.: ``./my_model_directory/configuration.json``.
cache_dir (:obj:`string`, optional, defaults to `None`):
Path to a directory in which a downloaded pre-trained model
configuration should be cached if the standard cache should not be used.
force_download (:obj:`boolean`, optional, defaults to `False`):
Force to (re-)download the model weights and configuration files and override the cached versions if they exist.
resume_download (:obj:`boolean`, optional, defaults to `False`):
Do not delete incompletely received file. Attempt to resume the download if such a file exists.
proxies (:obj:`Dict[str, str]`, optional, defaults to `None`):
A dictionary of proxy servers to use by protocol or endpoint, e.g.: :obj:`{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`.
The proxies are used on each request. See `the requests documentation <https://requests.readthedocs.io/en/master/user/advanced/#proxies>`__ for usage.
return_unused_kwargs (:obj:`boolean`, optional, defaults to `False`):
- If False, then this function returns just the final configuration object.
- If True, then this functions returns a tuple `(config, unused_kwargs)` where `unused_kwargs` is a dictionary consisting of the key/value pairs whose keys are not configuration attributes: ie the part of kwargs which has not been used to update `config` and is otherwise ignored.
kwargs (:obj:`Dict[str, any]`, optional, defaults to `{}`): key/value pairs with which to update the configuration object after loading.
- The values in kwargs of any keys which are configuration attributes will be used to override the loaded values.
- Behavior concerning key/value pairs whose keys are *not* configuration attributes is controlled by the `return_unused_kwargs` keyword parameter.
Examples::
config = AutoConfig.from_pretrained('bert-base-uncased') # Download configuration from S3 and cache.
config = AutoConfig.from_pretrained('./test/bert_saved_model/') # E.g. config (or model) was saved using `save_pretrained('./test/saved_model/')`
config = AutoConfig.from_pretrained('./test/bert_saved_model/my_configuration.json')
config = AutoConfig.from_pretrained('bert-base-uncased', output_attention=True, foo=False)
assert config.output_attention == True
config, unused_kwargs = AutoConfig.from_pretrained('bert-base-uncased', output_attention=True,
foo=False, return_unused_kwargs=True)
assert config.output_attention == True
assert unused_kwargs == {'foo': False}
"""
config_dict, _ = PretrainedConfig.get_config_dict(
pretrained_model_name_or_path, pretrained_config_archive_map=ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, **kwargs
)
if "model_type" in config_dict:
config_class = CONFIG_MAPPING[config_dict["model_type"]]
return config_class.from_dict(config_dict, **kwargs)
else:
# Fallback: use pattern matching on the string.
for pattern, config_class in CONFIG_MAPPING.items():
if pattern in pretrained_model_name_or_path:
return config_class.from_dict(config_dict, **kwargs)
raise ValueError(
"Unrecognized model in {}. "
"Should have a `model_type` key in its config.json, or contain one of the following strings "
"in its name: {}".format(pretrained_model_name_or_path, ", ".join(CONFIG_MAPPING.keys()))
)
xtune/src/transformers/configuration_bart.py
+105
View File
@@ -0,0 +1,105 @@
# coding=utf-8
# Copyright 2020 The Fairseq Authors and The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" BART configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
BART_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"bart-large": "https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large/config.json",
"bart-large-mnli": "https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-mnli/config.json",
"bart-large-cnn": "https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/config.json",
}
class BartConfig(PretrainedConfig):
r"""
Configuration class for Bart. Parameters are renamed from the fairseq implementation
"""
model_type = "bart"
pretrained_config_archive_map = BART_PRETRAINED_CONFIG_ARCHIVE_MAP
def __init__(
self,
activation_dropout=0.0,
vocab_size=50265,
pad_token_id=1,
eos_token_id=2,
d_model=1024,
encoder_ffn_dim=4096,
encoder_layers=12,
encoder_attention_heads=16,
decoder_ffn_dim=4096,
decoder_layers=12,
decoder_attention_heads=16,
encoder_layerdrop=0.0,
decoder_layerdrop=0.0,
attention_dropout=0.0,
dropout=0.1,
max_position_embeddings=1024,
init_std=0.02,
classifier_dropout=0.0,
output_past=False,
num_labels=3,
bos_token_id=0,
**common_kwargs
):
r"""
:class:`~transformers.BartConfig` is the configuration class for `BartModel`.
Examples:
config = BartConfig.from_pretrained('bart-large')
model = BartModel(config)
"""
super().__init__(
num_labels=num_labels,
output_past=output_past,
pad_token_id=pad_token_id,
bos_token_id=bos_token_id,
**common_kwargs,
)
self.vocab_size = vocab_size
self.d_model = d_model # encoder_embed_dim and decoder_embed_dim
self.eos_token_id = eos_token_id
self.encoder_ffn_dim = encoder_ffn_dim
self.encoder_layers = self.num_hidden_layers = encoder_layers
self.encoder_attention_heads = encoder_attention_heads
self.encoder_layerdrop = encoder_layerdrop
self.decoder_layerdrop = decoder_layerdrop
self.decoder_ffn_dim = decoder_ffn_dim
self.decoder_layers = decoder_layers
self.decoder_attention_heads = decoder_attention_heads
self.max_position_embeddings = max_position_embeddings
self.init_std = init_std # Normal(0, this parameter)
# 3 Types of Dropout
self.attention_dropout = attention_dropout
self.activation_dropout = activation_dropout
self.dropout = dropout
# Classifier stuff
self.classif_dropout = classifier_dropout
@property
def num_attention_heads(self):
return self.encoder_attention_heads
@property
def hidden_size(self):
return self.d_model
xtune/src/transformers/configuration_bert.py
+142
View File
@@ -0,0 +1,142 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" BERT model configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"bert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json",
"bert-large-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-config.json",
"bert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json",
"bert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-config.json",
"bert-base-multilingual-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json",
"bert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json",
"bert-base-chinese": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-config.json",
"bert-base-german-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-cased-config.json",
"bert-large-uncased-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-config.json",
"bert-large-cased-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-config.json",
"bert-large-uncased-whole-word-masking-finetuned-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json",
"bert-large-cased-whole-word-masking-finetuned-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-whole-word-masking-finetuned-squad-config.json",
"bert-base-cased-finetuned-mrpc": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-mrpc-config.json",
"bert-base-german-dbmdz-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-cased-config.json",
"bert-base-german-dbmdz-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-german-dbmdz-uncased-config.json",
"bert-base-japanese": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-config.json",
"bert-base-japanese-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-whole-word-masking-config.json",
"bert-base-japanese-char": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-config.json",
"bert-base-japanese-char-whole-word-masking": "https://s3.amazonaws.com/models.huggingface.co/bert/cl-tohoku/bert-base-japanese-char-whole-word-masking-config.json",
"bert-base-finnish-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-cased-v1/config.json",
"bert-base-finnish-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/TurkuNLP/bert-base-finnish-uncased-v1/config.json",
"bert-base-dutch-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/wietsedv/bert-base-dutch-cased/config.json",
}
class BertConfig(PretrainedConfig):
r"""
This is the configuration class to store the configuration of a :class:`~transformers.BertModel`.
It is used to instantiate an BERT model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the BERT `bert-base-uncased <https://huggingface.co/bert-base-uncased>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 30522):
Vocabulary size of the BERT model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.BertModel`.
hidden_size (:obj:`int`, optional, defaults to 768):
Dimensionality of the encoder layers and the pooler layer.
num_hidden_layers (:obj:`int`, optional, defaults to 12):
Number of hidden layers in the Transformer encoder.
num_attention_heads (:obj:`int`, optional, defaults to 12):
Number of attention heads for each attention layer in the Transformer encoder.
intermediate_size (:obj:`int`, optional, defaults to 3072):
Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
hidden_act (:obj:`str` or :obj:`function`, optional, defaults to "gelu"):
The non-linear activation function (function or string) in the encoder and pooler.
If string, "gelu", "relu", "swish" and "gelu_new" are supported.
hidden_dropout_prob (:obj:`float`, optional, defaults to 0.1):
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
attention_probs_dropout_prob (:obj:`float`, optional, defaults to 0.1):
The dropout ratio for the attention probabilities.
max_position_embeddings (:obj:`int`, optional, defaults to 512):
The maximum sequence length that this model might ever be used with.
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
type_vocab_size (:obj:`int`, optional, defaults to 2):
The vocabulary size of the `token_type_ids` passed into :class:`~transformers.BertModel`.
initializer_range (:obj:`float`, optional, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
The epsilon used by the layer normalization layers.
Example::
from transformers import BertModel, BertConfig
# Initializing a BERT bert-base-uncased style configuration
configuration = BertConfig()
# Initializing a model from the bert-base-uncased style configuration
model = BertModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = BERT_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "bert"
def __init__(
self,
vocab_size=30522,
hidden_size=768,
num_hidden_layers=12,
num_attention_heads=12,
intermediate_size=3072,
hidden_act="gelu",
hidden_dropout_prob=0.1,
attention_probs_dropout_prob=0.1,
max_position_embeddings=512,
type_vocab_size=2,
initializer_range=0.02,
layer_norm_eps=1e-12,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.hidden_size = hidden_size
self.num_hidden_layers = num_hidden_layers
self.num_attention_heads = num_attention_heads
self.hidden_act = hidden_act
self.intermediate_size = intermediate_size
self.hidden_dropout_prob = hidden_dropout_prob
self.attention_probs_dropout_prob = attention_probs_dropout_prob
self.max_position_embeddings = max_position_embeddings
self.type_vocab_size = type_vocab_size
self.initializer_range = initializer_range
self.layer_norm_eps = layer_norm_eps
xtune/src/transformers/configuration_camembert.py
+40
View File
@@ -0,0 +1,40 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" CamemBERT configuration """
import logging
from .configuration_roberta import RobertaConfig
logger = logging.getLogger(__name__)
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"camembert-base": "https://s3.amazonaws.com/models.huggingface.co/bert/camembert-base-config.json",
"umberto-commoncrawl-cased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-commoncrawl-cased-v1/config.json",
"umberto-wikipedia-uncased-v1": "https://s3.amazonaws.com/models.huggingface.co/bert/Musixmatch/umberto-wikipedia-uncased-v1/config.json",
}
class CamembertConfig(RobertaConfig):
"""
This class overrides :class:`~transformers.RobertaConfig`. Please check the
superclass for the appropriate documentation alongside usage examples.
"""
pretrained_config_archive_map = CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "camembert"
xtune/src/transformers/configuration_ctrl.py
+143
View File
@@ -0,0 +1,143 @@
# coding=utf-8
# Copyright 2018 Salesforce and HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Salesforce CTRL configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP = {"ctrl": "https://storage.googleapis.com/sf-ctrl/pytorch/ctrl-config.json"}
class CTRLConfig(PretrainedConfig):
"""
This is the configuration class to store the configuration of an :class:`~transformers.CTRLModel`.
It is used to instantiate an CTRL model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the `ctrl <https://huggingface.co/ctrl>`__ architecture from SalesForce.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 246534):
Vocabulary size of the CTRL model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.CTRLModel`.
n_positions (:obj:`int`, optional, defaults to 256):
The maximum sequence length that this model might ever be used with.
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
n_ctx (:obj:`int`, optional, defaults to 256):
Dimensionality of the causal mask (usually same as n_positions).
n_embd (:obj:`int`, optional, defaults to 1280):
Dimensionality of the embeddings and hidden states.
dff (:obj:`int`, optional, defaults to 8192):
Dimensionality of the inner dimension of the FFN.
n_layer (:obj:`int`, optional, defaults to 48):
Number of hidden layers in the Transformer encoder.
n_head (:obj:`int`, optional, defaults to 16):
Number of attention heads for each attention layer in the Transformer encoder.
resid_pdrop (:obj:`float`, optional, defaults to 0.1):
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
embd_pdrop (:obj:`int`, optional, defaults to 0.1):
The dropout ratio for the embeddings.
attn_pdrop (:obj:`float`, optional, defaults to 0.1):
The dropout ratio for the attention.
layer_norm_epsilon (:obj:`float`, optional, defaults to 1e-6):
The epsilon to use in the layer normalization layers
initializer_range (:obj:`float`, optional, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
Example::
from transformers import CTRLModel, CTRLConfig
# Initializing a CTRL configuration
configuration = CTRLConfig()
# Initializing a model from the configuration
model = CTRLModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "ctrl"
def __init__(
self,
vocab_size=246534,
n_positions=256,
n_ctx=256,
n_embd=1280,
dff=8192,
n_layer=48,
n_head=16,
resid_pdrop=0.1,
embd_pdrop=0.1,
attn_pdrop=0.1,
layer_norm_epsilon=1e-6,
initializer_range=0.02,
summary_type="cls_index",
summary_use_proj=True,
summary_activation=None,
summary_proj_to_labels=True,
summary_first_dropout=0.1,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.n_ctx = n_ctx
self.n_positions = n_positions
self.n_embd = n_embd
self.n_layer = n_layer
self.n_head = n_head
self.dff = dff
self.resid_pdrop = resid_pdrop
self.embd_pdrop = embd_pdrop
self.attn_pdrop = attn_pdrop
self.layer_norm_epsilon = layer_norm_epsilon
self.initializer_range = initializer_range
self.summary_type = summary_type
self.summary_use_proj = summary_use_proj
self.summary_activation = summary_activation
self.summary_first_dropout = summary_first_dropout
self.summary_proj_to_labels = summary_proj_to_labels
@property
def max_position_embeddings(self):
return self.n_positions
@property
def hidden_size(self):
return self.n_embd
@property
def num_attention_heads(self):
return self.n_head
@property
def num_hidden_layers(self):
return self.n_layer
xtune/src/transformers/configuration_distilbert.py
+143
View File
@@ -0,0 +1,143 @@
# coding=utf-8
# Copyright 2019-present, the HuggingFace Inc. team, The Google AI Language Team and Facebook, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" DistilBERT model configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"distilbert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-config.json",
"distilbert-base-uncased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-distilled-squad-config.json",
"distilbert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-config.json",
"distilbert-base-cased-distilled-squad": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-distilled-squad-config.json",
"distilbert-base-german-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-german-cased-config.json",
"distilbert-base-multilingual-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json",
"distilbert-base-uncased-finetuned-sst-2-english": "https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-uncased-finetuned-sst-2-english-config.json",
}
class DistilBertConfig(PretrainedConfig):
r"""
This is the configuration class to store the configuration of a :class:`~transformers.DistilBertModel`.
It is used to instantiate a DistilBERT model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the DistilBERT `distilbert-base-uncased <https://huggingface.co/distilbert-base-uncased>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 30522):
Vocabulary size of the DistilBERT model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.BertModel`.
max_position_embeddings (:obj:`int`, optional, defaults to 512):
The maximum sequence length that this model might ever be used with.
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
sinusoidal_pos_embds (:obj:`boolean`, optional, defaults to :obj:`False`):
Whether to use sinusoidal positional embeddings.
n_layers (:obj:`int`, optional, defaults to 6):
Number of hidden layers in the Transformer encoder.
n_heads (:obj:`int`, optional, defaults to 12):
Number of attention heads for each attention layer in the Transformer encoder.
dim (:obj:`int`, optional, defaults to 768):
Dimensionality of the encoder layers and the pooler layer.
hidden_dim (:obj:`int`, optional, defaults to 3072):
The size of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
attention_dropout (:obj:`float`, optional, defaults to 0.1):
The dropout ratio for the attention probabilities.
activation (:obj:`str` or :obj:`function`, optional, defaults to "gelu"):
The non-linear activation function (function or string) in the encoder and pooler.
If string, "gelu", "relu", "swish" and "gelu_new" are supported.
initializer_range (:obj:`float`, optional, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
qa_dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probabilities used in the question answering model
:class:`~tranformers.DistilBertForQuestionAnswering`.
seq_classif_dropout (:obj:`float`, optional, defaults to 0.2):
The dropout probabilities used in the sequence classification model
:class:`~tranformers.DistilBertForSequenceClassification`.
Example::
from transformers import DistilBertModel, DistilBertConfig
# Initializing a DistilBERT configuration
configuration = DistilBertConfig()
# Initializing a model from the configuration
model = DistilBertModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "distilbert"
def __init__(
self,
vocab_size=30522,
max_position_embeddings=512,
sinusoidal_pos_embds=False,
n_layers=6,
n_heads=12,
dim=768,
hidden_dim=4 * 768,
dropout=0.1,
attention_dropout=0.1,
activation="gelu",
initializer_range=0.02,
qa_dropout=0.1,
seq_classif_dropout=0.2,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.max_position_embeddings = max_position_embeddings
self.sinusoidal_pos_embds = sinusoidal_pos_embds
self.n_layers = n_layers
self.n_heads = n_heads
self.dim = dim
self.hidden_dim = hidden_dim
self.dropout = dropout
self.attention_dropout = attention_dropout
self.activation = activation
self.initializer_range = initializer_range
self.qa_dropout = qa_dropout
self.seq_classif_dropout = seq_classif_dropout
@property
def hidden_size(self):
return self.dim
@property
def num_attention_heads(self):
return self.n_heads
@property
def num_hidden_layers(self):
return self.n_layers
xtune/src/transformers/configuration_flaubert.py
+153
View File
@@ -0,0 +1,153 @@
# coding=utf-8
# Copyright 2019-present CNRS, Facebook Inc. and the HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Flaubert configuration, based on XLM. """
import logging
from .configuration_xlm import XLMConfig
logger = logging.getLogger(__name__)
FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"flaubert-small-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_small_cased/config.json",
"flaubert-base-uncased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_uncased/config.json",
"flaubert-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_base_cased/config.json",
"flaubert-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/flaubert/flaubert_large_cased/config.json",
}
class FlaubertConfig(XLMConfig):
"""
Configuration class to store the configuration of a `FlaubertModel`.
This is the configuration class to store the configuration of a :class:`~transformers.XLMModel`.
It is used to instantiate an XLM model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the `xlm-mlm-en-2048 <https://huggingface.co/xlm-mlm-en-2048>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
pre_norm (:obj:`bool`, `optional`, defaults to :obj:`False`):
Whether to apply the layer normalization before or after the feed forward layer following the
attention in each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)
layerdrop (:obj:`float`, `optional`, defaults to 0.0):
Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand
with Structured Dropout. ICLR 2020)
vocab_size (:obj:`int`, optional, defaults to 30145):
Vocabulary size of the Flaubert model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.FlaubertModel`.
emb_dim (:obj:`int`, optional, defaults to 2048):
Dimensionality of the encoder layers and the pooler layer.
n_layer (:obj:`int`, optional, defaults to 12):
Number of hidden layers in the Transformer encoder.
n_head (:obj:`int`, optional, defaults to 16):
Number of attention heads for each attention layer in the Transformer encoder.
dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probability for all fully connected
layers in the embeddings, encoder, and pooler.
attention_dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probability for the attention mechanism
gelu_activation (:obj:`boolean`, optional, defaults to :obj:`True`):
The non-linear activation function (function or string) in the
encoder and pooler. If set to `True`, "gelu" will be used instead of "relu".
sinusoidal_embeddings (:obj:`boolean`, optional, defaults to :obj:`False`):
Whether to use sinusoidal positional embeddings instead of absolute positional embeddings.
causal (:obj:`boolean`, optional, defaults to :obj:`False`):
Set this to `True` for the model to behave in a causal manner.
Causal models use a triangular attention mask in order to only attend to the left-side context instead
if a bidirectional context.
asm (:obj:`boolean`, optional, defaults to :obj:`False`):
Whether to use an adaptive log softmax projection layer instead of a linear layer for the prediction
layer.
n_langs (:obj:`int`, optional, defaults to 1):
The number of languages the model handles. Set to 1 for monolingual models.
use_lang_emb (:obj:`boolean`, optional, defaults to :obj:`True`)
Whether to use language embeddings. Some models use additional language embeddings, see
`the multilingual models page <http://huggingface.co/transformers/multilingual.html#xlm-language-embeddings>`__
for information on how to use them.
max_position_embeddings (:obj:`int`, optional, defaults to 512):
The maximum sequence length that this model might
ever be used with. Typically set this to something large just in case
(e.g., 512 or 1024 or 2048).
embed_init_std (:obj:`float`, optional, defaults to 2048^-0.5):
The standard deviation of the truncated_normal_initializer for
initializing the embedding matrices.
init_std (:obj:`int`, optional, defaults to 50257):
The standard deviation of the truncated_normal_initializer for
initializing all weight matrices except the embedding matrices.
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
The epsilon used by the layer normalization layers.
bos_index (:obj:`int`, optional, defaults to 0):
The index of the beginning of sentence token in the vocabulary.
eos_index (:obj:`int`, optional, defaults to 1):
The index of the end of sentence token in the vocabulary.
pad_index (:obj:`int`, optional, defaults to 2):
The index of the padding token in the vocabulary.
unk_index (:obj:`int`, optional, defaults to 3):
The index of the unknown token in the vocabulary.
mask_index (:obj:`int`, optional, defaults to 5):
The index of the masking token in the vocabulary.
is_encoder(:obj:`boolean`, optional, defaults to :obj:`True`):
Whether the initialized model should be a transformer encoder or decoder as seen in Vaswani et al.
summary_type (:obj:`string`, optional, defaults to "first"):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
Is one of the following options:
- 'last' => take the last token hidden state (like XLNet)
- 'first' => take the first token hidden state (like Bert)
- 'mean' => take the mean of all tokens hidden states
- 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
- 'attn' => Not implemented now, use multi-head attention
summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
Add a projection after the vector extraction
summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
'tanh' => add a tanh activation to the output, Other => no activation.
summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
summary_first_dropout (:obj:`float`, optional, defaults to 0.1):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
Add a dropout before the projection and activation
start_n_top (:obj:`int`, optional, defaults to 5):
Used in the SQuAD evaluation script for XLM and XLNet.
end_n_top (:obj:`int`, optional, defaults to 5):
Used in the SQuAD evaluation script for XLM and XLNet.
mask_token_id (:obj:`int`, optional, defaults to 0):
Model agnostic parameter to identify masked tokens when generating text in an MLM context.
lang_id (:obj:`int`, optional, defaults to 1):
The ID of the language used by the model. This parameter is used when generating
text in a given language.
"""
pretrained_config_archive_map = FLAUBERT_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "flaubert"
def __init__(self, layerdrop=0.0, pre_norm=False, **kwargs):
"""Constructs FlaubertConfig.
"""
super().__init__(**kwargs)
self.layerdrop = layerdrop
self.pre_norm = pre_norm
xtune/src/transformers/configuration_gpt2.py
+173
View File
@@ -0,0 +1,173 @@
# coding=utf-8
# Copyright 2018 The OpenAI Team Authors and HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" OpenAI GPT-2 configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"gpt2": "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json",
"gpt2-medium": "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-medium-config.json",
"gpt2-large": "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-large-config.json",
"gpt2-xl": "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-xl-config.json",
"distilgpt2": "https://s3.amazonaws.com/models.huggingface.co/bert/distilgpt2-config.json",
}
class GPT2Config(PretrainedConfig):
"""
This is the configuration class to store the configuration of a :class:`~transformers.GPT2Model`.
It is used to instantiate an GPT-2 model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the GPT-2 `small <https://huggingface.co/gpt2>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 50257):
Vocabulary size of the GPT-2 model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.GPT2Model`.
n_positions (:obj:`int`, optional, defaults to 1024):
The maximum sequence length that this model might ever be used with.
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
n_ctx (:obj:`int`, optional, defaults to 1024):
Dimensionality of the causal mask (usually same as n_positions).
n_embd (:obj:`int`, optional, defaults to 768):
Dimensionality of the embeddings and hidden states.
n_layer (:obj:`int`, optional, defaults to 12):
Number of hidden layers in the Transformer encoder.
n_head (:obj:`int`, optional, defaults to 12):
Number of attention heads for each attention layer in the Transformer encoder.
resid_pdrop (:obj:`float`, optional, defaults to 0.1):
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
embd_pdrop (:obj:`int`, optional, defaults to 0.1):
The dropout ratio for the embeddings.
attn_pdrop (:obj:`float`, optional, defaults to 0.1):
The dropout ratio for the attention.
layer_norm_epsilon (:obj:`float`, optional, defaults to 1e-5):
The epsilon to use in the layer normalization layers
initializer_range (:obj:`float`, optional, defaults to 16):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
summary_type (:obj:`string`, optional, defaults to "cls_index"):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.GPT2DoubleHeadsModel`.
Is one of the following options:
- 'last' => take the last token hidden state (like XLNet)
- 'first' => take the first token hidden state (like Bert)
- 'mean' => take the mean of all tokens hidden states
- 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
- 'attn' => Not implemented now, use multi-head attention
summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.GPT2DoubleHeadsModel`.
Add a projection after the vector extraction
summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.GPT2DoubleHeadsModel`.
'tanh' => add a tanh activation to the output, Other => no activation.
summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.GPT2DoubleHeadsModel`.
If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
summary_first_dropout (:obj:`float`, optional, defaults to 0.1):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.GPT2DoubleHeadsModel`.
Add a dropout before the projection and activation
Example::
from transformers import GPT2Model, GPT2Config
# Initializing a GPT2 configuration
configuration = GPT2Config()
# Initializing a model from the configuration
model = GPT2Model(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "gpt2"
def __init__(
self,
vocab_size=50257,
n_positions=1024,
n_ctx=1024,
n_embd=768,
n_layer=12,
n_head=12,
resid_pdrop=0.1,
embd_pdrop=0.1,
attn_pdrop=0.1,
layer_norm_epsilon=1e-5,
initializer_range=0.02,
summary_type="cls_index",
summary_use_proj=True,
summary_activation=None,
summary_proj_to_labels=True,
summary_first_dropout=0.1,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.n_ctx = n_ctx
self.n_positions = n_positions
self.n_embd = n_embd
self.n_layer = n_layer
self.n_head = n_head
self.resid_pdrop = resid_pdrop
self.embd_pdrop = embd_pdrop
self.attn_pdrop = attn_pdrop
self.layer_norm_epsilon = layer_norm_epsilon
self.initializer_range = initializer_range
self.summary_type = summary_type
self.summary_use_proj = summary_use_proj
self.summary_activation = summary_activation
self.summary_first_dropout = summary_first_dropout
self.summary_proj_to_labels = summary_proj_to_labels
@property
def max_position_embeddings(self):
return self.n_positions
@property
def hidden_size(self):
return self.n_embd
@property
def num_attention_heads(self):
return self.n_head
@property
def num_hidden_layers(self):
return self.n_layer
xtune/src/transformers/configuration_mmbt.py
+42
View File
@@ -0,0 +1,42 @@
# coding=utf-8
# Copyright (c) Facebook, Inc. and its affiliates.
# Copyright (c) HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" MMBT configuration """
import logging
logger = logging.getLogger(__name__)
class MMBTConfig(object):
"""Configuration class to store the configuration of a `MMBT Model`.
Args:
config (:obj:`~transformers.PreTrainedConfig`):
Config of the underlying Transformer models. Its values are
copied over to use a single config.
num_labels (:obj:`int` or :obj:`None`, optional, defaults to `None`):
Size of final Linear layer for classification.
modal_hidden_size (:obj:`int`, optional, defautls to 2048):
Embedding dimension of the non-text modality encoder.
"""
def __init__(self, config, num_labels=None, modal_hidden_size=2048):
self.__dict__ = config.__dict__
self.modal_hidden_size = modal_hidden_size
if num_labels:
self.num_labels = num_labels
xtune/src/transformers/configuration_openai.py
+177
View File
@@ -0,0 +1,177 @@
# coding=utf-8
# Copyright 2018 The OpenAI Team Authors and HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" OpenAI GPT configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"openai-gpt": "https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-config.json"
}
class OpenAIGPTConfig(PretrainedConfig):
"""
This is the configuration class to store the configuration of an :class:`~transformers.OpenAIGPTModel`.
It is used to instantiate an GPT model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the `GPT <https://huggingface.co/openai-gpt>`__ architecture from OpenAI.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 40478):
Vocabulary size of the GPT model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.CTRLModel`.
n_positions (:obj:`int`, optional, defaults to 512):
The maximum sequence length that this model might ever be used with.
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
n_ctx (:obj:`int`, optional, defaults to 512):
Dimensionality of the causal mask (usually same as n_positions).
n_embd (:obj:`int`, optional, defaults to 768):
Dimensionality of the embeddings and hidden states.
n_layer (:obj:`int`, optional, defaults to 12):
Number of hidden layers in the Transformer encoder.
n_head (:obj:`int`, optional, defaults to 12):
Number of attention heads for each attention layer in the Transformer encoder.
afn (:obj:`str` or :obj:`function`, optional, defaults to "gelu"):
The non-linear activation function (function or string) in the encoder and pooler.
If string, "gelu", "relu", "swish" and "gelu_new" are supported.
resid_pdrop (:obj:`float`, optional, defaults to 0.1):
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
embd_pdrop (:obj:`int`, optional, defaults to 0.1):
The dropout ratio for the embeddings.
attn_pdrop (:obj:`float`, optional, defaults to 0.1):
The dropout ratio for the attention.
layer_norm_epsilon (:obj:`float`, optional, defaults to 1e-5):
The epsilon to use in the layer normalization layers
initializer_range (:obj:`float`, optional, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
predict_special_tokens (:obj:`boolean`, optional, defaults to :obj:`True`):
Whether special tokens should be predicted when the model is has a language modeling head.
summary_type (:obj:`string`, optional, defaults to "cls_index"):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.OpenAIGPTDoubleHeadsModel`.
Is one of the following options:
- 'last' => take the last token hidden state (like XLNet)
- 'first' => take the first token hidden state (like Bert)
- 'mean' => take the mean of all tokens hidden states
- 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
- 'attn' => Not implemented now, use multi-head attention
summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.OpenAIGPTDoubleHeadsModel`.
Add a projection after the vector extraction
summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.OpenAIGPTDoubleHeadsModel`.
'tanh' => add a tanh activation to the output, Other => no activation.
summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.OpenAIGPTDoubleHeadsModel`.
If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
summary_first_dropout (:obj:`float`, optional, defaults to 0.1):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.OpenAIGPTDoubleHeadsModel`.
Add a dropout before the projection and activation
Example::
from transformers import OpenAIGPTConfig, OpenAIGPTModel
# Initializing a GPT configuration
configuration = OpenAIGPTConfig()
# Initializing a model from the configuration
model = OpenAIGPTModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "openai-gpt"
def __init__(
self,
vocab_size=40478,
n_positions=512,
n_ctx=512,
n_embd=768,
n_layer=12,
n_head=12,
afn="gelu",
resid_pdrop=0.1,
embd_pdrop=0.1,
attn_pdrop=0.1,
layer_norm_epsilon=1e-5,
initializer_range=0.02,
predict_special_tokens=True,
summary_type="cls_index",
summary_use_proj=True,
summary_activation=None,
summary_proj_to_labels=True,
summary_first_dropout=0.1,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.n_ctx = n_ctx
self.n_positions = n_positions
self.n_embd = n_embd
self.n_layer = n_layer
self.n_head = n_head
self.afn = afn
self.resid_pdrop = resid_pdrop
self.embd_pdrop = embd_pdrop
self.attn_pdrop = attn_pdrop
self.layer_norm_epsilon = layer_norm_epsilon
self.initializer_range = initializer_range
self.predict_special_tokens = predict_special_tokens
self.summary_type = summary_type
self.summary_use_proj = summary_use_proj
self.summary_activation = summary_activation
self.summary_first_dropout = summary_first_dropout
self.summary_proj_to_labels = summary_proj_to_labels
@property
def max_position_embeddings(self):
return self.n_positions
@property
def hidden_size(self):
return self.n_embd
@property
def num_attention_heads(self):
return self.n_head
@property
def num_hidden_layers(self):
return self.n_layer
xtune/src/transformers/configuration_roberta.py
+68
View File
@@ -0,0 +1,68 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" RoBERTa configuration """
import logging
from .configuration_bert import BertConfig
logger = logging.getLogger(__name__)
ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"roberta-base": "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-config.json",
"roberta-large": "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-config.json",
"roberta-large-mnli": "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-config.json",
"distilroberta-base": "https://s3.amazonaws.com/models.huggingface.co/bert/distilroberta-base-config.json",
"roberta-base-openai-detector": "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-openai-detector-config.json",
"roberta-large-openai-detector": "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-openai-detector-config.json",
}
class RobertaConfig(BertConfig):
r"""
This is the configuration class to store the configuration of an :class:`~transformers.RobertaModel`.
It is used to instantiate an RoBERTa model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the BERT `bert-base-uncased <https://huggingface.co/bert-base-uncased>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
The :class:`~transformers.RobertaConfig` class directly inherits :class:`~transformers.BertConfig`.
It reuses the same defaults. Please check the parent class for more information.
Example::
from transformers import RobertaConfig, RobertaModel
# Initializing a RoBERTa configuration
configuration = RobertaConfig()
# Initializing a model from the configuration
model = RobertaModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "roberta"
xtune/src/transformers/configuration_t5.py
+107
View File
@@ -0,0 +1,107 @@
# coding=utf-8
# Copyright 2010, The T5 Authors and HuggingFace Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" T5 model configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
T5_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"t5-small": "https://s3.amazonaws.com/models.huggingface.co/bert/t5-small-config.json",
"t5-base": "https://s3.amazonaws.com/models.huggingface.co/bert/t5-base-config.json",
"t5-large": "https://s3.amazonaws.com/models.huggingface.co/bert/t5-large-config.json",
"t5-3b": "https://s3.amazonaws.com/models.huggingface.co/bert/t5-3b-config.json",
"t5-11b": "https://s3.amazonaws.com/models.huggingface.co/bert/t5-11b-config.json",
}
class T5Config(PretrainedConfig):
r"""
:class:`~transformers.T5Config` is the configuration class to store the configuration of a
`T5Model`.
Arguments:
vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `T5Model`.
hidden_size: Size of the encoder layers and the pooler layer.
num_hidden_layers: Number of hidden layers in the Transformer encoder.
num_attention_heads: Number of attention heads for each attention layer in
the Transformer encoder.
intermediate_size: The size of the "intermediate" (i.e., feed-forward)
layer in the Transformer encoder.
hidden_act: The non-linear activation function (function or string) in the
encoder and pooler. If string, "gelu", "relu", "swish" and "gelu_new" are supported.
hidden_dropout_prob: The dropout probabilitiy for all fully connected
layers in the embeddings, encoder, and pooler.
attention_probs_dropout_prob: The dropout ratio for the attention
probabilities.
max_position_embeddings: The maximum sequence length that this model might
ever be used with. Typically set this to something large just in case
(e.g., 512 or 1024 or 2048).
type_vocab_size: The vocabulary size of the `token_type_ids` passed into
`T5Model`.
initializer_factor: A factor for initializing all weight matrices (should be kept to 1.0, used for initialization testing).
layer_norm_eps: The epsilon used by LayerNorm.
"""
pretrained_config_archive_map = T5_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "t5"
def __init__(
self,
vocab_size=32128,
n_positions=512,
d_model=512,
d_kv=64,
d_ff=2048,
num_layers=6,
num_heads=8,
relative_attention_num_buckets=32,
dropout_rate=0.1,
layer_norm_epsilon=1e-6,
initializer_factor=1.0,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.n_positions = n_positions
self.d_model = d_model
self.d_kv = d_kv
self.d_ff = d_ff
self.num_layers = num_layers
self.num_heads = num_heads
self.relative_attention_num_buckets = relative_attention_num_buckets
self.dropout_rate = dropout_rate
self.layer_norm_epsilon = layer_norm_epsilon
self.initializer_factor = initializer_factor
@property
def max_position_embeddings(self):
return self.n_positions
@property
def hidden_size(self):
return self.d_model
@property
def num_attention_heads(self):
return self.num_heads
@property
def num_hidden_layers(self):
return self.num_layers
xtune/src/transformers/configuration_transfo_xl.py
+211
View File
@@ -0,0 +1,211 @@
# coding=utf-8
# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Transformer XL configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"transfo-xl-wt103": "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-config.json",
}
class TransfoXLConfig(PretrainedConfig):
"""
This is the configuration class to store the configuration of an :class:`~transformers.TransfoXLModel`.
It is used to instantiate a Transformer XL model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the `Transformer XL <https://huggingface.co/transfo-xl-wt103>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 267735):
Vocabulary size of the Transformer XL model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.TransfoXLModel`.
cutoffs (:obj:`List[int]`, optional, defaults to :obj:`[20000, 40000, 200000]`):
Cutoffs for the adaptive softmax
d_model (:obj:`int`, optional, defaults to 1024):
Dimensionality of the model's hidden states.
d_embed (:obj:`int`, optional, defaults to 1024):
Dimensionality of the embeddings
n_head (:obj:`int`, optional, defaults to 16):
Number of attention heads for each attention layer in the Transformer encoder.
d_head (:obj:`int`, optional, defaults to 64):
Dimensionality of the model's heads.
d_inner (:obj:`int`, optional, defaults to 4096):
Inner dimension in FF
div_val (:obj:`int`, optional, defaults to 4):
Divident value for adapative input and softmax
pre_lnorm (:obj:`boolean`, optional, defaults to :obj:`False`):
Apply LayerNorm to the input instead of the output
n_layer (:obj:`int`, optional, defaults to 18):
Number of hidden layers in the Transformer encoder.
tgt_len (:obj:`int`, optional, defaults to 128):
Number of tokens to predict
ext_len (:obj:`int`, optional, defaults to 0):
Length of the extended context
mem_len (:obj:`int`, optional, defaults to 1600):
Length of the retained previous heads
clamp_len (:obj:`int`, optional, defaults to 1000):
use the same pos embeddings after clamp_len
same_length (:obj:`boolean`, optional, defaults to :obj:`True`):
Use the same attn length for all tokens
proj_share_all_but_first (:obj:`boolean`, optional, defaults to :obj:`True`):
True to share all but first projs, False not to share.
attn_type (:obj:`int`, optional, defaults to 0):
Attention type. 0 for Transformer-XL, 1 for Shaw et al, 2 for Vaswani et al, 3 for Al Rfou et al.
sample_softmax (:obj:`int`, optional, defaults to -1):
number of samples in sampled softmax
adaptive (:obj:`boolean`, optional, defaults to :obj:`True`):
use adaptive softmax
tie_weight (:obj:`boolean`, optional, defaults to :obj:`True`):
tie the word embedding and softmax weights
dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
dropatt (:obj:`float`, optional, defaults to 0):
The dropout ratio for the attention probabilities.
untie_r (:obj:`boolean`, optional, defaults to :obj:`True`):
Untie relative position biases
init (:obj:`string`, optional, defaults to `normal`):
Parameter initializer to use
init_range (:obj:`float`, optional, defaults to 0.01):
Parameters initialized by U(-init_range, init_range).
proj_init_std (:obj:`float`, optional, defaults to 0.01):
Parameters initialized by N(0, init_std)
init_std (:obj:`float`, optional, defaults to 0.02):
Parameters initialized by N(0, init_std)
layer_norm_epsilon (:obj:`float`, optional, defaults to 1e-5):
The epsilon to use in the layer normalization layers
Example::
from transformers import TransfoXLConfig, TransfoXLModel
# Initializing a Transformer XL configuration
configuration = TransfoXLConfig()
# Initializing a model from the configuration
model = TransfoXLModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "transfo-xl"
def __init__(
self,
vocab_size=267735,
cutoffs=[20000, 40000, 200000],
d_model=1024,
d_embed=1024,
n_head=16,
d_head=64,
d_inner=4096,
div_val=4,
pre_lnorm=False,
n_layer=18,
tgt_len=128,
ext_len=0,
mem_len=1600,
clamp_len=1000,
same_length=True,
proj_share_all_but_first=True,
attn_type=0,
sample_softmax=-1,
adaptive=True,
tie_weight=True,
dropout=0.1,
dropatt=0.0,
untie_r=True,
init="normal",
init_range=0.01,
proj_init_std=0.01,
init_std=0.02,
layer_norm_epsilon=1e-5,
**kwargs
):
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.cutoffs = []
self.cutoffs.extend(cutoffs)
self.tie_weight = tie_weight
if proj_share_all_but_first:
self.tie_projs = [False] + [True] * len(self.cutoffs)
else:
self.tie_projs = [False] + [False] * len(self.cutoffs)
self.d_model = d_model
self.d_embed = d_embed
self.d_head = d_head
self.d_inner = d_inner
self.div_val = div_val
self.pre_lnorm = pre_lnorm
self.n_layer = n_layer
self.n_head = n_head
self.tgt_len = tgt_len
self.ext_len = ext_len
self.mem_len = mem_len
self.same_length = same_length
self.attn_type = attn_type
self.clamp_len = clamp_len
self.sample_softmax = sample_softmax
self.adaptive = adaptive
self.dropout = dropout
self.dropatt = dropatt
self.untie_r = untie_r
self.init = init
self.init_range = init_range
self.proj_init_std = proj_init_std
self.init_std = init_std
self.layer_norm_epsilon = layer_norm_epsilon
@property
def max_position_embeddings(self):
return self.tgt_len + self.ext_len + self.mem_len
@property
def n_token(self): # Backward compatibility
return self.vocab_size
@n_token.setter
def n_token(self, value): # Backward compatibility
self.vocab_size = value
@property
def hidden_size(self):
return self.d_model
@property
def num_attention_heads(self):
return self.n_head
@property
def num_hidden_layers(self):
return self.n_layer
xtune/src/transformers/configuration_utils.py
+356
View File
@@ -0,0 +1,356 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Configuration base class and utilities."""
import copy
import json
import logging
import os
from typing import Dict, Optional, Tuple
from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
logger = logging.getLogger(__name__)
class PretrainedConfig(object):
r""" Base class for all configuration classes.
Handles a few parameters common to all models' configurations as well as methods for loading/downloading/saving configurations.
Note:
A configuration file can be loaded and saved to disk. Loading the configuration file and using this file to initialize a model does **not** load the model weights.
It only affects the model's configuration.
Class attributes (overridden by derived classes):
- ``pretrained_config_archive_map``: a python ``dict`` with `shortcut names` (string) as keys and `url` (string) of associated pretrained model configurations as values.
- ``model_type``: a string that identifies the model type, that we serialize into the JSON file, and that we use to recreate the correct object in :class:`~transformers.AutoConfig`.
Args:
finetuning_task (:obj:`string` or :obj:`None`, `optional`, defaults to :obj:`None`):
Name of the task used to fine-tune the model. This can be used when converting from an original (TensorFlow or PyTorch) checkpoint.
num_labels (:obj:`int`, `optional`, defaults to `2`):
Number of classes to use when the model is a classification model (sequences/tokens)
output_attentions (:obj:`bool`, `optional`, defaults to :obj:`False`):
Should the model returns attentions weights.
output_hidden_states (:obj:`string`, `optional`, defaults to :obj:`False`):
Should the model returns all hidden-states.
torchscript (:obj:`bool`, `optional`, defaults to :obj:`False`):
Is the model used with Torchscript (for PyTorch models).
"""
pretrained_config_archive_map = {} # type: Dict[str, str]
model_type = "" # type: str
def __init__(self, **kwargs):
# Attributes with defaults
self.output_attentions = kwargs.pop("output_attentions", False)
self.output_hidden_states = kwargs.pop("output_hidden_states", False)
self.output_past = kwargs.pop("output_past", True) # Not used by all models
self.torchscript = kwargs.pop("torchscript", False) # Only used by PyTorch models
self.use_bfloat16 = kwargs.pop("use_bfloat16", False)
self.pruned_heads = kwargs.pop("pruned_heads", {})
# Is decoder is used in encoder-decoder models to differentiate encoder from decoder
self.is_decoder = kwargs.pop("is_decoder", False)
# Parameters for sequence generation
self.max_length = kwargs.pop("max_length", 20)
self.do_sample = kwargs.pop("do_sample", False)
self.num_beams = kwargs.pop("num_beams", 1)
self.temperature = kwargs.pop("temperature", 1.0)
self.top_k = kwargs.pop("top_k", 50)
self.top_p = kwargs.pop("top_p", 1.0)
self.repetition_penalty = kwargs.pop("repetition_penalty", 1.0)
self.bos_token_id = kwargs.pop("bos_token_id", None)
self.pad_token_id = kwargs.pop("pad_token_id", None)
self.eos_token_ids = kwargs.pop("eos_token_ids", None)
self.length_penalty = kwargs.pop("length_penalty", 1.0)
self.num_return_sequences = kwargs.pop("num_return_sequences", 1)
# Fine-tuning task arguments
self.architectures = kwargs.pop("architectures", None)
self.finetuning_task = kwargs.pop("finetuning_task", None)
self.num_labels = kwargs.pop("num_labels", 2)
self.id2label = kwargs.pop("id2label", {i: "LABEL_{}".format(i) for i in range(self.num_labels)})
self.id2label = dict((int(key), value) for key, value in self.id2label.items())
self.label2id = kwargs.pop("label2id", dict(zip(self.id2label.values(), self.id2label.keys())))
self.label2id = dict((key, int(value)) for key, value in self.label2id.items())
# Additional attributes without default values
for key, value in kwargs.items():
try:
setattr(self, key, value)
except AttributeError as err:
logger.error("Can't set {} with value {} for {}".format(key, value, self))
raise err
def save_pretrained(self, save_directory):
"""
Save a configuration object to the directory `save_directory`, so that it
can be re-loaded using the :func:`~transformers.PretrainedConfig.from_pretrained` class method.
Args:
save_directory (:obj:`string`):
Directory where the configuration JSON file will be saved.
"""
assert os.path.isdir(
save_directory
), "Saving path should be a directory where the model and configuration can be saved"
# If we save using the predefined names, we can load using `from_pretrained`
output_config_file = os.path.join(save_directory, CONFIG_NAME)
self.to_json_file(output_config_file)
logger.info("Configuration saved in {}".format(output_config_file))
@classmethod
def from_pretrained(cls, pretrained_model_name_or_path, **kwargs) -> "PretrainedConfig":
r"""
Instantiate a :class:`~transformers.PretrainedConfig` (or a derived class) from a pre-trained model configuration.
Args:
pretrained_model_name_or_path (:obj:`string`):
either:
- a string with the `shortcut name` of a pre-trained model configuration to load from cache or
download, e.g.: ``bert-base-uncased``.
- a string with the `identifier name` of a pre-trained model configuration that was user-uploaded to
our S3, e.g.: ``dbmdz/bert-base-german-cased``.
- a path to a `directory` containing a configuration file saved using the
:func:`~transformers.PretrainedConfig.save_pretrained` method, e.g.: ``./my_model_directory/``.
- a path or url to a saved configuration JSON `file`, e.g.:
``./my_model_directory/configuration.json``.
cache_dir (:obj:`string`, `optional`):
Path to a directory in which a downloaded pre-trained model
configuration should be cached if the standard cache should not be used.
kwargs (:obj:`Dict[str, any]`, `optional`):
The values in kwargs of any keys which are configuration attributes will be used to override the loaded
values. Behavior concerning key/value pairs whose keys are *not* configuration attributes is
controlled by the `return_unused_kwargs` keyword parameter.
force_download (:obj:`bool`, `optional`, defaults to :obj:`False`):
Force to (re-)download the model weights and configuration files and override the cached versions if they exist.
resume_download (:obj:`bool`, `optional`, defaults to :obj:`False`):
Do not delete incompletely recieved file. Attempt to resume the download if such a file exists.
proxies (:obj:`Dict`, `optional`):
A dictionary of proxy servers to use by protocol or endpoint, e.g.:
:obj:`{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.`
The proxies are used on each request.
return_unused_kwargs: (`optional`) bool:
If False, then this function returns just the final configuration object.
If True, then this functions returns a :obj:`Tuple(config, unused_kwargs)` where `unused_kwargs` is a
dictionary consisting of the key/value pairs whose keys are not configuration attributes: ie the part
of kwargs which has not been used to update `config` and is otherwise ignored.
Returns:
:class:`PretrainedConfig`: An instance of a configuration object
Examples::
# We can't instantiate directly the base class `PretrainedConfig` so let's show the examples on a
# derived class: BertConfig
config = BertConfig.from_pretrained('bert-base-uncased') # Download configuration from S3 and cache.
config = BertConfig.from_pretrained('./test/saved_model/') # E.g. config (or model) was saved using `save_pretrained('./test/saved_model/')`
config = BertConfig.from_pretrained('./test/saved_model/my_configuration.json')
config = BertConfig.from_pretrained('bert-base-uncased', output_attention=True, foo=False)
assert config.output_attention == True
config, unused_kwargs = BertConfig.from_pretrained('bert-base-uncased', output_attention=True,
foo=False, return_unused_kwargs=True)
assert config.output_attention == True
assert unused_kwargs == {'foo': False}
"""
config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
return cls.from_dict(config_dict, **kwargs)
@classmethod
def get_config_dict(
cls, pretrained_model_name_or_path: str, pretrained_config_archive_map: Optional[Dict] = None, **kwargs
) -> Tuple[Dict, Dict]:
"""
From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used
for instantiating a Config using `from_dict`.
Parameters:
pretrained_model_name_or_path (:obj:`string`):
The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
pretrained_config_archive_map: (:obj:`Dict[str, str]`, `optional`) Dict:
A map of `shortcut names` to `url`. By default, will use the current class attribute.
Returns:
:obj:`Tuple[Dict, Dict]`: The dictionary that will be used to instantiate the configuration object.
"""
cache_dir = kwargs.pop("cache_dir", None)
force_download = kwargs.pop("force_download", False)
resume_download = kwargs.pop("resume_download", False)
proxies = kwargs.pop("proxies", None)
local_files_only = kwargs.pop("local_files_only", False)
if pretrained_config_archive_map is None:
pretrained_config_archive_map = cls.pretrained_config_archive_map
if pretrained_model_name_or_path in pretrained_config_archive_map:
config_file = pretrained_config_archive_map[pretrained_model_name_or_path]
elif os.path.isdir(pretrained_model_name_or_path):
config_file = os.path.join(pretrained_model_name_or_path, CONFIG_NAME)
elif os.path.isfile(pretrained_model_name_or_path) or is_remote_url(pretrained_model_name_or_path):
config_file = pretrained_model_name_or_path
else:
config_file = hf_bucket_url(pretrained_model_name_or_path, postfix=CONFIG_NAME)
try:
# Load from URL or cache if already cached
resolved_config_file = cached_path(
config_file,
cache_dir=cache_dir,
force_download=force_download,
proxies=proxies,
resume_download=resume_download,
local_files_only=local_files_only,
)
# Load config dict
if resolved_config_file is None:
raise EnvironmentError
config_dict = cls._dict_from_json_file(resolved_config_file)
except EnvironmentError:
if pretrained_model_name_or_path in pretrained_config_archive_map:
msg = "Couldn't reach server at '{}' to download pretrained model configuration file.".format(
config_file
)
else:
msg = (
"Model name '{}' was not found in model name list. "
"We assumed '{}' was a path, a model identifier, or url to a configuration file named {} or "
"a directory containing such a file but couldn't find any such file at this path or url.".format(
pretrained_model_name_or_path, config_file, CONFIG_NAME,
)
)
raise EnvironmentError(msg)
except json.JSONDecodeError:
msg = (
"Couldn't reach server at '{}' to download configuration file or "
"configuration file is not a valid JSON file. "
"Please check network or file content here: {}.".format(config_file, resolved_config_file)
)
raise EnvironmentError(msg)
if resolved_config_file == config_file:
logger.info("loading configuration file {}".format(config_file))
else:
logger.info("loading configuration file {} from cache at {}".format(config_file, resolved_config_file))
return config_dict, kwargs
@classmethod
def from_dict(cls, config_dict: Dict, **kwargs) -> "PretrainedConfig":
"""
Constructs a `Config` from a Python dictionary of parameters.
Args:
config_dict (:obj:`Dict[str, any]`):
Dictionary that will be used to instantiate the configuration object. Such a dictionary can be retrieved
from a pre-trained checkpoint by leveraging the :func:`~transformers.PretrainedConfig.get_config_dict`
method.
kwargs (:obj:`Dict[str, any]`):
Additional parameters from which to initialize the configuration object.
Returns:
:class:`PretrainedConfig`: An instance of a configuration object
"""
return_unused_kwargs = kwargs.pop("return_unused_kwargs", False)
config = cls(**config_dict)
if hasattr(config, "pruned_heads"):
config.pruned_heads = dict((int(key), value) for key, value in config.pruned_heads.items())
# Update config with kwargs if needed
to_remove = []
for key, value in kwargs.items():
if hasattr(config, key):
setattr(config, key, value)
to_remove.append(key)
for key in to_remove:
kwargs.pop(key, None)
logger.info("Model config %s", str(config))
if return_unused_kwargs:
return config, kwargs
else:
return config
@classmethod
def from_json_file(cls, json_file: str) -> "PretrainedConfig":
"""
Constructs a `Config` from the path to a json file of parameters.
Args:
json_file (:obj:`string`):
Path to the JSON file containing the parameters.
Returns:
:class:`PretrainedConfig`: An instance of a configuration object
"""
config_dict = cls._dict_from_json_file(json_file)
return cls(**config_dict)
@classmethod
def _dict_from_json_file(cls, json_file: str):
with open(json_file, "r", encoding="utf-8") as reader:
text = reader.read()
return json.loads(text)
def __eq__(self, other):
return self.__dict__ == other.__dict__
def __repr__(self):
return "{} {}".format(self.__class__.__name__, self.to_json_string())
def to_dict(self):
"""
Serializes this instance to a Python dictionary.
Returns:
:obj:`Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
"""
output = copy.deepcopy(self.__dict__)
if hasattr(self.__class__, "model_type"):
output["model_type"] = self.__class__.model_type
return output
def to_json_string(self):
"""
Serializes this instance to a JSON string.
Returns:
:obj:`string`: String containing all the attributes that make up this configuration instance in JSON format.
"""
return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
def to_json_file(self, json_file_path):
"""
Save this instance to a json file.
Args:
json_file_path (:obj:`string`):
Path to the JSON file in which this configuration instance's parameters will be saved.
"""
with open(json_file_path, "w", encoding="utf-8") as writer:
writer.write(self.to_json_string())
xtune/src/transformers/configuration_xlm.py
+255
View File
@@ -0,0 +1,255 @@
# coding=utf-8
# Copyright 2019-present, Facebook, Inc and the HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" XLM configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
XLM_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"xlm-mlm-en-2048": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-en-2048-config.json",
"xlm-mlm-ende-1024": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-ende-1024-config.json",
"xlm-mlm-enfr-1024": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-enfr-1024-config.json",
"xlm-mlm-enro-1024": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-enro-1024-config.json",
"xlm-mlm-tlm-xnli15-1024": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-tlm-xnli15-1024-config.json",
"xlm-mlm-xnli15-1024": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-xnli15-1024-config.json",
"xlm-clm-enfr-1024": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-clm-enfr-1024-config.json",
"xlm-clm-ende-1024": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-clm-ende-1024-config.json",
"xlm-mlm-17-1280": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-17-1280-config.json",
"xlm-mlm-100-1280": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-100-1280-config.json",
}
class XLMConfig(PretrainedConfig):
"""
This is the configuration class to store the configuration of a :class:`~transformers.XLMModel`.
It is used to instantiate an XLM model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the `xlm-mlm-en-2048 <https://huggingface.co/xlm-mlm-en-2048>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 30145):
Vocabulary size of the XLM model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.XLMModel`.
emb_dim (:obj:`int`, optional, defaults to 2048):
Dimensionality of the encoder layers and the pooler layer.
n_layer (:obj:`int`, optional, defaults to 12):
Number of hidden layers in the Transformer encoder.
n_head (:obj:`int`, optional, defaults to 16):
Number of attention heads for each attention layer in the Transformer encoder.
dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probability for all fully connected
layers in the embeddings, encoder, and pooler.
attention_dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probability for the attention mechanism
gelu_activation (:obj:`boolean`, optional, defaults to :obj:`True`):
The non-linear activation function (function or string) in the
encoder and pooler. If set to `True`, "gelu" will be used instead of "relu".
sinusoidal_embeddings (:obj:`boolean`, optional, defaults to :obj:`False`):
Whether to use sinusoidal positional embeddings instead of absolute positional embeddings.
causal (:obj:`boolean`, optional, defaults to :obj:`False`):
Set this to `True` for the model to behave in a causal manner.
Causal models use a triangular attention mask in order to only attend to the left-side context instead
if a bidirectional context.
asm (:obj:`boolean`, optional, defaults to :obj:`False`):
Whether to use an adaptive log softmax projection layer instead of a linear layer for the prediction
layer.
n_langs (:obj:`int`, optional, defaults to 1):
The number of languages the model handles. Set to 1 for monolingual models.
use_lang_emb (:obj:`boolean`, optional, defaults to :obj:`True`)
Whether to use language embeddings. Some models use additional language embeddings, see
`the multilingual models page <http://huggingface.co/transformers/multilingual.html#xlm-language-embeddings>`__
for information on how to use them.
max_position_embeddings (:obj:`int`, optional, defaults to 512):
The maximum sequence length that this model might
ever be used with. Typically set this to something large just in case
(e.g., 512 or 1024 or 2048).
embed_init_std (:obj:`float`, optional, defaults to 2048^-0.5):
The standard deviation of the truncated_normal_initializer for
initializing the embedding matrices.
init_std (:obj:`int`, optional, defaults to 50257):
The standard deviation of the truncated_normal_initializer for
initializing all weight matrices except the embedding matrices.
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
The epsilon used by the layer normalization layers.
bos_index (:obj:`int`, optional, defaults to 0):
The index of the beginning of sentence token in the vocabulary.
eos_index (:obj:`int`, optional, defaults to 1):
The index of the end of sentence token in the vocabulary.
pad_index (:obj:`int`, optional, defaults to 2):
The index of the padding token in the vocabulary.
unk_index (:obj:`int`, optional, defaults to 3):
The index of the unknown token in the vocabulary.
mask_index (:obj:`int`, optional, defaults to 5):
The index of the masking token in the vocabulary.
is_encoder(:obj:`boolean`, optional, defaults to :obj:`True`):
Whether the initialized model should be a transformer encoder or decoder as seen in Vaswani et al.
summary_type (:obj:`string`, optional, defaults to "first"):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
Is one of the following options:
- 'last' => take the last token hidden state (like XLNet)
- 'first' => take the first token hidden state (like Bert)
- 'mean' => take the mean of all tokens hidden states
- 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
- 'attn' => Not implemented now, use multi-head attention
summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
Add a projection after the vector extraction
summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
'tanh' => add a tanh activation to the output, Other => no activation.
summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
summary_first_dropout (:obj:`float`, optional, defaults to 0.1):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLMForSequenceClassification`.
Add a dropout before the projection and activation
start_n_top (:obj:`int`, optional, defaults to 5):
Used in the SQuAD evaluation script for XLM and XLNet.
end_n_top (:obj:`int`, optional, defaults to 5):
Used in the SQuAD evaluation script for XLM and XLNet.
mask_token_id (:obj:`int`, optional, defaults to 0):
Model agnostic parameter to identify masked tokens when generating text in an MLM context.
lang_id (:obj:`int`, optional, defaults to 1):
The ID of the language used by the model. This parameter is used when generating
text in a given language.
Example::
from transformers import XLMConfig, XLMModel
# Initializing a XLM configuration
configuration = XLMConfig()
# Initializing a model from the configuration
model = XLMModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = XLM_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "xlm"
def __init__(
self,
vocab_size=30145,
emb_dim=2048,
n_layers=12,
n_heads=16,
dropout=0.1,
attention_dropout=0.1,
gelu_activation=True,
sinusoidal_embeddings=False,
causal=False,
asm=False,
n_langs=1,
use_lang_emb=True,
max_position_embeddings=512,
embed_init_std=2048 ** -0.5,
layer_norm_eps=1e-12,
init_std=0.02,
bos_index=0,
eos_index=1,
pad_index=2,
unk_index=3,
mask_index=5,
is_encoder=True,
summary_type="first",
summary_use_proj=True,
summary_activation=None,
summary_proj_to_labels=True,
summary_first_dropout=0.1,
start_n_top=5,
end_n_top=5,
mask_token_id=0,
lang_id=0,
**kwargs
):
"""Constructs XLMConfig.
"""
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.emb_dim = emb_dim
self.n_layers = n_layers
self.n_heads = n_heads
self.dropout = dropout
self.attention_dropout = attention_dropout
self.gelu_activation = gelu_activation
self.sinusoidal_embeddings = sinusoidal_embeddings
self.causal = causal
self.asm = asm
self.n_langs = n_langs
self.use_lang_emb = use_lang_emb
self.layer_norm_eps = layer_norm_eps
self.bos_index = bos_index
self.eos_index = eos_index
self.pad_index = pad_index
self.unk_index = unk_index
self.mask_index = mask_index
self.is_encoder = is_encoder
self.max_position_embeddings = max_position_embeddings
self.embed_init_std = embed_init_std
self.init_std = init_std
self.summary_type = summary_type
self.summary_use_proj = summary_use_proj
self.summary_activation = summary_activation
self.summary_proj_to_labels = summary_proj_to_labels
self.summary_first_dropout = summary_first_dropout
self.start_n_top = start_n_top
self.end_n_top = end_n_top
self.mask_token_id = mask_token_id
self.lang_id = lang_id
if "n_words" in kwargs:
self.n_words = kwargs["n_words"]
@property
def n_words(self): # For backward compatibility
return self.vocab_size
@n_words.setter
def n_words(self, value): # For backward compatibility
self.vocab_size = value
@property
def hidden_size(self):
return self.emb_dim
@property
def num_attention_heads(self):
return self.n_heads
@property
def num_hidden_layers(self):
return self.n_layers
xtune/src/transformers/configuration_xlm_roberta.py
+43
View File
@@ -0,0 +1,43 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" XLM-RoBERTa configuration """
import logging
from .configuration_roberta import RobertaConfig
logger = logging.getLogger(__name__)
XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"xlm-roberta-base": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-base-config.json",
"xlm-roberta-large": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-large-config.json",
"xlm-roberta-large-finetuned-conll02-dutch": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-large-finetuned-conll02-dutch-config.json",
"xlm-roberta-large-finetuned-conll02-spanish": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-large-finetuned-conll02-spanish-config.json",
"xlm-roberta-large-finetuned-conll03-english": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-large-finetuned-conll03-english-config.json",
"xlm-roberta-large-finetuned-conll03-german": "https://s3.amazonaws.com/models.huggingface.co/bert/xlm-roberta-large-finetuned-conll03-german-config.json",
}
class XLMRobertaConfig(RobertaConfig):
"""
This class overrides :class:`~transformers.RobertaConfig`. Please check the
superclass for the appropriate documentation alongside usage examples.
"""
pretrained_config_archive_map = XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "xlm-roberta"
xtune/src/transformers/configuration_xlnet.py
+213
View File
@@ -0,0 +1,213 @@
# coding=utf-8
# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" XLNet configuration """
import logging
from .configuration_utils import PretrainedConfig
logger = logging.getLogger(__name__)
XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP = {
"xlnet-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-base-cased-config.json",
"xlnet-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-large-cased-config.json",
}
class XLNetConfig(PretrainedConfig):
"""
This is the configuration class to store the configuration of a :class:`~transformers.XLNetModel`.
It is used to instantiate an XLNet model according to the specified arguments, defining the model
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
the `xlnet-large-cased <https://huggingface.co/xlnet-large-cased>`__ architecture.
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
for more information.
Args:
vocab_size (:obj:`int`, optional, defaults to 32000):
Vocabulary size of the XLNet model. Defines the different tokens that
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.XLNetModel`.
d_model (:obj:`int`, optional, defaults to 1024):
Dimensionality of the encoder layers and the pooler layer.
n_layer (:obj:`int`, optional, defaults to 24):
Number of hidden layers in the Transformer encoder.
n_head (:obj:`int`, optional, defaults to 16):
Number of attention heads for each attention layer in the Transformer encoder.
d_inner (:obj:`int`, optional, defaults to 4096):
Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
ff_activation (:obj:`string`, optional, defaults to "gelu"):
The non-linear activation function (function or string) in the
encoder and pooler. If string, "gelu", "relu" and "swish" are supported.
untie_r (:obj:`boolean`, optional, defaults to :obj:`True`):
Untie relative position biases
attn_type (:obj:`string`, optional, defaults to "bi"):
The attention type used by the model. Set 'bi' for XLNet, 'uni' for Transformer-XL.
initializer_range (:obj:`float`, optional, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
The epsilon used by the layer normalization layers.
dropout (:obj:`float`, optional, defaults to 0.1):
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
mem_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):
The number of tokens to cache. The key/value pairs that have already been pre-computed
in a previous forward pass won't be re-computed. See the
`quickstart <https://huggingface.co/transformers/quickstart.html#using-the-past>`__
for more information.
reuse_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):
The number of tokens in the current batch to be cached and reused in the future.
bi_data (:obj:`boolean`, optional, defaults to :obj:`False`):
Whether to use bidirectional input pipeline. Usually set to `True` during
pretraining and `False` during finetuning.
clamp_len (:obj:`int`, optional, defaults to -1):
Clamp all relative distances larger than clamp_len.
Setting this attribute to -1 means no clamping.
same_length (:obj:`boolean`, optional, defaults to :obj:`False`):
Whether to use the same attention length for each token.
summary_type (:obj:`string`, optional, defaults to "last"):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
Is one of the following options:
- 'last' => take the last token hidden state (like XLNet)
- 'first' => take the first token hidden state (like Bert)
- 'mean' => take the mean of all tokens hidden states
- 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
- 'attn' => Not implemented now, use multi-head attention
summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
Add a projection after the vector extraction
summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
'tanh' => add a tanh activation to the output, Other => no activation.
summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
summary_last_dropout (:obj:`float`, optional, defaults to 0.1):
Argument used when doing sequence summary. Used in for the multiple choice head in
:class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
Add a dropout after the projection and activation
start_n_top (:obj:`int`, optional, defaults to 5):
Used in the SQuAD evaluation script for XLM and XLNet.
end_n_top (:obj:`int`, optional, defaults to 5):
Used in the SQuAD evaluation script for XLM and XLNet.
Example::
from transformers import XLNetConfig, XLNetModel
# Initializing a XLNet configuration
configuration = XLNetConfig()
# Initializing a model from the configuration
model = XLNetModel(configuration)
# Accessing the model configuration
configuration = model.config
Attributes:
pretrained_config_archive_map (Dict[str, str]):
A dictionary containing all the available pre-trained checkpoints.
"""
pretrained_config_archive_map = XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP
model_type = "xlnet"
def __init__(
self,
vocab_size=32000,
d_model=1024,
n_layer=24,
n_head=16,
d_inner=4096,
ff_activation="gelu",
untie_r=True,
attn_type="bi",
initializer_range=0.02,
layer_norm_eps=1e-12,
dropout=0.1,
mem_len=None,
reuse_len=None,
bi_data=False,
clamp_len=-1,
same_length=False,
summary_type="last",
summary_use_proj=True,
summary_activation="tanh",
summary_last_dropout=0.1,
start_n_top=5,
end_n_top=5,
**kwargs
):
"""Constructs XLNetConfig.
"""
super().__init__(**kwargs)
self.vocab_size = vocab_size
self.d_model = d_model
self.n_layer = n_layer
self.n_head = n_head
assert d_model % n_head == 0
self.d_head = d_model // n_head
self.ff_activation = ff_activation
self.d_inner = d_inner
self.untie_r = untie_r
self.attn_type = attn_type
self.initializer_range = initializer_range
self.layer_norm_eps = layer_norm_eps
self.dropout = dropout
self.mem_len = mem_len
self.reuse_len = reuse_len
self.bi_data = bi_data
self.clamp_len = clamp_len
self.same_length = same_length
self.summary_type = summary_type
self.summary_use_proj = summary_use_proj
self.summary_activation = summary_activation
self.summary_last_dropout = summary_last_dropout
self.start_n_top = start_n_top
self.end_n_top = end_n_top
@property
def max_position_embeddings(self):
return -1
@property
def n_token(self): # Backward compatibility
return self.vocab_size
@n_token.setter
def n_token(self, value): # Backward compatibility
self.vocab_size = value
@property
def hidden_size(self):
return self.d_model
@property
def num_attention_heads(self):
return self.n_head
@property
def num_hidden_layers(self):
return self.n_layer
xtune/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py
+61
View File
@@ -0,0 +1,61 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert ALBERT checkpoint."""
import argparse
import logging
import torch
from transformers import AlbertConfig, AlbertForMaskedLM, load_tf_weights_in_albert
logging.basicConfig(level=logging.INFO)
def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, albert_config_file, pytorch_dump_path):
# Initialise PyTorch model
config = AlbertConfig.from_json_file(albert_config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = AlbertForMaskedLM(config)
# Load weights from tf checkpoint
load_tf_weights_in_albert(model, config, tf_checkpoint_path)
# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path."
)
parser.add_argument(
"--albert_config_file",
default=None,
type=str,
required=True,
help="The config json file corresponding to the pre-trained ALBERT model. \n"
"This specifies the model architecture.",
)
parser.add_argument(
"--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
)
args = parser.parse_args()
convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.albert_config_file, args.pytorch_dump_path)
xtune/src/transformers/convert_bart_original_pytorch_checkpoint_to_pytorch.py
+112
View File
@@ -0,0 +1,112 @@
# coding=utf-8
# Copyright 2020 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert BART checkpoint."""
import argparse
import logging
from pathlib import Path
import fairseq
import torch
from packaging import version
from transformers import BartConfig, BartForMaskedLM, BartForSequenceClassification, BartModel, BartTokenizer
FAIRSEQ_MODELS = ["bart.large", "bart.large.mnli", "bart.large.cnn"]
if version.parse(fairseq.__version__) < version.parse("0.9.0"):
raise Exception("requires fairseq >= 0.9.0")
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
SAMPLE_TEXT = " Hello world! cécé herlolip"
rename_keys = [
("model.classification_heads.mnli.dense.weight", "classification_head.dense.weight"),
("model.classification_heads.mnli.dense.bias", "classification_head.dense.bias"),
("model.classification_heads.mnli.out_proj.weight", "classification_head.out_proj.weight"),
("model.classification_heads.mnli.out_proj.bias", "classification_head.out_proj.bias"),
]
IGNORE_KEYS = ["encoder.version", "decoder.version", "model.encoder.version", "model.decoder.version", "_float_tensor"]
def rename_key(dct, old, new):
val = dct.pop(old)
dct[new] = val
def convert_bart_checkpoint(checkpoint_path, pytorch_dump_folder_path):
"""
Copy/paste/tweak model's weights to our BERT structure.
"""
bart = torch.hub.load("pytorch/fairseq", checkpoint_path)
bart.eval() # disable dropout
bart.model.upgrade_state_dict(bart.model.state_dict())
hf_model_name = checkpoint_path.replace(".", "-")
config = BartConfig.from_pretrained(hf_model_name)
tokens = bart.encode(SAMPLE_TEXT).unsqueeze(0)
tokens2 = BartTokenizer.from_pretrained(hf_model_name).encode(SAMPLE_TEXT, return_tensors="pt").unsqueeze(0)
assert torch.eq(tokens, tokens2).all()
if checkpoint_path in ["bart.large", "bart.large.cnn"]:
state_dict = bart.model.state_dict()
for k in IGNORE_KEYS:
state_dict.pop(k, None)
state_dict["shared.weight"] = state_dict["decoder.embed_tokens.weight"]
model = BartModel(config)
their_output = bart.extract_features(tokens)
else: # MNLI Case
state_dict = bart.state_dict()
for k in IGNORE_KEYS:
state_dict.pop(k, None)
state_dict["model.shared.weight"] = state_dict["model.decoder.embed_tokens.weight"]
for src, dest in rename_keys:
rename_key(state_dict, src, dest)
model = BartForSequenceClassification(config)
their_output = bart.predict("mnli", tokens, return_logits=True)
# Load state dict
model.load_state_dict(state_dict)
model.eval()
# Check results
if checkpoint_path == "bart.large.cnn": # generate doesnt work yet
model = BartForMaskedLM(config, base_model=model)
assert "lm_head.weight" in model.state_dict()
assert model.lm_head.out_features == config.max_position_embeddings
model.eval()
our_outputs = model.model.forward(tokens)[0]
else:
our_outputs = model.forward(tokens)[0]
assert their_output.shape == our_outputs.shape
assert (their_output == our_outputs).all().item()
Path(pytorch_dump_folder_path).mkdir(exist_ok=True)
model.save_pretrained(pytorch_dump_folder_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument("fairseq_path", choices=FAIRSEQ_MODELS, type=str, help="")
parser.add_argument("pytorch_dump_folder_path", default=None, type=str, help="Path to the output PyTorch model.")
args = parser.parse_args()
convert_bart_checkpoint(
args.fairseq_path, args.pytorch_dump_folder_path,
)
xtune/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py
+61
View File
@@ -0,0 +1,61 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert BERT checkpoint."""
import argparse
import logging
import torch
from transformers import BertConfig, BertForPreTraining, load_tf_weights_in_bert
logging.basicConfig(level=logging.INFO)
def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path):
# Initialise PyTorch model
config = BertConfig.from_json_file(bert_config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = BertForPreTraining(config)
# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)
# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path."
)
parser.add_argument(
"--bert_config_file",
default=None,
type=str,
required=True,
help="The config json file corresponding to the pre-trained BERT model. \n"
"This specifies the model architecture.",
)
parser.add_argument(
"--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
)
args = parser.parse_args()
convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path)
xtune/src/transformers/convert_bert_pytorch_checkpoint_to_original_tf.py
+112
View File
@@ -0,0 +1,112 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert Huggingface Pytorch checkpoint to Tensorflow checkpoint."""
import argparse
import os
import numpy as np
import tensorflow as tf
import torch
from transformers import BertModel
def convert_pytorch_checkpoint_to_tf(model: BertModel, ckpt_dir: str, model_name: str):
"""
:param model:BertModel Pytorch model instance to be converted
:param ckpt_dir: Tensorflow model directory
:param model_name: model name
:return:
Currently supported HF models:
Y BertModel
N BertForMaskedLM
N BertForPreTraining
N BertForMultipleChoice
N BertForNextSentencePrediction
N BertForSequenceClassification
N BertForQuestionAnswering
"""
tensors_to_transpose = ("dense.weight", "attention.self.query", "attention.self.key", "attention.self.value")
var_map = (
("layer.", "layer_"),
("word_embeddings.weight", "word_embeddings"),
("position_embeddings.weight", "position_embeddings"),
("token_type_embeddings.weight", "token_type_embeddings"),
(".", "/"),
("LayerNorm/weight", "LayerNorm/gamma"),
("LayerNorm/bias", "LayerNorm/beta"),
("weight", "kernel"),
)
if not os.path.isdir(ckpt_dir):
os.makedirs(ckpt_dir)
state_dict = model.state_dict()
def to_tf_var_name(name: str):
for patt, repl in iter(var_map):
name = name.replace(patt, repl)
return "bert/{}".format(name)
def create_tf_var(tensor: np.ndarray, name: str, session: tf.Session):
tf_dtype = tf.dtypes.as_dtype(tensor.dtype)
tf_var = tf.get_variable(dtype=tf_dtype, shape=tensor.shape, name=name, initializer=tf.zeros_initializer())
session.run(tf.variables_initializer([tf_var]))
session.run(tf_var)
return tf_var
tf.reset_default_graph()
with tf.Session() as session:
for var_name in state_dict:
tf_name = to_tf_var_name(var_name)
torch_tensor = state_dict[var_name].numpy()
if any([x in var_name for x in tensors_to_transpose]):
torch_tensor = torch_tensor.T
tf_var = create_tf_var(tensor=torch_tensor, name=tf_name, session=session)
tf.keras.backend.set_value(tf_var, torch_tensor)
tf_weight = session.run(tf_var)
print("Successfully created {}: {}".format(tf_name, np.allclose(tf_weight, torch_tensor)))
saver = tf.train.Saver(tf.trainable_variables())
saver.save(session, os.path.join(ckpt_dir, model_name.replace("-", "_") + ".ckpt"))
def main(raw_args=None):
parser = argparse.ArgumentParser()
parser.add_argument("--model_name", type=str, required=True, help="model name e.g. bert-base-uncased")
parser.add_argument(
"--cache_dir", type=str, default=None, required=False, help="Directory containing pytorch model"
)
parser.add_argument("--pytorch_model_path", type=str, required=True, help="/path/to/<pytorch-model-name>.bin")
parser.add_argument("--tf_cache_dir", type=str, required=True, help="Directory in which to save tensorflow model")
args = parser.parse_args(raw_args)
model = BertModel.from_pretrained(
pretrained_model_name_or_path=args.model_name,
state_dict=torch.load(args.pytorch_model_path),
cache_dir=args.cache_dir,
)
convert_pytorch_checkpoint_to_tf(model=model, ckpt_dir=args.tf_cache_dir, model_name=args.model_name)
if __name__ == "__main__":
main()
xtune/src/transformers/convert_gpt2_original_tf_checkpoint_to_pytorch.py
+67
View File
@@ -0,0 +1,67 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert OpenAI GPT checkpoint."""
import argparse
import logging
import torch
from transformers import CONFIG_NAME, WEIGHTS_NAME, GPT2Config, GPT2Model, load_tf_weights_in_gpt2
logging.basicConfig(level=logging.INFO)
def convert_gpt2_checkpoint_to_pytorch(gpt2_checkpoint_path, gpt2_config_file, pytorch_dump_folder_path):
# Construct model
if gpt2_config_file == "":
config = GPT2Config()
else:
config = GPT2Config.from_json_file(gpt2_config_file)
model = GPT2Model(config)
# Load weights from numpy
load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path)
# Save pytorch-model
pytorch_weights_dump_path = pytorch_dump_folder_path + "/" + WEIGHTS_NAME
pytorch_config_dump_path = pytorch_dump_folder_path + "/" + CONFIG_NAME
print("Save PyTorch model to {}".format(pytorch_weights_dump_path))
torch.save(model.state_dict(), pytorch_weights_dump_path)
print("Save configuration file to {}".format(pytorch_config_dump_path))
with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
f.write(config.to_json_string())
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--gpt2_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path."
)
parser.add_argument(
"--pytorch_dump_folder_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
)
parser.add_argument(
"--gpt2_config_file",
default="",
type=str,
help="An optional config json file corresponding to the pre-trained OpenAI model. \n"
"This specifies the model architecture.",
)
args = parser.parse_args()
convert_gpt2_checkpoint_to_pytorch(args.gpt2_checkpoint_path, args.gpt2_config_file, args.pytorch_dump_folder_path)
xtune/src/transformers/convert_openai_original_tf_checkpoint_to_pytorch.py
+73
View File
@@ -0,0 +1,73 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert OpenAI GPT checkpoint."""
import argparse
import logging
import torch
from transformers import CONFIG_NAME, WEIGHTS_NAME, OpenAIGPTConfig, OpenAIGPTModel, load_tf_weights_in_openai_gpt
logging.basicConfig(level=logging.INFO)
def convert_openai_checkpoint_to_pytorch(openai_checkpoint_folder_path, openai_config_file, pytorch_dump_folder_path):
# Construct model
if openai_config_file == "":
config = OpenAIGPTConfig()
else:
config = OpenAIGPTConfig.from_json_file(openai_config_file)
model = OpenAIGPTModel(config)
# Load weights from numpy
load_tf_weights_in_openai_gpt(model, config, openai_checkpoint_folder_path)
# Save pytorch-model
pytorch_weights_dump_path = pytorch_dump_folder_path + "/" + WEIGHTS_NAME
pytorch_config_dump_path = pytorch_dump_folder_path + "/" + CONFIG_NAME
print("Save PyTorch model to {}".format(pytorch_weights_dump_path))
torch.save(model.state_dict(), pytorch_weights_dump_path)
print("Save configuration file to {}".format(pytorch_config_dump_path))
with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
f.write(config.to_json_string())
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--openai_checkpoint_folder_path",
default=None,
type=str,
required=True,
help="Path to the TensorFlow checkpoint path.",
)
parser.add_argument(
"--pytorch_dump_folder_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
)
parser.add_argument(
"--openai_config_file",
default="",
type=str,
help="An optional config json file corresponding to the pre-trained OpenAI model. \n"
"This specifies the model architecture.",
)
args = parser.parse_args()
convert_openai_checkpoint_to_pytorch(
args.openai_checkpoint_folder_path, args.openai_config_file, args.pytorch_dump_folder_path
)
xtune/src/transformers/convert_pytorch_checkpoint_to_tf2.py
+500
View File
@@ -0,0 +1,500 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Convert pytorch checkpoints to TensorFlow """
import argparse
import logging
import os
from transformers import (
ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP,
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP,
OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP,
ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP,
XLM_PRETRAINED_CONFIG_ARCHIVE_MAP,
XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
AlbertConfig,
BertConfig,
CamembertConfig,
CTRLConfig,
DistilBertConfig,
GPT2Config,
OpenAIGPTConfig,
RobertaConfig,
T5Config,
TFAlbertForMaskedLM,
TFBertForPreTraining,
TFBertForQuestionAnswering,
TFBertForSequenceClassification,
TFCamembertForMaskedLM,
TFCTRLLMHeadModel,
TFDistilBertForMaskedLM,
TFDistilBertForQuestionAnswering,
TFGPT2LMHeadModel,
TFOpenAIGPTLMHeadModel,
TFRobertaForMaskedLM,
TFRobertaForSequenceClassification,
TFT5WithLMHeadModel,
TFTransfoXLLMHeadModel,
TFXLMRobertaForMaskedLM,
TFXLMWithLMHeadModel,
TFXLNetLMHeadModel,
TransfoXLConfig,
XLMConfig,
XLMRobertaConfig,
XLNetConfig,
cached_path,
is_torch_available,
load_pytorch_checkpoint_in_tf2_model,
)
if is_torch_available():
import torch
import numpy as np
from transformers import (
BertForPreTraining,
BertForQuestionAnswering,
BertForSequenceClassification,
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
GPT2LMHeadModel,
GPT2_PRETRAINED_MODEL_ARCHIVE_MAP,
XLNetLMHeadModel,
XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
XLMWithLMHeadModel,
XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
XLMRobertaForMaskedLM,
TransfoXLLMHeadModel,
TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
OpenAIGPTLMHeadModel,
OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP,
RobertaForMaskedLM,
RobertaForSequenceClassification,
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
CamembertForMaskedLM,
CamembertForSequenceClassification,
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
DistilBertForMaskedLM,
DistilBertForQuestionAnswering,
DistilBertForSequenceClassification,
DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
CTRLLMHeadModel,
CTRL_PRETRAINED_MODEL_ARCHIVE_MAP,
AlbertForMaskedLM,
ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
T5WithLMHeadModel,
T5_PRETRAINED_MODEL_ARCHIVE_MAP,
)
else:
(
BertForPreTraining,
BertForQuestionAnswering,
BertForSequenceClassification,
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
GPT2LMHeadModel,
GPT2_PRETRAINED_MODEL_ARCHIVE_MAP,
XLNetLMHeadModel,
XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
XLMWithLMHeadModel,
XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
XLMRobertaForMaskedLM,
TransfoXLLMHeadModel,
TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
OpenAIGPTLMHeadModel,
OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP,
RobertaForMaskedLM,
RobertaForSequenceClassification,
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
CamembertForMaskedLM,
CamembertForSequenceClassification,
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
DistilBertForMaskedLM,
DistilBertForSequenceClassification,
DistilBertForQuestionAnswering,
DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
CTRLLMHeadModel,
CTRL_PRETRAINED_MODEL_ARCHIVE_MAP,
AlbertForMaskedLM,
ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
T5WithLMHeadModel,
T5_PRETRAINED_MODEL_ARCHIVE_MAP,
) = (
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
)
logging.basicConfig(level=logging.INFO)
MODEL_CLASSES = {
"bert": (
BertConfig,
TFBertForPreTraining,
BertForPreTraining,
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"bert-large-uncased-whole-word-masking-finetuned-squad": (
BertConfig,
TFBertForQuestionAnswering,
BertForQuestionAnswering,
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"bert-large-cased-whole-word-masking-finetuned-squad": (
BertConfig,
TFBertForQuestionAnswering,
BertForQuestionAnswering,
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"bert-base-cased-finetuned-mrpc": (
BertConfig,
TFBertForSequenceClassification,
BertForSequenceClassification,
BERT_PRETRAINED_MODEL_ARCHIVE_MAP,
BERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"gpt2": (
GPT2Config,
TFGPT2LMHeadModel,
GPT2LMHeadModel,
GPT2_PRETRAINED_MODEL_ARCHIVE_MAP,
GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"xlnet": (
XLNetConfig,
TFXLNetLMHeadModel,
XLNetLMHeadModel,
XLNET_PRETRAINED_MODEL_ARCHIVE_MAP,
XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"xlm": (
XLMConfig,
TFXLMWithLMHeadModel,
XLMWithLMHeadModel,
XLM_PRETRAINED_MODEL_ARCHIVE_MAP,
XLM_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"xlm-roberta": (
XLMRobertaConfig,
TFXLMRobertaForMaskedLM,
XLMRobertaForMaskedLM,
XLM_ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"transfo-xl": (
TransfoXLConfig,
TFTransfoXLLMHeadModel,
TransfoXLLMHeadModel,
TRANSFO_XL_PRETRAINED_MODEL_ARCHIVE_MAP,
TRANSFO_XL_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"openai-gpt": (
OpenAIGPTConfig,
TFOpenAIGPTLMHeadModel,
OpenAIGPTLMHeadModel,
OPENAI_GPT_PRETRAINED_MODEL_ARCHIVE_MAP,
OPENAI_GPT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"roberta": (
RobertaConfig,
TFRobertaForMaskedLM,
RobertaForMaskedLM,
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"roberta-large-mnli": (
RobertaConfig,
TFRobertaForSequenceClassification,
RobertaForSequenceClassification,
ROBERTA_PRETRAINED_MODEL_ARCHIVE_MAP,
ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"camembert": (
CamembertConfig,
TFCamembertForMaskedLM,
CamembertForMaskedLM,
CAMEMBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"distilbert": (
DistilBertConfig,
TFDistilBertForMaskedLM,
DistilBertForMaskedLM,
DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"distilbert-base-distilled-squad": (
DistilBertConfig,
TFDistilBertForQuestionAnswering,
DistilBertForQuestionAnswering,
DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"ctrl": (
CTRLConfig,
TFCTRLLMHeadModel,
CTRLLMHeadModel,
CTRL_PRETRAINED_MODEL_ARCHIVE_MAP,
CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"albert": (
AlbertConfig,
TFAlbertForMaskedLM,
AlbertForMaskedLM,
ALBERT_PRETRAINED_MODEL_ARCHIVE_MAP,
ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
"t5": (
T5Config,
TFT5WithLMHeadModel,
T5WithLMHeadModel,
T5_PRETRAINED_MODEL_ARCHIVE_MAP,
T5_PRETRAINED_CONFIG_ARCHIVE_MAP,
),
}
def convert_pt_checkpoint_to_tf(
model_type, pytorch_checkpoint_path, config_file, tf_dump_path, compare_with_pt_model=False, use_cached_models=True
):
if model_type not in MODEL_CLASSES:
raise ValueError("Unrecognized model type, should be one of {}.".format(list(MODEL_CLASSES.keys())))
config_class, model_class, pt_model_class, aws_model_maps, aws_config_map = MODEL_CLASSES[model_type]
# Initialise TF model
if config_file in aws_config_map:
config_file = cached_path(aws_config_map[config_file], force_download=not use_cached_models)
config = config_class.from_json_file(config_file)
config.output_hidden_states = True
config.output_attentions = True
print("Building TensorFlow model from configuration: {}".format(str(config)))
tf_model = model_class(config)
# Load weights from tf checkpoint
if pytorch_checkpoint_path in aws_model_maps:
pytorch_checkpoint_path = cached_path(
aws_model_maps[pytorch_checkpoint_path], force_download=not use_cached_models
)
# Load PyTorch checkpoint in tf2 model:
tf_model = load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path)
if compare_with_pt_model:
tfo = tf_model(tf_model.dummy_inputs, training=False) # build the network
state_dict = torch.load(pytorch_checkpoint_path, map_location="cpu")
pt_model = pt_model_class.from_pretrained(
pretrained_model_name_or_path=None, config=config, state_dict=state_dict
)
with torch.no_grad():
pto = pt_model(**pt_model.dummy_inputs)
np_pt = pto[0].numpy()
np_tf = tfo[0].numpy()
diff = np.amax(np.abs(np_pt - np_tf))
print("Max absolute difference between models outputs {}".format(diff))
assert diff <= 2e-2, "Error, model absolute difference is >2e-2: {}".format(diff)
# Save pytorch-model
print("Save TensorFlow model to {}".format(tf_dump_path))
tf_model.save_weights(tf_dump_path, save_format="h5")
def convert_all_pt_checkpoints_to_tf(
args_model_type,
tf_dump_path,
model_shortcut_names_or_path=None,
config_shortcut_names_or_path=None,
compare_with_pt_model=False,
use_cached_models=False,
remove_cached_files=False,
only_convert_finetuned_models=False,
):
assert os.path.isdir(args.tf_dump_path), "--tf_dump_path should be a directory"
if args_model_type is None:
model_types = list(MODEL_CLASSES.keys())
else:
model_types = [args_model_type]
for j, model_type in enumerate(model_types, start=1):
print("=" * 100)
print(" Converting model type {}/{}: {}".format(j, len(model_types), model_type))
print("=" * 100)
if model_type not in MODEL_CLASSES:
raise ValueError(
"Unrecognized model type {}, should be one of {}.".format(model_type, list(MODEL_CLASSES.keys()))
)
config_class, model_class, pt_model_class, aws_model_maps, aws_config_map = MODEL_CLASSES[model_type]
if model_shortcut_names_or_path is None:
model_shortcut_names_or_path = list(aws_model_maps.keys())
if config_shortcut_names_or_path is None:
config_shortcut_names_or_path = model_shortcut_names_or_path
for i, (model_shortcut_name, config_shortcut_name) in enumerate(
zip(model_shortcut_names_or_path, config_shortcut_names_or_path), start=1
):
print("-" * 100)
if "-squad" in model_shortcut_name or "-mrpc" in model_shortcut_name or "-mnli" in model_shortcut_name:
if not only_convert_finetuned_models:
print(" Skipping finetuned checkpoint {}".format(model_shortcut_name))
continue
model_type = model_shortcut_name
elif only_convert_finetuned_models:
print(" Skipping not finetuned checkpoint {}".format(model_shortcut_name))
continue
print(
" Converting checkpoint {}/{}: {} - model_type {}".format(
i, len(aws_config_map), model_shortcut_name, model_type
)
)
print("-" * 100)
if config_shortcut_name in aws_config_map:
config_file = cached_path(aws_config_map[config_shortcut_name], force_download=not use_cached_models)
else:
config_file = cached_path(config_shortcut_name, force_download=not use_cached_models)
if model_shortcut_name in aws_model_maps:
model_file = cached_path(aws_model_maps[model_shortcut_name], force_download=not use_cached_models)
else:
model_file = cached_path(model_shortcut_name, force_download=not use_cached_models)
if os.path.isfile(model_shortcut_name):
model_shortcut_name = "converted_model"
convert_pt_checkpoint_to_tf(
model_type=model_type,
pytorch_checkpoint_path=model_file,
config_file=config_file,
tf_dump_path=os.path.join(tf_dump_path, model_shortcut_name + "-tf_model.h5"),
compare_with_pt_model=compare_with_pt_model,
)
if remove_cached_files:
os.remove(config_file)
os.remove(model_file)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--tf_dump_path", default=None, type=str, required=True, help="Path to the output Tensorflow dump file."
)
parser.add_argument(
"--model_type",
default=None,
type=str,
help="Model type selected in the list of {}. If not given, will download and convert all the models from AWS.".format(
list(MODEL_CLASSES.keys())
),
)
parser.add_argument(
"--pytorch_checkpoint_path",
default=None,
type=str,
help="Path to the PyTorch checkpoint path or shortcut name to download from AWS. "
"If not given, will download and convert all the checkpoints from AWS.",
)
parser.add_argument(
"--config_file",
default=None,
type=str,
help="The config json file corresponding to the pre-trained model. \n"
"This specifies the model architecture. If not given and "
"--pytorch_checkpoint_path is not given or is a shortcut name"
"use the configuration associated to the shortcut name on the AWS",
)
parser.add_argument(
"--compare_with_pt_model", action="store_true", help="Compare Tensorflow and PyTorch model predictions."
)
parser.add_argument(
"--use_cached_models",
action="store_true",
help="Use cached models if possible instead of updating to latest checkpoint versions.",
)
parser.add_argument(
"--remove_cached_files",
action="store_true",
help="Remove pytorch models after conversion (save memory when converting in batches).",
)
parser.add_argument("--only_convert_finetuned_models", action="store_true", help="Only convert finetuned models.")
args = parser.parse_args()
# if args.pytorch_checkpoint_path is not None:
# convert_pt_checkpoint_to_tf(args.model_type.lower(),
# args.pytorch_checkpoint_path,
# args.config_file if args.config_file is not None else args.pytorch_checkpoint_path,
# args.tf_dump_path,
# compare_with_pt_model=args.compare_with_pt_model,
# use_cached_models=args.use_cached_models)
# else:
convert_all_pt_checkpoints_to_tf(
args.model_type.lower() if args.model_type is not None else None,
args.tf_dump_path,
model_shortcut_names_or_path=[args.pytorch_checkpoint_path]
if args.pytorch_checkpoint_path is not None
else None,
config_shortcut_names_or_path=[args.config_file] if args.config_file is not None else None,
compare_with_pt_model=args.compare_with_pt_model,
use_cached_models=args.use_cached_models,
remove_cached_files=args.remove_cached_files,
only_convert_finetuned_models=args.only_convert_finetuned_models,
)
xtune/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
+172
View File
@@ -0,0 +1,172 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert RoBERTa checkpoint."""
import argparse
import logging
import pathlib
import fairseq
import torch
from fairseq.models.roberta import RobertaModel as FairseqRobertaModel
from fairseq.modules import TransformerSentenceEncoderLayer
from packaging import version
from transformers.modeling_bert import BertIntermediate, BertLayer, BertOutput, BertSelfAttention, BertSelfOutput
from transformers.modeling_roberta import RobertaConfig, RobertaForMaskedLM, RobertaForSequenceClassification
if version.parse(fairseq.__version__) < version.parse("0.9.0"):
raise Exception("requires fairseq >= 0.9.0")
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
SAMPLE_TEXT = "Hello world! cécé herlolip"
def convert_roberta_checkpoint_to_pytorch(
roberta_checkpoint_path: str, pytorch_dump_folder_path: str, classification_head: bool
):
"""
Copy/paste/tweak roberta's weights to our BERT structure.
"""
roberta = FairseqRobertaModel.from_pretrained(roberta_checkpoint_path)
roberta.eval() # disable dropout
roberta_sent_encoder = roberta.model.decoder.sentence_encoder
config = RobertaConfig(
vocab_size=roberta_sent_encoder.embed_tokens.num_embeddings,
hidden_size=roberta.args.encoder_embed_dim,
num_hidden_layers=roberta.args.encoder_layers,
num_attention_heads=roberta.args.encoder_attention_heads,
intermediate_size=roberta.args.encoder_ffn_embed_dim,
max_position_embeddings=514,
type_vocab_size=1,
layer_norm_eps=1e-5, # PyTorch default used in fairseq
)
if classification_head:
config.num_labels = roberta.args.num_classes
print("Our BERT config:", config)
model = RobertaForSequenceClassification(config) if classification_head else RobertaForMaskedLM(config)
model.eval()
# Now let's copy all the weights.
# Embeddings
model.roberta.embeddings.word_embeddings.weight = roberta_sent_encoder.embed_tokens.weight
model.roberta.embeddings.position_embeddings.weight = roberta_sent_encoder.embed_positions.weight
model.roberta.embeddings.token_type_embeddings.weight.data = torch.zeros_like(
model.roberta.embeddings.token_type_embeddings.weight
) # just zero them out b/c RoBERTa doesn't use them.
model.roberta.embeddings.LayerNorm.weight = roberta_sent_encoder.emb_layer_norm.weight
model.roberta.embeddings.LayerNorm.bias = roberta_sent_encoder.emb_layer_norm.bias
for i in range(config.num_hidden_layers):
# Encoder: start of layer
layer: BertLayer = model.roberta.encoder.layer[i]
roberta_layer: TransformerSentenceEncoderLayer = roberta_sent_encoder.layers[i]
# self attention
self_attn: BertSelfAttention = layer.attention.self
assert (
roberta_layer.self_attn.k_proj.weight.data.shape
== roberta_layer.self_attn.q_proj.weight.data.shape
== roberta_layer.self_attn.v_proj.weight.data.shape
== torch.Size((config.hidden_size, config.hidden_size))
)
self_attn.query.weight.data = roberta_layer.self_attn.q_proj.weight
self_attn.query.bias.data = roberta_layer.self_attn.q_proj.bias
self_attn.key.weight.data = roberta_layer.self_attn.k_proj.weight
self_attn.key.bias.data = roberta_layer.self_attn.k_proj.bias
self_attn.value.weight.data = roberta_layer.self_attn.v_proj.weight
self_attn.value.bias.data = roberta_layer.self_attn.v_proj.bias
# self-attention output
self_output: BertSelfOutput = layer.attention.output
assert self_output.dense.weight.shape == roberta_layer.self_attn.out_proj.weight.shape
self_output.dense.weight = roberta_layer.self_attn.out_proj.weight
self_output.dense.bias = roberta_layer.self_attn.out_proj.bias
self_output.LayerNorm.weight = roberta_layer.self_attn_layer_norm.weight
self_output.LayerNorm.bias = roberta_layer.self_attn_layer_norm.bias
# intermediate
intermediate: BertIntermediate = layer.intermediate
assert intermediate.dense.weight.shape == roberta_layer.fc1.weight.shape
intermediate.dense.weight = roberta_layer.fc1.weight
intermediate.dense.bias = roberta_layer.fc1.bias
# output
bert_output: BertOutput = layer.output
assert bert_output.dense.weight.shape == roberta_layer.fc2.weight.shape
bert_output.dense.weight = roberta_layer.fc2.weight
bert_output.dense.bias = roberta_layer.fc2.bias
bert_output.LayerNorm.weight = roberta_layer.final_layer_norm.weight
bert_output.LayerNorm.bias = roberta_layer.final_layer_norm.bias
# end of layer
if classification_head:
model.classifier.dense.weight = roberta.model.classification_heads["mnli"].dense.weight
model.classifier.dense.bias = roberta.model.classification_heads["mnli"].dense.bias
model.classifier.out_proj.weight = roberta.model.classification_heads["mnli"].out_proj.weight
model.classifier.out_proj.bias = roberta.model.classification_heads["mnli"].out_proj.bias
else:
# LM Head
model.lm_head.dense.weight = roberta.model.decoder.lm_head.dense.weight
model.lm_head.dense.bias = roberta.model.decoder.lm_head.dense.bias
model.lm_head.layer_norm.weight = roberta.model.decoder.lm_head.layer_norm.weight
model.lm_head.layer_norm.bias = roberta.model.decoder.lm_head.layer_norm.bias
model.lm_head.decoder.weight = roberta.model.decoder.lm_head.weight
model.lm_head.decoder.bias = roberta.model.decoder.lm_head.bias
# Let's check that we get the same results.
input_ids: torch.Tensor = roberta.encode(SAMPLE_TEXT).unsqueeze(0) # batch of size 1
our_output = model(input_ids)[0]
if classification_head:
their_output = roberta.model.classification_heads["mnli"](roberta.extract_features(input_ids))
else:
their_output = roberta.model(input_ids)[0]
print(our_output.shape, their_output.shape, our_output[0][0], their_output[0][0])
max_absolute_diff = torch.max(torch.abs(our_output - their_output)).item()
print(f"max_absolute_diff = {max_absolute_diff}") # ~ 1e-7
success = torch.allclose(our_output, their_output, atol=1e-3)
print("Do both models output the same tensors?", "🔥" if success else "💩")
if not success:
raise Exception("Something went wRoNg")
pathlib.Path(pytorch_dump_folder_path).mkdir(parents=True, exist_ok=True)
print(f"Saving model to {pytorch_dump_folder_path}")
model.save_pretrained(pytorch_dump_folder_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--roberta_checkpoint_path", default=None, type=str, required=True, help="Path the official PyTorch dump."
)
parser.add_argument(
"--pytorch_dump_folder_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
)
parser.add_argument(
"--classification_head", action="store_true", help="Whether to convert a final classification head."
)
args = parser.parse_args()
convert_roberta_checkpoint_to_pytorch(
args.roberta_checkpoint_path, args.pytorch_dump_folder_path, args.classification_head
)
xtune/src/transformers/convert_t5_original_tf_checkpoint_to_pytorch.py
+61
View File
@@ -0,0 +1,61 @@
# coding=utf-8
# Copyright 2018 The T5 authors and HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert T5 checkpoint."""
import argparse
import logging
import torch
from transformers import T5Config, T5Model, load_tf_weights_in_t5
logging.basicConfig(level=logging.INFO)
def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, config_file, pytorch_dump_path):
# Initialise PyTorch model
config = T5Config.from_json_file(config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = T5Model(config)
# Load weights from tf checkpoint
load_tf_weights_in_t5(model, config, tf_checkpoint_path)
# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path."
)
parser.add_argument(
"--config_file",
default=None,
type=str,
required=True,
help="The config json file corresponding to the pre-trained T5 model. \n"
"This specifies the model architecture.",
)
parser.add_argument(
"--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
)
args = parser.parse_args()
convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.config_file, args.pytorch_dump_path)
xtune/src/transformers/convert_transfo_xl_original_tf_checkpoint_to_pytorch.py
+125
View File
@@ -0,0 +1,125 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert Transformer XL checkpoint and datasets."""
import argparse
import logging
import os
import pickle
import sys
import torch
import transformers.tokenization_transfo_xl as data_utils
from transformers import (
CONFIG_NAME,
WEIGHTS_NAME,
TransfoXLConfig,
TransfoXLLMHeadModel,
load_tf_weights_in_transfo_xl,
)
from transformers.tokenization_transfo_xl import CORPUS_NAME, VOCAB_FILES_NAMES
logging.basicConfig(level=logging.INFO)
# We do this to be able to load python 2 datasets pickles
# See e.g. https://stackoverflow.com/questions/2121874/python-pickling-after-changing-a-modules-directory/2121918#2121918
data_utils.Vocab = data_utils.TransfoXLTokenizer
data_utils.Corpus = data_utils.TransfoXLCorpus
sys.modules["data_utils"] = data_utils
sys.modules["vocabulary"] = data_utils
def convert_transfo_xl_checkpoint_to_pytorch(
tf_checkpoint_path, transfo_xl_config_file, pytorch_dump_folder_path, transfo_xl_dataset_file
):
if transfo_xl_dataset_file:
# Convert a pre-processed corpus (see original TensorFlow repo)
with open(transfo_xl_dataset_file, "rb") as fp:
corpus = pickle.load(fp, encoding="latin1")
# Save vocabulary and dataset cache as Dictionaries (should be better than pickles for the long-term)
pytorch_vocab_dump_path = pytorch_dump_folder_path + "/" + VOCAB_FILES_NAMES["pretrained_vocab_file"]
print("Save vocabulary to {}".format(pytorch_vocab_dump_path))
corpus_vocab_dict = corpus.vocab.__dict__
torch.save(corpus_vocab_dict, pytorch_vocab_dump_path)
corpus_dict_no_vocab = corpus.__dict__
corpus_dict_no_vocab.pop("vocab", None)
pytorch_dataset_dump_path = pytorch_dump_folder_path + "/" + CORPUS_NAME
print("Save dataset to {}".format(pytorch_dataset_dump_path))
torch.save(corpus_dict_no_vocab, pytorch_dataset_dump_path)
if tf_checkpoint_path:
# Convert a pre-trained TensorFlow model
config_path = os.path.abspath(transfo_xl_config_file)
tf_path = os.path.abspath(tf_checkpoint_path)
print("Converting Transformer XL checkpoint from {} with config at {}".format(tf_path, config_path))
# Initialise PyTorch model
if transfo_xl_config_file == "":
config = TransfoXLConfig()
else:
config = TransfoXLConfig.from_json_file(transfo_xl_config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = TransfoXLLMHeadModel(config)
model = load_tf_weights_in_transfo_xl(model, config, tf_path)
# Save pytorch-model
pytorch_weights_dump_path = os.path.join(pytorch_dump_folder_path, WEIGHTS_NAME)
pytorch_config_dump_path = os.path.join(pytorch_dump_folder_path, CONFIG_NAME)
print("Save PyTorch model to {}".format(os.path.abspath(pytorch_weights_dump_path)))
torch.save(model.state_dict(), pytorch_weights_dump_path)
print("Save configuration file to {}".format(os.path.abspath(pytorch_config_dump_path)))
with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
f.write(config.to_json_string())
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"--pytorch_dump_folder_path",
default=None,
type=str,
required=True,
help="Path to the folder to store the PyTorch model or dataset/vocab.",
)
parser.add_argument(
"--tf_checkpoint_path",
default="",
type=str,
help="An optional path to a TensorFlow checkpoint path to be converted.",
)
parser.add_argument(
"--transfo_xl_config_file",
default="",
type=str,
help="An optional config json file corresponding to the pre-trained BERT model. \n"
"This specifies the model architecture.",
)
parser.add_argument(
"--transfo_xl_dataset_file",
default="",
type=str,
help="An optional dataset file to be converted in a vocabulary.",
)
args = parser.parse_args()
convert_transfo_xl_checkpoint_to_pytorch(
args.tf_checkpoint_path,
args.transfo_xl_config_file,
args.pytorch_dump_folder_path,
args.transfo_xl_dataset_file,
)
xtune/src/transformers/convert_xlm_original_pytorch_checkpoint_to_pytorch.py
+79
View File
@@ -0,0 +1,79 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert OpenAI GPT checkpoint."""
import argparse
import json
import logging
import numpy
import torch
from transformers import CONFIG_NAME, WEIGHTS_NAME
from transformers.tokenization_xlm import VOCAB_FILES_NAMES
logging.basicConfig(level=logging.INFO)
def convert_xlm_checkpoint_to_pytorch(xlm_checkpoint_path, pytorch_dump_folder_path):
# Load checkpoint
chkpt = torch.load(xlm_checkpoint_path, map_location="cpu")
state_dict = chkpt["model"]
# We have the base model one level deeper than the original XLM repository
two_levels_state_dict = {}
for k, v in state_dict.items():
if "pred_layer" in k:
two_levels_state_dict[k] = v
else:
two_levels_state_dict["transformer." + k] = v
config = chkpt["params"]
config = dict((n, v) for n, v in config.items() if not isinstance(v, (torch.FloatTensor, numpy.ndarray)))
vocab = chkpt["dico_word2id"]
vocab = dict((s + "</w>" if s.find("@@") == -1 and i > 13 else s.replace("@@", ""), i) for s, i in vocab.items())
# Save pytorch-model
pytorch_weights_dump_path = pytorch_dump_folder_path + "/" + WEIGHTS_NAME
pytorch_config_dump_path = pytorch_dump_folder_path + "/" + CONFIG_NAME
pytorch_vocab_dump_path = pytorch_dump_folder_path + "/" + VOCAB_FILES_NAMES["vocab_file"]
print("Save PyTorch model to {}".format(pytorch_weights_dump_path))
torch.save(two_levels_state_dict, pytorch_weights_dump_path)
print("Save configuration file to {}".format(pytorch_config_dump_path))
with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
f.write(json.dumps(config, indent=2) + "\n")
print("Save vocab file to {}".format(pytorch_config_dump_path))
with open(pytorch_vocab_dump_path, "w", encoding="utf-8") as f:
f.write(json.dumps(vocab, indent=2) + "\n")
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--xlm_checkpoint_path", default=None, type=str, required=True, help="Path the official PyTorch dump."
)
parser.add_argument(
"--pytorch_dump_folder_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
)
args = parser.parse_args()
convert_xlm_checkpoint_to_pytorch(args.xlm_checkpoint_path, args.pytorch_dump_folder_path)
xtune/src/transformers/convert_xlnet_original_tf_checkpoint_to_pytorch.py
+114
View File
@@ -0,0 +1,114 @@
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert BERT checkpoint."""
import argparse
import logging
import os
import torch
from transformers import (
CONFIG_NAME,
WEIGHTS_NAME,
XLNetConfig,
XLNetForQuestionAnswering,
XLNetForSequenceClassification,
XLNetLMHeadModel,
load_tf_weights_in_xlnet,
)
GLUE_TASKS_NUM_LABELS = {
"cola": 2,
"mnli": 3,
"mrpc": 2,
"sst-2": 2,
"sts-b": 1,
"qqp": 2,
"qnli": 2,
"rte": 2,
"wnli": 2,
}
logging.basicConfig(level=logging.INFO)
def convert_xlnet_checkpoint_to_pytorch(
tf_checkpoint_path, bert_config_file, pytorch_dump_folder_path, finetuning_task=None
):
# Initialise PyTorch model
config = XLNetConfig.from_json_file(bert_config_file)
finetuning_task = finetuning_task.lower() if finetuning_task is not None else ""
if finetuning_task in GLUE_TASKS_NUM_LABELS:
print("Building PyTorch XLNetForSequenceClassification model from configuration: {}".format(str(config)))
config.finetuning_task = finetuning_task
config.num_labels = GLUE_TASKS_NUM_LABELS[finetuning_task]
model = XLNetForSequenceClassification(config)
elif "squad" in finetuning_task:
config.finetuning_task = finetuning_task
model = XLNetForQuestionAnswering(config)
else:
model = XLNetLMHeadModel(config)
# Load weights from tf checkpoint
load_tf_weights_in_xlnet(model, config, tf_checkpoint_path)
# Save pytorch-model
pytorch_weights_dump_path = os.path.join(pytorch_dump_folder_path, WEIGHTS_NAME)
pytorch_config_dump_path = os.path.join(pytorch_dump_folder_path, CONFIG_NAME)
print("Save PyTorch model to {}".format(os.path.abspath(pytorch_weights_dump_path)))
torch.save(model.state_dict(), pytorch_weights_dump_path)
print("Save configuration file to {}".format(os.path.abspath(pytorch_config_dump_path)))
with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
f.write(config.to_json_string())
if __name__ == "__main__":
parser = argparse.ArgumentParser()
# Required parameters
parser.add_argument(
"--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path."
)
parser.add_argument(
"--xlnet_config_file",
default=None,
type=str,
required=True,
help="The config json file corresponding to the pre-trained XLNet model. \n"
"This specifies the model architecture.",
)
parser.add_argument(
"--pytorch_dump_folder_path",
default=None,
type=str,
required=True,
help="Path to the folder to store the PyTorch model or dataset/vocab.",
)
parser.add_argument(
"--finetuning_task",
default=None,
type=str,
help="Name of a task on which the XLNet TensorFloaw model was fine-tuned",
)
args = parser.parse_args()
print(args)
convert_xlnet_checkpoint_to_pytorch(
args.tf_checkpoint_path, args.xlnet_config_file, args.pytorch_dump_folder_path, args.finetuning_task
)
xtune/src/transformers/data/__init__.py
+39
View File
@@ -0,0 +1,39 @@
# flake8: noqa
# There's no way to ignore "F401 '...' imported but unused" warnings in this
# module, but to preserve other warnings. So, don't check this module at all.
from .metrics import is_sklearn_available
from .processors import (
DataProcessor,
InputExample,
InputFeatures,
SingleSentenceClassificationProcessor,
SquadExample,
SquadFeatures,
SquadV1Processor,
SquadV2Processor,
glue_convert_examples_to_features,
glue_output_modes,
glue_processors,
glue_tasks_num_labels,
xglue_convert_examples_to_features,
xglue_convert_examples_to_vat_features,
xglue_output_modes,
xglue_processors,
xglue_tasks_num_labels,
xtreme_convert_examples_to_features,
xtreme_output_modes,
xtreme_processors,
xtreme_tasks_num_labels,
squad_convert_examples_to_features,
xnli_output_modes,
xnli_processors,
xnli_tasks_num_labels,
)
if is_sklearn_available():
from .metrics import glue_compute_metrics, xnli_compute_metrics, xglue_compute_metrics, xtreme_compute_metrics
xtune/src/transformers/data/metrics/__init__.py
+147
View File
@@ -0,0 +1,147 @@
# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
try:
from scipy.stats import pearsonr, spearmanr
from sklearn.metrics import matthews_corrcoef, f1_score, average_precision_score, ndcg_score, roc_auc_score
import numpy as np
_has_sklearn = True
except (AttributeError, ImportError):
_has_sklearn = False
def is_sklearn_available():
return _has_sklearn
if _has_sklearn:
def simple_accuracy(preds, labels):
return (preds == labels).mean()
def simple_ndcg(preds, labels, guids):
ndcgs = []
query2content = {}
for guid, pred, label in zip(guids, preds, labels):
query = guid.split("_")[0]
if not query in query2content:
query2content[query] = [[int(pred)], [int(label)]]
else:
query2content[query][0].append(int(pred))
query2content[query][1].append(int(label))
for key in query2content.keys():
if len(query2content[key][1]) < 2 or len(query2content[key][0]) < 2:
continue
ndcgs.append(ndcg_score(np.asarray([query2content[key][1]]), np.asarray([query2content[key][0]])))
return {"ndcg" : np.array(ndcgs).mean()}
def acc_and_f1(preds, labels):
acc = simple_accuracy(preds, labels)
f1 = f1_score(y_true=labels, y_pred=preds)
return {
"acc": acc,
"f1": f1,
"acc_and_f1": (acc + f1) / 2,
}
def acc_and_auc(preds, labels): # auc of pr curve is equal to average precision
acc = simple_accuracy(preds, labels)
auc = average_precision_score(labels, preds)
return {
"acc": acc,
"auc": auc,
"acc_and_auc": (acc + auc) / 2,
}
def acc_and_roc_auc(preds, labels): # auc of pr curve is equal to average precision
acc = simple_accuracy(preds, labels)
roc_auc = roc_auc_score(labels, preds)
return {
"acc": acc,
"roc_auc": roc_auc,
"acc_and_roc_auc": (acc + roc_auc) / 2,
}
def pearson_and_spearman(preds, labels):
pearson_corr = pearsonr(preds, labels)[0]
spearman_corr = spearmanr(preds, labels)[0]
return {
"pearson": pearson_corr,
"spearmanr": spearman_corr,
"corr": (pearson_corr + spearman_corr) / 2,
}
def xglue_compute_metrics(task_name, preds, labels, guids):
assert len(preds) == len(labels)
if task_name == "xnli":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "pawsx":
return acc_and_auc(preds, labels)
elif task_name == "qam":
return acc_and_auc(preds, labels)
elif task_name == "ads":
return acc_and_roc_auc(preds, labels)
elif task_name == "rel":
return simple_ndcg(preds, labels, guids)
elif task_name == "news":
return {"acc": simple_accuracy(preds, labels)}
else:
raise KeyError(task_name)
def xtreme_compute_metrics(task_name, preds, labels, guids):
assert len(preds) == len(labels)
if task_name == "xnli":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "pawsx":
return acc_and_auc(preds, labels)
else:
raise KeyError(task_name)
def glue_compute_metrics(task_name, preds, labels):
assert len(preds) == len(labels)
if task_name == "cola":
return {"mcc": matthews_corrcoef(labels, preds)}
elif task_name == "sst-2":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "mrpc":
return acc_and_f1(preds, labels)
elif task_name == "sts-b":
return pearson_and_spearman(preds, labels)
elif task_name == "qqp":
return acc_and_f1(preds, labels)
elif task_name == "mnli":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "mnli-mm":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "qnli":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "rte":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "wnli":
return {"acc": simple_accuracy(preds, labels)}
elif task_name == "hans":
return {"acc": simple_accuracy(preds, labels)}
else:
raise KeyError(task_name)
def xnli_compute_metrics(task_name, preds, labels):
assert len(preds) == len(labels)
if task_name == "xnli":
return {"acc": simple_accuracy(preds, labels)}
else:
raise KeyError(task_name)

Some files were not shown because too many files have changed in this diff Show More