chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,102 @@
|
||||
[[Back]](..)
|
||||
|
||||
# S2T Example: ST on CoVoST
|
||||
We replicate the experiments in
|
||||
[CoVoST 2 and Massively Multilingual Speech-to-Text Translation (Wang et al., 2020)](https://arxiv.org/abs/2007.10310).
|
||||
|
||||
## Data Preparation
|
||||
[Download](https://commonvoice.mozilla.org/en/datasets) and unpack Common Voice v4 to a path
|
||||
`${COVOST_ROOT}/${SOURCE_LANG_ID}`, then preprocess it with
|
||||
```bash
|
||||
# additional Python packages for S2T data processing/model training
|
||||
pip install pandas torchaudio sentencepiece
|
||||
|
||||
# En ASR
|
||||
python examples/speech_to_text/prep_covost_data.py \
|
||||
--data-root ${COVOST_ROOT} --vocab-type char --src-lang en
|
||||
# ST
|
||||
python examples/speech_to_text/prep_covost_data.py \
|
||||
--data-root ${COVOST_ROOT} --vocab-type char \
|
||||
--src-lang fr --tgt-lang en
|
||||
```
|
||||
The generated files (manifest, features, vocabulary and data configuration) will be added to
|
||||
`${COVOST_ROOT}/${SOURCE_LANG_ID}`.
|
||||
|
||||
Download our vocabulary files if you want to use our pre-trained models:
|
||||
- ASR: [En](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_asr_vocab_char.zip)
|
||||
- ST: [Fr-En](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_fr_en_st_vocab_char.zip), [De-En](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_de_en_st_vocab_char.zip), [Es-En](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_es_en_st_vocab_char.zip), [Ca-En](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_ca_en_st_vocab_char.zip), [En-De](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_de_st_vocab_char.zip), [En-Ca](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_ca_st_vocab_char.zip), [En-Fa](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_fa_st_vocab_char.zip), [En-Et](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_et_st_vocab_char.zip)
|
||||
|
||||
## ASR
|
||||
#### Training
|
||||
We train an En ASR model for encoder pre-training of all ST models:
|
||||
```bash
|
||||
fairseq-train ${COVOST_ROOT}/en \
|
||||
--config-yaml config_asr_en.yaml --train-subset train_asr_en --valid-subset dev_asr_en \
|
||||
--save-dir ${ASR_SAVE_DIR} --num-workers 4 --max-tokens 50000 --max-update 60000 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \
|
||||
--report-accuracy --arch s2t_transformer_s --dropout 0.15 --optimizer adam --lr 2e-3 \
|
||||
--lr-scheduler inverse_sqrt --warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8
|
||||
```
|
||||
where `ASR_SAVE_DIR` is the checkpoint root path. We set `--update-freq 8` to simulate 8 GPUs with 1 GPU.
|
||||
You may want to update it accordingly when using more than 1 GPU.
|
||||
|
||||
#### Inference & Evaluation
|
||||
```bash
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${ASR_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
fairseq-generate ${COVOST_ROOT}/en \
|
||||
--config-yaml config_asr_en.yaml --gen-subset test_asr_en --task speech_to_text \
|
||||
--path ${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME} --max-tokens 50000 --beam 5 \
|
||||
--scoring wer --wer-tokenizer 13a --wer-lowercase --wer-remove-punct
|
||||
```
|
||||
#### Results
|
||||
| --arch | Params | En | Model |
|
||||
|---|---|---|---|
|
||||
| s2t_transformer_s | 31M | 25.6 | [Download](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_asr_transformer_s.pt) |
|
||||
|
||||
## ST
|
||||
#### Training
|
||||
Fr-En as example:
|
||||
```bash
|
||||
fairseq-train ${COVOST_ROOT}/fr \
|
||||
--config-yaml config_st_fr_en.yaml --train-subset train_st_fr_en --valid-subset dev_st_fr_en \
|
||||
--save-dir ${ST_SAVE_DIR} --num-workers 4 --max-update 30000 --max-tokens 40000 \ # --max-tokens 50000 for en-*
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --report-accuracy \
|
||||
--arch s2t_transformer_s --encoder-freezing-updates 1000 --optimizer adam --lr 2e-3 \
|
||||
--lr-scheduler inverse_sqrt --warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8 \
|
||||
--load-pretrained-encoder-from ${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}
|
||||
```
|
||||
where `ST_SAVE_DIR` is the checkpoint root path. The ST encoder is pre-trained by En ASR for faster training and better
|
||||
performance: `--load-pretrained-encoder-from <ASR checkpoint path>`. We set `--update-freq 8` to simulate 8 GPUs with 1 GPU.
|
||||
You may want to update it accordingly when using more than 1 GPU.
|
||||
|
||||
#### Inference & Evaluation
|
||||
Average the last 10 checkpoints and evaluate on test split:
|
||||
```bash
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${ST_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${ST_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
fairseq-generate ${COVOST_ROOT}/fr \
|
||||
--config-yaml config_st_fr_en.yaml --gen-subset test_st_fr_en --task speech_to_text \
|
||||
--path ${ST_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 50000 --beam 5 --scoring sacrebleu
|
||||
```
|
||||
|
||||
## Interactive Decoding
|
||||
Launch the interactive console via
|
||||
```bash
|
||||
fairseq-interactive ${COVOST_ROOT}/fr --config-yaml config_st_fr_en.yaml \
|
||||
--task speech_to_text --path ${SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 50000 --beam 5
|
||||
```
|
||||
Type in WAV/FLAC/OGG audio paths (one per line) after the prompt.
|
||||
|
||||
#### Results
|
||||
| --arch | Params | Fr-En | De-En | Es-En | Ca-En | En-De | En-Ca | En-Fa | En-Et | Model |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| s2t_transformer_s | 31M | [27.2](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_fr_en_st_transformer_s.pt) | [17.7](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_de_en_st_transformer_s.pt) | [23.1](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_es_en_st_transformer_s.pt) | [19.3](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_ca_en_st_transformer_s.pt) | [16.1](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_de_st_transformer_s.pt) | [21.6](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_ca_st_transformer_s.pt) | [12.9](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_fa_st_transformer_s.pt) | [12.8](https://dl.fbaipublicfiles.com/fairseq/s2t/covost2_en_et_st_transformer_s.pt) | (<-Download) |
|
||||
|
||||
[[Back]](..)
|
||||
@@ -0,0 +1,69 @@
|
||||
[[Back]](..)
|
||||
|
||||
# S2T Example: Speech Recognition (ASR) on LibriSpeech
|
||||
[LibriSpeech](https://www.danielpovey.com/files/2015_icassp_librispeech.pdf) is a de-facto standard English ASR
|
||||
benchmark. We provide competitive
|
||||
vanilla [Transformer](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) baselines.
|
||||
|
||||
## Data preparation
|
||||
Download and preprocess LibriSpeech data with
|
||||
```bash
|
||||
# additional Python packages for S2T data processing/model training
|
||||
pip install pandas torchaudio sentencepiece
|
||||
|
||||
python examples/speech_to_text/prep_librispeech_data.py \
|
||||
--output-root ${LS_ROOT} --vocab-type unigram --vocab-size 10000
|
||||
```
|
||||
where `LS_ROOT` is the root path for downloaded data as well as generated files (manifest, features, vocabulary and
|
||||
data configuration).
|
||||
|
||||
[Download](https://dl.fbaipublicfiles.com/fairseq/s2t/librispeech_vocab_unigram10000.zip) our vocabulary files
|
||||
if you want to use our pre-trained models.
|
||||
|
||||
## Training
|
||||
```bash
|
||||
fairseq-train ${LS_ROOT} --save-dir ${SAVE_DIR} \
|
||||
--config-yaml config.yaml --train-subset train-clean-100,train-clean-360,train-other-500 --valid-subset dev-clean,dev-other \
|
||||
--num-workers 4 --max-tokens 40000 --max-update 300000 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --report-accuracy \
|
||||
--arch s2t_transformer_s --share-decoder-input-output-embed \
|
||||
--optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt --warmup-updates 10000 \
|
||||
--clip-norm 10.0 --seed 1 --update-freq 8
|
||||
```
|
||||
where `SAVE_DIR` is the checkpoint root path. Here we use `--arch s2t_transformer_s` (31M parameters) as example.
|
||||
For better performance, you may switch to `s2t_transformer_m` (71M, with `--lr 1e-3`) or `s2t_transformer_l`
|
||||
(268M, with `--lr 5e-4`). We set `--update-freq 8` to simulate 8 GPUs with 1 GPU. You may want to update it accordingly
|
||||
when using more than 1 GPU.
|
||||
|
||||
## Inference & Evaluation
|
||||
Average the last 10 checkpoints and evaluate on the 4 splits
|
||||
(`dev-clean`, `dev-other`, `test-clean` and `test-other`):
|
||||
```bash
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py --inputs ${SAVE_DIR} \
|
||||
--num-epoch-checkpoints 10 \
|
||||
--output "${SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
for SUBSET in dev-clean dev-other test-clean test-other; do
|
||||
fairseq-generate ${LS_ROOT} --config-yaml config.yaml --gen-subset ${SUBSET} \
|
||||
--task speech_to_text --path ${SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 50000 --beam 5 --scoring wer
|
||||
done
|
||||
```
|
||||
|
||||
## Interactive Decoding
|
||||
Launch the interactive console via
|
||||
```bash
|
||||
fairseq-interactive ${LS_ROOT} --config-yaml config.yaml --task speech_to_text \
|
||||
--path ${SAVE_DIR}/${CHECKPOINT_FILENAME} --max-tokens 50000 --beam 5
|
||||
```
|
||||
Type in WAV/FLAC/OGG audio paths (one per line) after the prompt.
|
||||
|
||||
## Results
|
||||
|
||||
| --arch | Params | dev-clean | dev-other | test-clean | test-other | Model |
|
||||
|---|---|---|---|---|---|---|
|
||||
| s2t_transformer_s | 30M | 3.8 | 8.9 | 4.4 | 9.0 | [Download](https://dl.fbaipublicfiles.com/fairseq/s2t/librispeech_transformer_s.pt) |
|
||||
| s2t_transformer_m | 71M | 3.2 | 8.0 | 3.4 | 7.9 | [Download](https://dl.fbaipublicfiles.com/fairseq/s2t/librispeech_transformer_m.pt) |
|
||||
| s2t_transformer_l | 268M | 3.0 | 7.5 | 3.2 | 7.5 | [Download](https://dl.fbaipublicfiles.com/fairseq/s2t/librispeech_transformer_l.pt) |
|
||||
|
||||
[[Back]](..)
|
||||
@@ -0,0 +1,201 @@
|
||||
[[Back]](..)
|
||||
|
||||
# S2T Example: Speech Translation (ST) on Multilingual TEDx
|
||||
|
||||
[Multilingual TEDx](https://arxiv.org/abs/2102.01757) is multilingual corpus for speech recognition and
|
||||
speech translation. The data is derived from TEDx talks in 8 source languages
|
||||
with translations to a subset of 5 target languages.
|
||||
|
||||
## Data Preparation
|
||||
[Download](http://openslr.org/100/) and unpack Multilingual TEDx data to a path
|
||||
`${MTEDX_ROOT}/${LANG_PAIR}`, then preprocess it with
|
||||
```bash
|
||||
# additional Python packages for S2T data processing/model training
|
||||
pip install pandas torchaudio soundfile sentencepiece
|
||||
|
||||
# Generate TSV manifests, features, vocabulary
|
||||
# and configuration for each language
|
||||
python examples/speech_to_text/prep_mtedx_data.py \
|
||||
--data-root ${MTEDX_ROOT} --task asr \
|
||||
--vocab-type unigram --vocab-size 1000
|
||||
python examples/speech_to_text/prep_mtedx_data.py \
|
||||
--data-root ${MTEDX_ROOT} --task st \
|
||||
--vocab-type unigram --vocab-size 1000
|
||||
|
||||
# Add vocabulary and configuration for joint data
|
||||
# (based on the manifests and features generated above)
|
||||
python examples/speech_to_text/prep_mtedx_data.py \
|
||||
--data-root ${MTEDX_ROOT} --task asr --joint \
|
||||
--vocab-type unigram --vocab-size 8000
|
||||
python examples/speech_to_text/prep_mtedx_data.py \
|
||||
--data-root ${MTEDX_ROOT} --task st --joint \
|
||||
--vocab-type unigram --vocab-size 8000
|
||||
```
|
||||
The generated files (manifest, features, vocabulary and data configuration) will be added to
|
||||
`${MTEDX_ROOT}/${LANG_PAIR}` (per-language data) and `MTEDX_ROOT` (joint data).
|
||||
|
||||
|
||||
## ASR
|
||||
#### Training
|
||||
Spanish as example:
|
||||
```bash
|
||||
fairseq-train ${MTEDX_ROOT}/es-es \
|
||||
--config-yaml config_asr.yaml --train-subset train_asr --valid-subset valid_asr \
|
||||
--save-dir ${ASR_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-epoch 200 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --report-accuracy \
|
||||
--arch s2t_transformer_xs --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --dropout 0.3 --label-smoothing 0.1 \
|
||||
--load-pretrained-encoder-from ${PRETRAINED_ENCODER} \
|
||||
--skip-invalid-size-inputs-valid-test \
|
||||
--keep-last-epochs 10 --update-freq 8 --patience 10
|
||||
```
|
||||
For joint model (using ASR data from all 8 languages):
|
||||
```bash
|
||||
fairseq-train ${MTEDX_ROOT} \
|
||||
--config-yaml config_asr.yaml \
|
||||
--train-subset train_es-es_asr,train_fr-fr_asr,train_pt-pt_asr,train_it-it_asr,train_ru-ru_asr,train_el-el_asr,train_ar-ar_asr,train_de-de_asr \
|
||||
--valid-subset valid_es-es_asr,valid_fr-fr_asr,valid_pt-pt_asr,valid_it-it_asr,valid_ru-ru_asr,valid_el-el_asr,valid_ar-ar_asr,valid_de-de_asr \
|
||||
--save-dir ${MULTILINGUAL_ASR_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-epoch 200 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --report-accuracy \
|
||||
--arch s2t_transformer_s --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --dropout 0.3 --label-smoothing 0.1 \
|
||||
--skip-invalid-size-inputs-valid-test \
|
||||
--keep-last-epochs 10 --update-freq 8 --patience 10 \
|
||||
--ignore-prefix-size 1
|
||||
```
|
||||
where `MULTILINGUAL_ASR_SAVE_DIR` is the checkpoint root path. We set `--update-freq 8` to simulate 8 GPUs
|
||||
with 1 GPU. You may want to update it accordingly when using more than 1 GPU.
|
||||
For multilingual models, we prepend target language ID token as target BOS, which should be excluded from
|
||||
the training loss via `--ignore-prefix-size 1`.
|
||||
|
||||
#### Inference & Evaluation
|
||||
```bash
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${ASR_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
|
||||
fairseq-generate ${MTEDX_ROOT}/es-es \
|
||||
--config-yaml config_asr.yaml --gen-subset test --task speech_to_text \
|
||||
--path ${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME} --max-tokens 50000 --beam 5 \
|
||||
--skip-invalid-size-inputs-valid-test \
|
||||
--scoring wer --wer-tokenizer 13a --wer-lowercase --wer-remove-punct --remove-bpe
|
||||
|
||||
# For models trained on joint data
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${MULTILINGUAL_ASR_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${MULTILINGUAL_ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
|
||||
for LANG in es fr pt it ru el ar de; do
|
||||
fairseq-generate ${MTEDX_ROOT} \
|
||||
--config-yaml config_asr.yaml --gen-subset test_${LANG}-${LANG}_asr --task speech_to_text \
|
||||
--prefix-size 1 --path ${MULTILINGUAL_ASR_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 40000 --beam 5 \
|
||||
--skip-invalid-size-inputs-valid-test \
|
||||
--scoring wer --wer-tokenizer 13a --wer-lowercase --wer-remove-punct --remove-bpe
|
||||
done
|
||||
```
|
||||
#### Results
|
||||
| Data | --arch | Params | Es | Fr | Pt | It | Ru | El | Ar | De |
|
||||
|--------------|--------------------|--------|------|------|------|------|------|-------|-------|-------|
|
||||
| Monolingual | s2t_transformer_xs | 10M | 46.4 | 45.6 | 54.8 | 48.0 | 74.7 | 109.5 | 104.4 | 111.1 |
|
||||
|
||||
|
||||
## ST
|
||||
#### Training
|
||||
Es-En as example:
|
||||
```bash
|
||||
fairseq-train ${MTEDX_ROOT}/es-en \
|
||||
--config-yaml config_st.yaml --train-subset train_st --valid-subset valid_st \
|
||||
--save-dir ${ST_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-epoch 200 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --report-accuracy \
|
||||
--arch s2t_transformer_xs --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --dropout 0.3 --label-smoothing 0.1 \
|
||||
--load-pretrained-encoder-from ${PRETRAINED_ENCODER} \
|
||||
--skip-invalid-size-inputs-valid-test \
|
||||
--keep-last-epochs 10 --update-freq 8 --patience 10
|
||||
```
|
||||
For multilingual model (all 12 directions):
|
||||
```bash
|
||||
fairseq-train ${MTEDX_ROOT} \
|
||||
--config-yaml config_st.yaml \
|
||||
--train-subset train_el-en_st,train_es-en_st,train_es-fr_st,train_es-it_st,train_es-pt_st,train_fr-en_st,train_fr-es_st,train_fr-pt_st,train_it-en_st,train_it-es_st,train_pt-en_st,train_pt-es_st,train_ru-en_st \
|
||||
--valid-subset valid_el-en_st,valid_es-en_st,valid_es-fr_st,valid_es-it_st,valid_es-pt_st,valid_fr-en_st,valid_fr-es_st,valid_fr-pt_st,valid_it-en_st,valid_it-es_st,valid_pt-en_st,valid_pt-es_st,valid_ru-en_st \
|
||||
--save-dir ${MULTILINGUAL_ST_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-epoch 200 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --report-accuracy \
|
||||
--arch s2t_transformer_s --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --dropout 0.3 --label-smoothing 0.1 \
|
||||
--skip-invalid-size-inputs-valid-test \
|
||||
--keep-last-epochs 10 --update-freq 8 --patience 10 \
|
||||
--ignore-prefix-size 1 \
|
||||
--load-pretrained-encoder-from ${PRETRAINED_ENCODER}
|
||||
```
|
||||
where `ST_SAVE_DIR` (`MULTILINGUAL_ST_SAVE_DIR`) is the checkpoint root path. The ST encoder is pre-trained by ASR
|
||||
for faster training and better performance: `--load-pretrained-encoder-from <(JOINT_)ASR checkpoint path>`. We set
|
||||
`--update-freq 8` to simulate 8 GPUs with 1 GPU. You may want to update it accordingly when using more than 1 GPU.
|
||||
For multilingual models, we prepend target language ID token as target BOS, which should be excluded from
|
||||
the training loss via `--ignore-prefix-size 1`.
|
||||
|
||||
#### Inference & Evaluation
|
||||
Average the last 10 checkpoints and evaluate on the `test` split:
|
||||
```bash
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${ST_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${ST_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
|
||||
fairseq-generate ${MTEDX_ROOT}/es-en \
|
||||
--config-yaml config_st.yaml --gen-subset test --task speech_to_text \
|
||||
--path ${ST_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 50000 --beam 5 --scoring sacrebleu --remove-bpe
|
||||
|
||||
# For multilingual models
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${MULTILINGUAL_ST_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${MULTILINGUAL_ST_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
|
||||
for LANGPAIR in es-en es-fr es-pt fr-en fr-es fr-pt pt-en pt-es it-en it-es ru-en el-en; do
|
||||
fairseq-generate ${MTEDX_ROOT} \
|
||||
--config-yaml config_st.yaml --gen-subset test_${LANGPAIR}_st --task speech_to_text \
|
||||
--prefix-size 1 --path ${MULTILINGUAL_ST_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 40000 --beam 5 \
|
||||
--skip-invalid-size-inputs-valid-test \
|
||||
--scoring sacrebleu --remove-bpe
|
||||
done
|
||||
```
|
||||
For multilingual models, we force decoding from the target language ID token (as BOS) via `--prefix-size 1`.
|
||||
|
||||
#### Results
|
||||
| Data | --arch | Params | Es-En | Es-Pt | Es-Fr | Fr-En | Fr-Es | Fr-Pt | Pt-En | Pt-Es | It-En | It-Es | Ru-En | El-En |
|
||||
|--------------|--------------------|-----|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
|
||||
| Bilingual | s2t_transformer_xs | 10M | 7.0 | 12.2 | 1.7 | 8.9 | 10.6 | 7.9 | 8.1 | 8.7 | 6.4 | 1.0 | 0.7 | 0.6 |
|
||||
| Multilingual | s2t_transformer_s | 31M | 12.3 | 17.4 | 6.1 | 12.0 | 13.6 | 13.2 | 12.0 | 13.7 | 10.7 | 13.1 | 0.6 | 0.8 |
|
||||
|
||||
|
||||
## Citation
|
||||
Please cite as:
|
||||
```
|
||||
@inproceedings{salesky2021mtedx,
|
||||
title={Multilingual TEDx Corpus for Speech Recognition and Translation},
|
||||
author={Elizabeth Salesky and Matthew Wiesner and Jacob Bremerman and Roldano Cattoni and Matteo Negri and Marco Turchi and Douglas W. Oard and Matt Post},
|
||||
booktitle={Proceedings of Interspeech},
|
||||
year={2021},
|
||||
}
|
||||
|
||||
@inproceedings{wang2020fairseqs2t,
|
||||
title = {fairseq S2T: Fast Speech-to-Text Modeling with fairseq},
|
||||
author = {Changhan Wang and Yun Tang and Xutai Ma and Anne Wu and Dmytro Okhonko and Juan Pino},
|
||||
booktitle = {Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations},
|
||||
year = {2020},
|
||||
}
|
||||
|
||||
@inproceedings{ott2019fairseq,
|
||||
title = {fairseq: A Fast, Extensible Toolkit for Sequence Modeling},
|
||||
author = {Myle Ott and Sergey Edunov and Alexei Baevski and Angela Fan and Sam Gross and Nathan Ng and David Grangier and Michael Auli},
|
||||
booktitle = {Proceedings of NAACL-HLT 2019: Demonstrations},
|
||||
year = {2019},
|
||||
}
|
||||
```
|
||||
|
||||
[[Back]](..)
|
||||
@@ -0,0 +1,155 @@
|
||||
[[Back]](..)
|
||||
|
||||
# S2T Example: Speech Translation (ST) on MuST-C
|
||||
|
||||
[MuST-C](https://www.aclweb.org/anthology/N19-1202) is multilingual speech-to-text translation corpus with
|
||||
8-language translations on English TED talks. We match the state-of-the-art performance in
|
||||
[ESPNet-ST](https://arxiv.org/pdf/2004.10234.pdf) with a simpler model training pipeline.
|
||||
|
||||
## Data Preparation
|
||||
[Download](https://ict.fbk.eu/must-c) and unpack MuST-C data to a path
|
||||
`${MUSTC_ROOT}/en-${TARGET_LANG_ID}`, then preprocess it with
|
||||
```bash
|
||||
# additional Python packages for S2T data processing/model training
|
||||
pip install pandas torchaudio soundfile sentencepiece
|
||||
|
||||
# Generate TSV manifests, features, vocabulary
|
||||
# and configuration for each language
|
||||
python examples/speech_to_text/prep_mustc_data.py \
|
||||
--data-root ${MUSTC_ROOT} --task asr \
|
||||
--vocab-type unigram --vocab-size 5000
|
||||
python examples/speech_to_text/prep_mustc_data.py \
|
||||
--data-root ${MUSTC_ROOT} --task st \
|
||||
--vocab-type unigram --vocab-size 8000
|
||||
|
||||
# Add vocabulary and configuration for joint data
|
||||
# (based on the manifests and features generated above)
|
||||
python examples/speech_to_text/prep_mustc_data.py \
|
||||
--data-root ${MUSTC_ROOT} --task asr --joint \
|
||||
--vocab-type unigram --vocab-size 10000
|
||||
python examples/speech_to_text/prep_mustc_data.py \
|
||||
--data-root ${MUSTC_ROOT} --task st --joint \
|
||||
--vocab-type unigram --vocab-size 10000
|
||||
```
|
||||
The generated files (manifest, features, vocabulary and data configuration) will be added to
|
||||
`${MUSTC_ROOT}/en-${TARGET_LANG_ID}` (per-language data) and `MUSTC_ROOT` (joint data).
|
||||
|
||||
Download our vocabulary files if you want to use our pre-trained models:
|
||||
- ASR: [En-De](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_de_asr_vocab_unigram5000.zip), [En-Nl](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_nl_asr_vocab_unigram5000.zip), [En-Es](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_es_asr_vocab_unigram5000.zip), [En-Fr](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_fr_asr_vocab_unigram5000.zip), [En-It](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_it_asr_vocab_unigram5000.zip), [En-Pt](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_pt_asr_vocab_unigram5000.zip), [En-Ro](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ro_asr_vocab_unigram5000.zip), [En-Ru](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ru_asr_vocab_unigram5000.zip), [Joint](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_joint_asr_vocab_unigram10000.zip)
|
||||
- ST: [En-De](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_de_st_vocab_unigram8000.zip), [En-Nl](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_nl_st_vocab_unigram8000.zip), [En-Es](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_es_st_vocab_unigram8000.zip), [En-Fr](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_fr_st_vocab_unigram8000.zip), [En-It](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_it_st_vocab_unigram8000.zip), [En-Pt](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_pt_st_vocab_unigram8000.zip), [En-Ro](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ro_st_vocab_unigram8000.zip), [En-Ru](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ru_st_vocab_unigram8000.zip), [Multilingual](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_multilingual_st_vocab_unigram10000.zip)
|
||||
|
||||
## ASR
|
||||
#### Training
|
||||
En-De as example:
|
||||
```bash
|
||||
fairseq-train ${MUSTC_ROOT}/en-de \
|
||||
--config-yaml config_asr.yaml --train-subset train_asr --valid-subset dev_asr \
|
||||
--save-dir ${ASR_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-update 100000 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --report-accuracy \
|
||||
--arch s2t_transformer_s --optimizer adam --lr 1e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8
|
||||
```
|
||||
For joint model (using ASR data from all 8 directions):
|
||||
```bash
|
||||
fairseq-train ${MUSTC_ROOT} \
|
||||
--config-yaml config_asr.yaml \
|
||||
--train-subset train_de_asr,train_nl_asr,train_es_asr,train_fr_asr,train_it_asr,train_pt_asr,train_ro_asr,train_ru_asr \
|
||||
--valid-subset dev_de_asr,dev_nl_asr,dev_es_asr,dev_fr_asr,dev_it_asr,dev_pt_asr,dev_ro_asr,dev_ru_asr \
|
||||
--save-dir ${JOINT_ASR_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-update 100000 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --report-accuracy \
|
||||
--arch s2t_transformer_s --optimizer adam --lr 1e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8
|
||||
```
|
||||
where `ASR_SAVE_DIR` (`JOINT_ASR_SAVE_DIR`) is the checkpoint root path. We set `--update-freq 8` to simulate 8 GPUs
|
||||
with 1 GPU. You may want to update it accordingly when using more than 1 GPU.
|
||||
|
||||
#### Inference & Evaluation
|
||||
```bash
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${ASR_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
fairseq-generate ${MUSTC_ROOT}/en-de \
|
||||
--config-yaml config_asr.yaml --gen-subset tst-COMMON_asr --task speech_to_text \
|
||||
--path ${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME} --max-tokens 50000 --beam 5 \
|
||||
--scoring wer --wer-tokenizer 13a --wer-lowercase --wer-remove-punct
|
||||
|
||||
# For models trained on joint data
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${JOINT_ASR_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${JOINT_ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
for LANG in de nl es fr it pt ro ru; do
|
||||
fairseq-generate ${MUSTC_ROOT} \
|
||||
--config-yaml config_asr.yaml --gen-subset tst-COMMON_${LANG}_asr --task speech_to_text \
|
||||
--path ${JOINT_ASR_SAVE_DIR}/${CHECKPOINT_FILENAME} --max-tokens 50000 --beam 5 \
|
||||
--scoring wer --wer-tokenizer 13a --wer-lowercase --wer-remove-punct
|
||||
done
|
||||
```
|
||||
#### Results
|
||||
| Data | --arch | Params | En-De | En-Nl | En-Es | En-Fr | En-It | En-Pt | En-Ro | En-Ru | Model |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| Single | s2t_transformer_s | 31M | [18.2](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_de_asr_transformer_s.pt) | [17.6](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_nl_asr_transformer_s.pt) | [17.7](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_es_asr_transformer_s.pt) | [17.2](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_fr_asr_transformer_s.pt) | [17.9](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_it_asr_transformer_s.pt) | [19.1](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_pt_asr_transformer_s.pt) | [18.1](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ro_asr_transformer_s.pt) | [17.7](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ru_asr_transformer_s.pt) | (<-Download) |
|
||||
| Joint | s2t_transformer_m | 76M | 16.8 | 16.7 | 16.9 | 16.9 | 17.0 | 17.4 | 17.0 | 16.9 | [Download](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_joint_asr_transformer_m.pt) |
|
||||
|
||||
## ST
|
||||
#### Training
|
||||
En-De as example:
|
||||
```bash
|
||||
fairseq-train ${MUSTC_ROOT}/en-de \
|
||||
--config-yaml config_st.yaml --train-subset train_st --valid-subset dev_st \
|
||||
--save-dir ${ST_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-update 100000 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --report-accuracy \
|
||||
--arch s2t_transformer_s --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8 \
|
||||
--load-pretrained-encoder-from ${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}
|
||||
```
|
||||
For multilingual model (all 8 directions):
|
||||
```bash
|
||||
fairseq-train ${MUSTC_ROOT} \
|
||||
--config-yaml config_st.yaml \
|
||||
--train-subset train_de_st,train_nl_st,train_es_st,train_fr_st,train_it_st,train_pt_st,train_ro_st,train_ru_st \
|
||||
--valid-subset dev_de_st,dev_nl_st,dev_es_st,dev_fr_st,dev_it_st,dev_pt_st,dev_ro_st,dev_ru_st \
|
||||
--save-dir ${MULTILINGUAL_ST_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-update 100000 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --report-accuracy \
|
||||
--arch s2t_transformer_s --ignore-prefix-size 1 --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8 \
|
||||
--load-pretrained-encoder-from ${JOINT_ASR_SAVE_DIR}/${CHECKPOINT_FILENAME}
|
||||
```
|
||||
where `ST_SAVE_DIR` (`MULTILINGUAL_ST_SAVE_DIR`) is the checkpoint root path. The ST encoder is pre-trained by ASR
|
||||
for faster training and better performance: `--load-pretrained-encoder-from <(JOINT_)ASR checkpoint path>`. We set
|
||||
`--update-freq 8` to simulate 8 GPUs with 1 GPU. You may want to update it accordingly when using more than 1 GPU.
|
||||
For multilingual models, we prepend target language ID token as target BOS, which should be excluded from
|
||||
the training loss via `--ignore-prefix-size 1`.
|
||||
|
||||
#### Inference & Evaluation
|
||||
Average the last 10 checkpoints and evaluate on the `tst-COMMON` split:
|
||||
```bash
|
||||
CHECKPOINT_FILENAME=avg_last_10_checkpoint.pt
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${ST_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${ST_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
fairseq-generate ${MUSTC_ROOT}/en-de \
|
||||
--config-yaml config_st.yaml --gen-subset tst-COMMON_st --task speech_to_text \
|
||||
--path ${ST_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 50000 --beam 5 --scoring sacrebleu
|
||||
|
||||
# For multilingual models
|
||||
python scripts/average_checkpoints.py \
|
||||
--inputs ${MULTILINGUAL_ST_SAVE_DIR} --num-epoch-checkpoints 10 \
|
||||
--output "${MULTILINGUAL_ST_SAVE_DIR}/${CHECKPOINT_FILENAME}"
|
||||
for LANG in de nl es fr it pt ro ru; do
|
||||
fairseq-generate ${MUSTC_ROOT} \
|
||||
--config-yaml config_st.yaml --gen-subset tst-COMMON_${LANG}_st --task speech_to_text \
|
||||
--prefix-size 1 --path ${MULTILINGUAL_ST_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--max-tokens 50000 --beam 5 --scoring sacrebleu
|
||||
done
|
||||
```
|
||||
For multilingual models, we force decoding from the target language ID token (as BOS) via `--prefix-size 1`.
|
||||
|
||||
#### Results
|
||||
| Data | --arch | Params | En-De | En-Nl | En-Es | En-Fr | En-It | En-Pt | En-Ro | En-Ru | Model |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| Bilingual | s2t_transformer_s | 31M | [22.7](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_de_st_transformer_s.pt) | [27.3](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_nl_st_transformer_s.pt) | [27.2](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_es_st_transformer_s.pt) | [32.9](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_fr_st_transformer_s.pt) | [22.7](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_it_st_transformer_s.pt) | [28.1](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_pt_st_transformer_s.pt) | [21.9](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ro_st_transformer_s.pt) | [15.3](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_ru_st_transformer_s.pt) | (<-Download) |
|
||||
| Multilingual | s2t_transformer_m | 76M | 24.5 | 28.6 | 28.2 | 34.9 | 24.6 | 31.1 | 23.8 | 16.0 | [Download](https://dl.fbaipublicfiles.com/fairseq/s2t/mustc_multilingual_st_transformer_m.pt) |
|
||||
|
||||
[[Back]](..)
|
||||
@@ -0,0 +1,190 @@
|
||||
# Simultaneous Speech Translation (SimulST) on MuST-C
|
||||
|
||||
This is a tutorial of training and evaluating a transformer *wait-k* simultaneous model on MUST-C English-Germen Dataset, from [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://www.aclweb.org/anthology/2020.aacl-main.58.pdf).
|
||||
|
||||
[MuST-C](https://www.aclweb.org/anthology/N19-1202) is multilingual speech-to-text translation corpus with 8-language translations on English TED talks.
|
||||
|
||||
## Data Preparation
|
||||
This section introduces the data preparation for training and evaluation.
|
||||
If you only want to evaluate the model, please jump to [Inference & Evaluation](#inference--evaluation)
|
||||
|
||||
[Download](https://ict.fbk.eu/must-c) and unpack MuST-C data to a path
|
||||
`${MUSTC_ROOT}/en-${TARGET_LANG_ID}`, then preprocess it with
|
||||
```bash
|
||||
# Additional Python packages for S2T data processing/model training
|
||||
pip install pandas torchaudio sentencepiece
|
||||
|
||||
# Generate TSV manifests, features, vocabulary,
|
||||
# global cepstral and mean estimation,
|
||||
# and configuration for each language
|
||||
cd fairseq
|
||||
|
||||
python examples/speech_to_text/prep_mustc_data.py \
|
||||
--data-root ${MUSTC_ROOT} --task asr \
|
||||
--vocab-type unigram --vocab-size 10000 \
|
||||
--cmvn-type global
|
||||
|
||||
python examples/speech_to_text/prep_mustc_data.py \
|
||||
--data-root ${MUSTC_ROOT} --task st \
|
||||
--vocab-type unigram --vocab-size 10000 \
|
||||
--cmvn-type global
|
||||
```
|
||||
|
||||
## ASR Pretraining
|
||||
We need a pretrained offline ASR model. Assuming the save directory of the ASR model is `${ASR_SAVE_DIR}`.
|
||||
The following command (and the subsequent training commands in this tutorial) assume training on 1 GPU (you can also train on 8 GPUs and remove the `--update-freq 8` option).
|
||||
```
|
||||
fairseq-train ${MUSTC_ROOT}/en-de \
|
||||
--config-yaml config_asr.yaml --train-subset train_asr --valid-subset dev_asr \
|
||||
--save-dir ${ASR_SAVE_DIR} --num-workers 4 --max-tokens 40000 --max-update 100000 \
|
||||
--task speech_to_text --criterion label_smoothed_cross_entropy --report-accuracy \
|
||||
--arch convtransformer_espnet --optimizer adam --lr 0.0005 --lr-scheduler inverse_sqrt \
|
||||
--warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8
|
||||
```
|
||||
A pretrained ASR checkpoint can be downloaded [here](https://dl.fbaipublicfiles.com/simultaneous_translation/must_c_v1_en_de_pretrained_asr)
|
||||
|
||||
## Simultaneous Speech Translation Training
|
||||
|
||||
### Wait-K with fixed pre-decision module
|
||||
Fixed pre-decision indicates that the model operate simultaneous policy on the boundaries of fixed chunks.
|
||||
Here is a example of fixed pre-decision ratio 7 (the simultaneous decision is made every 7 encoder states) and
|
||||
a wait-3 policy model. Assuming the save directory is `${ST_SAVE_DIR}`
|
||||
```bash
|
||||
fairseq-train ${MUSTC_ROOT}/en-de \
|
||||
--config-yaml config_st.yaml --train-subset train_st --valid-subset dev_st \
|
||||
--save-dir ${ST_SAVE_DIR} --num-workers 8 \
|
||||
--optimizer adam --lr 0.0001 --lr-scheduler inverse_sqrt --clip-norm 10.0 \
|
||||
--criterion label_smoothed_cross_entropy \
|
||||
--warmup-updates 4000 --max-update 100000 --max-tokens 40000 --seed 2 \
|
||||
--load-pretrained-encoder-from ${ASR_SAVE_DIR}/checkpoint_best.pt \
|
||||
--task speech_to_text \
|
||||
--arch convtransformer_simul_trans_espnet \
|
||||
--simul-type waitk_fixed_pre_decision \
|
||||
--waitk-lagging 3 \
|
||||
--fixed-pre-decision-ratio 7 \
|
||||
--update-freq 8
|
||||
|
||||
```
|
||||
### Monotonic multihead attention with fixed pre-decision module
|
||||
```
|
||||
fairseq-train ${MUSTC_ROOT}/en-de \
|
||||
--config-yaml config_st.yaml --train-subset train_st --valid-subset dev_st \
|
||||
--save-dir ${ST_SAVE_DIR} --num-workers 8 \
|
||||
--optimizer adam --lr 0.0001 --lr-scheduler inverse_sqrt --clip-norm 10.0 \
|
||||
--warmup-updates 4000 --max-update 100000 --max-tokens 40000 --seed 2 \
|
||||
--load-pretrained-encoder-from ${ASR_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--task speech_to_text \
|
||||
--criterion latency_augmented_label_smoothed_cross_entropy \
|
||||
--latency-weight-avg 0.1 \
|
||||
--arch convtransformer_simul_trans_espnet \
|
||||
--simul-type infinite_lookback_fixed_pre_decision \
|
||||
--fixed-pre-decision-ratio 7 \
|
||||
--update-freq 8
|
||||
```
|
||||
## Inference & Evaluation
|
||||
[SimulEval](https://github.com/facebookresearch/SimulEval) is used for evaluation.
|
||||
The following command is for evaluation.
|
||||
|
||||
```
|
||||
git clone https://github.com/facebookresearch/SimulEval.git
|
||||
cd SimulEval
|
||||
pip install -e .
|
||||
|
||||
simuleval \
|
||||
--agent ${FAIRSEQ}/examples/speech_to_text/simultaneous_translation/agents/fairseq_simul_st_agent.py
|
||||
--source ${SRC_LIST_OF_AUDIO}
|
||||
--target ${TGT_FILE}
|
||||
--data-bin ${MUSTC_ROOT}/en-de \
|
||||
--config config_st.yaml \
|
||||
--model-path ${ST_SAVE_DIR}/${CHECKPOINT_FILENAME} \
|
||||
--output ${OUTPUT} \
|
||||
--scores
|
||||
```
|
||||
|
||||
The source file `${SRC_LIST_OF_AUDIO}` is a list of paths of audio files. Assuming your audio files stored at `/home/user/data`,
|
||||
it should look like this
|
||||
|
||||
```bash
|
||||
/home/user/data/audio-1.wav
|
||||
/home/user/data/audio-2.wav
|
||||
```
|
||||
|
||||
Each line of target file `${TGT_FILE}` is the translation for each audio file input.
|
||||
```bash
|
||||
Translation_1
|
||||
Translation_2
|
||||
```
|
||||
The evaluation runs on the original MUSTC segmentation.
|
||||
The following command will generate the wav list and text file for a evaluation set `${SPLIT}` (chose from `dev`, `tst-COMMON` and `tst-HE`) in MUSTC to `${EVAL_DATA}`.
|
||||
```bash
|
||||
python ${FAIRSEQ}/examples/speech_to_text/seg_mustc_data.py \
|
||||
--data-root ${MUSTC_ROOT} --lang de \
|
||||
--split ${SPLIT} --task st \
|
||||
--output ${EVAL_DATA}
|
||||
```
|
||||
|
||||
The `--data-bin` and `--config` should be the same in previous section if you prepare the data from the scratch.
|
||||
If only for evaluation, a prepared data directory can be found [here](https://dl.fbaipublicfiles.com/simultaneous_translation/must_c_v1.0_en_de_databin.tgz). It contains
|
||||
- `spm_unigram10000_st.model`: a sentencepiece model binary.
|
||||
- `spm_unigram10000_st.txt`: the dictionary file generated by the sentencepiece model.
|
||||
- `gcmvn.npz`: the binary for global cepstral mean and variance.
|
||||
- `config_st.yaml`: the config yaml file. It looks like this.
|
||||
You will need to set the absolute paths for `sentencepiece_model` and `stats_npz_path` if the data directory is downloaded.
|
||||
```yaml
|
||||
bpe_tokenizer:
|
||||
bpe: sentencepiece
|
||||
sentencepiece_model: ABS_PATH_TO_SENTENCEPIECE_MODEL
|
||||
global_cmvn:
|
||||
stats_npz_path: ABS_PATH_TO_GCMVN_FILE
|
||||
input_channels: 1
|
||||
input_feat_per_channel: 80
|
||||
sampling_alpha: 1.0
|
||||
specaugment:
|
||||
freq_mask_F: 27
|
||||
freq_mask_N: 1
|
||||
time_mask_N: 1
|
||||
time_mask_T: 100
|
||||
time_mask_p: 1.0
|
||||
time_wrap_W: 0
|
||||
transforms:
|
||||
'*':
|
||||
- global_cmvn
|
||||
_train:
|
||||
- global_cmvn
|
||||
- specaugment
|
||||
vocab_filename: spm_unigram10000_st.txt
|
||||
```
|
||||
|
||||
Notice that once a `--data-bin` is set, the `--config` is the base name of the config yaml, not the full path.
|
||||
|
||||
Set `--model-path` to the model checkpoint.
|
||||
A pretrained checkpoint can be downloaded from [here](https://dl.fbaipublicfiles.com/simultaneous_translation/convtransformer_wait5_pre7), which is a wait-5 model with a pre-decision of 280 ms.
|
||||
|
||||
The result of this model on `tst-COMMON` is:
|
||||
```bash
|
||||
{
|
||||
"Quality": {
|
||||
"BLEU": 13.94974229366959
|
||||
},
|
||||
"Latency": {
|
||||
"AL": 1751.8031870037803,
|
||||
"AL_CA": 2338.5911762796536,
|
||||
"AP": 0.7931395378788959,
|
||||
"AP_CA": 0.9405103863210942,
|
||||
"DAL": 1987.7811616943081,
|
||||
"DAL_CA": 2425.2751560926167
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If `--output ${OUTPUT}` option is used, the detailed log and scores will be stored under the `${OUTPUT}` directory.
|
||||
|
||||
|
||||
The quality is measured by detokenized BLEU. So make sure that the predicted words sent to the server are detokenized.
|
||||
|
||||
The latency metrics are
|
||||
* Average Proportion
|
||||
* Average Lagging
|
||||
* Differentiable Average Lagging
|
||||
|
||||
Again they will also be evaluated on detokenized text.
|
||||
Reference in New Issue
Block a user