chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,265 @@
|
||||
# wav2vec 2.0
|
||||
|
||||
wav2vec 2.0 learns speech representations on unlabeled data as described in [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al., 2020)](https://arxiv.org/abs/2006.11477).
|
||||
|
||||
We learned speech representations in multiple languages as well in [Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al., 2020)](https://arxiv.org/abs/2006.13979).
|
||||
|
||||
We also combined wav2vec 2.0 with self-training in [Self-training and Pre-training are Complementary for Speech Recognition (Xu et al., 2020)](https://arxiv.org/abs/2010.11430).
|
||||
|
||||
## Pre-trained models
|
||||
|
||||
Model | Finetuning split | Dataset | Model
|
||||
|---|---|---|---
|
||||
Wav2Vec 2.0 Base | No finetuning | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt)
|
||||
Wav2Vec 2.0 Base | 10 minutes | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small_10m.pt)
|
||||
Wav2Vec 2.0 Base | 100 hours | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small_100h.pt)
|
||||
Wav2Vec 2.0 Base | 960 hours | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small_960h.pt)
|
||||
Wav2Vec 2.0 Large | No finetuning | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/libri960_big.pt)
|
||||
Wav2Vec 2.0 Large | 10 minutes | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_big_10m.pt)
|
||||
Wav2Vec 2.0 Large | 100 hours | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_big_100h.pt)
|
||||
Wav2Vec 2.0 Large | 960 hours | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_big_960h.pt)
|
||||
Wav2Vec 2.0 Large (LV-60)* | No finetuning | [Libri-Light](https://github.com/facebookresearch/libri-light) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_new.pt)
|
||||
Wav2Vec 2.0 Large (LV-60)* | 10 minutes | [Libri-Light](https://github.com/facebookresearch/libri-light) + [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_10m_new.pt)
|
||||
Wav2Vec 2.0 Large (LV-60)* | 100 hours | [Libri-Light](https://github.com/facebookresearch/libri-light) + [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_100h_new.pt)
|
||||
Wav2Vec 2.0 Large (LV-60)* | 960 hours | [Libri-Light](https://github.com/facebookresearch/libri-light) + [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec2_vox_960h_new.pt)
|
||||
Wav2Vec 2.0 Large (LV-60) + Self Training * | 10 minutes | [Libri-Light](https://github.com/facebookresearch/libri-light) + [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_10m_pl.pt)
|
||||
Wav2Vec 2.0 Large (LV-60) + Self Training * | 100 hours | [Libri-Light](https://github.com/facebookresearch/libri-light) + [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_100h_pl.pt)
|
||||
Wav2Vec 2.0 Large (LV-60) + Self Training * | 960 hours | [Libri-Light](https://github.com/facebookresearch/libri-light) + [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_vox_960h_pl.pt)
|
||||
|
||||
\* updated (Oct. 24, 2020)
|
||||
|
||||
We also release multilingual pre-trained wav2vec 2.0 (XLSR) models:
|
||||
|
||||
Model | Architecture | Hours | Languages | Datasets | Model
|
||||
|---|---|---|---|---|---
|
||||
XLSR-53 | Large | 56k | 53 | MLS, CommonVoice, BABEL | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/xlsr_53_56k.pt)
|
||||
|
||||
The XLSR model uses the following datasets for multilingual pretraining:
|
||||
|
||||
* **[MLS: Multilingual LibriSpeech](https://indico2.conference4me.psnc.pl/event/35/contributions/3585/attachments/1060/1101/Wed-2-6-10.pdf)** (8 languages, 50.7k hours): *Dutch, English, French, German, Italian, Polish, Portuguese, Spanish*
|
||||
|
||||
* **[CommonVoice](https://commonvoice.mozilla.org/en/languages)** (36 languages, 3.6k hours): *Arabic, Basque, Breton, Chinese (CN), Chinese (HK), Chinese (TW), Chuvash, Dhivehi, Dutch, English, Esperanto, Estonian, French, German, Hakh-Chin, Indonesian, Interlingua, Irish, Italian, Japanese, Kabyle, Kinyarwanda, Kyrgyz, Latvian, Mongolian, Persian, Portuguese, Russian, Sakha, Slovenian, Spanish, Swedish, Tamil, Tatar, Turkish, Welsh* (see also [finetuning splits]([https://dl.fbaipublicfiles.com/cpc_audio/common_voices_splits.tar.gz]) from [this paper](https://arxiv.org/abs/2002.02848)).
|
||||
|
||||
* **[Babel](https://catalog.ldc.upenn.edu/byyear)** (17 languages, 1.7k hours): *Assamese, Bengali, Cantonese, Cebuano, Georgian, Haitian, Kazakh, Kurmanji, Lao, Pashto, Swahili, Tagalog, Tamil, Tok, Turkish, Vietnamese, Zulu*
|
||||
|
||||
|
||||
## Training a new model with the CLI tools
|
||||
|
||||
Given a directory containing wav files to be used for pretraining (we recommend splitting each file into separate file 10 to 30 seconds in length)
|
||||
|
||||
### Prepare training data manifest:
|
||||
|
||||
First, install the `soundfile` library:
|
||||
```shell script
|
||||
pip install soundfile
|
||||
```
|
||||
|
||||
Next, run:
|
||||
|
||||
```shell script
|
||||
$ python examples/wav2vec/wav2vec_manifest.py /path/to/waves --dest /manifest/path --ext $ext --valid-percent $valid
|
||||
```
|
||||
|
||||
$ext should be set to flac, wav, or whatever format your dataset happens to use that soundfile can read.
|
||||
|
||||
$valid should be set to some reasonable percentage (like 0.01) of training data to use for validation.
|
||||
To use a pre-defined validation set (like dev-other from librispeech), set to it 0 and then overwrite valid.tsv with a
|
||||
separately pre-processed manifest file.
|
||||
|
||||
### Train a wav2vec 2.0 base model:
|
||||
|
||||
This configuration was used for the base model trained on the Librispeech dataset in the wav2vec 2.0 paper
|
||||
|
||||
Note that the input is expected to be single channel, sampled at 16 kHz
|
||||
|
||||
```shell script
|
||||
$ fairseq-hydra-train \
|
||||
task.data=/path/to/data \
|
||||
--config-dir /path/to/fairseq-py/examples/wav2vec/config/pretraining \
|
||||
--config-name wav2vec2_base_librispeech
|
||||
```
|
||||
|
||||
Note: you can simulate 64 GPUs by using k GPUs and adding command line parameters (before `--config-dir`)
|
||||
`distributed_training.distributed_world_size=k` `+optimization.update_freq='[x]'` where x = 64/k
|
||||
|
||||
### Train a wav2vec 2.0 large model:
|
||||
|
||||
This configuration was used for the large model trained on the Libri-light dataset in the wav2vec 2.0 paper
|
||||
|
||||
```shell script
|
||||
$ fairseq-hydra-train \
|
||||
task.data=/path/to/data \
|
||||
--config-dir /path/to/fairseq-py/examples/wav2vec/config/pretraining \
|
||||
--config-name wav2vec2_large_librivox
|
||||
```
|
||||
|
||||
Note: you can simulate 128 GPUs by using k GPUs and adding command line parameters (before `--config-dir`)
|
||||
`distributed_training.distributed_world_size=k` `+optimization.update_freq='[x]'` where x = 128/k
|
||||
|
||||
### Fine-tune a pre-trained model with CTC:
|
||||
|
||||
Fine-tuning a model requires parallel audio and labels file, as well as a vocabulary file in fairseq format.
|
||||
A letter vocabulary can be downloaded [here](https://dl.fbaipublicfiles.com/fairseq/wav2vec/dict.ltr.txt).
|
||||
An example [script](libri_labels.py) that generates labels for the Librispeech dataset from the tsv file produced by wav2vec_manifest.py can be used as follows:
|
||||
|
||||
```shell script
|
||||
split=train
|
||||
$ python libri_labels.py /path/to/tsv --output-dir /output/dir --output-name $split
|
||||
```
|
||||
|
||||
Fine-tuning on 100h of Librispeech with letter targets:
|
||||
```shell script
|
||||
$ fairseq-hydra-train \
|
||||
distributed_training.distributed_port=$PORT \
|
||||
task.data=/path/to/data \
|
||||
model.w2v_path=/path/to/model.pt \
|
||||
--config-dir /path/to/fairseq-py/examples/wav2vec/config/finetuning \
|
||||
--config-name base_100h
|
||||
```
|
||||
|
||||
There are other config files in the config/finetuning directory that can be used to fine-tune on other splits.
|
||||
You can specify the right config via the `--config-name` parameter.
|
||||
|
||||
Note: you can simulate 24 GPUs by using k GPUs and adding command line parameters (before `--config-dir`)
|
||||
`distributed_training.distributed_world_size=k` `+optimization.update_freq='[x]'` where x = 24/k
|
||||
|
||||
Decoding with a language model during training requires flashlight [python bindings](https://github.com/facebookresearch/flashlight/tree/master/bindings/python) (previously called [wav2letter](https://github.com/facebookresearch/wav2letter).
|
||||
If you want to use a language model, add `+criterion.wer_args='[/path/to/kenlm, /path/to/lexicon, 2, -1]'` to the command line.
|
||||
|
||||
### Evaluating a CTC model:
|
||||
|
||||
Evaluating a CTC model with a language model requires [flashlight python bindings](https://github.com/facebookresearch/flashlight/tree/master/bindings/python) (previously called [wav2letter](https://github.com/facebookresearch/wav2letter) to be installed.
|
||||
|
||||
Fairseq transformer language model used in the wav2vec 2.0 paper can be obtained from the [wav2letter model repository](https://github.com/facebookresearch/wav2letter/tree/master/recipes/sota/2019).
|
||||
Be sure to upper-case the language model vocab after downloading it.
|
||||
|
||||
Letter dictionary for pre-trained models can be found [here](https://dl.fbaipublicfiles.com/fairseq/wav2vec/dict.ltr.txt).
|
||||
|
||||
Next, run the evaluation command:
|
||||
|
||||
```shell script
|
||||
$subset=dev_other
|
||||
python examples/speech_recognition/infer.py /checkpoint/abaevski/data/speech/libri/10h/wav2vec/raw --task audio_pretraining \
|
||||
--nbest 1 --path /path/to/model --gen-subset $subset --results-path /path/to/save/results/for/sclite --w2l-decoder kenlm \
|
||||
--lm-model /path/to/kenlm.bin --lm-weight 2 --word-score -1 --sil-weight 0 --criterion ctc --labels ltr --max-tokens 4000000 \
|
||||
--post-process letter
|
||||
```
|
||||
|
||||
To get raw numbers, use --w2l-decoder viterbi and omit the lexicon. To use the transformer language model, use --w2l-decoder fairseqlm.
|
||||
|
||||
# wav2vec
|
||||
|
||||
Example to train a wav2vec model as described in [wav2vec: Unsupervised Pre-training for Speech Recognition (Schneider et al., 2019)](https://arxiv.org/abs/1904.05862).
|
||||
|
||||
## Pre-trained models
|
||||
|
||||
Description | Dataset | Model
|
||||
---|---|---
|
||||
Wav2Vec large | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_large.pt)
|
||||
|
||||
#### Example usage:
|
||||
```python
|
||||
import torch
|
||||
import fairseq
|
||||
|
||||
cp_path = '/path/to/wav2vec.pt'
|
||||
model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([cp_path])
|
||||
model = model[0]
|
||||
model.eval()
|
||||
|
||||
wav_input_16khz = torch.randn(1,10000)
|
||||
z = model.feature_extractor(wav_input_16khz)
|
||||
c = model.feature_aggregator(z)
|
||||
```
|
||||
|
||||
## Training a new model with the CLI tools
|
||||
|
||||
Given a directory containing wav files to be used for pretraining (we recommend splitting each file into separate files 10 to 30 seconds in length)
|
||||
|
||||
### Prepare training data manifest:
|
||||
|
||||
```
|
||||
$ python examples/wav2vec/wav2vec_manifest.py /path/to/waves --dest /manifest/path --ext wav
|
||||
```
|
||||
|
||||
### Train a wav2vec model:
|
||||
|
||||
```
|
||||
$ python train.py /manifest/path --save-dir /model/path --num-workers 6 --fp16 --max-update 400000 --save-interval 1 --no-epoch-checkpoints \
|
||||
--arch wav2vec --task audio_pretraining --min-lr 1e-06 --stop-min-lr 1e-09 --optimizer adam --lr 0.005 --lr-scheduler cosine \
|
||||
--conv-feature-layers [(512, 10, 5), (512, 8, 4), (512, 4, 2), (512, 4, 2), (512, 4, 2), (512, 1, 1), (512, 1, 1)] \
|
||||
--conv-aggregator-layers [(512, 2, 1), (512, 3, 1), (512, 4, 1), (512, 5, 1), (512, 6, 1), (512, 7, 1), (512, 8, 1), (512, 9, 1), (512, 10, 1), (512, 11, 1), (512, 12, 1), (512, 13, 1)] \
|
||||
--skip-connections-agg --residual-scale 0.5 --log-compression --warmup-updates 500 --warmup-init-lr 1e-07 --criterion wav2vec --num-negatives 10 \
|
||||
--max-sample-size 150000 --max-tokens 1500000 --skip-invalid-size-inputs-valid-test
|
||||
```
|
||||
|
||||
### Extract embeddings from the downstream task data:
|
||||
|
||||
```
|
||||
$ PYTHONPATH=/path/to/fairseq python examples/wav2vec/wav2vec_featurize.py --input /path/to/task/waves --output /path/to/output \
|
||||
--model /model/path/checkpoint_best.pt --split train valid test
|
||||
```
|
||||
|
||||
# vq-wav2vec
|
||||
|
||||
Example to train a vq-wav2vec model as described in [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (Baevski et al., 2019)](https://arxiv.org/abs/1910.05453).
|
||||
|
||||
These models are also used in [Effectiveness of self-supervised pre-training for speech recognition (Baevski et al., 2019)](https://arxiv.org/abs/1911.03912).
|
||||
|
||||
## Pre-trained models
|
||||
|
||||
Description | Dataset | Model
|
||||
---|---|---
|
||||
vq-wav2vec Gumbel | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/vq-wav2vec.pt)
|
||||
vq-wav2vec K-means | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/vq-wav2vec_kmeans.pt)
|
||||
Roberta on K-means codes | [Librispeech](http://www.openslr.org/12) | [download](https://dl.fbaipublicfiles.com/fairseq/wav2vec/bert_kmeans.tar)
|
||||
|
||||
#### Example usage:
|
||||
```python
|
||||
import torch
|
||||
import fairseq
|
||||
|
||||
cp = torch.load('/path/to/vq-wav2vec.pt')
|
||||
model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([cp])
|
||||
model = model[0]
|
||||
model.eval()
|
||||
|
||||
wav_input_16khz = torch.randn(1,10000)
|
||||
z = model.feature_extractor(wav_input_16khz)
|
||||
_, idxs = model.vector_quantizer.forward_idx(z)
|
||||
print(idxs.shape) # output: torch.Size([1, 60, 2]), 60 timesteps with 2 indexes corresponding to 2 groups in the model
|
||||
```
|
||||
|
||||
## Training a new model with the CLI tools
|
||||
|
||||
Given a directory containing wav files to be used for pretraining (we recommend splitting each file into separate file 10 to 30 seconds in length)
|
||||
|
||||
### Prepare training data manifest:
|
||||
|
||||
```
|
||||
$ python examples/wav2vec/wav2vec_manifest.py /path/to/waves --dest /manifest/path --ext wav
|
||||
```
|
||||
|
||||
### Train a gumbel vq-wav2vec model:
|
||||
|
||||
```
|
||||
$ python train.py /manifest/path --save-dir /model/path --num-workers 6 --fp16 --max-update 400000 \
|
||||
--save-interval 1 --no-epoch-checkpoints --arch wav2vec --task audio_pretraining --min-lr 1e-06 --stop-min-lr 1e-09 \
|
||||
--optimizer adam --lr 1e-05 --lr-scheduler cosine \
|
||||
--conv-feature-layers [(512, 10, 5), (512, 8, 4), (512, 4, 2), (512, 4, 2), (512, 4, 2), (512, 1, 1), (512, 1, 1), (512, 1, 1)] \
|
||||
--conv-aggregator-layers [(512, 2, 1), (512, 3, 1), (512, 4, 1), (512, 5, 1), (512, 6, 1), (512, 7, 1), (512, 8, 1), (512, 9, 1), (512, 10, 1), (512, 11, 1), (512, 12, 1), (512, 13, 1)] \
|
||||
--activation gelu --offset auto --skip-connections-agg --residual-scale 0.5 \
|
||||
--log-keys ["prob_perplexity","code_perplexity","temp"] --vq-type gumbel --vq-groups 2 --vq-depth 2 \
|
||||
--combine-groups --vq-vars 320 --vq-temp (2,0.5,0.999995) --prediction-steps 12 --warmup-updates 1000 \
|
||||
--warmup-init-lr 1e-07 --criterion wav2vec --num-negatives 10 --max-sample-size 150000 \
|
||||
--max-tokens 300000 --cross-sample-negatives 0 --update-freq 1 --seed 2 --skip-invalid-size-inputs-valid-test
|
||||
```
|
||||
|
||||
for k-means training, set vq-type with "kmeans" and add --loss-weights [1] argument. Pre-trained models were trained on 16 GPUs.
|
||||
|
||||
### Tokenize audio data (e.g. for BERT training):
|
||||
|
||||
```
|
||||
$ PYTHONPATH=/path/to/fairseq python examples/wav2vec/vq-wav2vec_featurize.py --data-dir /manifest/path --output-dir /path/to/output \
|
||||
--checkpoint /model/path/checkpoint_best.pt --split train valid test --extension tsv
|
||||
```
|
||||
@@ -0,0 +1,59 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: false
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 3200000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 2
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 80000
|
||||
lr: [0.00003]
|
||||
sentence_avg: true
|
||||
update_freq: [4]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.65
|
||||
mask_channel_prob: 0.5
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 0
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval: 50
|
||||
save_interval_updates: 10000
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: false
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 3200000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
validate_after_updates: 10000
|
||||
validate_interval: 50
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 2
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 20000
|
||||
lr: [0.00005]
|
||||
sentence_avg: true
|
||||
update_freq: [4]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.65
|
||||
mask_channel_prob: 0.5
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.05
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval: 1000
|
||||
save_interval_updates: 50
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: false
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 3200000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
validate_after_updates: 10000
|
||||
validate_interval: 1000
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 2
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 13000
|
||||
lr: [0.00005]
|
||||
sentence_avg: true
|
||||
update_freq: [4]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.65
|
||||
mask_channel_prob: 0.25
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval: 1000
|
||||
save_interval_updates: 50
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: false
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 3200000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
validate_after_updates: 10000
|
||||
validate_interval: 1000
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 2
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 13000
|
||||
lr: [0.00005]
|
||||
sentence_avg: true
|
||||
update_freq: [4]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.65
|
||||
mask_channel_prob: 0.25
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: false
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 3200000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 8
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 320000
|
||||
lr: [0.00001]
|
||||
sentence_avg: true
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.5
|
||||
mask_channel_prob: 0.1
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 0
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: true
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 1280000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 4
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 80000
|
||||
lr: [0.00003]
|
||||
sentence_avg: true
|
||||
update_freq: [5]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.5
|
||||
mask_channel_prob: 0.5
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval: 50
|
||||
save_interval_updates: 10000
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: true
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 1280000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
validate_after_updates: 10000
|
||||
validate_interval: 50
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 4
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 20000
|
||||
lr: [0.0001]
|
||||
sentence_avg: true
|
||||
update_freq: [5]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.75
|
||||
mask_channel_prob: 0.25
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval: 1000
|
||||
save_interval_updates: 50
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: true
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 1280000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
validate_after_updates: 10000
|
||||
validate_interval: 1000
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 4
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 13000
|
||||
lr: [0.0001]
|
||||
sentence_avg: true
|
||||
update_freq: [5]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.65
|
||||
mask_channel_prob: 0.25
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval: 1000
|
||||
save_interval_updates: 50
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: true
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 1280000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
validate_after_updates: 10000
|
||||
validate_interval: 1000
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 4
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 13000
|
||||
lr: [0.0003]
|
||||
sentence_avg: true
|
||||
update_freq: [5]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.75
|
||||
mask_channel_prob: 0.25
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
no_epoch_checkpoints: true
|
||||
best_checkpoint_metric: wer
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
normalize: true
|
||||
labels: ltr
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 1280000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
valid_subset: dev_other
|
||||
|
||||
distributed_training:
|
||||
ddp_backend: no_c10d
|
||||
distributed_world_size: 24
|
||||
|
||||
criterion:
|
||||
_name: ctc
|
||||
zero_infinity: true
|
||||
|
||||
optimization:
|
||||
max_update: 320000
|
||||
lr: [0.00003]
|
||||
sentence_avg: true
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-08
|
||||
|
||||
lr_scheduler:
|
||||
_name: tri_stage
|
||||
phase_ratio: [0.1, 0.4, 0.5]
|
||||
final_lr_scale: 0.05
|
||||
|
||||
model:
|
||||
_name: wav2vec_ctc
|
||||
w2v_path: ???
|
||||
apply_mask: true
|
||||
mask_prob: 0.5
|
||||
mask_channel_prob: 0.25
|
||||
mask_channel_length: 64
|
||||
layerdrop: 0.1
|
||||
activation_dropout: 0.1
|
||||
feature_grad_mult: 0.0
|
||||
freeze_finetune_updates: 10000
|
||||
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval_updates: 25000
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
max_sample_size: 250000
|
||||
min_sample_size: 32000
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 1400000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
|
||||
distributed_training:
|
||||
distributed_world_size: 64
|
||||
ddp_backend: no_c10d
|
||||
|
||||
criterion:
|
||||
_name: wav2vec
|
||||
infonce: true
|
||||
log_keys: ["prob_perplexity","code_perplexity","temp"]
|
||||
loss_weights: [0.1, 10]
|
||||
|
||||
optimization:
|
||||
max_update: 400000
|
||||
lr: [0.0005]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-06
|
||||
weight_decay: 0.01
|
||||
|
||||
lr_scheduler:
|
||||
_name: polynomial_decay
|
||||
warmup_updates: 32000
|
||||
|
||||
model:
|
||||
_name: wav2vec2
|
||||
quantize_targets: true
|
||||
final_dim: 256
|
||||
encoder_layerdrop: 0.05
|
||||
dropout_input: 0.1
|
||||
dropout_features: 0.1
|
||||
feature_grad_mult: 0.1
|
||||
@@ -0,0 +1,69 @@
|
||||
# @package _group_
|
||||
|
||||
common:
|
||||
fp16: true
|
||||
log_format: json
|
||||
log_interval: 200
|
||||
|
||||
checkpoint:
|
||||
save_interval_updates: 25000
|
||||
keep_interval_updates: 1
|
||||
no_epoch_checkpoints: true
|
||||
|
||||
task:
|
||||
_name: audio_pretraining
|
||||
data: ???
|
||||
max_sample_size: 320000
|
||||
min_sample_size: 32000
|
||||
normalize: true
|
||||
|
||||
dataset:
|
||||
num_workers: 6
|
||||
max_tokens: 1200000
|
||||
skip_invalid_size_inputs_valid_test: true
|
||||
|
||||
distributed_training:
|
||||
distributed_world_size: 128
|
||||
ddp_backend: no_c10d
|
||||
|
||||
criterion:
|
||||
_name: wav2vec
|
||||
infonce: true
|
||||
log_keys: ["prob_perplexity","code_perplexity","temp"]
|
||||
loss_weights: [0.1, 0]
|
||||
|
||||
optimization:
|
||||
max_update: 1000000
|
||||
lr: [0.005]
|
||||
|
||||
optimizer:
|
||||
_name: adam
|
||||
adam_betas: (0.9,0.98)
|
||||
adam_eps: 1e-06
|
||||
weight_decay: 0.01
|
||||
|
||||
lr_scheduler:
|
||||
_name: polynomial_decay
|
||||
warmup_updates: 32000
|
||||
|
||||
model:
|
||||
_name: wav2vec2
|
||||
quantize_targets: true
|
||||
extractor_mode: layer_norm
|
||||
layer_norm_first: true
|
||||
final_dim: 768
|
||||
latent_temp: [2.0,0.1,0.999995]
|
||||
encoder_layerdrop: 0.00
|
||||
dropout_input: 0.0
|
||||
dropout_features: 0.0
|
||||
dropout: 0.0
|
||||
attention_dropout: 0.0
|
||||
conv_bias: true
|
||||
|
||||
encoder_layers: 24
|
||||
encoder_embed_dim: 1024
|
||||
encoder_ffn_embed_dim: 4096
|
||||
encoder_attention_heads: 16
|
||||
|
||||
feature_grad_mult: 1.0
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) Facebook, Inc. and its affiliates.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
"""
|
||||
Helper script to pre-compute embeddings for a flashlight (previously called wav2letter++) dataset
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("tsv")
|
||||
parser.add_argument("--output-dir", required=True)
|
||||
parser.add_argument("--output-name", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
os.makedirs(args.output_dir, exist_ok=True)
|
||||
|
||||
transcriptions = {}
|
||||
|
||||
with open(args.tsv, "r") as tsv, open(
|
||||
os.path.join(args.output_dir, args.output_name + ".ltr"), "w"
|
||||
) as ltr_out, open(
|
||||
os.path.join(args.output_dir, args.output_name + ".wrd"), "w"
|
||||
) as wrd_out:
|
||||
root = next(tsv).strip()
|
||||
for line in tsv:
|
||||
line = line.strip()
|
||||
dir = os.path.dirname(line)
|
||||
if dir not in transcriptions:
|
||||
parts = dir.split(os.path.sep)
|
||||
trans_path = f"{parts[-2]}-{parts[-1]}.trans.txt"
|
||||
path = os.path.join(root, dir, trans_path)
|
||||
assert os.path.exists(path)
|
||||
texts = {}
|
||||
with open(path, "r") as trans_f:
|
||||
for tline in trans_f:
|
||||
items = tline.strip().split()
|
||||
texts[items[0]] = " ".join(items[1:])
|
||||
transcriptions[dir] = texts
|
||||
part = os.path.basename(line).split(".")[0]
|
||||
assert part in transcriptions[dir]
|
||||
print(transcriptions[dir][part], file=wrd_out)
|
||||
print(
|
||||
" ".join(list(transcriptions[dir][part].replace(" ", "|"))) + " |",
|
||||
file=ltr_out,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,250 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) Facebook, Inc. and its affiliates.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
"""
|
||||
Helper script to pre-compute embeddings for a flashlight (previously called wav2letter++) dataset
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import os.path as osp
|
||||
import pprint
|
||||
|
||||
import soundfile as sf
|
||||
import torch
|
||||
import fairseq
|
||||
from torch import nn
|
||||
from torch.utils.data import DataLoader
|
||||
|
||||
|
||||
try:
|
||||
import tqdm
|
||||
except:
|
||||
print("Install tqdm to use --log-format=tqdm")
|
||||
|
||||
|
||||
class FilesDataset:
|
||||
def __init__(self, files, labels):
|
||||
self.files = files
|
||||
if labels and osp.exists(labels):
|
||||
with open(labels, "r") as lbl_f:
|
||||
self.labels = [line.rstrip() for line in lbl_f]
|
||||
else:
|
||||
self.labels = labels
|
||||
|
||||
def __len__(self):
|
||||
return len(self.files)
|
||||
|
||||
def __getitem__(self, index):
|
||||
fname = self.files[index]
|
||||
|
||||
wav, sr = sf.read(fname)
|
||||
assert sr == 16000
|
||||
|
||||
wav = torch.from_numpy(wav).float()
|
||||
lbls = None
|
||||
if self.labels:
|
||||
if isinstance(self.labels, str):
|
||||
lbl_file = osp.splitext(fname)[0] + "." + self.labels
|
||||
with open(lbl_file, "r") as lblf:
|
||||
lbls = lblf.readline()
|
||||
assert lbls is not None
|
||||
else:
|
||||
lbls = self.labels[index]
|
||||
return wav, lbls
|
||||
|
||||
def collate(self, batch):
|
||||
return batch
|
||||
|
||||
|
||||
class ArgTypes:
|
||||
@staticmethod
|
||||
def existing_path(arg):
|
||||
arg = str(arg)
|
||||
assert osp.exists(arg), f"File {arg} does not exist"
|
||||
return arg
|
||||
|
||||
@staticmethod
|
||||
def mkdir(arg):
|
||||
arg = str(arg)
|
||||
os.makedirs(arg, exist_ok=True)
|
||||
return arg
|
||||
|
||||
|
||||
class DatasetWriter:
|
||||
def __init__(self):
|
||||
|
||||
self.args = self.load_config()
|
||||
pprint.pprint(self.args.__dict__)
|
||||
|
||||
self.model = self.load_model()
|
||||
|
||||
def __getattr__(self, attr):
|
||||
return getattr(self.args, attr)
|
||||
|
||||
def read_manifest(self, fname):
|
||||
|
||||
with open(fname, "r") as fp:
|
||||
lines = fp.read().split("\n")
|
||||
root = lines.pop(0).strip()
|
||||
fnames = [
|
||||
osp.join(root, line.split("\t")[0]) for line in lines if len(line) > 0
|
||||
]
|
||||
|
||||
return fnames
|
||||
|
||||
def process_splits(self):
|
||||
|
||||
if self.args.shard is not None or self.args.num_shards is not None:
|
||||
assert self.args.shard is not None and self.args.num_shards is not None
|
||||
|
||||
for split in self.splits:
|
||||
print(split)
|
||||
|
||||
if self.extension == "tsv":
|
||||
datadir = osp.join(self.data_dir, f"{split}.{self.extension}")
|
||||
print("Reading manifest file: ", datadir)
|
||||
files = self.read_manifest(datadir)
|
||||
else:
|
||||
datadir = osp.join(self.data_dir, split, f"**/*.{self.extension}")
|
||||
files = glob.glob(datadir, recursive=True)
|
||||
|
||||
assert len(files) > 0
|
||||
|
||||
if self.args.shard is not None:
|
||||
files = files[self.args.shard :: self.args.num_shards]
|
||||
|
||||
lbls = []
|
||||
with open(self.data_file(split), "w") as srcf:
|
||||
for line, lbl in self.iterate(files):
|
||||
print(line, file=srcf)
|
||||
if self.args.labels:
|
||||
lbls.append(lbl + "\n")
|
||||
|
||||
if self.args.labels:
|
||||
assert all(a is not None for a in lbls)
|
||||
with open(self.lbl_file(split), "w") as lblf:
|
||||
lblf.writelines(lbls)
|
||||
|
||||
def iterate(self, files):
|
||||
|
||||
data = self.load_data(files)
|
||||
for samples in tqdm.tqdm(data, total=len(files) // 32):
|
||||
|
||||
for wav, lbl in samples:
|
||||
x = wav.unsqueeze(0).float().cuda()
|
||||
|
||||
div = 1
|
||||
while x.size(-1) // div > self.args.max_size:
|
||||
div += 1
|
||||
|
||||
xs = x.chunk(div, dim=-1)
|
||||
|
||||
result = []
|
||||
for x in xs:
|
||||
torch.cuda.empty_cache()
|
||||
x = self.model.feature_extractor(x)
|
||||
if self.quantize_location == "encoder":
|
||||
with torch.no_grad():
|
||||
_, idx = self.model.vector_quantizer.forward_idx(x)
|
||||
idx = idx.squeeze(0).cpu()
|
||||
else:
|
||||
with torch.no_grad():
|
||||
z = self.model.feature_aggregator(x)
|
||||
_, idx = self.model.vector_quantizer.forward_idx(z)
|
||||
idx = idx.squeeze(0).cpu()
|
||||
result.append(idx)
|
||||
|
||||
idx = torch.cat(result, dim=0)
|
||||
yield " ".join("-".join(map(str, a.tolist())) for a in idx), lbl
|
||||
|
||||
def lbl_file(self, name):
|
||||
shard_part = "" if self.args.shard is None else f".{self.args.shard}"
|
||||
return osp.join(self.output_dir, f"{name}.lbl{shard_part}")
|
||||
|
||||
def data_file(self, name):
|
||||
shard_part = "" if self.args.shard is None else f".{self.args.shard}"
|
||||
return osp.join(self.output_dir, f"{name}.src{shard_part}")
|
||||
|
||||
def var_file(self):
|
||||
return osp.join(self.output_dir, f"vars.pt")
|
||||
|
||||
def load_config(self):
|
||||
|
||||
parser = argparse.ArgumentParser("Vector Quantized wav2vec features")
|
||||
|
||||
# Model Arguments
|
||||
parser.add_argument("--checkpoint", type=ArgTypes.existing_path, required=True)
|
||||
parser.add_argument("--data-parallel", action="store_true")
|
||||
|
||||
# Output Arguments
|
||||
parser.add_argument("--output-dir", type=ArgTypes.mkdir, required=True)
|
||||
|
||||
# Data Arguments
|
||||
parser.add_argument("--data-dir", type=ArgTypes.existing_path, required=True)
|
||||
parser.add_argument("--splits", type=str, nargs="+", required=True)
|
||||
parser.add_argument("--extension", type=str, required=True)
|
||||
parser.add_argument("--labels", type=str, required=False)
|
||||
|
||||
parser.add_argument("--shard", type=int, default=None)
|
||||
parser.add_argument("--num-shards", type=int, default=None)
|
||||
parser.add_argument("--max-size", type=int, default=1300000)
|
||||
|
||||
# Logger Arguments
|
||||
parser.add_argument(
|
||||
"--log-format", type=str, choices=["none", "simple", "tqdm"]
|
||||
)
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
def load_data(self, fnames):
|
||||
|
||||
dataset = FilesDataset(fnames, self.args.labels)
|
||||
loader = DataLoader(
|
||||
dataset, batch_size=32, collate_fn=dataset.collate, num_workers=8
|
||||
)
|
||||
return loader
|
||||
|
||||
def load_model(self):
|
||||
model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([self.checkpoint])
|
||||
model = model[0]
|
||||
|
||||
self.quantize_location = getattr(cfg.model, "vq", "encoder")
|
||||
|
||||
model.eval().float()
|
||||
model.cuda()
|
||||
|
||||
if self.data_parallel:
|
||||
model = nn.DataParallel(model)
|
||||
|
||||
return model
|
||||
|
||||
def __call__(self):
|
||||
|
||||
self.process_splits()
|
||||
|
||||
if hasattr(self.model.feature_extractor, "vars") and (
|
||||
self.args.shard is None or self.args.shard == 0
|
||||
):
|
||||
vars = (
|
||||
self.model.feature_extractor.vars.view(
|
||||
self.model.feature_extractor.banks,
|
||||
self.model.feature_extractor.num_vars,
|
||||
-1,
|
||||
)
|
||||
.cpu()
|
||||
.detach()
|
||||
)
|
||||
print("writing learned latent variable embeddings: ", vars.shape)
|
||||
torch.save(vars, self.var_file())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
write_data = DatasetWriter()
|
||||
|
||||
write_data()
|
||||
print("Done.")
|
||||
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) Facebook, Inc. and its affiliates.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
|
||||
"""
|
||||
Helper script to pre-compute embeddings for a flashlight (previously called wav2letter++) dataset
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
from shutil import copy
|
||||
|
||||
import h5py
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
import torch
|
||||
import tqdm
|
||||
import fairseq
|
||||
from torch import nn
|
||||
|
||||
|
||||
def read_audio(fname):
|
||||
""" Load an audio file and return PCM along with the sample rate """
|
||||
|
||||
wav, sr = sf.read(fname)
|
||||
assert sr == 16e3
|
||||
|
||||
return wav, 16e3
|
||||
|
||||
|
||||
class PretrainedWav2VecModel(nn.Module):
|
||||
def __init__(self, fname):
|
||||
super().__init__()
|
||||
|
||||
model, cfg, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([fname])
|
||||
model = model[0]
|
||||
model.eval()
|
||||
|
||||
self.model = model
|
||||
|
||||
def forward(self, x):
|
||||
with torch.no_grad():
|
||||
z = self.model.feature_extractor(x)
|
||||
if isinstance(z, tuple):
|
||||
z = z[0]
|
||||
c = self.model.feature_aggregator(z)
|
||||
return z, c
|
||||
|
||||
|
||||
class EmbeddingWriterConfig(argparse.ArgumentParser):
|
||||
def __init__(self):
|
||||
super().__init__("Pre-compute embeddings for flashlight datasets")
|
||||
|
||||
kwargs = {"action": "store", "type": str, "required": True}
|
||||
|
||||
self.add_argument("--input", "-i", help="Input Directory", **kwargs)
|
||||
self.add_argument("--output", "-o", help="Output Directory", **kwargs)
|
||||
self.add_argument("--model", help="Path to model checkpoint", **kwargs)
|
||||
self.add_argument("--split", help="Dataset Splits", nargs="+", **kwargs)
|
||||
self.add_argument(
|
||||
"--ext", default="wav", required=False, help="Audio file extension"
|
||||
)
|
||||
|
||||
self.add_argument(
|
||||
"--no-copy-labels",
|
||||
action="store_true",
|
||||
help="Do not copy label files. Useful for large datasets, use --targetdir in flashlight then.",
|
||||
)
|
||||
self.add_argument(
|
||||
"--use-feat",
|
||||
action="store_true",
|
||||
help="Use the feature vector ('z') instead of context vector ('c') for features",
|
||||
)
|
||||
self.add_argument("--gpu", help="GPU to use", default=0, type=int)
|
||||
|
||||
|
||||
class Prediction:
|
||||
""" Lightweight wrapper around a fairspeech embedding model """
|
||||
|
||||
def __init__(self, fname, gpu=0):
|
||||
self.gpu = gpu
|
||||
self.model = PretrainedWav2VecModel(fname).cuda(gpu)
|
||||
|
||||
def __call__(self, x):
|
||||
x = torch.from_numpy(x).float().cuda(self.gpu)
|
||||
with torch.no_grad():
|
||||
z, c = self.model(x.unsqueeze(0))
|
||||
|
||||
return z.squeeze(0).cpu().numpy(), c.squeeze(0).cpu().numpy()
|
||||
|
||||
|
||||
class H5Writer:
|
||||
""" Write features as hdf5 file in flashlight compatible format """
|
||||
|
||||
def __init__(self, fname):
|
||||
self.fname = fname
|
||||
os.makedirs(os.path.dirname(self.fname), exist_ok=True)
|
||||
|
||||
def write(self, data):
|
||||
channel, T = data.shape
|
||||
|
||||
with h5py.File(self.fname, "w") as out_ds:
|
||||
data = data.T.flatten()
|
||||
out_ds["features"] = data
|
||||
out_ds["info"] = np.array([16e3 // 160, T, channel])
|
||||
|
||||
|
||||
class EmbeddingDatasetWriter(object):
|
||||
"""Given a model and a flashlight dataset, pre-compute and store embeddings
|
||||
|
||||
Args:
|
||||
input_root, str :
|
||||
Path to the flashlight dataset
|
||||
output_root, str :
|
||||
Desired output directory. Will be created if non-existent
|
||||
split, str :
|
||||
Dataset split
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
input_root,
|
||||
output_root,
|
||||
split,
|
||||
model_fname,
|
||||
extension="wav",
|
||||
gpu=0,
|
||||
verbose=False,
|
||||
use_feat=False,
|
||||
):
|
||||
|
||||
assert os.path.exists(model_fname)
|
||||
|
||||
self.model_fname = model_fname
|
||||
self.model = Prediction(self.model_fname, gpu)
|
||||
|
||||
self.input_root = input_root
|
||||
self.output_root = output_root
|
||||
self.split = split
|
||||
self.verbose = verbose
|
||||
self.extension = extension
|
||||
self.use_feat = use_feat
|
||||
|
||||
assert os.path.exists(self.input_path), "Input path '{}' does not exist".format(
|
||||
self.input_path
|
||||
)
|
||||
|
||||
def _progress(self, iterable, **kwargs):
|
||||
if self.verbose:
|
||||
return tqdm.tqdm(iterable, **kwargs)
|
||||
return iterable
|
||||
|
||||
def require_output_path(self, fname=None):
|
||||
path = self.get_output_path(fname)
|
||||
os.makedirs(path, exist_ok=True)
|
||||
|
||||
@property
|
||||
def input_path(self):
|
||||
return self.get_input_path()
|
||||
|
||||
@property
|
||||
def output_path(self):
|
||||
return self.get_output_path()
|
||||
|
||||
def get_input_path(self, fname=None):
|
||||
if fname is None:
|
||||
return os.path.join(self.input_root, self.split)
|
||||
return os.path.join(self.get_input_path(), fname)
|
||||
|
||||
def get_output_path(self, fname=None):
|
||||
if fname is None:
|
||||
return os.path.join(self.output_root, self.split)
|
||||
return os.path.join(self.get_output_path(), fname)
|
||||
|
||||
def copy_labels(self):
|
||||
self.require_output_path()
|
||||
|
||||
labels = list(
|
||||
filter(
|
||||
lambda x: self.extension not in x, glob.glob(self.get_input_path("*"))
|
||||
)
|
||||
)
|
||||
for fname in tqdm.tqdm(labels):
|
||||
copy(fname, self.output_path)
|
||||
|
||||
@property
|
||||
def input_fnames(self):
|
||||
return sorted(glob.glob(self.get_input_path("*.{}".format(self.extension))))
|
||||
|
||||
def __len__(self):
|
||||
return len(self.input_fnames)
|
||||
|
||||
def write_features(self):
|
||||
|
||||
paths = self.input_fnames
|
||||
|
||||
fnames_context = map(
|
||||
lambda x: os.path.join(
|
||||
self.output_path, x.replace("." + self.extension, ".h5context")
|
||||
),
|
||||
map(os.path.basename, paths),
|
||||
)
|
||||
|
||||
for name, target_fname in self._progress(
|
||||
zip(paths, fnames_context), total=len(self)
|
||||
):
|
||||
wav, sr = read_audio(name)
|
||||
z, c = self.model(wav)
|
||||
feat = z if self.use_feat else c
|
||||
writer = H5Writer(target_fname)
|
||||
writer.write(feat)
|
||||
|
||||
def __repr__(self):
|
||||
|
||||
return "EmbeddingDatasetWriter ({n_files} files)\n\tinput:\t{input_root}\n\toutput:\t{output_root}\n\tsplit:\t{split})".format(
|
||||
n_files=len(self), **self.__dict__
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
args = EmbeddingWriterConfig().parse_args()
|
||||
|
||||
for split in args.split:
|
||||
|
||||
writer = EmbeddingDatasetWriter(
|
||||
input_root=args.input,
|
||||
output_root=args.output,
|
||||
split=split,
|
||||
model_fname=args.model,
|
||||
gpu=args.gpu,
|
||||
extension=args.ext,
|
||||
use_feat=args.use_feat,
|
||||
)
|
||||
|
||||
print(writer)
|
||||
writer.require_output_path()
|
||||
|
||||
print("Writing Features...")
|
||||
writer.write_features()
|
||||
print("Done.")
|
||||
|
||||
if not args.no_copy_labels:
|
||||
print("Copying label data...")
|
||||
writer.copy_labels()
|
||||
print("Done.")
|
||||
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) Facebook, Inc. and its affiliates.
|
||||
#
|
||||
# This source code is licensed under the MIT license found in the
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
"""
|
||||
Data pre-processing: build vocabularies and binarize training data.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import random
|
||||
|
||||
import soundfile
|
||||
|
||||
|
||||
def get_parser():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"root", metavar="DIR", help="root directory containing flac files to index"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--valid-percent",
|
||||
default=0.01,
|
||||
type=float,
|
||||
metavar="D",
|
||||
help="percentage of data to use as validation set (between 0 and 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dest", default=".", type=str, metavar="DIR", help="output directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ext", default="flac", type=str, metavar="EXT", help="extension to look for"
|
||||
)
|
||||
parser.add_argument("--seed", default=42, type=int, metavar="N", help="random seed")
|
||||
parser.add_argument(
|
||||
"--path-must-contain",
|
||||
default=None,
|
||||
type=str,
|
||||
metavar="FRAG",
|
||||
help="if set, path must contain this substring for a file to be included in the manifest",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(args):
|
||||
assert args.valid_percent >= 0 and args.valid_percent <= 1.0
|
||||
|
||||
if not os.path.exists(args.dest):
|
||||
os.makedirs(args.dest)
|
||||
|
||||
dir_path = os.path.realpath(args.root)
|
||||
search_path = os.path.join(dir_path, "**/*." + args.ext)
|
||||
rand = random.Random(args.seed)
|
||||
|
||||
with open(os.path.join(args.dest, "train.tsv"), "w") as train_f, open(
|
||||
os.path.join(args.dest, "valid.tsv"), "w"
|
||||
) as valid_f:
|
||||
print(dir_path, file=train_f)
|
||||
print(dir_path, file=valid_f)
|
||||
|
||||
for fname in glob.iglob(search_path, recursive=True):
|
||||
file_path = os.path.realpath(fname)
|
||||
|
||||
if args.path_must_contain and args.path_must_contain not in file_path:
|
||||
continue
|
||||
|
||||
frames = soundfile.info(fname).frames
|
||||
dest = train_f if rand.random() > args.valid_percent else valid_f
|
||||
print(
|
||||
"{}\t{}".format(os.path.relpath(file_path, dir_path), frames), file=dest
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = get_parser()
|
||||
args = parser.parse_args()
|
||||
main(args)
|
||||
Reference in New Issue
Block a user