nvidia-nemo--speech/examples/asr/asr_eou

Finetuning streming ASR model for integrated end-of-utterance (EOU) detection

This tutorial shows how to finetune a streaming ASR model (e.g., nvidia/nemotron-speech-streaming-en-0.6b) for integrated EOU detection (e.g., nvidia/parakeet_realtime_eou_120m-v1).

We use Nemotron-Speech-Streaming-En-0.6b as an example of pretrained ASR model.

Steps

1. Model preparation
2. Dataset preparation
3. Model training
4. Model evaluation

1. Model preparation

1.1. Download pretrained model

Download the Nemotron-Speech-Streaming-En-0.6b model from HuggingFace via:

wget https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b/resolve/main/nemotron-speech-streaming-en-0.6b.nemo

1.2. Add special tokens to tokenizer

By default, we use <EOU> and <EOB> for "end-of-utterance" and "end-of-backchannel" respectively, and add these two special tokens to the tokenizer of the pretrained model:

python <NeMo Root>/scripts/asr_eou/tokenizers/add_special_tokens_to_sentencepiece.py \
    --input_file /path/to/nemotron-speech-streaming-en-0.6b.nemo \
    --output_dir /path/to/asr_eou_tokenizer_dir

The output directory /path/to/asr_eou_tokenizer_dir will contain the updated tokenizer to be used when updateding the model config.

The special tokens are added to the end of the original vocabulary. For example, if the original vocabulary size is 1024, the new vocabulary size will be 1026, and the special tokens will be added at the indices 1024 and 1025.

1.3. Update model config for ASR-EOU model

We can extract the model config from the downloaded .nemo file by:

tar -xvf /path/to/nemotron-speech-streaming-en-0.6b.nemo -C /path/to/asr_model_dir

The output file /path/to/asr_model_dir/model_config.yaml is the model config to be updated for finetuning the ASR model into an ASR-EOU model.

In the model config file, we need to change the tokenizer to use the new tokenizer with special tokens:

tokenizer:
  dir: /path/to/asr_eou_tokenizer_dir
  type: bpe

We also need to add some additional configurations to the model section to specify how we want to initialize the weights for the special tokens:

model:
  token_init_method: "constant"  # choices=['min', 'max', 'mean', 'constant']
  token_init_weight_value: null  # only applicable when token_init_method='constant'
  token_init_bias_value: -1000.0  # only applicable when token_init_method='constant'

You may also need to change the optimization and loss parameters to suit your use cases. We empirically find that setting fastemit_lambda to 3e-2 is a good start.

loss:
    loss_name: "default"
    warprnnt_numba_kwargs:
      # FastEmit regularization: https://arxiv.org/abs/2010.11148
      # You may enable FastEmit to increase the accuracy and reduce the latency of the model for streaming
      # You may set it to lower values like 1e-3 for models with larger right context
      fastemit_lambda: 3e-2 

We also need to change the training, validation and test data paths in the model config file based on how we prepare the EOU labeled dataset illustrated in the next section.

For a full example of the model config file, please refer to examples/asr/conf/asr_eou/fastconformer_transducer_bpe_streaming_xlarge.yaml.

2. Dataset preparation

When finetuning the ASR model for EOU detection, we need to prepare the dataset in a specific format. But more importantly, we need to make sure the dataset used for finetuning meets the criteria that each sample should contain a single utterance, otherwise the model's EOU prediction accuracy will be degraded. In the case of using a small EOU dataset, we can blend the EOU dataset with the some normal ASR dataset which does not necessarily contain EOU labels, such that ASR WER is not significantly degraded. For lowest possible EOU latency, we recommend dropping the punctuations from the transcriptions to simplify the text processing.

2.1 Mainifest format

We expect the input data manifest to be JSONL format, with each line containing the following fields:

{
    "audio_filepath": "/path/to/audio.wav",
    "text": "The text of the audio.", # transcript of the utterance
    "offset": 0.0,  # offset of the audio, in seconds
    "duration": 3.0,  # duration of the audio, in seconds
    "sou_time": 0.2,  # start of utterance time, in seconds
    "eou_time": 1.5,  # end of utterance time, in seconds
    "is_backchannel": false  # [optional] whether the utterance is a backchannel phrase
  }

Your original input manifest should contain the fields audio_filepath, text, offset and duration, while the fields sou_time, eou_time and is_backchannel can be obtained by following steps.

2.2 Getting timestamps for end-of-utterance (EOU)

We recommend using forced alignment to get the timestamps for EOU. One way to do this is to use the Nemo Forced Aligner tool.

python <NeMo Root>/tools/nemo_forced_aligner/align_eou.py \
    pretrained_name="nvidia/parakeet-ctc-0.6b" \
    manifest_filepath=/path/to/asr_manifest.jsonl \
    output_manifest_filepath=/path/to/asr_eou_manifest.jsonl

The output manifest will contain the fields audio_filepath, text, offset, duration, sou_time and eou_time.

2.3 (Optional) Add end-of-backchannel (EOB) labels to dataset

Backchannel phrases refer to those phrases that are not part of the main conversation, but are used to acknowledge or respond to the speaker. For example, "uh-huh", "yeah", "right", "okay", "thanks", "sorry", etc. We can also train the model to detect backchannel phrases by adding end-of-backchannel (EOB) labels to the dataset, so that the cascaded system can leverage the EOU and EOB predictions to better understand the conversation. However, we can also treat EOB as a special case of EOU, and match the predicted EOU phrases with a list of predefined backchannel phrases to prediction EOB, which is more flexible in handling different backchannel phrases.

If you want to add end-of-backchannel (EOB) labels to training, you can use the following script to add the is_backchannel field to the manifest:

python <NeMo Root>/scripts/asr_eou/add_eob_labels.py \
    input_manifest=/path/to/asr_manifest.jsonl \
    output_manifest=/path/to/asr_eou_eob_manifest.jsonl

The add_eob_labels.py file has a list of predefined backchannel phrases, and you can edit it to add more backchannel phrases if needed. An easy way to figure out backchannel phrases is to find the most frequent one, two or three words utterances in the dataset, and manually check if they are backchannel phrases.

2.4 Creating tarred datasets for large-scale training.

For more efficient training, you can create tarred datasets for the ASR and EOU datasets by using scripts/speech_recognition/convert_to_tarred_audio_dataset.py script.

2.5 Creating input data config for blending ASR and EOU data

Please refer to the documentation for more details on how to specify the dataset configuration in the model config file.

An example of the train_input_config.yaml file is shown below, where we use 0.1 weight for the ASR dataset and 0.9 weight for the EOU dataset.

- input_cfg:
  - corpus: Librispeech
    language: en
    type: nemo
    manifest_filepath: /data/LibriSpeech/train_other_500.jsonl  # this is a normal ASR dataset
  tags:
    taskname: asr
  type: group
  weight: 0.1
- input_cfg:
  - corpus: LibriTTS
    language: en
    type: nemo
    manifest_filepath: /data/LibriTTS/train_clean_360_eou.jsonl  # this is a EOU manifest after adding sou_time and eou_time fields
  tags:
    taskname: eou
  type: group
  weight: 0.9

2.6 Creating evaluation dataset

We can create evaluation dataset by padding the audio signal with non-speech frames and/or adding noise to the clean audios.

Example usage with multiple manifests matching a pattern:

python <NeMo Root>/scripts/asr_eou/generate_noisy_eval_data.py \
    output_dir=/path/to/output/dir \
    data.manifest_filepath=/path/to/manifest/dir/ \
    data.pattern="*.json" \
    data.seed=42 \
    data.noise.manifest_path /path/to/noise_manifest.json

You can modify the yaml config to specify the augmentation parameters in scripts/asr_eou/conf/data.yaml.

2.7 Configuring dataset in model config

Now we can update the model config to use the prepared training and evaluation data config.

model:
  train_ds:
    input_cfg: /path/to/train_input_config.yaml
    manifest_filepath: null
    tarred_audio_filepaths: null
    ignore_eob_label: true  # ignore backchannel and treat them the same as EOU

    random_padding:
      prob: 0.9
      min_post_pad_duration: 3.0 # minimum duration of post-padding silence in seconds
      min_pre_pad_duration: 0.0 # minimum duration of pre-padding silence in seconds
      max_pad_duration: 6.0  # maximum duration of pre/post padding in seconds
      max_total_duration: 40.0  # maximum total duration of the padded audio in seconds
      pad_distribution: 'uniform'  # distribution of padding duration, 'uniform' or 'normal'
      normal_mean: 0.5  # mean of normal distribution used when pad_distribution='normal'
      normal_std: 2.0  # standard deviation of normal distribution  used when pad_distribution='normal'
      
    augmentor:
      white_noise:
        prob: 0.9
        min_level: -90
        max_level: -46
      gain:
        prob: 0.2
        min_gain_dbfs: -10.0
        max_gain_dbfs: 10.0
      noise:
        prob: 0.9
        manifest_path: /path/to/noise_manifest.json
        min_snr_db: 0
        max_snr_db: 20
        max_gain_db: 300.0

  validation_ds:
    input_cfg: null
    manifest_filepath: ["/path/to/eval_manifest1.json", "/path/to/eval_manifest2.json", ...]
    tarred_audio_filepaths: null
    ignore_eob_label: true  # ignore backchannel and treat them the same as EOU

For a full example of the model config file, please refer to examples/asr/conf/asr_eou/fastconformer_transducer_bpe_streaming_xlarge.yaml.

3. Model training

To start the training, you can run the following command:

#!/bin/bash

TRAIN_INPUT_CFG=/path/to/train_input_config.yaml
VAL_MANIFEST=/path/to/val_manifest.json
NOISE_MANIFEST=/path/to/noise_manifest.json

PRETRAINED_NEMO=/path/to/nemotron-speech-streaming-en-0.6b.nemo 

BATCH_SIZE=16
NUM_WORKERS=8
LIMIT_TRAIN_BATCHES=1000
VAL_CHECK_INTERVAL=1000
MAX_STEPS=1000000

EXP_NAME=nemotron_speech_streaming_en_0.6b_eou
SCRIPT=${NEMO_PATH}/examples/asr/asr_eou/speech_to_text_rnnt_eou_train.py
CONFIG_PATH=${NEMO_PATH}/examples/asr/conf/asr_eou
CONFIG_NAME=fastconformer_transducer_bpe_streaming_xlarge

CUDA_VISIBLE_DEVICES=0 python $SCRIPT \
    --config-path $CONFIG_PATH \
    --config-name $CONFIG_NAME \
    ++init_from_nemo_model=$PRETRAINED_NEMO \
    model.encoder.att_context_size="[70,1]" \
    model.train_ds.input_cfg=$TRAIN_INPUT_CFG \
    model.train_ds.augmentor.noise.manifest_path=$NOISE_MANIFEST \
    model.validation_ds.manifest_filepath=$VAL_MANIFEST \
    model.train_ds.batch_size=$BATCH_SIZE \
    model.train_ds.num_workers=$NUM_WORKERS \
    model.validation_ds.batch_size=$BATCH_SIZE \
    model.validation_ds.num_workers=$NUM_WORKERS \
    ~model.test_ds \
    trainer.limit_train_batches=$LIMIT_TRAIN_BATCHES \
    trainer.val_check_interval=$VAL_CHECK_INTERVAL \
    trainer.max_steps=$MAX_STEPS \
    exp_manager.name=$EXP_NAME

For lowest EOU latency, we set att_context_size to [70,1] in the model config file, which means the model lookahead is 1 frame (80ms), and the input chunk size is thus 2 frames (160ms).

4. Model evaluation

After training, we can evaluate the model on the evaluation dataset by running the following command:

TEST_MANIFEST="[/path/to/your/test_manifest.json,/path/to/your/test_manifest2.json,...]"
TEST_NAME="[test_name1,test_name2,...]"
TEST_BATCH=32
NUM_WORKERS=8

SAVE_PRED_TO_FILE=/path/to/predictions.json  # optional, if you want to save the predictions to a file, will slow down the evaluation speed. Set to `null` to disable.
PRETRAINED_NEMO=/path/to/EOU/model.nemo
CONFIG_NAME=fastconformer_transducer_bpe_streaming_xlarge

python speech_to_text_eou_eval.py \
    --config-name $CONFIG_NAME \
    ++save_pred_to_file=$SAVE_PRED_TO_FILE \
    ++init_from_nemo_model=$PRETRAINED_NEMO \
    ~model.train_ds \
    ~model.validation_ds \
    ++model.test_ds.defer_setup=true \
    ++model.test_ds.sample_rate=16000 \
    ++model.test_ds.manifest_filepath=$TEST_MANIFEST \
    ++model.test_ds.name=$TEST_NAME \
    ++model.test_ds.batch_size=$TEST_BATCH \
    ++model.test_ds.num_workers=$NUM_WORKERS \
    ++model.test_ds.drop_last=false \
    ++model.test_ds.force_finite=true \
    ++model.test_ds.shuffle=false \
    ++model.test_ds.pin_memory=true \
    exp_manager.create_wandb_logger=false

The script will show the WER metrics along with EOU metrics like latency, early cutoff rate, miss detection rate, etc.

5. Model deployment with voice agent

Please refer to the NeMo Voice Agent example for more details on how to deploy the ASR-EOU model with voice agent.