chore: import upstream snapshot with attribution

examples/asr/speech_classification/README.md

@@ -0,0 +1,105 @@
+# Speech Classification
+
+This directory contains example scripts to train speech classification and voice activity detection models. There are two types of VAD models: Frame-VAD and Segment-VAD.
+
+## Frame-VAD
+
+The frame-level VAD model predicts for each frame of the audio whether it has speech or not. For example, with the default config file (`../conf/marblenet/marblenet_3x2x64_20ms.yaml`), the model provides a probability for each frame of 20ms length.
+
+### Training
+```sh
+python speech_to_frame_label.py \
+    --config-path=<path to directory of configs, e.g. "../conf/marblenet">
+    --config-name=<name of config without .yaml, e.g. "marblenet_3x2x64_20ms"> \
+    model.train_ds.manifest_filepath="[<path to train manifest1>,<path to train manifest2>]" \
+    model.validation_ds.manifest_filepath=["<path to val manifest1>","<path to val manifest2>"] \
+    trainer.devices=-1 \
+    trainer.accelerator="gpu" \
+    strategy="ddp" \
+    trainer.max_epochs=100
+```
+
+The input manifest must be a manifest json file, where each line is a Python dictionary. The fields ["audio_filepath", "offset", "duration",  "label"] are required. An example of a manifest file is:
+```
+{"audio_filepath": "/path/to/audio_file1", "offset": 0, "duration": 10000,  "label": "0 1 0 0 1"}
+{"audio_filepath": "/path/to/audio_file2", "offset": 0, "duration": 10000,  "label": "0 0 0 1 1 1 1 0 0"}
+```
+For example, if you have a 1s audio file, you'll need to have 50 frame labels in the manifest entry like "0 0 0 0 1 1 0 1 .... 0 1".
+However, shorter label strings are also supported for smaller file sizes. For example, you can prepare the `label` in 40ms frame, and the model will properly repeat the label for each 20ms frame.
+
+
+### Inference
+python frame_vad_infer.py \
+    --config-path="../conf/vad" --config-name="frame_vad_infer_postprocess" \
+    dataset=<Path of manifest file containing evaluation data. Audio files should have unique names>
+
+The manifest json file should have the following format (each line is a Python dictionary):
+```
+{"audio_filepath": "/path/to/audio_file1.wav", "offset": 0, "duration": 10000}  
+{"audio_filepath": "/path/to/audio_file2.wav", "offset": 0, "duration": 10000}  
+```
+
+#### Evaluation
+If you want to evaluate tne model's AUROC and DER performance, you need to set `evaluate: True` in config yaml (e.g., `../conf/vad/frame_vad_infer_postprocess.yaml`), and also provide groundtruth in label strings:
+```
+{"audio_filepath": "/path/to/audio_file1.wav", "offset": 0, "duration": 10000, "label": "0 1 0 0 0 1 1 1 0"}
+```
+or RTTM files:
+```
+{"audio_filepath": "/path/to/audio_file1.wav", "offset": 0, "duration": 10000, "rttm_filepath": "/path/to/rttm_file1.rttm"}
+```
+
+
+## Segment-VAD
+
+Segment-level VAD predicts a single label for each segment of audio (e.g., 0.63s by default).
+
+### Training
+```sh
+python speech_to_label.py \
+    --config-path=<path to dir of configs, e.g. "../conf/marblenet"> \
+    --config-name=<name of config without .yaml, e.g., "marblenet_3x2x64"> \
+    model.train_ds.manifest_filepath="[<path to train manifest1>,<path to train manifest2>]" \
+    model.validation_ds.manifest_filepath=["<path to val manifest1>","<path to val manifest2>"] \
+    trainer.devices=-1 \
+    trainer.accelerator="gpu" \
+    strategy="ddp" \
+    trainer.max_epochs=100
+```
+
+The input manifest must be a manifest json file, where each line is a Python dictionary. The fields ["audio_filepath", "offset", "duration",  "label"] are required. An example of a manifest file is:
+```
+{"audio_filepath": "/path/to/audio_file1", "offset": 0, "duration": 0.63,  "label": "0"}
+{"audio_filepath": "/path/to/audio_file2", "offset": 0, "duration": 0.63,  "label": "1"}
+```
+
+
+### Inference
+```sh
+python vad_infer.py \
+    --config-path="../conf/vad" \
+    --config-name="vad_inference_postprocessing.yaml"
+    dataset=<Path of json file of evaluation data. Audio files should have unique names>
+```
+The manifest json file should have the following format (each line is a Python dictionary):
+```
+{"audio_filepath": "/path/to/audio_file1.wav", "offset": 0, "duration": 10000}  
+{"audio_filepath": "/path/to/audio_file2.wav", "offset": 0, "duration": 10000}  
+```
+
+
+## Visualization
+
+To visualize the VAD outputs, you can use the `nemo.collections.asr.parts.utils.vad_utils.plot_sample_from_rttm` function, which takes an audio file and an RTTM file as input, and plots the audio waveform and the VAD labels. Since the VAD inference script will output a json manifest `manifest_vad_out.json` by default, you can create a Jupyter Notebook with the following script and fill in the paths using the output manifest:
+```python
+from nemo.collections.asr.parts.utils.vad_utils import plot_sample_from_rttm
+
+plot_sample_from_rttm(
+    audio_file="/path/to/audio_file.wav",
+    rttm_file="/path/to/rttm_file.rttm",
+    offset=0.0,
+    duration=1000,
+    save_path="vad_pred.png"
+)
+```
+

examples/asr/speech_classification/frame_vad_infer.py

@@ -0,0 +1,211 @@
+# Copyright (c) 2023, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This script peforms VAD on each 20ms frames of the input audio files. 
+Postprocessing is also performed to generate speech segments and store them as RTTM files.
+Long audio files will be splitted into smaller chunks to avoid OOM issues, but the frames close
+to the split points might have worse performance due to truncated context.
+
+## Usage:
+python frame_vad_infer.py \
+    --config-path="../conf/vad" --config-name="frame_vad_infer_postprocess" \
+    input_manifest=<Path of manifest file containing evaluation data. Audio files should have unique names> \
+    output_dir=<Path of output directory>
+
+The manifest json file should have the following format (each line is a Python dictionary):
+{"audio_filepath": "/path/to/audio_file1", "offset": 0, "duration": 10000}  
+{"audio_filepath": "/path/to/audio_file2", "offset": 0, "duration": 10000}  
+
+If you want to evaluate tne model's AUROC and DER performance, you need to set `evaluate=True` in config yaml,
+and also provide groundtruth in either RTTM files or label strings:
+{"audio_filepath": "/path/to/audio_file1", "offset": 0, "duration": 10000, "label": "0 1 0 0 0 1 1 1 0"}
+or
+{"audio_filepath": "/path/to/audio_file1", "offset": 0, "duration": 10000, "rttm_filepath": "/path/to/rttm_file1.rttm"}
+
+"""
+
+import os
+from pathlib import Path
+
+import torch
+
+from nemo.collections.asr.parts.utils.manifest_utils import write_manifest
+from nemo.collections.asr.parts.utils.vad_utils import (
+    frame_vad_eval_detection_error,
+    frame_vad_infer_load_manifest,
+    generate_overlap_vad_seq,
+    generate_vad_frame_pred,
+    generate_vad_segment_table,
+    init_frame_vad_model,
+    prepare_manifest,
+)
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+
+@hydra_runner(config_path="../conf/vad", config_name="frame_vad_infer_postprocess")
+def main(cfg):
+    if not cfg.input_manifest:
+        raise ValueError("You must input the path of json file of evaluation data")
+    output_dir = cfg.output_dir if cfg.output_dir else "frame_vad_outputs"
+    if os.path.exists(output_dir):
+        logging.warning(
+            f"Output directory {output_dir} already exists, use this only if you're tuning post-processing params."
+        )
+    Path(output_dir).mkdir(parents=True, exist_ok=True)
+
+    cfg.frame_out_dir = os.path.join(output_dir, "frame_preds")
+    cfg.smoothing_out_dir = os.path.join(output_dir, "smoothing_preds")
+    cfg.rttm_out_dir = os.path.join(output_dir, "rttm_preds")
+
+    # each line of input_manifest should be have different audio_filepath and unique name to simplify edge cases or conditions
+    logging.info(f"Loading manifest file {cfg.input_manifest}")
+    manifest_orig, key_labels_map, key_rttm_map = frame_vad_infer_load_manifest(cfg)
+
+    # Prepare manifest for streaming VAD
+    manifest_vad_input = cfg.input_manifest
+    if cfg.prepare_manifest.auto_split:
+        logging.info("Split long audio file to avoid CUDA memory issue")
+        logging.debug("Try smaller split_duration if you still have CUDA memory issue")
+        config = {
+            'input': manifest_vad_input,
+            'window_length_in_sec': cfg.vad.parameters.window_length_in_sec,
+            'split_duration': cfg.prepare_manifest.split_duration,
+            'num_workers': cfg.num_workers,
+            'prepared_manifest_vad_input': cfg.prepared_manifest_vad_input,
+            'out_dir': output_dir,
+        }
+        manifest_vad_input = prepare_manifest(config)
+    else:
+        logging.warning(
+            "If you encounter CUDA memory issue, try splitting manifest entry by split_duration to avoid it."
+        )
+
+    torch.set_grad_enabled(False)
+    vad_model = init_frame_vad_model(cfg.vad.model_path)
+
+    # setup_test_data
+    vad_model.setup_test_data(
+        test_data_config={
+            'batch_size': 1,
+            'sample_rate': 16000,
+            'manifest_filepath': manifest_vad_input,
+            'labels': ['infer'],
+            'num_workers': cfg.num_workers,
+            'shuffle': False,
+            'normalize_audio_db': cfg.vad.parameters.normalize_audio_db,
+        }
+    )
+
+    vad_model = vad_model.to(device)
+    vad_model.eval()
+
+    if not os.path.exists(cfg.frame_out_dir):
+        logging.info(f"Frame predictions do not exist at {cfg.frame_out_dir}, generating frame prediction.")
+        os.mkdir(cfg.frame_out_dir)
+        extract_frame_preds = True
+    else:
+        logging.info(f"Frame predictions already exist at {cfg.frame_out_dir}, skipping frame prediction generation.")
+        extract_frame_preds = False
+
+    if extract_frame_preds:
+        logging.info("Generating frame-level prediction ")
+        pred_dir = generate_vad_frame_pred(
+            vad_model=vad_model,
+            window_length_in_sec=cfg.vad.parameters.window_length_in_sec,
+            shift_length_in_sec=cfg.vad.parameters.shift_length_in_sec,
+            manifest_vad_input=manifest_vad_input,
+            out_dir=cfg.frame_out_dir,
+        )
+        logging.info(f"Finish generating VAD frame level prediction. You can find the prediction in {pred_dir}")
+    else:
+        pred_dir = cfg.frame_out_dir
+
+    frame_length_in_sec = cfg.vad.parameters.shift_length_in_sec
+
+    # overlap smoothing filter
+    if cfg.vad.parameters.smoothing:
+        # Generate predictions with overlapping input segments. Then a smoothing filter is applied to decide the label for a frame spanned by multiple segments.
+        # smoothing_method would be either in majority vote (median) or average (mean)
+        logging.info("Generating predictions with overlapping input segments")
+        smoothing_pred_dir = generate_overlap_vad_seq(
+            frame_pred_dir=pred_dir,
+            smoothing_method=cfg.vad.parameters.smoothing,
+            overlap=cfg.vad.parameters.overlap,
+            window_length_in_sec=cfg.vad.parameters.window_length_in_sec,
+            shift_length_in_sec=cfg.vad.parameters.shift_length_in_sec,
+            num_workers=cfg.num_workers,
+            out_dir=cfg.smoothing_out_dir,
+        )
+        logging.info(
+            f"Finish generating predictions with overlapping input segments with smoothing_method={cfg.vad.parameters.smoothing} and overlap={cfg.vad.parameters.overlap}"
+        )
+        pred_dir = smoothing_pred_dir
+
+    # postprocessing and generate speech segments
+    logging.info("Converting frame level prediction to RTTM files.")
+    rttm_out_dir = generate_vad_segment_table(
+        vad_pred_dir=pred_dir,
+        postprocessing_params=cfg.vad.parameters.postprocessing,
+        frame_length_in_sec=frame_length_in_sec,
+        num_workers=cfg.num_workers,
+        use_rttm=cfg.vad.use_rttm,
+        out_dir=cfg.rttm_out_dir,
+    )
+    logging.info(
+        f"Finish generating speech semgents table with postprocessing_params: {cfg.vad.parameters.postprocessing}"
+    )
+
+    logging.info("Writing VAD output to manifest")
+    key_pred_rttm_map = {}
+    manifest_new = []
+    for entry in manifest_orig:
+        key = Path(entry['audio_filepath']).stem
+        entry['rttm_filepath'] = Path(os.path.join(rttm_out_dir, key + ".rttm")).absolute().as_posix()
+        if not Path(entry['rttm_filepath']).is_file():
+            logging.warning(f"Not able to find {entry['rttm_filepath']} for {entry['audio_filepath']}")
+            entry['rttm_filepath'] = ""
+        manifest_new.append(entry)
+        key_pred_rttm_map[key] = entry['rttm_filepath']
+
+    if not cfg.out_manifest_filepath:
+        out_manifest_filepath = os.path.join(output_dir, "manifest_vad_output.json")
+    else:
+        out_manifest_filepath = cfg.out_manifest_filepath
+    write_manifest(out_manifest_filepath, manifest_new)
+    logging.info(f"Finished writing VAD output to manifest: {out_manifest_filepath}")
+
+    if cfg.get("evaluate", False):
+        logging.info("Evaluating VAD results")
+        auroc, report = frame_vad_eval_detection_error(
+            pred_dir=pred_dir,
+            key_labels_map=key_labels_map,
+            key_rttm_map=key_rttm_map,
+            key_pred_rttm_map=key_pred_rttm_map,
+            frame_length_in_sec=frame_length_in_sec,
+        )
+        DetER = report.iloc[[-1]][('detection error rate', '%')].item()
+        FA = report.iloc[[-1]][('false alarm', '%')].item()
+        MISS = report.iloc[[-1]][('miss', '%')].item()
+        logging.info(f"AUROC: {auroc:.4f}")
+        logging.info(f"DetER={DetER:0.4f}, False Alarm={FA:0.4f}, Miss={MISS:0.4f}")
+        logging.info(f"with params: {cfg.vad.parameters.postprocessing}")
+    logging.info("Done!")
+
+
+if __name__ == "__main__":
+    main()  # pylint: disable=no-value-for-parameter

examples/asr/speech_classification/speech_to_frame_label.py

@@ -0,0 +1,71 @@
+# Copyright (c) 2023, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+The script trains a model that peforms classification on each frame of the input audio. 
+The default config (i.e., marblenet_3x2x64_20ms.yaml) outputs 20ms frames.
+
+## Training
+```sh
+python speech_to_frame_label.py \
+    --config-path=<path to dir of configs e.g. "../conf/marblenet">
+    --config-name=<name of config without .yaml e.g. "marblenet_3x2x64_20ms"> \
+    model.train_ds.manifest_filepath="<path to train manifest>" \
+    model.train_ds.augmentor.noise.manifest_path="<path to noise manifest>" \
+    model.validation_ds.manifest_filepath=["<path to val manifest>","<path to test manifest>"] \
+    trainer.devices=2 \
+    trainer.accelerator="gpu" \
+    strategy="ddp" \
+    trainer.max_epochs=200
+```
+
+The input manifest must be a manifest json file, where each line is a Python dictionary. The fields ["audio_filepath", "offset", "duration",  "label"] are required. An example of a manifest file is:
+```
+{"audio_filepath": "/path/to/audio_file1", "offset": 0, "duration": 10000,  "label": "0 1 0 0 1"}
+{"audio_filepath": "/path/to/audio_file2", "offset": 0, "duration": 10000,  "label": "0 0 0 1 1 1 1 0 0"}
+```
+For example, if you have a 1s audio file, you'll need to have 50 frame labels in the manifest entry like "0 0 0 0 1 1 0 1 .... 0 1".
+However, shorter label strings are also supported for smaller file sizes. For example, you can prepare the `label` in 40ms frame, and the model will properly repeat the label for each 20ms frame.
+
+"""
+
+import lightning.pytorch as pl
+from omegaconf import OmegaConf
+from nemo.collections.asr.models.classification_models import EncDecFrameClassificationModel
+
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+from nemo.utils.exp_manager import exp_manager
+
+
+@hydra_runner(config_path="../conf/marblenet", config_name="marblenet_3x2x64_20ms")
+def main(cfg):
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+
+    trainer = pl.Trainer(**cfg.trainer)
+    exp_manager(trainer, cfg.get("exp_manager", None))
+    model = EncDecFrameClassificationModel(cfg=cfg.model, trainer=trainer)
+
+    # Initialize the weights of the model from another model, if provided via config
+    model.maybe_init_from_pretrained_checkpoint(cfg)
+
+    trainer.fit(model)
+
+    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
+        if model.prepare_test(trainer):
+            trainer.test(model)
+
+
+if __name__ == '__main__':
+    main()  # noqa pylint: disable=no-value-for-parameter

examples/asr/speech_classification/speech_to_label.py

@@ -0,0 +1,180 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+# Task 1: Speech Command Recognition
+
+## Preparing the dataset
+Use the `process_speech_commands_data.py` script under <NEMO_ROOT>/scripts/dataset_processing in order to prepare the dataset.
+
+```sh
+python <NEMO_ROOT>/scripts/dataset_processing/process_speech_commands_data.py \
+    --data_root=<absolute path to where the data should be stored> \
+    --data_version=<either 1 or 2, indicating version of the dataset> \
+    --class_split=<either "all" or "sub", indicates whether all 30/35 classes should be used, or the 10+2 split should be used> \
+    --rebalance \
+    --log
+```
+
+## Train to convergence
+```sh
+python speech_to_label.py \
+    --config-path="../conf/marblenet" \
+    --config-name="marblenet_3x2x64" \
+    model.train_ds.manifest_filepath="<path to train manifest>" \
+    model.validation_ds.manifest_filepath=["<path to val manifest>","<path to test manifest>"] \
+    trainer.devices=2 \
+    trainer.accelerator="gpu" \
+    strategy="ddp" \
+    trainer.max_epochs=200 \
+    exp_manager.create_wandb_logger=True \
+    exp_manager.wandb_logger_kwargs.name="MarbleNet-3x2x64" \
+    exp_manager.wandb_logger_kwargs.project="MarbleNet" \
+    +trainer.precision=16
+```
+
+
+# Task 2: Voice Activity Detection
+
+## Preparing the dataset
+Use the `process_vad_data.py` script under <NEMO_ROOT>/scripts/dataset_processing in order to prepare the dataset.
+
+```sh
+python process_vad_data.py \
+    --out_dir=<output path to where the generated manifest should be stored> \
+    --speech_data_root=<path where the speech data are stored> \
+    --background_data_root=<path where the background data are stored> \
+    --rebalance_method=<'under' or 'over' of 'fixed'> \
+    --log
+    (Optional --demo (for demonstration in tutorial). If you want to use your own background noise data, make sure to delete --demo)
+```
+
+## Train to convergence
+```sh
+python speech_to_label.py \
+    --config-path=<path to dir of configs e.g. "conf">
+    --config-name=<name of config without .yaml e.g. "marblenet_3x2x64"> \
+    model.train_ds.manifest_filepath="<path to train manifest>" \
+    model.validation_ds.manifest_filepath=["<path to val manifest>","<path to test manifest>"] \
+    trainer.devices=2 \
+    trainer.accelerator="gpu" \
+    strategy="ddp" \
+    trainer.max_epochs=200 \
+    exp_manager.create_wandb_logger=True \
+    exp_manager.wandb_logger_kwargs.name="MarbleNet-3x2x64-vad" \
+    exp_manager.wandb_logger_kwargs.project="MarbleNet-vad" \
+    +trainer.precision=16
+```
+
+# Task 3: Language Identification
+
+## Preparing the dataset
+Use the `filelist_to_manifest.py` script under <NEMO_ROOT>/scripts/speaker_tasks in order to prepare the dataset.
+```
+
+## Train to convergence
+```sh
+python speech_to_label.py \
+    --config-path=<path to dir of configs e.g. "../conf/lang_id">
+    --config-name=<name of config without .yaml e.g. "titanet_large"> \
+    model.train_ds.manifest_filepath="<path to train manifest>" \
+    model.validation_ds.manifest_filepath="<path to val manifest>" \
+    model.train_ds.augmentor.noise.manifest_path="<path to noise manifest>" \
+    model.train_ds.augmentor.impulse.manifest_path="<path to impulse manifest>" \
+    model.decoder.num_classes=<num of languages> \
+    trainer.devices=2 \
+    trainer.max_epochs=40 \
+    exp_manager.create_wandb_logger=True \
+    exp_manager.wandb_logger_kwargs.name="titanet" \
+    exp_manager.wandb_logger_kwargs.project="langid" \
+    +exp_manager.checkpoint_callback_params.monitor="val_acc_macro" \
+    +exp_manager.checkpoint_callback_params.mode="max" \
+    +trainer.precision=16 \
+```
+
+
+# Optional: Use tarred dataset to speed up data loading. Apply to both tasks.
+## Prepare tarred dataset. 
+   Prepare ONE manifest that contains all training data you would like to include. Validation should use non-tarred dataset.
+   Note that it's possible that tarred datasets impacts validation scores because it drop values in order to have same amount of files per tarfile; 
+   Scores might be off since some data is missing. 
+
+   Use the `convert_to_tarred_audio_dataset.py` script under <NEMO_ROOT>/scripts/speech_recognition in order to prepare tarred audio dataset.
+   For details, please see TarredAudioToClassificationLabelDataset in <NEMO_ROOT>/nemo/collections/asr/data/audio_to_label.py
+
+python speech_to_label.py \
+    --config-path=<path to dir of configs e.g. "conf">
+    --config-name=<name of config without .yaml e.g. "marblenet_3x2x64"> \
+    model.train_ds.manifest_filepath=<path to train tarred_audio_manifest.json> \
+    model.train_ds.is_tarred=True \
+    model.train_ds.tarred_audio_filepaths=<path to train tarred audio dataset e.g. audio_{0..2}.tar> \
+    +model.train_ds.num_worker=<num_shards used generating tarred dataset> \
+    model.validation_ds.manifest_filepath=<path to validation audio_manifest.json>\
+    trainer.devices=2 \
+    trainer.accelerator="gpu" \
+    strategy="ddp" \
+    trainer.max_epochs=200 \
+    exp_manager.create_wandb_logger=True \
+    exp_manager.wandb_logger_kwargs.name="MarbleNet-3x2x64-vad" \
+    exp_manager.wandb_logger_kwargs.project="MarbleNet-vad" \
+    +trainer.precision=16
+
+# Fine-tune a model
+
+For documentation on fine-tuning this model, please visit -
+https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/configs.html#fine-tuning-configurations
+
+# Pretrained Models
+
+For documentation on existing pretrained models, please visit -
+https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speech_classification/results.html#
+
+"""
+import lightning.pytorch as pl
+import torch
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.models import EncDecClassificationModel, EncDecSpeakerLabelModel
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+from nemo.utils.exp_manager import exp_manager
+
+
+@hydra_runner(config_path="../conf/marblenet", config_name="marblenet_3x2x64")
+def main(cfg):
+
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+
+    trainer = pl.Trainer(**cfg.trainer)
+    exp_manager(trainer, cfg.get("exp_manager", None))
+
+    if 'titanet' in cfg.name.lower():
+        model = EncDecSpeakerLabelModel(cfg=cfg.model, trainer=trainer)
+    else:
+        model = EncDecClassificationModel(cfg=cfg.model, trainer=trainer)
+
+    # Initialize the weights of the model from another model, if provided via config
+    model.maybe_init_from_pretrained_checkpoint(cfg)
+    trainer.fit(model)
+    torch.distributed.destroy_process_group()
+
+    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
+        if trainer.is_global_zero:
+            trainer = pl.Trainer(devices=1, accelerator=cfg.trainer.accelerator, strategy=cfg.trainer.strategy)
+            if model.prepare_test(trainer):
+                trainer.test(model)
+
+
+if __name__ == '__main__':
+    main()  # noqa pylint: disable=no-value-for-parameter

examples/asr/speech_classification/vad_infer.py

@@ -0,0 +1,176 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+During inference, we perform frame-level prediction by two approaches: 
+    1) shift the window of length window_length_in_sec (e.g. 0.63s) by shift_length_in_sec (e.g. 10ms) to generate the frame and use the prediction of the window to represent the label for the frame;
+       [this script demonstrate how to do this approach]
+    2) generate predictions with overlapping input segments. Then a smoothing filter is applied to decide the label for a frame spanned by multiple segments. 
+       [get frame level prediction by this script and use vad_overlap_posterior.py in NeMo/scripts/voice_activity_detection
+       One can also find posterior about converting frame level prediction 
+       to speech/no-speech segment in start and end times format in that script.]
+       
+       Image https://raw.githubusercontent.com/NVIDIA/NeMo/main/tutorials/asr/images/vad_post_overlap_diagram.png 
+       will help you understand this method.
+
+This script will also help you perform postprocessing and generate speech segments if needed
+
+Usage:
+python vad_infer.py --config-path="../conf/vad" --config-name="vad_inference_postprocessing.yaml" dataset=<Path of json file of evaluation data. Audio files should have unique names>
+
+"""
+import json
+import os
+
+import torch
+
+from nemo.collections.asr.parts.utils.speaker_utils import write_rttm2manifest
+from nemo.collections.asr.parts.utils.vad_utils import (
+    generate_overlap_vad_seq,
+    generate_vad_frame_pred,
+    generate_vad_segment_table,
+    init_vad_model,
+    prepare_manifest,
+)
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+
+device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+
+
+@hydra_runner(config_path="../conf/vad", config_name="vad_inference_postprocessing.yaml")
+def main(cfg):
+    if not cfg.dataset:
+        raise ValueError("You must input the path of json file of evaluation data")
+
+    # each line of dataset should be have different audio_filepath and unique name to simplify edge cases or conditions
+    key_meta_map = {}
+    with open(cfg.dataset, 'r') as manifest:
+        for line in manifest.readlines():
+            audio_filepath = json.loads(line.strip())['audio_filepath']
+            uniq_audio_name = audio_filepath.split('/')[-1].rsplit('.', 1)[0]
+            if uniq_audio_name in key_meta_map:
+                raise ValueError("Please make sure each line is with different audio_filepath! ")
+            key_meta_map[uniq_audio_name] = {'audio_filepath': audio_filepath}
+
+    # Prepare manifest for streaming VAD
+    manifest_vad_input = cfg.dataset
+    if cfg.prepare_manifest.auto_split:
+        logging.info("Split long audio file to avoid CUDA memory issue")
+        logging.debug("Try smaller split_duration if you still have CUDA memory issue")
+        config = {
+            'input': manifest_vad_input,
+            'window_length_in_sec': cfg.vad.parameters.window_length_in_sec,
+            'split_duration': cfg.prepare_manifest.split_duration,
+            'num_workers': cfg.num_workers,
+            'prepared_manifest_vad_input': cfg.prepared_manifest_vad_input,
+        }
+        manifest_vad_input = prepare_manifest(config)
+    else:
+        logging.warning(
+            "If you encounter CUDA memory issue, try splitting manifest entry by split_duration to avoid it."
+        )
+
+    torch.set_grad_enabled(False)
+    vad_model = init_vad_model(cfg.vad.model_path)
+
+    # setup_test_data
+    vad_model.setup_test_data(
+        test_data_config={
+            'vad_stream': True,
+            'sample_rate': 16000,
+            'manifest_filepath': manifest_vad_input,
+            'labels': [
+                'infer',
+            ],
+            'num_workers': cfg.num_workers,
+            'shuffle': False,
+            'window_length_in_sec': cfg.vad.parameters.window_length_in_sec,
+            'shift_length_in_sec': cfg.vad.parameters.shift_length_in_sec,
+            'trim_silence': False,
+            'normalize_audio': cfg.vad.parameters.normalize_audio,
+        }
+    )
+
+    vad_model = vad_model.to(device)
+    vad_model.eval()
+
+    if not os.path.exists(cfg.frame_out_dir):
+        os.mkdir(cfg.frame_out_dir)
+    else:
+        logging.warning(
+            "Note frame_out_dir exists. If new file has same name as file inside existing folder, it will append result to existing file and might cause mistakes for next steps."
+        )
+
+    logging.info("Generating frame level prediction ")
+    pred_dir = generate_vad_frame_pred(
+        vad_model=vad_model,
+        window_length_in_sec=cfg.vad.parameters.window_length_in_sec,
+        shift_length_in_sec=cfg.vad.parameters.shift_length_in_sec,
+        manifest_vad_input=manifest_vad_input,
+        out_dir=cfg.frame_out_dir,
+    )
+    logging.info(
+        f"Finish generating VAD frame level prediction with window_length_in_sec={cfg.vad.parameters.window_length_in_sec} and shift_length_in_sec={cfg.vad.parameters.shift_length_in_sec}"
+    )
+    frame_length_in_sec = cfg.vad.parameters.shift_length_in_sec
+
+    # overlap smoothing filter
+    if cfg.vad.parameters.smoothing:
+        # Generate predictions with overlapping input segments. Then a smoothing filter is applied to decide the label for a frame spanned by multiple segments.
+        # smoothing_method would be either in majority vote (median) or average (mean)
+        logging.info("Generating predictions with overlapping input segments")
+        smoothing_pred_dir = generate_overlap_vad_seq(
+            frame_pred_dir=pred_dir,
+            smoothing_method=cfg.vad.parameters.smoothing,
+            overlap=cfg.vad.parameters.overlap,
+            window_length_in_sec=cfg.vad.parameters.window_length_in_sec,
+            shift_length_in_sec=cfg.vad.parameters.shift_length_in_sec,
+            num_workers=cfg.num_workers,
+            out_dir=cfg.smoothing_out_dir,
+        )
+        logging.info(
+            f"Finish generating predictions with overlapping input segments with smoothing_method={cfg.vad.parameters.smoothing} and overlap={cfg.vad.parameters.overlap}"
+        )
+        pred_dir = smoothing_pred_dir
+        frame_length_in_sec = 0.01
+
+    # postprocessing and generate speech segments
+    if cfg.gen_seg_table:
+        logging.info("Converting frame level prediction to speech/no-speech segment in start and end times format.")
+        table_out_dir = generate_vad_segment_table(
+            vad_pred_dir=pred_dir,
+            postprocessing_params=cfg.vad.parameters.postprocessing,
+            frame_length_in_sec=frame_length_in_sec,
+            num_workers=cfg.num_workers,
+            out_dir=cfg.table_out_dir,
+        )
+        logging.info(
+            f"Finish generating speech semgents table with postprocessing_params: {cfg.vad.parameters.postprocessing}"
+        )
+
+    if cfg.write_to_manifest:
+        for i in key_meta_map:
+            key_meta_map[i]['rttm_filepath'] = os.path.join(table_out_dir, i + ".txt")
+
+        if not cfg.out_manifest_filepath:
+            out_manifest_filepath = "vad_out.json"
+        else:
+            out_manifest_filepath = cfg.out_manifest_filepath
+        out_manifest_filepath = write_rttm2manifest(key_meta_map, out_manifest_filepath)
+        logging.info(f"Writing VAD output to manifest: {out_manifest_filepath}")
+
+
+if __name__ == '__main__':
+    main()