chore: import upstream snapshot with attribution

docs/source/asr/speaker_recognition/api.rst

@@ -0,0 +1,11 @@
+NeMo Speaker Recognition API
+=============================
+
+
+Model Classes
+-------------
+.. autoclass:: nemo.collections.asr.models.label_models.EncDecSpeakerLabelModel
+    :show-inheritance:
+    :members: setup_finetune_model, get_embedding, verify_speakers, verify_speakers_batch
+
+

docs/source/asr/speaker_recognition/configs.rst

@@ -0,0 +1,116 @@
+NeMo Speaker Recognition Configuration Files
+============================================
+
+This page covers NeMo configuration file setup that is specific to speaker recognition models.
+For general information about how to set up and run experiments that is common to all NeMo models (e.g.
+experiment manager and PyTorch Lightning trainer parameters), see the :doc:`../../core/core` page.
+
+The model section of NeMo speaker recognition configuration files will generally require information about the dataset(s) being
+used, the preprocessor for audio files, parameters for any augmentation being performed, as well as the
+model architecture specification.
+The sections on this page cover each of these in more detail.
+
+Example configuration files for all of the Speaker related scripts can be found in the
+config directory of the examples ``{NEMO_ROOT/examples/speaker_tasks/recognition/conf}``.
+
+
+Dataset Configuration
+---------------------
+
+Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
+``test_ds`` sections of your configuration file, respectively.
+Depending on the task, you may have arguments specifying the sample rate of your audio files, max time length to consider for each audio file , whether or not to shuffle the dataset, and so on.
+You may also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command line
+at run time.
+
+Any initialization parameters that are accepted for the Dataset class used in your experiment
+can be set in the config file.
+
+An example TitaNet train and validation configuration could look like (``{NEMO_ROOT}examples/speaker_tasks/recognition/conf/titanet-large.yaml``):
+
+.. code-block:: yaml
+
+  model:
+    train_ds:
+      manifest_filepath: ???
+      sample_rate: 16000
+      labels: None   # finds labels based on manifest file
+      batch_size: 32
+      trim_silence: False
+      shuffle: True
+
+    validation_ds:
+      manifest_filepath: ???
+      sample_rate: 16000
+      labels: None   # Keep None, to match with labels extracted during training
+      batch_size: 32
+      shuffle: False    # No need to shuffle the validation data
+
+      
+If you would like to use tarred dataset, have a look at the ASR :ref:`Tarred Datasets <Tarred_Datasets>` section.
+
+
+Preprocessor Configuration
+--------------------------
+Preprocessor helps to compute MFCC or mel spectrogram features that are given as inputs to model. 
+For details on how to write this section, refer to :ref:`Preprocessor Configuration <asr-configs-preprocessor-configuration>`.
+
+
+Augmentation Configurations
+---------------------------
+
+For TitaNet training we use on-the-fly augmentations with MUSAN and RIR impulses using ``noise`` augmentor section
+
+The following example sets up musan augmentation with audio files taken from manifest path and 
+minimum and maximum SNR specified with min_snr and max_snr respectively. This section can be added to 
+``train_ds`` part in model
+
+.. code-block:: yaml
+
+  model:
+    ...
+    train_ds:
+      ...
+      augmentor:
+        noise:
+          manifest_path: /path/to/musan/manifest_file
+          prob: 0.2  # probability to augment the incoming batch audio with augmentor data
+          min_snr_db: 5 
+          max_snr_db: 15        
+
+
+See the :class:`nemo.collections.asr.parts.preprocessing.perturb.AudioAugmentor`  API section for more details.
+
+
+Model Architecture Configurations
+---------------------------------
+
+Each configuration file should describe the model architecture being used for the experiment.
+Models in the NeMo ASR collection need a ``encoder`` section and a ``decoder`` section, with the ``_target_`` field
+specifying the module to use for each.
+
+The following sections go into more detail about the specific configurations of each model architecture.
+
+For more information about the TitaNet Encoder models, see the :doc:`Models <./models>` page.
+
+Decoder Configurations
+------------------------
+
+After features have been computed from TitaNet encoder, we pass these features to the decoder to compute embeddings and then to compute log probabilities
+for training models.
+
+.. code-block:: yaml
+
+  model:
+    ...
+    decoder:
+      _target_: nemo.collections.asr.modules.SpeakerDecoder
+      feat_in: *enc_feat_out
+      num_classes: 7205  # Total number of classes in voxceleb1,2 training manifest file 
+      pool_mode: attention # xvector, attention
+      emb_sizes: 192 # number of intermediate emb layers. can be comma separated for additional layers like 512,512
+      angular: true # if true then loss will be changed to angular softmax loss and consider scale and margin from loss section else train with cross-entropy loss
+    
+    loss:
+      scale: 30
+      margin 0.2

docs/source/asr/speaker_recognition/data/speaker_results.csv

@@ -0,0 +1,8 @@
+Model Name,Model Base Class,Model Card
+diar_sortformer_4spk-v1,SortformerEncLabelModel,"https://huggingface.co/nvidia/diar_sortformer_4spk-v1"
+diar_streaming_sortformer_4spk-v2,SortformerEncLabelModel,"https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2"
+diar_streaming_sortformer_4spk-v2.1,SortformerEncLabelModel,"https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1"
+titanet_large,EncDecSpeakerLabelModel,"https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_large"
+titanet_small,EncDecSpeakerLabelModel,"https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_small"
+speakerverification_speakernet,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet"
+ecapa_tdnn,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:ecapa_tdnn"

docs/source/asr/speaker_recognition/datasets.rst

@@ -0,0 +1,62 @@
+Datasets
+========
+
+.. _HI-MIA:
+
+HI-MIA
+--------
+
+Run the script to download and process ``hi-mia`` dataset in order to generate files in the supported format of  ``nemo_asr``. You should set the data folder of 
+hi-mia using ``--data_root``. These scripts are present in ``<nemo_root>/scripts``
+
+.. code-block:: bash
+
+    python get_hi-mia_data.py --data_root=<data directory> 
+
+After download and conversion, your `data` folder should contain directories with following set of files as:
+
+* `data/<set>/train.json`
+* `data/<set>/dev.json` 
+* `data/<set>/{set}_all.json` 
+* `data/<set>/utt2spk`
+
+
+All-other Datasets
+------------------
+
+These methods can be applied to any dataset to get similar training or inference manifest files.
+
+`filelist_to_manifest.py` script in `$<NeMo_root>/scripts/speaker_tasks/` folder generates manifest file from a text file containing paths to audio files. 
+
+sample `filelist.txt` file contents:
+
+.. code-block:: bash 
+
+    /data/datasets/voxceleb/data/dev/aac_wav/id00179/Q3G6nMr1ji0/00086.wav
+    /data/datasets/voxceleb/data/dev/aac_wav/id00806/VjpQLxHQQe4/00302.wav
+    /data/datasets/voxceleb/data/dev/aac_wav/id01510/k2tzXQXvNPU/00132.wav
+
+This list file is used to generate manifest file. This script has optional arguments to split the whole manifest file in to train and dev and also segment audio files to smaller segments for robust training (for testing, we don't need to create segments for each utterance).
+
+sample usage:
+
+.. code-block:: bash 
+    
+    python filelist_to_manifest.py --filelist=filelist.txt --id=-3 --out=speaker_manifest.json 
+
+This would create a manifest containing file contents as shown below:
+.. code-block:: json
+    
+    {"audio_filepath": "/data/datasets/voxceleb/data/dev/aac_wav/id00179/Q3G6nMr1ji0/00086.wav", "offset": 0, "duration": 4.16, "label": "id00179"}
+    {"audio_filepath": "/data/datasets/voxceleb/data/dev/aac_wav/id00806/VjpQLxHQQe4/00302.wav", "offset": 0, "duration": 12.288, "label": "id00806"}
+    {"audio_filepath": "/data/datasets/voxceleb/data/dev/aac_wav/id01510/k2tzXQXvNPU/00132.wav", "offset": 0, "duration": 4.608, "label": "id01510"}
+
+For other optional arguments like splitting manifest file to train and dev and for creating segements from each utterance refer to the arguments 
+described in the script.
+
+Tarred Datasets
+---------------
+
+Similarly to ASR, you can tar your audio files and use ASR Dataset class ``TarredAudioToSpeechLabelDataset`` (corresponding to the ``AudioToSpeechLabelDataset``) for this case.
+
+If you want to use tarred dataset, have a look at the ASR :ref:`Tarred Datasets <Tarred_Datasets>` section.

docs/source/asr/speaker_recognition/intro.rst

@@ -0,0 +1,23 @@
+Speaker Recognition (SR)
+========================
+
+Speaker recognition is a broad research area which solves two major tasks: speaker identification (what is the identity of the speaker?) and speaker verification (is the speaker who they claim to be?). We focus on text-independent speaker recognition when the identity of the speaker is based on how the speech is spoken, not necessarily in what is being said. Typically such speaker recognition systems operate on unconstrained speech utterances, which are converted into vectors of fixed length, called speaker embeddings. Speaker embeddings can also be used in automatic speech recognition (ASR) and speech synthesis.
+
+The goal of most speaker recognition systems is to get good speaker level representations that could help distinguish oneself from other speakers. To achieve this, we first train a neural network model in an end-to-end manner optimizing the encoder using cross-entropy or angular softmax loss. We modify the decoder to get these fixed size embeddings irrespective of the length of the audio input and employ a pooling strategy such as mean and variance based statistics pooling or attention based method to generate these embeddings.
+
+In speaker identification, we typically train on a larger training set with cross-entropy loss and fine-tune later on preferred set of labels where one would want to classify only known sets of speakers. 
+On the other hand, in speaker verification, we train an embedding extractor with angular softmax loss and compare the embeddings from one audio file coming from a single speaker with embeddings from an unknown speaker. For quantifying the similarity of the embeddings we use scoring techniques such as cosine similarity.
+
+The full documentation tree: 
+
+.. toctree::
+   :maxdepth: 8
+
+   models
+   configs
+   datasets
+   results
+   api
+   resources
+
+.. include:: resources.rst

docs/source/asr/speaker_recognition/models.rst

@@ -0,0 +1,72 @@
+Models
+======
+
+Examples of config files for all the below models can be found in the ``<NeMo_git_root>/examples/speaker_recognition/conf`` directory.
+
+For more information about the config files and how they should be structured, see the :doc:`./configs` page.
+
+Pretrained checkpoints for all of these models, as well as instructions on how to load them, can be found on the :doc:`./results` page.
+You can use the available checkpoints for immediate inference, or fine-tune them on your own datasets.
+The Checkpoints page also contains benchmark results for the available speaker recognition models.
+
+.. _TitaNet_model:
+
+TitaNet
+-----------
+
+TitaNet model :cite:`sr-models-koluguri2021titanet` is based on the ContextNet architecture :cite:`sr-models-han2020contextnet` for extracting speaker representations.
+We employ 1D depth-wise separable convolutions with Squeeze-and-Excitation (SE) layers with global context followed by channel attention based statistics pooling layer to map
+variable-length utterances to a fixed-length embedding (tvector). TitaNet is a scalable architecture and achieves state-of-the-art performance on speaker verification and diarization tasks.
+
+    .. image:: images/titanet_network.png
+        :align: center
+        :alt: speakernet model
+        :scale: 50%
+
+SpeakerNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecSpeakerLabelModel` class.
+
+SpeakerNet
+-----------
+
+The model is based on the QuartzNet ASR architecture :cite:`sr-models-koluguri2020speakernet`
+comprising of an encoder and decoder structure. We use the encoder of the QuartzNet model as a top-level feature extractor, and feed the output to the statistics pooling layer, where
+we compute the mean and variance across channel dimensions to capture the time-independent utterance-level speaker features.
+
+The QuartzNet encoder used for speaker embeddings shown in figure below has the following structure: a QuartzNet BxR
+model has B blocks, each with R sub-blocks. Each sub-block applies the following operations: a 1D convolution, batch norm, ReLU, and dropout. All sub-blocks in a block have the same number of output channels. These blocks are connected with residual connections. We use QuartzNet with 3 blocks, 2 sub-blocks, and 512 channels, as the Encoder for Speaker Embeddings. All conv layers have stride 1 and dilation 1.
+
+
+    .. image:: images/ICASPP_SpeakerNet.png
+        :align: center
+        :alt: speakernet model
+        :scale: 40%
+
+Top level acoustic Features, obtained from the output of
+encoder are used to compute intermediate features that are
+then passed to the decoder for getting utterance level speaker
+embeddings. The intermediate time-independent features are
+computed using a statistics pooling layer, where we compute the mean and standard deviation of features across
+time-channels, to get a time-independent feature representation S of size Batch_size × 3000.
+The intermediate features, S are passed through the Decoder consisting of two layers each of output size 512 for a
+linear transformation from S to the final number of classes
+N for the larger (L) model, and a single linear layer of output size 256 to the final number of classes N for the medium
+(M) model. We extract q-vectors after the final linear layer
+of fixed size 512, 256 for SpeakerNet-L and SpeakerNet-M
+models respectively.
+
+SpeakerNet models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecSpeakerLabelModel` class.
+
+ECAPA_TDNN
+----------
+
+The model is based on the paper "ECAPA_TDNN Embeddings for Speaker Diarization" :cite:`sr-models-Dawalatabad_2021` comprising an encoder of time dilation layers which are based on Emphasized Channel Attention, Propagation, and Aggregation. The ECAPA-TDNN model employs a channel and context dependent attention mechanism, Multi layer Feature Aggregation (MFA), as well as Squeeze-Excitation (SE) and residual blocks, due to faster training and inference we replacing residual blocks with group convolution blocks of single dilation. These models has shown good performance over various speaker tasks. 
+
+ecapa_tdnn models can be instantiated using the :class:`~nemo.collections.asr.models.EncDecSpeakerLabelModel` class.
+
+References
+-----------
+
+.. bibliography:: ../asr_all.bib
+    :style: plain
+    :labelprefix: SR-MODELS
+    :keyprefix: sr-models-

docs/source/asr/speaker_recognition/resources.rst

@@ -0,0 +1,22 @@
+
+Resource and Documentation Guide
+--------------------------------
+
+Hands-on speaker recognition tutorial notebooks can be found under
+`the speaker recognition tutorials folder <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/speaker_tasks/>`_. This and most other tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
+
+If you are looking for information about a particular SpeakerNet model, or would like to find out more about the model
+architectures available in the ``nemo_asr`` collection, check out the :doc:`Models <./models>` page.
+
+Documentation on dataset preprocessing can be found on the :doc:`Datasets <./datasets>` page.
+NeMo includes preprocessing and other scripts for speaker_recognition in <nemo/scripts/speaker_tasks/> folder, and this page contains instructions on running
+those scripts. It also includes guidance for creating your own NeMo-compatible dataset, if you have your own data.
+
+Information about how to load model checkpoints (either local files or pretrained ones from NGC), perform inference, as well as a list
+of the checkpoints available on NGC are located on the :doc:`Checkpoints <./results>` page.
+
+Documentation for configuration files specific to the ``nemo_asr`` models can be found on the
+:doc:`Configuration Files <./configs>` page.
+
+
+For a clear step-by-step tutorial we advise you to refer to the tutorials found in `folder <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/speaker_tasks/>`_.

docs/source/asr/speaker_recognition/results.rst

@@ -0,0 +1,144 @@
+Checkpoints
+===========
+
+There are two main ways to load pretrained checkpoints in NeMo:
+
+* Using the :code:`restore_from()` method to load a local checkpoint file (`.nemo`), or
+* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
+
+See the following sections for instructions and examples for each.
+
+Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning.
+For resuming an unfinished training experiment, please use the experiment manager to do so by setting the
+``resume_if_exists`` flag to True.
+
+Loading Local Checkpoints
+-------------------------
+
+NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
+You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
+
+If you have a local ``.nemo`` checkpoint that you'd like to load, simply use the :code:`restore_from()` method:
+
+.. code-block:: python
+
+  import nemo.collections.asr as nemo_asr
+  model = nemo_asr.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
+
+Where the model base class is the ASR model class of the original checkpoint, or the general `ASRModel` class.
+
+Speaker Label Inference
+------------------------
+
+The goal of speaker label inference is to infer speaker labels using a speaker model with known speaker labels from enrollment set. We provide `speaker_identification_infer.py` script for this purpose under `<NeMo_root>/examples/speaker_tasks/recognition` folder.
+Currently supported backends are cosine_similarity and neural classifier.
+
+The audio files should be 16KHz mono channel wav files.
+
+The script takes two manifest files: 
+
+* enrollment_manifest : This manifest contains enrollment data with known speaker labels.
+* test_manifest: This manifest contains test data for which we map speaker labels captured from enrollment manifest using one of provided backend
+
+sample format for each of these manifests is provided in `<NeMo_root>/examples/speaker_tasks/recognition/conf/speaker_identification_infer.yaml` config file.
+
+To infer speaker labels using cosine_similarity backend
+
+.. code-block:: bash
+  
+    python speaker_identification_infer.py data.enrollment_manifest=<path/to/enrollment_manifest> data.test_manifest=<path/to/test_manifest> backend.backend_model=cosine_similarity
+
+    
+Speaker Embedding Extraction
+-----------------------------
+Speaker Embedding Extraction, is to extract speaker embeddings for any wav file (from known or unknown speakers). We provide two ways to do this:
+
+* single Python liner for extracting embeddings from a single file 
+* Python script for extracting embeddings from a bunch of files provided through manifest file
+
+For extracting embeddings from a single file:
+
+.. code-block:: python
+
+  speaker_model = EncDecSpeakerLabelModel.from_pretrained(model_name="<pretrained_model_name or path/to/nemo/file>")
+  embs = speaker_model.get_embedding('<audio_path>')
+
+For extracting embeddings from a bunch of files:
+
+The audio files should be 16KHz mono channel wav files.
+
+Write audio files to a ``manifest.json`` file with lines as in format:
+
+.. code-block:: json
+    
+    {"audio_filepath": "<absolute path to dataset>/audio_file.wav", "duration": "duration of file in sec", "label": "speaker_id"}
+      
+This python call will download best pretrained model from NGC and writes embeddings pickle file to current working directory
+
+.. code-block:: bash
+  
+    python examples/speaker_tasks/recognition/extract_speaker_embeddings.py --manifest=manifest.json
+   
+or you can run `batch_inference()` to perform inference on the manifest with seleted batch_size to get embeddings
+
+.. code-block:: python
+
+  speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained(model_name="<pretrained_model_name or path/to/nemo/file>")
+  embs, logits, gt_labels, trained_labels = speaker_model.batch_inference(manifest, batch_size=32)
+
+Speaker Verification Inference
+------------------------------
+
+Speaker Verification is a task of verifying if two utterances are from the same speaker or not.
+
+We provide a helper function to verify the audio files (also in a batch) and return True if provided pair of audio files is from the same speaker, False otherwise.
+
+The audio files should be 16KHz mono channel wav files.
+
+.. code-block:: python
+
+  speaker_model = EncDecSpeakerLabelModel.from_pretrained(model_name="titanet_large")
+  decision = speaker_model.verify_speakers('path/to/one/audio_file','path/to/other/audio_file')
+  decisions = speaker_model.verify_speakers_batch([
+                                                  ('/path/to/audio_0_0', '/path/to/audio_0_1'),
+                                                  ('/path/to/audio_1_0', '/path/to/audio_1_1'),
+                                                  ('/path/to/audio_2_0', '/path/to/audio_2_1'),
+                                                  ('/path/to/audio_3_0', '/path/to/audio_3_1')
+                                                  ],  batch_size=4, device='cuda')
+
+
+NGC Pretrained Checkpoints
+--------------------------
+
+The SpeakerNet-ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
+`TitaNet <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_large>`_ , `ECAPA_TDNN <https://ngc.nvidia.com/catalog/models/nvidia:nemo:ecapa_tdnn>`_ and `Speaker_Verification <https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet>`_ model cards on NGC contain more information about each of the checkpoints available.
+
+The tables below list the speaker embedding extractor models available from NGC, and the models can be accessed via the
+:code:`from_pretrained()` method inside the EncDecSpeakerLabelModel Model class.
+
+In general, you can load any of these models with code in the following format:
+
+.. code-block:: python
+
+  import nemo.collections.asr as nemo_asr
+  model = nemo_asr.models.<MODEL_CLASS_NAME>.from_pretrained(model_name="<MODEL_NAME>")
+
+where the model name is the value under "Model Name" entry in the tables below.
+
+If you would like to programatically list the models available for a particular base class, you can use the
+:code:`list_available_models()` method.
+
+.. code-block:: python
+
+  nemo_asr.models.<MODEL_BASE_CLASS>.list_available_models()
+
+
+Speaker Recognition Models
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :file: data/speaker_results.csv
+   :align: left
+   :widths: 30, 30, 40
+   :header-rows: 1
+