chore: import upstream snapshot with attribution

docs/source/asr/speaker_diarization/api.rst

@@ -0,0 +1,26 @@
+NeMo Speaker Diarization API
+=============================
+
+
+Model Classes
+-------------
+
+.. autoclass:: nemo.collections.asr.models.ClusteringDiarizer
+    :show-inheritance:
+    :members:  
+
+.. autoclass:: nemo.collections.asr.models.SortformerEncLabelModel
+    :show-inheritance:
+    :members: list_available_models, setup_training_data, setup_validation_data, setup_test_data, process_signal, forward, forward_infer, frontend_encoder, diarize, training_step, validation_step, multi_validation_epoch_end, _get_aux_train_evaluations, _get_aux_validation_evaluations, _init_loss_weights, _init_eval_metrics, _reset_train_metrics, _reset_valid_metrics, _setup_diarize_dataloader, _diarize_forward, _diarize_output_processing, test_batch, _get_aux_test_batch_evaluations, on_validation_epoch_end
+
+Mixins
+------
+
+.. autoclass:: nemo.collections.asr.parts.mixins.DiarizationMixin
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.mixins.diarization.SpkDiarizationMixin
+    :show-inheritance:
+    :members: diarize, diarize_generator, _diarize_on_begin, _diarize_input_processing, _diarize_input_manifest_processing, _setup_diarize_dataloader, _diarize_forward, _diarize_output_processing, _diarize_on_end, _input_audio_to_rttm_processing, get_value_from_diarization_config
+    

docs/source/asr/speaker_diarization/configs.rst

@@ -0,0 +1,231 @@
+Speaker Diarization Configuration Files
+========================================
+
+.. note::
+
+   For the full configuration files, see the YAML configs on GitHub:
+
+   - `sortformer_diarizer_hybrid_loss_4spk-v1.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml>`__
+   - `streaming_sortformer_diarizer_4spk-v2.yaml <https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/diarization/conf/neural_diarizer/streaming_sortformer_diarizer_4spk-v2.yaml>`__
+
+Hydra Configurations for Sortformer Diarizer Training
+-----------------------------------------------------
+
+Sortformer Diarizer is an end-to-end speaker diarization model that is solely based on Transformer-encoder type of architecture.
+Model name convention for Sortformer Diarizer: ``sortformer_diarizer_<loss_type>_<speaker count limit>-<version>.yaml``
+
+* Example: ``<NeMo_root>/examples/speaker_tasks/diarization/neural_diarizer/conf/sortformer_diarizer_hybrid_loss_4spk-v1.yaml``
+
+Key parameters:
+
+.. code-block:: yaml
+
+  name: "SortformerDiarizer"
+  batch_size: 8
+
+  model:
+    sample_rate: 16000
+    pil_weight: 0.5       # Weight for Permutation Invariant Loss (PIL)
+    ats_weight: 0.5       # Weight for Arrival Time Sort (ATS) loss
+    max_num_of_spks: 4    # Maximum number of speakers per model
+
+    model_defaults:
+      fc_d_model: 512     # Hidden dimension size of the Fast-Conformer Encoder
+      tf_d_model: 192     # Hidden dimension size of the Transformer Encoder
+
+    train_ds:
+      manifest_filepath: ???
+      session_len_sec: 90
+      # ...
+
+    preprocessor:
+      _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+      normalize: "per_feature"
+      window_stride: 0.01
+      features: 80
+
+    encoder:
+      n_layers: 18
+      d_model: ${model.model_defaults.fc_d_model}
+      subsampling: dw_striding
+      subsampling_factor: 8
+
+    transformer_encoder:
+      num_layers: 18
+      hidden_size: ${model.model_defaults.tf_d_model}
+      num_attention_heads: 8
+
+Hydra Configurations for Streaming Sortformer Diarizer Training
+----------------------------------------------------------------
+
+Model name convention for Streaming Sortformer Diarizer: ``streaming_sortformer_diarizer_<speaker count limit>-<version>.yaml``
+
+* Example: ``<NeMo_root>/examples/speaker_tasks/diarization/neural_diarizer/conf/streaming_sortformer_diarizer_4spk-v2.yaml``
+
+The Streaming Sortformer config extends the offline config with ``streaming_mode: True`` and additional speaker cache parameters:
+
+.. code-block:: yaml
+
+  name: "StreamingSortformerDiarizer"
+  batch_size: 4
+
+  model:
+    sample_rate: 16000
+    pil_weight: 0.5
+    ats_weight: 0.5
+    max_num_of_spks: 4
+    streaming_mode: True
+
+    model_defaults:
+      fc_d_model: 512
+      tf_d_model: 192
+
+    preprocessor:
+      _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+      normalize: "NA"  # Required for streaming (no per-feature normalization)
+      window_stride: 0.01
+      features: 128
+
+    sortformer_modules:
+      num_spks: ${model.max_num_of_spks}
+      # Streaming-specific parameters
+      spkcache_len: 188          # Length of speaker cache buffer (frames for all speakers)
+      chunk_len: 188             # Number of frames processed per streaming chunk
+       chunk_left_context: 1
+      chunk_right_context: 1
+      # ...
+
+    encoder:
+      n_layers: 17
+      d_model: ${model.model_defaults.fc_d_model}
+      subsampling: dw_striding
+      subsampling_factor: 8
+
+    transformer_encoder:
+      num_layers: 18
+      hidden_size: ${model.model_defaults.tf_d_model}
+      num_attention_heads: 8
+
+See the full YAML configs on GitHub: `Sortformer <https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml>`__ · `Streaming Sortformer <https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/diarization/conf/neural_diarizer/streaming_sortformer_diarizer_4spk-v2.yaml>`__
+
+
+Hydra Configurations for (Streaming) Sortformer Diarization Post-processing
+-----------------------------------------------------------------------------
+
+Post-processing converts the floating point number based Tensor output to time stamp output. While generating the speaker-homogeneous segments, onset and offset threshold, 
+paddings can be considered to render the time stamps that can lead to the lowest diarization error rate (DER). This post-processing can be applied to both offline and streaming Sortformer diarizer.
+
+
+By default, post-processing is bypassed, and only binarization is performed. If you want to reproduce DER scores reported on NeMo model cards, you need to apply post-processing steps. Use batch_size = 1 to have the longest inference window and the highest possible accuracy.
+
+.. code-block:: yaml
+
+  parameters: 
+    onset: 0.64  # Onset threshold for detecting the beginning of a speech segment
+    offset: 0.74  # Offset threshold for detecting the end of a speech segment
+    pad_onset: 0.06  # Adds the specified duration at the beginning of each speech segment
+    pad_offset: 0.0  # Adds the specified duration at the end of each speech segment
+    min_duration_on: 0.1  # Removes short speech segments if the duration is less than the specified minimum duration
+    min_duration_off: 0.15  # Removes short silences if the duration is less than the specified minimum duration
+
+
+Hydra Configurations for Diarization Inference
+==============================================
+
+Example configuration files for speaker diarization inference can be found in ``<NeMo_root>/examples/speaker_tasks/diarization/conf/inference/``. Choose a yaml file that fits your targeted domain. For example, if you want to diarize audio recordings of telephonic speech, choose ``diar_infer_telephonic.yaml``.
+
+The configurations for all the components of diarization inference are included in a single file named ``diar_infer_<domain>.yaml``. Each ``.yaml`` file has a few different sections for the following modules: VAD, Speaker Embedding, Clustering and ASR.
+
+In speaker diarization inference, the datasets provided in manifest format denote the data that you would like to perform speaker diarization on. 
+
+Diarizer Configurations
+-----------------------
+
+An example ``diarizer``  Hydra configuration could look like:
+
+.. code-block:: yaml
+
+  diarizer:
+    manifest_filepath: ???
+    out_dir: ???
+    oracle_vad: False # If True, uses RTTM files provided in manifest file to get speech activity (VAD) timestamps
+    collar: 0.25 # Collar value for scoring
+    ignore_overlap: True # Consider or ignore overlap segments while scoring
+
+Under ``diarizer`` key, there are ``vad``, ``speaker_embeddings``, ``clustering`` and ``asr`` keys containing configurations for the inference of the corresponding modules.
+
+Configurations for Voice Activity Detector
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Parameters for VAD model are provided as in the following Hydra config example.
+
+.. code-block:: yaml
+
+  vad:
+    model_path: null # .nemo local model path or pretrained model name or none
+    external_vad_manifest: null # This option is provided to use external vad and provide its speech activity labels for speaker embeddings extraction. Only one of model_path or external_vad_manifest should be set
+
+    parameters: # Tuned parameters for CH109 (using the 11 multi-speaker sessions as dev set) 
+      window_length_in_sec: 0.15  # Window length in sec for VAD context input 
+      shift_length_in_sec: 0.01 # Shift length in sec for generate frame level VAD prediction
+      smoothing: "median" # False or type of smoothing method (eg: median)
+      overlap: 0.875 # Overlap ratio for overlapped mean/median smoothing filter
+      onset: 0.4 # Onset threshold for detecting the beginning and end of a speech 
+      offset: 0.7 # Offset threshold for detecting the end of a speech
+      pad_onset: 0.05 # Adding durations before each speech segment 
+      pad_offset: -0.1 # Adding durations after each speech segment 
+      min_duration_on: 0.2 # Threshold for short speech segment deletion
+      min_duration_off: 0.2 # Threshold for small non_speech deletion
+      filter_speech_first: True 
+
+Configurations for Speaker Embedding in Diarization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Parameters for speaker embedding model are provided in the following Hydra config example. Note that multiscale parameters either accept list or single floating point number.
+
+.. code-block:: yaml
+
+  speaker_embeddings:
+    model_path: ??? # .nemo local model path or pretrained model name (titanet_large, ecapa_tdnn or speakerverification_speakernet)
+    parameters:
+      window_length_in_sec: 1.5 # Window length(s) in sec (floating-point number). Either a number or a list. Ex) 1.5 or [1.5,1.25,1.0,0.75,0.5]
+      shift_length_in_sec: 0.75 # Shift length(s) in sec (floating-point number). Either a number or a list. Ex) 0.75 or [0.75,0.625,0.5,0.375,0.25]
+      multiscale_weights: null # Weight for each scale. should be null (for single scale) or a list matched with window/shift scale count. Ex) [1,1,1,1,1]
+      save_embeddings: False # Save embeddings as pickle file for each audio input.
+
+Configurations for Clustering in Diarization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Parameters for clustering algorithm are provided in the following Hydra config example.
+
+.. code-block:: yaml
+  
+  clustering:
+    parameters:
+      oracle_num_speakers: False # If True, use num of speakers value provided in the manifest file.
+      max_num_speakers: 20 # Max number of speakers for each recording. If oracle_num_speakers is passed, this value is ignored.
+      enhanced_count_thres: 80 # If the number of segments is lower than this number, enhanced speaker counting is activated.
+      max_rp_threshold: 0.25 # Determines the range of p-value search: 0 < p <= max_rp_threshold. 
+      sparse_search_volume: 30 # The higher the number, the more values will be examined with more time. 
+
+Configurations for Diarization with ASR
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following configuration needs to be appended under ``diarizer`` to run ASR with diarization to get a transcription with speaker labels. 
+
+.. code-block:: yaml
+
+  asr:
+    model_path: ??? # Provide NGC cloud ASR model name. stt_en_conformer_ctc_* models are recommended for diarization purposes.
+    parameters:
+      asr_based_vad: False # if True, speech segmentation for diarization is based on word-timestamps from ASR inference.
+      asr_based_vad_threshold: 50 # threshold (multiple of 10ms) for ignoring the gap between two words when generating VAD timestamps using ASR based VAD.
+      asr_batch_size: null # Batch size can be dependent on each ASR model. Default batch sizes are applied if set to null.
+      lenient_overlap_WDER: True # If true, when a word falls into speaker-overlapped regions, consider the word as a correctly diarized word.
+      decoder_delay_in_sec: null # Native decoder delay. null is recommended to use the default values for each ASR model.
+      word_ts_anchor_offset: null # Offset to set a reference point from the start of the word. Recommended range of values is [-0.05  0.2]. 
+      word_ts_anchor_pos: "start" # Select which part of the word timestamp we want to use. The options are: 'start', 'end', 'mid'.
+      fix_word_ts_with_VAD: False # Fix the word timestamp using VAD output. You must provide a VAD model to use this feature.
+      colored_text: False # If True, use colored text to distinguish speakers in the output transcript.
+      print_time: True # If True, the start of the end time of each speaker turn is printed in the output transcript.
+      break_lines: False # If True, the output transcript breaks the line to fix the line width (default is 90 chars)

docs/source/asr/speaker_diarization/data/diarization_results.csv

@@ -0,0 +1,6 @@
+Model Name,Model Base Class,Model Card
+vad_multilingual_marblenet,EncDecClassificationModel,"https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/vad_multilingual_marblenet"
+vad_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet"
+vad_telephony_marblenet,EncDecClassificationModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet"
+titanet_large,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:titanet_large"
+ecapa_tdnn,EncDecSpeakerLabelModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:ecapa_tdnn"

docs/source/asr/speaker_diarization/data/e2e_diar_models.csv

@@ -0,0 +1,4 @@
+Model Name,Model Base Class,Model Card
+diar_sortformer_4spk-v1,SortformerEncLabelModel,"https://huggingface.co/nvidia/diar_sortformer_4spk-v1"
+diar_streaming_sortformer_4spk-v2,SortformerEncLabelModel,"https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2"
+diar_streaming_sortformer_4spk-v2.1,SortformerEncLabelModel,"https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1"

docs/source/asr/speaker_diarization/datasets.rst

@@ -0,0 +1,183 @@
+Datasets
+========
+
+Data Preparation for Speaker Diarization Training (End-to-End Diarization)
+--------------------------------------------------------------------------
+
+Overview
+~~~~~~~~
+
+Speaker diarization training requires a manifest file in JSON-lines format. Each line describes one training segment and points to an audio file together with its corresponding RTTM ground-truth labels. The same manifest format is used for both training and inference.
+
+Use ``<NeMo_git_root>/scripts/speaker_tasks/pathfiles_to_diarize_manifest.py`` to generate a manifest from lists of audio and RTTM file paths:
+
+.. code-block:: bash
+
+  python <NeMo_git_root>/scripts/speaker_tasks/pathfiles_to_diarize_manifest.py \
+    --add_duration \
+    --paths2audio_files="/path/to/audio_file_path_list.txt" \
+    --paths2rttm_files="/path/to/rttm_file_list.txt" \
+    --manifest_filepath="/path/to/train_manifest.json"
+
+All three arguments are required. Audio and RTTM files are matched by their base filename (e.g., ``abcd01.wav`` pairs with ``abcd01.rttm``).
+
+.. note::
+  All provided files (audio, RTTM, text) must share the same base name, and each base name must be unique across the dataset.
+
+- Example ``audio_file_path_list.txt``:
+
+.. code-block:: bash
+
+  /path/to/abcd01.wav
+  /path/to/abcd02.wav
+
+- Example ``rttm_file_path_list.txt``:
+
+.. code-block:: bash
+
+  /path/to/abcd01.rttm
+  /path/to/abcd02.rttm
+
+The RTTM (Rich Transcription Time Marked) format provides per-speaker speech activity timestamps. Each line follows this structure:
+
+.. code-block:: bash
+
+  SPEAKER TS3012d.Mix-Headset 1 32.679 0.671 <NA> <NA> MTD046ID <NA> <NA>
+
+The generated ``train_manifest.json`` will contain one JSON line per segment:
+
+.. code-block:: bash
+
+  {"audio_filepath": "/path/to/abcd01.wav", "offset": 0, "duration": 90, "label": "infer", "text": "-", "num_speakers": 2, "rttm_filepath": "/path/to/rttm/abcd01.rttm"}
+
+For end-to-end speaker diarization training, this manifest format is sufficient. For cascaded speaker diarization training, the manifest should be further processed to generate pairwise two-speaker session files.
+
+
+Working with Long-Form Audio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Long audio recordings do **not** need to be physically split into shorter files. The NeMo diarization dataloader reads only the segment specified by the ``offset`` and ``duration`` fields in each manifest entry. The corresponding RTTM file is also windowed to the same time range automatically — there is no need to create separate RTTM files for each segment.
+
+To segment a long recording, create multiple manifest entries that reference the same audio and RTTM files but with different ``offset`` and ``duration`` values. Each entry must carry a unique ``uniq_id`` so that the dataloader can distinguish segments originating from the same source file. A recommended convention is ``<base_name>#<index>#<offset>#<duration>``.
+
+Below is an example where a 45-minute recording (``EN2001a.Mix-Headset.wav``) is divided into 90-second windows. The full session contains 4 speakers, but this particular segment has only 3 active speakers:
+
+.. code-block:: bash
+
+  {"uniq_id": "EN2001a.Mix-Headset#116#932.92#90.0", "offset": 932.92, "duration": 90, "num_speakers": 3, "audio_filepath": "/path/to/wav/EN2001a.Mix-Headset.wav", "rttm_filepath": "/path/to/rttm/EN2001a.Mix-Headset.rttm"}
+
+
+Handling ``num_speakers`` and Partial Speaker Presence
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A training segment may contain only a subset of the speakers present in the full recording. The ``num_speakers`` field should reflect the number of speakers **active in that specific segment**, not the total across the entire session.
+
+In practice, ``num_speakers`` is optional — the dataloader can infer the speaker count automatically from the RTTM labels within the segment's time window (see the ``DiarizationLabelDataset`` class in ``<NeMo_git_root>/nemo/collections/asr/data/audio_to_diar_label.py``).
+
+
+Speaker-Count Coverage in Training Data
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The model will not generalize to speaker counts it has never encountered during training. If the target scenario involves up to *N* speakers, the training set should include segments with every speaker count from 1 through *N*. When natural data is scarce for higher speaker counts, consider:
+
+- **Oversampling** segments that contain more speakers.
+- **Augmenting** with simulated multi-speaker mixtures (e.g., from LibriSpeech).
+- **Scaling data volume** with speaker count — for example, 100 hours for 1-speaker segments, 200 hours for 2-speaker, 300 hours for 3-speaker, and so on.
+
+Matching the speaker-count distribution between training and inference is critical for good diarization performance.
+
+
+Data Preparation for Diarization Inference: for Both End-to-end and Cascaded Systems
+------------------------------------------------------------------------------------
+
+As in dataset preparation for diarization trainiing, diarization inference is based on Hydra configurations which are fulfilled by ``.yaml`` files. See :doc:`NeMo Speaker Diarization Configuration Files <./configs>` for setting up the input Hydra configuration file for speaker diarization inference. Input data should be provided in line delimited JSON format as below:
+	
+.. code-block:: bash
+
+  {"audio_filepath": "/path/to/abcd.wav", "offset": 0, "duration": null, "label": "infer", "text": "-", "num_speakers": null, "rttm_filepath": "/path/to/rttm/abcd.rttm", "uem_filepath": "/path/to/uem/abcd.uem"}
+
+In each line of the input manifest file, ``audio_filepath`` item is mandatory while the rest of the items are optional and can be passed for desired diarization setting. We refer to this file as a manifest file. This manifest file can be created by using the script in ``<NeMo_git_root>/scripts/speaker_tasks/pathfiles_to_diarize_manifest.py``. The following example shows how to run ``pathfiles_to_diarize_manifest.py`` by providing path list files.
+
+.. code-block:: bash
+   
+    python pathfiles_to_diarize_manifest.py --paths2audio_files /path/to/audio_file_path_list.txt \
+                                            --paths2txt_files /path/to/transcript_file_path_list.txt \
+                                            --paths2rttm_files /path/to/rttm_file_path_list.txt \
+                                            --paths2uem_files /path/to/uem_file_path_list.txt \
+                                            --paths2ctm_files /path/to/ctm_file_path_list.txt \
+                                            --manifest_filepath /path/to/manifest_output/input_manifest.json 
+
+The ``--paths2audio_files`` and ``--manifest_filepath`` are required arguments. Note that we need to maintain consistency on unique filenames for every field (key) by only changing the filename extensions. For example, if there is an audio file named ``abcd.wav``, the rttm file should be named as ``abcd.rttm`` and the transcription file should be named as ``abcd.txt``. 
+
+- Example audio file path list ``audio_file_path_list.txt``
+
+.. code-block:: bash
+
+  /path/to/abcd01.wav
+  /path/to/abcd02.wav
+
+- Example RTTM file path list ``rttm_file_path_list.txt``
+
+.. code-block:: bash
+  
+  /path/to/abcd01.rttm
+  /path/to/abcd02.rttm
+   
+
+The path list files containing the absolute paths to these WAV, RTTM, TXT, CTM and UEM files should be provided as in the above example. ``pathsfiles_to_diarize_manifest.py`` script will match each file using the unique filename (e.g. ``abcd``). Finally, the absolute path of the created manifest file should be provided through Hydra configuration as shown below:
+
+.. code-block:: yaml
+   
+	diarizer.manifest_filepath="path/to/manifest/input_manifest.json"
+
+The following are descriptions about each field in an input manifest JSON file.
+
+.. note::
+	We expect all the provided files (e.g. audio, rttm, text) to have the same base name and the name should be unique (uniq-id).
+
+``audio_filepath`` (Required):
+  
+  a string containing absolute path to the audio file.
+
+``num_speakers`` (Optional):
+  
+  If the number of speakers is known, provide the integer number or assign null if not known. 
+	
+``rttm_filepath`` (Optional):
+  
+  To evaluate a diarization system with known rttm files, one needs to provide Rich Transcription Time Marked (RTTM) files as ground truth label files. If RTTM files are provided, the diarization evaluation will be initiated. Here is one line from a RTTM file as an example:
+
+.. code-block:: bash
+
+  SPEAKER TS3012d.Mix-Headset 1 331.573 0.671 <NA> <NA> MTD046ID <NA> <NA>
+
+``text`` (Optional):
+
+  Ground truth transcription for diarization with ASR inference. Provide the ground truth transcription of the given audio file in string format
+
+.. code-block:: bash
+
+  {"text": "this is an example transcript"}
+
+``uem_filepath`` (Optional):
+
+  The UEM file is used for specifying the scoring regions to be evaluated in the given audio file.
+  UEMfile follows the following convention: ``<uniq-id> <channel ID> <start time> <end time>``. ``<channel ID>`` is set to 1.
+
+  Example lines of UEM file:
+
+.. code-block:: bash
+  
+    TS3012d.Mix-Headset 1 12.31 108.98
+    TS3012d.Mix-Headset 1 214.00 857.09
+
+``ctm_filepath`` (Optional):
+    
+  The CTM file is used for the evaluation of word-level diarization results and word-timestamp alignment. The CTM file follows this convention: ``<session name> <channel ID> <start time> <duration> <word> <confidence> <type of token> <speaker>``. Note that the ``<speaker>`` should exactly match speaker IDs in RTTM. Since confidence is not required for evaluating diarization results, we assign ``<confidence>`` the value ``NA``. If the type of token is words, we assign ``<type of token>`` as ``lex``.  
+
+  Example lines of CTM file:
+
+.. code-block:: bash
+  
+   TS3012d.Mix-Headset 1 12.879 0.32 okay NA lex MTD046ID
+   TS3012d.Mix-Headset 1 13.203 0.24 yeah NA lex MTD046ID

docs/source/asr/speaker_diarization/intro.rst

@@ -0,0 +1,102 @@
+Speaker Diarization
+===================
+
+Speaker Diarization Overview
+----------------------------
+Speaker diarization is the process of segmenting audio recordings by speaker labels and aims to answer the question "who spoke when?". Speaker diarization makes a clear distinction when it is compared with speech recognition. As shown in the figure below, before we perform speaker diarization, we know "what is spoken" yet we do not know "who spoke it". Therefore, speaker diarization is an essential feature for a speech recognition system to enrich the transcription with speaker labels.
+
+.. image:: images/asr_sd_diagram.png
+        :align: center
+        :width: 800px
+        :alt: Speaker diarization pipeline- VAD, segmentation, speaker embedding extraction, clustering
+
+To figure out "who spoke when", speaker diarization systems need to capture the characteristics of unseen speakers and tell apart which regions in the audio recording belong to which speaker. To achieve this, speaker diarization systems extract voice characteristics, count the number of speakers, then assign the audio segments to the corresponding speaker index.
+
+Diarize speech with 3 lines of code
+------------------------------------
+After :ref:`installing NeMo<installation>`, you can diarize an audio file as follows:
+
+.. code-block:: python
+
+    from nemo.collections.asr.models import SortformerEncLabelModel
+    diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2.1").eval()
+    predicted_segments = diar_model.diarize(audio=["/path/to/your/audio.wav"], batch_size=1)
+
+For help choosing the right model for your use case, see :doc:`Choosing a Model <../../starthere/choosing_a_model>`.
+
+Types of Speaker Diarization Systems
+-------------------------------------
+
+.. image:: images/e2e_and_cascaded_diar_systems.png
+        :align: center
+        :width: 800px
+        :alt: End-to-End and Cascaded Diar Systems
+
+1. End-to-End Speaker Diarization System:
+
+End-to-end speaker diarization systems pursue a much more simplified version of a system where a single neural network model accepts raw audio signals and outputs speaker activity for each audio frame. Therefore, end-to-end diarization models have an advantage in ease of optimization and deployments.
+
+Currently, NeMo Speech AI provides the following end-to-end speaker diarization models:
+
+- **Sortformer Diarizer** : A transformer-based model that estimates speaker labels from the given audio input giving the speaker indexes in arrival-time order.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 40 20 40
+
+   * - Model
+     - Type
+     - HuggingFace Link
+   * - diar_sortformer_4spk-v1
+     - Offline
+     - `nvidia/diar_sortformer_4spk-v1 <https://huggingface.co/nvidia/diar_sortformer_4spk-v1>`__
+   * - diar_streaming_sortformer_4spk-v2
+     - Streaming
+     - `nvidia/diar_streaming_sortformer_4spk-v2 <https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2>`__
+   * - diar_streaming_sortformer_4spk-v2.1
+     - Streaming
+     - `nvidia/diar_streaming_sortformer_4spk-v2.1 <https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1>`__
+
+2. Cascaded Speaker Diarization System:
+
+Traditional cascaded (also referred to as modular or pipelined) speaker diarization systems consist of multiple modules such as voice activity detection (VAD), speaker embedding extraction, and clustering.
+Cascaded speaker diarization systems are more challenging to optimize and deploy together but still have the advantage of fewer restrictions on the number of speakers and session length.
+
+Cascaded NeMo Speech AI speaker diarization system consists of the following modules:
+
+- **Voice Activity Detector (VAD)**: A trainable model which detects the presence or absence of speech to generate timestamps for speech activity from the given audio recording.
+
+- **Speaker Embedding Extractor**: A trainable model that extracts speaker embedding vectors containing voice characteristics from raw audio signal.
+
+- **Clustering Module**: A non-trainable module that groups speaker embedding vectors into a number of clusters.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 30 40
+
+   * - Module
+     - Model
+     - Link
+   * - Voice Activity Detector
+     - Frame VAD Multilingual MarbleNet v2.0
+     - `HuggingFace <https://huggingface.co/nvidia/Frame_VAD_Multilingual_MarbleNet_v2.0>`__ · `NGC <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/vad_multilingual_marblenet>`__
+   * - Speaker Embedding Extractor
+     - TitaNet-Large
+     - `HuggingFace <https://huggingface.co/nvidia/speakerverification_en_titanet_large>`__ · `NGC <https://ngc.nvidia.com/catalog/models/nvidia:nemo:titanet_large>`__
+   * - Clustering Module
+     - Spectral Clustering
+     - Non-trainable (no checkpoint)
+
+
+The full documentation tree is as follows:
+
+.. toctree::
+   :maxdepth: 8
+
+   models
+   datasets
+   results
+   configs
+   api
+   resources
+

docs/source/asr/speaker_diarization/models.rst

@@ -0,0 +1,112 @@
+Models
+======
+
+This section gives a brief overview of the supported speaker diarization models in NeMo's ASR collection.
+
+Currently NeMo Speech AI supports two types of speaker diarization systems:
+
+1. **End-to-end Speaker Diarization:** Sortformer Diarizer 
+
+Sortformer is a Transformer encoder-based end-to-end speaker diarization model that generates predicted speaker labels directly from input audio clips. 
+We offer offline and online versions of Sortformer Diarizer. Online version of Sortformer diarizer can also be used for offline diarization by setting a long enough chunk size.
+
+2. **Cascaded (Pipelined) Speaker Diarization:** Clustering diarizer
+
+The clustering-based speaker diarization pipeline in NeMo Speech AI involves the use of the :doc:`MarbleNet <../speech_classification/models>` model for Voice Activity Detection (VAD) and the :doc:`TitaNet <../speaker_recognition/models>` model for speaker embedding extraction, followed by spectral clustering.
+
+.. _Sortformer Diarizer:
+
+Sortformer Diarizer
+-------------------
+Speaker diarization is all about figuring out who's speaking when in an audio recording. In the world of automatic speech recognition (ASR), this becomes even more important for handling conversations with multiple speakers. Multispeaker ASR (also known as speaker-attributed or multitalker ASR) uses this process to not just transcribe what's being said, but also to label each part of the transcript with the right speaker.
+
+As ASR technology continues to advance, speaker diarization is increasingly becoming part of the ASR workflow itself. Some systems now handle speaker labeling and transcription at the same time during decoding. This means you not only get accurate text—you're also getting insights into who said what, making it more useful for conversational analysis.
+
+However, despite significant advancements, integrating speaker diarization and ASR into a unified, seamless system remains a considerable challenge. A key obstacle lies in the need for extensive high-quality, annotated audio data featuring multiple speakers. Acquiring such data is far more complex than collecting monaural-speaker datasets. This challenge is particularly pronounced for low-resource languages and domains like healthcare, where strict privacy regulations further constrain data availability.
+
+On top of that, many real-world use cases need these models to handle really long audio files—sometimes hours of conversation at a time. Training on such lengthy data is even more complicated because it's hard to find or annotate. This creates a big gap between what's needed and what's available, making multispeaker ASR one of the toughest nuts to crack in the field of speech technology.
+
+.. image:: images/intro_comparison.png
+        :align: center
+        :width: 800px
+        :alt: Intro Comparison
+
+To tackle the complexities of multispeaker automatic speech recognition (ASR), we introduce `Sortformer <https://arxiv.org/abs/2409.06656>`__, a new approach that incorporates *Sort Loss* and techniques to align timestamps with text tokens. Traditional approaches like permutation-invariant loss (PIL) face challenges when applied in batchable and differentiable computational graphs, especially since token-based objectives struggle to incorporate speaker-specific attributes into PIL-based loss functions.
+
+To address this, we propose an arrival time sorting (ATS) approach. In this method, speaker tokens from ASR outputs and speaker timestamps from diarization outputs are sorted by their arrival times to resolve permutations. This approach allows the multispeaker ASR system to be trained or fine-tuned using token-based cross-entropy loss, eliminating the need for timestamp-based or frame-level objectives with PIL.
+
+.. image:: images/ats.png
+        :align: center
+        :width: 600px
+        :alt: Arrival Time Sort
+
+The ATS-based multispeaker ASR system is powered by an end-to-end neural diarizer model, Sortformer, which generates speaker-label timestamps in arrival time order (ATO). To train the neural diarizer to produce sorted outputs, we introduce Sort Loss, a method that creates gradients enabling the Transformer model to learn the ATS mechanism.
+
+.. image:: images/main_dataflow.png
+        :align: center
+        :width: 500px
+        :alt: Main Dataflow
+
+
+Additionally, as shown in the above figure, our diarization system integrates directly with the ASR encoder. By embedding speaker supervision data as speaker kernels into the ASR encoder states, the system seamlessly combines speaker and transcription information. This unified approach improves performance and simplifies the overall architecture.
+
+As a result, our end-to-end multispeaker ASR system is fully or partially trainable with token objectives, allowing both the ASR and speaker diarization modules to be trained or fine-tuned using these objectives. Additionally, during the multispeaker ASR training phase, no specialized loss calculation functions are needed when using Sortformer, as frameworks for standard single-speaker ASR models can be employed. These compatibilities greatly simplify and accelerate the training and fine-tuning process of multispeaker ASR systems.
+
+On top of all these benefits, *Sortformer* can be used as a stand-alone end-to-end speaker diarization model. By training a Sortformer diarizer model especially on high-quality simulated data with accurate time-stamps, you can boost the performance of multi-speaker ASR systems, just by integrating the *Sortformer* model as *Speaker Supervision* model in a computation graph.
+
+In this tutorial, we will walk you through the process of training a Sortformer diarizer model with toy dataset. Before starting, we will introduce the concepts of Sort-Loss calculation and the Hybrid loss technique.
+
+.. image:: images/sortformer.png
+        :align: center
+        :width: 500px
+        :alt: Sortformer Model with Hybrid Loss
+
+.. image:: images/loss_types.png
+        :align: center
+        :width: 1000px
+        :alt: PIL model VS SortLoss model
+
+*Sort Loss* is designed to compare the predicted outputs with the true labels, typically sorted in arrival-time order or another relevant metric. The key distinction that *Sortformer* introduces compared to previous end-to-end diarization systems such as `EEND-SA <https://arxiv.org/pdf/1909.06247>`__, `EEND-EDA <https://arxiv.org/abs/2106.10654>`__ lies in the organization of class presence :math:`\mathbf{\hat{Y}}`.
+
+The figure below illustrates the difference between *Sort Loss* and permutation-invariant loss (PIL) or permutation-free loss.
+
+- PIL is calculated by finding the permutation of the target that minimizes the loss value between the prediction and the target.
+
+- *Sort Loss* simply compares the arrival-time-sorted version of speaker activity outputs for both the prediction and the target. Note that sometimes the same ground-truth labels lead to different target matrices for *Sort Loss* and PIL.
+
+For example, the figure below shows two identical source target matrices (the two matrices at the top), but the resulting target matrices for *Sort Loss* and PIL are different.
+
+.. _Streaming Sortformer Diarizer:
+
+Streaming Sortformer Diarizer
+-----------------------------
+
+`Streaming Sortformer <https://www.arxiv.org/pdf/2507.18446>`__ is a streaming version of Sortformer diarizer. To handle live audio, Streaming Sortformer processes the sound in small, overlapping chunks. It employs an Arrival-Order Speaker Cache (AOSC) that stores frame-level acoustic embeddings for all speakers previously detected in the audio stream. This allows the model to compare speakers in the current chunk with those in the previous ones, ensuring a person is consistently identified with the same label throughout the stream.
+
+.. image:: images/cache_fifo_chunk.png
+        :align: center
+        :width: 800px
+        :alt: Chunk-wise processing with AOSC and FIFO buffer in Streaming Sortformer inference
+
+Streaming Sortformer employs a pre-encoder layer in the Fast-Conformer to generate a speaker cache. At each step, speaker cache is filtered to only retain the high-quality speaker cache vectors. Aside from speaker-cache management part, Streaming Sortformer follows the architecture of the offline version of Sortformer.
+
+.. image:: images/streaming_steps.png
+        :align: center
+        :width: 800px
+        :alt: The dataflow of step-wise Streaming Sortformer inference
+
+Below is the animated heatmap illustrating real-time speaker diarization for a three-speaker conversation using Streaming Sortformer. The heatmap shows how activities of speakers are detected in the current chunk and updated in the Arrival-Order Speaker Cache and FIFO queue.
+
+.. image:: images/aosc_3spk_example.gif
+        :align: center
+        :width: 800px
+        :alt: Streaming Sortformer Animated
+
+
+References
+-----------
+
+.. bibliography:: ../asr_all.bib
+    :style: plain
+    :labelprefix: SD-MODELS
+    :keyprefix: sd-models-

docs/source/asr/speaker_diarization/resources.rst

@@ -0,0 +1,22 @@
+
+Resource and Documentation Guide
+--------------------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - Resource
+     - Link
+   * - Tutorials
+     - `Speaker Tasks Notebooks <https://github.com/NVIDIA/NeMo/tree/main/tutorials/speaker_tasks>`__
+   * - Models
+     - :doc:`Model Architectures <./models>`
+   * - Datasets
+     - :doc:`Dataset Preprocessing <./datasets>`
+   * - Checkpoints
+     - :doc:`Pretrained Checkpoints <./results>`
+   * - Configs
+     - :doc:`Configuration Files <./configs>`
+   * - API
+     - :doc:`API Reference <./api>`

docs/source/asr/speaker_diarization/results.rst

@@ -0,0 +1,193 @@
+Checkpoints
+===========
+
+There are two main ways to load pretrained checkpoints in NeMo as introduced in the :doc:`ASR checkpoints <../results>` section.
+In speaker diarization, the diarizer loads checkpoints that are passed through the config file. 
+
+
+End-to-end Speaker Diarization Models
+=====================================
+
+Sortformer Diarizer Training
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use the following command to train a Sortformer diarizer model.
+  
+.. code-block:: bash  
+  
+  # Feed the config for Sortformer diarizer model training
+  python ${NEMO_ROOT}/examples/speaker_tasks/diarization/neural_diarizer/sortformer_diar_train.py --config-path='../conf/neural_diarizer' \  
+    --config-name='sortformer_diarizer_hybrid_loss_4spk-v1.yaml' \   
+    trainer.devices=1 \  
+    model.train_ds.manifest_filepath="<train_manifest_path>" \  
+    model.validation_ds.manifest_filepath="<dev_manifest_path>" \   
+    exp_manager.name='sample_train' \   
+    exp_manager.exp_dir=./sortformer_diar_train
+
+
+Sortformer Diarizer Inference with Post-processing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use the following command to run inference on a Sortformer diarizer model.
+
+.. code-block:: bash  
+
+  # Config for post-processing  
+  PP_YAML1=${NEMO_ROOT}/examples/speaker_tasks/diarization/conf/post_processing/sortformer_diar_4spk-v1_dihard3-dev.yaml  
+  PP_YAML2=${NEMO_ROOT}/examples/speaker_tasks/diarization/conf/post_processing/sortformer_diar_4spk-v1_callhome-part1.yaml   
+  python ${NEMO_ROOT}/examples/speaker_tasks/diarization/neural_diarizer/e2e_diarize_speech.py \  
+    batch_size=1 \  
+    model_path=/path/to/diar_sortformer_4spk-v1.nemo \  
+    postprocessing_yaml=${PP_YAML2} \  
+    dataset_manifest=/path/to/diarization_manifest.json  
+
+
+Streaming Sortformer Diarizer Training
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use the following command to train a Streaming Sortformer diarizer model.
+  
+.. code-block:: bash  
+  
+  # Feed the config for Sortformer diarizer model training
+  python ${NEMO_ROOT}/examples/speaker_tasks/diarization/neural_diarizer/sortformer_diar_train.py --config-path='../conf/neural_diarizer' \  
+    --config-name='streaming_sortformer_diarizer_4spk-v2.yaml' \  
+    trainer.devices=1 \   
+    model.streaming_mode=True \   
+    model.train_ds.manifest_filepath="<train_manifest_path>" \   
+    model.validation_ds.manifest_filepath="<dev_manifest_path>" \  
+    exp_manager.name='sample_train' \  
+    exp_manager.exp_dir=./sortformer_diar_train 
+
+
+Streaming Sortformer Diarizer Inference with Post-processing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Use the following command to run inference on a Streaming Sortformer diarizer model.
+
+.. code-block:: bash  
+
+  # Config for post-processing  
+  STREAM_PP_YAML1=${NEMO_ROOT}/examples/speaker_tasks/diarization/conf/post_processing/diar_streaming_sortformer_4spk-v2_dihard3-dev.yaml  
+  STREAM_PP_YAML2=${NEMO_ROOT}/examples/speaker_tasks/diarization/conf/post_processing/diar_streaming_sortformer_4spk-v2_callhome-part1.yaml   
+  python ${NEMO_ROOT}/examples/speaker_tasks/diarization/neural_diarizer/e2e_diarize_speech.py \  
+    batch_size=1 \  
+    model_path=/path/to/diar_streaming_sortformer_4spk-v2.nemo \  
+    postprocessing_yaml=${STREAM_PP_YAML2} \  
+    dataset_manifest=/path/to/diarization_manifest.json  
+
+
+HuggingFace Pretrained Checkpoints
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
+These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`__.
+The model cards on NGC contain more information about each of the checkpoints available.
+
+In general, you can load models with model name in the following format, 
+
+.. code-block:: bash
+
+  pip install -U "huggingface_hub[cli]"
+  huggingface-cli login
+
+Load Offline Sortformer Diarizer from HuggingFace
+
+.. code-block:: python
+
+  from nemo.collections.asr.models import SortformerEncLabelModel
+  diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_sortformer_4spk-v1")
+
+
+Load Streaming Sortformer Diarizer from HuggingFace
+
+.. code-block:: python
+
+  from nemo.collections.asr.models import SortformerEncLabelModel
+  diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2")
+
+where the model name is the value under "Model Name" entry in the tables below.
+
+End-to-end Speaker Diarization Models
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :file: /asr/speaker_diarization/data/e2e_diar_models.csv
+   :align: left
+   :widths: 30, 30, 40
+   :header-rows: 1
+
+
+
+Models for Cascaded Speaker Diarization Pipeline
+================================================
+
+Loading Local Checkpoints
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Load VAD models
+
+.. code-block:: bash
+
+  pretrained_vad_model='/path/to/vad_multilingual_marblenet.nemo' # local .nemo or pretrained vad model name
+  ...
+  # pass with hydra config
+  config.diarizer.vad.model_path=pretrained_vad_model
+
+
+Load speaker embedding models
+
+.. code-block:: bash
+
+  pretrained_speaker_model='/path/to/titanet-l.nemo' # local .nemo or pretrained speaker embedding model name
+  ...
+  # pass with hydra config
+  config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
+
+NeMo will automatically save checkpoints of a model you are training in a `.nemo` format.
+You can also manually save your models at any point using :code:`model.save_to(<checkpoint_path>.nemo)`.
+
+
+Inference
+^^^^^^^^^
+
+.. note::
+  For details and deep understanding, please refer to ``<NeMo_root>/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb``.
+
+Check out :doc:`Datasets <./datasets>` for preparing audio files and optional label files.
+
+Run and evaluate speaker diarizer with below command:
+
+.. code-block:: bash
+
+  # Have a look at the instruction inside the script and pass the arguments you might need. 
+  python <NeMo_root>/examples/speaker_tasks/diarization/offline_diarization.py 
+
+
+NGC Pretrained Checkpoints
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ASR collection has checkpoints of several models trained on various datasets for a variety of tasks.
+These checkpoints are obtainable via NGC `NeMo Automatic Speech Recognition collection <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`__.
+The model cards on NGC contain more information about each of the checkpoints available.
+
+In general, you can load models with model name in the following format, 
+
+.. code-block:: python
+
+  pretrained_vad_model='vad_multilingual_marblenet'
+  pretrained_speaker_model='titanet_large'
+  ...
+  config.diarizer.vad.model_path=pretrained_vad_model \
+  config.diarizer.speaker_embeddings.model_path=pretrained_speaker_model
+
+where the model name is the value under "Model Name" entry in the tables below.
+
+Models for Speaker Diarization Pipeline
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :file: /asr/speaker_diarization/data/diarization_results.csv
+   :align: left
+   :widths: 30, 30, 40
+   :header-rows: 1