chore: import upstream snapshot with attribution

2026-07-13 13:28:58 +08:00
commit ba4be087d5
2316 changed files with 2668701 additions and 0 deletions
@@ -0,0 +1,8 @@
+Speaker tasks in general are broadly classified into two tasks:
+- [Speaker Recognition](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_recognition/intro.html)
+- [Speaker Diarization](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_diarization/intro.html)
+
+**Speaker Recognition** is a research area which solves two major tasks: speaker identification (what is the identity of the speaker?) and speaker verification (is the speaker who they claim to be?). Whereas **Speaker Diarization** is a task segmenting audio recordings by speaker labels (Who Speaks When?). 
+
+In *recognition* folder we provide scripts for training, inference and verification of audio samples.   
+In *diarization* folder we provide scripts for inference of speaker diarization using pretrained VAD (optional) and Speaker embedding extractor  models
@@ -0,0 +1,331 @@
+# Speaker Diarization
+
+Documentation section for speaker related tasks can be found at:
+ - [Speaker Diarization](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_diarization/intro.html)
+ - [Speaker Identification and Verification](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_recognition/intro.html)
+
+
+## Features of NeMo Speaker Diarization
+- Provides pretrained speaker embedding extractor models and VAD models.
+- Does not need to be tuned on dev-set while showing the better performance than AHC+PLDA method in general.
+- Estimates the number of speakers in the given session.
+- Provides example script for asr transcription with speaker labels. 
+
+## Supported Pretrained Speaker Embedding Extractor models
+- [titanet_large](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_large)
+- [ecapa_tdnn](https://ngc.nvidia.com/catalog/models/nvidia:nemo:ecapa_tdnn)
+- [speakerverification_speakernet](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/speakerverification_speakernet)
+
+## Supported End-to-End Speaker Diarization models
+- [diar_sortformer_4spk-v1](https://huggingface.co/nvidia/diar_sortformer_4spk-v1) — Offline Sortformer diarizer (up to 4 speakers)
+- [diar_streaming_sortformer_4spk-v2](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2) — Streaming Sortformer diarizer (up to 4 speakers)
+- [diar_streaming_sortformer_4spk-v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) — Streaming Sortformer diarizer with improved meeting speech robustness
+
+## Supported Pretrained VAD models
+- [vad_multilingual_marblenet](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/vad_multilingual_marblenet)
+- [vad_marblenet](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/vad_marblenet)
+- [vad_telephony_marblenet](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/vad_telephony_marblenet)
+
+## Supported ASR models
+QuartzNet, CitriNet and Conformer-CTC models are supported. 
+Recommended models HuggingFace:
+- [stt_en_conformer_ctc_large](https://huggingface.co/nvidia/stt_en_conformer_ctc_large/tree/main)
+
+## Performance
+
+#### Clustering Diarizer 
+Diarization Error Rate (DER) table of `titanet_large.nemo` model on well known evaluation datasets. 
+
+|         Evaluation Condition           |   AMI(Lapel)   | AMI(MixHeadset)     |   CH109  | NIST SRE 2000 |  
+|:--------------------------------------:|:--------------:|:-------------------:|:--------:|:-------------:|
+|       Domain Configuration             |   Meeting      |        Meeting      |Telephonic| Telephonic    |
+|  Oracle VAD <br> Known # of Speakers   |     1.28       |         1.07        |  0.56    |     5.62      |
+|  Oracle VAD <br> Unknown # of Speakers |     1.28       |         1.4         |  0.88    |     4.33      | 
+
+* All models were tested using the domain specific `.yaml` files which can be found in `conf/inference/` folder.
+* The above result is based on the oracle Voice Activity Detection (VAD) result.
+* This result is based on [titanet_large.nemo](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_large) model.
+
+#### End-to-End Diarizer: Sortformer
+
+[Sortformer](https://arxiv.org/abs/2409.06656) is a novel end-to-end neural model for speaker diarization that resolves the permutation problem by following the arrival-time order of speech segments from each speaker. Sortformer consists of a Fast-Conformer (NEST) encoder followed by a Transformer encoder with sigmoid outputs for each speaker. Both offline and streaming variants are available.
+
+##### [diar_sortformer_4spk-v1](https://huggingface.co/nvidia/diar_sortformer_4spk-v1) (Offline)
+
+An offline Sortformer diarizer with an 18-layer NEST encoder (L-size) and 18-layer Transformer encoder (hidden size 192), supporting up to 4 speakers. Trained on ~7,180 hours of real conversations and simulated audio mixtures.
+
+```python
+from nemo.collections.asr.models import SortformerEncLabelModel
+diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_sortformer_4spk-v1")
+diar_model.eval()
+predicted_segments = diar_model.diarize(audio="/path/to/audio.wav", batch_size=1)
+```
+
+Diarization Error Rate (DER) — all evaluations include overlapping speech:
+
+| Dataset | Collar | DER (no PP) | DER (with PP) |
+|:-------:|:------:|:-----------:|:-------------:|
+| DIHARD3-Eval (<=4spk) | 0.0s | 16.28 | **14.76** |
+| CALLHOME-part2 (2spk) | 0.25s | 6.49 | **5.85** |
+| CALLHOME-part2 (3spk) | 0.25s | 10.01 | **8.46** |
+| CALLHOME-part2 (4spk) | 0.25s | 14.14 | **12.59** |
+| CH109 | 0.25s | **6.27** | 6.86 |
+
+##### [diar_streaming_sortformer_4spk-v2](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2) (Streaming)
+
+A streaming version of Sortformer using an Arrival-Order Speaker Cache (AOSC) for consistent speaker tracking across chunks. Uses a 17-layer NEST encoder and 18-layer Transformer encoder. Supports configurable latency from ultra-low (0.32s) to very-high (30.4s).
+
+```python
+from nemo.collections.asr.models import SortformerEncLabelModel
+diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2")
+diar_model.eval()
+diar_model.sortformer_modules.chunk_len = 340
+diar_model.sortformer_modules.chunk_right_context = 40
+diar_model.sortformer_modules.fifo_len = 40
+diar_model.sortformer_modules.spkcache_update_period = 300
+predicted_segments = diar_model.diarize(audio="/path/to/audio.wav", batch_size=1)
+```
+
+Diarization Error Rate (DER) with post-processing — all evaluations include overlapping speech:
+
+| Dataset | Collar | 30.4s latency | 10.0s latency | 1.04s latency | 0.32s latency |
+|:-------:|:------:|:-------------:|:-------------:|:-------------:|:-------------:|
+| DIHARD III Eval (<=4spk) | 0.0s | 13.45 | 13.75 | 13.24 | 13.44 |
+| DIHARD III Eval (>=5spk) | 0.0s | 41.40 | 41.41 | 42.56 | 43.73 |
+| CALLHOME-part2 (2spk) | 0.25s | 5.34 | 6.05 | 6.57 | 6.91 |
+| CALLHOME-part2 (3spk) | 0.25s | 9.22 | 9.88 | 10.05 | 10.45 |
+| CALLHOME-part2 (4spk) | 0.25s | 11.29 | 11.72 | 12.44 | 13.70 |
+| CH109 | 0.25s | 4.61 | 4.80 | 4.88 | 5.27 |
+
+##### [diar_streaming_sortformer_4spk-v2.1](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) (Streaming, improved)
+
+An improved streaming Sortformer with greater robustness for meeting speech scenarios. Same architecture as v2 but with additional training on meeting corpora (DiPCo, AliMeeting, NOTSOFAR1) and forced-alignment-based ground-truth RTTMs for AMI and AliMeeting.
+
+```python
+from nemo.collections.asr.models import SortformerEncLabelModel
+diar_model = SortformerEncLabelModel.from_pretrained("nvidia/diar_streaming_sortformer_4spk-v2.1")
+diar_model.eval()
+diar_model.sortformer_modules.chunk_len = 340
+diar_model.sortformer_modules.chunk_right_context = 40
+diar_model.sortformer_modules.fifo_len = 40
+diar_model.sortformer_modules.spkcache_update_period = 300
+predicted_segments = diar_model.diarize(audio="/path/to/audio.wav", batch_size=1)
+```
+
+Diarization Error Rate (DER) — all evaluations include overlapping speech:
+
+**Telephonic and General-Purpose Speech:**
+
+| Dataset | Collar | 30.4s latency | 1.04s latency |
+|:-------:|:------:|:-------------:|:-------------:|
+| DIHARD III Eval (<=4spk) | 0.0s | 14.84 | 15.09 |
+| DIHARD III Eval (full) | 0.0s | 19.49 | 20.21 |
+| CALLHOME-part2 (full) | 0.25s | 10.10 | 11.19 |
+| CH109 | 0.25s | 5.04 | 5.09 |
+
+**Meeting Speech (v2.1 vs v2 comparison):**
+
+| Dataset | Collar | v2.1 (30.4s) | v2 (30.4s) | v2.1 (1.04s) | v2 (1.04s) |
+|:-------:|:------:|:------------:|:----------:|:------------:|:----------:|
+| AliMeeting Test (near) | 0.0s | **11.73** | 19.63 | **12.60** | 19.98 |
+| AliMeeting Test (far) | 0.0s | **13.55** | 21.09 | **15.60** | 22.09 |
+| AMI Test (IHM) | 0.0s | **15.90** | 22.39 | **16.67** | 25.11 |
+| AMI Test (SDM) | 0.0s | **17.80** | 28.56 | **20.57** | 31.34 |
+| NOTSOFAR1 Eval SC (full) | 0.0s | **27.07** | 33.43 | **28.75** | 34.52 |
+
+##### Example inference script for end-to-end diarizer
+```bash
+python neural_diarizer/e2e_diarize_speech.py \
+    model_path=<path to .nemo or HuggingFace model name> \
+    dataset_manifest=<path to manifest file> \
+    batch_size=1
+```
+
+## Run Speaker Diarization on Your Audio Files
+
+#### Example script for clustering diarizer: with system-VAD
+```bash
+  python clustering_diarizer/offline_diar_infer.py \
+    diarizer.manifest_filepath=<path to manifest file> \
+    diarizer.out_dir='demo_output' \
+    diarizer.speaker_embeddings.parameters.save_embeddings=False \
+    diarizer.vad.model_path=<pretrained model name or path to .nemo> \
+    diarizer.speaker_embeddings.model_path=<pretrained speaker embedding model name or path to .nemo> 
+```
+
+If you have oracle VAD files and groundtruth RTTM files for evaluation:
+Provide rttm files in the input manifest file and enable oracle_vad as shown below. 
+```bash
+...
+    diarizer.oracle_vad=True \
+...
+```
+
+#### Arguments
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; To run speaker diarization on your audio recordings, you need to prepare the following file.
+
+- **`diarizer.manifest_filepath`: <manifest file>** Path to manifest file 
+
+Example: `manifest.json`
+
+```bash
+{"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, label: "infer", "text": "-", "num_speakers": null, "rttm_filepath": "/path/to/rttm/file", "uem_filepath"="/path/to/uem/filepath"}
+```
+Mandatory fields are `audio_filepath`, `offset`, `duration`, `label:"infer"` and `text: <ground truth or "-" >`  , and the rest are optional keys which can be passed based on the type of evaluation 
+
+Some of important options in config file: 
+
+- **`diarizer.vad.model_path`: voice activity detection model name or path to the model**
+
+Specify the name of VAD model, then the script will download the model from NGC. Currently, we have 'vad_multilingual_marblenet', 'vad_marblenet' and  'vad_telephony_marblenet' as options for VAD models.
+
+`diarizer.vad.model_path='vad_multilingual_marblenet'`
+
+
+Instead, you can also download the model from [vad_multilingual_marblenet](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/vad_multilingual_marblenet), [vad_marblenet](https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_marblenet) and [vad_telephony_marblenet](https://ngc.nvidia.com/catalog/models/nvidia:nemo:vad_telephony_marblenet) and specify the full path name to the model as below.
+
+`diarizer.vad.model_path='path/to/vad_multilingual_marblenet.nemo'`
+
+- **`diarizer.speaker_embeddings.model_path`: speaker embedding model name**
+
+Specify the name of speaker embedding model, then the script will download the model from NGC. Currently, we support 'titanet_large', 'ecapa_tdnn' and 'speakerverification_speakernet'.
+
+`diarizer.speaker_embeddings.model_path='titanet_large'`
+
+You could also download *.nemo files from [this link](https://ngc.nvidia.com/catalog/models?orderBy=scoreDESC&pageNumber=0&query=SpeakerNet&quickFilter=&filters=) and specify the full path name to the speaker embedding model file (`*.nemo`).
+
+`diarizer.speaker_embeddings.model_path='path/to/titanet_large.nemo'` 
+ 
+
+- **`diarizer.speaker_embeddings.parameters.multiscale_weights`: multiscale diarization**
+
+Multiscale diarization system employs multiple scales at the same time to obtain a finer temporal resolution. To use multiscale feature, at least two scales and scale weights should be provided. The scales should be provided in descending order, from the longest scale to the base scale (the shortest). If multiple scales are provided, multiscale_weights must be provided in list format. The following example shows how multiscale parameters are specified and the recommended parameters.
+
+#### Example script: single-scale and multiscale
+
+Single-scale setting:
+```bash
+  python offline_diar_infer.py \
+     ... <other parameters> ...
+     parameters.window_length_in_sec=1.5 \
+     parameters.shift_length_in_sec=0.75 \
+     parameters.multiscale_weights=null \
+```
+
+Multiscale setting (base scale - window_length 0.5 s and shift_length 0.25):
+
+```bash
+  python offline_diar_infer.py \
+     ... <other parameters> ...
+     parameters.window_length_in_sec=[1.5,1.0,0.5] \
+     parameters.shift_length_in_sec=[0.75,0.5,0.25] \
+     parameters.multiscale_weights=[0.33,0.33,0.33] \
+```
+ 
+<br/>
+
+## Run Speech Recognition with Clustering based Speaker Diarization
+
+Using the script `offline_diar_with_asr_infer.py`, you can transcribe your audio recording with speaker labels as shown below:
+
+```
+[00:03.34 - 00:04.46] speaker_0: back from the gym oh good how's it going 
+[00:04.46 - 00:09.96] speaker_1: oh pretty well it was really crowded today yeah i kind of assumed everyone would be at the shore uhhuh
+[00:12.10 - 00:13.97] speaker_0: well it's the middle of the week or whatever so
+```
+
+Currently, offline clustering diarization inference supports ConformerCTC ASR models (e.g.,`stt_en_conformer_ctc_large`). 
+
+#### Example script
+
+```bash
+python offline_diar_with_asr_infer.py \
+    diarizer.manifest_filepath=<path to manifest file> \
+    diarizer.out_dir='demo_asr_output' \
+    diarizer.speaker_embeddings.model_path=<pretrained model name or path to .nemo> \
+    diarizer.asr.model_path=<pretrained model name or path to .nemo> \
+    diarizer.speaker_embeddings.parameters.save_embeddings=False \
+    diarizer.asr.parameters.asr_based_vad=True
+```
+If you have reference rttm files or oracle number of speaker information, you can provide those file paths and number of speakers in the manifest file path and pass `diarizer.clustering.parameters.oracle_num_speakers=True` as shown in the following example.
+
+```bash
+python offline_diar_with_asr_infer.py \
+    diarizer.manifest_filepath=<path to manifest file> \
+    diarizer.out_dir='demo_asr_output' \
+    diarizer.speaker_embeddings.model_path=<pretrained model name or path to .nemo> \
+    diarizer.asr.model_path=<pretrained model name or path to .nemo> \
+    diarizer.speaker_embeddings.parameters.save_embeddings=False \
+    diarizer.asr.parameters.asr_based_vad=True \
+    diarizer.clustering.parameters.oracle_num_speakers=True
+```
+
+#### Output folders
+
+The above script will create a folder named `./demo_asr_output/`.
+For example, in `./demo_asr_output/`, you can check the results as below.
+
+```bash
+./asr_with_diar
+├── pred_rttms
+    └── my_audio1.json
+    └── my_audio1.txt
+    └── my_audio1.rttm
+    └── my_audio1_gecko.json
+│
+└── speaker_outputs
+    └── oracle_vad_manifest.json
+    └── subsegments_scale2_cluster.label
+    └── subsegments_scale0.json
+    └── subsegments_scale1.json
+    └── subsegments_scale2.json
+... 
+```
+
+`my_audio1.json` file contains word-by-word json output with speaker label and time stamps. We also provide a json output file for [gecko](https://gong-io.github.io/gecko/) tool, where you can visualize the diarization result along with the ASR output.
+
+Example: `./demo_asr_output/pred_rttms/my_audio1.json`
+```bash
+{
+    "status": "Success",
+    "session_id": "my_audio1",
+    "transcription": "back from the gym oh good ...",
+    "speaker_count": 2,
+    "words": [
+        {
+            "word": "back",
+            "start_time": 0.44,
+            "end_time": 0.56,
+            "speaker_label": "speaker_0"
+        },
+...
+        {
+            "word": "oh",
+            "start_time": 1.74,
+            "end_time": 1.88,
+            "speaker_label": "speaker_1"
+        },
+        {
+            "word": "good",
+            "start_time": 2.08,
+            "end_time": 3.28,
+            "speaker_label": "speaker_1"
+        },
+```
+
+`*.txt` files in `pred_rttms` folder contain transcriptions with speaker labels and corresponding time.
+
+Example: `./demo_asr_output/pred_rttms/my_audio1.txt`
+```
+[00:03.34 - 00:04.46] speaker_0: back from the gym oh good how's it going
+[00:04.46 - 00:09.96] speaker_1: pretty well it was really crowded today yeah i kind of assumed everylonewould be at the shore uhhuh
+[00:12.10 - 00:13.97] speaker_0: well it's the middle of the week or whatever so
+[00:13.97 - 00:15.78] speaker_1: but it's the fourth of july mm
+[00:16.90 - 00:21.80] speaker_0: so yeah people still work tomorrow do you have to work tomorrow did you drive off yesterday
+```
+ 
+In `speaker_outputs` folder we have three kinds of files as follows:
+ 
+ - `oracle_vad_manifest.json` file contains oracle VAD labels that are extracted from RTTM files.
+ - `subsegments_scale<scale_index>.json` is a manifest file for subsegments, which includes segment-by-segment start and end time with original wav file path. In multi-scale mode, this file is generated for each `<scale_index>`.
+ - `subsegments_scale<scale_index>_cluster.label` file contains the estimated cluster labels for each segment. This file is only generated for the base scale index in multi-scale diarization mode.
@@ -0,0 +1,47 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from lightning.pytorch import seed_everything
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.models import ClusteringDiarizer
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+
+"""
+This script demonstrates how to use run speaker diarization.
+Usage:
+  python offline_diar_infer.py \
+    diarizer.manifest_filepath=<path to manifest file> \
+    diarizer.out_dir='demo_output' \
+    diarizer.speaker_embeddings.model_path=<pretrained modelname or path to .nemo> \
+    diarizer.vad.model_path='vad_marblenet' \
+    diarizer.speaker_embeddings.parameters.save_embeddings=False
+
+Check out whole parameters in ./conf/offline_diarization.yaml and their meanings.
+For details, have a look at <NeMo_git_root>/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb
+"""
+
+seed_everything(42)
+
+
+@hydra_runner(config_path="../conf/inference", config_name="diar_infer_meeting.yaml")
+def main(cfg):
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+    sd_model = ClusteringDiarizer(cfg=cfg).to(cfg.device)
+    sd_model.diarize()
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,94 @@
+# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.parts.utils.decoder_timestamps_utils import ASRDecoderTimeStamps
+from nemo.collections.asr.parts.utils.diarization_utils import OfflineDiarWithASR
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+
+
+"""
+This script demonstrates how to run offline speaker diarization with asr.
+Usage:
+python offline_diar_with_asr_infer.py \
+    diarizer.manifest_filepath=<path to manifest file> \
+    diarizer.out_dir='demo_asr_output' \
+    diarizer.speaker_embeddings.model_path=<pretrained modelname or path to .nemo> \
+    diarizer.asr.model_path=<pretrained modelname or path to .nemo> \
+    diarizer.asr.parameters.asr_based_vad=True \
+    diarizer.speaker_embeddings.parameters.save_embeddings=False
+
+Check out whole parameters in ./conf/offline_diarization_with_asr.yaml and their meanings.
+For details, have a look at <NeMo_git_root>/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb
+Currently, the following NGC models are supported:
+
+    stt_en_quartznet15x5
+    stt_en_citrinet*
+    stt_en_conformer_ctc*
+
+"""
+
+
+@hydra_runner(config_path="../conf/inference", config_name="diar_infer_meeting.yaml")
+def main(cfg):
+
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+
+    # ASR inference for words and word timestamps
+    asr_decoder_ts = ASRDecoderTimeStamps(cfg.diarizer)
+    asr_model = asr_decoder_ts.set_asr_model()
+    word_hyp, word_ts_hyp = asr_decoder_ts.run_ASR(asr_model)
+
+    # Create a class instance for matching ASR and diarization results
+    asr_diar_offline = OfflineDiarWithASR(cfg.diarizer)
+    asr_diar_offline.word_ts_anchor_offset = asr_decoder_ts.word_ts_anchor_offset
+
+    # Diarization inference for speaker labels
+    diar_hyp, diar_score = asr_diar_offline.run_diarization(cfg, word_ts_hyp)
+    trans_info_dict = asr_diar_offline.get_transcript_with_speaker_labels(diar_hyp, word_hyp, word_ts_hyp)
+
+    # If RTTM is provided and DER evaluation
+    if diar_score is not None:
+        # Get session-level diarization error rate and speaker counting error
+        der_results = OfflineDiarWithASR.gather_eval_results(
+            diar_score=diar_score,
+            audio_rttm_map_dict=asr_diar_offline.AUDIO_RTTM_MAP,
+            trans_info_dict=trans_info_dict,
+            root_path=asr_diar_offline.root_path,
+        )
+
+        # Calculate WER and cpWER if reference CTM files exist
+        wer_results = OfflineDiarWithASR.evaluate(
+            hyp_trans_info_dict=trans_info_dict,
+            audio_file_list=asr_diar_offline.audio_file_list,
+            ref_ctm_file_list=asr_diar_offline.ctm_file_list,
+        )
+
+        # Print average DER, WER and cpWER
+        OfflineDiarWithASR.print_errors(der_results=der_results, wer_results=wer_results)
+
+        # Save detailed session-level evaluation results in `root_path`.
+        OfflineDiarWithASR.write_session_level_result_in_csv(
+            der_results=der_results,
+            wer_results=wer_results,
+            root_path=asr_diar_offline.root_path,
+            csv_columns=asr_diar_offline.csv_columns,
+        )
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,70 @@
+# This YAML file is created for all types of offline speaker diarization inference tasks in `<NeMo git root>/example/speaker_tasks/diarization` folder.
+# The inference parameters for VAD, speaker embedding extractor, clustering module, ASR decoder are all included in this YAML file.
+# All the keys under `diarizer` key (`vad`, `speaker_embeddings`, `clustering`, `asr`) can be selectively used for its own purpose and also can be ignored if the module is not used.
+# The configurations in this YAML file is optimized to show balanced performances on various types of domain. VAD is optimized on multilingual ASR datasets and diarizer is optimized on DIHARD3 development set.
+# An example line in an input manifest file (`.json` format):
+# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "infer", "text": "-", "num_speakers": null, "rttm_filepath": "/path/to/rttm/file", "uem_filepath": "/path/to/uem/file"}
+name: &name "ClusterDiarizer"
+
+num_workers: 1
+sample_rate: 16000
+batch_size: 64
+device: null # can specify a specific device, i.e: cuda:1 (default cuda if cuda available, else cpu)
+verbose: True # enable additional logging
+
+diarizer:
+  manifest_filepath: ???
+  out_dir: ???
+  oracle_vad: False # If True, uses RTTM files provided in the manifest file to get speech activity (VAD) timestamps
+  collar: 0.25 # Collar value for scoring
+  ignore_overlap: True # Consider or ignore overlap segments while scoring
+
+  vad:
+    model_path: vad_multilingual_marblenet # .nemo local model path or pretrained VAD model name
+    external_vad_manifest: null # This option is provided to use external vad and provide its speech activity labels for speaker embeddings extraction. Only one of model_path or external_vad_manifest should be set
+
+    parameters: # Tuned by detection error rate (false alarm + miss) on multilingual ASR evaluation datasets
+      window_length_in_sec: 0.63  # Window length in sec for VAD context input
+      shift_length_in_sec: 0.08 # Shift length in sec for generate frame level VAD prediction
+      smoothing: False # False or type of smoothing method (eg: median)
+      overlap: 0.5 # Overlap ratio for overlapped mean/median smoothing filter
+      onset: 0.5 # Onset threshold for detecting the beginning and end of a speech
+      offset: 0.3 # Offset threshold for detecting the end of a speech
+      pad_onset: 0.2 # Adding durations before each speech segment
+      pad_offset: 0.2 # Adding durations after each speech segment
+      min_duration_on: 0.5 # Threshold for short speech segment deletion
+      min_duration_off: 0.5 # Threshold for small non_speech deletion
+      filter_speech_first: True
+
+  speaker_embeddings:
+    model_path: titanet_large # .nemo local model path or pretrained model name (titanet_large, ecapa_tdnn or speakerverification_speakernet)
+    parameters:
+      window_length_in_sec: [1.9,1.2,0.5] # Window length(s) in sec (floating-point number). either a number or a list. ex) 1.5 or [1.5,1.0,0.5]
+      shift_length_in_sec: [0.95,0.6,0.25] # Shift length(s) in sec (floating-point number). either a number or a list. ex) 0.75 or [0.75,0.5,0.25]
+      multiscale_weights: [1,1,1] # Weight for each scale. should be null (for single scale) or a list matched with window/shift scale count. ex) [0.33,0.33,0.33]
+      save_embeddings: True # If True, save speaker embedding tensor.
+
+  clustering:
+    parameters:
+      oracle_num_speakers: False # If True, use num of speakers value provided in manifest file.
+      max_num_speakers: 8 # Max number of speakers for each recording. If an oracle number of speakers is passed, this value is ignored.
+      enhanced_count_thres: 80 # If the number of segments is lower than this number, enhanced speaker counting is activated.
+      max_rp_threshold: 0.25 # Determines the range of p-value search: 0 < p <= max_rp_threshold.
+      sparse_search_volume: 10 # The higher the number, the more values will be examined with more time.
+      maj_vote_spk_count: False  # If True, take a majority vote on multiple p-values to estimate the number of speakers.
+      chunk_cluster_count: 50 # Number of forced clusters (overclustering) per unit chunk in long-form audio clustering.
+      embeddings_per_chunk: 10000 # Number of embeddings in each chunk for long-form audio clustering. Adjust based on GPU memory capacity. (default: 10000, approximately 40 mins of audio)
+
+  asr:
+    model_path: null # Provide NGC cloud ASR model name. stt_en_conformer_ctc_* models are recommended for diarization purposes.
+    parameters:
+      asr_based_vad: False # if True, speech segmentation for diarization is based on word-timestamps from ASR inference.
+      asr_based_vad_threshold: 1.0 # Threshold (in sec) that caps the gap between two words when generating VAD timestamps using ASR based VAD.
+      asr_batch_size: null # Batch size can be dependent on each ASR model. Default batch sizes are applied if set to null.
+      decoder_delay_in_sec: null # Native decoder delay. null is recommended to use the default values for each ASR model.
+      word_ts_anchor_offset: null # Offset to set a reference point from the start of the word. Recommended range of values is [-0.05  0.2].
+      word_ts_anchor_pos: "start" # Select which part of the word timestamp we want to use. The options are: 'start', 'end', 'mid'.
+      fix_word_ts_with_VAD: False # Fix the word timestamp using VAD output. You must provide a VAD model to use this feature.
+      colored_text: False # If True, use colored text to distinguish speakers in the output transcript.
+      print_time: True # If True, the start and end time of each speaker turn is printed in the output transcript.
+      break_lines: False # If True, the output transcript breaks the line to fix the line width (default is 90 chars)
@@ -0,0 +1,70 @@
+# This YAML file is created for all types of offline speaker diarization inference tasks in `<NeMo git root>/example/speaker_tasks/diarization` folder.
+# The inference parameters for VAD, speaker embedding extractor, clustering module, ASR decoder are all included in this YAML file.
+# All the keys under `diarizer` key (`vad`, `speaker_embeddings`, `clustering`, `asr`) can be selectively used for its own purpose and also can be ignored if the module is not used.
+# The configurations in this YAML file is suitable for 3~5 speakers participating in a meeting and may not show the best performance on other types of dialogues.
+# An example line in an input manifest file (`.json` format):
+# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "infer", "text": "-", "num_speakers": null, "rttm_filepath": "/path/to/rttm/file", "uem_filepath": "/path/to/uem/file"}
+name: &name "ClusterDiarizer"
+
+num_workers: 1
+sample_rate: 16000
+batch_size: 64
+device: null # can specify a specific device, i.e: cuda:1 (default cuda if cuda available, else cpu)
+verbose: True # enable additional logging
+
+diarizer:
+  manifest_filepath: ???
+  out_dir: ???
+  oracle_vad: False # If True, uses RTTM files provided in the manifest file to get speech activity (VAD) timestamps
+  collar: 0.25 # Collar value for scoring
+  ignore_overlap: True # Consider or ignore overlap segments while scoring
+
+  vad:
+    model_path:  vad_multilingual_marblenet # .nemo local model path or pretrained VAD model name
+    external_vad_manifest: null # This option is provided to use external vad and provide its speech activity labels for speaker embeddings extraction. Only one of model_path or external_vad_manifest should be set
+
+    parameters: # Tuned parameters for CH109 (using the 11 multi-speaker sessions as dev set)
+      window_length_in_sec: 0.63  # Window length in sec for VAD context input
+      shift_length_in_sec: 0.01 # Shift length in sec for generate frame level VAD prediction
+      smoothing: False # False or type of smoothing method (eg: median)
+      overlap: 0.5 # Overlap ratio for overlapped mean/median smoothing filter
+      onset: 0.9 # Onset threshold for detecting the beginning and end of a speech
+      offset: 0.5 # Offset threshold for detecting the end of a speech
+      pad_onset: 0 # Adding durations before each speech segment
+      pad_offset: 0 # Adding durations after each speech segment
+      min_duration_on: 0 # Threshold for short speech segment deletion
+      min_duration_off: 0.6 # Threshold for small non_speech deletion
+      filter_speech_first: True
+
+  speaker_embeddings:
+    model_path: titanet_large # .nemo local model path or pretrained model name (titanet_large, ecapa_tdnn or speakerverification_speakernet)
+    parameters:
+      window_length_in_sec: [3.0,2.5,2.0,1.5,1.0,0.5] # Window length(s) in sec (floating-point number). either a number or a list. ex) 1.5 or [1.5,1.0,0.5]
+      shift_length_in_sec: [1.5,1.25,1.0,0.75,0.5,0.25] # Shift length(s) in sec (floating-point number). either a number or a list. ex) 0.75 or [0.75,0.5,0.25]
+      multiscale_weights: [1,1,1,1,1,1] # Weight for each scale. should be null (for single scale) or a list matched with window/shift scale count. ex) [0.33,0.33,0.33]
+      save_embeddings: True # If True, save speaker embedding tensor.
+
+  clustering:
+    parameters:
+      oracle_num_speakers: False # If True, use num of speakers value provided in manifest file.
+      max_num_speakers: 8 # Max number of speakers for each recording. If an oracle number of speakers is passed, this value is ignored.
+      enhanced_count_thres: 80 # If the number of segments is lower than this number, enhanced speaker counting is activated.
+      max_rp_threshold: 0.25 # Determines the range of p-value search: 0 < p <= max_rp_threshold.
+      sparse_search_volume: 30 # The higher the number, the more values will be examined with more time.
+      maj_vote_spk_count: False  # If True, take a majority vote on multiple p-values to estimate the number of speakers.
+      chunk_cluster_count: 50 # Number of forced clusters (overclustering) per unit chunk in long-form audio clustering.
+      embeddings_per_chunk: 10000 # Number of embeddings in each chunk for long-form audio clustering. Adjust based on GPU memory capacity. (default: 10000, approximately 40 mins of audio)
+
+  asr:
+    model_path: stt_en_conformer_ctc_large # Provide NGC cloud ASR model name. stt_en_conformer_ctc_* models are recommended for diarization purposes.
+    parameters:
+      asr_based_vad: False # if True, speech segmentation for diarization is based on word-timestamps from ASR inference.
+      asr_based_vad_threshold: 1.0 # Threshold (in sec) that caps the gap between two words when generating VAD timestamps using ASR based VAD.
+      asr_batch_size: null # Batch size can be dependent on each ASR model. Default batch sizes are applied if set to null.
+      decoder_delay_in_sec: null # Native decoder delay. null is recommended to use the default values for each ASR model.
+      word_ts_anchor_offset: null # Offset to set a reference point from the start of the word. Recommended range of values is [-0.05  0.2].
+      word_ts_anchor_pos: "start" # Select which part of the word timestamp we want to use. The options are: 'start', 'end', 'mid'.
+      fix_word_ts_with_VAD: False # Fix the word timestamp using VAD output. You must provide a VAD model to use this feature.
+      colored_text: False # If True, use colored text to distinguish speakers in the output transcript.
+      print_time: True # If True, the start and end time of each speaker turn is printed in the output transcript.
+      break_lines: False # If True, the output transcript breaks the line to fix the line width (default is 90 chars)
@@ -0,0 +1,70 @@
+# This YAML file is created for all types of offline speaker diarization inference tasks in `<NeMo git root>/example/speaker_tasks/diarization` folder.
+# The inference parameters for VAD, speaker embedding extractor, clustering module, ASR decoder are all included in this YAML file.
+# All the keys under `diarizer` key (`vad`, `speaker_embeddings`, `clustering`, `asr`) can be selectively used for its own purpose and also can be ignored if the module is not used.
+# The configurations in this YAML file is suitable for telephone recordings involving 2~8 speakers in a session and may not show the best performance on the other types of acoustic conditions or dialogues.
+# An example line in an input manifest file (`.json` format):
+# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "infer", "text": "-", "num_speakers": null, "rttm_filepath": "/path/to/rttm/file", "uem_filepath": "/path/to/uem/file"}
+name: &name "ClusterDiarizer"
+
+num_workers: 1
+sample_rate: 16000
+batch_size: 64
+device: null # can specify a specific device, i.e: cuda:1 (default cuda if cuda available, else cpu)
+verbose: True # enable additional logging
+
+diarizer:
+  manifest_filepath: ???
+  out_dir: ???
+  oracle_vad: False # If True, uses RTTM files provided in the manifest file to get speech activity (VAD) timestamps
+  collar: 0.25 # Collar value for scoring
+  ignore_overlap: True # Consider or ignore overlap segments while scoring
+
+  vad:
+    model_path: vad_multilingual_marblenet # .nemo local model path or pretrained VAD model name
+    external_vad_manifest: null # This option is provided to use external vad and provide its speech activity labels for speaker embeddings extraction. Only one of model_path or external_vad_manifest should be set
+
+    parameters: # Tuned parameters for CH109 (using the 11 multi-speaker sessions as dev set)
+      window_length_in_sec: 0.15  # Window length in sec for VAD context input
+      shift_length_in_sec: 0.01 # Shift length in sec for generate frame level VAD prediction
+      smoothing: "median" # False or type of smoothing method (eg: median)
+      overlap: 0.5 # Overlap ratio for overlapped mean/median smoothing filter
+      onset: 0.1 # Onset threshold for detecting the beginning and end of a speech
+      offset: 0.1 # Offset threshold for detecting the end of a speech
+      pad_onset: 0.1 # Adding durations before each speech segment
+      pad_offset: 0 # Adding durations after each speech segment
+      min_duration_on: 0 # Threshold for short speech segment deletion
+      min_duration_off: 0.2 # Threshold for small non_speech deletion
+      filter_speech_first: True
+
+  speaker_embeddings:
+    model_path: titanet_large # .nemo local model path or pretrained model name (titanet_large, ecapa_tdnn or speakerverification_speakernet)
+    parameters:
+      window_length_in_sec: [1.5,1.25,1.0,0.75,0.5] # Window length(s) in sec (floating-point number). either a number or a list. ex) 1.5 or [1.5,1.0,0.5]
+      shift_length_in_sec: [0.75,0.625,0.5,0.375,0.25] # Shift length(s) in sec (floating-point number). either a number or a list. ex) 0.75 or [0.75,0.5,0.25]
+      multiscale_weights: [1,1,1,1,1] # Weight for each scale. should be null (for single scale) or a list matched with window/shift scale count. ex) [0.33,0.33,0.33]
+      save_embeddings: True # If True, save speaker embedding tensor.
+
+  clustering:
+    parameters:
+      oracle_num_speakers: False # If True, use num of speakers value provided in manifest file.
+      max_num_speakers: 8 # Max number of speakers for each recording. If an oracle number of speakers is passed, this value is ignored.
+      enhanced_count_thres: 80 # If the number of segments is lower than this number, enhanced speaker counting is activated.
+      max_rp_threshold: 0.25 # Determines the range of p-value search: 0 < p <= max_rp_threshold.
+      sparse_search_volume: 30 # The higher the number, the more values will be examined with more time.
+      maj_vote_spk_count: False  # If True, take a majority vote on multiple p-values to estimate the number of speakers.
+      chunk_cluster_count: 50 # Number of forced clusters (overclustering) per unit chunk in long-form audio clustering.
+      embeddings_per_chunk: 10000 # Number of embeddings in each chunk for long-form audio clustering. Adjust based on GPU memory capacity. (default: 10000, approximately 40 mins of audio)
+
+  asr:
+    model_path: stt_en_conformer_ctc_large # Provide NGC cloud ASR model name. stt_en_conformer_ctc_* models are recommended for diarization purposes.
+    parameters:
+      asr_based_vad: False # if True, speech segmentation for diarization is based on word-timestamps from ASR inference.
+      asr_based_vad_threshold: 1.0 # Threshold (in sec) that caps the gap between two words when generating VAD timestamps using ASR based VAD.
+      asr_batch_size: null # Batch size can be dependent on each ASR model. Default batch sizes are applied if set to null.
+      decoder_delay_in_sec: null # Native decoder delay. null is recommended to use the default values for each ASR model.
+      word_ts_anchor_offset: null # Offset to set a reference point from the start of the word. Recommended range of values is [-0.05  0.2].
+      word_ts_anchor_pos: "start" # Select which part of the word timestamp we want to use. The options are: 'start', 'end', 'mid'.
+      fix_word_ts_with_VAD: False # Fix the word timestamp using VAD output. You must provide a VAD model to use this feature.
+      colored_text: False # If True, use colored text to distinguish speakers in the output transcript.
+      print_time: True # If True, the start and end time of each speaker turn is printed in the output transcript.
+      break_lines: False # If True, the output transcript breaks the line to fix the line width (default is 90 chars)
@@ -0,0 +1,191 @@
+# Sortformer Diarizer is an end-to-end speaker diarization model that is solely based on Transformer-encoder type of architecture.
+# Model name convention for Sortformer Diarizer: sortformer_diarizer_<loss_type>_<speaker count limit>-<version>.yaml
+# (Example) `sortformer_diarizer_hybrid_loss_4spk-v1.yaml`.
+# Sortformer Diarizer model checkpoint (.ckpt) and NeMo file (.nemo) contain Fast Conformer Encoder model (NEST Encoder) and the pre-trained NEST model is loaded along with the Transformer Encoder layers.
+# Example: a manifest line for training 
+# {"audio_filepath": "/path/to/audio01.wav", "offset": 390.83, "duration": 90.00, "text": "-", "num_speakers": 2, "rttm_filepath": "/path/to/audio01.rttm"}
+name: "SortFormerDiarizer"
+num_workers: 18
+batch_size: 8
+
+model: 
+  sample_rate: 16000
+  pil_weight: 0.5 # Weight for Permutation Invariant Loss (PIL) used in training the Sortformer diarizer model
+  ats_weight: 0.5 # Weight for Arrival Time Sort (ATS) loss in training the Sortformer diarizer model
+  max_num_of_spks: 4 # Maximum number of speakers per model; currently set to 4
+
+  model_defaults:
+    fc_d_model: 512 # Hidden dimension size of the Fast-conformer Encoder
+    tf_d_model: 192 # Hidden dimension size of the Transformer Encoder
+
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: ${model.sample_rate}
+    num_spks: ${model.max_num_of_spks}
+    session_len_sec: 90 # Maximum session length in seconds
+    soft_label_thres: 0.5 # Threshold for binarizing target values; higher values make the model more conservative in predicting speaker activity.
+    soft_targets: False # If True, use continuous values as target values when calculating cross-entropy loss
+    labels: null
+    batch_size: ${batch_size}
+    shuffle: True
+    num_workers: ${num_workers}
+    validation_mode: False
+    # lhotse config
+    use_lhotse: False
+    use_bucketing: True
+    num_buckets: 10
+    bucket_duration_bins: [10, 20, 30, 40, 50, 60, 70, 80, 90]
+    pin_memory: True
+    min_duration: 10
+    max_duration: 90
+    batch_duration: 400
+    quadratic_duration: 1200
+    bucket_buffer_size: 20000
+    shuffle_buffer_size: 10000
+    window_stride: ${model.preprocessor.window_stride}
+    subsampling_factor: ${model.encoder.subsampling_factor}
+
+  validation_ds:
+    manifest_filepath: ???
+    is_tarred: False
+    tarred_audio_filepaths: null
+    sample_rate: ${model.sample_rate}
+    num_spks: ${model.max_num_of_spks}
+    session_len_sec: 90 # Maximum session length in seconds
+    soft_label_thres: 0.5 # A threshold value for setting up the binarized labels. The higher the more conservative the model becomes. 
+    soft_targets: False
+    labels: null
+    batch_size: ${batch_size}
+    shuffle: False
+    num_workers: ${num_workers}
+    validation_mode: True
+    # lhotse config
+    use_lhotse: False
+    use_bucketing: False
+    drop_last: False
+    pin_memory: True
+    window_stride: ${model.preprocessor.window_stride}
+    subsampling_factor: ${model.encoder.subsampling_factor}
+  
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.025
+    sample_rate: ${model.sample_rate}
+    window_stride: 0.01
+    window: "hann"
+    features: 80
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  sortformer_modules:
+    _target_: nemo.collections.asr.modules.sortformer_modules.SortformerModules
+    num_spks: ${model.max_num_of_spks} # Number of speakers per model. This is currently fixed at 4.
+    dropout_rate: 0.5 # Dropout rate
+    fc_d_model: ${model.model_defaults.fc_d_model}
+    tf_d_model: ${model.model_defaults.tf_d_model} # Hidden layer size for linear layers in Sortformer Diarizer module
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConformerEncoder
+    feat_in: ${model.preprocessor.features}
+    feat_out: -1
+    n_layers: 18
+    d_model: ${model.model_defaults.fc_d_model}
+
+    # Sub-sampling parameters
+    subsampling: dw_striding # vggnet, striding, stacking or stacking_norm, dw_striding
+    subsampling_factor: 8 # must be power of 2 for striding and vggnet
+    subsampling_conv_channels: 256 # set to -1 to make it equal to the d_model
+    causal_downsampling: false
+    # Feed forward module's params
+    ff_expansion_factor: 4
+    # Multi-headed Attention Module's params
+    self_attention_model: rel_pos # rel_pos or abs_pos
+    n_heads: 8 # may need to be lower for smaller d_models
+    # [left, right] specifies the number of steps to be seen from left and right of each step in self-attention
+    att_context_size: [-1, -1] # -1 means unlimited context
+    att_context_style: regular # regular or chunked_limited
+    xscaling: true # scales up the input embeddings by sqrt(d_model)
+    untie_biases: true # unties the biases of the TransformerXL layers
+    pos_emb_max_len: 5000
+    # Convolution module's params
+    conv_kernel_size: 9
+    conv_norm_type: 'batch_norm' # batch_norm or layer_norm or groupnormN (N specifies the number of groups)
+    conv_context_size: null
+    # Regularization
+    dropout: 0.1 # The dropout used in most of the Conformer Modules
+    dropout_pre_encoder: 0.1 # The dropout used before the encoder
+    dropout_emb: 0.0 # The dropout used for embeddings
+    dropout_att: 0.1 # The dropout for multi-headed attention modules
+    # Set to non-zero to enable stochastic depth
+    stochastic_depth_drop_prob: 0.0
+    stochastic_depth_mode: linear  # linear or uniform
+    stochastic_depth_start_layer: 1
+  
+  transformer_encoder:
+    _target_: nemo.collections.asr.modules.transformer.transformer_encoders.TransformerEncoder
+    num_layers: 18
+    hidden_size: ${model.model_defaults.tf_d_model} # Needs to be multiple of num_attention_heads
+    inner_size: 768
+    num_attention_heads: 8
+    attn_score_dropout: 0.5
+    attn_layer_dropout: 0.5
+    ffn_dropout: 0.5
+    hidden_act: relu
+    pre_ln: False
+    pre_ln_final_layer_norm: True
+  
+  loss: 
+    _target_: nemo.collections.asr.losses.bce_loss.BCELoss
+    weight: null # Weight for binary cross-entropy loss. Either `null` or list type input. (e.g. [0.5,0.5])
+    reduction: mean
+
+  lr: 0.0001
+  optim:
+    name: adamw
+    lr: ${model.lr}
+    # optimizer arguments
+    betas: [0.9, 0.98]
+    weight_decay: 1e-3
+
+    sched:
+      name: InverseSquareRootAnnealing
+      warmup_steps: 2500
+      warmup_ratio: null
+      min_lr: 1e-06
+
+trainer:
+  devices: 1 # number of gpus (devices)
+  accelerator: gpu 
+  precision: 32 # 32, bf16, bf16-mixed
+  max_epochs: 800
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  strategy: ddp_find_unused_parameters_true # Could be "ddp"
+  accumulate_grad_batches: 1
+  deterministic: True
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+
+exp_manager:
+  use_datetime_version: False
+  exp_dir: null
+  name: ${name}
+  resume_if_exists: True
+  resume_from_checkpoint: null # The path to a checkpoint file to continue the training, restores the whole state including the epoch, step, LR schedulers, apex, etc.
+  resume_ignore_no_checkpoint: True
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
+  create_wandb_logger: False
+  checkpoint_callback_params:
+    monitor: "val_f1_acc"
+    mode: "max"
+    save_top_k: 9
+    every_n_epochs: 1
+  wandb_logger_kwargs:
+    resume: True
+    name: null
+    project: null
@@ -0,0 +1,235 @@
+# Sortformer Diarizer is an end-to-end speaker diarization model that is solely based on Transformer-encoder type of architecture.
+# Model name convention for Sortformer Diarizer: streaming_sortformer_diarizer_<speaker count limit>-<version>.yaml
+# (Example) `streaming_sortformer_diarizer_4spk-v2.yaml`.
+# Sortformer Diarizer model checkpoint (.ckpt) and NeMo file (.nemo) contain Fast Conformer Encoder model (NEST Encoder) and the pre-trained NEST model is loaded along with the Transformer Encoder layers.
+# Example: a manifest line for training 
+# {"audio_filepath": "/path/to/audio01.wav", "offset": 390.83, "duration": 90.00, "text": "-", "num_speakers": 2, "rttm_filepath": "/path/to/audio01.rttm"}
+name: "StreamingSortformerDiarizer"
+num_workers: 18
+batch_size: 4
+
+model: 
+  sample_rate: 16000
+  pil_weight: 0.5 # Weight for Permutation Invariant Loss (PIL) used in training the Sortformer diarizer model
+  ats_weight: 0.5 # Weight for Arrival Time Sort (ATS) loss in training the Sortformer diarizer model
+  max_num_of_spks: 4 # Maximum number of speakers per model; currently set to 4
+  streaming_mode: True
+
+  model_defaults:
+    fc_d_model: 512 # Hidden dimension size of the Fast-conformer Encoder
+    tf_d_model: 192 # Hidden dimension size of the Transformer Encoder
+
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: ${model.sample_rate}
+    num_spks: ${model.max_num_of_spks}
+    session_len_sec: 90 # Maximum session length in seconds
+    soft_label_thres: 0.5 # Threshold for binarizing target values; higher values make the model more conservative in predicting speaker activity.
+    soft_targets: False # If True, use continuous values as target values when calculating cross-entropy loss
+    labels: null
+    batch_size: ${batch_size}
+    shuffle: True
+    num_workers: ${num_workers}
+    validation_mode: False
+    # lhotse config
+    use_lhotse: False
+    use_bucketing: True
+    num_buckets: 10
+    bucket_duration_bins: [10, 20, 30, 40, 50, 60, 70, 80, 90]
+    pin_memory: True
+    min_duration: 10
+    max_duration: 90
+    batch_duration: 400
+    quadratic_duration: 1200
+    bucket_buffer_size: 20000
+    shuffle_buffer_size: 10000
+    window_stride: ${model.preprocessor.window_stride}
+    subsampling_factor: ${model.encoder.subsampling_factor}
+
+  validation_ds:
+    manifest_filepath: ???
+    is_tarred: False
+    tarred_audio_filepaths: null
+    sample_rate: ${model.sample_rate}
+    num_spks: ${model.max_num_of_spks}
+    session_len_sec: 90 # Maximum session length in seconds
+    soft_label_thres: 0.5 # A threshold value for setting up the binarized labels. The higher the more conservative the model becomes. 
+    soft_targets: False
+    labels: null
+    batch_size: ${batch_size}
+    shuffle: False
+    num_workers: ${num_workers}
+    validation_mode: True
+    # lhotse config
+    use_lhotse: False
+    use_bucketing: False
+    drop_last: False
+    pin_memory: True
+    window_stride: ${model.preprocessor.window_stride}
+    subsampling_factor: ${model.encoder.subsampling_factor}
+  
+  test_ds:
+    manifest_filepath: null
+    is_tarred: False
+    tarred_audio_filepaths: null
+    sample_rate: 16000
+    num_spks: ${model.max_num_of_spks}
+    session_len_sec: 90 # Maximum session length in seconds
+    soft_label_thres: 0.5
+    soft_targets: False
+    labels: null
+    batch_size: ${batch_size}
+    shuffle: False
+    seq_eval_mode: True
+    num_workers: ${num_workers}
+    validation_mode: True
+    # lhotse config
+    use_lhotse: False
+    use_bucketing: False
+    drop_last: False
+    pin_memory: True
+    window_stride: ${model.preprocessor.window_stride}
+    subsampling_factor: ${model.encoder.subsampling_factor}
+
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "NA"
+    window_size: 0.025
+    sample_rate: ${model.sample_rate}
+    window_stride: 0.01
+    window: "hann"
+    features: 128
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  sortformer_modules:
+    _target_: nemo.collections.asr.modules.sortformer_modules.SortformerModules
+    num_spks: ${model.max_num_of_spks} # Maximum number of speakers the model can handle
+    dropout_rate: 0.5 # Dropout rate
+    fc_d_model: ${model.model_defaults.fc_d_model} # Hidden dimension size for Fast Conformer encoder
+    tf_d_model: ${model.model_defaults.tf_d_model} # Hidden dimension size for Transformer encoder
+    # Streaming mode parameters
+    spkcache_len: 188 # Length of speaker cache buffer (total number of frames for all speakers)
+    fifo_len: 0 # Length of FIFO buffer for streaming processing (0 = disabled)
+    chunk_len: 188 # Number of frames processed in each streaming chunk
+    spkcache_update_period: 188 # Speaker cache update period in frames
+    chunk_left_context: 1 # Number of previous frames for each streaming chunk
+    chunk_right_context: 1 # Number of future frames for each streaming chunk
+    # Speaker cache update parameters
+    spkcache_sil_frames_per_spk: 3 # Number of silence frames allocated per speaker in the speaker cache
+    scores_add_rnd: 0 # Standard deviation of random noise added to scores in speaker cache update (training only)
+    pred_score_threshold: 0.25 # Probability threshold for internal scores processing in speaker cache update
+    max_index: 99999 # Maximum allowed index value for internal processing in speaker cache update
+    scores_boost_latest: 0.05 # Gain for scores for recently added frames in speaker cache update
+    sil_threshold: 0.2 # Threshold for determining silence frames to calculate average silence embedding
+    strong_boost_rate: 0.75 # Rate determining number of frames per speaker that receive strong score boosting
+    weak_boost_rate: 1.5 # Rate determining number of frames per speaker that receive weak score boosting
+    min_pos_scores_rate: 0.5 # Rate threshold for dropping overlapping frames when enough non-overlapping exist
+    # Self-attention parameters (training only)
+    causal_attn_rate: 0.5 # Proportion of batches that use self-attention with limited right context
+    causal_attn_rc: 7 # Right context size for self-attention with limited right context
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConformerEncoder
+    feat_in: ${model.preprocessor.features}
+    feat_out: -1
+    n_layers: 17
+    d_model: ${model.model_defaults.fc_d_model}
+
+    # Sub-sampling parameters
+    subsampling: dw_striding # vggnet, striding, stacking or stacking_norm, dw_striding
+    subsampling_factor: 8 # must be power of 2 for striding and vggnet
+    subsampling_conv_channels: 256 # set to -1 to make it equal to the d_model
+    causal_downsampling: false
+    # Feed forward module's params
+    ff_expansion_factor: 4
+    # Multi-headed Attention Module's params
+    self_attention_model: rel_pos # rel_pos or abs_pos
+    n_heads: 8 # may need to be lower for smaller d_models
+    # [left, right] specifies the number of steps to be seen from left and right of each step in self-attention
+    att_context_size: [-1, -1] # -1 means unlimited context
+    att_context_style: regular # regular or chunked_limited
+    xscaling: true # scales up the input embeddings by sqrt(d_model)
+    untie_biases: true # unties the biases of the TransformerXL layers
+    pos_emb_max_len: 5000
+    # Convolution module's params
+    conv_kernel_size: 9
+    conv_norm_type: 'batch_norm' # batch_norm or layer_norm or groupnormN (N specifies the number of groups)
+    conv_context_size: null
+    # Regularization
+    dropout: 0.1 # The dropout used in most of the Conformer Modules
+    dropout_pre_encoder: 0.1 # The dropout used before the encoder
+    dropout_emb: 0.0 # The dropout used for embeddings
+    dropout_att: 0.1 # The dropout for multi-headed attention modules
+    # Set to non-zero to enable stochastic depth
+    stochastic_depth_drop_prob: 0.0
+    stochastic_depth_mode: linear  # linear or uniform
+    stochastic_depth_start_layer: 1
+  
+  transformer_encoder:
+    _target_: nemo.collections.asr.modules.transformer.transformer_encoders.TransformerEncoder
+    num_layers: 18
+    hidden_size: ${model.model_defaults.tf_d_model} # Needs to be multiple of num_attention_heads
+    inner_size: 768
+    num_attention_heads: 8
+    attn_score_dropout: 0.5
+    attn_layer_dropout: 0.5
+    ffn_dropout: 0.5
+    hidden_act: relu
+    pre_ln: False
+    pre_ln_final_layer_norm: True
+  
+  loss: 
+    _target_: nemo.collections.asr.losses.bce_loss.BCELoss
+    weight: null # Weight for binary cross-entropy loss. Either `null` or list type input. (e.g. [0.5,0.5])
+    reduction: mean
+
+  lr: 0.0001
+  optim:
+    name: adamw
+    lr: ${model.lr}
+    # optimizer arguments
+    betas: [0.9, 0.98]
+    weight_decay: 1e-3
+
+    sched:
+      name: InverseSquareRootAnnealing
+      warmup_steps: 500
+      warmup_ratio: null
+      min_lr: 1e-06
+
+trainer:
+  devices: 1 # number of gpus (devices)
+  accelerator: gpu
+  precision: 32 # 32, bf16, bf16-mixed
+  max_epochs: 800
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  strategy: ddp_find_unused_parameters_true # Could be "ddp"
+  accumulate_grad_batches: 1
+  deterministic: True
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+
+exp_manager:
+  use_datetime_version: False
+  exp_dir: null
+  name: ${name}
+  resume_if_exists: True
+  resume_from_checkpoint: null # The path to a checkpoint file to continue the training, restores the whole state including the epoch, step, LR schedulers, apex, etc.
+  resume_ignore_no_checkpoint: True
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
+  create_wandb_logger: False
+  checkpoint_callback_params:
+    monitor: "val_f1_acc"
+    mode: "max"
+    save_top_k: 9
+    every_n_epochs: 1
+  wandb_logger_kwargs:
+    resume: True
+    name: null
+    project: null
@@ -0,0 +1,11 @@
+# Postprocessing parameters for timestamp outputs from speaker diarization models.
+# This speaker diarization postprocessing scheme is inspired by the postprocessing procedure in the following paper:
+# Medennikov, Ivan, et al. "Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario." (2020).
+# These parameters were optimized on CallHome Dataset from the NIST SRE 2000 Disc8, especially from the part1 (callhome1) specified in: Kaldi, “Kaldi x-vector recipe v2,” https://github.com/kaldi-asr/kaldi/blob/master/egs/callhome_diarization/v2/run.sh
+parameters: 
+  onset: 0.641 # Onset threshold for detecting the beginning and end of a speech
+  offset: 0.561 # Offset threshold for detecting the end of a speech
+  pad_onset: 0.229  # Adding durations before each speech segment
+  pad_offset: 0.079  # Adding durations after each speech segment
+  min_duration_on: 0.511 # Threshold for small non-speech deletion
+  min_duration_off: 0.296 # Threshold for short speech segment deletion
@@ -0,0 +1,11 @@
+# Postprocessing parameters for timestamp outputs from speaker diarization models.
+# This speaker diarization postprocessing scheme is inspired by the postprocessing procedure in the following paper:
+# Medennikov, Ivan, et al. "Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario." (2020).
+# These parameters were optimized on the development split of DIHARD3 dataset (See https://arxiv.org/pdf/2012.01477).
+parameters: 
+  onset: 0.56  # Onset threshold for detecting the beginning and end of a speech
+  offset: 1.0  # Offset threshold for detecting the end of a speech
+  pad_onset: 0.063  # Adding durations before each speech segment
+  pad_offset: 0.002  # Adding durations after each speech segment
+  min_duration_on: 0.007  # Threshold for small non-speech deletion
+  min_duration_off: 0.151  # Threshold for short speech segment deletion
@@ -0,0 +1,13 @@
+# Postprocessing parameters for timestamp outputs from speaker diarization models.
+# This speaker diarization postprocessing scheme is inspired by the postprocessing procedure in the following paper:
+# Medennikov, Ivan, et al. "Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario." (2020).
+# These parameters were optimized with hybrid-loss trained Sortformer model introduced in https://arxiv.org/pdf/2409.06656.
+# These parameters were optimized on CallHome Dataset from the NIST SRE 2000 Disc8, especially from the part1 (callhome1) specified in: Kaldi, “Kaldi x-vector recipe v2,” https://github.com/kaldi-asr/kaldi/blob/master/egs/callhome_diarization/v2/run.sh
+# Trial 24682 finished with value: 0.10257785779242055 and parameters: {'onset': 0.53, 'offset': 0.49, 'pad_onset': 0.23, 'pad_offset': 0.01, 'min_duration_on': 0.42, 'min_duration_off': 0.34}. Best is trial 24682 with value: 0.10257785779242055.
+parameters: 
+  onset: 0.53  # Onset threshold for detecting the beginning of a speech segment
+  offset: 0.49  # Offset threshold for detecting the end of a speech segment
+  pad_onset: 0.23  # Adds the specified duration at the beginning of each speech segment
+  pad_offset: 0.01  # Adds the specified duration at the end of each speech segment
+  min_duration_on: 0.42  # Removes short speech segments if the duration is less than the specified minimum duration
+  min_duration_off: 0.34  # Removes short silences if the duration is less than the specified minimum duration
@@ -0,0 +1,13 @@
+# Postprocessing parameters for timestamp outputs from speaker diarization models.
+# This speaker diarization postprocessing scheme is inspired by the postprocessing procedure in the following paper:
+# Medennikov, Ivan, et al. "Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario." (2020).
+# These parameters were optimized with hybrid-loss trained Sortformer model introduced in https://arxiv.org/pdf/2409.06656.
+# These parameters were optimized on the development split of DIHARD3 dataset (See https://arxiv.org/pdf/2012.01477).
+# Trial 732 finished with value: 0.12171946949255649 and parameters: {'onset': 0.64, 'offset': 0.74, 'pad_onset': 0.06, 'pad_offset': 0.0, 'min_duration_on': 0.1, 'min_duration_off': 0.15}. Best is trial 732 with value: 0.12171946949255649. 
+parameters: 
+  onset: 0.64  # Onset threshold for detecting the beginning of a speech segment
+  offset: 0.74  # Offset threshold for detecting the end of a speech segment
+  pad_onset: 0.06  # Adds the specified duration at the beginning of each speech segment
+  pad_offset: 0.0  # Adds the specified duration at the end of each speech segment
+  min_duration_on: 0.1  # Removes short speech segments if the duration is less than the specified minimum duration
+  min_duration_off: 0.15  # Removes short silences if the duration is less than the specified minimum duration
@@ -0,0 +1,463 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+"""
+This script provides an inference and evaluation script for end-to-end speaker diarization models.
+The performance of the diarization model is measured using the Diarization Error Rate (DER).
+If you want to evaluate its performance, the manifest JSON file should contain the corresponding RTTM
+(Rich Transcription Time Marked) file.
+Please refer to the NeMo Library Documentation for more details on data preparation for diarization inference:
+https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit
+/asr/speaker_diarization/datasets.html#data-preparation-for-inference
+
+Usage for diarization inference:
+
+The end-to-end speaker diarization model can be specified by "model_path".
+Data for diarization is fed through the "dataset_manifest".
+By default, post-processing is bypassed, and only binarization is performed.
+If you want to reproduce DER scores reported on NeMo model cards, you need to apply post-processing steps.
+Use batch_size = 1 to have the longest inference window and the highest possible accuracy.
+
+python $BASEPATH/neural_diarizer/e2e_diarize_speech.py \
+    model_path=/path/to/diar_sortformer_4spk_v1.nemo \
+    batch_size=1 \
+    dataset_manifest=/path/to/diarization_manifest.json
+
+"""
+import json
+import logging
+import os
+import tempfile
+from dataclasses import dataclass, is_dataclass
+from tempfile import NamedTemporaryFile
+from typing import Dict, List, Optional, Union
+
+import lightning.pytorch as pl
+import torch
+from omegaconf import OmegaConf
+from pytorch_lightning import seed_everything
+
+from nemo.collections.asr.metrics.der import score_labels
+from nemo.collections.asr.models import SortformerEncLabelModel
+from nemo.collections.asr.parts.utils.speaker_utils import (
+    audio_rttm_map,
+    get_uniqname_from_filepath,
+    timestamps_to_supervisions,
+)
+from nemo.collections.asr.parts.utils.transcribe_utils import read_and_maybe_sort_manifest
+from nemo.collections.asr.parts.utils.vad_utils import (
+    PostProcessingParams,
+    load_postprocessing_from_yaml,
+    predlist_to_timestamps,
+)
+from nemo.collections.common.parts.preprocessing.manifest import get_full_path
+from nemo.core.config import hydra_runner
+from nemo.utils.dependency import import_optional_dependency
+
+seed_everything(42)
+torch.backends.cudnn.deterministic = True
+
+
+@dataclass
+class DiarizationConfig:
+    """Diarization configuration parameters for inference."""
+
+    model_path: Optional[str] = None  # Path to a .nemo file
+    dataset_manifest: Optional[str] = None  # Path to dataset's JSON manifest
+    presort_manifest: Optional[bool] = True
+
+    postprocessing_yaml: Optional[str] = None  # Path to a yaml file for postprocessing configurations
+    no_der: bool = False
+    out_rttm_dir: Optional[str] = None
+    save_preds_tensors: bool = False
+    precision: str = "32"  # 32, bf16, bf16-mixed
+
+    # General configs
+    session_len_sec: float = -1  # End-to-end diarization session length in seconds
+    batch_size: int = 1
+    num_workers: int = 0
+    random_seed: Optional[int] = None  # seed number going to be used in seed_everything()
+    bypass_postprocessing: bool = True  # If True, postprocessing will be bypassed
+    log: bool = False  # If True, log will be printed
+
+    use_lhotse: bool = True
+    batch_duration: int = 100000
+
+    # Eval Settings: (0.25, False) should be default setting for sortformer eval.
+    collar: float = 0.25  # Collar in seconds for DER calculation
+    ignore_overlap: bool = False  # If True, DER will be calculated only for non-overlapping segments
+
+    # Streaming diarization configs
+    async_streaming: bool = False
+    spkcache_len: int = 188
+    spkcache_update_period: int = 144
+    fifo_len: int = 188
+    chunk_len: int = 6
+    chunk_left_context: int = 1
+    chunk_right_context: int = 7
+
+    # If `cuda` is a negative number, inference will be on CPU only.
+    cuda: Optional[int] = None
+    matmul_precision: str = "highest"  # Literal["highest", "high", "medium"]
+
+    # Optuna Config
+    launch_pp_optim: bool = False  # If True, launch optimization process for postprocessing parameters
+    optuna_study_name: str = "optim_postprocessing"
+    optuna_temp_dir: str = "/tmp/optuna"
+    optuna_storage: str = f"sqlite:///{optuna_study_name}.db"
+    optuna_log_file: str = f"{optuna_study_name}.log"
+    optuna_n_trials: int = 100000
+
+
+def optuna_suggest_params(postprocessing_cfg: PostProcessingParams, trial) -> PostProcessingParams:
+    """
+    Suggests hyperparameters for postprocessing using Optuna.
+    See the following link for `trial` instance in Optuna framework.
+    https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial
+
+    Args:
+        postprocessing_cfg (PostProcessingParams): The current postprocessing configuration.
+        trial (optuna.Trial): The Optuna trial object used to suggest hyperparameters.
+
+    Returns:
+        PostProcessingParams: The updated postprocessing configuration with suggested hyperparameters.
+    """
+    postprocessing_cfg.onset = trial.suggest_float("onset", 0.4, 0.8, step=0.01)
+    postprocessing_cfg.offset = trial.suggest_float("offset", 0.4, 0.9, step=0.01)
+    postprocessing_cfg.pad_onset = trial.suggest_float("pad_onset", 0.1, 0.5, step=0.01)
+    postprocessing_cfg.pad_offset = trial.suggest_float("pad_offset", 0.0, 0.2, step=0.01)
+    postprocessing_cfg.min_duration_on = trial.suggest_float("min_duration_on", 0.0, 0.75, step=0.01)
+    postprocessing_cfg.min_duration_off = trial.suggest_float("min_duration_off", 0.0, 0.75, step=0.01)
+    return postprocessing_cfg
+
+
+def get_tensor_path(cfg: DiarizationConfig) -> str:
+    """
+    Constructs the file path for saving or loading prediction tensors based on the configuration.
+
+    Args:
+        cfg (DiarizationConfig): The configuration object containing model and dataset details.
+
+    Returns:
+        str: The constructed file path for the prediction tensor.
+    """
+    tensor_filename = os.path.basename(cfg.dataset_manifest).replace("manifest.", "").replace(".json", "")
+    model_base_path = os.path.dirname(cfg.model_path)
+    model_id = os.path.basename(cfg.model_path).replace(".ckpt", "").replace(".nemo", "")
+    bpath = f"{model_base_path}/pred_tensors"
+    if not os.path.exists(bpath):
+        os.makedirs(bpath)
+    tensor_path = f"{bpath}/__{model_id}__{tensor_filename}.pt"
+    return tensor_path, model_id, tensor_filename
+
+
+def diarization_objective(
+    trial,
+    postprocessing_cfg: PostProcessingParams,
+    temp_out_dir: str,
+    infer_audio_rttm_dict: Dict[str, Dict[str, str]],
+    diar_model_preds_total_list: List[torch.Tensor],
+    collar: float = 0.25,
+    ignore_overlap: bool = False,
+) -> float:
+    """
+    Objective function for Optuna hyperparameter optimization in speaker diarization.
+
+    This function evaluates the diarization performance using a set of postprocessing parameters
+    suggested by Optuna. It converts prediction matrices to time-stamp segments, scores the
+    diarization results, and returns the Diarization Error Rate (DER) as the optimization metric.
+
+    Args:
+        trial (optuna.Trial): The Optuna trial object used to suggest hyperparameters.
+        postprocessing_cfg (PostProcessingParams): The current postprocessing configuration.
+        temp_out_dir (str): Temporary directory for storing intermediate outputs.
+        infer_audio_rttm_dict (Dict[str, Dict[str, str]]): Dictionary containing audio file paths,
+            offsets, durations, and RTTM file paths.
+        diar_model_preds_total_list (List[torch.Tensor]): List of prediction matrices containing
+            sigmoid values for each speaker.
+            Dimension: [(1, num_frames, num_speakers), ..., (1, num_frames, num_speakers)]
+        collar (float, optional): Collar in seconds for DER calculation. Defaults to 0.25.
+        ignore_overlap (bool, optional): If True, DER will be calculated only for non-overlapping segments.
+            Defaults to False.
+
+    Returns:
+        float: The Diarization Error Rate (DER) for the given set of postprocessing parameters.
+    """
+    with tempfile.TemporaryDirectory(dir=temp_out_dir, prefix="Diar_PostProcessing_") as _:
+        if trial is not None:
+            postprocessing_cfg = optuna_suggest_params(postprocessing_cfg, trial)
+        all_hyps, all_refs, all_uems = convert_pred_mat_to_segments(
+            audio_rttm_map_dict=infer_audio_rttm_dict,
+            postprocessing_cfg=postprocessing_cfg,
+            batch_preds_list=diar_model_preds_total_list,
+            unit_10ms_frame_count=8,
+            bypass_postprocessing=False,
+        )
+        metric, _, _ = score_labels(
+            AUDIO_RTTM_MAP=infer_audio_rttm_dict,
+            all_reference=all_refs,
+            all_hypothesis=all_hyps,
+            all_uem=all_uems,
+            collar=collar,
+            ignore_overlap=ignore_overlap,
+        )
+        der = abs(metric)
+    return der
+
+
+def run_optuna_hyperparam_search(
+    cfg: DiarizationConfig,  # type: DiarizationConfig
+    postprocessing_cfg: PostProcessingParams,
+    infer_audio_rttm_dict: Dict[str, Dict[str, str]],
+    preds_list: List[torch.Tensor],
+    temp_out_dir: str,
+):
+    """
+    Run Optuna hyperparameter optimization for speaker diarization.
+
+    Args:
+        cfg (DiarizationConfig): The configuration object containing model and dataset details.
+        postprocessing_cfg (PostProcessingParams): The current postprocessing configuration.
+        infer_audio_rttm_dict (dict): dictionary of audio file path, offset, duration and RTTM filepath.
+        preds_list (List[torch.Tensor]): list of prediction matrices containing sigmoid values for each speaker.
+            Dimension: [(1, num_frames, num_speakers), ..., (1, num_frames, num_speakers)]
+        temp_out_dir (str): temporary directory for storing intermediate outputs.
+    """
+    optuna = import_optional_dependency("optuna")
+
+    worker_function = lambda trial: diarization_objective(
+        trial=trial,
+        postprocessing_cfg=postprocessing_cfg,
+        temp_out_dir=temp_out_dir,
+        infer_audio_rttm_dict=infer_audio_rttm_dict,
+        diar_model_preds_total_list=preds_list,
+        collar=cfg.collar,
+    )
+    study = optuna.create_study(
+        direction="minimize", study_name=cfg.optuna_study_name, storage=cfg.optuna_storage, load_if_exists=True
+    )
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)  # Setup the root logger.
+    if cfg.optuna_log_file is not None:
+        logger.addHandler(logging.FileHandler(cfg.optuna_log_file, mode="a"))
+    logger.addHandler(logging.StreamHandler())
+    optuna.logging.enable_propagation()  # Propagate logs to the root logger.
+    study.optimize(worker_function, n_trials=cfg.optuna_n_trials)
+
+
+def convert_pred_mat_to_segments(
+    audio_rttm_map_dict: Dict[str, Dict[str, str]],
+    postprocessing_cfg,
+    batch_preds_list: List[torch.Tensor],
+    unit_10ms_frame_count: int = 8,
+    bypass_postprocessing: bool = False,
+    out_rttm_dir: str | None = None,
+):
+    """
+    Convert prediction matrix to time-stamp segments.
+
+    Args:
+        audio_rttm_map_dict (dict): dictionary of audio file path, offset, duration and RTTM filepath.
+        batch_preds_list (List[torch.Tensor]): list of prediction matrices containing sigmoid values for each speaker.
+            Dimension: [(1, num_frames, num_speakers), ..., (1, num_frames, num_speakers)]
+        unit_10ms_frame_count (int, optional): number of 10ms segments in a frame. Defaults to 8.
+        bypass_postprocessing (bool, optional): if True, postprocessing will be bypassed. Defaults to False.
+
+    Returns:
+       all_hypothesis (list): list of (uniq_id, list[SupervisionSegment]) per audio file.
+       all_reference (list): list of (uniq_id, list[SupervisionSegment]) per audio file.
+       all_uems (list): list of (uniq_id, list[SupervisionSegment]) per audio file.
+    """
+    all_hypothesis, all_reference, all_uems = [], [], []
+    cfg_vad_params = OmegaConf.structured(postprocessing_cfg)
+    total_speaker_timestamps = predlist_to_timestamps(
+        batch_preds_list=batch_preds_list,
+        audio_rttm_map_dict=audio_rttm_map_dict,
+        cfg_vad_params=cfg_vad_params,
+        unit_10ms_frame_count=unit_10ms_frame_count,
+        bypass_postprocessing=bypass_postprocessing,
+    )
+    for sample_idx, (uniq_id, audio_rttm_values) in enumerate(audio_rttm_map_dict.items()):
+        speaker_timestamps = total_speaker_timestamps[sample_idx]
+        if uniq_id is None:
+            if audio_rttm_values.get("uniq_id", None) is not None:
+                uniq_id = audio_rttm_values["uniq_id"]
+            else:
+                uniq_id = get_uniqname_from_filepath(audio_rttm_values["audio_filepath"])
+        all_hypothesis, all_reference, all_uems = timestamps_to_supervisions(
+            speaker_timestamps,
+            uniq_id,
+            audio_rttm_values,
+            all_hypothesis,
+            all_reference,
+            all_uems,
+            out_rttm_dir,
+        )
+    return all_hypothesis, all_reference, all_uems
+
+
+@hydra_runner(config_name="DiarizationConfig", schema=DiarizationConfig)
+def main(cfg: DiarizationConfig) -> Union[DiarizationConfig]:
+    """Main function for end-to-end speaker diarization inference."""
+    for key in cfg:
+        cfg[key] = None if cfg[key] == 'None' else cfg[key]
+
+    if is_dataclass(cfg):
+        cfg = OmegaConf.structured(cfg)
+
+    if cfg.random_seed:
+        pl.seed_everything(cfg.random_seed)
+
+    if cfg.model_path is None:
+        raise ValueError("cfg.model_path cannot be None. Please specify the path to the model.")
+
+    # setup GPU
+    torch.set_float32_matmul_precision(cfg.matmul_precision)
+    if cfg.cuda is None:
+        if torch.cuda.is_available():
+            device = [0]  # use 0th CUDA device
+            accelerator = 'gpu'
+            map_location = torch.device('cuda:0')
+        else:
+            device = 1
+            accelerator = 'cpu'
+            map_location = torch.device('cpu')
+    else:
+        device = [cfg.cuda]
+        accelerator = 'gpu'
+        map_location = torch.device(f'cuda:{cfg.cuda}')
+
+    if cfg.model_path.endswith(".ckpt"):
+        diar_model = SortformerEncLabelModel.load_from_checkpoint(
+            checkpoint_path=cfg.model_path, map_location=map_location, strict=False
+        )
+    elif cfg.model_path.endswith(".nemo"):
+        diar_model = SortformerEncLabelModel.restore_from(restore_path=cfg.model_path, map_location=map_location)
+    else:
+        raise ValueError("cfg.model_path must end with.ckpt or.nemo!")
+
+    diar_model._cfg.test_ds.session_len_sec = cfg.session_len_sec
+    trainer = pl.Trainer(devices=device, accelerator=accelerator, precision=cfg.precision)
+    diar_model.set_trainer(trainer)
+
+    if torch.cuda.is_bf16_supported() and cfg.precision.startswith("bf16"):
+        diar_model = diar_model.to(dtype=torch.bfloat16).eval()
+    else:
+        diar_model = diar_model.eval()
+
+    if cfg.presort_manifest:
+        audio_key = cfg.get('audio_key', 'audio_filepath')
+        with NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
+            for item in read_and_maybe_sort_manifest(cfg.dataset_manifest, try_sort=cfg.presort_manifest):
+                audio_file = get_full_path(audio_file=item[audio_key], manifest_file=cfg.dataset_manifest)
+                item[audio_key] = audio_file
+                f.write(json.dumps(item) + "\n")
+            sorted_manifest_path = f.name
+        diar_model._cfg.test_ds.manifest_filepath = sorted_manifest_path
+        infer_audio_rttm_dict = audio_rttm_map(sorted_manifest_path)
+    else:
+        diar_model._cfg.test_ds.manifest_filepath = cfg.dataset_manifest
+        infer_audio_rttm_dict = audio_rttm_map(cfg.dataset_manifest)
+    remove_path_after_done = sorted_manifest_path if sorted_manifest_path is not None else None
+
+    diar_model._cfg.test_ds.batch_size = cfg.batch_size
+    diar_model._cfg.test_ds.pin_memory = False
+
+    OmegaConf.set_struct(diar_model._cfg, False)
+    diar_model._cfg.test_ds.use_lhotse = cfg.use_lhotse
+    diar_model._cfg.test_ds.use_bucketing = False
+    diar_model._cfg.test_ds.drop_last = False
+    diar_model._cfg.test_ds.batch_duration = cfg.batch_duration
+    OmegaConf.set_struct(diar_model._cfg, True)
+
+    # Model setup for inference
+    diar_model._cfg.test_ds.num_workers = cfg.num_workers
+    diar_model.setup_test_data(test_data_config=diar_model._cfg.test_ds)
+
+    # Streaming mode setup (only if enabled)
+    if diar_model.streaming_mode:
+        diar_model.async_streaming = cfg.async_streaming
+        diar_model.sortformer_modules.chunk_len = cfg.chunk_len
+        diar_model.sortformer_modules.spkcache_len = cfg.spkcache_len
+        diar_model.sortformer_modules.chunk_left_context = cfg.chunk_left_context
+        diar_model.sortformer_modules.chunk_right_context = cfg.chunk_right_context
+        diar_model.sortformer_modules.fifo_len = cfg.fifo_len
+        diar_model.sortformer_modules.log = cfg.log
+        diar_model.sortformer_modules.spkcache_update_period = cfg.spkcache_update_period
+        diar_model.sortformer_modules._check_streaming_parameters()
+
+    postprocessing_cfg = load_postprocessing_from_yaml(cfg.postprocessing_yaml)
+    tensor_path, model_id, tensor_filename = get_tensor_path(cfg)
+    cfg.optuna_study_name = f"__{model_id}_{tensor_filename}"
+    cfg.optuna_storage: str = f"sqlite:///{cfg.optuna_temp_dir}/{cfg.optuna_study_name}.db"
+    cfg.optuna_log_file: str = f"{cfg.optuna_temp_dir}/{cfg.optuna_study_name}.log"
+
+    if os.path.exists(tensor_path) and cfg.save_preds_tensors:
+        logging.info(
+            f"A saved prediction tensor has been found. Loading the saved prediction tensors from {tensor_path}..."
+        )
+        diar_model_preds_total_list = torch.load(tensor_path)
+    else:
+        logging.info("No saved prediction tensors found. Running inference on the dataset...")
+        with torch.inference_mode(), torch.autocast(device_type=diar_model.device.type, dtype=diar_model.dtype):
+            diar_model.test_batch()
+
+        diar_model_preds_total_list = diar_model.preds_total_list
+        if cfg.save_preds_tensors:
+            torch.save(diar_model.preds_total_list, tensor_path)
+
+    if cfg.launch_pp_optim:
+        # Launch a hyperparameter optimization process if launch_pp_optim is True
+        run_optuna_hyperparam_search(
+            cfg=cfg,
+            postprocessing_cfg=postprocessing_cfg,
+            infer_audio_rttm_dict=infer_audio_rttm_dict,
+            preds_list=diar_model_preds_total_list,
+            temp_out_dir=cfg.optuna_temp_dir,
+        )
+
+    # Evaluation
+    if not cfg.no_der:
+        if cfg.out_rttm_dir is not None and not os.path.exists(cfg.out_rttm_dir):
+            os.mkdir(cfg.out_rttm_dir)
+
+        logging.info("Running offline diarization evaluation...")
+        all_hyps, all_refs, all_uems = convert_pred_mat_to_segments(
+            infer_audio_rttm_dict,
+            postprocessing_cfg=postprocessing_cfg,
+            batch_preds_list=diar_model_preds_total_list,
+            unit_10ms_frame_count=8,
+            bypass_postprocessing=cfg.bypass_postprocessing,
+            out_rttm_dir=cfg.out_rttm_dir,
+        )
+        logging.info(f"Evaluating the model on the {len(diar_model_preds_total_list)} audio segments...")
+        score_labels(
+            AUDIO_RTTM_MAP=infer_audio_rttm_dict,
+            all_reference=all_refs,
+            all_hypothesis=all_hyps,
+            all_uem=all_uems,
+            collar=cfg.collar,
+            ignore_overlap=cfg.ignore_overlap,
+        )
+        logging.info(f"PostProcessingParams: {postprocessing_cfg}")
+
+    # clean-up
+    if cfg.presort_manifest is not None:
+        if remove_path_after_done is not None:
+            os.unlink(remove_path_after_done)
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import lightning.pytorch as pl
+from lightning.pytorch import seed_everything
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.models import SortformerEncLabelModel
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+from nemo.utils.exp_manager import exp_manager
+
+"""
+Example training session (single node training)
+For training, you can use the following precisions: 32, bf16 and bf16-mixed.
+You can train with a larger batch size using BF16 mixed precision.
+
+python ./sortformer_diar_train.py --config-path='../conf/neural_diarizer' \
+    --config-name='sortformer_diarizer_hybrid_loss_4spk-v1.yaml' \
+    trainer.precision='bf16' \
+    trainer.devices=1 \
+    model.train_ds.manifest_filepath="<train_manifest_path>" \
+    model.validation_ds.manifest_filepath="<dev_manifest_path>" \
+    exp_manager.name='sample_train' \
+    exp_manager.exp_dir='./sortformer_diar_train'
+"""
+
+seed_everything(42)
+
+
+@hydra_runner(config_path="../conf/neural_diarizer", config_name="sortformer_diarizer_hybrid_loss_4spk-v1.yaml")
+def main(cfg):
+    """Main function for training the sortformer diarizer model."""
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+    trainer = pl.Trainer(**cfg.trainer)
+    exp_manager(trainer, cfg.get("exp_manager", None))
+    sortformer_model = SortformerEncLabelModel(cfg=cfg.model, trainer=trainer)
+    sortformer_model.maybe_init_from_pretrained_checkpoint(cfg)
+    trainer.fit(sortformer_model)
+
+    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
+        if sortformer_model.prepare_test(trainer):
+            trainer.test(sortformer_model)
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import lightning.pytorch as pl
+from lightning.pytorch import seed_everything
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.models import SortformerEncLabelModel
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+from nemo.utils.exp_manager import exp_manager
+
+"""
+Example training session (single node training)
+For training, you can use the following precisions: 32, bf16 and bf16-mixed.
+You can train with a larger batch size using BF16 mixed precision.
+
+python ./streaming_sortformer_diar_train.py --config-path='../conf/neural_diarizer' \
+    --config-name='streaming_sortformer_diarizer_4spk-v2.yaml' \
+    trainer.precision='bf16' \
+    trainer.devices=1 \
+    model.train_ds.manifest_filepath="<train_manifest_path>" \
+    model.validation_ds.manifest_filepath="<dev_manifest_path>" \
+    exp_manager.name='sample_train' \
+    exp_manager.exp_dir='./streaming_sortformer_diar_train'
+"""
+
+seed_everything(42)
+
+
+@hydra_runner(config_path="../conf/neural_diarizer", config_name="streaming_sortformer_diarizer_4spk-v2.yaml")
+def main(cfg):
+    """Main function for training the sortformer diarizer model."""
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+    trainer = pl.Trainer(**cfg.trainer)
+    exp_manager(trainer, cfg.get("exp_manager", None))
+    sortformer_model = SortformerEncLabelModel(cfg=cfg.model, trainer=trainer)
+    sortformer_model.maybe_init_from_pretrained_checkpoint(cfg)
+    trainer.fit(sortformer_model)
+
+    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
+        if sortformer_model.prepare_test(trainer):
+            trainer.test(sortformer_model)
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,105 @@
+# Speaker Recognition
+
+Documentation section for speaker related tasks can be found at:
+ - [Speaker Identification and Verification](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_recognition/intro.html)
+ - [Speaker Diarization](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_diarization/intro.html)
+
+## Performance
+|              MODEL             |          type         | EER (%)<br>Voxceleb-O (veri_test2.txt) |
+|:------------------------------:|:---------------------:|:--------------------------------------:|
+| [speakerverification_speakernet](https://ngc.nvidia.com/catalog/models/nvidia:nemo:speakerverification_speakernet) |        xvector        |                  1.96                  |
+|           [ecapa_tdnn](https://ngc.nvidia.com/catalog/models/nvidia:nemo:ecapa_tdnn)           | channel-<br>attention |                  0.92                  |
+|           [titanet_large](https://ngc.nvidia.com/catalog/models/nvidia:nemo:ecapa_tdnn)           | channel-<br>attention |                  0.66                  |
+
+## Training
+Speaker Recognition models can be trained in a similar way as other models in NeMo using train and dev manifest files. Steps on how to create manifest files for voxceleb are provided below.
+We provide three model configurations based on TitaNet, SpeakerNet and modified ECAPA_TDNN, with pretrained models provided for each of them.
+
+For training titanet_large (channel-attention) model:
+```bash
+python speaker_reco.py --config_path='conf' --config_name='titanet_large.yaml'
+```
+
+For training speakernet (x-vector) model:
+```bash
+python speaker_reco.py --config_path='conf' --config_name='SpeakerNet_verification_3x2x256.yaml'
+```
+
+For training ecapa_tdnn (channel-attention) model:
+```bash
+python speaker_reco.py --config_path='conf' --config_name='ecapa_tdnn.yaml'
+```
+For step by step tutorial see [notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb).
+
+### Fine Tuning
+For fine tuning on a pretrained .nemo speaker recognition model,
+```bash
+python speaker_reco_finetune.py --config_path='conf' --config_name='titanet-finetune.yaml'
+```
+for fine tuning tips see this [tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb)
+
+## Inference
+We provide generic scripts for manifest file creation, embedding extraction, Voxceleb evaluation and speaker ID inference. Hence most of the steps would be common and differ slightly based on your end application.
+
+We explain here the process for voxceleb EER calculation on voxceleb-O cleaned [trail file](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/meta/veri_test2.txt)
+
+### Manifest Creation
+We first generate manifest file to get embeddings. The embeddings are then used by `voxceleb_eval.py` script to get EER
+
+```bash
+# create list of files from voxceleb1 test folder (40 speaker test set)
+find <path/to/voxceleb1_test/directory/> -iname '*.wav' > voxceleb1_test_files.txt
+python <NeMo_root>/scripts/speaker_tasks/filelist_to_manifest.py --filelist voxceleb1_test_files.txt --id -3 --out voxceleb1_test_manifest.json
+```
+### Embedding Extraction
+Now using the manifest file created, we can extract embeddings to `data` folder using:
+```bash
+python extract_speaker_embeddings.py --manifest=voxceleb1_test_manifest.json --model_path='titanet_large' --embedding_dir='./'
+```
+If you have a single file, you may also be using the following one liner to get embeddings for the audio file:
+
+```python
+speaker_model = EncDecSpeakerLabelModel.from_pretrained(model_name="titanet_large")
+embs = speaker_model.get_embedding('audio_path')
+```
+
+### Voxceleb Evaluation
+``` bash
+python voxceleb_eval.py --trial_file='/path/to/trail/file' --emb='./embeddings/voxceleb1_test_manifest_embeddings.pt'
+```
+The above command gives the performance of models on voxceleb-o cleaned trial file.
+
+### SpeakerID inference
+Using data from an enrollment set, one can infer labels on a test set using various backends such as cosine-similarity or a neural classifier.
+
+To infer speaker labels using cosine_similarity backend
+```bash
+python speaker_identification_infer.py data.enrollment_manifest=<path/to/enrollment_manifest> data.test_manifest=<path/to/test_manifest> backend.backend_model=cosine_similarity
+```
+refer to conf/speaker_identification_infer.yaml for more options.
+
+## Voxceleb Data Preparation
+
+Scripts we provide for data preparation are very generic and can be applied to any dataset with a few path changes.
+For VoxCeleb datasets, we first download the datasets individually and make a list of audio files. Then we use the script to generate manifest files for training and validation.
+Download [voxceleb1](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html) and [voxceleb2](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html) data.
+
+Once downloaded and uncompressed, use programs such as ffmpeg to convert audio files from m4a format to wav format.
+Refer to the following sample command
+```bash
+ffmpeg -v 8 -i </path/to/m4a/file> -f wav -acodec pcm_s16le <path/to/wav/file>
+```
+
+Generate a list file that contains paths to all the dev audio files from voxceleb1 and voxceleb2 using find command as shown below:
+```bash
+find <path/to/voxceleb1/dev/folder/> -iname '*.wav' > voxceleb1_dev.txt
+find <path/to/voxceleb2/dev/folder/> -iname '*.wav' > voxceleb2_dev.txt
+cat voxceleb1_dev.txt voxceleb2_dev.txt > voxceleb12.txt
+```
+
+This list file is now used to generate training and validation manifest files using a script provided in `<NeMo_root>/scripts/speaker_tasks/`. This script has optional arguments to split the whole manifest file in to train and dev and also chunk audio files to smaller segments for robust training (for testing, we don't need this).
+
+```bash
+python <NeMo_root>/scripts/speaker_tasks/filelist_to_manifest.py --filelist voxceleb12.txt --id -3 --out voxceleb12_manifest.json --split --create_segments
+```
+This creates `train.json, dev.json` in the current working directory.
@@ -0,0 +1,154 @@
+name: &name "SpeakerNet"
+sample_rate: &sample_rate 16000
+repeat: &rep 2
+dropout: &drop 0.5
+separable: &separable True
+n_filters: &n_filters 512
+
+model:
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 64
+    shuffle: True
+    is_tarred: False
+    tarred_audio_filepaths: null
+    tarred_shard_strategy: "scatter"
+
+  validation_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 128
+    shuffle: False
+
+  test_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 1
+    shuffle: False
+    embedding_dir: '.'
+
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.02
+    sample_rate: *sample_rate
+    window_stride: 0.01
+    window: "hann"
+    features: &n_mels 64
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConvASREncoder
+    feat_in: *n_mels
+    activation: relu
+    conv_mask: true
+
+    jasper:
+      -   filters: *n_filters
+          repeat: 1
+          kernel: [3]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: *n_filters
+          repeat: *rep
+          kernel: [7]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: *n_filters
+          repeat: *rep
+          kernel: [11]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: *n_filters
+          repeat: *rep
+          kernel: [15]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: &enc_feat_out 1500
+          repeat: 1
+          kernel: [1]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: *separable
+
+  decoder:
+    _target_: nemo.collections.asr.modules.SpeakerDecoder
+    feat_in: *enc_feat_out
+    num_classes: 7205
+    pool_mode: 'xvector'
+    emb_sizes: [512,512]
+
+  loss:
+    _target_: from nemo.collections.common.losses.CrossEntropyLoss # you could also use AngularSoftmaxLoss
+
+  optim:
+    name: novograd
+    # _target_: nemo.core.optim.optimizers.Novograd
+    lr: .008
+    # optimizer arguments
+    args:
+      name: auto
+      # _target_: nemo.core.config.optimizers.NovogradParams
+      betas: [0.95, 0.5]
+      weight_decay: 0.001
+
+    # scheduler setup
+    sched:
+      name: CosineAnnealing
+      iters_per_batch: 1 # computed at runtime
+      max_steps: -1 # computed at runtime or explicitly set here
+
+      # scheduler config override
+      args:
+        name: auto
+        # _target_: nemo.core.config.schedulers.CosineAnnealingParams
+        warmup_steps: null
+        warmup_ratio: 0.1
+        min_lr: 0.0
+        last_epoch: -1
+
+trainer:
+  devices: 1 # number of gpus
+  max_epochs: 200
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  accelerator: gpu
+  strategy: ddp
+  accumulate_grad_batches: 1
+  deterministic: True
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+  gradient_clip_val: 1.0
+  
+    
+exp_manager:
+  exp_dir: null
+  name: *name
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
@@ -0,0 +1,133 @@
+name: &name "SpeakerNet"
+sample_rate: &sample_rate 16000
+repeat: &rep 2
+dropout: &drop 0.5
+separable: &separable True
+n_filters: &n_filters 256
+
+model:
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 64
+    shuffle: True
+    is_tarred: False
+    tarred_audio_filepaths: null
+    tarred_shard_strategy: "scatter"
+
+  validation_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 128
+    shuffle: False
+
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.02
+    sample_rate: *sample_rate
+    window_stride: 0.01
+    window: "hann"
+    features: &n_mels 64
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConvASREncoder
+    feat_in: *n_mels
+    activation: relu
+    conv_mask: true
+
+    jasper:
+      -   filters: *n_filters
+          repeat: 1
+          kernel: [3]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: *n_filters
+          repeat: *rep
+          kernel: [7]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: *n_filters
+          repeat: *rep
+          kernel: [11]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: *n_filters
+          repeat: *rep
+          kernel: [15]
+          stride: [1]
+          dilation: [1]
+          dropout: *drop
+          residual: true
+          separable: *separable
+
+      -   filters: &enc_feat_out 1500
+          repeat: 1
+          kernel: [1]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: *separable
+
+  decoder:
+    _target_: nemo.collections.asr.modules.SpeakerDecoder
+    feat_in: *enc_feat_out
+    num_classes: 7205
+    pool_mode: 'xvector'
+    emb_sizes: 256
+
+  loss:
+    _target_: nemo.collections.asr.losses.angularloss.AngularSoftmaxLoss # you could also use cross-entrophy loss
+    scale: 30
+    margin: 0.2
+
+  optim:
+    name: sgd
+    lr: .006
+    weight_decay: 0.001
+    momentum: 0.9
+
+    # scheduler setup
+    sched:
+      name: CosineAnnealing
+      warmup_ratio: 0.1
+      min_lr: 0.0
+
+trainer:
+  devices: 1 # number of gpus
+  max_epochs: 200
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  accelerator: gpu
+  strategy: ddp
+  accumulate_grad_batches: 1
+  deterministic: True
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+  gradient_clip_val: 1.0
+
+exp_manager:
+  exp_dir: null
+  name: *name
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
@@ -0,0 +1,106 @@
+name: "ECAPA_TDNN"
+
+model:
+
+  sample_rate: 16000
+
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: ${model.sample_rate}
+    labels: null
+    batch_size: 64
+    shuffle: True
+    augmentor:
+      noise:
+        manifest_path: null
+        prob: 0.5
+        min_snr_db: 0
+        max_snr_db: 15
+
+      speed:
+        prob: 0.5
+        sr: ${model.sample_rate}
+        resample_type: 'kaiser_fast'
+        min_speed_rate: 0.95
+        max_speed_rate: 1.05
+
+  validation_ds:
+    manifest_filepath: ???
+    sample_rate: ${model.sample_rate}
+    labels: null
+    batch_size: 128
+    shuffle: False
+
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.025
+    sample_rate: ${model.sample_rate}
+    window_stride: 0.01
+    window: "hann"
+    features: 80
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+    stft_conv: false
+
+  spec_augment:
+    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
+    freq_masks: 3
+    freq_width: 4
+    time_masks: 5
+    time_width: 0.03
+
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ECAPAEncoder
+    feat_in: ${model.preprocessor.features}
+    filters: [1024,1024,1024,1024,3072]
+    kernel_sizes: [5,3,3,3,1]
+    dilations: [1,1,1,1,1]
+    scale: 8
+
+
+  decoder:
+    _target_: nemo.collections.asr.modules.SpeakerDecoder
+    feat_in: 3072
+    num_classes: 7205
+    pool_mode: 'attention' #xvector,tap or attention
+    emb_sizes: 192
+
+  loss:
+    _target_: nemo.collections.asr.losses.angularloss.AngularSoftmaxLoss # you could also use cross-entrophy loss
+    scale: 30
+    margin: 0.2
+
+  optim:
+    name: sgd
+    lr: 0.08
+    weight_decay: 0.0002
+
+    # scheduler setup
+    sched:
+      name: CosineAnnealing
+      warmup_ratio: 0.1
+      min_lr: 0.0001
+
+trainer:
+  devices: 1 # number of gpus (trained on four nodes - each node has 8 gpus)
+  max_epochs: 250
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  accelerator: gpu
+  strategy: ddp
+  accumulate_grad_batches: 1
+  deterministic: False
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+  gradient_clip_val: 1.0
+
+exp_manager:
+  exp_dir: null
+  name: ${name}
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
@@ -0,0 +1,27 @@
+name: &name "SpeakerIdentificationInfer"
+
+data:
+    enrollment_manifest: ???
+    test_manifest: ???
+    out_manifest: './infer_output.json'
+    sample_rate: 16000
+
+backend:
+    backend_model: cosine_similarity # supported backends are cosine_similarity and neural_classifier
+
+    cosine_similarity:
+        model_path: titanet_large # or path to .nemo file
+        batch_size: 32
+
+    neural_classifier:
+        model_path: ??? # path to neural model trained/finetuned with enrollment dataset
+        batch_size: 32
+
+# json manifest line example
+#
+# enrollment_manifest:
+# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "<speaker_label>"}
+# 
+# test_manifest:
+# {"audio_filepath": "/path/to/audio_file", "offset": 0, "duration": null, "label": "infer"}
+#
@@ -0,0 +1,163 @@
+name: &name "TitaNet-Finetune"
+sample_rate: &sample_rate 16000
+
+init_from_pretrained_model:
+  speaker_tasks:
+    name: 'titanet_large'
+    include: ["preprocessor","encoder"]
+    exclude: ["decoder.final"] # Add specific layer names here to exlude or just ["decoder"] if to exclude all of decoder pretrained weights
+
+model:
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 64
+    shuffle: True
+    is_tarred: False
+    tarred_audio_filepaths: null
+    tarred_shard_strategy: "scatter"
+    augmentor:
+      speed:
+        prob: 0.3
+        sr: *sample_rate
+        resample_type: 'kaiser_fast'
+        min_speed_rate: 0.95
+        max_speed_rate: 1.05
+
+  validation_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 128
+    shuffle: False
+
+  model_defaults:
+    filters: 1024
+    repeat: 3
+    dropout: 0.1
+    separable: true
+    se: true
+    se_context_size: -1
+    kernel_size_factor: 1.0
+    
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.025
+    sample_rate: *sample_rate
+    window_stride: 0.01
+    window: "hann"
+    features: &n_mels 80
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConvASREncoder
+    feat_in: *n_mels
+    activation: relu
+    conv_mask: true
+
+    jasper:
+      -   filters: ${model.model_defaults.filters}
+          repeat: 1
+          kernel: [3]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat:  ${model.model_defaults.repeat}
+          kernel: [7]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat: ${model.model_defaults.repeat}
+          kernel: [11]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat: ${model.model_defaults.repeat}
+          kernel: [15]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: &enc_feat_out 3072
+          repeat: 1
+          kernel: [1]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+  decoder:
+    _target_: nemo.collections.asr.modules.SpeakerDecoder
+    feat_in: *enc_feat_out
+    num_classes: ???
+    pool_mode: 'attention'
+    emb_sizes: 192
+
+  loss:
+    _target_: nemo.collections.asr.losses.angularloss.AngularSoftmaxLoss # you could also use cross-entrophy loss
+    scale: 30
+    margin: 0.2
+
+  optim_param_groups:
+    encoder:
+      lr: .001
+
+  optim:
+    name: adamw
+    lr: .0001 #(original titanet-large was trained with 0.08 lr)
+    weight_decay: 0.0002
+
+    # scheduler setup
+    sched:
+      name: CosineAnnealing
+      warmup_ratio: 0.1
+      min_lr: 0.0
+
+trainer:
+  devices: 1 # number of gpus (original titanet-large was trained on 4 nodes with 8 gpus each)
+  max_epochs: 10
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  accelerator: gpu
+  strategy: ddp
+  deterministic: True
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+  gradient_clip_val: 1.0
+
+exp_manager:
+  exp_dir: null
+  name: *name
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
@@ -0,0 +1,168 @@
+name: &name "TitaNet-L"
+sample_rate: &sample_rate 16000
+
+model:
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 64
+    shuffle: True
+    is_tarred: False
+    tarred_audio_filepaths: null
+    tarred_shard_strategy: "scatter"
+    augmentor:
+      noise:
+        manifest_path: null
+        prob: 0.5
+        min_snr_db: 0
+        max_snr_db: 15
+
+      speed:
+        prob: 0.3
+        sr: *sample_rate
+        resample_type: 'kaiser_fast'
+        min_speed_rate: 0.95
+        max_speed_rate: 1.05
+
+  validation_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 128
+    shuffle: False
+    is_audio_pair: false  # set `true` to calculate EER during validation, see nemo/collections/asr/models/label_models.py for details
+
+  model_defaults:
+    filters: 1024
+    repeat: 3
+    dropout: 0.1
+    separable: true
+    se: true
+    se_context_size: -1
+    kernel_size_factor: 1.0
+    
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.025
+    sample_rate: *sample_rate
+    window_stride: 0.01
+    window: "hann"
+    features: &n_mels 80
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  spec_augment:
+    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
+    freq_masks: 3
+    freq_width: 4
+    time_masks: 5
+    time_width: 0.03
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConvASREncoder
+    feat_in: *n_mels
+    activation: relu
+    conv_mask: true
+
+    jasper:
+      -   filters: ${model.model_defaults.filters}
+          repeat: 1
+          kernel: [3]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat:  ${model.model_defaults.repeat}
+          kernel: [7]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat: ${model.model_defaults.repeat}
+          kernel: [11]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat: ${model.model_defaults.repeat}
+          kernel: [15]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: &enc_feat_out 3072
+          repeat: 1
+          kernel: [1]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+  decoder:
+    _target_: nemo.collections.asr.modules.SpeakerDecoder
+    feat_in: *enc_feat_out
+    num_classes: 7205
+    pool_mode: 'attention'
+    emb_sizes: 192
+
+  loss:
+    _target_: nemo.collections.asr.losses.angularloss.AngularSoftmaxLoss # you could also use cross-entrophy loss
+    scale: 30    
+    margin: 0.2  
+
+  optim:
+    name: sgd
+    lr: .006 #(original titanet-large was trained with 0.08 lr)
+    weight_decay: 0.0002
+    momentum: 0.9
+
+    # scheduler setup
+    sched:
+      name: CosineAnnealing
+      warmup_ratio: 0.1
+      min_lr: 0.0
+
+trainer:
+  devices: 1 # number of gpus (original titanet-large was trained on 4 nodes with 8 gpus each)
+  max_epochs: 250
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  accelerator: gpu
+  strategy: ddp
+  deterministic: True
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+  gradient_clip_val: 1.0
+
+exp_manager:
+  exp_dir: null
+  name: *name
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
@@ -0,0 +1,172 @@
+name: &name "TitaNet-S"
+sample_rate: &sample_rate 16000
+
+model:
+  train_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 64
+    shuffle: True
+    is_tarred: False
+    tarred_audio_filepaths: null
+    tarred_shard_strategy: "scatter"
+    augmentor:
+      noise:
+        manifest_path: null
+        prob: 0.5
+        min_snr_db: 5
+        max_snr_db: 15
+
+      impulse:
+        manifest_path: null
+        prob: 0.5 
+
+      speed:
+        prob: 0.5
+        sr: *sample_rate
+        resample_type: 'kaiser_fast'
+        min_speed_rate: 0.95
+        max_speed_rate: 1.05
+
+  validation_ds:
+    manifest_filepath: ???
+    sample_rate: 16000
+    labels: null
+    batch_size: 128
+    shuffle: False
+
+  model_defaults:
+    filters: 256
+    repeat: 3
+    dropout: 0.1
+    separable: true
+    se: true
+    se_context_size: -1
+    kernel_size_factor: 1.0
+    
+  preprocessor:
+    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    normalize: "per_feature"
+    window_size: 0.025
+    sample_rate: *sample_rate
+    window_stride: 0.01
+    window: "hann"
+    features: &n_mels 80
+    n_fft: 512
+    frame_splicing: 1
+    dither: 0.00001
+
+  spec_augment:
+    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
+    freq_masks: 3
+    freq_width: 4
+    time_masks: 5
+    time_width: 0.03
+
+  encoder:
+    _target_: nemo.collections.asr.modules.ConvASREncoder
+    feat_in: *n_mels
+    activation: relu
+    conv_mask: true
+
+    jasper:
+      -   filters: ${model.model_defaults.filters}
+          repeat: 1
+          kernel: [3]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat:  ${model.model_defaults.repeat}
+          kernel: [7]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat: ${model.model_defaults.repeat}
+          kernel: [11]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: ${model.model_defaults.filters}
+          repeat: ${model.model_defaults.repeat}
+          kernel: [15]
+          stride: [1]
+          dilation: [1]
+          dropout: ${model.model_defaults.dropout}
+          residual: true
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+      -   filters: &enc_feat_out 3072
+          repeat: 1
+          kernel: [1]
+          stride: [1]
+          dilation: [1]
+          dropout: 0.0
+          residual: false
+          separable: ${model.model_defaults.separable}
+          se: ${model.model_defaults.se}
+          se_context_size: ${model.model_defaults.se_context_size}
+
+  decoder:
+    _target_: nemo.collections.asr.modules.SpeakerDecoder
+    feat_in: *enc_feat_out
+    num_classes: 7205
+    pool_mode: 'attention'
+    emb_sizes: 192
+
+  loss:
+    _target_: nemo.collections.asr.losses.angularloss.AngularSoftmaxLoss
+    scale: 30
+    margin: 0.2
+
+  optim:
+    name: sgd
+    lr: .006 #(original titanet-large was trained with 0.08 lr)
+    weight_decay: 0.001
+    momentum: 0.9
+
+    # scheduler setup
+    sched:
+      name: CosineAnnealing
+      warmup_ratio: 0.1
+      min_lr: 0.0
+
+trainer:
+  devices: 1 # number of gpus (original titanet-large was trained on 4 nodes with 8 gpus each)
+  max_epochs: 250
+  max_steps: -1 # computed at runtime if not set
+  num_nodes: 1
+  accelerator: gpu
+  strategy: ddp
+  accumulate_grad_batches: 1
+  deterministic: True
+  enable_checkpointing: False
+  logger: False
+  log_every_n_steps: 1  # Interval of logging.
+  val_check_interval: 1.0  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
+  gradient_clip_val: 1.0
+
+exp_manager:
+  exp_dir: null
+  name: *name
+  create_tensorboard_logger: True
+  create_checkpoint_callback: True
@@ -0,0 +1,131 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This is a helper script to extract speaker embeddings based on manifest file
+Usage:
+python extract_speaker_embeddings.py --manifest=/path/to/manifest/file'
+--model_path='/path/to/.nemo/file'(optional)
+--embedding_dir='/path/to/embedding/directory'
+
+Args:
+--manifest: path to manifest file containing audio_file paths for which embeddings need to be extracted
+--model_path(optional): path to .nemo speaker verification model file to extract embeddings, if not passed SpeakerNet-M model would
+    be downloaded from NGC and used to extract embeddings
+--embeddings_dir(optional): path to directory where embeddings need to stored default:'./'
+
+
+"""
+
+import json
+import os
+from argparse import ArgumentParser
+
+import numpy as np
+import torch
+
+from nemo.collections.asr.models.label_models import EncDecSpeakerLabelModel
+from nemo.collections.asr.parts.utils.speaker_utils import embedding_normalize
+from nemo.utils import logging
+
+
+def get_embeddings(speaker_model, manifest_file, batch_size=1, embedding_dir='./', device='cuda'):
+    """
+    save embeddings to cached file
+    Args:
+        speaker_model: NeMo <EncDecSpeakerLabel> model
+        manifest_file: path to the manifest file containing the audio file path from which the
+                       embeddings should be extracted
+        batch_size: batch_size for inference
+        embedding_dir: path to directory to store embeddings file
+        device: compute device to perform operations
+    """
+
+    all_embs, _, _, _ = speaker_model.batch_inference(manifest_file, batch_size=batch_size, device=device)
+    all_embs = np.asarray(all_embs)
+    all_embs = embedding_normalize(all_embs)
+    out_embeddings = {}
+
+    with open(manifest_file, 'r', encoding='utf-8') as manifest:
+        for i, line in enumerate(manifest.readlines()):
+            line = line.strip()
+            dic = json.loads(line)
+            uniq_name = '@'.join(dic['audio_filepath'].split('/')[-3:])
+            out_embeddings[uniq_name] = all_embs[i]
+
+    embedding_dir = os.path.join(embedding_dir, 'embeddings')
+    if not os.path.exists(embedding_dir):
+        os.makedirs(embedding_dir, exist_ok=True)
+
+    prefix = manifest_file.split('/')[-1].rsplit('.', 1)[-2]
+
+    name = os.path.join(embedding_dir, prefix)
+    embeddings_file = name + '_embeddings.pt'
+    torch.save(out_embeddings, embeddings_file)
+    logging.info("Saved embedding files to {}".format(embedding_dir))
+
+
+def main():
+    parser = ArgumentParser()
+    parser.add_argument(
+        "--manifest",
+        type=str,
+        required=True,
+        help="Path to manifest file",
+    )
+    parser.add_argument(
+        "--model_path",
+        type=str,
+        default='titanet_large',
+        required=False,
+        help="path to .nemo speaker verification model file to extract embeddings, if not passed SpeakerNet-M model would be downloaded from NGC and used to extract embeddings",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=1,
+        required=False,
+        help="batch size",
+    )
+    parser.add_argument(
+        "--embedding_dir",
+        type=str,
+        default='./',
+        required=False,
+        help="path to directory where embeddings need to stored default:'./'",
+    )
+    args = parser.parse_args()
+    torch.set_grad_enabled(False)
+
+    if args.model_path.endswith('.nemo'):
+        logging.info(f"Using local speaker model from {args.model_path}")
+        speaker_model = EncDecSpeakerLabelModel.restore_from(restore_path=args.model_path)
+    elif args.model_path.endswith('.ckpt'):
+        speaker_model = EncDecSpeakerLabelModel.load_from_checkpoint(checkpoint_path=args.model_path)
+    else:
+        speaker_model = EncDecSpeakerLabelModel.from_pretrained(model_name="titanet_large")
+        logging.info(f"using pretrained titanet_large speaker model from NGC")
+
+    device = 'cuda'
+    if not torch.cuda.is_available():
+        device = 'cpu'
+        logging.warning("Running model on CPU, for faster performance it is adviced to use atleast one NVIDIA GPUs")
+
+    get_embeddings(
+        speaker_model, args.manifest, batch_size=args.batch_size, embedding_dir=args.embedding_dir, device=device
+    )
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,123 @@
+# Copyright (c) 2022, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+
+import numpy as np
+import torch
+from lightning.pytorch import seed_everything
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.data.audio_to_label import AudioToSpeechLabelDataset
+from nemo.collections.asr.models import EncDecSpeakerLabelModel
+from nemo.collections.asr.parts.features import WaveformFeaturizer
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+
+seed_everything(42)
+
+
+@hydra_runner(config_path="conf", config_name="speaker_identification_infer")
+def main(cfg):
+
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+
+    device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+    enrollment_manifest = cfg.data.enrollment_manifest
+    test_manifest = cfg.data.test_manifest
+    out_manifest = cfg.data.out_manifest
+    sample_rate = cfg.data.sample_rate
+
+    backend = cfg.backend.backend_model.lower()
+
+    featurizer = WaveformFeaturizer(sample_rate=sample_rate)
+    dataset = AudioToSpeechLabelDataset(manifest_filepath=enrollment_manifest, labels=None, featurizer=featurizer)
+    enroll_id2label = dataset.id2label
+
+    if backend == 'cosine_similarity':
+        model_path = cfg.backend.cosine_similarity.model_path
+        batch_size = cfg.backend.cosine_similarity.batch_size
+        if model_path.endswith('.nemo'):
+            speaker_model = EncDecSpeakerLabelModel.restore_from(model_path)
+        else:
+            speaker_model = EncDecSpeakerLabelModel.from_pretrained(model_path)
+
+        enroll_embs, _, enroll_truelabels, _ = speaker_model.batch_inference(
+            enrollment_manifest,
+            batch_size,
+            sample_rate,
+            device=device,
+        )
+
+        test_embs, _, _, _ = speaker_model.batch_inference(
+            test_manifest,
+            batch_size,
+            sample_rate,
+            device=device,
+        )
+
+        # length normalize
+        enroll_embs = enroll_embs / (np.linalg.norm(enroll_embs, ord=2, axis=-1, keepdims=True))
+        test_embs = test_embs / (np.linalg.norm(test_embs, ord=2, axis=-1, keepdims=True))
+
+        # reference embedding
+        reference_embs = []
+        keyslist = list(enroll_id2label.values())
+        for label_id in keyslist:
+            indices = np.where(enroll_truelabels == label_id)
+            embedding = (enroll_embs[indices].sum(axis=0).squeeze()) / len(indices)
+            reference_embs.append(embedding)
+
+        reference_embs = np.asarray(reference_embs)
+
+        scores = np.matmul(test_embs, reference_embs.T)
+        matched_labels = scores.argmax(axis=-1)
+
+    elif backend == 'neural_classifier':
+        model_path = cfg.backend.neural_classifier.model_path
+        batch_size = cfg.backend.neural_classifier.batch_size
+
+        if model_path.endswith('.nemo'):
+            speaker_model = EncDecSpeakerLabelModel.restore_from(model_path)
+        else:
+            speaker_model = EncDecSpeakerLabelModel.from_pretrained(model_path)
+
+        if speaker_model.decoder.final.out_features != len(enroll_id2label):
+            raise ValueError(
+                "number of labels mis match. Make sure you trained or finetuned neural classifier with labels from enrollement manifest_filepath"
+            )
+
+        _, test_logits, _, _ = speaker_model.batch_inference(
+            test_manifest,
+            batch_size,
+            sample_rate,
+            device=device,
+        )
+        matched_labels = test_logits.argmax(axis=-1)
+
+    with open(test_manifest, 'rb') as f1, open(out_manifest, 'w', encoding='utf-8') as f2:
+        lines = f1.readlines()
+        for idx, line in enumerate(lines):
+            line = line.strip()
+            item = json.loads(line)
+            item['infer'] = enroll_id2label[matched_labels[idx]]
+            json.dump(item, f2)
+            f2.write('\n')
+
+    logging.info("Inference labels have been written to {} manifest file".format(out_manifest))
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,84 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+import lightning.pytorch as pl
+import torch
+from lightning.pytorch import seed_everything
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.models import EncDecSpeakerLabelModel
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+from nemo.utils.exp_manager import exp_manager
+
+"""
+Basic run (on GPU for 10 epochs for 2 class training):
+EXP_NAME=sample_run
+python ./speaker_reco.py --config-path='conf' --config-name='SpeakerNet_recognition_3x2x512.yaml' \
+    trainer.max_epochs=10  \
+    model.train_ds.batch_size=64 model.validation_ds.batch_size=64 \
+    model.train_ds.manifest_filepath="<train_manifest>" model.validation_ds.manifest_filepath="<dev_manifest>" \
+    model.test_ds.manifest_filepath="<test_manifest>" \
+    trainer.devices=1 \
+    model.decoder.params.num_classes=2 \
+    exp_manager.name=$EXP_NAME +exp_manager.use_datetime_version=False \
+    exp_manager.exp_dir='./speaker_exps'
+
+See https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb for notebook tutorial
+
+Optional: Use tarred dataset to speech up data loading.
+   Prepare ONE manifest that contains all training data you would like to include. Validation should use non-tarred dataset.
+   Note that it's possible that tarred datasets impacts validation scores because it drop values in order to have same amount of files per tarfile; 
+   Scores might be off since some data is missing. 
+   
+   Use the `convert_to_tarred_audio_dataset.py` script under <NEMO_ROOT>/speech_recognition/scripts in order to prepare tarred audio dataset.
+   For details, please see TarredAudioToClassificationLabelDataset in <NEMO_ROOT>/nemo/collections/asr/data/audio_to_label.py
+"""
+
+seed_everything(42)
+
+
+@hydra_runner(config_path="conf", config_name="SpeakerNet_verification_3x2x256.yaml")
+def main(cfg):
+
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+    trainer = pl.Trainer(**cfg.trainer)
+    log_dir = exp_manager(trainer, cfg.get("exp_manager", None))
+    speaker_model = EncDecSpeakerLabelModel(cfg=cfg.model, trainer=trainer)
+
+    # save labels to file
+    if log_dir is not None:
+        with open(os.path.join(log_dir, 'labels.txt'), 'w') as f:
+            if speaker_model.labels is not None:
+                for label in speaker_model.labels:
+                    f.write(f'{label}\n')
+
+    trainer.fit(speaker_model)
+
+    if not trainer.fast_dev_run:
+        model_path = os.path.join(log_dir, '..', 'spkr.nemo')
+        speaker_model.save_to(model_path)
+
+    torch.distributed.destroy_process_group()
+    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
+        if trainer.is_global_zero:
+            trainer = pl.Trainer(devices=1, accelerator=cfg.trainer.accelerator, strategy=cfg.trainer.strategy)
+            if speaker_model.prepare_test(trainer):
+                trainer.test(speaker_model)
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,57 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+import lightning.pytorch as pl
+import torch
+from lightning.pytorch import seed_everything
+from omegaconf import OmegaConf
+
+from nemo.collections.asr.models import EncDecSpeakerLabelModel
+from nemo.core.config import hydra_runner
+from nemo.utils import logging
+from nemo.utils.exp_manager import exp_manager
+
+seed_everything(42)
+
+
+@hydra_runner(config_path="conf", config_name="titanet-finetune.yaml")
+def main(cfg):
+
+    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')
+    trainer = pl.Trainer(**cfg.trainer)
+    log_dir = exp_manager(trainer, cfg.get("exp_manager", None))
+    speaker_model = EncDecSpeakerLabelModel(cfg=cfg.model, trainer=trainer)
+    speaker_model.maybe_init_from_pretrained_checkpoint(cfg)
+
+    # save labels to file
+    if log_dir is not None:
+        with open(os.path.join(log_dir, 'labels.txt'), 'w') as f:
+            if speaker_model.labels is not None:
+                for label in speaker_model.labels:
+                    f.write(f'{label}\n')
+
+    trainer.fit(speaker_model)
+
+    torch.distributed.destroy_process_group()
+    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
+        if trainer.is_global_zero:
+            trainer = pl.Trainer(devices=1, accelerator=cfg.trainer.accelerator, strategy=cfg.trainer.strategy)
+            if speaker_model.prepare_test(trainer):
+                trainer.test(speaker_model)
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,108 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import os
+import sys
+
+import numpy as np
+from scipy.interpolate import interp1d
+from scipy.optimize import brentq
+from sklearn.metrics import roc_curve
+from tqdm import tqdm
+
+"""
+This script faciliates to get EER % based on cosine-smilarity
+for Voxceleb dataset.
+
+Args:
+    trial_file str: path to voxceleb trial file
+    emb : path to cached file of embeddings dictionary (generated from spkr_get_emb.py)
+    save_kaldi_emb: if required pass this argument to save kaldi embeddings for KALDI PLDA training later
+    Note: order of audio files in manifest file should match the embeddings
+"""
+
+
+def get_acc(trial_file='', emb='', save_kaldi_emb=False):
+
+    trial_score = open('trial_score.txt', 'w')
+    dirname = os.path.dirname(trial_file)
+    emb = torch.load(emb)
+    trial_embs = []
+    keys = []
+    all_scores = []
+    all_keys = []
+
+    # for each trials in trial file
+    with open(trial_file, 'r') as f:
+        tmp_file = f.readlines()
+        for line in tqdm(tmp_file):
+            line = line.strip()
+            truth, x_speaker, y_speaker = line.split()
+
+            x_speaker = x_speaker.split('/')
+            x_speaker = '@'.join(x_speaker)
+
+            y_speaker = y_speaker.split('/')
+            y_speaker = '@'.join(y_speaker)
+
+            X = emb[x_speaker]
+            Y = emb[y_speaker]
+
+            if save_kaldi_emb and x_speaker not in keys:
+                keys.append(x_speaker)
+                trial_embs.extend([X])
+
+            if save_kaldi_emb and y_speaker not in keys:
+                keys.append(y_speaker)
+                trial_embs.extend([Y])
+
+            score = np.dot(X, Y) / ((np.dot(X, X) * np.dot(Y, Y)) ** 0.5)
+            score = (score + 1) / 2
+
+            all_scores.append(score)
+            trial_score.write(str(score) + "\t" + truth)
+            truth = int(truth)
+            all_keys.append(truth)
+
+            trial_score.write('\n')
+    trial_score.close()
+
+    if save_kaldi_emb:
+        np.save(dirname + '/all_embs_voxceleb.npy', np.asarray(trial_embs))
+        np.save(dirname + '/all_ids_voxceleb.npy', np.asarray(keys))
+        print("Saved KALDI PLDA related embeddings to {}".format(dirname))
+
+    return np.asarray(all_scores), np.asarray(all_keys)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--trial_file", help="path to voxceleb trial file", type=str, required=True)
+    parser.add_argument("--emb", help="path to numpy file of embeddings", type=str, required=True)
+    parser.add_argument(
+        "--save_kaldi_emb",
+        help=":save kaldi embeddings for KALDI PLDA training later",
+        required=False,
+        action='store_true',
+    )
+
+    args = parser.parse_args()
+    trial_file, emb, save_kaldi_emb = args.trial_file, args.emb, args.save_kaldi_emb
+
+    y_score, y = get_acc(trial_file=trial_file, emb=emb, save_kaldi_emb=save_kaldi_emb)
+    fpr, tpr, thresholds = roc_curve(y, y_score, pos_label=1)
+
+    eer = brentq(lambda x: 1.0 - x - interp1d(fpr, tpr)(x), 0.0, 1.0)
+    sys.stdout.write("{0:.2f}\n".format(eer * 100))