chore: import upstream snapshot with attribution

docs/source/tts/api.rst

@@ -0,0 +1,68 @@
+.. _nemo tts collection api:
+
+NeMo TTS API
+============
+
+Model Classes
+-------------
+
+MagpieTTS (Codec-based TTS)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+MagpieTTS is an end-to-end TTS model that generates audio codes from transcript and optional context (audio or text). It supports multiple architectures (e.g. multi-encoder context, decoder context) and can be used for standard, long-form, and streaming inference.
+
+.. autoclass:: nemo.collections.tts.models.MagpieTTSModel
+    :show-inheritance:
+    :members: infer_batch, do_tts, generate_long_form_speech, list_available_models, audio_to_codes, codes_to_audio, has_baked_context_embedding, num_baked_speakers, get_baked_context_embeddings_batch, create_longform_chunk_state, embed_text, prepare_context_tensors
+
+Mel-Spectrogram Generators
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. autoclass:: nemo.collections.tts.models.FastPitchModel
+    :show-inheritance:
+    :members:
+    :exclude-members: setup_training_data, setup_validation_data, training_step, on_validation_epoch_end, validation_step, setup_test_data, on_train_epoch_start
+
+
+Vocoders
+~~~~~~~~
+.. autoclass:: nemo.collections.tts.models.HifiGanModel
+    :show-inheritance:
+    :members:
+    :exclude-members: setup_training_data, setup_validation_data, training_step, on_validation_epoch_end, validation_step, setup_test_data, on_train_epoch_start
+
+
+Codecs
+~~~~~~
+.. autoclass:: nemo.collections.tts.models.AudioCodecModel
+    :show-inheritance:
+    :members:
+    :exclude-members: setup_training_data, setup_validation_data, training_step, on_validation_epoch_end, validation_step, setup_test_data, on_train_epoch_start
+
+
+Base Classes
+----------------
+
+The classes below are the base of the TTS pipeline.
+
+.. autoclass:: nemo.collections.tts.models.base.MelToSpec
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.tts.models.base.SpectrogramGenerator
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.tts.models.base.Vocoder
+    :show-inheritance:
+    :members:
+
+
+Dataset Processing Classes
+--------------------------
+.. autoclass:: nemo.collections.tts.data.dataset.TTSDataset
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.tts.data.dataset.VocoderDataset
+    :show-inheritance:
+    :members:

docs/source/tts/checkpoints.rst

@@ -0,0 +1,164 @@
+Checkpoints
+===========
+
+There are two main ways to load pretrained checkpoints in NeMo as described in :doc:`../asr/results`.
+
+* Using the :code:`restore_from()` method to load a local checkpoint file (``.nemo``), or
+* Using the :code:`from_pretrained()` method to download and set up a checkpoint from NGC.
+
+Note that these instructions are for loading fully trained checkpoints for evaluation or fine-tuning. For resuming an unfinished
+training experiment, use the Experiment Manager to do so by setting the ``resume_if_exists`` flag to ``True``.
+
+Local Checkpoints
+-------------------------
+
+* **Save Model Checkpoints**: NeMo automatically saves final model checkpoints with ``.nemo`` suffix. You could also manually save any model checkpoint using :code:`model.save_to(<checkpoint_path>.nemo)`.
+* **Load Model Checkpoints**: if you'd like to load a checkpoint saved at ``<path/to/checkpoint/file.nemo>``, use the :code:`restore_from()` method below, where ``<MODEL_BASE_CLASS>`` is the TTS model class of the original checkpoint.
+
+.. code-block:: python
+
+    import nemo.collections.tts as nemo_tts
+    model = nemo_tts.models.<MODEL_BASE_CLASS>.restore_from(restore_path="<path/to/checkpoint/file.nemo>")
+
+NGC Pretrained Checkpoints
+--------------------------
+
+The NGC `NeMo Text to Speech collection <https://catalog.ngc.nvidia.com/orgs/nvidia/collections/nemo_tts>`_ aggregates model cards that contain detailed information about  checkpoints of various models trained on various datasets. The tables below in :ref:`Checkpoints<NGC TTS Models>` list part of available TTS models from NGC including speech/text aligners, acoustic models, and vocoders.
+
+Load Model Checkpoints
+^^^^^^^^^^^^^^^^^^^^^^
+The models can be accessed via the :code:`from_pretrained()` method inside the TTS Model class. In general, you can load any of these models with code in the following format,
+
+.. code-block:: python
+
+    import nemo.collections.tts as nemo_tts
+    model = nemo_tts.models.<MODEL_BASE_CLASS>.from_pretrained(model_name="<MODEL_NAME>")
+
+where ``<MODEL_NAME>`` is the value in ``Model Name`` column in the tables in :ref:`Checkpoints<NGC TTS Models>`. These names are predefined in the each model's member function ``self.list_available_models()``. For example, the available NGC FastPitch model names can be found,
+
+.. code-block:: shell-session
+
+    In [1]: import nemo.collections.tts as nemo_tts
+
+    In [2]: nemo_tts.models.FastPitchModel.list_available_models()
+    Out[2]:
+    [PretrainedModelInfo(
+        pretrained_model_name=tts_en_fastpitch,
+        description=This model is trained on LJSpeech sampled at 22050Hz with and can be used to generate female English voices with an American accent. It is ARPABET-based.,
+        location=https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_fastpitch/versions/1.8.1/files/tts_en_fastpitch_align.nemo,
+        class_=<class 'nemo.collections.tts.models.fastpitch.FastPitchModel'>
+     ),
+     PretrainedModelInfo(
+        pretrained_model_name=tts_en_fastpitch_ipa,
+        description=This model is trained on LJSpeech sampled at 22050Hz with and can be used to generate female English voices with an American accent. It is IPA-based.,
+        location=https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_fastpitch/versions/IPA_1.13.0/files/tts_en_fastpitch_align_ipa.nemo,
+        class_=<class 'nemo.collections.tts.models.fastpitch.FastPitchModel'>
+     ),
+     PretrainedModelInfo(
+        pretrained_model_name=tts_en_fastpitch_multispeaker,
+        description=This model is trained on HiFITTS sampled at 44100Hz with and can be used to generate male and female English voices with an American accent.,
+        location=https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_multispeaker_fastpitchhifigan/versions/1.10.0/files/tts_en_fastpitch_multispeaker.nemo,
+        class_=<class 'nemo.collections.tts.models.fastpitch.FastPitchModel'>
+     ),
+     PretrainedModelInfo(
+        pretrained_model_name=tts_de_fastpitch_singlespeaker,
+        description=This model is trained on a single male speaker data in OpenSLR Neutral German Dataset sampled at 22050Hz and can be used to generate male German voices.,
+        location=https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitchhifigan/versions/1.10.0/files/tts_de_fastpitch_align.nemo,
+        class_=<class 'nemo.collections.tts.models.fastpitch.FastPitchModel'>
+     ),
+     PretrainedModelInfo(
+        pretrained_model_name=tts_de_fastpitch_multispeaker_5,
+        description=This model is trained on 5 speakers in HUI-Audio-Corpus-German clean subset sampled at 44100Hz with and can be used to generate male and female German voices.,
+        location=https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitch_multispeaker_5/versions/1.11.0/files/tts_de_fastpitch_multispeaker_5.nemo,
+        class_=<class 'nemo.collections.tts.models.fastpitch.FastPitchModel'>
+     )]
+
+From the above key-value pair ``pretrained_model_name=tts_en_fastpitch``, you could get the model name ``tts_en_fastpitch`` and load it by running,
+
+.. code-block:: python
+
+    model = nemo_tts.models.FastPitchModel.from_pretrained(model_name="tts_en_fastpitch")
+
+If you would like to programmatically list the models available for a particular base class, you can use the
+:code:`list_available_models()` method,
+
+.. code-block:: python
+
+    nemo_tts.models.<MODEL_BASE_CLASS>.list_available_models()
+
+Inference and Audio Generation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+NeMo TTS supports both cascaded and end-to-end models to synthesize audios. Most of steps in between are the same except that cascaded models need to load an extra vocoder model before generating audios. Below code snippet demonstrates steps of generating a audio sample from a text input using a cascaded FastPitch and HiFiGAN models. Please refer to :ref:`NeMo TTS Collection API` for detailed implementation of model classes.
+
+.. code-block:: python
+
+    import nemo.collections.tts as nemo_tts
+    # Load mel spectrogram generator
+    spec_generator = nemo_tts.models.FastPitchModel.from_pretrained("tts_en_fastpitch")
+    # Load vocoder
+    vocoder = nemo_tts.models.HifiGanModel.from_pretrained(model_name="tts_en_hifigan")
+    # Generate audio
+    import soundfile as sf
+    parsed = spec_generator.parse("You can type your sentence here to get nemo to produce speech.")
+    spectrogram = spec_generator.generate_spectrogram(tokens=parsed)
+    audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)
+    # Save the audio to disk in a file called speech.wav
+    sf.write("speech.wav", audio.to('cpu').numpy(), 22050)
+
+
+Fine-Tuning on Different Datasets
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are multiple TTS tutorials provided in the directory of `tutorials/tts/ <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts>`_. Most of these tutorials demonstrate how to instantiate a pre-trained model, and prepare the model for fine-tuning on datasets with the same language or different languages, the same speaker or different speakers.
+
+* **cross-lingual fine-tuning**: https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts/FastPitch_GermanTTS_Training.ipynb
+* **cross-speaker fine-tuning**: https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts/FastPitch_Finetuning.ipynb
+
+
+.. _NGC TTS Models:
+
+NGC TTS Models
+--------------
+
+This section summarizes a full list of available NeMo TTS models that have been released in `NGC NeMo Text to Speech Collection <https://catalog.ngc.nvidia.com/orgs/nvidia/collections/nemo_tts/entities>`_. You can download model checkpoints of your interest via either way below,
+
+* :code:`wget '<CHECKPOINT_URL_IN_THE_TABLE>'`
+* :code:`curl -LO '<CHECKPOINT_URL_IN_THE_TABLE>'`
+
+
+Speech/Text Aligners
+^^^^^^^^^^^^^^^^^^^^
+
+.. csv-table::
+   :file: data/ngc_models_aligner.csv
+   :align: left
+   :header-rows: 1
+
+Mel-Spectrogram Generators
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. csv-table::
+   :file: data/ngc_models_am.csv
+   :align: left
+   :header-rows: 1
+
+Vocoders
+^^^^^^^^
+.. csv-table::
+   :file: data/ngc_models_vocoder.csv
+   :align: left
+   :header-rows: 1
+
+End2End models
+^^^^^^^^^^^^^^
+.. csv-table::
+   :file: data/ngc_models_e2e.csv
+   :align: left
+   :header-rows: 1
+
+Codec models
+^^^^^^^^^^^^
+.. csv-table::
+   :file: data/models_codec.csv
+   :align: left
+   :header-rows: 1

docs/source/tts/configs.rst

@@ -0,0 +1,264 @@
+NeMo TTS Configuration Files
+============================
+This section describes the NeMo configuration file setup that is specific to models in the TTS collection. For general information
+about how to set up and run experiments that is common to all NeMo models (e.g. Experiment Manager and PyTorch Lightning trainer
+parameters), see the :doc:`../core/core` section.
+
+The model section of the NeMo TTS configuration files generally requires information about the dataset(s) being used, the preprocessor
+for audio files, parameters for any augmentation being performed, as well as the model architecture specification. The sections on
+this page cover each of these in more detail.
+
+Example configuration files for all of the NeMo TTS scripts can be found in the
+`config directory of the examples <https://github.com/NVIDIA/NeMo/tree/stable/examples/tts/conf>`_.
+
+Dataset Configuration
+---------------------
+
+Training, validation, and test parameters are specified using the ``model.train_ds``, ``model.validation_ds``, and ``model.test_ds`` sections in the configuration file, respectively. Depending on the task, there may be arguments specifying the sample rate of the audio files, supplementary data such as speech/text alignment priors and speaker IDs, etc., the threshold to trim leading and trailing silence from an audio signal, pitch normalization parameters, and so on. You may also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command-line at runtime.
+
+Any initialization parameter that is accepted for the class `nemo.collections.tts.data.dataset.TTSDataset
+<https://github.com/NVIDIA/NeMo/tree/stable/nemo/collections/tts/data/dataset.py>`_  can be set in the config
+file. Refer to the `Dataset Processing Classes <./api.html#datasets>`__ section of the API for a list of datasets classes and their respective parameters. An example TTS train and validation configuration should look similar to the following:
+
+.. code-block:: yaml
+
+  model:
+    train_ds:
+      dataset:
+        _target_: nemo.collections.tts.data.dataset.TTSDataset
+        manifest_filepath: ???
+        sample_rate: 44100
+        sup_data_path: ???
+        sup_data_types: ["align_prior_matrix", "pitch"]
+        n_fft: 2048
+        win_length: 2048
+        hop_length: 512
+        window: hann
+        n_mels: 80
+        lowfreq: 0
+        highfreq: null
+        max_duration: null
+        min_duration: 0.1
+        ignore_file: null
+        trim: false
+        pitch_fmin: 65.40639132514966
+        pitch_fmax: 2093.004522404789
+        pitch_norm: true
+        pitch_mean: 212.35873413085938
+        pitch_std: 68.52806091308594
+        use_beta_binomial_interpolator: true
+
+      dataloader_params:
+        drop_last: false
+        shuffle: true
+        batch_size: 32
+        num_workers: 12
+        pin_memory: true
+
+
+Audio Preprocessor Configuration
+--------------------------------
+
+If you are loading audio files for your experiment, you will likely want to use a preprocessor to convert from the raw audio signal to features (e.g. mel-spectrogram or MFCC). The ``preprocessor`` section of the config specifies the audio preprocessor to be used via the ``_target_`` field, as well as any initialization parameters for that preprocessor. An example of specifying a preprocessor is as follows. Refer to the :ref:`Audio Preprocessors <asr-audio-preprocessors>` in the ASR API section for the preprocessor options, expected arguments, and defaults.
+
+.. code-block:: yaml
+
+  model:
+    preprocessor:
+      _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+      features: 80
+      lowfreq: 0
+      highfreq: null
+      n_fft: 2048
+      n_window_size: 2048
+      window_size: false
+      n_window_stride: 512
+      window_stride: false
+      pad_to: 1
+      pad_value: 0
+      sample_rate: 44100
+      window: hann
+      normalize: null
+      preemph: null
+      dither: 0.0
+      frame_splicing: 1
+      log: true
+      log_zero_guard_type: add
+      log_zero_guard_value: 1e-05
+      mag_power: 1.0
+
+Text Normalizer Configuration
+------------------------------
+Text normalization (TN) converts text from written form into its verbalized form, and it is an essential preprocessing step before text-to-speech Synthesis. TN ensures that TTS can handle all input texts without skipping unknown symbols. For example, "$123" is converted to "one hundred and twenty three dollars". Currently, NeMo supports text normalizers for English, German, Spanish, and Chinese. Refer to the ``nemo_text_processing`` package for more details. Below shows an example of specifying text normalizer for English.
+
+.. code-block:: yaml
+
+  model:
+    text_normalizer:
+      _target_: nemo_text_processing.text_normalization.normalize.Normalizer
+      lang: en
+      input_case: cased
+
+    text_normalizer_call_kwargs:
+      verbose: false
+      punct_pre_process: true
+      punct_post_process: true
+
+Tokenizer Configuration
+------------------------
+Tokenization converts input text string to a list of integer tokens. It may pad leading and/or trailing whitespaces to a string. NeMo tokenizer supports grapheme-only inputs, phoneme-only inputs, or a mixer of grapheme and phoneme inputs to disambiguate pronunciations of heteronyms for English, German, and Spanish. It also utilizes a grapheme-to-phoneme (G2P) tool to transliterate out-of-vocabulary (OOV) words. Please refer to the :doc:`G2P section <./g2p>` and `TTS tokenizer collection <https://github.com/NVIDIA/NeMo/tree/stable/nemo/collections/common/tokenizers/text_to_speech/tts_tokenizers.py>`_ for more details. Note that G2P integration to NeMo TTS tokenizers pipeline is upcoming soon. The following example sets up a ``EnglishPhonemesTokenizer`` with a mixer of grapheme and phoneme inputs where each word shown in the heteronym list is transliterated into graphemes or phonemes by a 50% chance.
+
+.. code-block:: yaml
+
+  model:
+    text_tokenizer:
+      _target_: nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.EnglishPhonemesTokenizer
+      punct: true
+      stresses: true
+      chars: true
+      apostrophe: true
+      pad_with_space: true
+      g2p:
+        _target_: nemo.collections.tts.g2p.models.en_us_arpabet.EnglishG2p
+        phoneme_dict: ${phoneme_dict_path}
+        heteronyms: ${heteronyms_path}
+      phoneme_probability: 0.5
+
+
+Model Architecture Configuration
+--------------------------------
+Each configuration file should describe the model architecture being used for the experiment. Models in the NeMo TTS collection need several module sections with the ``_target_`` field specifying which model architecture or component is used. Please refer to `TTS module collection <https://github.com/NVIDIA/NeMo/tree/stable/nemo/collections/tts/modules>`_ for details. Below shows an example of FastPitch model architecture,
+
+.. code-block:: yaml
+
+  model:
+    input_fft: #n_embed and padding_idx are added by the model
+      _target_: nemo.collections.tts.modules.transformer.FFTransformerEncoder
+      n_layer: 6
+      n_head: 1
+      d_model: 384
+      d_head: 64
+      d_inner: 1536
+      kernel_size: 3
+      dropout: 0.1
+      dropatt: 0.1
+      dropemb: 0.0
+      d_embed: 384
+
+    output_fft:
+      _target_: nemo.collections.tts.modules.transformer.FFTransformerDecoder
+      n_layer: 6
+      n_head: 1
+      d_model: 384
+      d_head: 64
+      d_inner: 1536
+      kernel_size: 3
+      dropout: 0.1
+      dropatt: 0.1
+      dropemb: 0.0
+
+    alignment_module:
+      _target_: nemo.collections.tts.modules.aligner.AlignmentEncoder
+      n_text_channels: 384
+
+    duration_predictor:
+      _target_: nemo.collections.tts.modules.fastpitch.TemporalPredictor
+      input_size: 384
+      kernel_size: 3
+      filter_size: 256
+      dropout: 0.1
+      n_layers: 2
+
+    pitch_predictor:
+      _target_: nemo.collections.tts.modules.fastpitch.TemporalPredictor
+      input_size: 384
+      kernel_size: 3
+      filter_size: 256
+      dropout: 0.1
+      n_layers: 2
+
+    optim:
+      name: adamw
+      lr: 1e-3
+      betas: [0.9, 0.999]
+      weight_decay: 1e-6
+
+      sched:
+        name: NoamAnnealing
+        warmup_steps: 1000
+        last_epoch: -1
+        d_model: 1  # Disable scaling based on model dim
+
+Finetuning Configuration
+--------------------------
+
+All TTS scripts support easy finetuning by partially/fully loading the pretrained weights from a checkpoint into the **currently instantiated model**. Note that the currently instantiated model should have parameters that match the pre-trained checkpoint (such that weights may load properly). In order to directly finetune a pre-existing checkpoint, please follow the tutorial of `Finetuning FastPitch for a new speaker. <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts/FastPitch_Finetuning.ipynb>`_
+
+Pre-trained weights can be provided in multiple ways:
+
+1) Providing a path to a NeMo model (via ``init_from_nemo_model``)
+2) Providing a name of a pretrained NeMo model (which will be downloaded via the cloud) (via ``init_from_pretrained_model``)
+3) Providing a path to a Pytorch Lightning checkpoint file (via ``init_from_ptl_ckpt``)
+
+There are multiple TTS model finetuning scripts in `examples/tts/<model>_finetune.py <https://github.com/NVIDIA/NeMo/tree/stable/examples/tts/>`_. You can finetune any model by substituting the ``<model>`` tag. An example of finetuning a HiFiGAN model is shown below.
+
+Fine-tuning via a NeMo model
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: sh
+    :emphasize-lines: 13
+
+    python examples/tts/hifigan_finetune.py \
+        --config-path=<path to dir of configs> \
+        --config-name=<name of config without .yaml>) \
+        model/train_ds=train_ds_finetune \
+        model/validation_ds=val_ds_finetune \
+        train_dataset="<path to manifest file>" \
+        validation_dataset="<path to manifest file>" \
+        model.optim.lr=0.00001 \
+        ~model.optim.sched \
+        trainer.devices=-1 \
+        trainer.accelerator='gpu' \
+        trainer.max_epochs=50 \
+        +init_from_nemo_model="<path to .nemo model file>"
+
+
+Fine-tuning via a NeMo pretrained model name
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: sh
+    :emphasize-lines: 13
+
+    python examples/tts/hifigan_finetune.py \
+        --config-path=<path to dir of configs> \
+        --config-name=<name of config without .yaml>) \
+        model/train_ds=train_ds_finetune \
+        model/validation_ds=val_ds_finetune \
+        train_dataset="<path to manifest file>" \
+        validation_dataset="<path to manifest file>" \
+        model.optim.lr=0.00001 \
+        ~model.optim.sched \
+        trainer.devices=-1 \
+        trainer.accelerator='gpu' \
+        trainer.max_epochs=50 \
+        +init_from_pretrained_model="<name of pretrained checkpoint>"
+
+Fine-tuning via a Pytorch Lightning checkpoint
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: sh
+    :emphasize-lines: 13
+
+    python examples/tts/hifigan_finetune.py \
+        --config-path=<path to dir of configs> \
+        --config-name=<name of config without .yaml>) \
+        model/train_ds=train_ds_finetune \
+        model/validation_ds=val_ds_finetune \
+        train_dataset="<path to manifest file>" \
+        validation_dataset="<path to manifest file>" \
+        model.optim.lr=0.00001 \
+        ~model.optim.sched \
+        trainer.devices=-1 \
+        trainer.accelerator='gpu' \
+        trainer.max_epochs=50 \
+        +init_from_ptl_ckpt="<name of pytorch lightning checkpoint>"

docs/source/tts/data/datasets.csv

@@ -0,0 +1,14 @@
+Language,Locale,Dataset Name,#spk-total,#spk-F,#spk-M,#hours-total,#hour-F,#hour-M,Sampling Rate,URL
+English,en-US,LJSpeech,1,1,0,23.92,23.92,0.00,"22,050Hz",https://keithito.com/LJ-Speech-Dataset/
+English,en-US,LibriTTS (clean),1230,592,638,262.62,133.97,128.65,"24,000Hz",https://www.openslr.org/60/
+English,en-US,HiFiTTS,10,6,4,291.60,158.30,133.30,"44,100Hz",http://www.openslr.org/109/
+German,de-DE,Thorsten Müller Neutral 21.02 dataset,1,0,1,20.91,0.00,20.91,"22,050Hz",https://zenodo.org/record/5525342/files/thorsten-neutral_v03.tgz?download=1
+German,de-DE,Thorsten Müller Neutral 22.10 dataset,1,0,1,11.21,0.00,11.21,"22,050Hz",https://zenodo.org/record/7265581/files/ThorstenVoice-Dataset_2022.10.zip?download=1
+German,de-DE,HUI-Audio-Corpus-German (clean),118,n/a,n/a,253.85,0.00,0.00,"44,100Hz",https://github.com/iisys-hof/HUI-Audio-Corpus-German
+Spanish,es-AR,Crowdsourced high-quality Argentinian Spanish,44,31,13,8.03,5.61,2.42,"48,000Hz",https://www.openslr.org/61/
+Spanish,es-CL,Crowdsourced high-quality Chilean Spanish,31,13,18,7.15,2.84,4.31,"48,000Hz",https://www.openslr.org/71/
+Spanish,es-CO,Crowdsourced high-quality Colombian Spanish,33,16,17,7.58,3.74,3.84,"48,000Hz",https://www.openslr.org/72/
+Spanish,es-PE,Crowdsourced high-quality Peruvian Spanish,38,18,20,9.22,4.35,4.87,"48,000Hz",https://www.openslr.org/73/
+Spanish,es-PR,Crowdsourced high-quality Puerto Rico Spanish,5,5,0,1.00,1.00,0.00,"48,000Hz",https://www.openslr.org/74/
+Spanish,es-VE,Crowdsourced high-quality Venezuelan Spanish,23,11,12,4.81,2.41,2.40,"48,000Hz",https://www.openslr.org/75/
+Chinese,zh-CN,SFSpeech Chinese/English Bilingual Speech,1,1,0,4.50,4.50,0.00,"22,050Hz",https://catalog.ngc.nvidia.com/orgs/nvidia/resources/sf_bilingual_speech_zh_en

docs/source/tts/data/models_codec.csv

@@ -0,0 +1,11 @@
+Model Name,Dataset,Sampling Rate,Model Class,Overview,Checkpoint
+audio_codec_16khz_small,Libri-Light,16000Hz,nemo.collections.tts.models.AudioCodecModel,`audio_codec_16khz_small <https://ngc.nvidia.com/catalog/models/nvidia:nemo:audio_codec_16khz_small>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/audio_codec_16khz_small/versions/v1/files/audio_codec_16khz_small.nemo``
+mel_codec_22khz_medium,LibriVox and Common Voice,22050Hz,nemo.collections.tts.models.AudioCodecModel,`mel_codec_22khz_medium <https://ngc.nvidia.com/catalog/models/nvidia:nemo:mel_codec_22khz_medium>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/mel_codec_22khz_medium/versions/v1/files/mel_codec_22khz_medium.nemo``
+mel_codec_44khz_medium,LibriVox and Common Voice,44100Hz,nemo.collections.tts.models.AudioCodecModel,`mel_codec_44khz_medium <https://ngc.nvidia.com/catalog/models/nvidia:nemo:mel_codec_44khz_medium>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/mel_codec_44khz_medium/versions/v1/files/mel_codec_44khz_medium.nemo``
+mel_codec_22khz_fullband_medium,LibriVox and Common Voice,22050Hz,nemo.collections.tts.models.AudioCodecModel,`mel_codec_22khz_fullband_medium <https://ngc.nvidia.com/catalog/models/nvidia:nemo:mel_codec_22khz_fullband_medium>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/mel_codec_22khz_fullband_medium/versions/v1/files/mel_codec_22khz_fullband_medium.nemo``
+mel_codec_44khz_fullband_medium,LibriVox and Common Voice,44100Hz,nemo.collections.tts.models.AudioCodecModel,`mel_codec_44khz_fullband_medium <https://ngc.nvidia.com/catalog/models/nvidia:nemo:mel_codec_44khz_fullband_medium>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/mel_codec_44khz_fullband_medium/versions/v1/files/mel_codec_44khz_fullband_medium.nemo``
+nvidia/low-frame-rate-speech-codec-22khz,LibriVox and Common Voice,22050Hz,nemo.collections.tts.models.AudioCodecModel,`audio_codec_low_frame_rate_22khz <https://huggingface.co/nvidia/low-frame-rate-speech-codec-22khz>`_,``https://huggingface.co/nvidia/low-frame-rate-speech-codec-22khz/resolve/main/low-frame-rate-speech-codec-22khz.nemo``
+nvidia/audio-codec-22khz,LibriVox and Common Voice,22050Hz,nemo.collections.tts.models.AudioCodecModel,`audio-codec-22khz <https://huggingface.co/nvidia/audio-codec-22khz>`_,``https://huggingface.co/nvidia/audio-codec-22khz/resolve/main/audio-codec-22khz.nemo``
+nvidia/audio-codec-44khz,LibriVox and Common Voice,44100Hz,nemo.collections.tts.models.AudioCodecModel,`audio-codec-44khz <https://huggingface.co/nvidia/audio-codec-44khz>`_,``https://huggingface.co/nvidia/audio-codec-44khz/resolve/main/audio-codec-44khz.nemo``
+nvidia/mel-codec-22khz,LibriVox and Common Voice,22050Hz,nemo.collections.tts.models.AudioCodecModel,`mel-codec-22khz <https://huggingface.co/nvidia/mel-codec-22khz>`_,``https://huggingface.co/nvidia/mel-codec-22khz/resolve/main/mel-codec-22khz.nemo``
+nvidia/mel-codec-44khz,LibriVox and Common Voice,44100Hz,nemo.collections.tts.models.AudioCodecModel,`mel-codec-44khz <https://huggingface.co/nvidia/mel-codec-44khz>`_,``https://huggingface.co/nvidia/mel-codec-44khz/resolve/main/mel-codec-44khz.nemo``

docs/source/tts/data/ngc_models_aligner.csv

@@ -0,0 +1,2 @@
+Locale,Model Name,Dataset,Sampling Rate,Model Class,Overview,Checkpoint
+en-US,tts_en_fastpitch_align,LJSpeech,22050Hz,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_en_fastpitch (aligner) <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_en_fastpitch>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_fastpitch/versions/1.8.1/files/tts_en_fastpitch_align.nemo``

docs/source/tts/data/ngc_models_am.csv

@@ -0,0 +1,9 @@
+Locale,Model Name,Dataset,Sampling Rate,#Spk,Symbols,Model Class,Overview,Checkpoint
+en-US,tts_en_fastpitch,LJSpeech,22050Hz,1,ARPABET,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_en_fastpitch <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_en_fastpitch>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_fastpitch/versions/1.8.1/files/tts_en_fastpitch_align.nemo``
+en-US,tts_en_fastpitch_ipa,LJSpeech,22050Hz,1,IPA,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_en_fastpitch <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_en_fastpitch>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_fastpitch/versions/IPA_1.13.0/files/tts_en_fastpitch_align_ipa.nemo``
+en-US,tts_en_fastpitch_multispeaker,HiFiTTS,44100Hz,10,ARPABET,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_en_multispeaker_fastpitchhifigan <https://ngc.nvidia.com/models/nvidia:nemo:tts_en_multispeaker_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_multispeaker_fastpitchhifigan/versions/1.10.0/files/tts_en_fastpitch_multispeaker.nemo``
+de-DE,tts_de_fastpitch_multispeaker_5,HUI Audio Corpus German,44100Hz,5,ARPABET,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_de_fastpitch_multispeaker_5 <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_de_fastpitch_multispeaker_5>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitch_multispeaker_5/versions/1.11.0/files/tts_de_fastpitch_multispeaker_5.nemo``
+de-DE,tts_de_fastpitch_singleSpeaker_thorstenNeutral_2102,Thorsten Müller Neutral 21.02 dataset,22050Hz,1,Graphemes,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_de_fastpitchhifigan <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_de_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitchhifigan/versions/1.15.0/files/tts_de_fastpitch_thorstens2102.nemo``
+de-DE,tts_de_fastpitch_singleSpeaker_thorstenNeutral_2210,Thorsten Müller Neutral 22.10 dataset,22050Hz,1,Graphemes,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_de_fastpitchhifigan <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_de_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitchhifigan/versions/1.15.0/files/tts_de_fastpitch_thorstens2210.nemo``
+es,tts_es_fastpitch_multispeaker,OpenSLR crowdsourced Latin American Spanish,44100Hz,174,IPA,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_es_multispeaker_fastpitchhifigan <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_es_multispeaker_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_es_multispeaker_fastpitchhifigan/versions/1.15.0/files/tts_es_fastpitch_multispeaker.nemo``
+zh-CN,tts_zh_fastpitch_sfspeech,SFSpeech Chinese/English Bilingual Speech,22050Hz,1,pinyin,nemo.collections.tts.models.fastpitch.FastPitchModel,`tts_zh_fastpitch_hifigan_sfspeech <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_zh_fastpitch_hifigan_sfspeech>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_zh_fastpitch_hifigan_sfspeech/versions/1.15.0/files/tts_zh_fastpitch_sfspeech.nemo``

docs/source/tts/data/ngc_models_e2e.csv

@@ -0,0 +1,2 @@
+Locale,Model Name,Dataset,Sampling Rate,#Spk,Model Class,Overview,Checkpoint
+en-US,tts_en_mixertts,LJSpeech,22050Hz,1,nemo.collections.tts.models.MixerTTSModel,`tts_en_mixertts <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_en_mixertts>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_mixertts/versions/1.0.0/files/tts_en_mixertts.nemo``

docs/source/tts/data/ngc_models_vocoder.csv

@@ -0,0 +1,10 @@
+Locale,Model Name,Spectrogram Generator,Dataset,Sampling Rate,#Spk,Model Class,Overview,Checkpoint
+en-US,tts_en_hifigan,librosa.filters.mel,LJSpeech,22050Hz,1,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_en_hifigan <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_hifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_hifigan/versions/1.0.0rc1/files/tts_hifigan.nemo``
+en-US,tts_en_lj_hifigan_ft_mixertts,Mixer-TTS,LJSpeech,22050Hz,1,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_en_lj_hifigan <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_en_lj_hifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_lj_hifigan/versions/1.6.0/files/tts_en_lj_hifigan_ft_mixertts.nemo``
+en-US,tts_en_lj_hifigan_ft_mixerttsx,Mixer-TTS-X,LJSpeech,22050Hz,1,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_en_lj_hifigan <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_en_lj_hifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_lj_hifigan/versions/1.6.0/files/tts_en_lj_hifigan_ft_mixerttsx.nemo``
+en-US,tts_en_hifitts_hifigan_ft_fastpitch,FastPitch,HiFiTTS,44100Hz,10,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_en_multispeaker_fastpitchhifigan <https://ngc.nvidia.com/models/nvidia:nemo:tts_en_multispeaker_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_multispeaker_fastpitchhifigan/versions/1.10.0/files/tts_en_hifitts_hifigan_ft_fastpitch.nemo``
+de-DE,tts_de_hui_hifigan_ft_fastpitch_multispeaker_5,FastPitch,HUI Audio Corpus German,44100Hz,5,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_de_fastpitch_multispeaker_5 <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_de_fastpitch_multispeaker_5>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitch_multispeaker_5/versions/1.11.0/files/tts_de_hui_hifigan_ft_fastpitch_multispeaker_5.nemo``
+de-DE,tts_de_hifigan_singleSpeaker_thorstenNeutral_2102,FastPitch,Thorsten Müller Neutral 21.02 dataset,22050Hz,1,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_de_fastpitchhifigan <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_de_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitchhifigan/versions/1.15.0/files/tts_de_hifigan_thorstens2102.nemo``
+de-DE,tts_de_hifigan_singleSpeaker_thorstenNeutral_2210,FastPitch,Thorsten Müller Neutral 22.10 dataset,22050Hz,1,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_de_fastpitchhifigan <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_de_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_de_fastpitchhifigan/versions/1.15.0/files/tts_de_hifigan_thorstens2210.nemo``
+es,tts_es_hifigan_ft_fastpitch_multispeaker,FastPitch,OpenSLR crowdsourced Latin American Spanish,44100Hz,174,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_es_multispeaker_fastpitchhifigan <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_es_multispeaker_fastpitchhifigan>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_es_multispeaker_fastpitchhifigan/versions/1.15.0/files/tts_es_hifigan_ft_fastpitch_multispeaker.nemo``
+zh-CN,tts_zh_hifigan_sfspeech,FastPitch,SFSpeech Chinese/English Bilingual Speech,22050Hz,1,nemo.collections.tts.models.hifigan.HifiGanModel,`tts_zh_fastpitch_hifigan_sfspeech <https://ngc.nvidia.com/catalog/models/nvidia:nemo:tts_zh_fastpitch_hifigan_sfspeech>`_,``https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_zh_fastpitch_hifigan_sfspeech/versions/1.15.0/files/tts_zh_hifigan_sfspeech.nemo``

docs/source/tts/datasets.rst

@@ -0,0 +1,237 @@
+Data Preprocessing
+==================
+
+NeMo TTS recipes support most of public TTS datasets that consist of multiple languages, multiple emotions, and multiple speakers. Current recipes covered English (en-US), German (de-DE), Spanish (es-ES), and Mandarin Chinese (zh-CN), while the support for many other languages is under planning. NeMo provides corpus-specific data preprocessing scripts, as shown in the directory of `scripts/data_processing/tts/ <https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/>`_, to convert common public TTS datasets into the format expected by the dataloaders as defined in `nemo/collections/tts/data/dataset.py <https://github.com/NVIDIA/NeMo/tree/stable/nemo/collections/tts/data/dataset.py>`_. The ``nemo_tts`` collection expects each dataset to consist of a set of utterances in individual audio files plus a ``JSON`` manifest that describes the dataset, with information about one utterance per line. We recommend ``WAV`` files as they are the default and have been most thoroughly tested. NeMo supports any original sampling rates of audios, although our scripts of extracting supplementary data and model training all specify the common target sampling rates as either 44100 Hz or 22050 Hz. If the original sampling rate mismatches the target sampling rate, the `feature preprocessing <https://github.com/NVIDIA/NeMo/blob/stable/nemo/collections/asr/parts/preprocessing/features.py>`__ can automatically resample the original sampling rate into the target one.
+
+There should be one ``JSON`` manifest file per dataset that will be passed in, therefore, if the user wants separate training and validation datasets, they should also have separate manifests. Otherwise, they will be loading validation data with their training data and vice versa. Each line of the manifest should be in the following format:
+
+.. code-block:: json
+
+    {
+      "audio_filepath": "/path/to/audio.wav",
+      "text": "the transcription of the utterance",
+      "normalized_text": "the normalized transcription of the utterance",
+      "speaker": 5,
+      "duration": 15.4
+    }
+
+where ``"audio_filepath"`` provides an absolute path to the ``.wav`` file corresponding to the utterance so that audio files can be located anywhere without the constraint of being organized in the same directory as the manifest itself; ``"text"`` contains the full transcript (either graphemes or phonemes or their mixer) for the utterance; ``"normalized_text"`` contains normalized ``"text"`` that helps to bypass the normalization steps but it is fully optional; ``"speaker"`` refers to the integer speaker ID; ``"duration"`` describes the duration of the utterance in seconds.
+
+Each entry in the manifest (describing one audio file) should be bordered by ``"{"`` and ``"}"`` and must be placed on one line. The ``"key": value`` pairs should be separated by a commas as shown above. NeMo enforces no blank lines in the manifest so that the total number of lines indicates the total number of audio files in the dataset.
+
+Once there is a manifest that describes each audio file in the dataset, assign the ``JSON`` manifest file path in the experiment config file, for example, ``training_ds.manifest_filepath=<path/to/manifest.json>``.
+
+Following the instructions below on how to run the corpus-specific scripts, you can get started with either directly processing those public datasets, or creating your custom scripts to preprocess your custom datasets scripts.
+
+TTS Data Collection Classes and Datasets
+----------------------------------------
+The ``nemo.collections.tts.data`` package provides dataset classes and helpers used by TTS models. The descriptions below are drawn from the docstrings in the source; see the code for full argument lists and defaults.
+
+**dataset.py**
+
+* **TTSDataset** — Dataset for training spectrogram generators and end-to-end TTS models. It loads main data (audio, text) and optional supplementary types (log mel, durations, align prior matrix, pitch, energy, speaker id). Some supplementary data is computed on the fly and saved under ``sup_data_path``. Expects JSON-line manifests with fields such as ``audio_filepath``, ``text``, optional ``normalized_text``, ``mel_filepath``, ``duration``. Supports trimming, STFT/mel args, min/max duration, ignore lists, and optional pitch augmentation and segment max duration.
+
+* **VocoderDataset** — Dataset for training and fine-tuning vocoders with optional pre-computed mel-spectrograms. Manifest lines include ``audio_filepath``, optional ``duration`` and ``mel_filepath`` (.npy or .pt). Supports ``n_segments`` for random segments, min/max duration, ignore file, trim, and ``load_precomputed_mel`` with ``hop_length``.
+
+* **FastPitchSSLDataset** — Dataset for FastPitchModel_SSL. Requires supplementary data from ``scripts/ssl_tts/make_supdata.py``. Manifest should include ``audio_filepath``, optional ``speaker`` and ``duration``. Supports ``ssl_content_emb_type`` (probs/embedding/log_probs/embedding_and_probs), pitch conditioning and normalization (speaker_wise/global/none), ``sup_data_dir``, ``speaker_stats_pitch_fp``, and ``speaker_conditioning_type`` (per_sample/mean/interpolate).
+
+* **DistributedBucketSampler** — Sampler that keeps similar input lengths in a batch. Length groups are given by boundaries; samples outside those boundaries are removed. Integrates with distributed training (num_replicas, rank, shuffle).
+
+**text_to_speech_dataset.py**
+
+* **DatasetMeta** — Dataclass with ``manifest_path``, ``audio_dir``, ``feature_dir``, ``sample_weight``, ``tokenizer_names``.
+
+* **DatasetSample** — Dataclass with ``dataset_name``, ``manifest_entry``, ``audio_dir``, ``feature_dir``, ``text``, ``speaker``, ``speaker_index``, ``tokenizer_names``.
+
+* **TextToSpeechDataset** — Class for processing and loading text-to-speech training examples. Arguments include ``dataset_meta``, ``sample_rate``, ``text_tokenizer``, optional ``weighted_sampling_steps_per_epoch``, ``speaker_path``, ``featurizers``, ``feature_processors``, ``align_prior_hop_length``, ``min_duration``, ``max_duration``, and ``volume_norm``.
+
+* **MagpieTTSDataset** — For Magpie-TTS. In addition to the manifest structure for TextToSpeechDataset, supports ``context_audio_filepath``, ``context_audio_duration``, ``target_audio_codes_path``, ``context_audio_codes_path``. If code paths are missing or ``load_cached_codes_if_available=False``, audio is loaded and codes computed in the model. Supports codec frame rate, BOS/EOS ids, prior scaling, tokenizer config, 16 kHz audio for speaker verification, text conditioning tokenizer, context duration min/max, and text context remapping.
+
+* **MagpieTTSDatasetDPO** — For DPO training. Manifests are created with ``scripts/magpietts/dpo/create_text_contextpairs.py`` and ``scripts/magpietts/dpo/create_preference_pairs.py``. Returns chosen/rejected pairs with reward labels.
+
+* **LongFormTTSInferenceDataset** — Dataset for longform TTS inference with sentence-level text chunking. Inherits MagpieTTSDataset for context audio and text conditioning; adds sentence-level chunking. Arguments include ``dataset_meta``, ``sample_rate``, ``tokenizer_name``, codec/token IDs, and context duration and text conditioning options.
+
+**text_to_speech_dataset_lhotse.py**
+
+* **MagpieTTSLhotseDataset** — PyTorch Dataset for MagpieTTS using Lhotse CutSets, for target utterances with optional text or audio context. It loads pre-computed audio codes or raw audio, applies volume normalization, and tokenizes text. Context audio/codes are sliced or repeated to fit ``context_duration_min``–``context_duration_max``. Optional 16 kHz audio for speaker verification and alignment priors. Tokenizers are initialized lazily per worker. Supports ``load_cached_codes_if_available``, ``dataset_type``, ``pad_context_text_to_max_duration``, and ``use_text_conditioning_tokenizer`` with ``tokenizer_config``.
+
+**vocoder_dataset.py**
+
+* **DatasetMeta** — Dataclass with ``manifest_path``, ``audio_dir``, ``sample_weight``, optional ``audio_tar_filepaths``.
+
+* **DatasetSample** — Dataclass with ``dataset_name``, ``manifest_entry``, ``audio_dir``.
+
+* **VocoderDataset** — Class for processing and loading vocoder training examples. Uses ``dataset_meta`` and ``sample_rate``; audio is resampled if needed. Supports featurizers and feature processors.
+
+* **TarredVocoderDataset** — Similar to VocoderDataset but loads tarred audio (WebDataset). Accepts a comma-separated JSON manifest and path(s) to tarballs; supports brace expansion. Options include ``n_samples``, ``shuffle_n``, min/max/trunc duration, ``feature_processors``, and ``shard_strategy`` (scatter vs replicate) for DDP. The helper **preprocess_manifest** filters by duration and builds samples/weights; **audio_collate_fn** stacks batched audio and lengths.
+
+Public TTS Datasets
+------------------------------
+This table below summarizes the statistics for a collection of high-quality public datasets used by NeMo TTS. We recommend to start customizing the scripts for your custom datasets that have close sampling rate and number of speakers.
+
+.. csv-table::
+   :file: data/datasets.csv
+   :align: right
+   :header-rows: 1
+
+Corpus-Specific Data Preprocessing
+----------------------------------
+NeMo implements model-agnostic data preprocessing scripts that wrap up steps of **downloading raw datasets, extracting files, and/or normalizing raw texts, and generating data manifest files**. Most scripts are able to be reused for any datasets with only minor adaptations. Most TTS models work out-of-the-box with the LJSpeech dataset, so it would be straightforward to start adapting your custom script from `LJSpeech script <https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/ljspeech/get_data.py>`_. For some models that may require supplementary data for training and validating, such as speech/text alignment prior, pitch, speaker ID, emotion ID, energy, etc, you may need an extra step of **supplementary data extraction** by calling `script/dataset_processing/tts/extract_sup_data.py <https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/extract_sup_data.py>`_ . The following sub-sections demonstrate detailed instructions for running data preprocessing scripts.
+
+LJSpeech
+~~~~~~~~
+* Dataset URL: https://keithito.com/LJ-Speech-Dataset/
+* Dataset Processing Script: https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/ljspeech/get_data.py
+* Command Line Instruction:
+
+.. code-block:: shell-session
+
+    $ python scripts/dataset_processing/tts/ljspeech/get_data.py \
+        --data-root <your_local_dataset_root>
+
+    $ python scripts/dataset_processing/tts/extract_sup_data.py \
+        --config-path ljspeech/ds_conf \
+        --config-name ds_for_fastpitch_align.yaml \
+        manifest_filepath=<your_path_to_train_manifest> \
+        sup_data_path=<your_path_to_where_to_save_supplementary_data>
+
+
+LibriTTS
+~~~~~~~~
+* Dataset URL: https://www.openslr.org/60/
+* Dataset Processing Script: https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/libritts/get_data.py
+* Command Line Instruction:
+
+.. code-block:: console
+
+    $ python scripts/dataset_processing/tts/libritts/get_data.py \
+        --data-root <your_local_dataset_root> \
+        --data-sets "ALL"
+        --num-workers 4
+
+    $ python scripts/dataset_processing/tts/extract_sup_data.py \
+        --config-path ljspeech/ds_conf \
+        --config-name ds_for_fastpitch_align.yaml \
+        manifest_filepath=<your_path_to_train_manifest> \
+        sup_data_path=<your_path_to_where_to_save_supplementary_data>
+
+.. note::
+    LibriTTS original sampling rate is **24000 Hz**, we re-use LJSpeech's config to down-sample it to **22050 Hz**.
+
+
+HiFiTTS
+~~~~~~~
+The texts of this dataset has been normalized already. So there is no extra need to preprocess the data again. But we still need a download script and split it into manifests.
+
+* Dataset URL: http://www.openslr.org/109/
+* Dataset Processing Script: TBD
+* Command Line Instruction: TBD
+
+
+Thorsten Müller's German Neutral-TTS Datasets
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+There are two German neutral datasets released by Thorsten Müller for now, 21.02 and 22.10, respectively. Version 22.10 has been recorded with a better recording setup, such as recording chamber and better microphone. So it is advised to train models on the 22.10 version because its audio quality is better and it has a way more natural speech flow and higher character rate per second speech. The two datasets are described below and defined in `scripts/dataset_processing/tts/thorsten_neutral/get_data.py <https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/thorsten_neutral/get_data.py>`_.
+
+.. code-block:: python
+
+    # Thorsten Müller published two neural voice datasets, 21.02 and 22.10.
+    THORSTEN_NEUTRAL = {
+        "21_02": {
+            "url": "https://zenodo.org/record/5525342/files/thorsten-neutral_v03.tgz?download=1",
+            "dir_name": "thorsten-de_v03",
+            "metadata": ["metadata.csv"],
+        },
+        "22_10": {
+            "url": "https://zenodo.org/record/7265581/files/ThorstenVoice-Dataset_2022.10.zip?download=1",
+            "dir_name": "ThorstenVoice-Dataset_2022.10",
+            "metadata": ["metadata_train.csv", "metadata_dev.csv", "metadata_test.csv"],
+        },
+    }
+
+* Thorsten Müller's German Datasets repo: https://github.com/thorstenMueller/Thorsten-Voice
+* Dataset Processing Script: https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/thorsten_neutral/get_data.py
+* Command Line Instruction:
+
+.. code-block:: bash
+
+    # Version 22.10
+    $ python scripts/dataset_processing/tts/thorsten_neutral/get_data.py \
+        --data-root <your_local_dataset_root> \
+        --manifests-root <your_local_manifest_root> \
+        --data-version "22_10" \
+        --val-size 100 \
+        --test-size 100 \
+        --seed-for-ds-split 100 \
+        --normalize-text
+
+    # Version 21.02
+    $ python scripts/dataset_processing/tts/thorsten_neutral/get_data.py \
+        --data-root <your_local_dataset_root> \
+        --manifests-root <your_local_manifest_root> \
+        --data-version "21_02" \
+        --val-size 100 \
+        --test-size 100 \
+        --seed-for-ds-split 100 \
+        --normalize-text
+
+    # extract pitch and compute pitch normalization params for each version.
+    $ python scripts/dataset_processing/tts/extract_sup_data.py \
+        --config-path thorsten_neutral/ds_conf \
+        --config-name ds_for_fastpitch_align.yaml \
+        manifest_filepath=<your_path_to_train_manifest> \
+        sup_data_path=<your_path_to_where_to_save_supplementary_data>
+
+HUI Audio Corpus German
+~~~~~~~~~~~~~~~~~~~~~~~
+* Dataset URL: https://github.com/iisys-hof/HUI-Audio-Corpus-German
+* Dataset Processing Script: https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/hui_acg/get_data.py
+* Command Line Instruction:
+
+.. code-block:: bash
+
+    $ python scripts/dataset_processing/tts/hui_acg/get_data.py \
+        --data-root <your_local_dataset_root> \
+        --manifests-root <your_local_manifest_root> \
+        --set-type clean \
+        --min-duration 0.1 \
+        --max-duration 15 \
+        --val-num-utts-per-speaker 1 \
+        --test-num-utts-per-speaker 1 \
+        --seed-for-ds-split 100
+
+    $ python scripts/dataset_processing/tts/hui_acg/phonemizer.py \
+        --json-manifests <your_path_to_train_manifest> <your_path_to_val_manifest> <your_path_to_test_manifest> \
+        --preserve-punctuation
+
+    $ python scripts/dataset_processing/tts/extract_sup_data.py \
+        --config-path hui_acg/ds_conf \
+        --config-name ds_for_fastpitch_align.yaml \
+        manifest_filepath=<your_path_to_train_manifest> \
+        sup_data_path=<your_path_to_where_to_save_supplementary_data>
+
+
+SFSpeech Chinese/English Bilingual Speech
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Dataset URL: https://catalog.ngc.nvidia.com/orgs/nvidia/resources/sf_bilingual_speech_zh_en
+* Dataset Processing Script: https://github.com/NVIDIA/NeMo/tree/stable/scripts/dataset_processing/tts/sfbilingual/get_data.py
+* Command Line Instruction: please refer details in Section 1 (NGC Registry CLI installation), Section 2 (Downloading SFSpeech Dataset), and Section 3 (Creatiung Data Manifests) from https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_ChineseTTS_Training.ipynb. Below code block briefly describes the steps.
+
+.. code-block:: bash
+
+    # [prerequisite] Install and setup 'ngc' cli tool by following document https://docs.ngc.nvidia.com/cli/cmd.html
+
+    $ ngc registry resource download-version "nvidia/sf_bilingual_speech_zh_en:v1"
+
+    $ unzip sf_bilingual_speech_zh_en_vv1/SF_bilingual.zip -d <your_local_dataset_root>
+
+    $ python scripts/dataset_processing/tts/sfbilingual/get_data.py \
+        --data-root <your_local_dataset_root>/SF_bilingual \
+        --val-size 0.005 \
+        --test-size 0.01 \
+        --seed-for-ds-split 100
+
+    $ python scripts/dataset_processing/tts/extract_sup_data.py \
+        --config-path sfbilingual/ds_conf \
+        --config-name ds_for_fastpitch_align.yaml \
+        manifest_filepath=<your_path_to_train_manifest> \
+        sup_data_path=<your_path_to_where_to_save_supplementary_data>

docs/source/tts/g2p.rst

@@ -0,0 +1,138 @@
+.. _g2p:
+
+Grapheme-to-Phoneme Models
+==========================
+
+Grapheme-to-phoneme conversion (G2P) is the task of transducing graphemes (i.e., orthographic symbols) to phonemes (i.e., units of the sound system of a language).
+For example, for `International_Phonetic_Alphabet (IPA): <https://en.wikipedia.org/wiki/International_Phonetic_Alphabet>`__ ``"Swifts, flushed from chimneys …" → "ˈswɪfts, ˈfɫəʃt ˈfɹəm ˈtʃɪmniz …"``.
+
+Modern text-to-speech (TTS) synthesis models can learn pronunciations from raw text input and its corresponding audio data,
+but by relying on grapheme input during training, such models fail to provide a reliable way of correcting wrong pronunciations. As a result, many TTS systems use phonetic input
+during training to directly access and correct pronunciations at inference time. G2P systems allow users to enforce the desired pronunciation by providing a phonetic transcript of the input.
+
+G2P models convert out-of-vocabulary words (OOV), e.g. proper names and loaner words, as well as heteronyms in their phonetic form to improve the quality of the syntesized text.
+
+*Heteronyms* represent words that have the same spelling but different pronunciations, e.g., “read” in “I will read the book.” vs. “She read her project last week.”  A single model that can handle OOVs and heteronyms and replace dictionary lookups can significantly simplify and improve the quality of synthesized speech.
+
+We support the following G2P models:
+
+* **ByT5 G2P** a text-to-text model that is based on ByT5 :cite:`g2p--xue2021byt5` neural network model that was originally proposed in :cite:`g2p--vrezavckova2021t5g2p` and :cite:`g2p--zhu2022byt5`.
+
+* **G2P-Conformer** CTC model -  uses a Conformer encoder :cite:`g2p--ggulati2020conformer` followed by a linear decoder; the model is trained with CTC-loss. G2P-Conformer model has about 20 times fewer parameters than the ByT5 model and is a non-autoregressive model that makes it faster during inference.
+
+The models can be trained using words or sentences as input.
+If trained with sentence-level input, the models can handle out-of-vocabulary (OOV) and heteronyms along with unambiguous words in a single pass.
+See :ref:`Sentence-level Dataset Preparation Pipeline <sentence_level_dataset_pipeline>` on how to label data for G2P model training.
+
+Model Training, Evaluation and Inference
+----------------------------------------
+
+The section covers both ByT5 and G2P-Conformer models.
+
+The models take input data in `.json` manifest format, and there should be separate training and validation manifests.
+Each line of the manifest should be in the following format:
+
+.. code::
+
+  {"text_graphemes": "Swifts, flushed from chimneys.", "text": "ˈswɪfts, ˈfɫəʃt ˈfɹəm ˈtʃɪmniz."}
+
+Manifest fields:
+
+* ``text`` - name of the field in manifest_filepath for ground truth phonemes
+
+* ``text_graphemes`` - name of the field in manifest_filepath for input grapheme text
+
+The models can handle input with and without punctuation marks.
+
+To train ByT5 G2P model and evaluate it after at the end of the training, run:
+
+.. code::
+
+    python examples/text_processing/g2p/g2p_train_and_evaluate.py \
+        # (Optional: --config-path=<Path to dir of configs> --config-name=<name of config without .yaml>) \
+        model.train_ds.manifest_filepath="<Path to manifest file>" \
+        model.validation_ds.manifest_filepath="<Path to manifest file>" \
+        model.test_ds.manifest_filepath="<Path to manifest file>" \
+        trainer.devices=1 \
+        do_training=True \
+        do_testing=True
+
+Example of the config file: ``NeMo/examples/tts/g2p/conf/g2p_t5.yaml``.
+
+
+To train G2P-Conformer model and evaluate it after at the end of the training, run:
+
+.. code-block::
+
+    python examples/text_processing/g2p/g2p_train_and_evaluate.py \
+        # (Optional: --config-path=<Path to dir of configs> --config-name=<name of config without .yaml>) \
+        model.train_ds.manifest_filepath="<Path to manifest file>" \
+        model.validation_ds.manifest_filepath="<Path to manifest file>" \
+        model.test_ds.manifest_filepath="<Path to manifest file>" \
+        model.tokenizer.dir=<Path to pretrained tokenizer> \
+        model.tokenizer_grapheme.do_lower=False \
+        model.tokenizer_grapheme.add_punctuation=True \
+        trainer.devices=1 \
+        do_training=True \
+        do_testing=True
+
+Example of the config file: ``NeMo/examples/text_processing/g2p/conf/g2p_conformer_ctc.yaml``.
+
+
+To evaluate a pretrained G2P model, run:
+
+.. code-block::
+
+    python examples/text_processing/g2p/g2p_train_and_evaluate.py \
+        # (Optional: --config-path=<Path to dir of configs> --config-name=<name of config without .yaml>) \
+        pretrained_model="<Path to .nemo file or pretrained model name from list_available_models()>" \
+        model.test_ds.manifest_filepath="<Path to manifest file>" \
+        trainer.devices=1 \
+        do_training=False \
+        do_testing=True
+
+To run inference with a pretrained G2P model, run:
+
+.. code-block::
+
+    python g2p_inference.py \
+        pretrained_model=<Path to .nemo file or pretrained model name for G2PModel from list_available_models()>" \
+        manifest_filepath="<Path to .json manifest>" \
+        output_file="<Path to .json manifest to save prediction>" \
+        batch_size=32 \
+        num_workers=4 \
+        pred_field="pred_text"
+
+Model's predictions will be saved in `pred_field` of the `output_file`.
+
+.. _sentence_level_dataset_pipeline:
+
+Sentence-level Dataset Preparation Pipeline
+-------------------------------------------
+
+Here is the overall overview of the data labeling pipeline for sentence-level G2P model training:
+
+    .. image:: images/data_labeling_pipeline.png
+        :align: center
+        :alt: Data labeling pipeline for sentence-level G2P model training
+        :scale: 70%
+
+Here we describe the automatic phoneme-labeling process for generating augmented data. The figure below shows the phoneme-labeling steps to prepare data for sentence-level G2P model training. We first convert known unambiguous words to their phonetic pronunciations with dictionary lookups, e.g. CMU dictionary.
+Next, we automatically label heteronyms using an Aligner :cite:`g2p--badlani2022one`. More details on how to disambiguate heteronyms with a pretrained Aligner model could be found in `NeMo/tutorials/tts/Aligner_Inference_Examples.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Aligner_Inference_Examples.ipynb>`__ in `Google's Colab <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/tts/Aligner_Inference_Examples.ipynb>`_.
+Finally, we mask-out OOV words with a special masking token, “<unk>” in the figure below (note, we use `model.tokenizer_grapheme.unk_token="҂"` symbol during G2P model training.)
+Using this unknown token forces a G2P model to produce the same masking token as a phonetic representation during training. During inference, the model generates phoneme predictions for OOV words without emitting the masking token as long as this token is not included in the grapheme input.
+
+
+Requirements
+------------
+
+G2P requires the NeMo ASR collection to be installed. See :ref:`installation` and include the ``asr`` extra.
+
+
+References
+----------
+
+.. bibliography:: tts_all.bib
+    :style: plain
+    :labelprefix: g2p-
+    :keyprefix: g2p--

docs/source/tts/intro.rst

@@ -0,0 +1,24 @@
+Text-to-Speech (TTS)
+====================
+
+Text-to-Speech (TTS) synthesis refers to a system that converts textual inputs into natural human speech. The synthesized speech is expected to sound intelligible and natural. With the resurgence of deep neural networks, TTS research has achieved tremendous progress. NeMo implementation focuses on the state-of-the-art neural TTS where both **cascaded** and **end-to-end** (upcoming) systems are included,
+
+1. **Cascaded TTS** follows a three-stage process. *Text analysis stage* transliterates grapheme inputs into phonemes by either looking up in a canonical dictionary or using a grapheme-to-phoneme (G2P) conversion; *acoustic modeling stage* generates acoustic features from phoneme inputs or from a mixer of graphemes and phonemes. NeMo chooses mel-spectrograms to represent expressive acoustic features, so we would use the term in the context, mel-spectrogram generators or acoustic models, interchangeably; *vocoder stage* synthesizes waveform audios from acoustic features accordingly.
+2. **End-to-End TTS** alternatively integrates the above three stages as a single model so that it directly synthesizes audios from graphemes/phonemes inputs without any intermediate processes.
+
+We will illustrate details in the following sections.
+
+.. toctree::
+    :maxdepth: 2
+
+    models
+    datasets
+    checkpoints
+    configs
+    g2p
+    magpietts
+    magpietts-finetuning
+    magpietts-po
+    magpietts-longform
+
+.. include:: resources.rst

docs/source/tts/magpietts-finetuning.rst

@@ -0,0 +1,220 @@
+.. _magpie-tts-finetuning:
+
+======================
+Magpie-TTS Finetuning
+======================
+
+Finetuning a pretrained Magpie-TTS checkpoint lets you adapt the model to new voices or new languages without training from scratch. The pretrained model has already learned general speech patterns, prosody, and acoustic modeling, so finetuning requires far less data and compute than pretraining. This guide covers two common finetuning scenarios:
+
+- **Adding new speakers in an existing language** — adapt the model to speak in voices not seen during pretraining, using a small dataset of target-speaker audio.
+- **Adding a new language** — extend the model to synthesize speech in a language absent from the pretraining data, using a multilingual dataset configuration.
+
+For preference optimization (DPO/GRPO) on top of a finetuned checkpoint, see :doc:`Magpie-TTS Preference Optimization <magpietts-po>`.
+
+
+Prerequisites
+#############
+
+Before finetuning, you will need:
+
+- A pretrained Magpie-TTS checkpoint (``pretrained.ckpt`` or ``pretrained.nemo``). Public checkpoints (``https://huggingface.co/nvidia/magpie_tts_multilingual_357m``) are available on Hugging Face.
+- The audio codec model (``https://huggingface.co/nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps``), available on Hugging Face alongside the TTS checkpoint.
+- A prepared dataset. For faster finetuning audio codec tokens must be pre-extracted from your audio files. See the *Dataset Preparation* section below.
+- NeMo installed from source or with the local Dockerfile. See :ref:`installation` for installation instructions.
+
+
+Dataset Preparation
+-------------------
+
+Training uses ``MagpieTTSDataset`` with ``dataset_meta`` entries (see ``DatasetMeta`` in ``nemo/collections/tts/data/text_to_speech_dataset.py``). Each line in ``manifest_path`` file is one training example.
+
+**Optional cached codec codes.** If each line includes ``target_audio_codes_path`` and ``context_audio_codes_path`` (paths to saved tensors) and ``model.load_cached_codes_if_available=true``, the dataloader can skip on-the-fly codec encoding. If those keys are absent, the codec runs during training and loads waveform from ``audio_filepath`` and ``context_audio_filepath`` (slower but no separate extraction step).
+
+**Minimum fields** (paths relative to ``audio_dir`` / ``feature_dir`` unless you use absolute paths):
+
+.. code-block:: json
+
+    {
+      "audio_filepath": "relative/path/to/audio.wav",
+      "text": "transcript of the utterance",
+      "duration": 5.2,
+      "context_audio_filepath": "relative/path/to/context.wav",
+      "context_text": "transcript of the context audio",
+      "target_audio_codes_path": "/optional/path/to/target_codes.pt",
+      "context_audio_codes_path": "/optional/path/to/context_codes.pt"
+    }
+
+The ``context_audio_filepath`` is the reference audio used for voice cloning during training. It should come from the same speaker as ``audio_filepath``. A minimum context duration of about 3 seconds and a high speaker similarity (for example ≥ 0.6 with TitaNet) are recommended for best results.
+
+**Registering datasets in config.** For each named split in ``train_ds_meta`` and ``val_ds_meta``, set ``manifest_path``, ``audio_dir``, ``feature_dir``, ``sample_weight`` (training), and ``tokenizer_names``: a list of keys that exist under ``model.text_tokenizers`` in the config. The dataloader picks the tokenizer for each sample from that list (see ``DatasetMeta``).
+
+
+.. _magpie-tts-new-speaker:
+
+Adding New Speakers in an Existing Language
+###########################################
+
+This scenario adapts a pretrained checkpoint to new speakers in a language the model already supports (for example adding new English speakers to a checkpoint trained on English data). You are teaching new voice characteristics while keeping the same text tokenizer. Mixing in some public Magpie-TTS data can reduce regression; see the `Magpie-TTS dataset <https://huggingface.co/nvidia/magpie_tts_multilingual_357m#training-dataset>`_ on Hugging Face.
+
+Key training choices:
+
+- **Low learning rate** (``5e-6``): the pretrained model is already well-converged; a high LR can destroy learned representations.
+- **Disable alignment prior** (``alignment_loss_scale=0.0``, ``prior_scaling_factor=null``): the prior helps pretraining but can over-constrain finetuning.
+- **Tokenizer**: use ``tokenizer_names: [english_phoneme]`` (or the tokenizer that matches your transcripts) on each ``train_ds_meta`` / ``val_ds_meta`` entry.
+
+``magpietts.yaml`` trains with ``max_epochs`` and top-level ``batch_size``. Validation mixes all ``val_ds_meta`` entries in a single dataloader (joint validation metrics).
+
+.. code-block:: bash
+
+    python examples/tts/magpietts.py \
+        --config-path=examples/tts/conf/magpietts \
+        --config-name=magpietts \
+        +init_from_ptl_ckpt=/path/to/pretrained.ckpt \
+        exp_manager.exp_dir=/path/to/output \
+        +train_ds_meta.en_sft.manifest_path=/path/to/train.json \
+        +train_ds_meta.en_sft.audio_dir=/path/to/audio \
+        +train_ds_meta.en_sft.feature_dir=/path/to/features \
+        +train_ds_meta.en_sft.sample_weight=1.0 \
+        "+train_ds_meta.en_sft.tokenizer_names=[english_phoneme]" \
+        +val_ds_meta.en_val.manifest_path=/path/to/val.json \
+        +val_ds_meta.en_val.audio_dir=/path/to/audio \
+        +val_ds_meta.en_val.feature_dir=/path/to/audio \
+        +val_ds_meta.en_val.sample_weight=1.0 \
+        "+val_ds_meta.en_val.tokenizer_names=[english_phoneme]" \
+        model.codecmodel_path=nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps \
+        model.context_duration_min=5.0 \
+        model.context_duration_max=5.0 \
+        model.alignment_loss_scale=0.0 \
+        model.prior_scaling_factor=null \
+        model.optim.lr=5e-6 \
+        ~model.optim.sched \
+        model.load_cached_codes_if_available=true \
+        trainer.precision=32 \
+        trainer.devices=8 \
+        trainer.num_nodes=1 \
+        batch_size=16 \
+        max_epochs=500
+
+The ``+init_from_ptl_ckpt`` flag loads the pretrained checkpoint weights before training begins. The ``+`` prefix is required because this key is not present in the base config.
+
+``~model.optim.sched`` removes the learning rate schedule so the LR stays constant during finetuning.
+
+``trainer.precision=32`` is recommended for finetuning stability. Mixed precision (``bf16`` or ``16``) can cause loss instability on small datasets.
+
+
+.. _magpie-tts-new-language:
+
+Adding a New Language
+#####################
+
+This scenario extends the model to one or more languages not present in the pretraining data. Use the same ``magpietts`` config and combine multiple manifests with per-language ``sample_weight``.
+
+**Tokenizers**
+
+- Define each new tokenizer under ``model.text_tokenizers`` (for example an ``AutoTokenizer`` with ``google/byt5-small`` for scripts outside the IPA vocabulary).
+- **How it is applied:** each ``train_ds_meta`` / ``val_ds_meta`` entry lists ``tokenizer_names`` (keys under ``model.text_tokenizers``). The dataloader uses those names to select which tokenizer encodes each sample’s transcript (see ``DatasetMeta`` in ``nemo/collections/tts/data/text_to_speech_dataset.py``).
+
+**Per-language entries**
+
+Each language is a separate key under ``train_ds_meta`` / ``val_ds_meta`` with ``manifest_path``, ``audio_dir``, ``feature_dir``, ``sample_weight``, and ``tokenizer_names``.
+
+**Sample weights**
+
+Upsample low-resource languages with a higher ``sample_weight`` so they are not drowned out by high-resource languages.
+
+Align transcript format with the tokenizer you choose (IPA phonemes for ``english_phoneme`` / IPA-style tokenizers, raw text for byte-level models, and so on). Audio codes can be cached as in *Dataset Preparation*.
+
+.. code-block:: bash
+
+    python examples/tts/magpietts.py \
+        --config-name=magpietts \
+        +init_from_ptl_ckpt=/path/to/pretrained.ckpt \
+        exp_manager.exp_dir=/path/to/output \
+        +model.text_tokenizers.your_language_chartokenizer._target_=AutoTokenizer \
+        +model.text_tokenizers.your_language_chartokenizer.pretrained_model="google/byt5-small" \
+        +train_ds_meta.your_language.manifest_path=/path/to/your_lang_train.json \
+        +train_ds_meta.your_language.audio_dir=/path/to/your_lang_audio \
+        +train_ds_meta.your_language.feature_dir=/path/to/your_lang_audio \
+        +train_ds_meta.your_language.sample_weight=1.0 \
+        "+train_ds_meta.your_language.tokenizer_names=[your_language_chartokenizer]" \
+        +val_ds_meta.your_language_dev.manifest_path=/path/to/your_lang_val.json \
+        +val_ds_meta.your_language_dev.audio_dir=/path/to/your_lang_audio \
+        +val_ds_meta.your_language_dev.feature_dir=/path/to/your_lang_audio \
+        +val_ds_meta.your_language_dev.sample_weight=1.0 \
+        "+val_ds_meta.your_language_dev.tokenizer_names=[your_language_chartokenizer]" \
+        model.codecmodel_path=nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps \
+        model.context_duration_min=5.0 \
+        model.context_duration_max=5.0 \
+        model.alignment_loss_scale=0.0 \
+        model.prior_scaling_factor=null \
+        model.optim.lr=1e-5 \
+        ~model.optim.sched \
+        model.load_cached_codes_if_available=true \
+        trainer.precision=32 \
+        trainer.devices=8 \
+        trainer.num_nodes=1 \
+        max_epochs=500
+
+
+Mixing Multiple Languages
+--------------------------
+
+Add one ``train_ds_meta`` entry per language. Increase ``sample_weight`` for low-resource languages. You can mix public Magpie-TTS data with your own; see the `Magpie-TTS dataset <https://huggingface.co/nvidia/magpie_tts_multilingual_357m#training-dataset>`_ on Hugging Face.
+
+.. code-block:: bash
+
+        # High-resource languages — standard weight
+        +train_ds_meta.spanish.manifest_path=/path/to/spanish_train.json \
+        +train_ds_meta.spanish.audio_dir=/path/to/spanish_audio \
+        +train_ds_meta.spanish.feature_dir=/path/to/spanish_audio \
+        +train_ds_meta.spanish.sample_weight=1.0 \
+        "+train_ds_meta.spanish.tokenizer_names=[spanish_phoneme_or_chartokenizer]" \
+        +train_ds_meta.french.manifest_path=/path/to/french_train.json \
+        +train_ds_meta.french.audio_dir=/path/to/french_audio \
+        +train_ds_meta.french.feature_dir=/path/to/french_audio \
+        +train_ds_meta.french.sample_weight=1.0 \
+        "+train_ds_meta.french.tokenizer_names=[french_chartokenizer]" \
+        # Low-resource language — upsampled 10x
+        +train_ds_meta.low_resource_lang.manifest_path=/path/to/low_resource_train.json \
+        +train_ds_meta.low_resource_lang.audio_dir=/path/to/low_resource_audio \
+        +train_ds_meta.low_resource_lang.feature_dir=/path/to/low_resource_audio \
+        +train_ds_meta.low_resource_lang.sample_weight=5.0 \
+        "+train_ds_meta.low_resource_lang.tokenizer_names=[low_resource_chartokenizer]"
+
+With ``model.load_cached_codes_if_available=true``, precomputed ``target_audio_codes_path`` / ``context_audio_codes_path`` in the manifest avoid recomputing codec codes at train time.
+
+
+Preference Optimization After Finetuning
+#########################################
+
+After supervised finetuning, you can further improve quality with GRPO. For commands and hyperparameters, see :doc:`Magpie-TTS Preference Optimization <magpietts-po>` (the GRPO example uses ``--config-name=magpietts`` with ``+mode=onlinepo_train``).
+
+
+Key Hyperparameter Reference
+#############################
+
+.. list-table::
+   :widths: 35 25 40
+   :header-rows: 1
+
+   * - Parameter
+     - Typical Value
+     - Notes
+   * - ``model.optim.lr``
+     - ``5e-6`` (same-language speakers), ``1e-5`` (multilingual)
+     - Much lower than pretraining LR to preserve learned features
+   * - ``max_epochs``
+     - tens to hundreds
+     - Shorter runs for small datasets; monitor validation loss
+   * - ``model.alignment_loss_scale``
+     - ``0.0``
+     - Disable alignment prior during finetuning
+   * - ``model.prior_scaling_factor``
+     - ``null``
+     - Disable alignment prior during finetuning
+   * - ``trainer.precision``
+     - ``32``
+     - Recommended for finetuning stability
+   * - ``model.cfg_unconditional_prob``
+     - ``0.1``
+     - Classifier-free guidance dropout rate during training

docs/source/tts/magpietts-longform.rst

@@ -0,0 +1,258 @@
+.. _magpie-tts-longform:
+
+==============================
+Magpie-TTS Longform Inference
+==============================
+
+This document describes how longform (multi-sentence) text-to-speech inference works in Magpie-TTS.
+
+
+Overview
+########
+
+Magpie-TTS supports generating speech for long text inputs by processing them in smaller, sentence-level chunks while maintaining prosodic continuity across the entire utterance. This approach overcomes the context window limitations of the underlying transformer architecture.
+
+
+When Longform is Used
+#####################
+
+Longform inference is automatically triggered based on word count thresholds (approximately 20 seconds of audio):
+
+.. list-table:: Language Word Thresholds
+   :header-rows: 1
+   :widths: 30 30
+
+   * - Language
+     - Word Threshold
+   * - English
+     - 45 words
+   * - Spanish
+     - 73 words
+   * - French
+     - 69 words
+   * - German
+     - 50 words
+   * - Italian
+     - 53 words
+   * - Vietnamese
+     - 50 words
+   * - Japanese
+     - 50 words
+   * - Hindi
+     - 50 words
+
+.. note::
+
+   Longform is best supported for English. Mandarin currently falls back to standard inference.
+
+
+Algorithm
+#########
+
+The longform inference algorithm processes long text through the following steps:
+
+
+Step 1: Sentence Splitting
+--------------------------
+
+The input text is split into individual sentences using punctuation markers (``.``, ``?``, ``!``, ``...``). The splitting is intelligent and handles abbreviations like "Dr.", "Mr.", "a.m." by checking if the period is followed by a space.
+
+**Example:**
+
+::
+
+    Input:  "Dr. Smith arrived early. How are you today?"
+    Output: ["Dr. Smith arrived early.", "How are you today?"]
+
+
+Step 2: State Initialization
+----------------------------
+
+A ``ChunkState`` object is created to track information across sentence chunks:
+
+- **History text tokens**: Text from previous chunks for context
+- **History encoder context**: Encoder outputs that provide continuity
+- **Attention tracking**: Monitors which positions have been attended to
+
+
+Step 3: Iterative Chunk Processing
+----------------------------------
+
+For each sentence chunk, the following sub-steps are performed:
+
+1. **Context Preparation**: Prepend history text and encoder context from previous chunks to maintain prosodic continuity.
+
+2. **Attention Prior Application**: Apply a learned attention prior that guides the model to attend to the correct text positions, preventing repetition or skipping.
+
+3. **Autoregressive Generation**: Generate audio codes token-by-token using the transformer decoder with temperature sampling.
+
+4. **State Update**: Update the chunk state with:
+
+   - New history text (last N tokens)
+   - New encoder context
+   - Updated attention tracking
+
+5. **Code Collection**: Store the generated audio codes for this chunk.
+
+
+Step 4: Code Concatenation
+--------------------------
+
+After all chunks are processed, concatenate the audio codes from each chunk along the time dimension into a single sequence.
+
+
+Step 5: Audio Decoding
+----------------------
+
+Pass the concatenated codes through the neural audio codec decoder to produce the final waveform.
+
+
+Key Components
+--------------
+
+1. **Sentence Splitting** (``split_by_sentence``): Intelligently splits text on sentence boundaries while handling abbreviations (e.g., "Dr.", "Mr.").
+
+2. **Chunk State** (``ChunkState``): Maintains context across chunks:
+
+   - ``history_text``: Text tokens from previous chunks
+   - ``history_context_tensor``: Encoder outputs for continuity
+   - ``last_attended_timesteps``: Attention tracking for smooth transitions
+
+3. **Attention Prior**: Guides the model's attention to maintain proper alignment and prevent repetition/skipping.
+
+
+Usage
+#####
+
+
+Method 1: Using ``do_tts`` (Recommended for Simple Use Cases)
+-------------------------------------------------------------
+
+The ``do_tts`` method automatically detects whether longform inference is needed:
+
+.. code-block:: python
+
+    import torch
+    from nemo.collections.tts.models import MagpieTTSModel
+
+    # Load model
+    model = MagpieTTSModel.restore_from("path/to/magpietts.nemo")
+    model.eval()
+    model.cuda()
+
+    # Short text - uses standard inference automatically
+    short_audio, short_len = model.do_tts(
+        transcript="Hello, how are you?",
+        language="en",
+    )
+
+    # Long text - automatically switches to longform inference
+    long_text = """
+    The quick brown fox jumps over the lazy dog. This sentence contains every 
+    letter of the alphabet. Sphinx of black quartz, judge my vow. Pack my box 
+    with five dozen liquor jugs. How vexingly quick daft zebras jump. The five 
+    boxing wizards jump quickly. Jackdaws love my big sphinx of quartz.
+    """
+
+    long_audio, long_len = model.do_tts(
+        transcript=long_text,
+        language="en",
+        apply_TN=True,  # Apply text normalization
+        temperature=0.7,
+        topk=80,
+        use_cfg=True,
+        cfg_scale=2.5,
+    )
+
+    # Save audio
+    import soundfile as sf
+    sf.write("output.wav", long_audio[0].cpu().numpy(), 22050)
+
+
+Method 2: Using CLI (``magpietts_inference.py``)
+------------------------------------------------
+
+For batch inference from manifests:
+
+.. code-block:: bash
+
+    # Auto-detect longform based on text length (default)
+    python examples/tts/magpietts_inference.py \
+        --nemo_files /path/to/magpietts.nemo \
+        --datasets_json_path /path/to/evalset_config.json \
+        --out_dir /path/to/output \
+        --codecmodel_path /path/to/codec.nemo \
+        --longform_mode auto
+
+    # Force longform inference for all inputs
+    python examples/tts/magpietts_inference.py \
+        --nemo_files /path/to/magpietts.nemo \
+        --datasets_json_path /path/to/evalset_config.json \
+        --out_dir /path/to/output \
+        --codecmodel_path /path/to/codec.nemo \
+        --longform_mode always \
+        --longform_max_decoder_steps 50000
+
+**Longform CLI Options:**
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 15 60
+
+   * - Option
+     - Default
+     - Description
+   * - ``--longform_mode``
+     - ``auto``
+     - ``auto``: detect from text, ``always``: force longform, ``never``: disable
+
+
+Configuration Dataclasses
+#########################
+
+
+``ChunkedInferenceConfig``
+--------------------------
+
+Immutable tuning parameters (set in model):
+
+.. literalinclude:: ../../../nemo/collections/tts/models/magpietts.py
+   :language: python
+   :pyobject: ChunkedInferenceConfig
+
+
+``ChunkState``
+--------------
+
+Mutable state passed between chunk iterations:
+
+.. literalinclude:: ../../../nemo/collections/tts/models/magpietts.py
+   :language: python
+   :pyobject: ChunkState
+
+
+Best Practices
+##############
+
+1. **Use ``apply_TN=True``** for raw text to ensure proper normalization before synthesis.
+
+2. **Increase ``max_decoder_steps``** for very long texts (default 50000 is usually sufficient).
+
+3. **Use ``longform_mode="auto"``** (default) to let the system decide based on text length.
+
+4. **For non-English languages**, be aware that longform performance may vary. English is best supported.
+
+
+Limitations
+###########
+
+- **Mandarin (zh)**: Currently falls back to standard inference due to character-based tokenization complexities.
+- **Prosodic boundaries**: While the algorithm maintains continuity, natural paragraph breaks may not always be perfectly preserved in non-English languages.
+
+
+See Also
+########
+
+- :doc:`magpietts`: Main Magpie-TTS documentation
+- :doc:`magpietts-po`: Preference Optimization Guide
+

docs/source/tts/magpietts-po.rst

@@ -0,0 +1,284 @@
+.. _magpie-tts-po:
+
+=======================================
+Magpie-TTS Preference Optimization
+=======================================
+
+Preference optimization is a powerful technique for improving the quality of Magpie-TTS outputs by learning from ranked examples. Rather than relying solely on supervised learning with ground-truth audio, preference optimization teaches the model to distinguish between good and bad generations, allowing it to internalize quality metrics like intelligibility and speaker similarity directly into its generation process.
+
+Magpie-TTS supports two complementary approaches to preference optimization: offline alignment using Direct Preference Optimization (DPO) and online optimization using Group Relative Policy Optimization (GRPO). While DPO requires pre-generating preference data before training, GRPO generates candidates on the fly and is generally recommended for its simplicity and effectiveness.
+
+
+Offline Preference Alignment (DPO)
+##################################
+
+Direct Preference Optimization works by fine-tuning the model on pairs of chosen and rejected outputs. The training objective encourages the model to increase the likelihood of generating outputs similar to the chosen examples while decreasing the likelihood of rejected ones. This approach is particularly useful when you have access to human quality judgments or want fine-grained control over the preference data.
+
+The DPO pipeline consists of four distinct steps, each building on the output of the previous one.
+
+
+Step 1: Create Text-Context Pairs
+---------------------------------
+
+The first step is to assemble a collection of text-context pairs that will be used for preference data generation. A well-designed dataset should include a mix of challenging texts (such as tongue twisters, technical terms, and complex sentence structures) alongside regular transcripts. These texts are paired with various speaker contexts—either audio samples for voice cloning or text descriptions for style conditioning.
+
+The diversity of this dataset is crucial for robust preference optimization. Including challenging examples helps the model learn to handle edge cases, while regular transcripts ensure it maintains quality on typical inputs. You can also include examples with text contexts to improve style conditioning capabilities.
+
+.. code-block:: bash
+
+    python scripts/magpietts/dpo/create_text_contextpairs.py \
+        --challenging_texts /path/to/challenging_texts.txt \
+        --regular_texts_for_audiocontext /path/to/regular_texts_for_audiocontext.txt \
+        --regular_texts_for_textcontext /path/to/regular_texts_for_textcontext.txt \
+        --audio_contexts /path/to/audio_context_list.json \
+        --text_contexts /path/to/text_context_list.txt \
+        --output_manifest /path/to/text_context_pairs.json \
+        --nsamples_perpair 6
+
+The ``nsamples_perpair`` parameter specifies how many audio samples will be generated for each text-context pair in the next step. Setting this to 6 provides enough variety to create meaningful preference pairs while keeping computation manageable. The output manifest serves as input for the generation step.
+
+
+Step 2: Generate Audio Samples
+------------------------------
+
+With the text-context pairs prepared, the next step is to generate multiple audio samples for each pair using a base Magpie-TTS checkpoint. The generation process also computes quality metrics—Character Error Rate (CER) and Speaker Similarity (SSIM)—for each output, which will be used to create preference pairs.
+
+This step can be parallelized across multiple GPUs and nodes to speed up generation. Each generated audio file is accompanied by a JSON file containing the computed metrics.
+
+.. code-block:: bash
+
+    python examples/tts/magpietts.py \
+        --config-name=magpietts_po_inference \
+        mode=test \
+        batch_size=64 \
+        +init_from_ptl_ckpt=/path/to/magpie_checkpoint \
+        exp_manager.exp_dir=/path/to/po_experiment \
+        +test_ds_meta.textcontextpairs.manifest_path=/path/to/text_context_pairs.json \
+        +test_ds_meta.textcontextpairs.audio_dir="/" \
+        +test_ds_meta.textcontextpairs.feature_dir="/" \
+        model.codecmodel_path=/path/to/codec_model.nemo \
+        model.prior_scaling_factor=null \
+        model.load_cached_codes_if_available=false
+
+.. note::
+
+    The manifest contains absolute audio paths, so ``audio_dir`` is set to ``"/"``. Adjust the model configuration parameters to match your base checkpoint architecture.
+
+
+Step 3: Create Preference Pairs
+-------------------------------
+
+Once audio samples are generated, you need to create chosen-rejected pairs based on the computed metrics. The script analyzes the CER and SSIM scores for each group of samples and selects the best and worst performers to form preference pairs.
+
+.. code-block:: bash
+
+    python scripts/magpietts/dpo/create_preference_pairs.py \
+        --input_manifest /path/to/text_context_pairs.json \
+        --generated_audio_dir /path/to/po_experiment/MagpieTTS-PO-Infer/version_0/audio \
+        --group_size 6 \
+        --cer_threshold 0.01 \
+        --val_size 256
+
+The ``cer_threshold`` parameter filters out pairs where even the chosen example has poor intelligibility (CER > 0.01). This ensures the model learns from genuinely good examples rather than just "less bad" ones. The script outputs train and validation manifests in the ``manifests/`` subdirectory.
+
+
+Step 4: DPO Fine-tuning
+-----------------------
+
+The final step is fine-tuning the base model on the preference pairs using the DPO loss. This teaches the model to prefer generating outputs similar to the chosen examples over the rejected ones.
+
+.. code-block:: bash
+
+    python examples/tts/magpietts.py \
+        batch_size=4 \
+        +init_from_ptl_ckpt=/path/to/magpie_checkpoint \
+        +mode="dpo_train" \
+        max_epochs=10 \
+        exp_manager.exp_dir=/path/to/dpo_experiment \
+        exp_manager.checkpoint_callback_params.always_save_nemo=false \
+        model.train_ds.datasets._target_="nemo.collections.tts.data.text_to_speech_dataset.MagpieTTSDatasetDPO" \
+        model.validation_ds.datasets._target_="nemo.collections.tts.data.text_to_speech_dataset.MagpieTTSDatasetDPO" \
+        +train_ds_meta.dpopreftrain.manifest_path="/path/to/manifests/" \
+        +train_ds_meta.dpopreftrain.audio_dir="/" \
+        +train_ds_meta.dpopreftrain.feature_dir="/" \
+        +val_ds_meta.dpoprefval.manifest_path="/path/to/manifests/dpo_val_manifest.json" \
+        +val_ds_meta.dpoprefval.audio_dir="/" \
+        +val_ds_meta.dpoprefval.feature_dir="/" \
+        +model.dpo_beta=0.01 \
+        +model.dpo_sft_loss_weight=0.0 \
+        model.codecmodel_path=/path/to/codec_model.nemo \
+        model.alignment_loss_scale=0.001 \
+        model.prior_scaling_factor=null \
+        trainer.val_check_interval=200 \
+        trainer.log_every_n_steps=10 \
+        model.optim.lr=2e-7 \
+        ~model.optim.sched
+
+Key parameters for DPO training include ``dpo_beta``, which controls the strength of the preference signal, and a low learning rate (2e-7) to ensure stable fine-tuning.
+
+
+Online Preference Optimization (GRPO)
+#####################################
+
+Group Relative Policy Optimization offers a more streamlined approach that eliminates the need for pre-generating preference data. Instead, GRPO generates multiple candidate outputs for each training example on the fly, computes reward signals based on quality metrics, and optimizes the model to maximize these rewards through policy gradient methods.
+
+GRPO is generally recommended over DPO for several reasons. It continuously adapts to the model's current capabilities rather than relying on static preference data. It requires less setup and storage since there's no need to pre-generate and store audio samples. Additionally, it can optimize for multiple reward signals simultaneously, including CER, SSIM, and PESQ.
+
+
+Setting Up GRPO Training
+------------------------
+
+The GRPO training process starts with preparing text-context pairs, similar to DPO but without the need for multiple samples per pair:
+
+.. code-block:: bash
+
+    python scripts/magpietts/dpo/create_text_contextpairs.py \
+        --challenging_texts /path/to/challenging_texts.txt \
+        --regular_texts_for_audiocontext /path/to/regular_texts_for_audiocontext.txt \
+        --regular_texts_for_textcontext /path/to/regular_texts_for_textcontext.txt \
+        --audio_contexts /path/to/audio_context_list.json \
+        --text_contexts /path/to/text_context_list.txt \
+        --output_manifest /path/to/text_context_pairs_grpo.json \
+        --nsamples_perpair 1
+
+Note that ``nsamples_perpair`` is set to 1 since GRPO generates candidates during training.
+
+
+GRPO Training Configuration
+---------------------------
+
+GRPO training requires careful configuration of several hyperparameters. The following table summarizes the key parameters:
+
+.. list-table:: GRPO Hyperparameters
+   :header-rows: 1
+   :widths: 30 15 55
+
+   * - Parameter
+     - Default
+     - Description
+   * - ``num_generations_per_item``
+     - 12
+     - Number of candidate outputs generated per training example
+   * - ``reference_free``
+     - true
+     - If true, skips KL divergence term and optimizes rewards directly
+   * - ``grpo_beta``
+     - 0.0
+     - Coefficient for KL loss (only used when reference_free=false)
+   * - ``cer_reward_weight``
+     - 0.33
+     - Weight of Character Error Rate in the reward function
+   * - ``ssim_reward_weight``
+     - 0.33
+     - Weight of Speaker Similarity in the reward function
+   * - ``pesq_reward_weight``
+     - 0.33
+     - Weight of PESQ score in the reward function
+   * - ``use_pesq``
+     - true
+     - Whether to include PESQ in the reward computation
+   * - ``reward_asr_model``
+     - (none)
+     - ASR model for CER computation; set to ``whisper`` for multilingual
+   * - ``inference_temperature``
+     - 0.8
+     - Sampling temperature for candidate generation
+   * - ``inference_topk``
+     - 2016
+     - Top-k sampling parameter (2016 effectively disables it)
+   * - ``loss_type``
+     - "grpo"
+     - Loss function variant; can be "grpo" or "dr_grpo"
+   * - ``scale_rewards``
+     - true
+     - Whether to normalize advantages by standard deviation
+
+
+GRPO Training Command
+---------------------
+
+The following command demonstrates a complete GRPO training setup for multilingual models:
+
+.. code-block:: bash
+
+    python examples/tts/magpietts.py \
+        --config-name=magpietts \
+        batch_size=2 \
+        +init_from_ptl_ckpt=/path/to/magpie_checkpoint \
+        model.codecmodel_path=/path/to/codec_model.nemo \
+        +mode="onlinepo_train" \
+        max_epochs=20 \
+        exp_manager.exp_dir=/path/to/grpo_experiment \
+        +exp_manager.version=0 \
+        exp_manager.checkpoint_callback_params.always_save_nemo=false \
+        +train_ds_meta.dpopreftrain.manifest_path=/path/to/train_manifest.json \
+        +train_ds_meta.dpopreftrain.audio_dir="/" \
+        +train_ds_meta.dpopreftrain.feature_dir="/" \
+        +val_ds_meta.dpoprefval.manifest_path=/path/to/val_manifest.json \
+        +val_ds_meta.dpoprefval.audio_dir="/" \
+        +val_ds_meta.dpoprefval.feature_dir="/" \
+        +model.grpo_beta=0.0 \
+        +model.num_generations_per_item=12 \
+        +model.reference_free=true \
+        +model.inference_cfg_prob=0.5 \
+        +model.inference_cfg_scale=2.5 \
+        +model.cer_reward_weight=0.45 \
+        +model.ssim_reward_weight=0.45 \
+        +model.pesq_reward_weight=0.1 \
+        +model.use_pesq=true \
+        +model.reward_asr_model="whisper" \
+        model.cfg_unconditional_prob=0.0 \
+        +model.inference_topk=2016 \
+        +model.inference_temperature=0.7 \
+        +model.use_kv_cache_during_online_po=true \
+        +model.loss_type="grpo" \
+        +model.max_decoder_steps=430 \
+        model.decoder.p_dropout=0.0 \
+        model.encoder.p_dropout=0.0 \
+        model.alignment_loss_scale=0.0 \
+        model.prior_scaling_factor=null \
+        ~trainer.check_val_every_n_epoch \
+        +trainer.val_check_interval=50 \
+        trainer.log_every_n_steps=10 \
+        model.optim.lr=1e-7 \
+        ~model.optim.sched \
+        exp_manager.checkpoint_callback_params.monitor="val_cer_gt" \
+        exp_manager.checkpoint_callback_params.mode="min" \
+        trainer.precision=32 \
+        +trainer.gradient_clip_val=2.5
+
+
+Important GRPO Training Considerations
+--------------------------------------
+
+Several configuration choices are critical for stable GRPO training:
+
+**Disable Dropout**: Set ``p_dropout=0.0`` for all modules (encoder, decoder). This is essential when not using reference-free mode, as dropout causes the KL divergence loss to become unstable.
+
+**Disable Attention Priors and CTC Loss**: Set ``alignment_loss_scale=0.0`` and ``prior_scaling_factor=null``. These training signals can interfere with the preference optimization objective.
+
+**Use Small Batch Size**: Since GRPO generates ``num_generations_per_item`` samples for each batch item, the effective batch size becomes ``batch_size * num_generations_per_item``. A batch size of 2 with 12 generations per item results in 24 forward passes per step.
+
+**Frequent Validation**: GRPO steps take longer than standard training steps due to the generation overhead. Configure more frequent validation with ``val_check_interval=50`` to monitor progress.
+
+**Low Learning Rate**: Use a learning rate around 1e-7 to ensure stable optimization. The preference signal is noisy, and aggressive updates can destabilize training.
+
+**Model-Specific Overrides**: Ensure your GRPO configuration matches the base model architecture, including attention heads, number of layers, local transformer settings, and tokenizer configuration.
+
+
+Advanced: Local Transformer Optimization
+----------------------------------------
+
+For models with a Local Transformer, GRPO can optimize both with and without the LT by setting ``use_local_transformer_prob`` between 0 and 1. This trains the model to produce high-quality outputs regardless of whether the Local Transformer is used during inference, providing flexibility in the speed-quality trade-off at deployment time.
+
+.. code-block:: bash
+
+    +model.use_local_transformer_prob=0.5  # 50% of generations use LT
+
+
+See Also
+########
+
+- :doc:`magpietts`: Main Magpie-TTS documentation
+- `Preference Optimization Source Code <https://github.com/NVIDIA-NeMo/NeMo/blob/main/nemo/collections/tts/models/magpietts_preference_optimization.py>`__
+

docs/source/tts/magpietts.rst

@@ -0,0 +1,167 @@
+.. _magpie-tts:
+
+==========
+Magpie-TTS
+==========
+
+Large language models have revolutionized text-to-speech synthesis, enabling the generation of remarkably natural and expressive speech. However, these models often suffer from a critical limitation: hallucinations. Users frequently encounter issues such as repeated words, missing phrases, or speech that drifts out of alignment with the input text. These artifacts significantly degrade the user experience and limit the deployment of LLM-based TTS systems in production environments.
+
+Magpie-TTS [1]_ addresses these challenges head-on by introducing monotonic alignment techniques that ensure robust, hallucination-free speech synthesis. Developed by NVIDIA, the model combines the expressiveness of modern neural TTS with the reliability required for real-world applications. At its core, Magpie-TTS uses CTC (Connectionist Temporal Classification) loss and attention priors to enforce monotonic cross-attention between text and audio, effectively preventing the model from skipping, repeating, or misaligning content during generation.
+
+The model follows an encoder-decoder transformer architecture that operates on discrete audio tokens produced by a neural audio codec. This design choice enables high-quality waveform reconstruction while maintaining the flexibility of sequence-to-sequence modeling. Magpie-TTS supports voice cloning through audio context conditioning, allowing users to synthesize speech in any voice given just a few seconds of reference audio.
+
+
+Model Architecture
+##################
+
+Magpie-TTS processes text through a transformer encoder that captures the linguistic structure and phonetic content of the input. The encoder uses a stack of self-attention layers with causal masking, producing contextual representations that guide the subsequent audio generation. For phoneme-based models, the text is first converted using an IPA tokenizer with grapheme-to-phoneme conversion, while character-level models can operate directly on raw text using byte-level tokenization.
+
+The decoder autoregressively generates discrete audio tokens by attending to both the encoded text and any provided audio context. The cross-attention mechanism between the decoder and encoder is where the attention prior comes into play, biasing the model toward monotonic alignment patterns that match the natural left-to-right progression of speech. This architectural choice is what distinguishes Magpie-TTS from conventional LLM-based approaches and eliminates most hallucination artifacts.
+
+For voice cloning, Magpie-TTS employs a dedicated context encoder that processes the reference audio. The context encoding is injected into the decoder, allowing the model to capture speaker characteristics such as pitch, timbre, and speaking style. The model also supports text-based context conditioning, enabling control over speaking style through textual descriptions rather than audio examples.
+
+
+Frame Stacking and Local Transformer
+------------------------------------
+
+A key innovation in Magpie-TTS is the two-stage decoding architecture [4]_ that dramatically accelerates inference. The base decoder can be configured to process multiple consecutive audio frames in a single forward pass through a technique called frame stacking. Rather than generating one frame at a time, the base decoder produces a single latent representation for each group (or "stack") of frames, and a lightweight Local Transformer then expands this into individual frame-level predictions.
+
+This two-stage approach delivers significant speed improvements for two reasons. First, the Local Transformer uses far fewer parameters than the base decoder, making each of its forward passes much faster. Second, the Local Transformer only attends to the current frame stack and its corresponding latent, rather than the entire audio sequence, resulting in much shorter attention contexts. The ``frame_stacking_factor`` parameter controls how many frames are grouped together, with values up to 4 tested successfully. Setting this to 1 (the default) disables frame stacking entirely.
+
+The Local Transformer handles the prediction of tokens across multiple codebooks within each time frame. The audio codec uses multiple codebooks to represent audio at different levels of detail, and the Local Transformer can operate in either autoregressive mode, where codebooks are predicted sequentially, or MaskGit mode, which uses iterative parallel decoding for even faster inference. When frame stacking is enabled, using the Local Transformer is typically necessary to maintain high synthesis quality.
+
+
+Model Configurations
+--------------------
+
+Magpie-TTS supports two model configurations to suit different use cases:
+
+- **decoder_context_tts**: The standard configuration where text goes to the encoder and both context audio and target audio are processed by the decoder
+- **decoder_ce**: Adds a learned context embedding network between the context encoder and decoder for more flexible voice conditioning
+
+
+Key Features
+############
+
+The attention prior mechanism is central to Magpie-TTS's robustness. During training, the model learns to produce cross-attention patterns that follow a monotonic diagonal, matching the natural progression of speech from left to right through the text. This is achieved through a combination of CTC-based alignment loss and soft attention priors that guide the attention weights without being overly restrictive. During inference, the attention prior can be applied dynamically, further reducing the risk of hallucinations on out-of-distribution inputs.
+
+Classifier-free guidance (CFG) [2]_ provides another dimension of control over the generation process. By training the model to occasionally drop conditioning information, Magpie-TTS learns to generate both conditioned and unconditioned outputs. At inference time, the difference between these outputs can be amplified to increase the fidelity and consistency of the generated speech. Users can adjust the CFG scale to balance between adherence to the conditioning and diversity in the output.
+
+Magpie-TTS is designed for multilingual synthesis from the ground up. The model uses a flexible tokenization scheme that can handle multiple languages, with support for language-specific phoneme tokenizers as well as universal byte-level tokenization. This enables training on diverse multilingual datasets and synthesizing speech in languages beyond English. The 357M parameter multilingual checkpoint available on Hugging Face demonstrates these capabilities across a range of languages.
+
+Long-form speech generation (beta) is supported by Magpie-TTS for English language only. This feature is currently in beta. The long input text should have punctuation, especially sentence boundaries. See the :doc:`Longform Inference Guide <magpietts-longform>` for detailed usage instructions.
+
+Training
+########
+
+Training Magpie-TTS requires preparing your dataset in NeMo's manifest format, where each entry specifies the audio file path, transcript, and optional speaker information. The model expects pre-computed audio codec tokens, which can be generated using the accompanying audio codec model. For voice cloning, you should also include context audio segments that the model will learn to condition on.
+
+The training script uses Hydra for configuration management, making it easy to customize model architecture, training hyperparameters, and dataset settings. The default configuration provides a solid starting point with a 6-layer encoder and 12-layer decoder, though these can be scaled up for higher quality or down for faster iteration.
+
+.. code-block:: bash
+
+    python examples/tts/magpietts.py \
+        --config-name=magpietts \
+        max_epochs=100 \
+        batch_size=16 \
+        model.codecmodel_path=/path/to/audio_codec.nemo \
+        train_ds_meta.dataset_name.manifest_path=/path/to/train_manifest.json \
+        train_ds_meta.dataset_name.audio_dir=/path/to/audio \
+        val_ds_meta.dataset_name.manifest_path=/path/to/val_manifest.json \
+        val_ds_meta.dataset_name.audio_dir=/path/to/audio \
+        exp_manager.exp_dir=/path/to/experiments
+
+
+Preference Optimization
+-----------------------
+
+Beyond standard supervised training, Magpie-TTS supports preference optimization techniques that can further refine output quality by learning from ranked examples. Two approaches are available: offline preference alignment using Direct Preference Optimization (DPO) and online optimization using Group Relative Policy Optimization (GRPO) (recommended).
+
+**Offline Preference Alignment (DPO)** follows a four-step pipeline. First, you create a set of text-context pairs that will be used for preference data generation—typically a mix of challenging texts (tongue twisters, complex sentences) and regular transcripts paired with various speaker contexts. Second, you generate multiple audio samples for each pair using a base checkpoint, computing quality metrics like Character Error Rate (CER) and Speaker Similarity (SSIM) for each generation. Third, you create chosen-rejected pairs by selecting the best and worst outputs based on these metrics. Finally, you fine-tune the model on these preference pairs using the DPO loss.
+
+.. code-block:: bash
+
+    # Step 1: Create text-context pairs
+    python scripts/magpietts/dpo/create_text_contextpairs.py \
+        --challenging_texts /path/to/challenging_texts.txt \
+        --audio_contexts /path/to/audio_context_list.json \
+        --output_manifest /path/to/text_context_pairs.json \
+        --nsamples_perpair 6
+
+    # Step 4: DPO fine-tuning
+    python examples/tts/magpietts.py \
+        +init_from_ptl_ckpt=/path/to/base_checkpoint \
+        +mode="dpo_train" \
+        +model.dpo_beta=0.01 \
+        model.optim.lr=2e-7
+
+**Group Relative Policy Optimization (GRPO)** [3]_ offers a simpler alternative that generates preference data on-the-fly during training. Instead of pre-generating audio samples, GRPO produces multiple candidates for each training example and computes rewards based on CER, SSIM, and optionally PESQ (Perceptual Evaluation of Speech Quality). The model then learns to maximize these rewards through policy gradient optimization.
+
+GRPO is particularly effective because it continuously adapts to the model's current capabilities rather than relying on static preference data. Key hyperparameters include ``num_generations_per_item`` (typically 12) which controls how many candidates are generated per example, and the reward weights that balance different quality metrics. For multilingual models, setting ``reward_asr_model="whisper"`` enables proper CER computation across languages.
+
+.. code-block:: bash
+
+    python examples/tts/magpietts.py \
+        +init_from_ptl_ckpt=/path/to/base_checkpoint \
+        +mode="onlinepo_train" \
+        batch_size=2 \
+        +model.num_generations_per_item=12 \
+        +model.reference_free=true \
+        +model.cer_reward_weight=0.33 \
+        +model.ssim_reward_weight=0.33 \
+        +model.pesq_reward_weight=0.33 \
+        +model.use_pesq=true \
+        +model.loss_type="grpo" \
+        model.decoder.p_dropout=0.0 \
+        model.optim.lr=1e-7
+
+.. note::
+
+    When using GRPO, it's important to disable dropout in all modules (``p_dropout=0.0``) to ensure stable KL divergence computation. Also disable attention priors and CTC loss during GRPO training, as these can interfere with the preference optimization process.
+
+For comprehensive documentation on preference optimization, including step-by-step instructions for both DPO and GRPO, see the :doc:`Preference Optimization Guide <magpietts-po>`.
+
+Inference
+#########
+
+Running inference with Magpie-TTS involves loading the trained model and providing the text to synthesize along with optional context audio for voice cloning. The inference script supports batched generation for efficient processing of multiple utterances and includes built-in evaluation metrics for assessing output quality.
+
+Several parameters control the generation behavior. The temperature setting affects the randomness of token sampling, with lower values producing more deterministic output and higher values introducing more variation. Top-k sampling restricts the token selection to the k most likely candidates, helping to avoid low-probability artifacts. When using classifier-free guidance, the CFG scale parameter controls how strongly the conditioning influences the output.
+
+.. code-block:: bash
+
+    python examples/tts/magpietts_inference.py \
+        --nemo_files /path/to/magpietts_model.nemo \
+        --codecmodel_path /path/to/audio_codec.nemo \
+        --datasets your_evaluation_set \
+        --out_dir /path/to/output \
+        --temperature 0.6 \
+        --topk 80 \
+        --use_cfg \
+        --cfg_scale 2.5
+
+For production deployments, enabling the attention prior during inference adds an extra layer of robustness. The prior gently biases the cross-attention toward monotonic patterns, catching any potential alignment drift before it manifests as audible artifacts. This is especially valuable when processing text that differs from the training distribution.
+
+When using models trained with frame stacking, you can enable the Local Transformer during inference with ``--use_local_transformer``. The MaskGit decoding mode can be activated for faster inference at a slight quality trade-off, with ``--maskgit_n_steps`` controlling the number of refinement iterations.
+
+To enable Long-form speech generation (beta) set ``--longform_mode`` to ``auto or always``; this will either automatically detect the long form text or always use the long form inference pipeline. For comprehensive documentation on longform inference, see the :doc:`Longform Inference Guide <magpietts-longform>`.
+
+Resources
+#########
+
+To get started with Magpie-TTS, you can download the pretrained multilingual checkpoint from `Hugging Face <https://huggingface.co/nvidia/magpie_tts_multilingual_357m>`__ and try it out in the interactive `demo space <https://huggingface.co/spaces/nvidia/magpie_tts_multilingual_demo>`__. For deeper technical details, refer to [1]_, [2]_, [3]_, and [4]_. The complete source code is available in the `NeMo GitHub repository <https://github.com/NVIDIA/NeMo/tree/main/nemo/collections/tts/models/magpietts.py>`__.
+
+Additional documentation on advanced features can be found in the repository:
+
+- `Frame Stacking Guide <https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/tts/README_frame_stacking.md>`__: Detailed explanation of the two-stage decoding architecture
+
+References
+##########
+
+.. [1] `Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment <https://arxiv.org/pdf/2406.17957>`__
+
+.. [2] `Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance <https://aclanthology.org/2025.emnlp-main.1076.pdf>`__
+
+.. [3] `ALIGN2SPEAK: IMPROVING TTS FOR LOW RESOURCE LANGUAGES VIA ASR-GUIDED ONLINE PREFERENCE OPTIMIZATION <https://arxiv.org/pdf/2509.21718?>`__
+
+.. [4] `FRAME-STACKED LOCAL TRANSFORMERS FOR EFFICIENT MULTI-CODEBOOK SPEECH GENERATION <https://arxiv.org/pdf/2509.19592>`__

docs/source/tts/models.rst

@@ -0,0 +1,83 @@
+Models
+=======
+This section provides a brief overview of TTS models that NeMo's TTS collection currently supports.
+
+* **Model Recipes** can be accessed through `examples/tts/*.py <https://github.com/NVIDIA/NeMo/tree/stable/examples/tts>`_.
+* **Configuration Files** can be found in the directory of `examples/tts/conf/ <https://github.com/NVIDIA/NeMo/tree/stable/examples/tts/conf>`_. For detailed information about TTS configuration files and how they
+  should be structured, please refer to the section :doc:`./configs`.
+* **Pretrained Model Checkpoints** are available for any users for immediately synthesizing speech or fine-tuning models on
+  your custom datasets. Please follow the section :doc:`./checkpoints` for instructions on how to use those pretrained models.
+
+
+Mel-Spectrogram Generators
+--------------------------
+
+.. _FastPitch_model:
+
+FastPitch
+~~~~~~~~~
+FastPitch is a fully-parallel text-to-speech synthesis model based on FastSpeech, conditioned on fundamental frequency contours. The model predicts pitch contours during inference. By altering these predictions, the generated speech can be more expressive, better match the semantic of the utterance, and in the end more engaging to the listener. Uniformly increasing or decreasing pitch with FastPitch generates speech that resembles the voluntary modulation of voice. Conditioning on frequency contours improves the overall quality of synthesized speech, making it comparable to the state of the art. It does not introduce an overhead, and FastPitch retains the favorable, fully-parallel Transformers architecture, with over 900x real-time factor for mel-spectrogram synthesis of a typical utterance. The architecture of FastPitch is shown below. It is based on FastSpeech and consists of two feed-forward Transformer (FFTr) stacks. The first FFTr operates in the resolution of input tokens, and the other one in the resolution of the output frames. Please refer to :cite:`tts-models-lancucki2021fastpitch` for details.
+
+    .. image:: images/fastpitch_model.png
+        :align: center
+        :alt: fastpitch model
+        :scale: 30%
+
+SSL FastPitch
+~~~~~~~~~~~~~
+This **experimental** version of FastPitch takes in content and speaker embeddings generated by an SSL Disentangler and generates mel-spectrograms, with the goal that voice characteristics are taken from the speaker embedding while the content of speech is determined by the content embedding. Voice conversion can be done using this model by swapping the speaker embedding input to that of a target speaker, while keeping the content embedding the same. More details to come.
+
+
+End-to-End LLM-based TTS
+------------------------
+
+MagpieTTS
+~~~~~~~~~
+MagpieTTS is an encoder-decoder transformer TTS model that operates on discrete audio tokens from a neural audio codec. It uses monotonic alignment (CTC loss and attention priors) to reduce hallucinations and supports voice cloning via audio or text context conditioning. For architecture, training, inference, and preference optimization (DPO/GRPO), see :doc:`Magpie-TTS documentation <magpietts>`. To adapt a pretrained checkpoint to new speakers or new languages, see :doc:`Magpie-TTS Finetuning <magpietts-finetuning>`.
+
+
+Vocoders
+--------
+
+HiFiGAN
+~~~~~~~
+HiFi-GAN focuses on designing a vocoder model that efficiently synthesizes raw waveform audios from the intermediate mel-spectrograms. It consists of one generator and two discriminators (multi-scale and multi-period). The generator and discriminators are trained adversarially with two additional loses for improving training stability and model performance. The generator is a fully convolutional neural network which takes a mel-spectrogram as input and upsamples it through transposed convolutions until the length of the output sequence matches the temporal resolution of raw waveforms. Every transposed convolution is followed by a multi-receptive field fusion (MRF) module. The architecture of the generator is shown below (left). Multi-period discriminator (MPD) is a mixer of sub-discriminators, each of which only accepts equally spaced samples of an input audio. The sub-discriminators are designed to capture different implicit structures from each other by looking at different parts of an input audio. While MPD only accepts disjoint samples, multi-scale discriminator (MSD) is added to consecutively evaluate the audio sequence. MSD is a mixer of 3 sub-discriminators operating on different input scales (raw audio, x2 average-pooled audio, and x4 average-pooled audio). HiFi-GAN could achieve both higher computational efficiency and sample quality than the best publicly available auto-regressive or flow-based models, such as WaveNet and WaveGlow. Please refer to :cite:`tts-models-kong2020hifi` for details.
+
+    .. figure:: images/hifigan_g_model.png
+        :alt: hifigan_g model
+        :scale: 30%
+
+(a) Generator
+
+    .. figure:: images/hifigan_d_model.png
+        :alt: hifigan_d model
+        :scale: 30%
+
+(b) Discriminators
+
+
+
+Codecs
+------
+
+Audio Codec
+~~~~~~~~~~~
+
+The NeMo Audio Codec model is a non-autoregressive convolutional encoder-quantizer-decoder model for coding or tokenization of raw audio signals or mel-spectrogram features.
+The NeMo Audio Codec model supports residual vector quantizer (RVQ) :cite:`tts-models-zeghidour2022soundstream` and finite scalar quantizer (FSQ) :cite:`tts-models-mentzer2023finite` for quantization of the encoder output.
+This model is trained end-to-end using generative loss, discriminative loss, and reconstruction loss, similar to other neural audio codecs such as SoundStream :cite:`tts-models-zeghidour2022soundstream` and EnCodec :cite:`tts-models-defossez2022encodec`.
+For further information refer to the ``Audio Codec Training`` tutorial in the TTS tutorial section.
+
+    .. image:: images/audiocodec_model.png
+        :align: center
+        :alt: audiocodec model
+        :scale: 35%
+
+
+References
+----------
+
+.. bibliography:: tts_all.bib
+    :style: plain
+    :labelprefix: TTS-MODELS
+    :keyprefix: tts-models-

docs/source/tts/resources.rst

@@ -0,0 +1,12 @@
+Resources and Documentation
+===========================
+
+Hands-on TTS tutorial notebooks can be found under `the TTS tutorials folder <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts/>`_. If you are a beginner to NeMo, consider trying out the tutorials of `NeMo Primer <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/00_NeMo_Primer.ipynb>`_ and `NeMo Model <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/01_NeMo_Models.ipynb>`_. If you are also a beginner to TTS, consider trying out the `NeMo TTS Primer Tutorial <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts/NeMo_TTS_Primer.ipynb>`_. These tutorials can be run on Google Colab by specifying the link to the notebooks' GitHub pages on Colab.
+
+If you are looking for information about a particular TTS model, or would like to find out more about the model architectures available in the directory of `nemo.collections.tts <https://github.com/NVIDIA/NeMo/tree/stable/nemo/collections/tts/models>`_, refer to the :doc:`Models <./models>` section.
+
+NeMo includes preprocessing scripts for several common TTS datasets. The :doc:`Data Preprocessing <./datasets>` section contains instructions on how to run those scripts. You can also creating your own NeMo-compatible dataset preprocessing script by following the guidance.
+
+Information about how to load model checkpoints (either local files or pretrained ones from NGC), as well as a list of the checkpoints available on NGC are located on the :doc:`Checkpoints <./checkpoints>` section.
+
+Documentation regarding the configuration files specific to the NeMo TTS models can be found on the :doc:`Configuration Files <./configs>` section.

docs/source/tts/tts_all.bib

@@ -0,0 +1,136 @@
+@inproceedings{shen2018natural,
+  title={Natural tts synthesis by conditioning wavenet on mel spectrogram predictions},
+  author={Shen, Jonathan and Pang, Ruoming and Weiss, Ron J and Schuster, Mike and Jaitly, Navdeep and Yang, Zongheng and Chen, Zhifeng and Zhang, Yu and Wang, Yuxuan and Skerrv-Ryan, Rj and others},
+  booktitle={2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)},
+  pages={4779--4783},
+  year={2018},
+  organization={IEEE}
+}
+
+@inproceedings{lancucki2021fastpitch,
+  title={Fastpitch: Parallel text-to-speech with pitch prediction},
+  author={{\L}a{\'n}cucki, Adrian},
+  booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  pages={6588--6592},
+  year={2021},
+  organization={IEEE}
+}
+
+@inproceedings{tatanov2022mixer,
+  title={{Mixer-TTS}: non-autoregressive, fast and compact text-to-speech model conditioned on language model embeddings},
+  author={Tatanov, Oktai and Beliaev, Stanislav and Ginsburg, Boris},
+  booktitle={ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  pages={7482--7486},
+  year={2022},
+  organization={IEEE}
+}
+
+@inproceedings{shih2021rad,
+  title={{RAD-TTS}: Parallel flow-based {TTS} with robust alignment learning and diverse synthesis},
+  author={Shih, Kevin J and Valle, Rafael and Badlani, Rohan and Lancucki, Adrian and Ping, Wei and Catanzaro, Bryan},
+  booktitle={ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models},
+  year={2021}
+}
+
+@article{kong2020hifi,
+  title={{HiFi-GAN}: Generative adversarial networks for efficient and high fidelity speech synthesis},
+  author={Kong, Jungil and Kim, Jaehyeon and Bae, Jaekyoung},
+  journal={Advances in Neural Information Processing Systems},
+  volume={33},
+  pages={17022--17033},
+  year={2020}
+}
+
+@inproceedings{prenger2019waveglow,
+  title={Waveglow: A flow-based generative network for speech synthesis},
+  author={Prenger, Ryan and Valle, Rafael and Catanzaro, Bryan},
+  booktitle={ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  pages={3617--3621},
+  year={2019},
+  organization={IEEE}
+}
+
+@inproceedings{jang21_interspeech,
+  author={Won Jang and Dan Lim and Jaesam Yoon and Bongwan Kim and Juntae Kim},
+  title={{UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation}},
+  year=2021,
+  booktitle={Proc. Interspeech 2021},
+  pages={2207--2211},
+  doi={10.21437/Interspeech.2021-1016}
+}
+
+@inproceedings{badlani2022one,
+  title={One {TTS} alignment to rule them all},
+  author={Badlani, Rohan and {\L}a{\'n}cucki, Adrian and Shih, Kevin J and Valle, Rafael and Ping, Wei and Catanzaro, Bryan},
+  booktitle={ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
+  pages={6092--6096},
+  year={2022},
+  organization={IEEE}
+}
+
+@article{xue2021byt5,
+  title={ByT5: Towards a token-free future with pre-trained byte-to-byte models 2021},
+  author={Xue, Linting and Barua, Aditya and Constant, Noah and Al-Rfou, Rami and Narang, Sharan and Kale, Mihir and Roberts, Adam and Raffel, Colin},
+  journal={arXiv preprint arXiv:2105.13626},
+  year={2021}
+}
+
+@article{vrezavckova2021t5g2p,
+  title={T5g2p: Using text-to-text transfer transformer for grapheme-to-phoneme conversion},
+  author={{\v{R}}ez{\'a}{\v{c}}kov{\'a}, Mark{\'e}ta and {\v{S}}vec, Jan and Tihelka, Daniel},
+  year={2021},
+  journal={International Speech Communication Association}
+}
+
+@article{zhu2022byt5,
+  title={ByT5 model for massively multilingual grapheme-to-phoneme conversion},
+  author={Zhu, Jian and Zhang, Cong and Jurgens, David},
+  journal={arXiv preprint arXiv:2204.03067},
+  year={2022}
+}
+
+@article{ggulati2020conformer,
+  title={Conformer: Convolution-augmented transformer for speech recognition},
+  author={Gulati, Anmol and Qin, James and Chiu, Chung-Cheng and Parmar, Niki and Zhang, Yu and Yu, Jiahui and Han, Wei and Wang, Shibo and Zhang, Zhengdong and Wu, Yonghui and others},
+  journal={arXiv preprint arXiv:2005.08100},
+  year={2020}
+}
+
+@inproceedings{gorman2018improving,
+  title={Improving homograph disambiguation with supervised machine learning},
+  author={Gorman, Kyle and Mazovetskiy, Gleb and Nikolaev, Vitaly},
+  booktitle={Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
+  year={2018}
+}
+@inproceedings{kim2021conditional,
+  title={Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech},
+  author={Kim, Jaehyeon and Kong, Jungil and Son, Juhee},
+  booktitle={International Conference on Machine Learning},
+  pages={5530--5540},
+  year={2021},
+  organization={PMLR}
+}
+
+@article{zeghidour2022soundstream,
+  author={Zeghidour, Neil and Luebs, Alejandro and Omran, Ahmed and Skoglund, Jan and Tagliasacchi, Marco},
+  journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing}, 
+  title={{SoundStream}: An End-to-End Neural Audio Codec}, 
+  year={2022},
+  volume={30},
+  pages={495-507},
+  doi={10.1109/TASLP.2021.3129994}
+}
+
+@article{defossez2022encodec,
+  title={High fidelity neural audio compression},
+  author={D{\'e}fossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
+  journal={arXiv preprint arXiv:2210.13438},
+  year={2022}
+}
+
+@article{mentzer2023finite,
+  title={Finite scalar quantization: {VQ-VAE} made simple},
+  author={Mentzer, Fabian and Minnen, David and Agustsson, Eirikur and Tschannen, Michael},
+  journal={arXiv preprint arXiv:2309.15505},
+  year={2023}
+}