wehub/nvidia-nemo--speech
1
0
Fork 0
Code Issues Pull Requests Actions 1 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Create PR to main with cherry-pick from release / cherry-pick (push) Failing after 0s
Details
CICD NeMo / pre-flight (push) Failing after 0s
Details
CICD NeMo / configure (push) Has been skipped
Details
Build, validate, and release Neural Modules / pre-flight (push) Failing after 1s
Details
CICD NeMo / code-linting (push) Has been skipped
Details
Build, validate, and release Neural Modules / release (push) Has been skipped
Details
Build, validate, and release Neural Modules / release-summary (push) Has been cancelled
Details
CICD NeMo / cicd-test-container-build (push) Has been cancelled
Details
CICD NeMo / cicd-import-tests (push) Has been cancelled
Details
CICD NeMo / L0_Setup_Test_Data_And_Models (push) Has been cancelled
Details
CICD NeMo / cicd-main-unit-tests (push) Has been cancelled
Details
CICD NeMo / cicd-main-speech (push) Has been cancelled
Details
CICD NeMo / Nemo_CICD_Test (push) Has been cancelled
Details
CICD NeMo / Coverage (e2e) (push) Has been cancelled
Details
CICD NeMo / Coverage (unit-test) (push) Has been cancelled
Details
CodeQL / Analyze (python) (push) Has been cancelled
Details
CICD NeMo / cicd-wait-in-queue (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:28:58 +08:00
commit ba4be087d5
2316 changed files with 2668701 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tutorials/tts/Audio_Codec_Inference.ipynb
+478
View File
@@ -0,0 +1,478 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "7X-TwhdTGmlc"
},
"source": [
"# License"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fCQUeZRPGnoe"
},
"source": [
"> Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n",
">\n",
"> Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
">\n",
"> http://www.apache.org/licenses/LICENSE-2.0\n",
">\n",
"> Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rtBDkKqVGZJ8"
},
"source": [
"# Introduction"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pZ2QSsXuGbMe"
},
"source": [
"In this tutorial we show how use NeMo **neural audio codecs** at inference time. To learn more about training and finetuning neural audio codecs in NeMo, check the [Audio Codec Training tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Audio_Codec_Training.ipynb).\n",
"\n",
"An audio codec typically consists of an encoder, a quantizer and a decoder, with a typical architecture depicted in the figure below.\n",
"An audio codec can be used to encode an input audio signal into a sequence of discrete values.\n",
"In this tutorial, the discrete values will be referred to as **audio tokens**.\n",
"The obtained audio tokens can be decoded into an output audio signal.\n",
"\n",
"Audio tokens can be used to represent the input audio for an automatic speech recognition (ASR) model [[1](https://arxiv.org/abs/2309.10922), [2](https://arxiv.org/pdf/2407.03495)], or to represent the output audio of a text-to-speech (TTS) system [[3](https://arxiv.org/abs/2406.05298), [4](https://arxiv.org/pdf/2406.17957)].\n",
"\n",
"NeMo provides several neural audio codec models, inlcuding audio codecs and mel codecs at different sampling rates.\n",
"The list of the available models can be found [here](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tts/checkpoints.html#codec-models).\n",
"\n",
"<div>\n",
"<img src=\"https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/nemo_audio_codec.png\" width=\"800\", height=\"400\"/>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3OZassNG5xff"
},
"source": [
"# Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WZvQvPkIhRi3"
},
"outputs": [],
"source": [
"BRANCH = 'main'\n",
"# Install NeMo library. If you are running locally (rather than on Google Colab), follow the instructions at https://github.com/NVIDIA/NeMo#Installation\n",
"\n",
"if 'google.colab' in str(get_ipython()):\n",
" !python -m pip install \"nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "v8NGOM0EzK8W"
},
"outputs": [],
"source": [
"import math\n",
"import wget\n",
"import os\n",
"import librosa\n",
"import torch\n",
"import numpy as np\n",
"import IPython.display as ipd\n",
"import matplotlib.pyplot as plt\n",
"from pathlib import Path\n",
"\n",
"\n",
"# Utility for displaying signals and metrics\n",
"def show_signal(signal: np.ndarray, sample_rate: int = 16000, tag: str = 'Signal'):\n",
" \"\"\"Show the time-domain signal and its spectrogram.\n",
" \"\"\"\n",
" fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 2.5))\n",
"\n",
" # show waveform\n",
" t = np.arange(0, len(signal)) / sample_rate\n",
"\n",
" ax[0].plot(t, signal)\n",
" ax[0].set_xlim(0, t.max())\n",
" ax[0].grid()\n",
" ax[0].set_xlabel('time / s')\n",
" ax[0].set_ylabel('amplitude')\n",
" ax[0].set_title(tag)\n",
"\n",
" n_fft = 1024\n",
" hop_length = 256\n",
"\n",
" D = librosa.amplitude_to_db(np.abs(librosa.stft(signal, n_fft=n_fft, hop_length=hop_length)), ref=np.max)\n",
" img = librosa.display.specshow(D, y_axis='linear', x_axis='time', sr=sample_rate, n_fft=n_fft, hop_length=hop_length, ax=ax[1])\n",
" ax[1].set_title(tag)\n",
"\n",
" plt.tight_layout()\n",
" plt.colorbar(img, format=\"%+2.f dB\", ax=ax)\n",
"\n",
"\n",
"# Utility for displaying a latent representation\n",
"def show_latent(latent: np.ndarray, tag: str):\n",
" plt.figure(figsize = (16, 3))\n",
" img = plt.imshow(latent, aspect='equal')\n",
" plt.colorbar(img, ax=plt.gca())\n",
" plt.title(tag)\n",
" plt.xlabel('Time frame')\n",
" plt.ylabel('Latent vector index')\n",
" plt.tight_layout()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8ZKDMTwsEY1K"
},
"outputs": [],
"source": [
"# Working directory\n",
"ROOT_DIR = Path().absolute() / 'codec_tutorial'\n",
"\n",
"# Create dataset directory\n",
"DATA_DIR = ROOT_DIR / 'data'\n",
"DATA_DIR.mkdir(parents=True, exist_ok=True)\n",
"\n",
"audio_path = DATA_DIR / 'LJ023-0089.wav'\n",
"audio_url = \"https://multilangaudiosamples.s3.us-east-2.amazonaws.com/LJ023-0089.wav\"\n",
"\n",
"if not os.path.exists(audio_path):\n",
" wget.download(audio_url, audio_path.as_posix())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KAbH7N427FdT"
},
"source": [
"# Load a model from NGC"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ODgdGgsAAUku"
},
"source": [
"Any of the [pretrained checkpoints](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tts/checkpoints.html#codec-models) could be used for inference.\n",
"Here, we use `mel_codec_22khz_fullband_medium`, which works for 22.05 kHz audio signals.\n",
"\n",
"The model can be easily restored from NGC:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XqAYWR65aKTx"
},
"outputs": [],
"source": [
"from nemo.collections.tts.models.audio_codec import AudioCodecModel\n",
"\n",
"# Optionally specify a pretrained model to fine-tune from. To train from scratch, set this to 'None'.\n",
"model_name = 'mel_codec_22khz_fullband_medium'\n",
"codec_model = AudioCodecModel.from_pretrained(model_name)\n",
"codec_model.freeze()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZnnjL28pEY1L"
},
"source": [
"Show information about the loaded model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4xsfeHVyEY1L"
},
"outputs": [],
"source": [
"print(f'Loaded model from NeMo:')\n",
"print(f'\\tmodel name : {model_name}')\n",
"print(f'\\tsample rate : {codec_model.sample_rate} Hz')\n",
"print(f'\\tlatent dimension : {codec_model.vector_quantizer.codebook_dim}')\n",
"\n",
"print('\\n\\nModel summary:')\n",
"print(codec_model.summarize())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fM4QPsLTnzK7"
},
"source": [
"# Inference"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tkZC6Dl7KRl6"
},
"source": [
"## Processing audio\n",
"\n",
"Here we use the codec model to process the input audio by applying the complete model. The input signal is encoded, quantized, dequantized and decoded. Finally, a reconstructed signal is obtained."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sYzvAYr2vo1K"
},
"outputs": [],
"source": [
"input_audio, sr = librosa.load(audio_path, sr=codec_model.sample_rate)\n",
"\n",
"# Shape (batch, time)\n",
"input_audio_tensor = torch.from_numpy(input_audio).unsqueeze(dim=0).to(codec_model.device)\n",
"\n",
"# Shape (batch,)\n",
"input_audio_len = torch.tensor([input_audio_tensor.size(-1)]).to(codec_model.device)\n",
"\n",
"# Process audio using the codec model\n",
"output_audio_tensor, _ = codec_model(audio=input_audio_tensor, audio_len=input_audio_len)\n",
"\n",
"# Output audio\n",
"output_audio = output_audio_tensor.squeeze().cpu().numpy()\n",
"\n",
"# Show signals\n",
"show_signal(input_audio, tag='Input audio', sample_rate=codec_model.sample_rate)\n",
"show_signal(output_audio, tag='Output audio', sample_rate=codec_model.sample_rate)\n",
"\n",
"# Play audio\n",
"print('Input audio')\n",
"ipd.display(ipd.Audio(input_audio, rate=codec_model.sample_rate))\n",
"\n",
"print('Output audio')\n",
"ipd.display(ipd.Audio(output_audio, rate=codec_model.sample_rate))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rynZYwg2VP5d"
},
"source": [
"## Audio tokens\n",
"\n",
"Audio tokens can be easily computed by using the `encode` method of the `AudioCodec` model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ASKM_jKVEY1L"
},
"outputs": [],
"source": [
"# Convert audio to tokens\n",
"tokens, tokens_len = codec_model.encode(audio=input_audio_tensor, audio_len=input_audio_len)\n",
"\n",
"print('tokens information:')\n",
"print(f'\\tshape (batch, codebook, time frame) : {tokens.size()}')\n",
"print(f'\\tdtype : {tokens.dtype}')\n",
"print(f'\\tmin : {tokens.min()}')\n",
"print(f'\\tmax : {tokens.max()}')\n",
"\n",
"# Number of codebooks should match the number of codebooks/groups\n",
"if hasattr(codec_model.vector_quantizer, 'num_groups'):\n",
" # Group FSQ\n",
" assert tokens.size(1) == codec_model.vector_quantizer.num_groups\n",
" print(f'\\tnum_groups : {tokens.size(1)}')\n",
"elif hasattr(codec_model.vector_quantizer, 'codebooks'):\n",
" # RVQ\n",
" assert tokens.size(1) == len(codec_model.vector_quantizer.codebooks)\n",
" print(f'\\tnum_codebooks : {tokens.size(1)}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CmliPMnDEY1L"
},
"source": [
"Similarly, audio can be easily reconstructed from audio tokens using the `decode` method of the `AudioCodec` models."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RTQ1M9PMEY1L"
},
"outputs": [],
"source": [
"# Convert tokens back to audio\n",
"output_audio_from_tokens_tensor, _ = codec_model.decode(tokens=tokens, tokens_len=tokens_len)\n",
"output_audio_from_tokens = output_audio_from_tokens_tensor.squeeze().cpu().numpy()\n",
"\n",
"# Show signals\n",
"show_signal(output_audio_from_tokens, tag='Output audio from tokens', sample_rate=codec_model.sample_rate)\n",
"show_signal(output_audio_from_tokens - output_audio, tag='Difference compared to forward pass', sample_rate=codec_model.sample_rate)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kGqotZkqEY1M"
},
"source": [
"## Latent representation\n",
"\n",
"Continuous (non-discrete) latent representation at the output of the encoder can be easily computed using the `encode_audio` method of the `AudioCodec` model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "r-89-gG3EY1M"
},
"outputs": [],
"source": [
"# Convert audio to the encoded representation\n",
"encoded, encoded_len = codec_model.encode_audio(audio=input_audio_tensor, audio_len=input_audio_len)\n",
"\n",
"print('encoded information:')\n",
"print(f'\\tshape (batch, codebook, time frame) : {encoded.size()}')\n",
"print(f'\\tdtype : {encoded.dtype}')\n",
"print(f'\\tmin : {encoded.min()}')\n",
"print(f'\\tmax : {encoded.max()}')\n",
"\n",
"\n",
"# Show the encoded representation\n",
"show_latent(encoded.squeeze().cpu().numpy(), tag='Encoder output')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3Ory1U1uEY1M"
},
"source": [
"The encoded representation can be easily converted to tokens, dequantized into a continuous latent representation and decoded back to audio."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "btmqUWNkEY1M"
},
"outputs": [],
"source": [
"# Encoder output to tokens\n",
"tokens = codec_model.quantize(encoded=encoded, encoded_len=encoded_len)\n",
"\n",
"# Tokens back to a continuous vector\n",
"dequantized = codec_model.dequantize(tokens=tokens, tokens_len=encoded_len)\n",
"\n",
"# Reconstruct audio\n",
"output_audio_from_latent_tensor, _ = codec_model.decode_audio(inputs=dequantized, input_len=encoded_len)\n",
"output_audio_from_latent = output_audio_from_latent_tensor.squeeze().cpu().numpy()\n",
"\n",
"# Show dequantized latent representation\n",
"show_latent(dequantized.squeeze().cpu().numpy(), tag='Decoder input')\n",
"\n",
"# Show signals\n",
"show_signal(output_audio_from_latent, tag='Output audio from latent', sample_rate=codec_model.sample_rate)\n",
"show_signal(output_audio_from_latent - output_audio, tag='Difference compared to forward pass', sample_rate=codec_model.sample_rate)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cMvU0WxlEY1M"
},
"source": [
"# Related information"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_LtyHHuLkNDv"
},
"source": [
"To learn more about audio codec models in NeMo, look at our [documentation](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tts/models.html#codecs).\n",
"\n",
"For more information on training and finetuning neural audio codecs in NeMo, check the [Audio Codec Training tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Audio_Codec_Training.ipynb)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LeqV3VvJVOb-"
},
"source": [
"# References"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Rvu4w2x_3RSY"
},
"source": [
"1. [Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition](https://arxiv.org/abs/2309.10922)\n",
"2. [Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations](https://arxiv.org/pdf/2407.03495)\n",
"3. [Spectral Codecs: Spectrogram-Based Audio Codecs for High Quality Speech Synthesis](https://arxiv.org/abs/2406.05298)\n",
"4. [Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment](https://arxiv.org/pdf/2406.17957)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"colab": {
"provenance": [],
"toc_visible": true
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tutorials/tts/Audio_Codec_Training.ipynb
+796
View File
@@ -0,0 +1,796 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "7X-TwhdTGmlc"
},
"source": [
"# License"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fCQUeZRPGnoe"
},
"source": [
"> Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n",
">\n",
"> Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
">\n",
"> http://www.apache.org/licenses/LICENSE-2.0\n",
">\n",
"> Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rtBDkKqVGZJ8"
},
"source": [
"# Introduction"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pZ2QSsXuGbMe"
},
"source": [
"In this tutorial we show how to use NeMo to train and fine-tune **neural audio codecs**.\n",
"\n",
"Neural audio codecs are deep learning models that compress audio into a low bitrate representation. The compact embedding space created by these models can be useful for various speech tasks, such as TTS and ASR.\n",
"\n",
"<div>\n",
"<img src=\"https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/nemo_audio_codec.png\" width=\"800\", height=\"400\"/>\n",
"</div>\n",
"\n",
"Audio codec models typically have an *encoder-quantizer-decoder* structure. The **encoder** takes an input audio signal and encodes it into a sequence of embeddings. The **quantizer** discretizes the embeddings to create a lookup table known as a **codebook**. The embeddings saved in the codebook are referred to as **audio codes**. The **decoder** takes the audio codes as input and attempts to reconstruct the original audio signal.\n",
"\n",
"To store compressed audio we only need to save the codebook index for each embedding in an audio sequence. This is how audio codec models achieve low bitrates. The codebook indices for an audio are referred to **audio tokens**. It is becoming common for speech generation models to synthesize speech by predicting audio tokens.\n",
"\n",
"In NeMo we have implementations of the [SEANet encoder and decoder](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/modules/encodec_modules.py#L146) used by [EnCodec](https://github.com/facebookresearch/encodec). As well as a [ResNet encoder](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/modules/audio_codec_modules.py#L1035) and [HiFi-GAN decoder](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/modules/audio_codec_modules.py#L875). For quantizers we support [Residual Vector Quantizer](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/modules/encodec_modules.py#L694) (**RVQ**) and [Finite Scalar Quantizer](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/modules/audio_codec_modules.py#L409) (**FSQ**).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3OZassNG5xff"
},
"source": [
"# Install"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WZvQvPkIhRi3"
},
"outputs": [],
"source": [
"BRANCH = 'main'\n",
"# Install NeMo library. If you are running locally (rather than on Google Colab), comment out the below line\n",
"# and instead follow the instructions at https://github.com/NVIDIA/NeMo#Installation\n",
"!python -m pip install \"nemo_toolkit[tts] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "v8NGOM0EzK8W"
},
"outputs": [],
"source": [
"from pathlib import Path"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tvsgWO_WhV3M"
},
"outputs": [],
"source": [
"# Directory where tutorialscripts will run and outputs will be saved.\n",
"ROOT_DIR = Path().absolute() / \"codec_tutorial\"\n",
"\n",
"# Nemo code paths\n",
"NEMO_DIR = ROOT_DIR / \"nemo\"\n",
"NEMO_SCRIPT_DIR = NEMO_DIR / \"scripts\" / \"dataset_processing\" / \"tts\"\n",
"NEMO_EXAMPLES_DIR = NEMO_DIR / \"examples\" / \"tts\"\n",
"NEMO_CONFIG_DIR = NEMO_EXAMPLES_DIR / \"conf\"\n",
"\n",
"nemo_download_dir = str(NEMO_DIR)\n",
"# Download local version of NeMo scripts. If you are running locally and want to use your own local NeMo code,\n",
"# comment out the below line and set NEMO_ROOT_DIR to your local path.\n",
"!git clone -b $BRANCH https://github.com/NVIDIA-NeMo/Speech.git $nemo_download_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KAbH7N427FdT"
},
"source": [
"# Configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ODgdGgsAAUku"
},
"source": [
"Predefined model configurations are available in https://github.com/NVIDIA/NeMo/tree/main/examples/tts/conf/audio_codec.\n",
"\n",
"Configurations available include:\n",
"\n",
"* **audio_codec_*.yaml**: Audio codec configurations optimized for various sampling rates.\n",
"* **mel_codec_*.yaml**: A mel-spectrogram based codec designed to maximize the performance of speech synthesis models.\n",
"* **encodec_*.yaml**: A reproduction of the original [EnCodec](https://arxiv.org/abs/2210.13438) model setup.\n",
"* **audio_codec_low_frame_rate_22050.yaml**: The [Low Frame-rate Speech Codec](https://arxiv.org/abs/2409.12117): an audio codec that achieves high-quality audio compression with a 1.89 kbps bitrate and 21.5 frames per second.\n",
"\n",
"This tutorial can be run with any of our predefined configs. As a default we have selected `audio_codec_16000.yaml`, which works for 16kHz audio."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SPtjS2LkzE9Q"
},
"outputs": [],
"source": [
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iCPJFKg63Dsv"
},
"outputs": [],
"source": [
"CONFIG_FILENAME = \"audio_codec_22050.yaml\"\n",
"CONFIG_DIR = NEMO_CONFIG_DIR / \"audio_codec\"\n",
"\n",
"config_filepath = CONFIG_DIR / CONFIG_FILENAME\n",
"\n",
"if not config_filepath.exists():\n",
" raise ValueError(f\"Config file does not exist {config_filepath}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "QE0HYh7FjAR3"
},
"outputs": [],
"source": [
"# Read model name and sample rate from model configuration\n",
"omega_conf = OmegaConf.load(config_filepath)\n",
"MODEL_NAME = omega_conf.name\n",
"SAMPLE_RATE = omega_conf.sample_rate\n",
"print(f\"Training {MODEL_NAME} with sample rate {SAMPLE_RATE}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "W7F--_0maLh5"
},
"source": [
"We provide pretrained model checkpoints for fine-tuning.\n",
"\n",
"A list of models available on NGC can be found [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/models/audio_codec.py#L645).\n",
"\n",
"A list of models available on Hugging Face can be found [here](https://huggingface.co/collections/nvidia/nemo-audio-codecs-674f57ab6cb1324f997b5d5b). To use a checkpoint from hugging face, add \"nvidia/\" before the model name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cADIAIDUcGWd"
},
"outputs": [],
"source": [
"from nemo.collections.tts.models.audio_codec import AudioCodecModel\n",
"\n",
"pretrained_model_name = \"nvidia/audio-codec-22khz\"\n",
"\n",
"if pretrained_model_name is None:\n",
" MODEL_CHECKPOINT_PATH = None\n",
"else:\n",
" MODEL_CHECKPOINT_PATH = AudioCodecModel.from_pretrained(model_name=pretrained_model_name, return_model_file=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fM4QPsLTnzK7"
},
"source": [
"# Dataset Preparation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tkZC6Dl7KRl6"
},
"source": [
"For our tutorial, we use a subset of [VCTK](https://datashare.ed.ac.uk/handle/10283/2950) dataset with 5 speakers (p225-p229)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sYzvAYr2vo1K"
},
"outputs": [],
"source": [
"import tarfile\n",
"import wget\n",
"\n",
"from nemo.collections.asr.parts.utils.manifest_utils import read_manifest, write_manifest"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "aoxN1QsUzX-k"
},
"outputs": [],
"source": [
"# Create dataset directory\n",
"DATA_DIR = ROOT_DIR / \"data\"\n",
"\n",
"DATA_DIR.mkdir(parents=True, exist_ok=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mArlQd5Hk36b"
},
"outputs": [],
"source": [
"# Download the dataset\n",
"dataset_url = \"https://vctk-subset.s3.amazonaws.com/vctk_subset_multispeaker.tar.gz\"\n",
"dataset_tar_filepath = DATA_DIR / \"vctk.tar.gz\"\n",
"\n",
"if not dataset_tar_filepath.exists():\n",
" wget.download(dataset_url, out=str(dataset_tar_filepath))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "p987cjtOy9C7"
},
"outputs": [],
"source": [
"# Extract the dataset\n",
"with tarfile.open(dataset_tar_filepath) as tar_f:\n",
" tar_f.extractall(DATA_DIR)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ko6dxYJW0i3G"
},
"outputs": [],
"source": [
"DATASET_DIR = DATA_DIR / \"vctk_subset_multispeaker\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "We5FHYQt5BeO"
},
"outputs": [],
"source": [
"# Visualize the raw dataset\n",
"train_raw_filepath = DATASET_DIR / \"train.json\"\n",
"!head $train_raw_filepath"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i3jsk2HCMSU5"
},
"source": [
"## Manifest Processing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "N8WuAGJsMHRn"
},
"source": [
"The downloaded manifest is formatted for TTS training, which contains metadata such as text and speaker.\n",
"\n",
"For codec training we need `audio_filepath`. The `audio_filepath` field can either be an *absolute path*, or a *relative path* with the root directory provided as an argument to each script. Here we use relative paths.\n",
"\n",
"If you include `duration` the training script will automatically calculate the total size of each dataset used, and can be useful for filtering based on utterance length."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zoCRrKQ20VZP"
},
"outputs": [],
"source": [
"def update_manifest(data_type):\n",
" input_filepath = DATASET_DIR / f\"{data_type}.json\"\n",
" output_filepath = DATASET_DIR / f\"{data_type}_raw.json\"\n",
"\n",
" entries = read_manifest(input_filepath)\n",
" new_entries = []\n",
" for entry in entries:\n",
" # Provide relative path instead of absolute path\n",
" audio_filepath = entry[\"audio_filepath\"].replace(\"audio/\", \"\")\n",
" duration = round(entry[\"duration\"], 2)\n",
" new_entry = {\n",
" \"audio_filepath\": audio_filepath,\n",
" \"duration\": duration\n",
" }\n",
" new_entries.append(new_entry)\n",
"\n",
" write_manifest(output_path=output_filepath, target_manifest=new_entries, ensure_ascii=False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PaCc3GCG1UbH"
},
"outputs": [],
"source": [
"update_manifest(\"dev\")\n",
"update_manifest(\"train\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bVLIB3Ip1Aqn"
},
"outputs": [],
"source": [
"# Visualize updated 'audio_filepath' field.\n",
"train_filepath = DATASET_DIR / \"train_raw.json\"\n",
"!head $train_filepath"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "alrRDWio41qi"
},
"source": [
"## Audio Preprocessing"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4WfEaMwpUsFt"
},
"source": [
"Next we process the audio data using [preprocess_audio.py](https://github.com/NVIDIA/NeMo/blob/main/scripts/dataset_processing/tts/preprocess_audio.py).\n",
"\n",
"During this step we can apply the following transformations:\n",
"\n",
"1. Resample the audio from 48khz to the target sample rate for codec training.\n",
"2. Remove long silence from the beginning and end of each audio file. This can be done using an *energy* based approach which will work on clean audio, or using *voice activity detection (VAD)* which is slower but also works on audio with background or static noise (eg. from a microphone). Here we suggest VAD because some audio in VCTK has background noise."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WEvIefjnd7AG"
},
"outputs": [],
"source": [
"import IPython.display as ipd"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-qEuCH8S4vFP"
},
"outputs": [],
"source": [
"# Python wrapper to invoke the given bash script with the given input args\n",
"def run_script(script, args):\n",
" args = ' \\\\'.join(args)\n",
" cmd = f\"python {script} \\\\{args}\"\n",
"\n",
" print(cmd.replace(\" \\\\\", \"\\n\"))\n",
" print()\n",
" !$cmd"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0kQ1UDnGfdX6"
},
"outputs": [],
"source": [
"audio_preprocessing_script = NEMO_SCRIPT_DIR / \"preprocess_audio.py\"\n",
"\n",
"# Directory with raw audio data\n",
"input_audio_dir = DATASET_DIR / \"audio\"\n",
"# Directory to write preprocessed audio to\n",
"output_audio_dir = DATASET_DIR / \"audio_preprocessed\"\n",
"# Whether to overwrite existing audio, if it exists in the output directory\n",
"overwrite_audio = True\n",
"# Whether to overwrite output manifest, if it exists\n",
"overwrite_manifest = True\n",
"# Number of threads to parallelize audio processing across\n",
"num_workers = 4\n",
"# Format of output audio files. Use \"flac\" to compress to a smaller file size.\n",
"output_format = \"flac\"\n",
"# Method for silence trimming. Can use \"energy.yaml\" or \"vad.yaml\".\n",
"trim_config_path = NEMO_CONFIG_DIR / \"trim\" / \"vad.yaml\"\n",
"\n",
"def preprocess_audio(data_type):\n",
" input_filepath = DATASET_DIR / f\"{data_type}_raw.json\"\n",
" output_filepath = DATASET_DIR / f\"{data_type}_manifest.json\"\n",
"\n",
" args = [\n",
" f\"--input_manifest={input_filepath}\",\n",
" f\"--output_manifest={output_filepath}\",\n",
" f\"--input_audio_dir={input_audio_dir}\",\n",
" f\"--output_audio_dir={output_audio_dir}\",\n",
" f\"--num_workers={num_workers}\",\n",
" f\"--output_sample_rate={SAMPLE_RATE}\",\n",
" f\"--output_format={output_format}\",\n",
" f\"--trim_config_path={trim_config_path}\"\n",
" ]\n",
" if overwrite_manifest:\n",
" args.append(\"--overwrite_manifest\")\n",
" if overwrite_audio:\n",
" args.append(\"--overwrite_audio\")\n",
"\n",
" run_script(audio_preprocessing_script, args)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ai0zbXSOriuY"
},
"outputs": [],
"source": [
"preprocess_audio(\"dev\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NUKnidQYfgDo"
},
"outputs": [],
"source": [
"preprocess_audio(\"train\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "x2yhJtsj2lDR"
},
"source": [
"Before we proceed, it is important to verify that the audio processing works as expected. Let's listen to an audio file before and after processing.\n",
"\n",
"Note that the processed audio is shorter because we trimmed the leading and trailing silence."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "AfdHUHAWuF-G"
},
"outputs": [],
"source": [
"!ls $processed_audio_filepath"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_fM3GwJxkjOA"
},
"outputs": [],
"source": [
"audio_file = \"p228_009.wav\"\n",
"audio_filepath = input_audio_dir / audio_file\n",
"processed_audio_filepath = output_audio_dir / audio_file.replace(\".wav\", \".flac\")\n",
"\n",
"print(\"Original audio.\")\n",
"ipd.display(ipd.Audio(audio_filepath, rate=SAMPLE_RATE))\n",
"\n",
"print(\"Processed audio.\")\n",
"ipd.display(ipd.Audio(processed_audio_filepath, rate=SAMPLE_RATE))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oRO842MUyODC"
},
"source": [
"# Audio Codec Training"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E4wUKYOfH8ax"
},
"source": [
"Here we show how to train an audio codec model from scratch. Instructions and checkpoints for fine-tuning will be provided later.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pqfl9jAYMJob"
},
"outputs": [],
"source": [
"import os\n",
"import torch\n",
"from omegaconf import OmegaConf"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jK2rr-Kr6Qg8"
},
"outputs": [],
"source": [
"dataset_name = \"vctk\"\n",
"audio_dir = DATASET_DIR / \"audio_preprocessed\"\n",
"train_manifest_filepath = DATASET_DIR / \"train_manifest.json\"\n",
"dev_manifest_filepath = DATASET_DIR / \"dev_manifest.json\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Vr4D-NB-yQx8"
},
"outputs": [],
"source": [
"audio_codec_training_script = NEMO_EXAMPLES_DIR / \"audio_codec.py\"\n",
"\n",
"# The total number of training steps will be (epochs * steps_per_epoch)\n",
"epochs = 10\n",
"steps_per_epoch = 10\n",
"\n",
"# Name of the experiment that will determine where it is saved locally and in TensorBoard and WandB\n",
"run_id = \"test_run\"\n",
"exp_dir = ROOT_DIR / \"exps\"\n",
"codec_exp_output_dir = exp_dir / MODEL_NAME / run_id\n",
"# Directory where predicted audio will be stored periodically throughout training\n",
"codec_log_dir = codec_exp_output_dir / \"logs\"\n",
"# Optionally log visualization of learned codes.\n",
"log_dequantized = True\n",
"# Optionally log predicted audio and other artifacts to WandB\n",
"log_to_wandb = False\n",
"# Optionally log predicted audio and other artifacts to Tensorboard\n",
"log_to_tensorboard = False\n",
"\n",
"if torch.cuda.is_available():\n",
" accelerator=\"gpu\"\n",
" batch_size = 4\n",
" devices = -1\n",
"else:\n",
" import multiprocessing\n",
" accelerator=\"cpu\"\n",
" batch_size = 2\n",
" devices = multiprocessing.cpu_count()\n",
"\n",
"args = [\n",
" f\"--config-path={CONFIG_DIR}\",\n",
" f\"--config-name={CONFIG_FILENAME}\",\n",
" f\"max_epochs={epochs}\",\n",
" f\"weighted_sampling_steps_per_epoch={steps_per_epoch}\",\n",
" f\"batch_size={batch_size}\",\n",
" f\"log_dir={codec_log_dir}\",\n",
" f\"exp_manager.exp_dir={exp_dir}\",\n",
" f\"+exp_manager.version={run_id}\",\n",
" f\"model.log_config.log_wandb={log_to_wandb}\",\n",
" f\"model.log_config.log_tensorboard={log_to_tensorboard}\",\n",
" f\"model.log_config.generators.0.log_dequantized={log_dequantized}\",\n",
" f\"trainer.accelerator={accelerator}\",\n",
" f\"trainer.devices={devices}\",\n",
" f\"+train_ds_meta.{dataset_name}.manifest_path={train_manifest_filepath}\",\n",
" f\"+train_ds_meta.{dataset_name}.audio_dir={audio_dir}\",\n",
" f\"+val_ds_meta.{dataset_name}.manifest_path={dev_manifest_filepath}\",\n",
" f\"+val_ds_meta.{dataset_name}.audio_dir={audio_dir}\",\n",
" f\"+log_ds_meta.{dataset_name}.manifest_path={dev_manifest_filepath}\",\n",
" f\"+log_ds_meta.{dataset_name}.audio_dir={audio_dir}\"\n",
"]\n",
"\n",
"if MODEL_CHECKPOINT_PATH is not None:\n",
" args.append(f\"+init_from_nemo_model={MODEL_CHECKPOINT_PATH}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Bn8lQG0PxWGi"
},
"outputs": [],
"source": [
"# If an error occurs, log the entire stacktrace.\n",
"os.environ[\"HYDRA_FULL_ERROR\"] = \"1\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yUxFCNrE3Ywi"
},
"outputs": [],
"source": [
"# Do the model training. For some configurations this step might hang when using CPU.\n",
"run_script(audio_codec_training_script, args)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BBPIpS-lL6z9"
},
"source": [
"During training, the model will automatically save predictions for all audio files specified in the `log_ds_meta` manifest."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rSFOm1Sg46Lh"
},
"outputs": [],
"source": [
"codec_log_epoch_dir = codec_log_dir / \"epoch_10\" / dataset_name\n",
"!ls $codec_log_epoch_dir"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oCJs7oCLMIjD"
},
"source": [
"This makes it easy to listen to the audio to determine how well the model is performing. We can decide to stop training when either:\n",
"\n",
"* The predicted audio sounds almost identical to the original audio.\n",
"* The predicted audio stops improving in between epochs.\n",
"\n",
"**Note that when training from scratch, the dataset in this tutorial is too small to get good audio quality.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "G6k4ymzfJ5Y6"
},
"outputs": [],
"source": [
"audio_filepath_ground_truth = output_audio_dir / \"p228_009.flac\"\n",
"audio_filepath_reconstructed = codec_log_epoch_dir / \"p228_009_audio_out.wav\"\n",
"\n",
"print(\"Ground truth audio.\")\n",
"ipd.display(ipd.Audio(audio_filepath_ground_truth, rate=SAMPLE_RATE))\n",
"\n",
"print(\"Reconstructed audio.\")\n",
"ipd.display(ipd.Audio(audio_filepath_reconstructed, rate=SAMPLE_RATE))\n",
"\n",
"dequantized_filepath = codec_log_epoch_dir / \"p228_009_dequantized.png\"\n",
"ipd.Image(dequantized_filepath)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rynZYwg2VP5d"
},
"source": [
"# Related Information"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_LtyHHuLkNDv"
},
"source": [
"To learn more about audio codec models in NeMo, look at our [documentation](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/tts/models.html#codecs).\n",
"\n",
"For more information on how to download and run pre-trained audio codec models, visit [NGC](https://catalog.ngc.nvidia.com/models?filters=&orderBy=scoreDESC&query=codec)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LeqV3VvJVOb-"
},
"source": [
"# References"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Rvu4w2x_3RSY"
},
"source": [
"1. [EnCodec](https://arxiv.org/abs/2210.13438)\n",
"2. [Finite Scalar Quantization (FSQ)](https://arxiv.org/abs/2309.15505)\n",
"3. [HiFi-GAN](https://arxiv.org/abs/2010.05646)\n",
"4. [SEANet](https://arxiv.org/abs/2009.02095)\n",
"5. [Spectral Codecs](https://arxiv.org/abs/2406.05298)\n",
"6. [Low Frame-rate Speech Codec](https://arxiv.org/abs/2409.12117)"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tutorials/tts/Evaluation_MelCepstralDistortion.ipynb
+631
View File
@@ -0,0 +1,631 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "f0408502",
"metadata": {},
"source": [
"# Model Evaluation: Mel Cepstral Distortion with Dynamic Time Warping"
]
},
{
"cell_type": "markdown",
"id": "8a749b10",
"metadata": {},
"source": [
"In this tutorial, we will learn how to calculate **mel cepstral distortion (MCD)** with **dynamic time warping (DTW)** between a synthesized mel spec and a reference mel spec. MCD DTW can be useful for comparing models trained on the same training data.\n",
"\n",
"MCD is an objective measure of speech quality that is calculated between pairs of TTS-generated mel spectrograms and ground truth mel spectrograms. Two mel spectrograms are similar if the MCD value between them is low; as you might expect, the MCD of a mel spec with itself is 0.\n",
"\n",
"MCD DTW is a modification of MCD that works with non-aligned mels by using a dynamic time warping cost matrix. (Vanilla MCD can only be measured for two mels that are both the same length, and which are assumed to be aligned.) The scale depends on factors such as the mel extractor and reduction algorithm (mean of DTW cost or min DTW path cost)."
]
},
{
"cell_type": "markdown",
"id": "8801b8cb",
"metadata": {},
"source": [
"## License\n",
"\n",
"> Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n",
">\n",
"> Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"> you may not use this file except in compliance with the License.\n",
"> You may obtain a copy of the License at\n",
">\n",
"> http://www.apache.org/licenses/LICENSE-2.0\n",
">\n",
"> Unless required by applicable law or agreed to in writing, software\n",
"> distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"> See the License for the specific language governing permissions and\n",
"> limitations under the License."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "09cff1ab",
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"You can either run this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"BRANCH = 'main'\n",
"# If you're using Google Colab and not running locally, uncomment and run this cell.\n",
"# !pip install librosa numpy matplotlib"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8b9b6e1c",
"metadata": {},
"outputs": [],
"source": [
"## Import libraries\n",
"import IPython.display as ipd\n",
"import librosa\n",
"import numpy as np\n",
"import math\n",
"import matplotlib.pyplot as plt\n",
"import os"
]
},
{
"cell_type": "markdown",
"id": "398b01e0",
"metadata": {},
"source": [
"## Defining Parameters for Spectrogram Generation\n",
"\n",
"In this section we define parameters required to generate our mel spectrograms. More information about these parameters can found in the Librosa documentation for [stft](https://librosa.org/doc/main/generated/librosa.stft.html), [mels](https://librosa.org/doc/main/generated/librosa.filters.mel.html) and [mfcc](https://librosa.org/doc/main/generated/librosa.feature.mfcc.html) generation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "191012de",
"metadata": {},
"outputs": [],
"source": [
"## Mel spec params\n",
"n_fft=1024\n",
"hop_length=256\n",
"win_length=None\n",
"window='hann'\n",
"n_mels = 80\n",
"sr = 22050\n",
"\n",
"## Mfcc params\n",
"n_mfcc=34"
]
},
{
"cell_type": "markdown",
"id": "54830c8c",
"metadata": {},
"source": [
"A little bit about these parameters: \n",
" - `n_fft`: Number of fft_components or length of windowed signal after padding in STFT. \n",
" - `hop_length`: Number of audio samples between adjacent STFT columns. \n",
" - `window`: Window to use in STFT. \n",
" - `win_length`: Length of the window to be used. \n",
" - `n_mels`: Number of number of mel bands to generate. \n",
" - `sr`: Sample rate of the samples. \n",
" - `n_mfcc`: Number of MFCCs to generate."
]
},
{
"cell_type": "markdown",
"id": "4cb9e37c",
"metadata": {},
"source": [
"## Loading and Visualizing Data\n",
"\n",
"Lets apply the algorithm on a synthesized mel and ground truth mel pair for understanding.\n",
"\n",
"First, we need a function to generate mel spectrograms from audio files. Mel spectrograms are generated using the [librosa mel extractor](https://librosa.org/doc/main/generated/librosa.filters.mel.html)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "042ceb7f",
"metadata": {},
"outputs": [],
"source": [
"def wav2mel(filename):\n",
" \"\"\"\n",
" Function to load an audio file and generate/return mel specs:\n",
" Args:\n",
" filename: Full path of the audio file.\n",
" Returns:\n",
" mels: Corresponding mel spectrogram.\n",
" \"\"\"\n",
" wav_, _ = librosa.load(filename, sr=sr) # load() returns an (audio data, sample rate) tuple\n",
" mels = librosa.feature.melspectrogram(\n",
" y=wav_,\n",
" sr=sr,\n",
" n_fft=n_fft,\n",
" hop_length=hop_length,\n",
" win_length=win_length,\n",
" window=window,\n",
" n_mels=n_mels\n",
" )\n",
" mels = librosa.power_to_db(mels, ref=np.max)\n",
" return mels"
]
},
{
"cell_type": "markdown",
"id": "8546edb5",
"metadata": {},
"source": [
"We also need a function to convert mels to MFCC, we will use Librosa's [mfcc](https://librosa.org/doc/main/generated/librosa.feature.mfcc.html) generation:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c1a0663d",
"metadata": {},
"outputs": [],
"source": [
"def mel2mfcc(mels):\n",
" mfcc = librosa.feature.mfcc(S=mels, n_mfcc=n_mfcc)\n",
" return mfcc"
]
},
{
"cell_type": "markdown",
"id": "6efac554",
"metadata": {},
"source": [
"For this tutorial, we have already generated mels from trained [FastPitch](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_fastpitch) and [RAD-TTS](https://github.com/NVIDIA/radtts) models, and we have the ground truth sample that they correspond to. Let's first download the tarball with the data and expand it."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "156290a9",
"metadata": {},
"outputs": [],
"source": [
"# Untar the example files.\n",
"!wget https://tts-tutorial-data.s3.us-east-2.amazonaws.com/MCD_DTW_examples.tar\n",
"!tar -xvf MCD_DTW_examples.tar"
]
},
{
"cell_type": "markdown",
"id": "7aeabe75",
"metadata": {},
"source": [
"Now we can generate a mel spectrogram for the ground truth audio, load mel specs for generated audio and generate MFCC(Mel frequency cepstral coefficient) for both."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d1a71f88",
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"## Generate spectrograms\n",
"gt_mels = wav2mel(\"MCD_DTW/gt/sample_0.wav\")\n",
"synt_mels = np.load(\"MCD_DTW/fastpitch/mels/mels_0.npy\")\n",
"\n",
"## Generate MFCCs\n",
"gt_mfcc = mel2mfcc(gt_mels)\n",
"synt_mfcc = mel2mfcc(synt_mels)"
]
},
{
"cell_type": "markdown",
"id": "faa5df64",
"metadata": {},
"source": [
"### Visualizing the Audio\n",
"\n",
"Let's first listen to the ground truth and synthesized audio."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "08f08b89",
"metadata": {},
"outputs": [],
"source": [
"print(\"Ground truth audio:\")\n",
"audio_gt, sr_gt = librosa.load(\"MCD_DTW/gt/sample_0.wav\")\n",
"ipd.display(ipd.Audio(audio_gt, rate=sr_gt))\n",
"\n",
"print(\"Synthesized audio:\")\n",
"audio_fp, sr_fp = librosa.load(\"MCD_DTW/fastpitch/audio/sample_0.wav\")\n",
"ipd.display(ipd.Audio(audio_fp, rate=sr_fp))"
]
},
{
"cell_type": "markdown",
"id": "21b5c780",
"metadata": {},
"source": [
"Now, let's take a look at the mel spectrograms."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "550d9b12",
"metadata": {},
"outputs": [],
"source": [
"# Visualize mel spectrograms\n",
"fig, ax = plt.subplots(2, 1, figsize=(12,10))\n",
"\n",
"_ = ax[0].pcolormesh(gt_mels, cmap='viridis')\n",
"_ = ax[0].set_title(\"Ground Truth Mel Spectrogram\")\n",
"\n",
"_ = ax[1].pcolormesh(synt_mels, cmap='viridis')\n",
"_ = ax[1].set_title(\"FastPitch Synthesized Mel Spectrogram\")"
]
},
{
"cell_type": "markdown",
"id": "93c4c301",
"metadata": {},
"source": [
"## Calculating the DTW matrix\n",
"\n",
"Dynamic time warping finds the most optimal path to align two sequences of different length, and in doing so measures the similarity between the two time series that are not in sync.\n",
"\n",
"DTW uses dynamic programming to **calculate the cost of every alignment path and chooses the path with least accumulated cost**. The value of accumulated cost matrix $D$ at index $(x_a, y_b)$ is the minimum distance between the points, where $x$ and $y$ are two time series. Formally, the cost matrix can be defined as:\n",
" \n",
"$D(a,b) = min(D(a-1, b), D(a, b-1), D(a-1, b-1)) + c(x_{a}, y_{b})$ \n",
"$D(1, b) = \\sum(c(1, y_{b}))$ \n",
"$D(a, 1) = \\sum(c(x_{a}, 1))$ \n",
" \n",
"Where ***x*** and ***y*** are the audio signals and ***c*** is the log cost function we have defined.\n",
"\n",
"We will use [the DTW function from Librosa](https://librosa.org/doc/main/generated/librosa.sequence.dtw.html) on MFCC. It returns the DTW accumulated cost matrix and DTW optimum path.\n",
"\n",
"In the following function, we define the cost function for DTW to pass in."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8a4adae2",
"metadata": {},
"outputs": [],
"source": [
"## Define the cost function for calculating DTW\n",
"def log_spec_dB_dist(x, y):\n",
" log_spec_dB_const = 10.0 / math.log(10.0) * math.sqrt(2.0)\n",
" diff = x - y\n",
" return log_spec_dB_const * math.sqrt(np.inner(diff, diff))"
]
},
{
"cell_type": "markdown",
"id": "ca8c6974",
"metadata": {},
"source": [
"Now we can run Librosa's `dtw()`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0c4c5338",
"metadata": {},
"outputs": [],
"source": [
"# dtw_cost: Shape (N, M), where N and M are the lengths of gt_mfcc, synt_mfcc respectively. Cost matrix.\n",
"# dtw_min_path: Shape (N, 2). Pairs of coordinates with of the min-cost path from bottom-right to top-left.\n",
"dtw_cost, dtw_min_path = librosa.sequence.dtw(gt_mfcc, synt_mfcc, metric=log_spec_dB_dist)"
]
},
{
"cell_type": "markdown",
"id": "913b6548",
"metadata": {},
"source": [
"Reduction of the DTW matrix can be done by either taking the mean of the entire cost matrix or averaging the DTW cost for the minimum cost path per frame. We will use the DTW cost along the min cost path."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6c85ab38",
"metadata": {},
"outputs": [],
"source": [
"# First, sum up the costs over the path\n",
"path_cost_matrix = dtw_cost[dtw_min_path[:, 0], dtw_min_path[:, 1]]\n",
"path_cost = np.sum(path_cost_matrix)\n",
"\n",
"# Average over path length\n",
"path_length = dtw_min_path.shape[0]\n",
"reduced_dtw_cost = path_cost/path_length\n",
"\n",
"# Average over number of frames\n",
"frames = synt_mels.shape[1]\n",
"mcd = reduced_dtw_cost/frames\n",
"\n",
"print(f\"MCD_DTW is: {mcd}\")"
]
},
{
"cell_type": "markdown",
"id": "24801d61",
"metadata": {},
"source": [
"## MCD DTW Usecase\n",
"\n",
"MCD is a very useful metric to **compare the convergence of two models**. Therefore in this section, we will calculate the average MCD for audio files generated by two different models: FastPitch and RAD-TTS.\n",
"\n",
"But first, let's put the calculations we have just performed in functions for better readability and reusability.\n",
"\n",
"Here are the helper functions for getting the average cost of DTW along a path."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e4181d60",
"metadata": {},
"outputs": [],
"source": [
"def extract_path_cost(D, wp):\n",
" \"\"\"\n",
" Get the path cost from D(cost matrix), wp (warped path)\n",
" Returns the sum of the path cost \n",
" \"\"\"\n",
" path_cost = D[wp[:, 0], wp[:, 1]]\n",
" return np.sum(path_cost)\n",
"\n",
"def extract_frame_avg_path_cost(D, wp):\n",
" \"\"\"\n",
" Get the average path cost over the length of the given path\n",
" \"\"\"\n",
" path_cost = extract_path_cost(D, wp)\n",
" path_length = wp.shape[0]\n",
" frame_avg_path_cost = path_cost / float(path_length)\n",
" return frame_avg_path_cost"
]
},
{
"cell_type": "markdown",
"id": "08ca0f47",
"metadata": {},
"source": [
"And a function for calculating MCD for a synthetic mel spectrogram:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "99800a97",
"metadata": {},
"outputs": [],
"source": [
"def cal_mcd(gt_mfcc, synt_mfcc, cost_function, dtw_type='path_cost'):\n",
" \"\"\"\n",
" Calculates MCD between a ground truth and synthetic mel.\n",
" \"\"\"\n",
" frames = synt_mfcc.shape[1]\n",
" path_cost = 0\n",
" \n",
" # dynamic time warping for MCD\n",
" dtw_cost, dtw_min_path = librosa.sequence.dtw(gt_mfcc, synt_mfcc, metric=cost_function)\n",
" if dtw_type == 'mean':\n",
" path_cost = np.mean(dtw_cost)\n",
" else:\n",
" path_cost = extract_frame_avg_path_cost(dtw_cost, dtw_min_path)\n",
" \n",
" mcd = path_cost / frames\n",
" \n",
" return mcd, frames"
]
},
{
"cell_type": "markdown",
"id": "f1489ecf",
"metadata": {},
"source": [
"This last function will calculate the MCDs for all the synthetic mels in a directory.\n",
"\n",
"We'll get a better comparison if we compute the MCDs of (much) more than one synthesized spectrogram per model! This function streamlines the process by letting us pass in (1) a directory with all the synthesized mel spectrograms from a TTS model, and (2) a directory with all the ground-truth audio.\n",
"\n",
"The function assumes that, after sorting the contents of each directory, wav *#i* in the ground truth directory will correspond to mel *#i* in the synthesized mel directory. In our case, the directories for ground truth and synthesized FastPitch mels look like this:\n",
"\n",
"```bash\n",
"% ls gt/\n",
"sample_0.wav sample_1.wav sample_2.wav sample_3.wav sample_4.wav\n",
"% ls fastpitch/mels/\n",
"mels_0.npy mels_1.npy mels_2.npy mels_3.npy mels_4.npy\n",
"```\n",
"\n",
"We only have ten files each to compare as a toy example, for a full evaluation you will likely want more!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "07bb51df",
"metadata": {},
"outputs": [],
"source": [
"def cal_mcd_dir(synt_dir, gt_dir):\n",
" \"\"\"\n",
" Calculate MCDs for pairs of synthetic and ground truth audio files.\n",
" \n",
" Args:\n",
" synt_dir: Path to the directory that contains all the synthetic mels.\n",
" gt_dir: Path to the directory that contains all the ground truth wavs.\n",
" These should correspond 1:1 with the mels in synt_dir.\n",
" \n",
" Returns:\n",
" List of MCDs (where length is the number of files in each directory)\n",
" \"\"\"\n",
" mcds = []\n",
" \n",
" synt_filenames = os.listdir(synt_dir)\n",
" synt_filepaths = [os.path.join(synt_dir, filename) for filename in synt_filenames]\n",
" synt_filepaths.sort()\n",
" gt_filenames = os.listdir(gt_dir)\n",
" gt_filepaths = [os.path.join(gt_dir, filename) for filename in gt_filenames]\n",
" gt_filepaths.sort()\n",
"\n",
" for synt_melname, gt_audio in zip(synt_filepaths, gt_filepaths):\n",
" synt_mels = np.load(synt_melname)\n",
" synt_mfcc = mel2mfcc(synt_mels)\n",
" \n",
" gt_mels = wav2mel(gt_audio)\n",
" gt_mfcc = mel2mfcc(gt_mels)\n",
" \n",
" mcd, _ = cal_mcd(gt_mfcc, synt_mfcc, log_spec_dB_dist)\n",
" mcds.append(mcd)\n",
" return mcds"
]
},
{
"cell_type": "markdown",
"id": "fc295283",
"metadata": {
"scrolled": true
},
"source": [
"### Calculate MCD DTW on Synthesized Files from Each Model\n",
"\n",
"We can now calculate the MCD DTW for the synthesized FastPitch mels (compared to the ground truth) as well as for the synthesized RAD-TTS mels (ditto), then compare them. This will take a few seconds."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2bc8da46",
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"%%capture --no-display\n",
"mels_dir_m1 = \"MCD_DTW/fastpitch/mels/\"\n",
"mels_dir_m2 = \"MCD_DTW/radtts/mels/\"\n",
"mels_dir_gt = \"MCD_DTW/gt/\"\n",
"\n",
"mcds_m1 = cal_mcd_dir(mels_dir_m1, mels_dir_gt) # FastPitch\n",
"mcds_m2 = cal_mcd_dir(mels_dir_m2, mels_dir_gt) # RAD-TTS"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6bd1ef33",
"metadata": {},
"outputs": [],
"source": [
"print(f\"Average MCD for Model 1 (FastPitch) is: {sum(mcds_m1)/len(mcds_m1):.2f}\")\n",
"print(f\"Average MCD for Model 2 (RAD-TTS) is: {sum(mcds_m2)/len(mcds_m2):.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "f9ef5363",
"metadata": {},
"source": [
"We're measuring divergence from the ground truth for each of these, so **lower is better**!"
]
},
{
"cell_type": "markdown",
"id": "71e998a9",
"metadata": {},
"source": [
"### Plotting MCD"
]
},
{
"cell_type": "markdown",
"id": "7f81c5b5",
"metadata": {},
"source": [
"We can also plot the MCD DTW values for both models, with the model with a lower MCD DTW value being closer to ground truth."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3f8d71bf",
"metadata": {},
"outputs": [],
"source": [
"x_axis = np.linspace(1, len(mcds_m1), len(mcds_m1)) ## Define an x axis for for plotting in matplotlib\n",
"plt.plot(x_axis, mcds_m1, label=\"FastPitch\")\n",
"plt.plot(x_axis, mcds_m2, label=\"RAD-TTS\")\n",
"\n",
"plt.title(\"MCD DTW value for each file\")\n",
"plt.ylabel(\"MCD_DTW value\")\n",
"\n",
"plt.grid()\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "fd4eb0a6",
"metadata": {},
"source": [
"From the graph above, we can see that in general the value of MCD is greater for RAD-TTS mel spectrograms than for the FastPitch ones. This is also reflected in the average MCD value for both models that we computed before.\n",
"\n",
"Therefore, we can conclude that FastPitch has better convergence than RAD-TTS. However, we cannot evaluate the quality of audio generated by these models using MCD alone! **MCD is a great tool for testing model convergence, but generated audio may have pronunciation and quality artifacts.** Therefore MCD evaluation should be followed by a MOS (Mean Opinion Score) and CMOS (Comparative Mean Opinion Score) evaluation."
]
},
{
"cell_type": "markdown",
"id": "d66a52d5",
"metadata": {},
"source": [
"## Additional NeMo Resources\n",
"\n",
"If you are unsure where to begin for training a TTS model, you may want to start with the [FastPitch and Mixer-TTS Training notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_MixerTTS_Training.ipynb) or the [NeMo TTS Primer notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/NeMo_TTS_Primer.ipynb). For fine-tuning, there is also the [FastPitch Fine-Tuning notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb).\n",
"\n",
"For some guidance on how to load a trained model and perform inference to generate mels or waveforms, check out how it's done in the [Inference notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Inference_ModelSelect.ipynb). Important functions to know are include `from_pretrained()` (if loading from an NGC model) and `restore_from()` (if loading a `.nemo` file). See the [NeMo Primer notebook](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/00_NeMo_Primer.ipynb) for more general information about model training, saving, and loading."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
tutorials/tts/NeMo_TTS_Primer.ipynb
+2069
View File
File diff suppressed because it is too large Load Diff
tutorials/tts/Pronunciation_customization.ipynb
+306
View File
@@ -0,0 +1,306 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"http://developer.download.nvidia.com/notebooks/dlsw-notebooks/riva_tts_tts-python-basics/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
"\n",
"# How do I customize TTS pronunciations?\n",
"\n",
"This tutorial walks you through the basics of NeMo TTS pronunciation customization. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"You can either run this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
"Instructions for setting up Colab are as follows:\n",
"1. Open a new Python 3 notebook.\n",
"2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
"3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
"4. Run this cell to set up dependencies.\n",
"\"\"\"\n",
"\n",
"BRANCH = 'main'\n",
"# # If you're using Google Colab and not running locally, uncomment and run this cell.\n",
"# !apt-get install sox libsndfile1 ffmpeg\n",
"# !pip install wget text-unidecode \n",
"# !python -m pip install \"nemo_toolkit[all] @ git+https://github.com/NVIDIA-NeMo/Speech.git@$BRANCH\"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Grapheme-to-phoneme (G2P) Overview\n",
"\n",
"Modern **text-to-speech** (TTS) models can learn pronunciations from raw text input and its corresponding audio data.\n",
"Sometimes, however, it is desirable to customize pronunciations, for example, for domain-specific terms. As a result, many TTS systems use grapheme and phonetic input during training to directly access and correct pronunciations at inference time.\n",
"\n",
"\n",
"[The International Phonetic Alphabet (IPA)](https://en.wikipedia.org/wiki/International_Phonetic_Alphabet) and [ARPABET](https://en.wikipedia.org/wiki/ARPABET) are the most common phonetic alphabets. \n",
"\n",
"There are two ways to customize pronunciations:\n",
"\n",
"1. pass phonemes as an input to the TTS model, note that the request-time overrides are best suited for one-off adjustments\n",
"2. configure TTS model with the desired domain-specific terms using custom phonetic dictionary\n",
"\n",
"Both methods require users to convert graphemes into phonemes (G2P). \n",
"\n",
"#### All words for G2P purposes could be divided into the following groups:\n",
"* *known* words - words that are present in the model's phonetic dictionary\n",
"* *out-of-vocabulary (OOV)* words - words that are missing from the model's phonetic dictionary. \n",
"* *[heteronyms](https://en.wikipedia.org/wiki/Heteronym_&#40;linguistics&#41;)* - words with the same spelling but different pronunciations and/or meanings, e.g., *bass* (the fish) and *bass* (the musical instrument).\n",
"\n",
"#### Important NeMo flags:\n",
"* `your_spec_generator_model.vocab.g2p.phoneme_dict` - phoneme dictionary that maps words to their phonetic transcriptions, e.g., [ARPABET-based CMU Dictionary](https://raw.githubusercontent.com/NVIDIA/NeMo/stable/scripts/tts_dataset_files/cmudict-0.7b_nv22.10) or [IPA-based CMU Dictionary](https://github.com/NVIDIA/NeMo/blob/stable/scripts/tts_dataset_files/ipa_cmudict-0.7b_nv23.01.txt)\n",
"* `your_spec_generator_model.vocab.g2p.heteronyms` - list of the model's heteronyms, grapheme form of these words will be used even if the word is present in the phoneme dictionary.\n",
"* `your_spec_generator_model.vocab.g2p.ignore_ambiguous_words`: if is set to **True**, words with more than one phonetic representation in the pronunciation dictionary are ignored. This flag is relevant to the words with multiple valid phonetic transcriptions in the dictionary that are not in `your_spec_generator_model.vocab.g2p.heteronyms` list.\n",
"* `your_spec_generator_model.vocab.phoneme_probability` - phoneme probability flag in the Tokenizer and the same from in the G2P module: `your_spec_generator_model.vocab.g2p.phoneme_probability` ([0, 1]). If a word is present in the phoneme dictionary, we still want our TTS model to see graphemes and phonemes during training to handle OOV words during inference. The `phoneme_probability` determines the probability of an unambiguous dictionary word appearing in phonetic form during model training, `(1 - phoneme_probability)` is the probability of the graphemes. This flag is set to `1` in the parse() method during inference.\n",
"\n",
"To ensure the desired pronunciation, we need to add a new entry to the model's phonetic dictionary. If the target word is already in the dictionary, we need to remove the default pronunciation so that only the target pronunciation is present. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Default G2P\n",
"\n",
"Below we show how to analyze default G2P output of the NeMo models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import nemo.collections.tts as nemo_tts\n",
"import IPython.display as ipd\n",
"\n",
"# Load mel spectrogram generator\n",
"spec_generator = nemo_tts.models.FastPitchModel.from_pretrained(\"tts_en_fastpitch_ipa\").eval()\n",
"# Load vocoder\n",
"vocoder = nemo_tts.models.HifiGanModel.from_pretrained(model_name=\"tts_en_hifigan\").eval()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# helper functions\n",
"\n",
"def generate_audio(input_text):\n",
" # parse() sets phoneme probability to 1, i.e. dictionary phoneme transcriptions are used for known words\n",
" parsed = spec_generator.parse(input_text)\n",
" spectrogram = spec_generator.generate_spectrogram(tokens=parsed)\n",
" audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)\n",
" display(ipd.Audio(audio.detach().to('cpu').numpy(), rate=22050))\n",
" \n",
"def display_postprocessed_text(text):\n",
" # to use dictionary entries for known words, not needed for generate_audio() as parse() handles this\n",
" spec_generator.vocab.phoneme_probability = 1\n",
" spec_generator.vocab.g2p.phoneme_probability = 1\n",
" print(f\"Input before tokenization: {' '.join(spec_generator.vocab.g2p(text))}\\n\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"text = \"paracetamol can help reduce fever.\"\n",
"generate_audio(text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Expected results if you run the tutorial:\n",
"<audio controls src=\"https://github.com/NVIDIA/NeMo/raw/main/tutorials/tts/audio_samples/default_ipa.wav\" type=\"audio/ogg\"></audio> \n",
"\n",
"\n",
"During preprocessing, unambiguous dictionary words are converted to phonemes, while OOV and words with multiple entries are kept as graphemes. For example, **paracetamol** is missing from the phoneme dictionary, and **can** has 2 forms."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"display_postprocessed_text(text)\n",
"for word in [\"paracetamol\", \"can\"]:\n",
" word = word.upper()\n",
" phoneme_forms = spec_generator.vocab.g2p.phoneme_dict[word]\n",
" print(f\"Number of phoneme forms for wordPhoneme forms for '{word}': {len(phoneme_forms)} -- {phoneme_forms}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Input customization\n",
"\n",
"One can pass phonemes directly as input to the model to customize pronunciation.\n",
"\n",
"Let's replace the word **paracetamol** with the desired phonemic transcription in our example by adding vertical bars around each phone, e.g., `ˌpæɹəˈsitəmɔl` -> `|ˌ||p||æ||ɹ||ə||ˈ||s||i||t||ə||m||ɔ||l|`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(f\"Original text: {text}\")\n",
"\n",
"new_pronunciation = \"ˌpæɹəˈsitəmɔl\"\n",
"phoneme_form = \"\".join([f\"|{s}|\" for s in new_pronunciation])\n",
"text_with_phonemes = text.replace(\"paracetamol\", phoneme_form)\n",
"print(f\"Text with phonemes: {text_with_phonemes}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"generate_audio(text_with_phonemes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Expected results if you run the tutorial:\n",
"<audio controls src=\"https://github.com/NVIDIA/NeMo/raw/main/tutorials/tts/audio_samples/phonemes_as_input.wav\" type=\"audio/ogg\"></audio> \n",
"\n",
"\n",
"## Dictionary customization\n",
"\n",
"Below we show how to customize phonetic dictionary for NeMo TTS models. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's add a new entry to the dictionary for the word **paracetamol**. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# we download IPA-based CMU Dictionary and add a custom entry for the target word\n",
"ipa_cmu_dict = \"ipa_cmudict-0.7b_nv23.01.txt\"\n",
"if os.path.exists(ipa_cmu_dict):\n",
" ! rm $ipa_cmu_dict\n",
"\n",
"! wget https://raw.githubusercontent.com/NVIDIA/NeMo/main/scripts/tts_dataset_files/$ipa_cmu_dict\n",
"\n",
"with open(ipa_cmu_dict, \"a\") as f:\n",
" f.write(f\"PARACETAMOL {new_pronunciation}\\n\")\n",
" \n",
"! tail $ipa_cmu_dict"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# let's now use our updated dictionary as the model's phonetic dictionary\n",
"spec_generator.vocab.g2p.replace_dict(ipa_cmu_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Paracetamol** is no longer an OOV, and the model uses the phonetic form we provided:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"display_postprocessed_text(text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, let's use the new phoneme dictionary for synthesis."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"generate_audio(text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Expected results if you run the tutorial:\n",
"<audio controls src=\"https://github.com/NVIDIA/NeMo/raw/main/tutorials/tts/audio_samples/new_dict_entry.wav\" type=\"audio/ogg\"></audio> "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Resources\n",
"* [TTS pipeline customization](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-custom.html#tts-pipeline-configuration)\n",
"* [Overview of TTS in NeMo](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/NeMo_TTS_Primer.ipynb)\n",
"* [G2P models in NeMo](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/tts/g2p.html)\n",
"* [Riva TTS documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-overview.html)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "38",
"language": "python",
"name": "38"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
tutorials/tts/audio_samples/default_ipa.wav
BIN
View File
Binary file not shown.
tutorials/tts/audio_samples/new_dict_entry.wav
BIN
View File
Binary file not shown.
tutorials/tts/audio_samples/phonemes_as_input.wav
BIN
View File
Binary file not shown.
tutorials/tts/images/fastpitch-pitch.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 82 KiB

tutorials/tts/images/fastpitch_diagram.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

tutorials/tts/images/hifi_gan_discriminator_diagram.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

tutorials/tts/images/sampling_rate.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.1 KiB

tutorials/tts/images/tacotron2_diagram.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 197 KiB

tutorials/tts/images/tts_pipeline.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 75 KiB

tutorials/tts/images/wavenet_diagram.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB