wehub/nvidia-nemo--speech
1
0
Fork 0
Code Issues Pull Requests Actions 430 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Create PR to main with cherry-pick from release / cherry-pick (push) Failing after 0s
Details
CICD NeMo / pre-flight (push) Failing after 0s
Details
CICD NeMo / configure (push) Has been skipped
Details
Build, validate, and release Neural Modules / pre-flight (push) Failing after 1s
Details
CICD NeMo / code-linting (push) Has been skipped
Details
Build, validate, and release Neural Modules / release (push) Has been skipped
Details
Build, validate, and release Neural Modules / release-summary (push) Has been cancelled
Details
CICD NeMo / cicd-test-container-build (push) Has been cancelled
Details
CICD NeMo / cicd-import-tests (push) Has been cancelled
Details
CICD NeMo / L0_Setup_Test_Data_And_Models (push) Has been cancelled
Details
CICD NeMo / cicd-main-unit-tests (push) Has been cancelled
Details
CICD NeMo / cicd-main-speech (push) Has been cancelled
Details
CICD NeMo / Nemo_CICD_Test (push) Has been cancelled
Details
CICD NeMo / Coverage (e2e) (push) Has been cancelled
Details
CICD NeMo / Coverage (unit-test) (push) Has been cancelled
Details
CodeQL / Analyze (python) (push) Has been cancelled
Details
CICD NeMo / cicd-wait-in-queue (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:28:58 +08:00
commit ba4be087d5
2316 changed files with 2668701 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/_static/css/custom.css
+381
View File
@@ -0,0 +1,381 @@
@import url("theme.css");
body {
font-size: 100%;
font-family: 'NVIDIA Sans', sans-serif;
}
/* Width of template */
.wy-nav-content {
max-width: 1200px !important;
}
/* Standard Text Formatting */
h1 {
color: #76b900;
text-align: center;
/* background-color: #ffffff; */
}
h2 {
color: #ffffff;
/* background-color: #ffffff; */
/* #76b900 */
Padding: 5px;
}
h3 {
padding-top: 0px;
border-top: solid 3px #000000;
/* #76b900 */
border-bottom: solid 3px #000000;
/* #76b900 */
}
p {
margin-bottom: 24px;
}
/* Link Colors */
/*
a {
color: #76b900;
}
/*
/*
a:visited {
color: #218219;
}
*/
.container-xl {
margin-right: unset;
margin-left: unset;
}
section {
overflow-x: auto;
}
/* ----------------------------------------------TABLES--------------------------------------- */
section table {
overflow-x: auto;
display: block;
}
table {
font-size: small;
}
/* Table head Color */
thead td {
background-color: #333333 !important;
}
.row-odd p {
/*padding-bottom: 0px;*/
/*margin-bottom: 0px;*/
}
/* even rows*/
.row-even tr {
background-color: #e5f1e6 !important;
}
/* odd rows*/
.wy-table-responsive table tr {
background-color: #ffffff !important;
}
.wy-table-responsive table td {
white-space: normal;
}
/* Removes bottom margin in tables*/
.rst-content .line-block {
margin-bottom: 0px;
}
.wy-table-responsive {
overflow: visible !important;
}
/* reduces the size of text in multiline table columns. */
.rst-content table.docutils td {
font-size: 80%;
}
.rst-content dl:not(.docutils) dt {
background-color: inherit;
color: #000000;
border-top: solid 0px #000000;
}
.rst-content dl:not(.docutils) dt:before {
color: #333333;
}
.rst-content .line-block {
margin-bottom: 0px;
}
.wy-side-nav-search,
.wy-nav-top {
background-color: #000000;
padding: 0;
}
.wy-side-nav-search img {
padding: 0px;
padding: 0px 0px;
margin-bottom: 0;
}
.wy-side-nav-search input[type=text] {
border-radius: 0px;
}
.wy-menu-vertical p.caption {
color: #76b900;
}
.wy-side-nav-search>a img.logo,
.wy-side-nav-search .wy-dropdown>a img.logo {
margin: 0px 0px 0px 0px;
}
.wy-nav-content {
margin: 0;
min-height: 100%;
height: 100%;
background: #ffffff;
}
/* NVIDIA green hover on table rows */
table tbody tr:hover,
table.table tbody tr:hover,
table.docutils tbody tr:hover,
.pst-scrollable-table-container table tbody tr:hover,
.rst-content table tbody tr:hover {
background-color: rgba(118, 185, 0, 0.15) !important;
}
/* List (numbered, bulleted) padding Fix */
.wy-plain-list-decimal li {
margin-top: -6px;
margin-bottom: -6px;
}
.rst-content .section ol.loweralpha {
margin-top: -6px;
margin-bottom: 12px;
}
.wy-plain-list-disc,
.rst-content .toctree-wrapper ul,
article ul {
margin-top: 0px !important;
margin-bottom: 12px;
}
/* Alert Boxes */
/* Background color of Alert Box Title */
.rst-content .section ul {
margin-top: -12px;
margin-bottom: 16px;
}
.wy-alert.wy-alert-info .wy-alert-title,
.rst-content .note .wy-alert-title,
.rst-content .wy-alert-info.attention .wy-alert-title,
.rst-content .wy-alert-info.caution .wy-alert-title,
.rst-content .wy-alert-info.danger .wy-alert-title,
.rst-content .wy-alert-info.error .wy-alert-title,
.rst-content .wy-alert-info.hint .wy-alert-title,
.rst-content .wy-alert-info.important .wy-alert-title,
.rst-content .wy-alert-info.tip .wy-alert-title,
.rst-content .wy-alert-info.warning .wy-alert-title,
.rst-content .seealso .wy-alert-title,
.rst-content .wy-alert-info.admonition-todo .wy-alert-title,
.rst-content .wy-alert-info.admonition .wy-alert-title,
.wy-alert.wy-alert-info .rst-content .admonition-title,
.rst-content .wy-alert.wy-alert-info .admonition-title,
.rst-content .note .admonition-title,
.rst-content .wy-alert-info.attention .admonition-title,
.rst-content .wy-alert-info.caution .admonition-title,
.rst-content .wy-alert-info.danger .admonition-title,
.rst-content .wy-alert-info.error .admonition-title,
.rst-content .wy-alert-info.hint .admonition-title,
.rst-content .wy-alert-info.important .admonition-title,
.rst-content .wy-alert-info.tip .admonition-title,
.rst-content .wy-alert-info.warning .admonition-title,
.rst-content .seealso .admonition-title,
.rst-content .wy-alert-info.admonition-todo .admonition-title,
.rst-content .wy-alert-info.admonition .admonition-title {
background: #76b900;
}
/* Background and Font Color of Alert Box Main Body*/
.wy-alert.wy-alert-info,
.rst-content .note,
.rst-content .wy-alert-info.attention,
.rst-content .wy-alert-info.caution,
.rst-content .wy-alert-info.danger,
.rst-content .wy-alert-info.error,
.rst-content .wy-alert-info.hint,
.rst-content .wy-alert-info.important,
.rst-content .wy-alert-info.tip,
.rst-content .wy-alert-info.warning,
.rst-content .seealso,
.rst-content .wy-alert-info.admonition-todo,
.rst-content .wy-alert-info.admonition {
background: #333333;
color: #999999;
}
.section {
margin-top: 50px;
}
/* Logo */
.navbar-brand-box {
background-color: #ffffff;
}
/* ---------------------------------------------- Media Queries --------------------------------------- */
@media (min-width: 1200px) {
.container-xl {
max-width: 100%;
}
}
@media (min-width: none) {
body {
font-size: 18px;
}
#site-navigation nav ul.nav {
font-size: 18px;
}
#site-navigation nav.bd-links p {
font-size: 18px;
}
#site-navigation {
width: 350px;
}
.toc-h2 {
font-size: 18px;
}
.toc-h3 {
font-size: 1rem;
}
.toc-h4 {
font-size: 0.85rem;
}
.header-article .bd-toc {
font-size: 18px;
}
#main-content>div {
margin-left: 10%;
margin-right: 10%;
}
}
/* ---------------------------------------------- NVIDIA Sans --------------------------------------- */
:root {
--md-text-font: "NVIDIA Sans";
/* --md-code-font: "NVIDIA Sans"; */
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/5/2/52891dda673228d54e5d57bf1e4a3880d4b22405.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/e/0/e090b7dda7a582522c7f9045c6ce949cce60134f.woff) format("woff");
font-weight: 300;
font-style: normal;
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/a/1/a107baabcbf6b241099122336bce7429bcfd377a.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/3/a/3a6060a4e3bce70e5552ba0de8af4b22c6cf9144.woff) format("woff");
font-weight: 300;
font-style: italic;
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/9/9/9920d2b172b01d92fc9c1c0e521dcf45b59c47c3.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/6/c/6c7d947928a7e4ef3e80ed409bef6c243f2148cb.woff) format("woff");
font-weight: 400;
font-style: normal;
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/e/8/e8e63fe1244372cd942d957f44a5616a1eba0644.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/0/f/0f1fb2af0283ab09d36e7097bb07d895c3228f12.woff) format("woff");
font-weight: 400;
font-style: italic;
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/7/9/79d3c513a9cd72c59f65354f39f89ca52dc17dd2.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/2/5/2581ac533f5d01f4985d8a7245b0766b4630ced8.woff) format("woff");
font-weight: 500;
font-style: normal;
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/3/9/39d9ef1ee9770dd503f19bb2ace2fdb4eff3bb50.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/7/b/7bb5d5e2e71b2e13c8098b2e67c0a0ed9258e6c7.woff) format("woff");
font-weight: 500;
font-style: italic;
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/0/5/05276a55a43eb3f74981ec1e93252727afcd9d16.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/9/c/9cfec7ed941b06564aa4d5ca14610e81542d070f.woff) format("woff");
font-weight: 700;
font-style: normal;
}
@font-face {
font-family: "NVIDIA Sans";
src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/a/e/aebd14d09ba56f541e1b8735fb051e33710f9ae7.woff2) format("woff2"),
url(https://aws1.discourse-cdn.com/nvidia/original/3X/e/d/edbdabef43acc5c12e84a94baaa5542c9404cfeb.woff) format("woff");
font-weight: 700;
font-style: italic;
}
docs/source/_static/js/pk_scripts.js
+19
View File
@@ -0,0 +1,19 @@
document.addEventListener("DOMContentLoaded", function () {
var params = window.location.search.substring(1).split("&").reduce(function (params, param) {
if (!param) {
return params;
}
var values = param.split("=");
var name = values[0];
var value = values[1];
params[name] = value;
return params;
}, {});
var form = document.getElementById("feedback-form");
for (var name in params) {
var input = form.querySelector("[name=" + name + "]");
input.value = params[name];
}
});
docs/source/apis.rst
+48
View File
@@ -0,0 +1,48 @@
:orphan:
=========
NeMo APIs
=========
You can learn more about the underlying principles of the NeMo codebase in this section.
The `NeMo Toolkit codebase <https://github.com/NVIDIA/NeMo>`__ is composed of a `core <https://github.com/NVIDIA/NeMo/tree/main/nemo/core>`__ section which contains the main building blocks of the framework, and various `collections <https://github.com/NVIDIA/NeMo/tree/main/nemo/collections>`__ which help you
build specialized AI models.
You can learn more about aspects of the NeMo "core" by following the links below:
.. toctree::
:maxdepth: 1
:name: core
:titlesonly:
core/core
core/neural_modules
core/exp_manager
core/neural_types
core/adapters/intro
You can learn more about aspects of the NeMo APIs by following the links below:
.. toctree::
:maxdepth: 1
:name: API
:titlesonly:
core/api
common/intro
asr/api
tts/api
audio/api
Alternatively, you can jump straight to the documentation for the individual collections:
* :doc:`Automatic Speech Recognition (ASR) <../asr/intro>`
* :doc:`Text-to-Speech (TTS) <../tts/intro>`
* :doc:`Audio Processing <../audio/intro>`
* :doc:`SpeechLM2 <../speechlm2/intro>`
docs/source/asr/all_chkpt.rst
+402
View File
@@ -0,0 +1,402 @@
:orphan:
All Checkpoints
===============
English
^^^^^^^
.. csv-table::
:file: data/benchmark_en.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
German
^^^^^^
.. csv-table::
:file: data/benchmark_de.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Spanish
^^^^^^^
.. csv-table::
:file: data/benchmark_es.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
French
^^^^^^
.. csv-table::
:file: data/benchmark_fr.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Arabic
^^^^^^^
.. csv-table::
:file: data/benchmark_ar.csv
:align: left
:widths: 50,50
:header-rows: 1
------------------------------
Russian
^^^^^^^
.. csv-table::
:file: data/benchmark_ru.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Portuguese
^^^^^^^^^^
.. csv-table::
:file: data/benchmark_pt.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Belarusian
^^^^^^^^^^
.. csv-table::
:file: data/benchmark_be.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Japanese
^^^^^^^^
.. csv-table::
:file: data/benchmark_jp.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Armenian
^^^^^^^^
.. csv-table::
:file: data/benchmark_hy.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Georgian
^^^^^^^^
.. csv-table::
:file: data/benchmark_ka.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Kazakh
^^^^^^
.. csv-table::
:file: data/benchmark_kz.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Persian
^^^^^^^
.. csv-table::
:file: data/benchmark_fa.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Uzbek
^^^^^
.. csv-table::
:file: data/benchmark_uz.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Ukrainian
^^^^^^^^^
.. csv-table::
:file: data/benchmark_ua.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Polish
^^^^^^
.. csv-table::
:file: data/benchmark_pl.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Italian
^^^^^^^
.. csv-table::
:file: data/benchmark_it.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Croatian
^^^^^^^^
.. csv-table::
:file: data/benchmark_hr.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Esperanto
^^^^^^^^^
.. csv-table::
:file: data/benchmark_eo.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Kabyle
^^^^^^
.. csv-table::
:file: data/benchmark_kab.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Dutch
^^^^^
.. csv-table::
:file: data/benchmark_nl.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Catalan
^^^^^^^
.. csv-table::
:file: data/benchmark_ca.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Hindi
^^^^^^^
.. csv-table::
:file: data/benchmark_hi.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Marathi
^^^^^^^
.. csv-table::
:file: data/benchmark_mr.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Mandarin
^^^^^^^^
.. csv-table::
:file: data/benchmark_zh.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Czech
^^^^^
.. csv-table::
:file: data/benchmark_cs.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Bulgarian
^^^^^^^^^
.. csv-table::
:file: data/benchmark_bg.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Danish
^^^^^^
.. csv-table::
:file: data/benchmark_da.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Estonian
^^^^^^^^
.. csv-table::
:file: data/benchmark_et.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Finnish
^^^^^^^
.. csv-table::
:file: data/benchmark_fi.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Greek
^^^^^
.. csv-table::
:file: data/benchmark_el.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Hungarian
^^^^^^^^^
.. csv-table::
:file: data/benchmark_hu.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Latvian
^^^^^^^
.. csv-table::
:file: data/benchmark_lv.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Lithuanian
^^^^^^^^^^
.. csv-table::
:file: data/benchmark_lt.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Maltese
^^^^^^^
.. csv-table::
:file: data/benchmark_mt.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Romanian
^^^^^^^^^
.. csv-table::
:file: data/benchmark_ro.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Slovak
^^^^^^
.. csv-table::
:file: data/benchmark_sk.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Slovenian
^^^^^^^^^
.. csv-table::
:file: data/benchmark_sl.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Swedish
^^^^^^^
.. csv-table::
:file: data/benchmark_sv.csv
:align: left
:widths: 50,50
:header-rows: 1
-----------------------------
Kinyarwanda
^^^^^^^^^^^
.. csv-table::
:file: data/benchmark_rw.csv
:align: left
:widths: 50,50
:header-rows: 1
docs/source/asr/api.rst
+367
View File
@@ -0,0 +1,367 @@
NeMo ASR API
============
Model Classes
-------------
.. autoclass:: nemo.collections.asr.models.EncDecCTCModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecCTCModelBPE
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecRNNTModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecRNNTBPEModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecRNNTBPEModelWithPrompt
:show-inheritance:
:members: transcribe, set_inference_prompt, initialize_prompt_feature, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecHybridRNNTCTCBPEModelWithPrompt
:show-inheritance:
:members: transcribe, set_inference_prompt, initialize_prompt_feature, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecMultiTalkerRNNTBPEModel
:show-inheritance:
:members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecClassificationModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. autoclass:: nemo.collections.asr.models.EncDecSpeakerLabelModel
:show-inheritance:
:members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
.. _asr-api-modules:
Modules
-------
.. autoclass:: nemo.collections.asr.modules.ConvASREncoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConvASRDecoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.ConvASRDecoderClassification
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.SpeakerDecoder
:show-inheritance:
:members:
.. _conformer-encoder-api:
.. autoclass:: nemo.collections.asr.modules.ConformerEncoder
:show-inheritance:
:members:
.. _rnn-encoder-api:
.. autoclass:: nemo.collections.asr.modules.RNNEncoder
:show-inheritance:
:members:
.. _rnnt-decoder-api:
.. autoclass:: nemo.collections.asr.modules.RNNTDecoder
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.StatelessTransducerDecoder
:show-inheritance:
:members:
.. _rnnt-joint-api:
.. autoclass:: nemo.collections.asr.modules.RNNTJoint
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.SampledRNNTJoint
:show-inheritance:
:members:
Mixins
------
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRBPEMixin
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.mixins.transcription.TranscriptionMixin
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.mixins.transcription.TranscribeConfig
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.mixins.interctc_mixin.InterCTCMixin
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.mixins.multitalker_asr_mixins.SpeakerKernelMixin
:show-inheritance:
:members:
.. _asr-api-datasets:
Datasets
--------
Character Encoding Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToCharDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset
:show-inheritance:
:members:
Text-to-Text Datasets
~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.text_to_text.TextToTextDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.text_to_text.TextToTextIterableDataset
:show-inheritance:
:members:
Speaker-Tagged Datasets for Multitalker ASR models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.audio_to_text_lhotse_speaker.LhotseSpeechToTextSpkBpeDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_diar_label_lhotse.LhotseAudioToSpeechE2ESpkDiarDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.data_simulation.MultiSpeakerSimulator
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.data_simulation.RIRMultiSpeakerSimulator
:show-inheritance:
:members:
Subword Encoding Datasets
~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToBPEDataset
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset
:show-inheritance:
:members:
.. _asr-audio-preprocessors:
Audio Preprocessors
-------------------
.. autoclass:: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.AudioToMFCCPreprocessor
:show-inheritance:
:members:
.. _asr-api-audio-augmentors:
Audio Augmentors
----------------
.. autoclass:: nemo.collections.asr.modules.SpectrogramAugmentation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.modules.CropOrPadSpectrogramAugmentation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.SpeedPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TimeStretchPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.GainPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ImpulsePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ShiftPerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.NoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.WhiteNoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.RirAndNoisePerturbation
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TranscodePerturbation
:show-inheritance:
:members:
Miscellaneous Classes
---------------------
CTC Decoding
~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.submodules.ctc_decoding.CTCDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.ctc_decoding.CTCBPEDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.ctc_greedy_decoding.GreedyCTCInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.ctc_beam_decoding.BeamCTCInfer
:show-inheritance:
:members:
RNNT Decoding
~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_decoding.RNNTDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_decoding.RNNTBPEDecoding
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyRNNTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedRNNTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_beam_decoding.BeamRNNTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_beam_decoding.BeamBatchedRNNTInfer
:show-inheritance:
:members:
TDT Decoding
~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyTDTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedTDTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.tdt_beam_decoding.BeamTDTInfer
:show-inheritance:
:members:
.. autoclass:: nemo.collections.asr.parts.submodules.tdt_beam_decoding.BeamBatchedTDTInfer
:show-inheritance:
:members:
Hypotheses
~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis
:show-inheritance:
:no-members:
.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses
:show-inheritance:
:no-members:
Adapter Networks
~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.MultiHeadAttentionAdapter
:show-inheritance:
:members:
:member-order: bysource
.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.RelPositionMultiHeadAttentionAdapter
:show-inheritance:
:members:
:member-order: bysource
.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.PositionalEncodingAdapter
:show-inheritance:
:members:
:member-order: bysource
.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.RelPositionalEncodingAdapter
:show-inheritance:
:members:
:member-order: bysource
Adapter Strategies
~~~~~~~~~~~~~~~~~~
.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.MHAResidualAddAdapterStrategy
:show-inheritance:
:members:
:member-order: bysource
:undoc-members: adapter_module_names
docs/source/asr/asr_all.bib
+1093
View File
File diff suppressed because it is too large Load Diff
docs/source/asr/asr_checkpoints.rst
+396
View File
@@ -0,0 +1,396 @@
.. _asr-checkpoints-list:
=======================
ASR Model Checkpoints
=======================
This page lists all supported ASR model checkpoints released by NVIDIA NeMo.
Benchmark scores for each model can be found on its `HuggingFace model card <https://huggingface.co/nvidia>`__.
For community fine-tunes built on these checkpoints, see :doc:`Featured Community Checkpoints <./featured_community_checkpoints>`.
Glossary
--------
.. list-table::
:header-rows: 1
* - Term
- Definition
* - **ASR**
- Automatic Speech Recognition — transcribing speech to text
* - **AST**
- Automatic Speech Translation — translating speech to text from one language to another
* - **AED**
- Attention Encoder-Decoder — autoregressive decoder using cross-attention (Canary family)
* - **CTC**
- Connectionist Temporal Classification — non-autoregressive decoder
* - **RNN-T**
- Recurrent Neural Network Transducer — autoregressive streaming-friendly decoder
* - **TDT**
- Token-and-Duration Transducer — extends RNN-T with duration prediction for faster inference
* - **Hybrid**
- Joint RNN-T + CTC model — both decoders trained together, either usable at inference
* - **PnC**
- Punctuation and Capitalization in the output
* - **SALM**
- Speech Augmented Language Model — combines a speech encoder with a large language model
* - **Streaming**
- Real-time / cache-aware inference capability
* - **EU4**
- English, German, Spanish, French
* - **EU25**
- English, German, Spanish, French, Italian, Polish, Portuguese, Dutch, Russian, Ukrainian, Belarusian, Croatian, Czech, Bulgarian, Danish, Estonian, Finnish, Greek, Hungarian, Latvian, Lithuanian, Maltese, Romanian, Slovak, Slovenian, Swedish
Canary Models (AED)
-------------------
Multi-task encoder-decoder models supporting ASR, AST, PnC, and timestamps across multiple languages.
.. list-table::
:header-rows: 1
* - Model
- Decoder
- Capabilities
- Language
- Size
* - `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`__
- AED
- ASR, AST, PnC, timestamps
- EU25
- 1B
* - `canary-qwen-2.5b <https://huggingface.co/nvidia/canary-qwen-2.5b>`__
- SALM
- ASR, AST, PnC, timestamps
- EU25
- 2.5B
* - `canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`__
- AED
- ASR, AST, PnC, timestamps, fast
- EU4
- 1B
* - `canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`__
- AED
- ASR, AST, PnC, timestamps, fast
- EU4
- 180M
* - `canary-1b <https://huggingface.co/nvidia/canary-1b>`__
- AED
- ASR, AST, PnC
- EU4
- 1B
Parakeet Models
-----------------
High-accuracy ASR models built on the FastConformer encoder architecture.
Parakeet, Nemotron Speech, and the ``stt_*_fastconformer_*`` models below all share the same underlying FastConformer encoder;
the different names reflect release branding, not architectural differences.
.. list-table::
:header-rows: 1
* - Model
- Decoder
- Capabilities
- Language
- Size
* - `parakeet-tdt-0.6b-v3 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3>`__
- TDT
- ASR, PnC, timestamps
- English
- 0.6B
* - `parakeet-tdt-0.6b-v2 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2>`__
- TDT
- ASR, PnC, timestamps
- English
- 0.6B
* - `parakeet-tdt-1.1b <https://huggingface.co/nvidia/parakeet-tdt-1.1b>`__
- TDT
- ASR, timestamps
- English
- 1.1B
* - `parakeet-tdt_ctc-1.1b <https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b>`__
- Hybrid TDT+CTC
- ASR, timestamps
- English
- 1.1B
* - `parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`__
- Hybrid TDT+CTC
- ASR, timestamps
- Japanese
- 0.6B
* - `parakeet-tdt_ctc-110m <https://huggingface.co/nvidia/parakeet-tdt_ctc-110m>`__
- Hybrid TDT+CTC
- ASR, timestamps
- English
- 110M
* - `parakeet-rnnt-1.1b <https://huggingface.co/nvidia/parakeet-rnnt-1.1b>`__
- RNN-T
- ASR, timestamps
- English
- 1.1B
* - `parakeet-rnnt-0.6b <https://huggingface.co/nvidia/parakeet-rnnt-0.6b>`__
- RNN-T
- ASR, timestamps
- English
- 0.6B
* - `parakeet-ctc-1.1b <https://huggingface.co/nvidia/parakeet-ctc-1.1b>`__
- CTC
- ASR
- English
- 1.1B
* - `parakeet-ctc-0.6b <https://huggingface.co/nvidia/parakeet-ctc-0.6b>`__
- CTC
- ASR
- English
- 0.6B
* - `parakeet-ctc-0.6b-Vietnamese <https://huggingface.co/nvidia/parakeet-ctc-0.6b-Vietnamese>`__
- CTC
- ASR
- Vietnamese
- 0.6B
* - `parakeet-rnnt-110m-da-dk <https://huggingface.co/nvidia/parakeet-rnnt-110m-da-dk>`__
- RNN-T
- ASR
- Danish
- 110M
Streaming Models
-----------------
Cache-aware models for real-time / low-latency inference.
.. list-table::
:header-rows: 1
* - Model
- Decoder
- Capabilities
- Language
- Size
* - `nemotron-3.5-asr-streaming-0.6b <https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b>`__
- Hybrid
- ASR, streaming
- 40 languages
- 0.6B
* - `multitalker-parakeet-streaming-0.6b-v1 <https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1>`__
- RNN-T
- ASR, multitalker, streaming
- English
- 0.6B
* - `parakeet_realtime_eou_120m-v1 <https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1>`__
- RNN-T
- ASR, end-of-utterance, streaming
- English
- 120M
* - `stt_en_fastconformer_hybrid_large_streaming_multi <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_streaming_multi>`__
- Hybrid
- ASR, streaming, multiple look-aheads
- English
- Large
* - `stt_en_fastconformer_hybrid_medium_streaming_80ms_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms_pc>`__
- Hybrid
- ASR, PnC, streaming
- English
- Medium
* - `stt_en_fastconformer_hybrid_medium_streaming_80ms <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms>`__
- Hybrid
- ASR, streaming
- English
- Medium
* - `stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc>`__
- Hybrid
- ASR, PnC, streaming
- Georgian
- Large
* - `stt_en_fastconformer_hybrid_large_streaming_1040ms <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_fastconformer_hybrid_large_streaming_1040ms>`__
- Hybrid
- ASR, streaming
- English
- Large
FastConformer English Models (Non-Streaming)
----------------------------------------------
.. list-table::
:header-rows: 1
* - Model
- Decoder
- Capabilities
- Language
- Size
* - `stt_en_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- English
- Large
* - `stt_en_fastconformer_ctc_large <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_large>`__
- CTC
- ASR
- English
- Large
* - `stt_en_fastconformer_ctc_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xlarge>`__
- CTC
- ASR
- English
- XLarge
* - `stt_en_fastconformer_ctc_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xxlarge>`__
- CTC
- ASR
- English
- XXLarge
* - `stt_en_fastconformer_transducer_large <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_large>`__
- RNN-T
- ASR
- English
- Large
* - `stt_en_fastconformer_transducer_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xlarge>`__
- RNN-T
- ASR
- English
- XLarge
* - `stt_en_fastconformer_transducer_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xxlarge>`__
- RNN-T
- ASR
- English
- XXLarge
* - `stt_en_fastconformer_tdt_large <https://huggingface.co/nvidia/stt_en_fastconformer_tdt_large>`__
- TDT
- ASR
- English
- Large
FastConformer Multilingual Models
----------------------------------
Single-language FastConformer Hybrid models. Models with ``_pc`` suffix support punctuation and capitalization.
.. list-table::
:header-rows: 1
* - Model
- Decoder
- Capabilities
- Language
- Size
* - `stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`__
- Hybrid
- ASR, PnC
- Multilingual EU
- Large
* - `stt_de_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_de_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- German
- Large
* - `stt_es_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Spanish
- Large
* - `stt_es_fastconformer_hybrid_large_pc_nc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc_nc>`__
- Hybrid
- ASR, Punctuation only
- Spanish
- Large
* - `stt_fr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_fr_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- French
- Large
* - `stt_it_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_it_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Italian
- Large
* - `stt_ru_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Russian
- Large
* - `stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Ukrainian
- Large
* - `stt_pl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pl_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Polish
- Large
* - `stt_hr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hr_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Croatian
- Large
* - `stt_be_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_be_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Belarusian
- Large
* - `stt_nl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_nl_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Dutch
- Large
* - `stt_pt_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pt_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Portuguese
- Large
* - `stt_fa_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large>`__
- Hybrid
- ASR
- Farsi
- Large
* - `stt_ka_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Georgian
- Large
* - `stt_hy_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Armenian
- Large
* - `stt_ar_fastconformer_hybrid_large_pc_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0>`__
- Hybrid
- ASR, PnC
- Arabic
- Large
* - `stt_ar_fastconformer_hybrid_large_pcd_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pcd_v1.0>`__
- Hybrid
- ASR, PnC (diacritized)
- Arabic
- Large
* - `stt_uz_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_uz_fastconformer_hybrid_large_pc>`__
- Hybrid
- ASR, PnC
- Uzbek
- Large
* - `stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`__
- Hybrid
- ASR
- Kazakh + Russian
- Large
Loading Models
--------------
All models (except SALM — see :doc:`SpeechLM2 </speechlm2/intro>`) can be loaded via the ``from_pretrained()`` API:
.. code-block:: python
import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")
docs/source/asr/asr_customization/legacy_language_modeling_and_customization.rst
+297
View File
@@ -0,0 +1,297 @@
.. _ngram_modeling:
****************************
N-gram Language Model Fusion
****************************
In this approach, an N-gram LM is trained on text data, then it is used in fusion with beam search decoding to find the
best candidates. The beam search decoders in NeMo support language models trained with KenLM library (
`https://github.com/kpu/kenlm <https://github.com/kpu/kenlm>`__).
The beam search decoders and KenLM library are not installed by default in NeMo.
You need to install them to be able to use beam search decoding and N-gram LM.
Please refer to `scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh>`__
on how to install them. Alternatively, you can build Docker image
`scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__ with all the necessary dependencies.
Please, refer to :ref:`train-ngram-lm` for more details on how to train an N-gram LM using KenLM library.
NeMo supports both character-based and BPE-based models for N-gram LMs. An N-gram LM can be used with beam search
decoders on top of the ASR models to produce more accurate candidates. The beam search decoder would incorporate
the scores produced by the N-gram LM into its score calculations as the following:
.. code-block::
final_score = acoustic_score + beam_alpha*lm_score + beam_beta*seq_length
where acoustic_score is the score predicted by the acoustic encoder and lm_score is the one estimated by the LM.
The parameter 'beam_alpha' determines the weight given to the N-gram language model, while 'beam_beta' is a penalty term that accounts for sequence length in the scores. A larger 'beam_alpha' places more emphasis on the language model and less on the acoustic model. Negative values for 'beam_beta' penalize longer sequences, encouraging the decoder to prefer shorter predictions. Conversely, positive values for 'beam_beta' favor longer candidates.
Evaluate by Beam Search Decoding and N-gram LM
==============================================
NeMo's beam search decoders are capable of using the KenLM's N-gram models to find the best candidates.
The script to evaluate an ASR model with beam search decoding and N-gram models can be found at
`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__.
This script has a large number of possible argument overrides; therefore, it is recommended that you use ``python eval_beamsearch_ngram_ctc.py --help`` to see the full list of arguments.
You can evaluate an ASR model using the following:
.. code-block::
python eval_beamsearch_ngram_ctc.py nemo_model_file=<path to the .nemo file of the model> \
input_manifest=<path to the evaluation JSON manifest file \
kenlm_model_file=<path to the binary KenLM model> \
beam_width=[<list of the beam widths, separated with commas>] \
beam_alpha=[<list of the beam alphas, separated with commas>] \
beam_beta=[<list of the beam betas, separated with commas>] \
preds_output_folder=<optional folder to store the predictions> \
probs_cache_file=null \
decoding_mode=beamsearch_ngram \
decoding_strategy="<Beam library such as beam, pyctcdecode or flashlight>"
It can evaluate a model in the following three modes by setting the argument ``--decoding_mode``:
* greedy: Just greedy decoding is done and no beam search decoding is performed.
* beamsearch: The beam search decoding is done, but without using the N-gram language model. Final results are equivalent to setting the weight of LM (beam_beta) to zero.
* beamsearch_ngram: The beam search decoding is done with N-gram LM.
In ``beamsearch`` mode, the evaluation is performed using beam search decoding without any language model. The performance is reported in terms of Word Error Rate (WER) and Character Error Rate (CER). Moreover, when the best candidate is selected among the candidates, it is also reported as the best WER/CER. This can serve as an indicator of the quality of the predicted candidates.
The script initially loads the ASR model and predicts the outputs of the model's encoder as log probabilities. This part is computed in batches on a device specified by --device, which can be either a CPU (`--device=cpu`) or a single GPU (`--device=cuda:0`).
The batch size for this part is specified by ``--acoustic_batch_size``. Using the largest feasible batch size can speed up the calculation of log probabilities. Additionally, you can use `--use_amp` to accelerate the calculation and allow for larger --acoustic_batch_size values.
Currently, multi-GPU support is not available for calculating log probabilities. However, using ``--probs_cache_file`` can help. This option stores the log probabilities produced by the model's encoder in a pickle file, allowing you to skip the first step in future runs.
The following is the list of the important arguments for the evaluation script:
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| **Argument** | **Type** | **Default** | **Description** |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| nemo_model_file | str | Required | The path of the `.nemo` file of the ASR model to extract the tokenizer. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| input_manifest | str | Required | Path to the training file, it can be a text file or JSON manifest. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| kenlm_model_file | str | Required | The path to store the KenLM binary model file. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| preds_output_folder | str | None | The path to an optional folder to store the predictions. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| probs_cache_file | str | None | The cache file for storing the outputs of the model. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| acoustic_batch_size | int | 16 | The batch size to calculate log probabilities. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| use_amp | bool | False | Whether to use AMP if available to calculate log probabilities. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| device | str | cuda | The device to load the model onto to calculate log probabilities. |
| | | | It can `cpu`, `cuda`, `cuda:0`, `cuda:1`, ... |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| decoding_mode | str | beamsearch_ngram | The decoding scheme to be used for evaluation. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| beam_width | float | Required | List of the width or list of the widths of the beam search decoding. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| beam_alpha | float | Required | List of the alpha parameter for the beam search decoding. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| beam_beta | float | Required | List of the beta parameter for the beam search decoding. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| beam_batch_size | int | 128 | The batch size to be used for beam search decoding. |
| | | | Larger batch size can be a little faster, but uses larger memory. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| decoding_strategy | str | beam | String argument for type of decoding strategy for the model. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| decoding | Dict | BeamCTC | Subdict of beam search configs. Values found via |
| | Config | InferConfig | python eval_beamsearch_ngram_ctc.py --help |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| text_processing.do_lowercase | bool | ``False`` | Whether to make the training text all lower case. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| text_processing.punctuation_marks | str | ``""`` | String with punctuation marks to process. Example: ".\,?" |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| text_processing.rm_punctuation | bool | ``False`` | Whether to remove punctuation marks from text. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
| text_processing.separate_punctuation | bool | ``True`` | Whether to separate punctuation with the previous word by space. |
+--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
The width of the beam search (``--beam_width``) specifies the number of top candidates or predictions the beam search decoder will consider. Larger beam widths result in more accurate but slower predictions.
.. note::
The ``eval_beamsearch_ngram_ctc.py`` script contains the entire subconfig used for CTC Beam Decoding.
Therefore it is possible to forward arguments for various beam search libraries such as ``flashlight``
and ``pyctcdecode`` via the ``decoding`` subconfig.
To learn more about evaluating the ASR models with N-gram LM, refer to the tutorial here: Offline ASR Inference with Beam Search and External Language Model Rescoring
`Offline ASR Inference with Beam Search and External Language Model Rescoring <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/Offline_ASR.ipynb>`_
Beam Search Engines
-------------------
NeMo ASR CTC supports multiple beam search engines for decoding. The default engine is beam, which is the OpenSeq2Seq decoding library.
OpenSeq2Seq (``beam``)
~~~~~~~~~~~~~~~~~~~~~~
CPU-based beam search engine that is quite efficient and supports char and subword models. It requires a character/subword
KenLM model to be provided.
The config for this decoding library is described above.
Flashlight (``flashlight``)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Flashlight is a C++ library for ASR decoding provided at `https://github.com/flashlight/flashlight <https://github.com/flashlight/flashlight>`_. It is a CPU- and CUDA-based beam search engine that is quite efficient and supports char and subword models. It requires an ARPA KenLM file.
It supports several advanced features, such as lexicon-based decoding, lexicon-free decoding, beam pruning threshold, and more.
.. code-block:: python
@dataclass
class FlashlightConfig:
lexicon_path: Optional[str] = None
boost_path: Optional[str] = None
beam_size_token: int = 16
beam_threshold: float = 20.0
unk_weight: float = -math.inf
sil_weight: float = 0.0
.. code-block::
# Lexicon-based decoding
python eval_beamsearch_ngram_ctc.py ... \
decoding_strategy="flashlight" \
decoding.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon' \
decoding.beam.flashlight_cfg.beam_size_token = 32 \
decoding.beam.flashlight_cfg.beam_threshold = 25.0
# Lexicon-free decoding
python eval_beamsearch_ngram_ctc.py ... \
decoding_strategy="flashlight" \
decoding.beam.flashlight_cfg.beam_size_token = 32 \
decoding.beam.flashlight_cfg.beam_threshold = 25.0
PyCTCDecode (``pyctcdecode``)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
PyCTCDecode is a Python library for ASR decoding provided at `https://github.com/kensho-technologies/pyctcdecode <https://github.com/kensho-technologies/pyctcdecode>`_. It is a CPU-based beam search engine that is somewhat efficient for a pure Python library, and supports char and subword models. It requires a character/subword KenLM ARPA / BINARY model to be provided.
It has advanced features, such as word boosting, which can be useful for transcript customization.
.. code-block:: python
@dataclass
class PyCTCDecodeConfig:
beam_prune_logp: float = -10.0
token_min_logp: float = -5.0
prune_history: bool = False
hotwords: Optional[List[str]] = None
hotword_weight: float = 10.0
.. code-block::
# PyCTCDecoding
python eval_beamsearch_ngram_ctc.py ... \
decoding_strategy="pyctcdecode" \
decoding.beam.pyctcdecode_cfg.beam_prune_logp = -10. \
decoding.beam.pyctcdecode_cfg.token_min_logp = -5. \
decoding.beam.pyctcdecode_cfg.hotwords=[<List of str words>] \
decoding.beam.pyctcdecode_cfg.hotword_weight=10.0
Hyperparameter Grid Search
--------------------------
Beam search decoding with N-gram LM has three main hyperparameters: `beam_width`, `beam_alpha`, and `beam_beta`.
The accuracy of the model is dependent on the values of these parameters, specifically, beam_alpha and beam_beta. To perform grid search, you can specify a single value or a list of values for each of these parameters. In this case, it would perform the beam search decoding on all combinations of the three hyperparameters.
For example, the following set of parameters would result in 212=4 beam search decodings:
.. code-block::
python eval_beamsearch_ngram_ctc.py ... \
beam_width=[64,128] \
beam_alpha=[1.0] \
beam_beta=[1.0,0.5]
Beam Search ngram Decoding for Transducer Models (RNNT and HAT)
===============================================================
You can also find a similar script to evaluate an RNNT/HAT model with beam search decoding and N-gram models at:
`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_transducer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_transducer.py>`_
.. code-block::
python eval_beamsearch_ngram_transducer.py nemo_model_file=<path to the .nemo file of the model> \
input_manifest=<path to the evaluation JSON manifest file \
kenlm_model_file=<path to the binary KenLM model> \
beam_width=[<list of the beam widths, separated with commas>] \
beam_alpha=[<list of the beam alphas, separated with commas>] \
preds_output_folder=<optional folder to store the predictions> \
probs_cache_file=null \
decoding_strategy=<greedy_batch or maes decoding>
maes_prefix_alpha=[<list of the maes prefix alphas, separated with commas>] \
maes_expansion_gamma=[<list of the maes expansion gammas, separated with commas>] \
hat_subtract_ilm=<in case of HAT model: subtract internal LM or not (True/False)> \
hat_ilm_weight=[<in case of HAT model: list of the HAT internal LM weights, separated with commas>] \
.. _wfst-ctc-decoding:
WFST CTC decoding
=================
Weighted Finite-State Transducers (WFST) are finite-state machines with input and output symbols on each transition and some weight element of a semiring. WFSTs can act as N-gram LMs in a special type of LM-forced beam search, called WFST decoding.
.. note::
More precisely, WFST decoding is more of a greedy N-depth search with LM.
Thus, it is asymptotically worse than conventional beam search decoding algorithms, but faster.
**WARNING**
At the moment, NeMo supports WFST decoding only for CTC models and word-based LMs.
To run WFST decoding in NeMo, one needs to provide a NeMo ASR model and either an ARPA LM or a WFST LM (advanced). An ARPA LM can be built from source text with KenLM as follows: ``<kenlm_bin_path>/lmplz -o <ngram_length> --arpa <out_arpa_path> --prune <ngram_prune>``.
The script to evaluate an ASR model with WFST decoding and N-gram models can be found at
`scripts/asr_language_modeling/ngram_lm/eval_wfst_decoding_ctc.py
<https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_wfst_decoding_ctc.py>`__.
This script has a large number of possible argument overrides, therefore it is advised to use ``python eval_wfst_decoding_ctc.py --help`` to see the full list of arguments.
You may evaluate an ASR model as the following:
.. code-block::
python eval_wfst_decoding_ctc.py nemo_model_file=<path to the .nemo file of the model> \
input_manifest=<path to the evaluation JSON manifest file> \
arpa_model_file=<path to the ARPA LM model> \
decoding_wfst_file=<path to the decoding WFST file> \
beam_width=[<list of the beam widths, separated with commas>] \
lm_weight=[<list of the LM weight multipliers, separated with commas>] \
open_vocabulary_decoding=<whether to use open vocabulary mode for WFST decoding> \
decoding_mode=<decoding mode, affects output. Usually "nbest"> \
decoding_search_type=<WFST decoding library. Usually "riva"> \
preds_output_folder=<optional folder to store the predictions> \
probs_cache_file=null
.. note::
Since WFST decoding is LM-forced (the search goes over the WIDEST graph), only word sequences accepted by the WFST can appear in the decoding results.
To circumvent this restriction, one can pass ``open_vocabulary_decoding=true`` (experimental feature).
Quick start example
-------------------
.. code-block::
wget -O - https://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz | \
gunzip -c | tr '[:upper:]' '[:lower:]' > 3-gram.pruned.1e-7.arpa && \
python eval_wfst_decoding_ctc.py nemo_model_file="stt_en_conformer_ctc_small_ls" \
input_manifest="<data_dir>/Librispeech/test_other.json" \
arpa_model_file="3-gram.pruned.1e-7.arpa" \
decoding_wfst_file="3-gram.pruned.1e-7.fst" \
beam_width=[8] \
lm_weight=[0.5,0.6,0.7,0.8,0.9]
.. note::
Building a decoding WFST is a long process, so it is better to provide a ``decoding_wfst_file`` path even if you don't have it.
This way, the decoding WFST will be buffered to the specified file path and there will be no need to re-build it on the next run.
docs/source/asr/asr_customization/neural_rescoring.rst
+105
View File
@@ -0,0 +1,105 @@
.. _neural_rescoring:
****************
Neural Rescoring
****************
When using the neural rescoring approach, a neural network is used to score candidates. A candidate is the text transcript predicted by the ASR model's decoder. The top K candidates produced by beam search decoding (with a beam width of K) are given to a neural language model for ranking. The language model assigns a score to each candidate, which is usually combined with the scores from beam search decoding to produce the final scores and rankings.
Train Neural Rescorer
=====================
An example script to train such a language model with Transformer can be found at `examples/nlp/language_modeling/transformer_lm.py <https://github.com/NVIDIA/NeMo/blob/stable/examples/nlp/language_modeling/transformer_lm.py>`__.
It trains a ``TransformerLMModel`` which can be used as a neural rescorer for an ASR system. For more information on language models training, see LLM/NLP documentation.
You can also use a pretrained language model from the Hugging Face library, such as Transformer-XL and GPT, instead of training your model.
Models like BERT and RoBERTa are not supported by this script because they are trained as Masked Language Models. As a result, they are not efficient or effective for scoring sentences out of the box.
Evaluation
==========
Given a trained TransformerLMModel `.nemo` file or a pretrained HF model, the script available at
`scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__
can be used to re-score beams obtained with ASR model. You need the `.tsv` file containing the candidates produced
by the acoustic model and the beam search decoding to use this script. The candidates can be the result of just the beam
search decoding or the result of fusion with an N-gram LM. You can generate this file by specifying `--preds_output_folder` for
`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__.
The neural rescorer would rescore the beams/candidates by using two parameters of `rescorer_alpha` and `rescorer_beta`, as follows:
.. code-block::
final_score = beam_search_score + rescorer_alpha*neural_rescorer_score + rescorer_beta*seq_length
The parameter `rescorer_alpha` specifies the importance placed on the neural rescorer model, while `rescorer_beta` is a penalty term that accounts for sequence length in the scores. These parameters have similar effects to `beam_alpha` and `beam_beta` in the beam search decoder and N-gram language model.
Use the following steps to evaluate a neural LM:
#. Obtain `.tsv` file with beams and their corresponding scores. Scores can be from a regular beam search decoder or
in fusion with an N-gram LM scores. For a given beam size `beam_size` and a number of examples
for evaluation `num_eval_examples`, it should contain (`num_eval_examples` x `beam_size`) lines of
form `beam_candidate_text \t score`. This file can be generated by `scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__
#. Rescore the candidates by `scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__.
.. code-block::
python eval_neural_rescorer.py
--lm_model=[path to .nemo file of the LM or the name of a HF pretrained model]
--beams_file=[path to beams .tsv file]
--beam_size=[size of the beams]
--eval_manifest=[path to eval manifest .json file]
--batch_size=[batch size used for inference on the LM model]
--alpha=[the value for the parameter rescorer_alpha]
--beta=[the value for the parameter rescorer_beta]
--scores_output_file=[the optional path to store the rescored candidates]
The candidates, along with their new scores, are stored at the file specified by `--scores_output_file`.
The following is the list of the arguments for the evaluation script:
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| **Argument** |**Type**| **Default** | **Description** |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| lm_model | str | Required | The path of the '.nemo' file of an ASR model, or the name of a |
| | | | Hugging Face pretrained model like 'transfo-xl-wt103' or 'gpt2'. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| eval_manifest | str | Required | Path to the evaluation manifest file (.json manifest file). |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beams_file | str | Required | Path to beams file (.tsv) containing the candidates and their scores. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beam_size | int | Required | The width of the beams (number of candidates) generated by the decoder. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| alpha | float | None | The value for parameter rescorer_alpha |
| | | | Not passing value would enable linear search for rescorer_alpha. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| beta | float | None | The value for parameter rescorer_beta |
| | | | Not passing value would enable linear search for rescorer_beta. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| batch_size | int | 16 | The batch size used to calculate the scores. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| max_seq_length | int | 512 | Maximum sequence length (in tokens) for the input. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| scores_output_file | str | None | The optional file to store the rescored beams. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| use_amp | bool | ``False`` | Whether to use AMP if available calculate the scores. |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
| device | str | cuda | The device to load LM model onto to calculate the scores |
| | | | It can be 'cpu', 'cuda', 'cuda:0', 'cuda:1', ... |
+---------------------+--------+------------------+-------------------------------------------------------------------------+
Hyperparameter Linear Search
----------------------------
The hyperparameter linear search script also supports linear search for parameters `alpha` and `beta`. If any of the two is not
provided, a linear search is performed to find the best value for that parameter. When linear search is used, initially
`beta` is set to zero and the best value for `alpha` is found, then `alpha` is fixed with
that value and another linear search is done to find the best value for `beta`.
If any of the of these two parameters is already specified, then search for that one is skipped. After each search for a
parameter, the plot of WER% for different values of the parameter is also shown.
It is recommended to first use the linear search for both parameters on a validation set by not providing any values for `--alpha` and `--beta`.
Then check the WER curves and decide on the best values for each parameter. Finally, evaluate the best values on the test set.
docs/source/asr/asr_customization/ngpulm_language_modeling_and_customization.rst
+346
View File
@@ -0,0 +1,346 @@
.. _ngpulm_ngram_modeling:
***************************************************************
NGPU-LM (GPU-based N-gram Language Model) Language Model Fusion
***************************************************************
ASR systems can achieve significantly improved accuracy by leveraging **external language model (LM) shallow fusion** during the decoding process.
This technique integrates knowledge from an external LM without requiring the ASR model itself to be retrained.
**How Shallow Fusion Works:**
During shallow fusion, the output probabilities generated by the ASR model are combined with those from a separate, external language model.
The final transcription is then determined by selecting the word sequence that yields the highest combined score.
These external LMs are typically trained on vast text datasets, allowing them to capture the statistical patterns, syntactic structures, and contextual dependencies of language.
This enables them to predict more plausible word sequences, thereby correcting potential errors from the ASR model.
**Domain Adaptation Benefits:**
Shallow fusion is particularly valuable for **adapting ASR systems to new or specialized domains**.
By training the external LM on domain-specific text-such as medical, legal, or technical documents-it learns the vocabulary of that field.
This specialized knowledge guides the ASR decoding process towards more accurate and contextually relevant transcriptions.
Traditionally, shallow fusion has been performed during **beam search decoding**, a method that explores multiple promising hypotheses to find the most likely transcription.
NGPU-LM
=======
A widely used library for training traditional n-gram language models is KenLM.
While KenLM (https://github.com/kpu/kenlm) is known for its efficient CPU-based implementation, its reliance on the CPU can limit performance in high-throughput scenarios, especially when dealing with large-scale data.
NGPU-LM on contrast is a GPU-accelerated implementation of a statistical n-gram language model.
It uses a **universal trie-based data structure**, which enables fast, batched queries. For full details, please refer to the paper [ngpulm]_.
This enables shallow fusion during **greedy decoding**, creating a middle ground between standard greedy decoding and full beam search with a language model.
It preserves the speed and simplicity of greedy decoding while regaining much of the accuracy typically achieved with beam search with external LM fusion.
While not as accurate as full beam search, greedy decoding with NGPU-LM fusion offers a compelling balance between speed and accuracy.
NeMo provides efficient, fully GPU-based beam search implementations for all major ASR model types,
allowing **beam decoding to operate with real-time factors (RTFx) close to those of greedy decoding**.
At a batch size of 32, the RTFx difference between beam and greedy decoding is only about 20%.
These implementations incorporate NGPU-LM, enabling fast, fully GPU-based decoding and customization.
This enables users to customize decoding while maintaining reasonable speed, even in beam search mode.
For full details, please refer to the [beamsearch]_.
NGPU-LM fusion is supported for BPE-based ASR models (CTC, RNNT, TDT, AED) during both greedy and beam decoding.
Train NGPU-LM
=============
NGPU-LM is built using `.ARPA` files generated by the KenLM library. You can train an n-gram LM using the following script:
`train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
The generated `.ARPA` files can be directly used for GPU-based decoding.
However, for faster performance, it is recommended to convert the model to the `.nemo` format by setting the ``save_nemo`` flag to ``true``.
.. code-block::
python train_kenlm.py nemo_model_file=<path to the .nemo file of the model> \
train_paths=<list of paths to the training text or JSON manifest files> \
kenlm_bin_path=<path to the bin folder of KenLM library> \
kenlm_model_file=<path to store the binary KenLM model> \
ngram_length=<order of N-gram model> \
preserve_arpa=true \
save_nemo=True
For a complete list of arguments and usage details, refer to the :ref:`train-ngram-lm`.
.. note::
It is recommended that you use 6 as the order of the N-gram model for BPE-based models. Higher orders may require re-compiling KenLM to support them.
.. _ctc-decoding-with-ngpulm:
Decoding with NGPU-LM
=====================
To run inference with NGPU-LM fusion, the ``ngram_lm_model`` and ``ngram_lm_alpha`` fields must be specified in the decoding configuration.
.. note::
For CTC, RNNT, and TDT models, these fields should be set within the respective ``greedy`` or ``beam`` sub-configurations.
For AED models running in greedy mode, set the beam size to 1 and specify these fields under the ``beam`` sub-configuration.
Examples for different model types are provided below.
CTC Decoding with NGPU-LM
-------------------------
**Greedy Search:**
You can run NGPU-LM shallow fusion during greedy CTC decoding using the following command:
.. code-block:: bash
python examples/asr/speech_to_text_eval.py \
pretrained_name=nvidia/parakeet-ctc-1.1b \
amp=false \
amp_dtype=bfloat16 \
matmul_precision=high \
compute_dtype=bfloat16 \
presort_manifest=true \
cuda=0 \
batch_size=32 \
dataset_manifest=<path to the evaluation JSON manifest file> \
ctc_decoding.greedy.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
ctc_decoding.greedy.ngram_lm_alpha=0.2 \
ctc_decoding.greedy.allow_cuda_graphs=True \
ctc_decoding.strategy="greedy_batch"
**Beam Search:**
During CTC beam search, each hypothesis is scored using the following formula:
.. code-block::
final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length
where:
- ``acoustic_score`` is the score predicted by the ASR.
- ``lm_score`` is the score predicted by the NGPU-LM LM.
- ``ngram_lm_alpha`` is the weight given to the language model.
- ``beam_beta`` is a penalty term that accounts for sequence length in the scores.
For running fully batched GPU-based CTC decoding with NGPU-LM, you can use the following command:
The following is the list of the adjustable arguments of batched CTC decoding algorithm ``beam_batch``:
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| **Argument** | **Type** | **Default** | **Description** |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| ngram_lm_alpha | float | Required | Weight factor applied to the language model scores. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| beam_size | int | 4 | Beam size. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| beam_beta | float | 1 | Penalty applied to word insertions to control the trade-off between insertion and deletion errors during beam search decoding. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| beam_threshold | float | 20 | Threshold used to prune candidate hypotheses by comparing their scores to the best hypothesis. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
.. code-block:: bash
python examples/asr/speech_to_text_eval.py \
pretrained_name=nvidia/parakeet-ctc-1.1b \
amp=false \
amp_dtype=bfloat16 \
matmul_precision=high \
compute_dtype=bfloat16 \
presort_manifest=true \
cuda=0 \
batch_size=32 \
dataset_manifest=<path to the evaluation JSON manifest file> \
ctc_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
ctc_decoding.beam.ngram_lm_alpha=0.2 \
ctc_decoding.beam.beam_size=12 \
ctc_decoding.beam.beam_beta=1.0 \
ctc_decoding.strategy="beam_batch" \
ctc_decoding.beam.allow_cuda_graphs=True
RNN-T/TDT decoding with NGPU-LM
-------------------------------
**Greedy Search:**
You can run NGPU-LM shallow fusion during greedy RNN-T / TDT decoding using the following command:
.. code-block:: bash
python examples/asr/speech_to_text_eval.py \
pretrained_name=nvidia/parakeet-rnnt-1.1b \
amp=false \
amp_dtype=bfloat16 \
matmul_precision=high \
compute_dtype=bfloat16 \
presort_manifest=true \
cuda=0 \
batch_size=32 \
dataset_manifest=<path to the evaluation JSON manifest file> \
rnnt_decoding.greedy.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
rnnt_decoding.greedy.ngram_lm_alpha=0.2 \
rnnt_decoding.greedy.allow_cuda_graphs=True \
rnnt_decoding.strategy="greedy_batch"
.. note::
To run the inference with TDT model, you need to provide pretrained TDT model in ``pretrained_name`` field (for example ``nvidia/parakeet-tdt_ctc-1.1b`` ).
**Beam Search:**
During RNN-T / TDT beam search, each hypothesis is scored using the following formula:
.. code-block::
final_score = acoustic_score + ngram_lm_alpha * lm_score
where:
- ``acoustic_score`` is the score predicted by the ASR.
- ``lm_score`` is the score predicted by the NGPU-LM LM.
- ``ngram_lm_alpha`` is the weight given to the language model.
Final hypotheses is chosen based on the normalized score ``final_score / seq_length``.
*Blank Scoring in Transducer Models*
Transducer models include a blank symbol (``∅``) for frame transitions, while LMs do not model blanks.
During shallow fusion, the LM is typically applied only to non-blank tokens:
.. math::
\ln p_{\text{tot}}[k] =
\begin{cases}
\ln p[k] + \lambda \ln p_{\text{LM}}[k], & k \in V \\
\ln p[\emptyset], & k = \emptyset
\end{cases}
This can lead to excessive blank predictions at higher LM weights, increasing deletion errors.
NeMo supports a blank-aware scoring method that adjusts LM contributions to better balance predictions:
.. math::
\ln p_{\text{tot}}[k] =
\begin{cases}
\ln p[k] + \lambda \ln((1 - p[\emptyset]) \cdot p_{\text{LM}}[k]), & k \in V \\
(1 + \lambda) \ln p[\emptyset], & k = \emptyset
\end{cases}
*Early vs. Late Pruning*
In shallow fusion, LM and ASR scores can be combined at different stages:
- **Early pruning:** ASR selects top hypotheses, then LM rescoring is applied. Efficient for small beams.
- **Late pruning:** ASR and LM scores are combined before pruning. More accurate but requires full-vocab LM queries.
For Transducer models, late pruning with the blank-aware scoring method generally yields better performance than the standard approach.
*Beam Search Strategies:*
In NeMo fully batched implementation of following strategies are supported:
- **malsd_batch:** fully batched implemention of modified Alignment-Length Synchronous Decoding [alsd]_, supporting both RNNT and TDT models.
- **maes_batch:** fully batched implemention of modified Adaptive Expansion Search [aes]_, supporting for only RNNT models. CudaGraphs are not supported.
The following is the list of the adjustable arguments of batched CTC decoding algorithm ``beam_batch``:
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| **Argument** | **Type** | **Strategy** | **Default** | **Description** |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| ngram_lm_alpha | float | malsd_batch, maes_batch | Required | Weight factor applied to the language model scores. |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| beam_size | int | malsd_batch, maes_batch | 4 | Beam size. |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| pruning_mode | str | malsd_batch, maes_batch | late | Mode for hypotheses pruning. Can be ``early`` or ``late``. |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| blank_lm_score_mode | str | malsd_batch, maes_batch | lm_weighted_full | Mode for blank symbol scoring. Can be ``no_score`` or ``lm_weighted_full`` |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| max_symbols_per_step | int | malsd_batch | 10 | Max symbols to emit on each step to avoid infinite looping. |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| maes_num_step | int | maes_batch | 2 | Number of adaptive steps to take. |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| maes_expansion_beta | float | maes_batch | 1.0 | Maximum number of prefix expansions allowed, in addition to the beam size. |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
| maes_expansion_gamma | float | maes_batch | 2.3 | Threshold used to prune candidate hypotheses by comparing their scores to the best hypothesis. |
+-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
You can run NGPU-LM shallow fusion during beam RNN-T / TDT decoding using the following command:
.. code-block:: bash
python examples/asr/speech_to_text_eval.py \
pretrained_name=nvidia/parakeet-rnnt-1.1b \
amp=false \
amp_dtype=bfloat16 \
matmul_precision=high \
compute_dtype=bfloat16 \
presort_manifest=true \
cuda=0 \
batch_size=32 \
dataset_manifest=<path to the evaluation JSON manifest file> \
rnnt_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
rnnt_decoding.beam.ngram_lm_alpha=0.2 \
rnnt_decoding.beam.beam_size=12 \
rnnt_decoding.beam.pruning_mode="late" \
rnnt_decoding.beam.blank_lm_score_mode="lm_weighted_full" \
rnnt_decoding.beam.allow_cuda_graphs=True \
rnnt_decoding.strategy="malsd_batch"
.. note::
To run the inference with TDT model, you need to provide pretrained TDT model in ``pretrained_name`` field (for example ``nvidia/parakeet-tdt_ctc-1.1b`` ).
AED Decoding with NGPU-LM
-------------------------
**Beam Search:**
You can run NGPU-LM shallow fusion during greedy CTC decoding using the following command:
.. code-block:: bash
python examples/asr/speech_to_text_eval.py \
pretrained_name="nvidia/canary-1b" \
amp=false \
amp_dtype=bfloat16 \
matmul_precision=high \
compute_dtype=bfloat16 \
presort_manifest=true \
cuda=0 \
batch_size=32 \
dataset_manifest=<dataset_manifest> \
multitask_decoding.beam.beam_size=4 \
multitask_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
multitask_decoding.beam.ngram_lm_alpha=0.2 \
multitask_decoding.strategy="beam"
.. note::
For greedy decoding with NGPU-LM, use beam search with beam_size=1.
References
==========
.. [ngpulm] V. Bataev, A. Andrusenko, L. Grigoryan, A. Laptev, V. Lavrukhin, and B. Ginsburg.
*NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding*.
arXiv:2505.22857, 2025. Available at: https://arxiv.org/abs/2505.22857
.. [beamsearch] L. Grigoryan, V. Bataev, A. Andrusenko, H. Xu, V. Lavrukhin, and B. Ginsburg.
*Pushing the Limits of Beam Search Decoding for Transducer-based ASR Models*.
arXiv:2506.00185, 2025. Available at: https://arxiv.org/abs/2506.00185
.. [alsd] G. Saon, Z. Tüske, and K. Audhkhasi.
*Alignment-Length Synchronous Decoding for RNN Transducer*.
In: ICASSP 2020 – IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7804–7808, 2020.
doi: https://doi.org/10.1109/ICASSP40776.2020.9053040
.. [aes] J. Kim, Y. Lee, and E. Kim.
*Accelerating RNN Transducer Inference via Adaptive Expansion Search*.
IEEE Signal Processing Letters, vol. 27, pp. 2019–2023, 2020.
doi: https://doi.org/10.1109/LSP.2020.3036335
docs/source/asr/asr_customization/ngram_utils.rst
+151
View File
@@ -0,0 +1,151 @@
.. _ngram-utils:
Scripts for building and merging N-gram Language Models
=======================================================
.. _train-ngram-lm:
Train N-gram LM
===============
NeMo utilizes the KenLM library (`https://github.com/kpu/kenlm`) for building efficient n-gram language models.
.. note::
KenLM is not installed by default in NeMo.
Please see the installation instructions in the script:
`scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh>`__.
Alternatively, you can build a Docker image with all required dependencies using:
`scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__.
The script for training an n-gram language model with KenLM is available here:
`scripts/asr_language_modeling/ngram_lm/train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
This script supports training n-gram LMs on both character-level and BPE-level encodings, which are automatically detected from the model type. The resulting language models can then be used with beam search decoders integrated on top of ASR models.
You can train an n-gram model using the following command:
.. code-block::
python train_kenlm.py nemo_model_file=<path to the .nemo file of the model> \
train_paths=<list of paths to the training text or JSON manifest files> \
kenlm_bin_path=<path to the bin folder of KenLM library> \
kenlm_model_file=<path to store the binary KenLM model> \
ngram_length=<order of N-gram model> \
preserve_arpa=true
The `train_paths` parameter allows for various input types, such as a list of text files, JSON manifests, or directories, to be used as the training data.
If the file's extension is anything other than `.json`, it assumes that data format is plain text. For plain text format, each line should contain one
sample. For the JSON manifests, the file must contain JSON-formatted samples per each line like this:
.. code-block::
{"audio_filepath": "/data_path/file1.wav", "text": "The transcript of the audio file."}
This code extracts the `text` field from each line to create the training text file. After the N-gram model is trained, it is stored at the path specified by `kenlm_model_file`.
The following is the list of the arguments for the training script:
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| **Argument** | **Type** | **Default** | **Description** |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| nemo_model_file | str | Required | The path to `.nemo` file of the ASR model, or name of a pretrained NeMo model to extract a tokenizer. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| train_paths | List[str] | Required | List of training files or folders. Files can be a plain text file or ".json" manifest or ".json.gz". |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| kenlm_model_file | str | Required | The path to store the KenLM binary model file. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| kenlm_bin_path | str | Required | The path to the bin folder of KenLM. It is a folder named `bin` under where KenLM is installed. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| ngram_length** | int | Required | Specifies order of N-gram LM. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| ngram_prune | List[int] | [0] | List of thresholds to prune N-grams. Example: [0,0,1]. See Pruning section on the https://kheafield.com/code/kenlm/estimation |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| cache_path | str | ``""`` | Cache path to save tokenized files. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| preserve_arpa | bool | ``False`` | Whether to preserve the intermediate ARPA file after construction of the BIN file. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| verbose | int | 1 | Verbose level. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
| save_nemo | bool | ``False`` | Whether to save LM in .nemo format. |
+------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
..note::
It is recommended that you use 6 as the order of the N-gram model for BPE-based models. Higher orders may require re-compiling KenLM to support them.
Combine N-gram Language Models
==============================
Before combining N-gram LMs, install the required OpenGrm NGram library using `scripts/installers/install_opengrm.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/install_opengrm.sh>`__.
Alternatively, you can use Docker image `scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__ with all the necessary dependencies.
Alternatively, you can use the Docker image at:
`scripts/asr_language_modeling/ngram_lm/ngram_merge.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/ngram_merge.py>`__, which includes all the necessary dependencies.
This script interpolates two ARPA N-gram language models and creates a KenLM binary file that can be used with the beam search decoders on top of ASR models.
You can specify weights (`--alpha` and `--beta`) for each of the models (`--ngram_a` and `--ngram_b`) correspondingly: `alpha` * `ngram_a` + `beta` * `ngram_b`.
This script supports both character level and BPE level encodings and models which are detected automatically from the type of the model.
To combine two N-gram models, you can use the following command:
.. code-block::
python ngram_merge.py --kenlm_bin_path <path to the bin folder of KenLM library> \
--ngram_bin_path <path to the bin folder of OpenGrm Ngram library> \
--arpa_a <path to the ARPA N-gram model file A> \
--alpha <weight of N-gram model A> \
--ar
pa_b <path to the ARPA N-gram model file B> \
--beta <weight of N-gram model B> \
--out_path <path to folder to store the output files>
If you provide `--test_file` and `--nemo_model_file`, This script supports both character-level and BPE-level encodings and models, which are detected automatically based on the type of the model.
Note, the result of each step during the process is cached in the temporary file in the `--out_path`, to speed up further run.
You can use the `--force` flag to discard the cache and recalculate everything from scratch.
.. code-block::
python ngram_merge.py --kenlm_bin_path <path to the bin folder of KenLM library> \
--ngram_bin_path <path to the bin folder of OpenGrm Ngram library> \
--arpa_a <path to the ARPA N-gram model file A> \
--alpha <weight of N-gram model A> \
--arpa_b <path to the ARPA N-gram model file B> \
--beta <weight of N-gram model B> \
--out_path <path to folder to store the output files>
--nemo_model_file <path to the .nemo file of the model> \
--test_file <path to the test file> \
--symbols <path to symbols (.syms) file> \
--force <flag to recalculate and rewrite all cached files>
The following is the list of the arguments for the opengrm script:
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| **Argument** |**Type**| **Default** | **Description** |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| kenlm_bin_path | str | Required | The path to the bin folder of KenLM library. It is a folder named `bin` under where KenLM is installed. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| ngram_bin_path | str | Required | The path to the bin folder of OpenGrm Ngram. It is a folder named `bin` under where OpenGrm Ngram is installed. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| arpa_a | str | Required | Path to the ARPA N-gram model file A. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| alpha | float | Required | Weight of N-gram model A. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| arpa_b | int | Required | Path to the ARPA N-gram model file B. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| beta | float | Required | Weight of N-gram model B. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| out_path | str | Required | Path for writing temporary and resulting files. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| test_file | str | None | Path to test file to count perplexity if provided. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| symbols | str | None | Path to symbols (.syms) file. Could be calculated if it is not provided. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| nemo_model_file | str | None | The path to '.nemo' file of the ASR model, or name of a pretrained NeMo model. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
| force | bool | ``False`` | Whether to recompile and rewrite all files. |
+----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
docs/source/asr/asr_customization/word_boosting.rst
+362
View File
@@ -0,0 +1,362 @@
.. _word_boosting:
****************************************************
Word Boosting
****************************************************
.. _word_boosting_gpupb:
GPU-PB
========================
GPU-PB is a GPU-accelerated Phrase-Boosting method supported for CTC, RNN-T/TDT, and AED (Canary) models based on NGPU-LM infrastructure.
The method supports greedy and beam search decoding, including CUDA graphs mode. GPU-PB is compatible with NGPU-LM at the same decoding run.
GPU-PB is applied only at the decoding step in shallow fusion mode. You do not need to retrain the ASR model.
During greedy or beam search decoding, GPU-PB rescales ASR model scores with a boosting tree at the token level.
The boosting tree is built from a context phrases list, which is provided by the user.
**NOTE**: for ASR models that support capitalization by default (e.g., Canary or parakeet-tdt-0.6b-v2), you need to capitalize all the key phrases in advance (and capitalize the full word for abbreviations).
You can use LLM for this task.
More details about GPU-PB method can be found in the `original paper <https://arxiv.org/abs/2508.07014>`__.
Usage
-----
We support three ways to pass the context phrases into the decoding script:
1. Build a boosting tree for a specific ASR model (step 0.0) and use it for all the decoding evaluation by ``boosting_tree.model_path`` (step 1.1-3.1).
2. Provide a file with context phrases ``boosting_tree.key_phrases_file`` - one phrase per line (step 1.1-3.1).
3. Provide a python list of context phrases ``boosting_tree.key_phrases_list`` (step 1.1-3.1).
The use of the Phrase-Boosting tree is controlled by ``boosting_tree`` config (``BoostingTreeModelConfig``) for all the models.
For prepared boosting tree use ``boosting_tree.model_path=${PATH_TO_BTREE}``.
We recommend to provide the list of context phrases directly into ``speech_to_text_eval.py`` by ``boosting_tree.key_phrases_file=${KEY_WORDS_LIST}``.
List of the most important parameters:
* ``strategy`` - The strategy to use for decoding depending on the model type (CTC - greedy_batch or beam_batch; RNN-T/TDT - greedy_batch or malsd_batch; AED - beam).
* ``model_path``, ``key_phrases_file``, ``key_phrases_list`` - The way to pass the context phrases into the decoding script.
* ``context_score`` - The score for each arc transition in the context graph (1.0 is recommended).
* ``depth_scaling`` - The scaling factor for the depth of the context graph (2.0 is recommended for CTC, RNN-T and TDT, 1.0 for Canary).
* ``boosting_tree_alpha`` - Weight of the GPU-PB boosting tree during shallow fusion decoding (tune it according to your data).
**0.0. [Optional] Build the boosting tree for a specific ASR model:**
.. code-block::
python scripts/asr_context_biasing/build_gpu_boosting_tree.py \
asr_model_path=${ASR_NEMO_MODEL_FILE} \
key_phrases_file=${CONTEXT_BIASING_LIST} \
save_to=${PATH_TO_SAVE_BTREE} \
context_score=${CONTEXT_SCORE} \
depth_scaling=${DEPTH_SCALING} \
use_triton=True
**1.1. CTC greedy batch decoding:**
.. code-block::
python examples/asr/speech_to_text_eval.py \
model_path=${MODEL_NAME} \
dataset_manifest=${EVAL_MANIFEST} \
batch_size=${BATCH_SIZE} \
output_filename=${OUT_MANIFEST} \
ctc_decoding.strategy="greedy_batch" \
ctc_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
ctc_decoding.greedy.boosting_tree.context_score=1.0 \
ctc_decoding.greedy.boosting_tree.depth_scaling=2.0 \
ctc_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}
**1.2. CTC beam batch decoding:**
.. code-block::
python examples/asr/speech_to_text_eval.py \
model_path=${MODEL_NAME} \
dataset_manifest=${EVAL_MANIFEST} \
batch_size=${BATCH_SIZE} \
output_filename=${OUT_MANIFEST} \
ctc_decoding.strategy="beam_batch" \
ctc_decoding.beam.beam_size=${BEAM_SIZE} \
ctc_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
ctc_decoding.beam.boosting_tree.context_score=1.0 \
ctc_decoding.beam.boosting_tree.depth_scaling=2.0 \
ctc_decoding.beam.boosting_tree_alpha=${BT_ALPHA}
**2.1. RNN-T/TDT greedy batch decoding:**
.. code-block::
python examples/asr/speech_to_text_eval.py \
model_path=${MODEL_NAME} \
dataset_manifest=${EVAL_MANIFEST} \
batch_size=${BATCH_SIZE} \
output_filename=${OUT_MANIFEST} \
rnnt_decoding.strategy="greedy_batch" \
rnnt_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
rnnt_decoding.greedy.boosting_tree.context_score=1.0 \
rnnt_decoding.greedy.boosting_tree.depth_scaling=2.0 \
rnnt_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}
**2.2. RNN-T/TDT beam (malsd_batch) decoding:**
.. code-block::
python examples/asr/speech_to_text_eval.py \
model_path=${MODEL_NAME} \
dataset_manifest=${EVAL_MANIFEST} \
batch_size=${BATCH_SIZE} \
output_filename=${OUT_MANIFEST} \
rnnt_decoding.strategy="malsd_batch" \
rnnt_decoding.beam.beam_size=${BEAM_SIZE} \
rnnt_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
rnnt_decoding.beam.boosting_tree.context_score=1.0 \
rnnt_decoding.beam.boosting_tree.depth_scaling=2.0 \
rnnt_decoding.beam.boosting_tree_alpha=${BT_ALPHA}
**3.1. AED (Canary) greedy (beam_size=1) or beam (beam_size>1) decoding:**
.. code-block::
python examples/asr/speech_to_text_eval.py \
model_path=${MODEL_NAME} \
dataset_manifest=${EVAL_MANIFEST} \
batch_size=${BATCH_SIZE} \
output_filename=${OUT_MANIFEST} \
multitask_decoding.strategy="beam" \
multitask_decoding.beam.beam_size=${BEAM_SIZE} \
multitask_decoding.beam.boosting_tree.key_phrases_file=${CONTEXT_BIASING_LIST} \
multitask_decoding.beam.boosting_tree.context_score=1.0 \
multitask_decoding.beam.boosting_tree.depth_scaling=1.0 \
multitask_decoding.beam.boosting_tree_alpha=${BT_ALPHA} \
gt_lang_attr_name="target_lang" \
gt_text_attr_name="text"
Results evaluation
------------------
You can compute the F-score for the list of context phrases directly from the decoding manifest.
.. code-block::
python scripts/asr_context_biasing/compute_key_words_fscore.py \
--input_manifest=${DECODING_MANIFEST} \
--key_words_file=${CONTEXT_PHRASES_LIST}
.. _word_boosting_per_stream:
Per-Stream Phrase Boosting
==========================
Per-stream (per-utterance) phrase boosting extends GPU-PB to allow specifying different key phrases for each audio stream or utterance in a batch.
This is useful when different utterances require different context biasing (e.g., different speaker names, product terms, or domain vocabulary per audio).
Per-stream boosting is currently supported for **greedy label-looping decoding with Transducers (RNN-T, TDT)**, including cache-aware streaming models.
Manifest-based Usage
--------------------
Specify per-utterance key phrases in your manifest using the ``biasing_request`` field:
.. code-block:: json
{"audio_filepath": "/data/file1.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["one phrase"]}}}
{"audio_filepath": "/data/file2.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["other phrases", "and this one"]}}}
Use the streaming inference script with ``use_per_stream_biasing=true``:
.. code-block:: bash
python examples/asr/asr_streaming_inference/asr_streaming_infer.py \
--config-path="../conf/asr_streaming_inference/" \
--config-name=cache_aware_rnnt.yaml \
audio_file="<manifest_with_boosting_requests>" \
output_filename="result.jsonl" \
asr.model_name="nvidia/parakeet-rnnt-1.1b" \
asr.decoding.greedy.enable_per_stream_biasing=True
Python API Usage
----------------
.. code-block:: python
from omegaconf import open_dict
from nemo.collections.asr.models import EncDecRNNTBPEModel
from nemo.collections.asr.parts.context_biasing.biasing_multi_model import BiasingRequestItemConfig
from nemo.collections.asr.parts.context_biasing.boosting_graph_batched import BoostingTreeModelConfig
from nemo.collections.asr.parts.utils.rnnt_utils import Hypothesis
asr_model = EncDecRNNTBPEModel.from_pretrained("nvidia/parakeet-rnnt-1.1b")
asr_model.to("cuda")
with open_dict(asr_model.cfg.decoding):
asr_model.cfg.decoding.strategy = "greedy_batch"
asr_model.cfg.decoding.greedy.loop_labels = True
asr_model.cfg.decoding.greedy.enable_per_stream_biasing = True
asr_model.change_decoding_strategy(asr_model.cfg.decoding)
biasing_requests = [
BiasingRequestItemConfig(
boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["one phrase"]),
boosting_model_alpha=2.0,
),
None, # no biasing for this utterance
BiasingRequestItemConfig(
boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["other phrases"]),
boosting_model_alpha=1.0,
),
]
results = asr_model.transcribe(
audio=["file1.wav", "file2.wav", "file3.wav"],
partial_hypothesis=[
Hypothesis.empty_with_biasing_cfg(biasing_cfg=req) if req else None
for req in biasing_requests
],
return_hypotheses=True,
)
Caching
-------
Building a boosting model from a phrase list has some overhead. NeMo provides caching mechanisms to speed up repeated use of the same phrases:
.. list-table::
:header-rows: 1
:widths: 15 60 25
* - Strategy
- Description
- Recommended For
* - Memory
- Set ``cache_key`` on ``BiasingRequestItemConfig`` to cache compiled models in memory by a string key.
- Repeated phrase sets
* - Disk
- Set ``model_path`` on ``BoostingTreeModelConfig`` to save/load compiled models from disk.
- Persistent caching
* - Decoder
- Set ``auto_manage_multi_model=False`` and manually manage models in the decoder's multi-model.
- Advanced use cases
With memory caching, per-stream boosting achieves near-zero overhead compared to global (shared) boosting.
.. _word_boosting_flashlight:
Flashlight-based Word Boosting
==============================
The Flashlight decoder supports word boosting during CTC decoding using a KenLM binary and corresponding lexicon. Word boosting only works in lexicon-decoding mode and does not function in lexicon-free mode. It allows you to bias the decoder for certain words by manually increasing or decreasing the probability of emitting specific words. This can be very helpful if you have uncommon or industry-specific terms that you want to ensure are transcribed correctly.
For more information, go to `word boosting <https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-customizing.html#word-boosting>`__
To use word boosting in NeMo, create a simple tab-separated text file. Each line should contain a word to be boosted, followed by a tab, and then the boosted score for that word.
For example:
.. code-block::
nvidia 40
geforce 50
riva 80
turing 30
badword -100
Positive scores boost words higher in the LM decoding step so they show up more frequently, whereas negative scores
squelch words so they show up less frequently. The recommended range for the boost score is +/- 20 to 100.
The boost file handles both in-vocabulary words and OOV words just fine, so you can specify both IV and OOV words with corresponding scores.
You can then pass this file to your Flashlight config object during decoding:
.. code-block::
# Lexicon-based decoding
python eval_beamsearch_ngram_ctc.py ... \
decoding_strategy="flashlight" \
decoding.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon' \
decoding.beam.flashlight_cfg.boost_path='/path/to/my_boost_file.boost' \
decoding.beam.flashlight_cfg.beam_size_token = 32 \
decoding.beam.flashlight_cfg.beam_threshold = 25.0
.. _word_boosting_ctcws:
CTC-WS: Context-biasing (Word Boosting) without External LM
===========================================================
NeMo toolkit supports a fast context-biasing method for CTC and Transducer (RNN-T) ASR models with CTC-based Word Spotter.
The method involves decoding CTC log probabilities with a context graph built for words and phrases from the context-biasing list.
The spotted context-biasing candidates (with their scores and time intervals) are compared by scores with words from the greedy CTC decoding results to improve recognition accuracy and prevent false accepts of context-biasing.
A Hybrid Transducer-CTC model (a shared encoder trained together with CTC and Transducer output heads) enables the use of the CTC-WS method for the Transducer model.
Context-biasing candidates obtained by CTC-WS are also filtered by the scores with greedy CTC predictions and then merged with greedy Transducer results.
Scheme of the CTC-WS method:
.. image:: https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/asset-post-v1.22.0-ctcws_scheme_1.png
:align: center
:alt: CTC-WS scheme
:width: 80%
High-level overview of the context-biasing words replacement with CTC-WS method:
.. image:: https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/asset-post-v1.22.0-ctcws_scheme_2.png
:align: center
:alt: CTC-WS high level overview
:width: 80%
More details about CTC-WS context-biasing can be found in the `tutorial <https://github.com/NVIDIA/NeMo/tree/main/tutorials/asr/ASR_Context_Biasing.ipynb>`__.
To use CTC-WS context-biasing, you need to create a context-biasing text file that contains words/phrases to be boosted, with its transcriptions (spellings) separated by underscore.
Multiple transcriptions can be useful for abbreviations ("gpu" -> "g p u"), compound words ("nvlink" -> "nv link"),
or words with common mistakes in the case of our ASR model ("nvidia" -> "n video").
Example of the context-biasing file:
.. code-block::
nvidia_nvidia
omniverse_omniverse
gpu_gpu_g p u
dgx_dgx_d g x_d gx
nvlink_nvlink_nv link
ray tracing_ray tracing
The main script for CTC-WS context-biasing in NeMo is:
.. code-block::
{NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py
Context-biasing is managed by ``apply_context_biasing`` parameter [true or false].
Other important context-biasing parameters are:
* ``beam_threshold`` - threshold for CTC-WS beam pruning.
* ``context_score`` - per token weight for context biasing.
* ``ctc_ali_token_weight`` - per token weight for CTC alignment (prevents false acceptances of context-biasing words).
All the context-biasing parameters are selected according to the default values in the script.
You can tune them according to your data and ASR model (list all the values in the [] separated by commas)
for example: ``beam_threshold=[7.0,8.0,9.0]``, ``context_score=[3.0,4.0,5.0]``, ``ctc_ali_token_weight=[0.5,0.6,0.7]``.
The script will run the recognition with all the combinations of the parameters and will select the best one based on WER value.
.. code-block::
# Context-biasing with the CTC-WS method for CTC ASR model
python {NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py \
nemo_model_file={ctc_model_name} \
input_manifest={test_nemo_manifest} \
preds_output_folder={exp_dir} \
decoder_type="ctc" \
acoustic_batch_size=64 \
apply_context_biasing=true \
context_file={cb_list_file_modified} \
beam_threshold=[7.0] \
context_score=[3.0] \
ctc_ali_token_weight=[0.5]
To use Transducer head of the Hybrid Transducer-CTC model, you need to set ``decoder_type=rnnt``.
docs/source/asr/asr_language_modeling_and_customization.rst
+213
View File
@@ -0,0 +1,213 @@
.. _asr_language_modeling_and_customization:
#######################################
ASR Language Modeling and Customization
#######################################
NeMo supports decoding-time customization techniques such as *language modeling* and *word boosting*,
which improve transcription accuracy by incorporating external knowledge or domain-specific vocabulary—without retraining the model.
Decoder Types
-------------
NeMo ASR models use different decoder architectures. The table below summarizes them:
.. list-table::
:header-rows: 1
* - Decoder
- Type
- Description
- Models
* - **CTC**
- Non-autoregressive
- Connectionist Temporal Classification. Fast inference, supports LM fusion and word boosting.
- Parakeet-CTC, FastConformer-CTC
* - **RNN-T**
- Autoregressive
- Recurrent Neural Network Transducer. Strong accuracy, streaming-friendly.
- Parakeet-RNNT, FastConformer-Transducer
* - **TDT**
- Autoregressive
- Token-and-Duration Transducer. Extends RNN-T with duration prediction for better timestamps.
- Parakeet-TDT
* - **AED**
- Autoregressive
- Attention Encoder-Decoder. Multi-task capable (ASR + AST), prompt-based language control.
- Canary-1B, Canary-1B-V2, Canary-1B-Flash
* - **Hybrid**
- Both
- Joint RNN-T + CTC training. Use either decoder at inference time.
- FastConformer Hybrid models
Language Modeling
-----------------
In NeMo two approaches of external language modeling are supported:
- **Language Model Fusion:**
Language model (LM) fusion integrates scores from an external statistical n-gram model into the ASR decoder.
This helps guide decoding toward more likely word sequences based on text corpora.
NeMo provides two approaches for language model shallow fusion with ASR systems:
**1. NGPU-LM (Recommended for Production)**
GPU-accelerated LM fusion for all major model types: CTC, RNN-T, TDT, and AED models.
- Customization during both greedy and beam decoding.
- Fast beam decoding for all major model types, offering only 20% RTFx difference between beam and greedy decoding.
- Integration with NGPU-LM GPU-based ngram LM.
For details, please refer to :ref:`ngpulm_ngram_modeling`
**2. KenLM (Traditional CPU-based)**
CPU-based LM fusion using the KenLM library.
.. note::
These approaches, especially beam decoding, can be extremely slow and are retained in the repository primarily for backward compatibility.
If possible, we recommend using NGPU-LM for improved performance.
For details, please refer to :ref:`ngram_modeling`
- **Neural Rescoring:**
When using the neural rescoring approach, a neural network is used to score candidates. A candidate is the text transcript predicted by the ASR model’s decoder.
The top K candidates produced by beam search decoding (with a beam width of K) are given to a neural language model for ranking.
The language model assigns a score to each candidate, which is usually combined with the scores from beam search decoding to produce the final scores and rankings.
For details, please refer to :ref:`neural_rescoring`.
Word Boosting
-------------
Word boosting increases the likelihood of specific words or phrases during decoding by applying a positive bias, helping the model better recognize names,
uncommon terms, and custom vocabulary.
- :ref:`word_boosting_gpupb` (preferred): GPU-accelerated phrase-boosting for CTC, RNN-T/TDT, and AED (Canary) models supporting greedy and beam search decoding.
- :ref:`word_boosting_flashlight`: Word-boosting method for CTC models with external n-gram LM.
- :ref:`word_boosting_ctcws`: Word-boosting method for hybrid (Transducer-CTC) models without LM.
For details, please refer to: :ref:`word_boosting`.
LM Training
-----------
NeMo provides tools for training n-gram language models that can be used for language model fusion or word-boosting.
For details, please refer to: :ref:`ngram-utils`.
CUDA Graphs
-----------
CUDA graphs accelerate decoding by capturing and replaying GPU operations, eliminating kernel launch overhead.
Support varies by decoder strategy:
.. list-table::
:header-rows: 1
* - Strategy
- Config Parameter
- Default
- Notes
* - ``greedy_batch`` (RNN-T, TDT)
- ``use_cuda_graph_decoder``
- ``true``
- Requires ``loop_labels=True`` and ``blank_as_pad=True``
* - ``maes_batch``, ``malsd_batch`` (beam)
- ``allow_cuda_graphs``
- ``true``
- Batched beam search strategies
* - Non-batched ``greedy`` / ``beam``
- N/A
- N/A
- Not supported; standard decoding used
To disable CUDA graphs (e.g. for debugging or when preserving alignments with frame-looping):
**Via Python (at runtime):**
.. code-block:: python
model.disable_cuda_graphs()
**Greedy decoding** — use ``use_cuda_graph_decoder=true/false``:
.. code-block:: bash
python examples/asr/speech_to_text_eval.py \
pretrained_name="nvidia/parakeet-rnnt-1.1b" \
dataset_manifest=<dataset_manifest> \
batch_size=32 \
output_filename=decoded.jsonl \
rnnt_decoding.strategy="greedy_batch" \
rnnt_decoding.greedy.use_cuda_graph_decoder=true
**Beam decoding** — use ``allow_cuda_graphs=true/false``:
.. code-block:: bash
python examples/asr/speech_to_text_eval.py \
pretrained_name="nvidia/parakeet-rnnt-1.1b" \
dataset_manifest=<dataset_manifest> \
batch_size=32 \
output_filename=decoded.jsonl \
rnnt_decoding.strategy="malsd_batch" \
rnnt_decoding.beam.max_symbols_per_step=10 \
rnnt_decoding.beam.beam_size=12 \
rnnt_decoding.beam.allow_cuda_graphs=true
When unsupported, NeMo falls back to standard decoding automatically.
Confidence Estimation
---------------------
NeMo supports per-frame, per-token, and per-word confidence scores during decoding.
Confidence estimation helps applications decide when to trust ASR output and when to request human review.
.. code-block:: yaml
decoding:
confidence_cfg:
preserve_frame_confidence: false
preserve_token_confidence: false
preserve_word_confidence: false
exclude_blank: true
aggregation: "mean" # mean, min, max, prod
method_cfg:
name: "entropy" # max_prob or entropy
entropy_type: "tsallis" # gibbs, tsallis, renyi
alpha: 0.33
entropy_norm: "exp" # lin or exp
**Confidence methods:**
* ``max_prob``: Maximum token probability as confidence. Simple and fast.
* ``entropy``: Normalized entropy of the log-likelihood vector (default). Entropy types:
- ``gibbs``: Standard Gibbs entropy
- ``tsallis``: Tsallis entropy (default, recommended)
- ``renyi``: Renyi entropy
**Aggregation** combines frame-level scores into token/word scores: ``mean``, ``min``, ``max``, or ``prod``.
For TDT models, set ``tdt_include_duration_confidence: true`` to include duration prediction confidence.
.. toctree::
:maxdepth: 1
:hidden:
asr_customization/ngpulm_language_modeling_and_customization
asr_customization/neural_rescoring
asr_customization/legacy_language_modeling_and_customization
asr_customization/ngram_utils
asr_customization/word_boosting
docs/source/asr/configs.rst
+925
View File
@@ -0,0 +1,925 @@
NeMo ASR Configuration Files
============================
This section describes the NeMo configuration file setup that is specific to models in the ASR collection. For general information
about how to set up and run experiments that is common to all NeMo models (e.g. Experiment Manager and PyTorch Lightning trainer
parameters), see the :doc:`../core/core` section.
The model section of the NeMo ASR configuration files generally requires information about the dataset(s) being used, the preprocessor
for audio files, parameters for any augmentation being performed, as well as the model architecture specification. The sections on
this page cover each of these in more detail.
Example configuration files for all of the NeMo ASR scripts can be found in the
`config directory of the examples <https://github.com/NVIDIA/NeMo/tree/stable/examples/asr/conf>`_.
.. _asr-configs-dataset-configuration:
Dataset Configuration
---------------------
Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
``test_ds`` sections in the configuration file, respectively. Depending on the task, there may be arguments specifying the sample rate
of the audio files, the vocabulary of the dataset (for character prediction), whether or not to shuffle the dataset, and so on. You may
also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command-line at runtime.
Any initialization parameter that is accepted for the Dataset class used in the experiment can be set in the config file.
Refer to the :ref:`Datasets <asr-api-datasets>` section of the API for a list of Datasets and their respective parameters.
An example ASR train and validation configuration should look similar to the following:
.. code-block:: yaml
# Specified at the beginning of the config file
labels: &labels [" ", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m",
"n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "'"]
model:
train_ds:
manifest_filepath: ???
sample_rate: 16000
labels: *labels # Uses the labels above
batch_size: 32
trim_silence: True
max_duration: 16.7
shuffle: True
num_workers: 8
pin_memory: true
# tarred datasets
is_tarred: false # If set to true, uses the tarred version of the Dataset
tarred_audio_filepaths: null # Not used if is_tarred is false
shuffle_n: 2048 # Not used if is_tarred is false
# bucketing params
bucketing_strategy: "synced_randomized"
bucketing_batch_size: null
bucketing_weights: null
validation_ds:
manifest_filepath: ???
sample_rate: 16000
labels: *labels # Uses the labels above
batch_size: 32
shuffle: False # No need to shuffle the validation data
num_workers: 8
pin_memory: true
There are two ways to test/validate on more than one manifest:
- Specify a list in the `manifest_filepath` field. Results will be reported for each, the first one being used for overall loss / WER (specify `val_dl_idx` if you wish to change that). In this case, all manifests will share configuration parameters.
- Use the ds_item key and pass a list of config objects to it. This allows you to use differently configured datasets for validation, e.g.
.. code-block:: yaml
model:
validation_ds:
ds_item:
- name: dataset1
manifest_filepath: ???
# Config parameters for dataset1
...
- name: dataset2
manifest_filepath: ???
# Config parameters for dataset2
...
By default, dataloaders are set up when the model is instantiated. However, dataloader setup can be deferred to
model's `setup()` method by setting ``defer_setup`` in the configuration.
For example, training data setup can be deferred as follows:
.. code-block:: yaml
model:
train_ds:
# Configure training data as usual
...
# Defer train dataloader setup from `__init__` to `setup`
defer_setup: true
.. _asr-configs-metric-configuration:
.. _asr-configs-preprocessor-configuration:
Preprocessor Configuration
--------------------------
If you are loading audio files for your experiment, you will likely want to use a preprocessor to convert from the
raw audio signal to features (e.g. mel-spectrogram or MFCC). The ``preprocessor`` section of the config specifies the audio
preprocessor to be used via the ``_target_`` field, as well as any initialization parameters for that preprocessor.
An example of specifying a preprocessor is as follows:
.. code-block:: yaml
model:
...
preprocessor:
# _target_ is the audio preprocessor module you want to use
_target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
normalize: "per_feature"
window_size: 0.02
...
# Other parameters for the preprocessor
Refer to the :ref:`Audio Preprocessors <asr-audio-preprocessors>` API section for the preprocessor options, expected arguments,
and defaults.
.. _asr-configs-augmentation-configurations:
Augmentation Configurations
---------------------------
There are a few on-the-fly spectrogram augmentation options for NeMo ASR, which can be specified by the
configuration file using a ``spec_augment`` section.
For example, there are options for `Cutout <https://arxiv.org/abs/1708.04552>`_ and
`SpecAugment <https://arxiv.org/abs/1904.08779>`_ available via the ``SpectrogramAugmentation`` module.
The following example sets up both ``Cutout`` (via the ``rect_*`` parameters) and ``SpecAugment`` (via the ``freq_*``
and ``time_*`` parameters).
.. code-block:: yaml
model:
...
spec_augment:
_target_: nemo.collections.asr.modules.SpectrogramAugmentation
# Cutout parameters
rect_masks: 5 # Number of rectangles to cut from any given spectrogram
rect_freq: 50 # Max cut of size 50 along the frequency dimension
rect_time: 120 # Max cut of size 120 along the time dimension
# SpecAugment parameters
freq_masks: 2 # Cut two frequency bands
freq_width: 15 # ... of width 15 at maximum
time_masks: 5 # Cut out 10 time bands
time_width: 25 # ... of width 25 at maximum
You can use any combination of ``Cutout``, frequency/time ``SpecAugment``, or neither of them.
You can also add audio augmentation pipelines via an ``augmentor`` section in ``train_ds``.
.. caution::
The ``augmentor`` pipeline is not supported by the Lhotse dataloader, which provides its own set of augmentation options.
See :doc:`Lhotse Dataloading </dataloaders>` for details.
Augmentors are applied on-the-fly to audio data in the data layer. The following example
adds white noise (probability 0.5, level between -50 dB and -10 dB) and room impulse response
augmentation (probability 0.3, from a manifest of impulse responses):
.. code-block:: yaml
model:
...
train_ds:
...
augmentor:
white_noise:
prob: 0.5
min_level: -50
max_level: -10
impulse:
prob: 0.3
manifest_path: /path/to/impulse_manifest.json
Refer to the :ref:`Audio Augmentors <asr-api-audio-augmentors>` API section for more details.
Metric Configurations
---------------------
NeMo ASR models supports WER and BLEU metric logging during training and validation. All metrics are based on the TorchMetrics backend, allowing for distributed training without additional code.
Word Error Rate (WER)
~~~~~~~~~~~~~~~~~~~~~
WER is the default metric for all ASR models and measures transcription accuracy at the word or character level.
.. code-block:: yaml
model:
use_cer: false # Set to true for Character Error Rate instead (default: false)
log_prediction: true # Whether to log a sample prediction during training (default: true)
batch_dim_index: 0 # Index of batch dimension in prediction tensors output. Set to 1 for RNNT models.
BLEU Score
~~~~~~~~~~
BLEU score can be used for ASR models to evaluate translation quality. NeMo's BLEU implementation is based on SacreBLEU for standardized, reproducible scoring:
.. code-block:: yaml
model:
bleu_tokenizer: "13a" # SacreBLEU tokenizer type (see below). (default: "13a")
n_gram: 4 # Maximum n-gram order for BLEU calculation. (default: 4)
lowercase: false # Whether to lowercase before computing BLEU. (default: False)
weights: null # Optional custom weights for n-gram orders. (default: null)
smooth: false # Whether to apply smoothing to BLEU calculation. (default: False)
check_cuts_for_bleu_tokenizers: false # Enable per-sample tokenizer selection. (See below for more details.) (default: False)
log_prediction: true # Whether to log sample predictions. (default: True)
batch_dim_index: 0 # Index of batch dimension in prediction tensors output. Set to 1 for RNNT models. (default: 0)
BLEU score relies on TorchMetrics' SacreBLEU implementation and supports all SacreBLEU tokenization options. Valid strings may be passed to ``bleu_tokenizer`` parameter to configure base tokenizer behavior during BLEU calculation. Available options are:
* ``"13a"`` - Default WMT tokenizer (mteval-v13a script compatible)
* ``"none"`` - No tokenization applied
* ``"intl"`` - International tokenization (mteval-v14 script compatible)
* ``"char"`` - Character-level tokenization (language-agnostic)
* ``"zh"`` - Chinese tokenization (separates Chinese characters, uses 13a for non-Chinese)
* ``"ja-mecab"`` - Japanese tokenization using MeCab morphological analyzer
* ``"ko-mecab"`` - Korean tokenization using MeCab-ko morphological analyzer
* ``"flores101"`` / ``"flores200"`` - SentencePiece models from Flores datasets
**Note** Due to their unique orthographies, it is highly recommended to use ``zh``, ``ja-mecab``, or ``ko-mecab`` tokenizers for Chinese, Japanese, and Korean target evaluations, respectively. For more information on SacreBLEU tokenizers, please refer to the `SacreBLEU documentation <https://github.com/mjpost/sacrebleu>`__.
**Dynamic Tokenizer Selection**
In multilingual training scenarios, it is somtimes desireable to configure the BLEU tokenizer per sample to avoid sub-optimal parsing (e.g. tokenizing Chinese characters as English words). This can be toggled with ``check_cuts_for_bleu_tokenizers: true``. When enabled with Lhotse dataloading, BLEU will check individual ``cuts`` in a batch's Lhotse ``CutSet`` for the ``bleu_tokenizer`` attribute. If found, the tokenizer will be used for that sample. If not, the default ``bleu_tokenizer`` from config will be used.
MultiTask Metrics
~~~~~~~~~~~~~~~~~
Multiple metrics can be configured simultaneously using a ``MultiTaskMetric`` config. This is done by specifying in the config each desired metric as a DictConfig entry with a custom key name and ``_target_`` path, along with desired properties. All properties specified within a metric config will be passed only to the metric class. All properties specified at the top level of the config will be inherited by all submetrics.
.. code-block:: yaml
model:
multitask_metrics_config:
log_prediction: true
metrics:
wer:
_target_: nemo.collections.asr.metrics.wer.WER
use_cer: true
constraint: ".task==transcribe" # Only apply WER to transcription samples
bleu:
_target_: nemo.collections.asr.metrics.bleu.BLEU
bleu_tokenizer: flores101
lowercase: true
check_cuts_for_bleu_tokenizers: true
constraint: ".task==translate" # Only apply BLEU to translation samples
**Metric Constraints**
Each metric within ``MultiTaskMetric`` can be configured with an optional boolean ``constraint`` pattern that filters batch samples before metric computation. This allows validation to be limited to only applicable samples in a batch (e.g. only apply WER to transcription samples, only apply BLEU to translation samples). Constraint patterns match against property keywords in the batch's Lhotse CutSet.
.. code-block:: yaml
model:
multitask_metrics_config:
metrics:
pnc_wer:
_target_: nemo.collections.asr.metrics.wer.WER
constraint: ".task==transcribe and .pnc==true"
multilingual_bleu:
_target_: nemo.collections.asr.metrics.bleu.BLEU
constraint: "(.source_lang!=.target_lang) or .task==translate"
**Note:** MultiTaskMetric is currently only supported for AED multitask models.
Tokenizer Configurations
------------------------
Some models utilize sub-word encoding via an external tokenizer instead of explicitly defining their vocabulary.
For such models, a ``tokenizer`` section is added to the model config. ASR models currently support two types of
custom tokenizers:
- Google Sentencepiece tokenizers (tokenizer type of ``bpe`` in the config)
- HuggingFace WordPiece tokenizers (tokenizer type of ``wpe`` in the config)
- Aggregate tokenizers ((tokenizer type of ``agg`` in the config), see below)
In order to build custom tokenizers, refer to the ``ASR_with_Subword_Tokenization`` notebook available in the
ASR tutorials directory.
The following example sets up a ``SentencePiece Tokenizer`` at a path specified by the user:
.. code-block:: yaml
model:
...
tokenizer:
dir: "<path to the directory that contains the custom tokenizer files>"
type: "bpe" # can be "bpe" or "wpe"
The Aggregate (``agg``) tokenizer feature makes it possible to combine tokenizers in order to train multilingual
models. The config file would look like this:
.. code-block:: yaml
model:
...
tokenizer:
type: "agg" # aggregate tokenizer
langs:
en:
dir: "<path to the directory that contains the tokenizer files>"
type: "bpe" # can be "bpe" or "wpe"
es:
dir: "<path to the directory that contains the tokenizer files>"
type: "bpe" # can be "bpe" or "wpe"
In the above config file, each language is associated with its own pre-trained tokenizer, which gets assigned
a token id range in the order the tokenizers are listed. To train a multilingual model, one needs to populate the
``lang`` field in the manifest file, allowing the routing of each sample to the correct tokenizer. At inference time,
the routing is done based on the inferred token id range.
For models which utilize sub-word tokenization, we share the decoder module (``ConvASRDecoder``) with character tokenization models.
All parameters are shared, but for models which utilize sub-word encoding, there are minor differences when setting up the config. For
such models, the tokenizer is utilized to fill in the missing information when the model is constructed automatically.
For example, a decoder config corresponding to a sub-word tokenization model should look similar to the following:
.. code-block:: yaml
model:
...
decoder:
_target_: nemo.collections.asr.modules.ConvASRDecoder
feat_in: *enc_final
num_classes: -1 # filled with vocabulary size from tokenizer at runtime
vocabulary: [] # filled with vocabulary from tokenizer at runtime
On-the-fly Code Switching
-------------------------
Nemo supports creating code-switched synthetic utterances on-the-fly during training/validation/testing. This allows you to create ASR models which
support intra-utterance code switching. If you have Nemo formatted audio data on disk (either JSON manifests or tarred audio data), you
can easily mix as many of these audio sources together as desired by adding some extra parameters to your `train_ds`, `validation_ds`, and `test_ds`.
Please note that this allows you to mix any kind of audio sources together to create synthetic utterances which sample from all sources. The most
common use case for this is blending different languages together to create a multilingual code-switched model, but you can also blend
together different audio sources from the same languages (or language families), to create noise robust data, or mix fast and slow speech from the
same language.
For multilingual code-switched models, we recommend using AggTokenizer for your Tokenizer if mixing different languages.
The following example shows how to mix 3 different languages: English (en), German (de), and Japanese (ja) added to the `train_ds` model block, however
you can add similar logic to your `validation_ds` and `test_ds` blocks for on-the-fly code-switched validation and test data too. This example mixes
together 3 languages, but you can use as many as you want. However, be advised that the more languages you add, the higher your `min_duration` and `max_duration`
need to be set to ensure all languages are sampled into each synthetic utterance, and setting these hyperparameters higher will use more VRAM per mini-batch during
training and evaluation.
.. code-block:: yaml
model:
train_ds:
manifest_filepath: [/path/to/EN/tarred_manifest.json, /path/to/DE/tarred_manifest.json, /path/to/JA/tarred_manifest.json]
tarred_audio_filepaths: ['/path/to/EN/tars/audio__OP_0..511_CL_.tar', '/path/to/DE/tars/audio__OP_0..1023_CL_.tar', '/path/to/JA/tars/audio__OP_0..2047_CL_.tar']
is_code_switched: true
is_tarred: true
shuffle: true
code_switched: # add this block for code-switching
min_duration: 12 # the minimum number of seconds for each synthetic code-switched utterance
max_duration: 20 # the maximum number of seconds for each synthetic code-switched utterance
min_monolingual: 0.3 # the minimum percentage of utterances which will be pure monolingual (0.3 = 30%)
probs: [0.25, 0.5, 0.25] # the probability to sample each language (matches order of `language` above) if not provided, assumes uniform distribution
force_monochannel: true # if your source data is multi-channel, then setting this to True will force the synthetic utterances to be mono-channel
sampling_scales: 0.75 # allows you to down/up sample individual languages. Can set this as an array for individual languages, or a scalar for all languages
seed: 123 # add a seed for replicability in future runs (highly useful for `validation_ds` and `test_ds`)
Model Architecture Configurations
---------------------------------
Each configuration file should describe the model architecture being used for the experiment. Models in the NeMo ASR collection need
an ``encoder`` section and a ``decoder`` section, with the ``_target_`` field specifying the module to use for each.
Here is the list of the parameters in the model section which are shared among most of the ASR models:
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| **Parameter** | **Datatype** | **Description** | **Supported Values** |
+=========================+==================+===============================================================================================================+=================================+
| :code:`log_prediction` | bool | Whether a random sample should be printed in the output at each step, along with its predicted transcript. | |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
| :code:`ctc_reduction` | string | Specifies the reduction type of CTC loss. Defaults to ``mean_batch`` which would take the average over the | :code:`none`, |
| | | batch after taking the average over the length of each sample. | :code:`mean_batch` |
| | | | :code:`mean`, :code:`sum` |
+-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
For more information about the ASR models, refer to the :doc:`Featured Models <./featured_models>` section.
.. _asr-configs-conformer-ctc:
Conformer-CTC
~~~~~~~~~~~~~
The config files for Conformer-CTC model contain character-based encoding and sub-word encoding at
``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_char.yaml`` and ``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_bpe.yaml``
respectively. Some components of the configs of :ref:`Conformer-CTC <Conformer-CTC_model>` include the following datasets:
* ``train_ds``, ``validation_ds``, and ``test_ds``
* opimizer (``optim``)
* augmentation (``spec_augment``)
* ``decoder``
* ``trainer``
* ``exp_manager``
There should be a tokenizer section where you can
specify the tokenizer if you want to use sub-word encoding instead of character-based encoding.
The encoder section includes the details about the Conformer-CTC encoder architecture. You may find more information in the
config files and also :ref:`nemo.collections.asr.modules.ConformerEncoder <conformer-encoder-api>`.
Conformer-Transducer
~~~~~~~~~~~~~~~~~~~~
Please refer to the model page of :ref:`Conformer-Transducer <Conformer-Transducer_model>` for more information on this model.
Transducer Configurations
-------------------------
All CTC-based ASR model configs can be modified to support Transducer loss training. Below, we discuss the modifications required in the config to enable Transducer training. All modifications are made to the ``model`` config.
Model Defaults
~~~~~~~~~~~~~~
It is a subsection to the model config representing the default values shared across the entire model represented as ``model.model_defaults``.
There are three values that are primary components of a transducer model. They are :
* ``enc_hidden``: The hidden dimension of the final layer of the Encoder network.
* ``pred_hidden``: The hidden dimension of the final layer of the Prediction network.
* ``joint_hidden``: The hidden dimension of the intermediate layer of the Joint network.
One can access these values inside the config by using OmegaConf interpolation as follows :
.. code-block:: yaml
model:
...
model_defaults:
enc_hidden: 256
pred_hidden: 256
joint_hidden: 256
...
decoder:
...
prednet:
pred_hidden: ${model.model_defaults.pred_hidden}
Acoustic Encoder Model
~~~~~~~~~~~~~~~~~~~~~~
The transducer model is comprised of three models combined. One of these models is the Acoustic (encoder) model. We should be able to drop in any CTC Acoustic model config into this section of the transducer config.
The only condition that needs to be met is that **the final layer of the acoustic model must have the hidden dimension defined in ``model_defaults.enc_hidden``**.
Decoder / Prediction Model
~~~~~~~~~~~~~~~~~~~~~~~~~~
The Prediction model is generally an autoregressive, causal model that consumes text tokens and returns embeddings that will be used by the Joint model. The base config for an LSTM based Prediction network can be found in the ``decoder`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
**This config can be copy-pasted into any custom transducer model with no modification.**
Let us discuss some of the important arguments:
* ``blank_as_pad``: In ordinary transducer models, the embedding matrix does not acknowledge the ``Transducer Blank`` token (similar to CTC Blank). However, this causes the autoregressive loop to be more complicated and less efficient. Instead, this flag which is set by default, will add the ``Transducer Blank`` token to the embedding matrix - and use it as a pad value (zeros tensor). This enables more efficient inference without harming training. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
* ``prednet.pred_hidden``: The hidden dimension of the LSTM and the output dimension of the Prediction network.
.. code-block:: yaml
decoder:
_target_: nemo.collections.asr.modules.RNNTDecoder
normalization_mode: null
random_state_sampling: false
blank_as_pad: true
prednet:
pred_hidden: ${model.model_defaults.pred_hidden}
pred_rnn_layers: 1
t_max: null
dropout: 0.0
Joint Model
~~~~~~~~~~~
The Joint model is a simple feed-forward Multi-Layer Perceptron network. This MLP accepts the output of the Acoustic and Prediction models and computes a joint probability distribution over the entire vocabulary space. The base config for the Joint network can be found in the ``joint`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
**This config can be copy-pasted into any custom transducer model with no modification.**
The Joint model config has several essential components which we discuss below :
* ``log_softmax``: Due to the cost of computing softmax on such large tensors, the Numba CUDA implementation of RNNT loss will implicitly compute the log softmax when called (so its inputs should be logits). The CPU version of the loss doesn't face such memory issues so it requires log-probabilities instead. Since the behaviour is different for CPU-GPU, the ``None`` value will automatically switch behaviour dependent on whether the input tensor is on a CPU or GPU device.
* ``preserve_memory``: This flag will call ``torch.cuda.empty_cache()`` at certain critical sections when computing the Joint tensor. While this operation might allow us to preserve some memory, the empty_cache() operation is tremendously slow and will slow down training by an order of magnitude or more. It is available to use but not recommended.
* ``fuse_loss_wer``: This flag performs "batch splitting" and then "fused loss + metric" calculation. It will be discussed in detail in the next tutorial that will train a Transducer model.
* ``fused_batch_size``: When the above flag is set to True, the model will have two distinct "batch sizes". The batch size provided in the three data loader configs (``model.*_ds.batch_size``) will now be the ``Acoustic model`` batch size, whereas the ``fused_batch_size`` will be the batch size of the ``Prediction model``, the ``Joint model``, the ``transducer loss`` module and the ``decoding`` module.
* ``jointnet.joint_hidden``: The hidden intermediate dimension of the joint network.
.. code-block:: yaml
joint:
_target_: nemo.collections.asr.modules.RNNTJoint
log_softmax: null # sets it according to cpu/gpu device
# fused mode
fuse_loss_wer: false
fused_batch_size: 16
jointnet:
joint_hidden: ${model.model_defaults.joint_hidden}
activation: "relu"
dropout: 0.0
Sampled Softmax Joint Model
^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are some situations where a large vocabulary with a Transducer model - such as for multilingual models with a large
number of languages. In this setting, we need to consider the cost of memory of training Transducer networks which does
not allow large vocabulary.
For such cases, one can instead utilize the ``SampledRNNTJoint`` module instead of the usual ``RNNTJoint`` module, in order
to compute the loss using a sampled subset of the vocabulary rather than the full vocabulary file.
It adds only one additional parameter :
* ``n_samples``: Specifies the minimum number of tokens to sample from the vocabulary space,
excluding the RNNT blank token. If a given value is larger than the entire vocabulary size,
then the full vocabulary will be used.
The only difference in config required is to replace ``nemo.collections.asr.modules.RNNTJoint`` with ``nemo.collections.asr.modules.SampledRNNTJoint``
.. code-block:: yaml
joint:
_target_: nemo.collections.asr.modules.SampledRNNTJoint
n_samples: 500
... # All other arguments from RNNTJoint can be used after this.
Effect of Batch Splitting / Fused Batch step
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The following information below explain why memory is an issue when training Transducer models and how NeMo tackles the issue with its Fused Batch step. The material can be read for a thorough understanding, otherwise, it can be skipped. You can also follow these steps in the "ASR_with_Transducers" tutorial.
**Diving deeper into the memory costs of Transducer Joint**
One of the significant limitations of Transducers is the exorbitant memory cost of computing the Joint module. The Joint module is comprised of two steps.
1) Projecting the Acoustic and Transcription feature dimensions to some standard hidden dimension (specified by model.model_defaults.joint_hidden)
2) Projecting this intermediate hidden dimension to the final vocabulary space to obtain the transcription.
Take the following example.
BS=32 ; T (after 2x stride) = 800, U (with character encoding) = 400-450 tokens, Vocabulary size V = 28 (26 alphabet chars, space and apostrophe). Let the hidden dimension of the Joint model be 640 (Most Google Transducer papers use hidden dimension of 640).
* Memory (Hidden, gb) = 32 x 800 x 450 x 640 x 4 = 29.49 gigabytes (4 bytes per float).
* Memory (Joint, gb) = 32 x 800 x 450 x 28 x 4 = 1.290 gigabytes (4 bytes per float)
**NOTE**: This is just for the forward pass! We need to double this memory to store gradients! This much memory is also just for the Joint model **alone**. Far more memory is required for the Prediction model as well as the large Acoustic model itself and its gradients!
Even with mixed precision, that's ~30 GB of GPU RAM for just 1 part of the network + its gradients.
Effect of Fused Batch Step
^^^^^^^^^^^^^^^^^^^^^^^^^^
The fundamental problem is that the joint tensor grows in size when ``[T x U]`` grows in size. This growth in memory cost is due to many reasons - either by model construction (downsampling) or the choice of dataset preprocessing (character tokenization vs. sub-word tokenization).
Another dimension that NeMo can control is **batch**. Due to how we batch our samples, small and large samples all get clumped together into a single batch. So even though the individual samples are not all as long as the maximum length of T and U in that batch, when a batch of such samples is constructed, it will consume a significant amount of memory for the sake of compute efficiency.
So as is always the case - **trade-off compute speed for memory savings**.
The fused operation goes as follows :
1) Forward the entire acoustic model in a single pass. (Use global batch size here for acoustic model - found in ``model.*_ds.batch_size``)
2) Split the Acoustic Model's logits by ``fused_batch_size`` and loop over these sub-batches.
3) Construct a sub-batch of same ``fused_batch_size`` for the Prediction model. Now the target sequence length is U_sub-batch < U.
4) Feed this U_sub-batch into the Joint model, along with a sub-batch from the Acoustic model (with T_sub-batch < T). Remember, we only have to slice off a part of the acoustic model here since we have the full batch of samples (B, T, D) from the acoustic model.
5) Performing steps (3) and (4) yields T_sub-batch and U_sub-batch. Perform sub-batch joint step - costing an intermediate (B, T_sub-batch, U_sub-batch, V) in memory.
6) Compute loss on sub-batch and preserve in a list to be later concatenated.
7) Compute sub-batch metrics (such as Character / Word Error Rate) using the above Joint tensor and sub-batch of ground truth labels. Preserve the scores to be averaged across the entire batch later.
8) Delete the sub-batch joint matrix (B, T_sub-batch, U_sub-batch, V). Only gradients from .backward() are preserved now in the computation graph.
9) Repeat steps (3) - (8) until all sub-batches are consumed.
10) Cleanup step. Compute full batch WER and log. Concatenate loss list and pass to PTL to compute the equivalent of the original (full batch) Joint step. Delete ancillary objects necessary for sub-batching.
Transducer Decoding
~~~~~~~~~~~~~~~~~~~
Models which have been trained with CTC can transcribe text simply by performing a regular argmax over the output of their decoder. For transducer-based models, the three networks must operate in a synchronized manner in order to transcribe the acoustic features. The base config for the Transducer decoding step can be found in the ``decoding`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
**This config can be copy-pasted into any custom transducer model with no modification.**
The most important component at the top level is the ``strategy``. It can take one of many values:
* ``greedy``: This is sample-level greedy decoding. It is generally exceptionally slow as each sample in the batch will be decoded independently. For publications, this should be used alongside batch size of 1 for exact results.
* ``greedy_batch``: This is the general default and should nearly match the ``greedy`` decoding scores (if the acoustic features are not affected by feature mixing in batch mode). Even for small batch sizes, this strategy is significantly faster than ``greedy``.
* ``beam``: Runs beam search with the implicit language model of the Prediction model. It will generally be quite slow, and might need some tuning of the beam size to get better transcriptions.
* ``tsd``: Time synchronous decoding. Please refer to the paper: `Alignment-Length Synchronous Decoding for RNN Transducer <https://ieeexplore.ieee.org/document/9053040>`_ for details on the algorithm implemented. Time synchronous decoding (TSD) execution time grows by the factor T * max_symmetric_expansions. For longer sequences, T is greater and can therefore take a long time for beams to obtain good results. TSD also requires more memory to execute.
* ``alsd``: Alignment-length synchronous decoding. Please refer to the paper: `Alignment-Length Synchronous Decoding for RNN Transducer <https://ieeexplore.ieee.org/document/9053040>`_ for details on the algorithm implemented. Alignment-length synchronous decoding (ALSD) execution time is faster than TSD, with a growth factor of T + U_max, where U_max is the maximum target length expected during execution. Generally, T + U_max < T * max_symmetric_expansions. However, ALSD beams are non-unique. Therefore it is required to use larger beam sizes to achieve the same (or close to the same) decoding accuracy as TSD. For a given decoding accuracy, it is possible to attain faster decoding via ALSD than TSD.
* ``maes``: Modified Adaptive Expansion Search Decoding. Please refer to the paper `Accelerating RNN Transducer Inference via Adaptive Expansion Search <https://ieeexplore.ieee.org/document/9250505>`_. Modified Adaptive Synchronous Decoding (mAES) execution time is adaptive w.r.t the number of expansions (for tokens) required per timestep. The number of expansions can usually be constrained to 1 or 2, and in most cases 2 is sufficient. This beam search technique can possibly obtain superior WER while sacrificing some evaluation time.
.. code-block:: yaml
decoding:
strategy: "greedy_batch"
# preserve decoding alignments
preserve_alignments: false
# Overrides the fused batch size after training.
# Setting it to -1 will process whole batch at once when combined with `greedy_batch` decoding strategy
fused_batch_size: -1
# greedy strategy config
greedy:
max_symbols: 10
# beam strategy config
beam:
beam_size: 2
score_norm: true
softmax_temperature: 1.0 # scale the logits by some temperature prior to softmax
tsd_max_sym_exp: 10 # for Time Synchronous Decoding, int > 0
alsd_max_target_len: 5.0 # for Alignment-Length Synchronous Decoding, float > 1.0
maes_num_steps: 2 # for modified Adaptive Expansion Search, int > 0
maes_prefix_alpha: 1 # for modified Adaptive Expansion Search, int > 0
maes_expansion_beta: 2 # for modified Adaptive Expansion Search, int >= 0
maes_expansion_gamma: 2.3 # for modified Adaptive Expansion Search, float >= 0
Transducer Loss
~~~~~~~~~~~~~~~
This section configures the type of Transducer loss itself, along with possible sub-sections. By default, an optimized implementation of Transducer loss will be used which depends on Numba for CUDA acceleration. The base config for the Transducer loss section can be found in the ``loss`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
**This config can be copy-pasted into any custom transducer model with no modification.**
The loss config is based on a resolver pattern and can be used as follows:
1) ``loss_name``: ``default`` is generally a good option. Will select one of the available resolved losses and match the kwargs from a sub-configs passed via explicit ``{loss_name}_kwargs`` sub-config.
2) ``{loss_name}_kwargs``: This sub-config is passed to the resolved loss above and can be used to configure the resolved loss.
.. code-block:: yaml
loss:
loss_name: "default"
warprnnt_numba_kwargs:
fastemit_lambda: 0.0
FastEmit Regularization
^^^^^^^^^^^^^^^^^^^^^^^
FastEmit Regularization is supported for the default Numba based WarpRNNT loss. Recently proposed regularization approach - `FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization <https://arxiv.org/abs/2010.11148>`_ allows us near-direct control over the latency of transducer models.
Refer to the above paper for results and recommendations of ``fastemit_lambda``.
For decoding customization (confidence scores, CUDA graphs, language models, word boosting), see :doc:`ASR Language Modeling and Customization <./asr_language_modeling_and_customization>`.
InterCTC Config
---------------
All CTC-based models also support `InterCTC loss <https://arxiv.org/abs/2102.03216>`_. To use it, you need to specify
2 parameters as in example below
.. code-block:: yaml
model:
# ...
interctc:
loss_weights: [0.3]
apply_at_layers: [8]
which can be used to reproduce the default setup from the paper (assuming the total number of layers is 18).
You can also specify multiple CTC losses from different layers, e.g., to get 2 losses from layers 3 and 8 with
weights 0.1 and 0.3, specify:
.. code-block:: yaml
model:
# ...
interctc:
loss_weights: [0.1, 0.3]
apply_at_layers: [3, 8]
Note that the final-layer CTC loss weight is automatically computed to normalize
all weight to 1 (0.6 in the example above).
Stochastic Depth Config
-----------------------
`Stochastic Depth <https://arxiv.org/abs/2102.03216>`_ is a useful technique for regularizing ASR model training.
Currently it's only supported for :ref:`nemo.collections.asr.modules.ConformerEncoder <conformer-encoder-api>`. To
use it, specify the following parameters in the encoder config file to reproduce the default setup from the paper:
.. code-block:: yaml
model:
# ...
encoder:
# ...
stochastic_depth_drop_prob: 0.3
stochastic_depth_mode: linear # linear or uniform
stochastic_depth_start_layer: 1
See :ref:`documentation of ConformerEncoder <conformer-encoder-api>` for more details. Note that stochastic depth
is supported for both CTC and Transducer model variations (or any other kind of model/loss that's using
conformer as encoder).
.. _Hybrid-Transducer-CTC-Prompt_model__Config:
Hybrid-Transducer-CTC with Prompt Conditioning Configuration
------------------------------------------------------------
The :ref:`Hybrid-Transducer-CTC model with prompt conditioning <Hybrid-Transducer-CTC-Prompt_model>`
(``EncDecHybridRNNTCTCBPEModelWithPrompt``) extends the base hybrid model to support prompt-based multilingual ASR/AST.
**Key Configuration Parameters:**
The model introduces several prompt-specific configuration parameters in the ``model_defaults`` section:
.. code-block:: yaml
model:
model_defaults:
# Prompt Feature Configuration
initialize_prompt_feature: true # Enable prompt conditioning
num_prompts: 128 # Number of supported prompt categories
prompt_dictionary: { # Mapping from identifiers to prompt indices
# Language prompts (0-99)
'en-US': 0,
'de-DE': 1,
'fr-FR': 2,
'es-ES': 3,
# Task/domain prompts (100-127)
'pnc': 100, # Punctuation mode
'no_pnc': 101, # No punctuation mode
}
**Dataset Configuration:**
The model requires training data with prompt annotations when using Lhotse datasets:
.. code-block:: yaml
model:
train_ds:
use_lhotse: true
initialize_prompt_feature: true
prompt_field: "target_lang" # Field name for prompt extraction
prompt_dictionary: ${model.model_defaults.prompt_dictionary}
num_prompts: ${model.model_defaults.num_prompts}
validation_ds:
use_lhotse: true
initialize_prompt_feature: true
prompt_field: "target_lang"
prompt_dictionary: ${model.model_defaults.prompt_dictionary}
num_prompts: ${model.model_defaults.num_prompts}
**Manifest Format:**
Training manifests should include prompt information:
.. code-block:: json
{
"audio_filepath": "/path/to/audio.wav",
"text": "transcription text",
"duration": 10.5,
"target_lang": "en-US"
}
**Example Configuration:**
A complete example configuration can be found at:
``<NeMo_git_root>/examples/asr/conf/fastconformer/hybrid_transducer_ctc/fastconformer_hybrid_transducer_ctc_bpe_prompt.yaml``
**Training Command:**
.. code-block:: bash
python <NeMo_git_root>/examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe_prompt.py \
--config-path=<NeMo_git_root>/examples/asr/conf/fastconformer/hybrid_transducer_ctc/ \
--config-name=fastconformer_hybrid_transducer_ctc_bpe_prompt.yaml \
model.train_ds.manifest_filepath=<path_to_train_manifest> \
model.validation_ds.manifest_filepath=<path_to_val_manifest> \
model.tokenizer.dir=<path_to_tokenizer> \
model.test_ds.manifest_filepath=<path_to_test_manifest>
.. _RNNT-Prompt_model__Config:
RNN-T with Prompt Conditioning Configuration
--------------------------------------------
The :ref:`RNN-T model with prompt conditioning <RNNT-Prompt_model>`
(``EncDecRNNTBPEModelWithPrompt``) is the RNN-T-only counterpart of the hybrid prompt model
(no auxiliary CTC head). It targets cache-aware streaming multilingual ASR using the same
one-hot language-ID prompt concatenation as the hybrid variant.
**Key Configuration Parameters:**
The prompt-specific parameters live in the ``model_defaults`` section, mirroring the hybrid
variant:
.. code-block:: yaml
model:
model_defaults:
# Prompt Feature Configuration
initialize_prompt_feature: true # Enable prompt conditioning
num_prompts: 128 # Number of supported prompt categories
prompt_dictionary: { # Mapping from identifiers to prompt indices
'en-US': 0,
'de-DE': 1,
'fr-FR': 2,
'es-ES': 3,
# ... additional language codes ...
'auto': 127, # Per-sample dynamic language (read from manifest)
}
**Dataset Configuration:**
The model uses the same index-based Lhotse dataset
(``LhotseSpeechToTextBpeDatasetWithPromptIndex``) as the hybrid model:
.. code-block:: yaml
model:
train_ds:
use_lhotse: true
initialize_prompt_feature: true
prompt_field: "target_lang" # Field name for per-sample prompt extraction
prompt_dictionary: ${model.model_defaults.prompt_dictionary}
num_prompts: ${model.model_defaults.num_prompts}
validation_ds:
use_lhotse: true
initialize_prompt_feature: true
prompt_field: "target_lang"
prompt_dictionary: ${model.model_defaults.prompt_dictionary}
num_prompts: ${model.model_defaults.num_prompts}
**Manifest Format:**
Identical to the hybrid model — each entry needs a ``target_lang`` field:
.. code-block:: json
{
"audio_filepath": "/path/to/audio.wav",
"text": "transcription text",
"duration": 10.5,
"target_lang": "en-US"
}
**Example Configuration:**
A cache-aware streaming RNN-T prompt config ships at:
``<NeMo_git_root>/examples/asr/conf/fastconformer/cache_aware_streaming/fastconformer_transducer_bpe_streaming_prompt.yaml``
**Training Command:**
.. code-block:: bash
python <NeMo_git_root>/examples/asr/asr_transducer/speech_to_text_rnnt_bpe_prompt.py \
--config-path=<NeMo_git_root>/examples/asr/conf/fastconformer/cache_aware_streaming/ \
--config-name=fastconformer_transducer_bpe_streaming_prompt.yaml \
model.train_ds.manifest_filepath=<path_to_train_manifest> \
model.validation_ds.manifest_filepath=<path_to_val_manifest> \
model.tokenizer.dir=<path_to_tokenizer> \
model.test_ds.manifest_filepath=<path_to_test_manifest>
**Streaming Inference:**
The standard cache-aware streaming inference script accepts ``target_lang`` (and the optional
``strip_lang_tags`` / ``lang_tag_pattern`` flags) for prompt-conditioned models:
.. code-block:: bash
python <NeMo_git_root>/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
model_path=<path_to_nemo_checkpoint> \
dataset_manifest=<path_to_manifest> \
target_lang=<en-US|auto|...> \
strip_lang_tags=true
docs/source/asr/data/asrlm_results.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Model Base Class,Model Card
asrlm_en_transformer_large_ls,TransformerLMModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls"
1 Model Name Model Base Class Model Card
2 asrlm_en_transformer_large_ls TransformerLMModel https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls
docs/source/asr/data/benchmark_ar.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`stt_ar_fastconformer_hybrid_large_pcd_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pcd_v1.0>`_,EncDecHybridRNNTCTCBPEModel
`stt_ar_fastconformer_hybrid_large_pc_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_ar_fastconformer_hybrid_large_pcd_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pcd_v1.0>`_ EncDecHybridRNNTCTCBPEModel
3 `stt_ar_fastconformer_hybrid_large_pc_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_be.csv
+4
View File
@@ -0,0 +1,4 @@
Model,Model Base Class
`stt_be_conformer_transducer_large <https://huggingface.co/nvidia/stt_be_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_be_conformer_ctc_large <https://huggingface.co/nvidia/stt_be_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_be_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_be_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_be_conformer_transducer_large <https://huggingface.co/nvidia/stt_be_conformer_transducer_large>`_ EncDecRNNTBPEModel
3 `stt_be_conformer_ctc_large <https://huggingface.co/nvidia/stt_be_conformer_ctc_large>`_ EncDecCTCModelBPE
4 `stt_be_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_be_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_bg.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_by.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
3 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_ca.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`stt_ca_conformer_ctc_large <https://huggingface.co/nvidia/stt_ca_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_ca_conformer_transducer_large <https://huggingface.co/nvidia/stt_ca_conformer_transducer_large>`_,EncDecRNNTBPEModel
1 Model Model Base Class
2 `stt_ca_conformer_ctc_large <https://huggingface.co/nvidia/stt_ca_conformer_ctc_large>`_ EncDecCTCModelBPE
3 `stt_ca_conformer_transducer_large <https://huggingface.co/nvidia/stt_ca_conformer_transducer_large>`_ EncDecRNNTBPEModel
docs/source/asr/data/benchmark_canary.csv
+4
View File
@@ -0,0 +1,4 @@
Model,Language
`canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_,"English, French, German, Spanish"
`canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_,"English, French, German, Spanish"
`canary-1b <https://huggingface.co/nvidia/canary-1b>`_ ,"English, French, German, Spanish"
1 Model Language
2 `canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_ English, French, German, Spanish
3 `canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_ English, French, German, Spanish
4 `canary-1b <https://huggingface.co/nvidia/canary-1b>`_ English, French, German, Spanish
docs/source/asr/data/benchmark_cn.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`stt_zh_conformer_transducer_large <https://huggingface.co/nvidia/stt_zh_conformer_transducer_large>`_,EncDecRNNTBPEModel
1 Model Model Base Class
2 `stt_zh_conformer_transducer_large <https://huggingface.co/nvidia/stt_zh_conformer_transducer_large>`_ EncDecRNNTBPEModel
docs/source/asr/data/benchmark_code_switching.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Language
`stt_enes_conformer_ctc_large_codesw <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_ctc_large_codesw>`_,"English, Spanish"
`stt_enes_conformer_transducer_large_codesw <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large_codesw>`_,"English, Spanish"
1 Model Language
2 `stt_enes_conformer_ctc_large_codesw <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_ctc_large_codesw>`_ English, Spanish
3 `stt_enes_conformer_transducer_large_codesw <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large_codesw>`_ English, Spanish
docs/source/asr/data/benchmark_cs.csv
+4
View File
@@ -0,0 +1,4 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_ca_conformer_ctc_large <https://huggingface.co/nvidia/stt_ca_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_ca_conformer_transducer_large <https://huggingface.co/nvidia/stt_ca_conformer_transducer_large>`_,EncDecRNNTBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_ca_conformer_ctc_large <https://huggingface.co/nvidia/stt_ca_conformer_ctc_large>`_ EncDecCTCModelBPE
4 `stt_ca_conformer_transducer_large <https://huggingface.co/nvidia/stt_ca_conformer_transducer_large>`_ EncDecRNNTBPEModel
docs/source/asr/data/benchmark_da.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_de.csv
+10
View File
@@ -0,0 +1,10 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`canary-1b <https://huggingface.co/nvidia/canary-1b>`_,EncDecMultiTaskModel
`stt_de_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_de_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_de_conformer_ctc_large <https://huggingface.co/nvidia/stt_de_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_de_conformer_transducer_large <https://huggingface.co/nvidia/stt_de_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_,EncDecHybridRNNTCTCBPEModel
`canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_,EncDecMultiTaskModel
`canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `canary-1b <https://huggingface.co/nvidia/canary-1b>`_ EncDecMultiTaskModel
4 `stt_de_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_de_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
5 `stt_de_conformer_ctc_large <https://huggingface.co/nvidia/stt_de_conformer_ctc_large>`_ EncDecCTCModelBPE
6 `stt_de_conformer_transducer_large <https://huggingface.co/nvidia/stt_de_conformer_transducer_large>`_ EncDecRNNTBPEModel
7 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
8 `stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_ EncDecHybridRNNTCTCBPEModel
9 `canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_ EncDecMultiTaskModel
10 `canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_el.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_en.csv
+35
View File
@@ -0,0 +1,35 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`parakeet-tdt-1.1b <https://huggingface.co/nvidia/parakeet-tdt-1.1b>`_, EncDecRNNTBPEModel
`parakeet-tdt_ctc-1.1b <https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b>`_, ASRModel
`parakeet-tdt_ctc-110m <https://huggingface.co/nvidia/parakeet-tdt_ctc-110m>`_, ASRModel
`canary-1b <https://huggingface.co/nvidia/canary-1b>`_, EncDecMultiTaskModel
`stt_en_conformer_ctc_large <https://huggingface.co/nvidia/stt_en_conformer_ctc_large>`_, EncDecCTCModelBPE
`parakeet-ctc-0.6b <https://huggingface.co/nvidia/parakeet-ctc-0.6b>`_, EncDecCTCModelBPE
`parakeet-ctc-1.1b <https://huggingface.co/nvidia/parakeet-ctc-1.1b>`_, EncDecCTCModelBPE
`stt_en_conformer_transducer_xlarge <https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge>`_, EncDecRNNTBPEModel
`stt_en_fastconformer_ctc_large <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_large>`_, EncDecCTCModelBPE
`stt_en_fastconformer_hybrid_large_streaming_multi <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_streaming_multi>`_, EncDecHybridRNNTCTCBPEModel
`stt_en_fastconformer_ctc_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xxlarge>`_, EncDecCTCTBPEModel
`stt_en_conformer_transducer_large <https://huggingface.co/nvidia/stt_en_conformer_transducer_large>`_, EncDecRNNTBPEModel
`stt_en_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_pc>`_, EncDecHybridRNNTCTCBPEModel
`stt_en_conformer_ctc_small <https://huggingface.co/nvidia/stt_en_conformer_ctc_small>`_, EncDecCTCModelBPE
`stt_en_fastconformer_transducer_large <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_large>`_, EncDecRNNTBPEModel
`stt_en_fastconformer_transducer_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xlarge>`_, EncDecRNNTBPEModel
`stt_en_fastconformer_transducer_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xxlarge>`_, EncDecRNNTBPEModel
`stt_en_fastconformer_ctc_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xlarge>`_, EncDecCTCTBPEModel
`stt_enes_conformer_ctc_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_ctc_large>`_, EncDecCTCModelBPE
`stt_enes_conformer_transducer_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large>`_, EncDecRNNTBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_, EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_, EncDecHybridRNNTCTCBPEModel
`canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_, EncDecMultiTaskModel
`low-frame-rate-speech-codec-22khz <https://huggingface.co/nvidia/low-frame-rate-speech-codec-22khz>`_, EncDecCTCModelBPE
`stt_en_fastconformer_hybrid_medium_streaming_80ms_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms_pc>`_, EncDecHybridRNNTCTCBPEModel
`parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`_, ASRModel
`parakeet-tdt-0.6b-v2 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2>`_, ASRModel
`multitalker-parakeet-streaming-0.6b-v1 <https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1>`_, EncDecMultiTalkerRNNTBPEModel
`canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_, EncDecMultiTaskModel
`parakeet-rnnt-1.1b <https://huggingface.co/nvidia/parakeet-rnnt-1.1b>`_, EncDecRNNTBPEModel
`stt_en_fastconformer_hybrid_medium_streaming_80ms <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms>`_, EncDecHybridRNNTCTCBPEModel
`parakeet-rnnt-0.6b <https://huggingface.co/nvidia/parakeet-rnnt-0.6b>`_, EncDecRNNTBPEModel
`stt_en_fastconformer_tdt_large <https://huggingface.co/nvidia/stt_en_fastconformer_tdt_large>`_, EncDecRNNTModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `parakeet-tdt-1.1b <https://huggingface.co/nvidia/parakeet-tdt-1.1b>`_ EncDecRNNTBPEModel
4 `parakeet-tdt_ctc-1.1b <https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b>`_ ASRModel
5 `parakeet-tdt_ctc-110m <https://huggingface.co/nvidia/parakeet-tdt_ctc-110m>`_ ASRModel
6 `canary-1b <https://huggingface.co/nvidia/canary-1b>`_ EncDecMultiTaskModel
7 `stt_en_conformer_ctc_large <https://huggingface.co/nvidia/stt_en_conformer_ctc_large>`_ EncDecCTCModelBPE
8 `parakeet-ctc-0.6b <https://huggingface.co/nvidia/parakeet-ctc-0.6b>`_ EncDecCTCModelBPE
9 `parakeet-ctc-1.1b <https://huggingface.co/nvidia/parakeet-ctc-1.1b>`_ EncDecCTCModelBPE
10 `stt_en_conformer_transducer_xlarge <https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge>`_ EncDecRNNTBPEModel
11 `stt_en_fastconformer_ctc_large <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_large>`_ EncDecCTCModelBPE
12 `stt_en_fastconformer_hybrid_large_streaming_multi <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_streaming_multi>`_ EncDecHybridRNNTCTCBPEModel
13 `stt_en_fastconformer_ctc_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xxlarge>`_ EncDecCTCTBPEModel
14 `stt_en_conformer_transducer_large <https://huggingface.co/nvidia/stt_en_conformer_transducer_large>`_ EncDecRNNTBPEModel
15 `stt_en_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
16 `stt_en_conformer_ctc_small <https://huggingface.co/nvidia/stt_en_conformer_ctc_small>`_ EncDecCTCModelBPE
17 `stt_en_fastconformer_transducer_large <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_large>`_ EncDecRNNTBPEModel
18 `stt_en_fastconformer_transducer_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xlarge>`_ EncDecRNNTBPEModel
19 `stt_en_fastconformer_transducer_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xxlarge>`_ EncDecRNNTBPEModel
20 `stt_en_fastconformer_ctc_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xlarge>`_ EncDecCTCTBPEModel
21 `stt_enes_conformer_ctc_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_ctc_large>`_ EncDecCTCModelBPE
22 `stt_enes_conformer_transducer_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large>`_ EncDecRNNTBPEModel
23 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
24 `stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_ EncDecHybridRNNTCTCBPEModel
25 `canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_ EncDecMultiTaskModel
26 `low-frame-rate-speech-codec-22khz <https://huggingface.co/nvidia/low-frame-rate-speech-codec-22khz>`_ EncDecCTCModelBPE
27 `stt_en_fastconformer_hybrid_medium_streaming_80ms_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms_pc>`_ EncDecHybridRNNTCTCBPEModel
28 `parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`_ ASRModel
29 `parakeet-tdt-0.6b-v2 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2>`_ ASRModel
30 `multitalker-parakeet-streaming-0.6b-v1 <https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1>`_ EncDecMultiTalkerRNNTBPEModel
31 `canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_ EncDecMultiTaskModel
32 `parakeet-rnnt-1.1b <https://huggingface.co/nvidia/parakeet-rnnt-1.1b>`_ EncDecRNNTBPEModel
33 `stt_en_fastconformer_hybrid_medium_streaming_80ms <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms>`_ EncDecHybridRNNTCTCBPEModel
34 `parakeet-rnnt-0.6b <https://huggingface.co/nvidia/parakeet-rnnt-0.6b>`_ EncDecRNNTBPEModel
35 `stt_en_fastconformer_tdt_large <https://huggingface.co/nvidia/stt_en_fastconformer_tdt_large>`_ EncDecRNNTModel
docs/source/asr/data/benchmark_eo.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`stt_eo_conformer_transducer_large <https://huggingface.co/nvidia/stt_eo_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_eo_conformer_ctc_large <https://huggingface.co/nvidia/stt_eo_conformer_ctc_large>`_,EncDecCTCModelBPE
1 Model Model Base Class
2 `stt_eo_conformer_transducer_large <https://huggingface.co/nvidia/stt_eo_conformer_transducer_large>`_ EncDecRNNTBPEModel
3 `stt_eo_conformer_ctc_large <https://huggingface.co/nvidia/stt_eo_conformer_ctc_large>`_ EncDecCTCModelBPE
docs/source/asr/data/benchmark_es.csv
+13
View File
@@ -0,0 +1,13 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`canary-1b <https://huggingface.co/nvidia/canary-1b>`_,EncDecMultiTaskModel
`stt_es_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_es_conformer_transducer_large <https://huggingface.co/nvidia/stt_es_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_es_conformer_ctc_large <https://huggingface.co/nvidia/stt_es_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_enes_conformer_ctc_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_enes_conformer_transducer_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_,EncDecHybridRNNTCTCBPEModel
`stt_es_fastconformer_hybrid_large_pc_nc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc_nc>`_,EncDecHybridRNNTCTCBPEModel
`canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_,EncDecMultiTaskModel
`canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `canary-1b <https://huggingface.co/nvidia/canary-1b>`_ EncDecMultiTaskModel
4 `stt_es_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
5 `stt_es_conformer_transducer_large <https://huggingface.co/nvidia/stt_es_conformer_transducer_large>`_ EncDecRNNTBPEModel
6 `stt_es_conformer_ctc_large <https://huggingface.co/nvidia/stt_es_conformer_ctc_large>`_ EncDecCTCModelBPE
7 `stt_enes_conformer_ctc_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_ctc_large>`_ EncDecCTCModelBPE
8 `stt_enes_conformer_transducer_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large>`_ EncDecRNNTBPEModel
9 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
10 `stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_ EncDecHybridRNNTCTCBPEModel
11 `stt_es_fastconformer_hybrid_large_pc_nc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc_nc>`_ EncDecHybridRNNTCTCBPEModel
12 `canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_ EncDecMultiTaskModel
13 `canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_et.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_fa.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`stt_fa_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_fa_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_fastconformer_hybrid.csv
+19
View File
@@ -0,0 +1,19 @@
Model,Language
`stt_ar_fastconformer_hybrid_large_pc_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0>`_,Arabic
`stt_hy_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc>`_,Armenian
`stt_be_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_be_fastconformer_hybrid_large_pc>`_,Belarusian
`stt_hr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hr_fastconformer_hybrid_large_pc>`_,Croatian
`stt_nl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_nl_fastconformer_hybrid_large_pc>`_,Dutch
`stt_en_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_pc>`_,English
`stt_fr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_fr_fastconformer_hybrid_large_pc>`_,French
`stt_ka_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc>`_,Georgian
`stt_de_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_de_fastconformer_hybrid_large_pc>`_,German
`stt_it_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_it_fastconformer_hybrid_large_pc>`_,Italian
`stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`_,"Kazakh, Russian"
`stt_pt_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pt_fastconformer_hybrid_large_pc>`_,Portugese
`stt_fa_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large>`_,Persian
`stt_pl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pl_fastconformer_hybrid_large_pc>`_,Polish
`stt_ru_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc>`_,Russian
`stt_es_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc>`_,Spanish
`stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`_,Ukrainian
`stt_uz_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_uz_fastconformer_hybrid_large_pc>`_,Uzbek
1 Model Language
2 `stt_ar_fastconformer_hybrid_large_pc_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0>`_ Arabic
3 `stt_hy_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc>`_ Armenian
4 `stt_be_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_be_fastconformer_hybrid_large_pc>`_ Belarusian
5 `stt_hr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hr_fastconformer_hybrid_large_pc>`_ Croatian
6 `stt_nl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_nl_fastconformer_hybrid_large_pc>`_ Dutch
7 `stt_en_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_pc>`_ English
8 `stt_fr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_fr_fastconformer_hybrid_large_pc>`_ French
9 `stt_ka_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc>`_ Georgian
10 `stt_de_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_de_fastconformer_hybrid_large_pc>`_ German
11 `stt_it_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_it_fastconformer_hybrid_large_pc>`_ Italian
12 `stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`_ Kazakh, Russian
13 `stt_pt_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pt_fastconformer_hybrid_large_pc>`_ Portugese
14 `stt_fa_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large>`_ Persian
15 `stt_pl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pl_fastconformer_hybrid_large_pc>`_ Polish
16 `stt_ru_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc>`_ Russian
17 `stt_es_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc>`_ Spanish
18 `stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`_ Ukrainian
19 `stt_uz_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_uz_fastconformer_hybrid_large_pc>`_ Uzbek
docs/source/asr/data/benchmark_fi.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_fr.csv
+11
View File
@@ -0,0 +1,11 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`canary-1b <https://huggingface.co/nvidia/canary-1b>`_,EncDecMultiTaskModel
`stt_fr_conformer_ctc_large <https://huggingface.co/nvidia/stt_fr_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_fr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_fr_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_fr_conformer_transducer_large <https://huggingface.co/nvidia/stt_fr_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_fr_no_hyphen_conformer_ctc_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_,EncDecHybridRNNTCTCBPEModel
`canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_,EncDecMultiTaskModel
`canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `canary-1b <https://huggingface.co/nvidia/canary-1b>`_ EncDecMultiTaskModel
4 `stt_fr_conformer_ctc_large <https://huggingface.co/nvidia/stt_fr_conformer_ctc_large>`_ EncDecCTCModelBPE
5 `stt_fr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_fr_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
6 `stt_fr_conformer_transducer_large <https://huggingface.co/nvidia/stt_fr_conformer_transducer_large>`_ EncDecRNNTBPEModel
7 `stt_fr_no_hyphen_conformer_ctc_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_fr_conformer_ctc_large>`_ EncDecCTCModelBPE
8 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
9 `stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`_ EncDecHybridRNNTCTCBPEModel
10 `canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_ EncDecMultiTaskModel
11 `canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_hi.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Model Base Class
`stt_hi_conformer_ctc_medium <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_hi_conformer_ctc_medium>`_,EncDecCTCModelBPE
1 Model Name Model Base Class
2 `stt_hi_conformer_ctc_medium <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_hi_conformer_ctc_medium>`_ EncDecCTCModelBPE
docs/source/asr/data/benchmark_hr.csv
+6
View File
@@ -0,0 +1,6 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_hr_conformer_transducer_large <https://huggingface.co/nvidia/stt_hr_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_hr_conformer_ctc_large <https://huggingface.co/nvidia/stt_hr_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_hr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hr_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_hr_conformer_transducer_large <https://huggingface.co/nvidia/stt_hr_conformer_transducer_large>`_ EncDecRNNTBPEModel
4 `stt_hr_conformer_ctc_large <https://huggingface.co/nvidia/stt_hr_conformer_ctc_large>`_ EncDecCTCModelBPE
5 `stt_hr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hr_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
6 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_hu.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_hy.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`stt_hy_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_hy_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_it.csv
+6
View File
@@ -0,0 +1,6 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_it_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_it_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_it_conformer_ctc_large <https://huggingface.co/nvidia/stt_it_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_it_conformer_transducer_large <https://huggingface.co/nvidia/stt_it_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_it_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_it_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
4 `stt_it_conformer_ctc_large <https://huggingface.co/nvidia/stt_it_conformer_ctc_large>`_ EncDecCTCModelBPE
5 `stt_it_conformer_transducer_large <https://huggingface.co/nvidia/stt_it_conformer_transducer_large>`_ EncDecRNNTBPEModel
6 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_jp.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`_,ASRModel
1 Model Model Base Class
2 `parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`_ ASRModel
docs/source/asr/data/benchmark_ka.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`stt_ka_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_ka_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
3 `stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_kab.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`stt_kab_conformer_transducer_large <https://huggingface.co/nvidia/stt_kab_conformer_transducer_large>`_,EncDecRNNTBPEModel
1 Model Model Base Class
2 `stt_kab_conformer_transducer_large <https://huggingface.co/nvidia/stt_kab_conformer_transducer_large>`_ EncDecRNNTBPEModel
docs/source/asr/data/benchmark_kz.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_lt.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_lv.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_mr.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Model Base Class
`stt_mr_conformer_ctc_medium <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_mr_conformer_ctc_medium>`_,EncDecCTCModelBPE
1 Model Name Model Base Class
2 `stt_mr_conformer_ctc_medium <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_mr_conformer_ctc_medium>`_ EncDecCTCModelBPE
docs/source/asr/data/benchmark_mt.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_multilingual.csv
+7
View File
@@ -0,0 +1,7 @@
Model,Model Base Class,Model Card
stt_enes_conformer_ctc_large,EncDecCTCModelBPE,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_ctc_large"
stt_enes_conformer_transducer_large,EncDecRNNTBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_enes_conformer_transducer_large"
stt_multilingual_fastconformer_hybrid_large_pc,EncDecHybridRNNTCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc"
stt_multilingual_fastconformer_hybrid_large_pc_blend_eu,EncDecHybridRNNTCTCBPEModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc_blend_eu"`canary-1b <https://huggingface.co/nvidia/canary-1b>`_,EncDecMultiTaskModel
`canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`_,EncDecMultiTaskModel
`canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`_,EncDecMultiTaskModel
Can't render this file because it contains an unexpected character in line 5 and column 191.
docs/source/asr/data/benchmark_nl.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_nl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_nl_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_nl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_nl_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_parakeet.csv
+10
View File
@@ -0,0 +1,10 @@
Model,Model Base Class
`parakeet-tdt-0.6b-v3 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3>`_, ASRModel
`parakeet-tdt-0.6b-v2 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2>`_,ASRModel
`parakeet-rnnt-0.6b <https://huggingface.co/nvidia/parakeet-rnnt-0.6b>`_,EncDecRNNTBPEModel
`parakeet-rnnt-1.1b <https://huggingface.co/nvidia/parakeet-rnnt-1.1b>`_,EncDecRNNTBPEModel
`parakeet-ctc-0.6b <https://huggingface.co/nvidia/parakeet-ctc-0.6b>`_,EncDecCTCModelBPE
`parakeet-tdt-1.1b <https://huggingface.co/nvidia/parakeet-tdt-1.1b>`_,EncDecRNNTBPEModel
`parakeet-ctc-1.1b <https://huggingface.co/nvidia/parakeet-ctc-1.1b>`_,EncDecCTCModelBPE
`parakeet-tdt_ctc-1.1b <https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b>`_,ASRModel
`parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`_,ASRModel
1 Model Model Base Class
2 `parakeet-tdt-0.6b-v3 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3>`_ ASRModel
3 `parakeet-tdt-0.6b-v2 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2>`_ ASRModel
4 `parakeet-rnnt-0.6b <https://huggingface.co/nvidia/parakeet-rnnt-0.6b>`_ EncDecRNNTBPEModel
5 `parakeet-rnnt-1.1b <https://huggingface.co/nvidia/parakeet-rnnt-1.1b>`_ EncDecRNNTBPEModel
6 `parakeet-ctc-0.6b <https://huggingface.co/nvidia/parakeet-ctc-0.6b>`_ EncDecCTCModelBPE
7 `parakeet-tdt-1.1b <https://huggingface.co/nvidia/parakeet-tdt-1.1b>`_ EncDecRNNTBPEModel
8 `parakeet-ctc-1.1b <https://huggingface.co/nvidia/parakeet-ctc-1.1b>`_ EncDecCTCModelBPE
9 `parakeet-tdt_ctc-1.1b <https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b>`_ ASRModel
10 `parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`_ ASRModel
docs/source/asr/data/benchmark_pl.csv
+4
View File
@@ -0,0 +1,4 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_pl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pl_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_pl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pl_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
4 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_pt.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_pt_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pt_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_pt_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pt_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_ro.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_ru.csv
+7
View File
@@ -0,0 +1,7 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_ru_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_ru_conformer_transducer_large <https://huggingface.co/nvidia/stt_ru_conformer_transducer_large>`_,EncDecRNNTBPEModel
`stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`_,EncDecHybridRNNTCTCBPEModel
`stt_ru_conformer_ctc_large <https://huggingface.co/nvidia/stt_ru_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_ru_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
4 `stt_ru_conformer_transducer_large <https://huggingface.co/nvidia/stt_ru_conformer_transducer_large>`_ EncDecRNNTBPEModel
5 `stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`_ EncDecHybridRNNTCTCBPEModel
6 `stt_ru_conformer_ctc_large <https://huggingface.co/nvidia/stt_ru_conformer_ctc_large>`_ EncDecCTCModelBPE
7 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_rw.csv
+3
View File
@@ -0,0 +1,3 @@
Model,Model Base Class
`stt_rw_conformer_ctc_large <https://huggingface.co/nvidia/stt_rw_conformer_ctc_large>`_,EncDecCTCModelBPE
`stt_rw_conformer_transducer_large <https://huggingface.co/nvidia/stt_rw_conformer_transducer_large>`_,EncDecRNNTBPEModel
1 Model Model Base Class
2 `stt_rw_conformer_ctc_large <https://huggingface.co/nvidia/stt_rw_conformer_ctc_large>`_ EncDecCTCModelBPE
3 `stt_rw_conformer_transducer_large <https://huggingface.co/nvidia/stt_rw_conformer_transducer_large>`_ EncDecRNNTBPEModel
docs/source/asr/data/benchmark_sk.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_sl.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_sv.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
docs/source/asr/data/benchmark_ua.csv
+4
View File
@@ -0,0 +1,4 @@
Model,Model Base Class
`canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_,EncDecMultiTaskModel
`stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
`stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`_ EncDecMultiTaskModel
3 `stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
4 `stt_multilingual_fastconformer_hybrid_large_pc <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_multilingual_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_uz.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`stt_uz_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_uz_fastconformer_hybrid_large_pc>`_,EncDecHybridRNNTCTCBPEModel
1 Model Model Base Class
2 `stt_uz_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_uz_fastconformer_hybrid_large_pc>`_ EncDecHybridRNNTCTCBPEModel
docs/source/asr/data/benchmark_zh.csv
+2
View File
@@ -0,0 +1,2 @@
Model,Model Base Class
`stt_zh_conformer_transducer_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_conformer_transducer_large>`_,EncDecRNNTModel
1 Model Model Base Class
2 `stt_zh_conformer_transducer_large <https://ngc.nvidia.com/catalog/models/nvidia:nemo:stt_zh_conformer_transducer_large>`_ EncDecRNNTModel
docs/source/asr/data/scores/be/conformer_be.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,MCV Test-Set v10 (be)
stt_be_conformer_ctc_large,be,4.7 %
stt_be_conformer_transducer_large,be,3.8 %
1 Model Name Language MCV Test-Set v10 (be)
2 stt_be_conformer_ctc_large be 4.7 %
3 stt_be_conformer_transducer_large be 3.8 %
docs/source/asr/data/scores/by/fastconformer_by.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set v12.0 (be),MCV Test-Set v12.0 (be)
stt_by_fastconformer_hybrid_large_pc,by,2.7 %,2.7 %
1 Model Name Language MCV Dev-Set v12.0 (be) MCV Test-Set v12.0 (be)
2 stt_by_fastconformer_hybrid_large_pc by 2.7 % 2.7 %
docs/source/asr/data/scores/ca/conformer_ca.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,MCV Dev-Set (v??) (ca),MCV Dev-Set v9.0 (ca),MCV Test-Set v9.0 (ca)
stt_ca_conformer_ctc_large,ca,,4.70,4.27
stt_ca_conformer_transducer_large,ca,,4.43,3.85
1 Model Name Language MCV Dev-Set (v??) (ca) MCV Dev-Set v9.0 (ca) MCV Test-Set v9.0 (ca)
2 stt_ca_conformer_ctc_large ca 4.70 4.27
3 stt_ca_conformer_transducer_large ca 4.43 3.85
docs/source/asr/data/scores/de/conformer_de.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,MCV Dev-Set (v??) (de),MCV Dev-Set v12.0 (de),MCV Dev-Set v7.0 (de),MCV Test-Set v12.0 (de),MCV Test-Set v7.0 (de),MLS Dev (en),MLS Test (en),VoxPopuli Dev (de),VoxPopuli Test (de)
stt_de_conformer_ctc_large,de,,,5.84,,6.68,3.85,4.63,12.56,10.51
stt_de_conformer_transducer_large,de,,,4.75,,5.36,3.46,4.19,11.21,9.14
1 Model Name Language MCV Dev-Set (v??) (de) MCV Dev-Set v12.0 (de) MCV Dev-Set v7.0 (de) MCV Test-Set v12.0 (de) MCV Test-Set v7.0 (de) MLS Dev (en) MLS Test (en) VoxPopuli Dev (de) VoxPopuli Test (de)
2 stt_de_conformer_ctc_large de 5.84 6.68 3.85 4.63 12.56 10.51
3 stt_de_conformer_transducer_large de 4.75 5.36 3.46 4.19 11.21 9.14
docs/source/asr/data/scores/de/fastconformer_de.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set (v??) (de),MCV Dev-Set v12.0 (de),MCV Dev-Set v7.0 (de),MCV Test-Set v12.0 (de),MCV Test-Set v7.0 (de),MLS Dev (en),MLS Test (en),VoxPopuli Dev (de),VoxPopuli Test (de)
stt_de_fastconformer_hybrid_large_pc,de,,4.2 %,,4.9 %,,3.3 %,3.8 %,10.8 %,8.7 %
1 Model Name Language MCV Dev-Set (v??) (de) MCV Dev-Set v12.0 (de) MCV Dev-Set v7.0 (de) MCV Test-Set v12.0 (de) MCV Test-Set v7.0 (de) MLS Dev (en) MLS Test (en) VoxPopuli Dev (de) VoxPopuli Test (de)
2 stt_de_fastconformer_hybrid_large_pc de 4.2 % 4.9 % 3.3 % 3.8 % 10.8 % 8.7 %
docs/source/asr/data/scores/en/conformer_en.csv
+28
View File
@@ -0,0 +1,28 @@
Model Name,Language,EuroParl Test Set (en),Fisher Test Set (en),Librispeech Dev-Clean,Librispeech Dev-Other,Librispeech Test-Clean,Librispeech Test-Other,MCV Test-Set v11.0 (en),MCV Test-Set v8.0 (en),MLS Dev (en),MLS Test (en),NSC Part1,NSC Part6,Peoples Speech Test v1,SLR 83 Test,SPGI Test,VoxPopuli Test (en),WSJ Dev 93,WSJ Eval 92
stt_en_conformer_ctc_small,en,,,3.6,8.1,3.7,8.1,,,,,,,,,,,,
stt_en_conformer_ctc_medium,en,,,2.5,5.8,2.6,5.9,,,,,,,,,,,,
stt_en_conformer_ctc_large,en,,,1.9,4.4,2.1,4.5,,,,,,,,,,,,
stt_en_conformer_ctc_xlarge,en,,,1.77 %,3.79 %,2.00 %,3.74 %,,7.88 %,,5.99 %,,6.44 %,22.90 %,5.50 %,,,2.36 %,
stt_en_conformer_ctc_small_ls,en,,,3.3,8.8,3.4,8.8,,,,,,,,,,,,
stt_en_conformer_ctc_medium_ls,en,,,2.7,7.4,3.0,7.3,,,,,,,,,,,,
stt_en_conformer_ctc_large_ls,en,,,2.4,6.2,2.7,6.0,,,,,,,,,,,,
stt_en_conformer_transducer_small,en,,,2.8,6.6,2.5,6.6,,,,,,,,,,,,
stt_en_conformer_transducer_medium,en,,,2.0,4.6,2.1,4.7,,,,,,,,,,,,
stt_en_conformer_transducer_large,en,,,1.6,3.5,1.7,3.7,,,,,,,,,,,,
stt_en_conformer_transducer_large_ls,en,,,2.1,5.0,2.3,5.1,,,,,,,,,,,,
stt_en_conformer_transducer_xlarge,en,,,1.48 %,2.95 %,1.62 %,3.01 %,,6.46 %,4.59 %,5.32 %,5.70 %,6.47 %,21.32 %,,,,2.05 %,1.17 %
stt_en_conformer_transducer_xxlarge,en,,,1.52 %,3.09 %,1.72 %,3.14 %,,,5.29 %,5.85 %,6.64 %,,,,,,2.42 %,1.49 %
stt_en_fastconformer_hybrid_large_streaming_80ms (CTC),en,,,,,3.5 %,8.1 %,,,10.2 %,7.2 %,,,,,,,3.5 %,2.3 %
stt_en_fastconformer_hybrid_large_streaming_480ms (CTC),en,,,,,3.6 %,7.5 %,,,9.8 %,7.0 %,,,,,,,3.5 %,2.1 %
stt_en_fastconformer_hybrid_large_streaming_1040ms (CTC),en,,,,,2.7 %,6.4 %,,,9.0 %,7.0 %,,,,,,,3.2 %,1.9 %
stt_en_fastconformer_hybrid_large_streaming_80ms (RNNT),en,,,,,2.7 %,6.5 %,,,9.1 %,6.9 %,,,,,,,3.2 %,1.9 %
stt_en_fastconformer_hybrid_large_streaming_480ms (RNNT),en,,,,,2.7 %,6.1 %,,,8.5 %,6.7 %,,,,,,,3.1 %,1.8 %
stt_en_fastconformer_hybrid_large_streaming_1040ms (RNNT),en,,,,,2.3 %,5.5 %,,,8.0 %,6.6 %,,,,,,,2.9 %,1.6 %
stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 0ms),en,,,,,,7.0 %,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 80ms),en,,,,,,6.4 %,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 480),en,,,,,,5.7 %,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 1040),en,,,,,,5.4 %,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 0ms),en,,,,,,8.4 %,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 80ms),en,,,,,,7.8 %,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 480),en,,,,,,6.7 %,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 1040),en,,,,,,6.2 %,,,,,,,,,,,,
1 Model Name Language EuroParl Test Set (en) Fisher Test Set (en) Librispeech Dev-Clean Librispeech Dev-Other Librispeech Test-Clean Librispeech Test-Other MCV Test-Set v11.0 (en) MCV Test-Set v8.0 (en) MLS Dev (en) MLS Test (en) NSC Part1 NSC Part6 Peoples Speech Test v1 SLR 83 Test SPGI Test VoxPopuli Test (en) WSJ Dev 93 WSJ Eval 92
2 stt_en_conformer_ctc_small en 3.6 8.1 3.7 8.1
3 stt_en_conformer_ctc_medium en 2.5 5.8 2.6 5.9
4 stt_en_conformer_ctc_large en 1.9 4.4 2.1 4.5
5 stt_en_conformer_ctc_xlarge en 1.77 % 3.79 % 2.00 % 3.74 % 7.88 % 5.99 % 6.44 % 22.90 % 5.50 % 2.36 %
6 stt_en_conformer_ctc_small_ls en 3.3 8.8 3.4 8.8
7 stt_en_conformer_ctc_medium_ls en 2.7 7.4 3.0 7.3
8 stt_en_conformer_ctc_large_ls en 2.4 6.2 2.7 6.0
9 stt_en_conformer_transducer_small en 2.8 6.6 2.5 6.6
10 stt_en_conformer_transducer_medium en 2.0 4.6 2.1 4.7
11 stt_en_conformer_transducer_large en 1.6 3.5 1.7 3.7
12 stt_en_conformer_transducer_large_ls en 2.1 5.0 2.3 5.1
13 stt_en_conformer_transducer_xlarge en 1.48 % 2.95 % 1.62 % 3.01 % 6.46 % 4.59 % 5.32 % 5.70 % 6.47 % 21.32 % 2.05 % 1.17 %
14 stt_en_conformer_transducer_xxlarge en 1.52 % 3.09 % 1.72 % 3.14 % 5.29 % 5.85 % 6.64 % 2.42 % 1.49 %
15 stt_en_fastconformer_hybrid_large_streaming_80ms (CTC) en 3.5 % 8.1 % 10.2 % 7.2 % 3.5 % 2.3 %
16 stt_en_fastconformer_hybrid_large_streaming_480ms (CTC) en 3.6 % 7.5 % 9.8 % 7.0 % 3.5 % 2.1 %
17 stt_en_fastconformer_hybrid_large_streaming_1040ms (CTC) en 2.7 % 6.4 % 9.0 % 7.0 % 3.2 % 1.9 %
18 stt_en_fastconformer_hybrid_large_streaming_80ms (RNNT) en 2.7 % 6.5 % 9.1 % 6.9 % 3.2 % 1.9 %
19 stt_en_fastconformer_hybrid_large_streaming_480ms (RNNT) en 2.7 % 6.1 % 8.5 % 6.7 % 3.1 % 1.8 %
20 stt_en_fastconformer_hybrid_large_streaming_1040ms (RNNT) en 2.3 % 5.5 % 8.0 % 6.6 % 2.9 % 1.6 %
21 stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 0ms) en 7.0 %
22 stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 80ms) en 6.4 %
23 stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 480) en 5.7 %
24 stt_en_fastconformer_hybrid_large_streaming_multi (RNNT - 1040) en 5.4 %
25 stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 0ms) en 8.4 %
26 stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 80ms) en 7.8 %
27 stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 480) en 6.7 %
28 stt_en_fastconformer_hybrid_large_streaming_multi (CTC - 1040) en 6.2 %
docs/source/asr/data/scores/en/fastconformer_en.csv
+4
View File
@@ -0,0 +1,4 @@
Model Name,Language,EuroParl Test Set (en),Fisher Test Set (en),Librispeech Dev-Clean,Librispeech Dev-Other,Librispeech Test-Clean,Librispeech Test-Other,MCV Test-Set v11.0 (en),MCV Test-Set v8.0 (en),MLS Dev (en),MLS Test (en),NSC Part1,NSC Part6,Peoples Speech Test v1,SLR 83 Test,SPGI Test,VoxPopuli Test (en),WSJ Dev 93,WSJ Eval 92
stt_en_fastconformer_ctc_large,en,,,1.9,4.2,2.1,4.2,,,,,,,,,,,,
stt_en_fastconformer_transducer_large,en,,,2.0,3.8,1.8,3.8,,,,,,,,,,,,
stt_en_fastconformer_hybrid_large_pc,en,8.0 %,10.3 %,,,2.0 %,4.1 %,8.2 %,,,4.5 %,4.6 %,,,,2.3 %,4.5 %,,
1 Model Name Language EuroParl Test Set (en) Fisher Test Set (en) Librispeech Dev-Clean Librispeech Dev-Other Librispeech Test-Clean Librispeech Test-Other MCV Test-Set v11.0 (en) MCV Test-Set v8.0 (en) MLS Dev (en) MLS Test (en) NSC Part1 NSC Part6 Peoples Speech Test v1 SLR 83 Test SPGI Test VoxPopuli Test (en) WSJ Dev 93 WSJ Eval 92
2 stt_en_fastconformer_ctc_large en 1.9 4.2 2.1 4.2
3 stt_en_fastconformer_transducer_large en 2.0 3.8 1.8 3.8
4 stt_en_fastconformer_hybrid_large_pc en 8.0 % 10.3 % 2.0 % 4.1 % 8.2 % 4.5 % 4.6 % 2.3 % 4.5 %
docs/source/asr/data/scores/enes/conformer_enes.csv
+5
View File
@@ -0,0 +1,5 @@
Model Name,Language,Fisher-Dev-En,Fisher-Dev-Es,Fisher-Test-En,Fisher-Test-Es,Librispeech Dev-Clean,Librispeech Dev-Other,Librispeech Test-Clean,Librispeech Test-Other,MCV Dev-Set v7.0 (en),MCV Dev-Set v7.0 (es),MCV Test-Set v7.0 (en),MCV Test-Set v7.0 (es),MLS Dev (en),MLS Dev (es),MLS Test (en),MLS Test (es),VoxPopuli Dev (en),VoxPopuli Dev (es),VoxPopuli Test (en),VoxPopuli Test (es)
stt_enes_conformer_ctc_large,enes,,16.7 %,,,2.2 %,5.5 %,2.6 %,5.5 %,5.8 %,,,,,3.5 %,,,,5.7 %,,
stt_enes_conformer_ctc_large_codesw,enes,,16.51 %,,16.31 %,2.22 %,5.36 %,2.55 %,5.38 %,,5.00 %,,5.51 %,,3.46 %,,3.73 %,,5.58 %,,6.63 %
stt_enes_conformer_transducer_large,enes,,16.2 %,,,2.0 %,4.6 %,2.2 %,4.6 %,5.0 %,,,,,3.3 %,,,,5.3 %,,
stt_enes_conformer_transducer_large_codesw,enes,15.70 %,,15.66 %,,1.97 %,4.54 %,2.17 %,4.53 %,4.51 %,,5.06 %,,3.27 %,,3.67 %,,5.28 %,,6.54 %,
1 Model Name Language Fisher-Dev-En Fisher-Dev-Es Fisher-Test-En Fisher-Test-Es Librispeech Dev-Clean Librispeech Dev-Other Librispeech Test-Clean Librispeech Test-Other MCV Dev-Set v7.0 (en) MCV Dev-Set v7.0 (es) MCV Test-Set v7.0 (en) MCV Test-Set v7.0 (es) MLS Dev (en) MLS Dev (es) MLS Test (en) MLS Test (es) VoxPopuli Dev (en) VoxPopuli Dev (es) VoxPopuli Test (en) VoxPopuli Test (es)
2 stt_enes_conformer_ctc_large enes 16.7 % 2.2 % 5.5 % 2.6 % 5.5 % 5.8 % 3.5 % 5.7 %
3 stt_enes_conformer_ctc_large_codesw enes 16.51 % 16.31 % 2.22 % 5.36 % 2.55 % 5.38 % 5.00 % 5.51 % 3.46 % 3.73 % 5.58 % 6.63 %
4 stt_enes_conformer_transducer_large enes 16.2 % 2.0 % 4.6 % 2.2 % 4.6 % 5.0 % 3.3 % 5.3 %
5 stt_enes_conformer_transducer_large_codesw enes 15.70 % 15.66 % 1.97 % 4.54 % 2.17 % 4.53 % 4.51 % 5.06 % 3.27 % 3.67 % 5.28 % 6.54 %
docs/source/asr/data/scores/eo/conformer_eo.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,MCV Dev-Set v11.0 (eo),MCV Test-Set v11.0 (eo)
stt_eo_conformer_ctc_large,eo,2.9 %,4.8 %
stt_eo_conformer_transducer_large,eo,2.4 %,4.0 %
1 Model Name Language MCV Dev-Set v11.0 (eo) MCV Test-Set v11.0 (eo)
2 stt_eo_conformer_ctc_large eo 2.9 % 4.8 %
3 stt_eo_conformer_transducer_large eo 2.4 % 4.0 %
docs/source/asr/data/scores/es/conformer_es.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,Call Home Dev Test (es),Call Home Eval Test (es),Call Home Train (es),Fisher Dev Set (es),Fisher Test Set (es),MCV Dev-Set (v??) (es),MCV Dev-Set v12.0 (es),MCV Dev-Set v7.0 (es),MCV Test-Set (v??) (es),MCV Test-Set v12.0 (es),MCV Test-Set v7.0 (es),MLS Dev (en),MLS Test (en),VoxPopuli Dev (es),VoxPopuli Test (es)
stt_es_conformer_ctc_large,es,23.7 %,25.3 %,22.4 %,18.3 %,18.5 %,,,6.3 %,,,6.9 %,4.3 %,4.2 %,6.1 %,7.5 %
stt_es_conformer_transducer_large,es,18.0 %,19.4 %,17.2 %,14.7 %,14.8 %,,,4.6 %,,,5.2 %,2.7 %,3.2 %,4.7 %,6.0 %
1 Model Name Language Call Home Dev Test (es) Call Home Eval Test (es) Call Home Train (es) Fisher Dev Set (es) Fisher Test Set (es) MCV Dev-Set (v??) (es) MCV Dev-Set v12.0 (es) MCV Dev-Set v7.0 (es) MCV Test-Set (v??) (es) MCV Test-Set v12.0 (es) MCV Test-Set v7.0 (es) MLS Dev (en) MLS Test (en) VoxPopuli Dev (es) VoxPopuli Test (es)
2 stt_es_conformer_ctc_large es 23.7 % 25.3 % 22.4 % 18.3 % 18.5 % 6.3 % 6.9 % 4.3 % 4.2 % 6.1 % 7.5 %
3 stt_es_conformer_transducer_large es 18.0 % 19.4 % 17.2 % 14.7 % 14.8 % 4.6 % 5.2 % 2.7 % 3.2 % 4.7 % 6.0 %
docs/source/asr/data/scores/es/fastconformer_es.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,Call Home Dev Test (es),Call Home Eval Test (es),Call Home Train (es),Fisher Dev Set (es),Fisher Test Set (es),MCV Dev-Set (v??) (es),MCV Dev-Set v12.0 (es),MCV Dev-Set v7.0 (es),MCV Test-Set (v??) (es),MCV Test-Set v12.0 (es),MCV Test-Set v7.0 (es),MLS Dev (en),MLS Test (en),VoxPopuli Dev (es),VoxPopuli Test (es)
stt_es_fastconformer_hybrid_large_pc,es,,,,29.4 %,28.9 %,,7.1 %,,,7.5 %,,10.6 %,11.8 %,8.6 %,9.8 %
1 Model Name Language Call Home Dev Test (es) Call Home Eval Test (es) Call Home Train (es) Fisher Dev Set (es) Fisher Test Set (es) MCV Dev-Set (v??) (es) MCV Dev-Set v12.0 (es) MCV Dev-Set v7.0 (es) MCV Test-Set (v??) (es) MCV Test-Set v12.0 (es) MCV Test-Set v7.0 (es) MLS Dev (en) MLS Test (en) VoxPopuli Dev (es) VoxPopuli Test (es)
2 stt_es_fastconformer_hybrid_large_pc es 29.4 % 28.9 % 7.1 % 7.5 % 10.6 % 11.8 % 8.6 % 9.8 %
docs/source/asr/data/scores/fr/conformer_fr.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,MCV Dev-Set (v??) (fr),MCV Dev-Set v7.0 (fr),MCV Dev-Set v7.0 (fr) (No Hyphen),MCV Test-Set v7.0 (fr),MCV Test-Set v7.0 (fr) (No Hyphen),MLS Dev (en),MLS Dev (en) (No Hyphen),MLS Test (en),MLS Test (en) (No Hyphen)
stt_fr_conformer_ctc_large,fr,,8.35,7.88,9.63,9.01,5.88,5.90,4.91,4.63
stt_fr_conformer_transducer_large,fr,,6.85,,7.95,,5.05,,4.10,
1 Model Name Language MCV Dev-Set (v??) (fr) MCV Dev-Set v7.0 (fr) MCV Dev-Set v7.0 (fr) (No Hyphen) MCV Test-Set v7.0 (fr) MCV Test-Set v7.0 (fr) (No Hyphen) MLS Dev (en) MLS Dev (en) (No Hyphen) MLS Test (en) MLS Test (en) (No Hyphen)
2 stt_fr_conformer_ctc_large fr 8.35 7.88 9.63 9.01 5.88 5.90 4.91 4.63
3 stt_fr_conformer_transducer_large fr 6.85 7.95 5.05 4.10
docs/source/asr/data/scores/hr/conformer_hr.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,ParlaSpeech Dev-Set v1.0 (hr),ParlaSpeech Test-Set v1.0 (hr),Parlaspeech Dev-Set (v??) (hr),Parlaspeech Test-Set (v??) (hr)
stt_hr_conformer_ctc_large,hr,4.43,4.70,,
stt_hr_conformer_transducer_large,hr,4.56,4.69,,
1 Model Name Language ParlaSpeech Dev-Set v1.0 (hr) ParlaSpeech Test-Set v1.0 (hr) Parlaspeech Dev-Set (v??) (hr) Parlaspeech Test-Set (v??) (hr)
2 stt_hr_conformer_ctc_large hr 4.43 4.70
3 stt_hr_conformer_transducer_large hr 4.56 4.69
docs/source/asr/data/scores/hr/fastconformer_hr.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,ParlaSpeech Dev-Set v1.0 (hr),ParlaSpeech Test-Set v1.0 (hr),Parlaspeech Dev-Set (v??) (hr),Parlaspeech Test-Set (v??) (hr)
stt_hr_fastconformer_hybrid_large_pc,hr,,,4.5 %,4.2 %
1 Model Name Language ParlaSpeech Dev-Set v1.0 (hr) ParlaSpeech Test-Set v1.0 (hr) Parlaspeech Dev-Set (v??) (hr) Parlaspeech Test-Set (v??) (hr)
2 stt_hr_fastconformer_hybrid_large_pc hr 4.5 % 4.2 %
docs/source/asr/data/scores/it/conformer_it.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,MCV Dev-Set (v??) (it),MCV Dev-Set v11.0 (it),MCV Dev-Set v12.0 (it),MCV Test-Set v11.0 (it),MCV Test-Set v12.0 (it),MLS Dev (en),MLS Test (en),VoxPopuli Dev (it),VoxPopuli Test (it)
stt_it_conformer_ctc_large,it,,5.38,,5.92,,13.16,10.62,13.43,16.75
stt_it_conformer_transducer_large,it,,4.80,,5.24,,14.62,12.18,12.00,15.15
1 Model Name Language MCV Dev-Set (v??) (it) MCV Dev-Set v11.0 (it) MCV Dev-Set v12.0 (it) MCV Test-Set v11.0 (it) MCV Test-Set v12.0 (it) MLS Dev (en) MLS Test (en) VoxPopuli Dev (it) VoxPopuli Test (it)
2 stt_it_conformer_ctc_large it 5.38 5.92 13.16 10.62 13.43 16.75
3 stt_it_conformer_transducer_large it 4.80 5.24 14.62 12.18 12.00 15.15
docs/source/asr/data/scores/it/fastconformer_it.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set (v??) (it),MCV Dev-Set v11.0 (it),MCV Dev-Set v12.0 (it),MCV Test-Set v11.0 (it),MCV Test-Set v12.0 (it),MLS Dev (en),MLS Test (en),VoxPopuli Dev (it),VoxPopuli Test (it)
stt_it_fastconformer_hybrid_large_pc,it,,,5.2 %,,5.8 %,13.6 %,11.5 %,12.7 %,15.6 %
1 Model Name Language MCV Dev-Set (v??) (it) MCV Dev-Set v11.0 (it) MCV Dev-Set v12.0 (it) MCV Test-Set v11.0 (it) MCV Test-Set v12.0 (it) MLS Dev (en) MLS Test (en) VoxPopuli Dev (it) VoxPopuli Test (it)
2 stt_it_fastconformer_hybrid_large_pc it 5.2 % 5.8 % 13.6 % 11.5 % 12.7 % 15.6 %
docs/source/asr/data/scores/kab/conformer_kab.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Test-Set v10.0 (kab)
stt_kab_conformer_transducer_large,kab,18.86
1 Model Name Language MCV Test-Set v10.0 (kab)
2 stt_kab_conformer_transducer_large kab 18.86
docs/source/asr/data/scores/nl/fastconformer_nl.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Test-Set v12.0 (nl),MLS Test (nl)
stt_nl_fastconformer_hybrid_large_pc,nl,9.2 %,12.1 %
1 Model Name Language MCV Test-Set v12.0 (nl) MLS Test (nl)
2 stt_nl_fastconformer_hybrid_large_pc nl 9.2 % 12.1 %
docs/source/asr/data/scores/pl/fastconformer_pl.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set (v??) (pl),MCV Dev-Set v12.0 (pl),MCV Test-Set v12.0 (pl),MLS Dev (en),MLS Test (en),VoxPopuli Dev (pl),VoxPopuli Test (pl)
stt_pl_fastconformer_hybrid_large_pc,pl,,6.0 %,8.7 %,7.1 %,5.8 %,11.3 %,8.5 %
1 Model Name Language MCV Dev-Set (v??) (pl) MCV Dev-Set v12.0 (pl) MCV Test-Set v12.0 (pl) MLS Dev (en) MLS Test (en) VoxPopuli Dev (pl) VoxPopuli Test (pl)
2 stt_pl_fastconformer_hybrid_large_pc pl 6.0 % 8.7 % 7.1 % 5.8 % 11.3 % 8.5 %
docs/source/asr/data/scores/ru/conformer_ru.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,GOLOS Crowd Test-Set (v??) (ru),GOLOS Farfield Test-Set (v??) (ru),Librispeech Test,MCV Dev-Set (v??) (ru),MCV Dev-Set v10.0 (ru),MCV Test-Set v10.0 (ru)
stt_ru_conformer_ctc_large,ru,2.8 %,7.1 %,13.5 %,,3.9 %,4.3 %
stt_ru_conformer_transducer_large,ru,2.7%,7.6%,12.0%,,3.5%,4.0%
1 Model Name Language GOLOS Crowd Test-Set (v??) (ru) GOLOS Farfield Test-Set (v??) (ru) Librispeech Test MCV Dev-Set (v??) (ru) MCV Dev-Set v10.0 (ru) MCV Test-Set v10.0 (ru)
2 stt_ru_conformer_ctc_large ru 2.8 % 7.1 % 13.5 % 3.9 % 4.3 %
3 stt_ru_conformer_transducer_large ru 2.7% 7.6% 12.0% 3.5% 4.0%
docs/source/asr/data/scores/rw/conformer_rw.csv
+3
View File
@@ -0,0 +1,3 @@
Model Name,Language,MCV Test-Set v9.0 (rw)
stt_rw_conformer_ctc_large,rw,18.2 %
stt_rw_conformer_transducer_large,rw,16.2 %
1 Model Name Language MCV Test-Set v9.0 (rw)
2 stt_rw_conformer_ctc_large rw 18.2 %
3 stt_rw_conformer_transducer_large rw 16.2 %
docs/source/asr/data/scores/ua/fastconformer_ua.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Test-Set v12.0 (ua)
stt_ua_fastconformer_hybrid_large_pc,ua,5.2 %
1 Model Name Language MCV Test-Set v12.0 (ua)
2 stt_ua_fastconformer_hybrid_large_pc ua 5.2 %
docs/source/asr/data/scores/zh/conformer_zh.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,AIShell Dev-Android v2,AIShell Dev-Ios v1,AIShell Dev-Ios v2,AIShell Dev-Mic v2,AIShell Test-Android v2,AIShell Test-Ios v1,AIShell Test-Ios v2,AIShell Test-Mic v2
stt_zh_conformer_transducer_large,zh,3.4,,3.2,3.4,3.4,,3.2,3.4
1 Model Name Language AIShell Dev-Android v2 AIShell Dev-Ios v1 AIShell Dev-Ios v2 AIShell Dev-Mic v2 AIShell Test-Android v2 AIShell Test-Ios v1 AIShell Test-Ios v2 AIShell Test-Mic v2
2 stt_zh_conformer_transducer_large zh 3.4 3.2 3.4 3.4 3.2 3.4
docs/source/asr/data/scores_pc/by/fastconformer_by.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set v12.0 (be),MCV Test-Set v12.0 (be)
stt_by_fastconformer_hybrid_large_pc,by,3.8 %,3.9 %
1 Model Name Language MCV Dev-Set v12.0 (be) MCV Test-Set v12.0 (be)
2 stt_by_fastconformer_hybrid_large_pc by 3.8 % 3.9 %
docs/source/asr/data/scores_pc/de/fastconformer_de.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set v12.0 (de),MCV Test-Set v12.0 (de),MLS Dev (en),MLS Test (en),VoxPopuli Dev (de),VoxPopuli Test (de)
stt_de_fastconformer_hybrid_large_pc,de,4.7 %,5.4 %,10.1 %,11.1 %,12.6 %,10.4 %
1 Model Name Language MCV Dev-Set v12.0 (de) MCV Test-Set v12.0 (de) MLS Dev (en) MLS Test (en) VoxPopuli Dev (de) VoxPopuli Test (de)
2 stt_de_fastconformer_hybrid_large_pc de 4.7 % 5.4 % 10.1 % 11.1 % 12.6 % 10.4 %
docs/source/asr/data/scores_pc/en/fastconformer_en.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,EuroParl Test Set (en),Fisher Test Set (en),Librispeech Test-Clean,Librispeech Test-Other,MCV Test-Set v11.0 (en),MLS Test (en),NSC Part1,SPGI Test,VoxPopuli Test (en)
stt_en_fastconformer_hybrid_large_pc,en,12.5 %,19.0 %,7.3 %,9.2 %,10.1 %,12.7 %,7.2 %,5.1 %,6.7 %
1 Model Name Language EuroParl Test Set (en) Fisher Test Set (en) Librispeech Test-Clean Librispeech Test-Other MCV Test-Set v11.0 (en) MLS Test (en) NSC Part1 SPGI Test VoxPopuli Test (en)
2 stt_en_fastconformer_hybrid_large_pc en 12.5 % 19.0 % 7.3 % 9.2 % 10.1 % 12.7 % 7.2 % 5.1 % 6.7 %
docs/source/asr/data/scores_pc/es/fastconformer_es.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,Fisher Dev Set (es),Fisher Test Set (es),MCV Dev-Set v12.0 (es),MCV Test-Set v12.0 (es),MLS Dev (en),MLS Test (en),VoxPopuli Dev (es),VoxPopuli Test (es)
stt_es_fastconformer_hybrid_large_pc,es,14.7 %,14.6 %,4.5 %,5.0 %,3.1 %,3.9 %,4.4 %,5.6 %
1 Model Name Language Fisher Dev Set (es) Fisher Test Set (es) MCV Dev-Set v12.0 (es) MCV Test-Set v12.0 (es) MLS Dev (en) MLS Test (en) VoxPopuli Dev (es) VoxPopuli Test (es)
2 stt_es_fastconformer_hybrid_large_pc es 14.7 % 14.6 % 4.5 % 5.0 % 3.1 % 3.9 % 4.4 % 5.6 %
docs/source/asr/data/scores_pc/hr/fastconformer_hr.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,Parlaspeech Dev-Set (v??) (hr),Parlaspeech Test-Set (v??) (hr)
stt_hr_fastconformer_hybrid_large_pc,hr,10.4 %,8.7 %
1 Model Name Language Parlaspeech Dev-Set (v??) (hr) Parlaspeech Test-Set (v??) (hr)
2 stt_hr_fastconformer_hybrid_large_pc hr 10.4 % 8.7 %
docs/source/asr/data/scores_pc/it/fastconformer_it.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set v12.0 (it),MCV Test-Set v12.0 (it),MLS Dev (en),MLS Test (en),VoxPopuli Dev (it),VoxPopuli Test (it)
stt_it_fastconformer_hybrid_large_pc,it,7.8 %,8.2 %,26.4 %,22.5 %,16.8 %,19.6 %
1 Model Name Language MCV Dev-Set v12.0 (it) MCV Test-Set v12.0 (it) MLS Dev (en) MLS Test (en) VoxPopuli Dev (it) VoxPopuli Test (it)
2 stt_it_fastconformer_hybrid_large_pc it 7.8 % 8.2 % 26.4 % 22.5 % 16.8 % 19.6 %
docs/source/asr/data/scores_pc/nl/fastconformer_nl.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Test-Set v12.0 (nl),MLS Test (nl)
stt_nl_fastconformer_hybrid_large_pc,nl,32.1 %,25.1 %
1 Model Name Language MCV Test-Set v12.0 (nl) MLS Test (nl)
2 stt_nl_fastconformer_hybrid_large_pc nl 32.1 % 25.1 %
docs/source/asr/data/scores_pc/pl/fastconformer_pl.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Dev-Set v12.0 (pl),MCV Test-Set v12.0 (pl),MLS Dev (en),MLS Test (en),VoxPopuli Dev (pl),VoxPopuli Test (pl)
stt_pl_fastconformer_hybrid_large_pc,pl,8.9 %,11.0 %,16.0 %,11.0 %,14.0 %,11.4 %
1 Model Name Language MCV Dev-Set v12.0 (pl) MCV Test-Set v12.0 (pl) MLS Dev (en) MLS Test (en) VoxPopuli Dev (pl) VoxPopuli Test (pl)
2 stt_pl_fastconformer_hybrid_large_pc pl 8.9 % 11.0 % 16.0 % 11.0 % 14.0 % 11.4 %
docs/source/asr/data/scores_pc/ua/fastconformer_ua.csv
+2
View File
@@ -0,0 +1,2 @@
Model Name,Language,MCV Test-Set v12.0 (ua)
stt_ua_fastconformer_hybrid_large_pc,ua,7.3 %
1 Model Name Language MCV Test-Set v12.0 (ua)
2 stt_ua_fastconformer_hybrid_large_pc ua 7.3 %
docs/source/asr/datasets.rst
+206
View File
@@ -0,0 +1,206 @@
Datasets
========
NeMo ASR models expect data as a set of audio files plus a manifest file describing each utterance.
.. _section-with-manifest-format-explanation:
Manifest Format
---------------
Each line of the manifest is a JSON object:
.. code-block:: json
{"audio_filepath": "/path/to/audio.wav", "text": "the transcription of the utterance", "duration": 23.147}
* ``audio_filepath`` — absolute or relative path to the audio file (WAV recommended)
* ``text`` — the transcript
* ``duration`` — duration in seconds
There should be one manifest per dataset split (train, validation, test). Pass it via ``training_ds.manifest_filepath=<path>``.
.. _canary-manifest-format:
Canary Manifest Format
~~~~~~~~~~~~~~~~~~~~~~
Canary multi-task models require additional manifest keys to control transcription, translation, punctuation, and other behaviors.
The required and optional keys differ between Canary v1 and Canary Flash / v2.
**Canary v1** (e.g., ``canary-1b``):
.. code-block:: json
{"audio_filepath": "audio.wav", "text": "hello world", "duration": 3.5, "source_lang": "en", "task": "asr", "target_lang": "en", "pnc": "yes"}
.. list-table::
:header-rows: 1
* - Key
- Required
- Description
* - ``source_lang``
- Yes
- Input audio language (ISO code, e.g. ``en``, ``de``, ``es``)
* - ``target_lang``
- Yes
- Output transcription language
* - ``task``
- Yes
- ``"asr"`` (transcribe) or ``"ast"`` (translate)
* - ``pnc``
- Yes
- ``"yes"`` or ``"no"`` — enable punctuation and capitalization
**Canary Flash / v2** (e.g., ``canary-1b-flash``, ``canary-1b-v2``):
The ``task`` field has been removed; the model infers ASR vs translation from the language pair.
Additional optional keys control features like timestamps, ITN, and diarization.
.. code-block:: json
{"audio_filepath": "audio.wav", "text": "hello world", "duration": 3.5, "source_lang": "en", "target_lang": "en", "pnc": "yes"}
.. list-table::
:header-rows: 1
* - Key
- Required
- Description
* - ``source_lang``
- Yes
- Input audio language (ISO code)
* - ``target_lang``
- Yes
- Output transcription language. Same as ``source_lang`` for ASR; different for translation.
* - ``pnc``
- No (default: ``"yes"``)
- ``"yes"`` or ``"no"`` — punctuation and capitalization
* - ``itn``
- No (default: ``"no"``)
- ``"yes"`` or ``"no"`` — inverse text normalization
* - ``timestamp``
- No (default: ``"no"``)
- ``"yes"`` or ``"no"`` — predict word-level timestamps
* - ``diarize``
- No (default: ``"no"``)
- ``"yes"`` or ``"no"`` — diarize speech
* - ``decodercontext``
- No (default: ``""``)
- Previous transcript or other context to bias predictions
* - ``emotion``
- No (default: ``"undefined"``)
- Speaker emotion hint (``"neutral"``, ``"angry"``, ``"happy"``, ``"sad"``, ``"undefined"``)
During fine-tuning, these keys are read from the manifest and encoded as prompt tokens.
During inference, they can be provided either in the manifest or as arguments to ``model.transcribe()``.
.. _Tarred_Datasets:
Tarred Datasets
---------------
For cluster training with distributed file systems, tar your audio files to avoid reading many small files.
Use ``is_tarred: true`` in the config and provide tarball paths via ``tarred_audio_filepaths``.
NeMo uses `WebDataset <https://github.com/tmbdev/webdataset>`_ for tarred data.
**Convert to tarred format:**
.. code-block:: bash
python scripts/speech_recognition/convert_to_tarred_audio_dataset.py \
--manifest_path=<manifest> \
--target_dir=<output_dir> \
--num_shards=64 \
--max_duration=<float representing maximum duration of audio samples> \
--min_duration=<float representing minimum duration of audio samples> \
--shuffle --shuffle_seed=0
This script shuffles the entries in the given manifest (if ``--shuffle`` is set, which we recommend), filter
audio files according to ``min_duration`` and ``max_duration``, and tar the remaining audio files to the directory
``--target_dir`` in ``n`` shards, along with separate manifest and metadata files.
The files in the target directory should look similar to the following:
.. code::
target_dir/
├── audio_1.tar
├── audio_2.tar
├── ...
├── metadata.yaml
├── tarred_audio_manifest.json
├── sharded_manifests/
├── manifest_1.json
├── ...
└── manifest_N.json
Note that file structures are flattened such that all audio files are at the top level in each tarball. This ensures that
filenames are unique in the tarred dataset and the filepaths do not contain "-sub" and forward slashes in each ``audio_filepath`` are
simply converted to underscores. For example, a manifest entry for ``/data/directory1/file.wav`` would be ``_data_directory1_file.wav``
in the tarred dataset manifest, and ``/data/directory2/file.wav`` would be converted to ``_data_directory2_file.wav``.
Sharded manifests are generated by default; this behavior can be toggled via the ``no_shard_manifests`` flag.
To use an existing tarred dataset instead of a non-tarred dataset, set ``is_tarred: true`` in
the experiment config file. Then, pass in the paths to all of the audio tarballs in ``tarred_audio_filepaths``, either as a list
of filepaths, e.g. ``['/data/shard1.tar', '/data/shard2.tar']``, or in a single brace-expandable string, e.g.
``'/data/shard_{1..64}.tar'`` or ``'/data/shard__OP_1..64_CL_'`` (recommended, see note below).
.. note::
For brace expansion, there may be cases where ``{x..y}`` syntax cannot be used due to shell interference. This occurs most commonly
inside SLURM scripts. Therefore, we provide a few equivalent replacements. Supported opening braces (equivalent to ``{``) are ``(``,
``[``, ``<`` and the special tag ``_OP_``. Supported closing braces (equivalent to ``}``) are ``)``, ``]``, ``>`` and the special
tag ``_CL_``. For SLURM based tasks, we suggest the use of the special tags for ease of use.
As with non-tarred datasets, the manifest file should be passed in ``manifest_filepath``. The dataloader assumes that the length
of the manifest after filtering is the correct size of the dataset for reporting training progress.
The ``tarred_shard_strategy`` field of the config file can be set if you have multiple shards and are running an experiment with
multiple workers. It defaults to ``scatter``, which preallocates a set of shards per worker which do not change during runtime.
Note that this strategy, on specific occasions (when the number of shards is not divisible with ``world_size``), will not sample
the entire dataset. As an alternative the ``replicate`` strategy, will preallocate the entire set of shards to every worker and not
change it during runtime. The benefit of this strategy is that it allows each worker to sample data points from the entire dataset
independently of others. Note, though, that more than one worker may sample the same shard, and even sample the same data points!
As such, there is no assured guarantee that all samples in the dataset will be sampled at least once during 1 epoch. Note that
for these reasons it is not advisable to use tarred datasets as validation and test datasets.
For more information about the individual tarred datasets and the parameters available, including shuffling options,
see the corresponding class APIs in the :ref:`Datasets <asr-api-datasets>` section.
.. warning::
If using multiple workers, the number of shards should be divisible by the world size to ensure an even
split among workers. If it is not divisible, logging will give a warning but training will proceed, but likely hang at the last epoch.
In addition, if using distributed processing, each shard must have the same number of entries after filtering is
applied such that each worker ends up with the same number of files. We currently do not check for this in any dataloader, but the user's
program may hang if the shards are uneven.
.. _Bucketing_Datasets:
Bucketing
---------
The script ``scripts/speech_recognition/convert_to_tarred_audio_dataset.py`` offers a ``--buckets_num`` option that enables
static bucketing by sorting data into separate duration-based buckets at pre-processing time.
This approach is deprecated in favor of :ref:`dynamic bucketing <lhotse-dataloading>` enabled with Lhotse, which doesn't require special pre-processing.
If you do wish to proceed with static bucketing, pass the tarred datasets as a list of lists in your training config:
.. code-block:: yaml
train_ds:
manifest_filepath: [[bucket1/manifest.json], [bucket2/manifest.json], ...]
tarred_audio_filepaths: [[bucket1/audio__OP_0..63_CL_.tar], [bucket2/audio__OP_0..63_CL_.tar], ...]
bucketing_batch_size: null # set to a list of ints for adaptive batch sizes per bucket
Lhotse Dataloading
------------------
NeMo supports `Lhotse <https://github.com/lhotse-speech/lhotse>`_ for advanced dataloading with dynamic batch sizes, dynamic bucketing, OOMptimizer, and multi-dataset configuration.
See :doc:`Lhotse Dataloading </dataloaders>` for full documentation.
docs/source/asr/examples/kinyarwanda_asr.rst
+636
View File
@@ -0,0 +1,636 @@
:orphan:
Example With MCV
================
########################################################################
Kinyarwanda ASR using Mozilla Common Voice Dataset
########################################################################
In this example, we describe essential steps of training an ASR model for a new language (Kinyarwanda). Namely,
* Data preprocessing
* Building tokenizers
* Tarred datasets and bucketing
* Training from scratch and finetuning
* Inference and evaluation
**************************
Kinyarwanda Speech Dataset
**************************
We use `Mozilla Common Voice <https://commonvoice.mozilla.org/rw>`_ dataset for Kinyarwanda which is a large dataset with 2000+ hours of audio data.
**Note**: You should download this dataset by yourself.
Mozilla distributes the dataset in tsv+mp3 format.
After downloading and unpacking, the dataset has the following structure
.. code-block:: bash
├── cv-corpus-9.0-2022-04-27
│ └── rw
│ ├── clips [here are all audio files, e.g. common_voice_rw_26260276.mp3]
│ ├── dev.tsv
│ ├── invalidated.tsv
│ ├── other.tsv
│ ├── reported.tsv
│ ├── test.tsv
│ ├── train.tsv
│ └── validated.tsv
Mozilla provides **train/dev/test** split of the data, so we can just use it.
Let's look at the format of a .tsv file
.. code-block:: bash
head train.tsv
.. code-block:: bash
client_id path sentence up_votes down_votes age gender accents locale segment
e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a common_voice_rw_26273273.mp3 kandi tuguwe neza kugira ngo twakire amagambo y’ukuri, 2 0 twenties male rw
e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a common_voice_rw_26273478.mp3 Simbi na we akajya kwiga nubwo byari bigoye 2 0 twenties male rw
e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a common_voice_rw_26273483.mp3 Inshuti yanjye yaje kunsura ku biro byanjye. 2 0 twenties male rw
e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a common_voice_rw_26273488.mp3 Grand Canyon ni ahantu hazwi cyane ba mukerarugendo. 2 0 twenties male rw
Each line corresponds to one record (usually one sentence) and contains:
* name of the audio file
* corresponding transcription
* meta information: client_id, age, gender, etc.
Resampling and creating manifests
#################################
To be able to use a dataset with NeMo Toolkit, we first need to
* Convert *.tsv* files to *.json* manifests
* Convert *.mp3* files to *.wav* with sample rate of 16000
To convert a .tsv file to .json manifest, we used the following script
.. code-block:: bash
python tsv_to_json.py \
--tsv=cv-corpus-9.0-2022-04-27/rw/train.tsv \
--folder=cv-corpus-9.0-2022-04-27/rw/clips \
--sampling_count=-1
**tsv_to_json.py**:
.. code-block:: python
import pandas as pd
import json
import tqdm
import argparse
parser = argparse.ArgumentParser("MCV TSV-to-JSON converter")
parser.add_argument("--tsv", required=True, type=str, help="Input TSV file")
parser.add_argument("--sampling_count", required=True, type=int, help="Number of examples, you want, use -1 for all examples")
parser.add_argument("--folder", required=True, type=str, help="Relative path to folder with audio files")
args = parser.parse_args()
df = pd.read_csv(args.tsv, sep='\t')
with open(args.tsv.replace('.tsv', '.json'), 'w') as fo:
mod = 1
if args.sampling_count > 0:
mod = len(df) // args.sampling_count
for idx in tqdm.tqdm(range(len(df))):
if idx % mod != 0:
continue
item = {
'audio_filepath': args.folder + "/" + df['path'][idx],
'text': df['sentence'][idx],
'up_votes': int(df['up_votes'][idx]), 'down_votes': int(df['down_votes'][idx]),
'age': df['age'][idx], 'gender': df['gender'][idx], 'accents': df['accents'][idx],
'client_id': df['client_id'][idx]
}
fo.write(json.dumps(item) + "\n")
This script will create a corresponding **train.json** manifest near the initial **train.tsv**. It will look like this:
.. code-block:: bash
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273273.mp3", "text": "kandi tuguwe neza kugira ngo twakire amagambo y\u2019ukuri,", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273478.mp3", "text": "Simbi na we akajya kwiga nubwo byari bigoye", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273483.mp3", "text": "Inshuti yanjye yaje kunsura ku biro byanjye.", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
{"audio_filepath": "cv-corpus-9.0-2022-04-27/rw/clips/common_voice_rw_26273488.mp3", "text": "Grand Canyon ni ahantu hazwi cyane ba mukerarugendo.", "up_votes": 2, "down_votes": 0, "age": "twenties", "gender": "male", "accents": NaN, "client_id": "e2a04c0ecacf81302f4270a3dddaa7a131420f6b7319208473af17d4adf3724ad9a3b6cdee107e2f321495db86f114a50c396e0928464a58dfad472130e7514a"}
For resampling we used the following script:
.. code-block:: bash
mkdir train
python ../decode_resample.py \
--manifest=cv-corpus-9.0-2022-04-27/rw/train.json \
--destination_folder=./train
**decode_resample.py**:
.. code-block:: python
import argparse
import os
import json
import sox
from sox import Transformer
import tqdm
import multiprocessing
from tqdm.contrib.concurrent import process_map
parser = argparse.ArgumentParser()
parser.add_argument('--manifest', required=True, type=str, help='path to the original manifest')
parser.add_argument("--num_workers", default=multiprocessing.cpu_count(), type=int, help="Workers to process dataset.")
parser.add_argument("--destination_folder", required=True, type=str, help="Destination folder where audio files will be stored")
args = parser.parse_args()
def process(x):
if not isinstance(x['text'], str):
x['text'] = ''
else:
x['text'] = x['text'].lower().strip()
_, file_with_ext = os.path.split(x['audio_filepath'])
name, ext = os.path.splitext(file_with_ext)
output_wav_path = args.destination_folder + "/" + name + '.wav'
if not os.path.exists(output_wav_path):
tfm = Transformer()
tfm.rate(samplerate=16000)
tfm.channels(n_channels=1)
tfm.build(input_filepath=x['audio_filepath'],
output_filepath=output_wav_path)
x['duration'] = sox.file_info.duration(output_wav_path)
x['audio_filepath'] = output_wav_path
return x
def load_data(manifest):
data = []
with open(manifest, 'r') as f:
for line in tqdm.tqdm(f):
item = json.loads(line)
data.append(item)
return data
data = load_data(args.manifest)
data_new = process_map(process, data, max_workers=args.num_workers, chunksize=100)
with open(args.manifest.replace('.json', '_decoded.json'), 'w') as f:
for item in tqdm.tqdm(data_new):
f.write(json.dumps(item) + '\n')
It will write the resampled .wav-files to the specified directory and save a new json manifest with corrected audiopaths.
**Note:** You need to repeat these steps for **test.tsv** and **dev.tsv** as well.
******************
Data Preprocessing
******************
Before we start training the model on the above manifest files, we need to preprocess the text data. Data pre-processing is done to reduce ambiguity in transcripts. This is an essential step, and often requires moderate expertise in the language.
We used the following script
**prepare_dataset_kinyarwanda.py**:
.. code-block:: python
import json
import os
import re
from collections import defaultdict
from nemo.collections.asr.parts.utils.manifest_utils import read_manifest, write_manifest
from tqdm.auto import tqdm
def write_processed_manifest(data, original_path):
original_manifest_name = os.path.basename(original_path)
new_manifest_name = original_manifest_name.replace(".json", "_processed.json")
manifest_dir = os.path.split(original_path)[0]
filepath = os.path.join(manifest_dir, new_manifest_name)
write_manifest(filepath, data)
print(f"Finished writing manifest: {filepath}")
return filepath
# calculate the character set
def get_charset(manifest_data):
charset = defaultdict(int)
for row in tqdm(manifest_data, desc="Computing character set"):
text = row['text']
for character in text:
charset[character] += 1
return charset
# Preprocessing steps
def remove_special_characters(data):
chars_to_ignore_regex = "[\.\,\?\:\-!;()«»…\]\[/\*–‽+&_\\½√>€™$•¼}{~—=“\"”″‟„]"
apostrophes_regex = "[’'‘`ʽ']"
data["text"] = re.sub(chars_to_ignore_regex, " ", data["text"]) # replace punctuation by space
data["text"] = re.sub(apostrophes_regex, "'", data["text"]) # replace different apostrophes by one
data["text"] = re.sub(r"'+", "'", data["text"]) # merge multiple apostrophes
# remove spaces where apostrophe marks a deleted vowel
# this rule is taken from https://huggingface.co/lucio/wav2vec2-large-xlsr-kinyarwanda-apostrophied
data["text"] = re.sub(r"([b-df-hj-np-tv-z])' ([aeiou])", r"\1'\2", data["text"])
data["text"] = re.sub(r" '", " ", data["text"]) # delete apostrophes at the beginning of word
data["text"] = re.sub(r"' ", " ", data["text"]) # delete apostrophes at the end of word
data["text"] = re.sub(r" +", " ", data["text"]) # merge multiple spaces
return data
def replace_diacritics(data):
data["text"] = re.sub(r"[éèëēê]", "e", data["text"])
data["text"] = re.sub(r"[ãâāá]", "a", data["text"])
data["text"] = re.sub(r"[úūü]", "u", data["text"])
data["text"] = re.sub(r"[ôōó]", "o", data["text"])
data["text"] = re.sub(r"[ćç]", "c", data["text"])
data["text"] = re.sub(r"[ïī]", "i", data["text"])
data["text"] = re.sub(r"[ñ]", "n", data["text"])
return data
def remove_oov_characters(data):
oov_regex = "[^ 'aiuenrbomkygwthszdcjfvplxq]"
data["text"] = re.sub(oov_regex, "", data["text"]) # delete oov characters
data["text"] = data["text"].strip()
return data
# Processing pipeline
def apply_preprocessors(manifest, preprocessors):
for processor in preprocessors:
for idx in tqdm(range(len(manifest)), desc=f"Applying {processor.__name__}"):
manifest[idx] = processor(manifest[idx])
print("Finished processing manifest !")
return manifest
# List of pre-processing functions
PREPROCESSORS = [
remove_special_characters,
replace_diacritics,
remove_oov_characters,
]
train_manifest = "train_decoded.json"
dev_manifest = "dev_decoded.json"
test_manifest = "test_decoded.json"
train_data = read_manifest(train_manifest)
dev_data = read_manifest(dev_manifest)
test_data = read_manifest(test_manifest)
# Apply preprocessing
train_data_processed = apply_preprocessors(train_data, PREPROCESSORS)
dev_data_processed = apply_preprocessors(dev_data, PREPROCESSORS)
test_data_processed = apply_preprocessors(test_data, PREPROCESSORS)
# Write new manifests
train_manifest_cleaned = write_processed_manifest(train_data_processed, train_manifest)
dev_manifest_cleaned = write_processed_manifest(dev_data_processed, dev_manifest)
test_manifest_cleaned = write_processed_manifest(test_data_processed, test_manifest)
It performs the following operations:
* Remove all punctuation except for apostrophes
* Replace different kinds of apostrophes by one
* Lowercase
* Replace rare characters with diacritics (e.g. [éèëēê] => e)
* Delete all remaining out-of-vocabulary (OOV) characters
The final Kinyarwanda alphabet in all trancripts consists of Latin letters, space and apostrophe.
*******************
Building Tokenizers
*******************
Though it is possible to train character-based ASR model, usually we get some improvement in quality and speed if we predict longer units. The commonly used tokenization algorithm is called `Byte-pair encoding <https://en.wikipedia.org/wiki/Byte_pair_encoding>`_. This is a deterministic tokenization algorithm based on corpus statistics. It splits the words to subtokens and the beginning of word is marked by special symbol so it's easy to restore the original words.
NeMo toolkit supports on-the-fly subword tokenization, so you need not modify the transcripts, but need to pass your tokenizer via the model config. NeMo supports both Word Piece Tokenizer (via HuggingFace) and Sentence Piece Tokenizer (via Google SentencePiece library)
For Kinyarwanda experiments we used 128 subtokens for the CTC model and 1024 subtokens for the Transducer model. The tokenizers for these models were built using the text transcripts of the train set with this script. For vocabulary of size 1024 we restrict maximum subtoken length to 4 symbols (2 symbols for size 128) to avoid populating vocabulary with specific frequent words from the dataset. This does not affect the model performance and potentially helps to adapt to other domain without retraining tokenizer.
We used the following script from NeMo toolkit to create `Sentencepiece <https://github.com/google/sentencepiece>`_ tokenizers with different vocabulary sizes (128 and 1024 subtokens)
.. code-block:: bash
python ${NEMO_ROOT}/scripts/tokenizers/process_asr_text_tokenizer.py \
--manifest=dev_decoded_processed.json,train_decoded_processed.json \
--vocab_size=1024 \
--data_root=tokenizer_bpe_maxlen_4 \
--tokenizer="spe" \
--spe_type=bpe \
--spe_character_coverage=1.0 \
--spe_max_sentencepiece_length=4 \
--log
python ${NEMO_ROOT}/scripts/tokenizers/process_asr_text_tokenizer.py \
--manifest=dev_decoded_processed.json,train_decoded_processed.json \
--vocab_size=128 \
--data_root=tokenizer_bpe_maxlen_2 \
--tokenizer="spe" \
--spe_type=bpe \
--spe_character_coverage=1.0 \
--spe_max_sentencepiece_length=2 \
--log
Most of the arguments are similar to those explained in the `ASR with Subword Tokenization tutorial <https://github.com/NVIDIA/NeMo/tree/stable/tutorials/asr/ASR_with_Subword_Tokenization.ipynb>`_.
The resulting tokenizer is a folder like that:
.. code-block:: bash
├── tokenizer_spe_bpe_v1024_max_4
│ ├── tokenizer.model
│ ├── tokenizer.vocab
│ └── vocab.txt
Remember that you will need to pass the path to tokenizer in the model config.
You can see all the subtokens in the **vocab.txt** file.
*****************************
Tarred datasets and bucketing
*****************************
There are two useful techniques for training on large datasets.
* Tarred dataset allows to store the dataset as large .tar files instead of small separate audio files. It speeds up the training and minimizes the load on the network in the cluster.
* Bucketing groups utterances with similar duration. It reduces padding and speeds up the training.
The NeMo toolkit provides a script to implement both of these techniques.
.. code-block:: bash
## create tarred dataset with 1 bucket
python ${NEMO_ROOT}/scripts/speech_recognition/convert_to_tarred_audio_dataset.py \
--manifest_path=train_decoded_processed.json \
--target_dir=train_tarred_1bk \
--num_shards=1024 \
--max_duration=11.0 \
--min_duration=1.0 \
--shuffle \
--shuffle_seed=1 \
--sort_in_shards \
--workers=-1
## create tarred dataset with 4 buckets
python ${NEMO_ROOT}/scripts/speech_recognition/convert_to_tarred_audio_dataset.py \
--manifest_path=train_decoded_processed.json \
--target_dir=train_tarred_4bk \
--num_shards=1024 \
--max_duration=11.0 \
--min_duration=1.0 \
--shuffle \
--shuffle_seed=1 \
--sort_in_shards \
--workers=-1 \
--buckets_num=4
**Note**: we only need to process train data, dev and test are usually much smaller and can be used as is.
Our final dataset folder looks like this:
.. code-block:: bash
├── dev [15988 .wav files]
├── dev_decoded_processed.json (dev manifest)
├── test [16213 .wav files]
├── test_decoded_processed.json (test manifest)
└── train_tarred_1bk
├── metadata.yaml
├── tarred_audio_manifest.json
└── [1024 .tar files]
In case of 4 buckets it will look like:
.. code-block:: bash
└── train_tarred_4bk
├── bucket1
├── metadata.yaml
├── tarred_audio_manifest.json
└── [1024 .tar files]
├── bucket2
...
├── bucket3
└── bucket4
************************************
Training from scratch and finetuning
************************************
ASR models
##########
Our goal was to train two ASR models with different architectures: :ref:`Conformer-CTC <Conformer-CTC_model>` and :ref:`Conformer-Transducer <Conformer-Transducer_model>`, with around 120 million parameters.
The CTC model predicts output tokens for each timestep. The outputs are assumed to be independent of each other. As a result the CTC models work faster but they can produce outputs that are inconsistent with each other. CTC models are often combined with external language models in production. In contrast, the Transducer models contain the decoding part which generates the output tokens one by one and the next token prediction depends on this history. Due to autoregressive nature of decoding the inference speed is several times slower than that of CTC models, but the quality is usually better because it can incorporate language model information within the same model.
Training scripts and configs
############################
To train a Conformer-CTC model, we use `speech_to_text_ctc_bpe.py <https://github.com/NVIDIA/NeMo/tree/stable/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py>`_ with the default config `conformer_ctc_bpe.yaml <https://github.com/NVIDIA/NeMo/tree/stable/examples/asr/conf/conformer/conformer_ctc_bpe.yaml>`_.
To train a Conformer-Transducer model, we use `speech_to_text_rnnt_bpe.py <https://github.com/NVIDIA/NeMo/tree/stable/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py>`_ with the default config `conformer_transducer_bpe.yaml <https://github.com/NVIDIA/NeMo/tree/stable/examples/asr/conf/conformer/conformer_transducer_bpe.yaml>`_.
Any options of default config can be overwritten from command line.
Usually we should provide the options related to the dataset and tokenizer.
This is an example of how we can run the training script:
.. code-block:: bash
TOKENIZER=tokenizers/tokenizer_spe_bpe_v1024_max_4/
TRAIN_MANIFEST=data/train_tarred_1bk/tarred_audio_manifest.json
TRAIN_FILEPATHS=data/train_tarred_1bk/audio__OP_0..1023_CL_.tar
VAL_MANIFEST=data/dev_decoded_processed.json
TEST_MANIFEST=data/test_decoded_processed.json
python ${NEMO_ROOT}/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py \
--config-path=../conf/conformer/ \
--config-name=conformer_ctc_bpe \
exp_manager.name="Some name of our experiment" \
exp_manager.resume_if_exists=true \
exp_manager.resume_ignore_no_checkpoint=true \
exp_manager.exp_dir=results/ \
model.tokenizer.dir=$TOKENIZER \
model.train_ds.is_tarred=true \
model.train_ds.tarred_audio_filepaths=$TRAIN_FILEPATHS \
model.train_ds.manifest_filepath=$TRAIN_MANIFEST \
model.validation_ds.manifest_filepath=$VAL_MANIFEST \
model.test_ds.manifest_filepath=$TEST_MANIFEST
The option *exp_manager.resume_if_exists=true* allows to resume training. Actually you can stop training at any moment and then continue from the last checkpoint.
When the training is finished, the final model will be saved as *.nemo* file inside the folder that we specified in *exp_manager.exp_dir*.
Training dynamics
#################
The figure below shows the training dynamics when we train Kinyarwanda models **from scratch**. In these experiments we used the hyperparameters from the default configs, the training was run on 2 nodes with 16 gpus per node, training batch size was 32. We see that Transducer model achieves better quality than CTC.
.. image:: ../images/kinyarwanda_from_scratch.png
:align: center
:alt: Training dynamics of Kinyarwanda models trained from scratch
:width: 800px
Finetuning from another model
#############################
Often it's a good idea to initialize our ASR model with the weights of some other pretrained model, for example, a model for another language. It usually makes our model to converge faster and achieve better quality, especially if the dataset for our target language is small.
Though Kinyarwanda dataset is rather large, we also tried finetuning Kinyarwanda Conformer-Transducer model from different pretrained checkpoints, namely:
* English Conformer-Transducer checkpoint
* Self-supervised Learning (SSL) checkpoint trained on English data
* SSL checkpoint trained on multilingual data
To initialize from **non-SSL checkpoint** we should simply add the option `+init_from_pretrained_model`:
.. code-block:: bash
INIT_MODEL='stt_en_conformer_ctc_large'
python ${NEMO_ROOT}/examples/asr/asr_ctc/speech_to_text_ctc_bpe.py
...[same options as in the previous example]...
+init_from_pretrained_model=${INIT_MODEL}
In that case the pretrained model `stt_en_conformer_ctc_large <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_conformer_ctc_large>`_ will be automatically downloaded from NVIDIA GPU Cloud(NGC) and used to initialize weights before training.
To initialize from **SSL checkpoint** we should edit our training script like the following code:
.. code-block:: python
import nemo.collections.asr as nemo_asr
ssl_model = nemo_asr.models.ssl_models.SpeechEncDecSelfSupervisedModel.from_pretrained(model_name='ssl_en_conformer_large')
# define fine-tune model
asr_model = nemo_asr.models.EncDecCTCModelBPE(cfg=cfg.model, trainer=trainer)
# load ssl checkpoint
asr_model.load_state_dict(ssl_model.state_dict(), strict=False)
del ssl_model
When using finetuning you probably will need to change the some hyperparameters from the default config, especially the learning rate and learning rate policy. In the experiments below we used *model.optim.sched.name=CosineAnnealing* and *model.optim.lr=1e-3*.
The figure below compares the training dynamics for three Conformer-Transducer models. They differ only by how they are initialized. We see that finetuning leads to faster convergence and better quality. Initializing from SSL gives lowest WER at earlier stages, but in a longer period it performs worse.
.. image:: ../images/kinyarwanda_finetuning.png
:align: center
:alt: Training dynamics of Kinyarwanda models trained from scratch and finetuned from different pretrained checkpoints
:width: 800px
************************
Inference and evaluation
************************
Running the inference
#####################
To run the inference we need a pretrained model. This can be either a `.nemo` file that we get after the training is finished, or any published ASR model from `HF <https://huggingface.co/nvidia>`__ or `NGC <https://catalog.ngc.nvidia.com/?filters=application%7CAutomatic+Speech+Recognition%7Cuscs_automatic_speech_recognition&query=nemo>`__.
We run the inference using the following script:
.. code-block:: bash
python ${NEMO_ROOT}/examples/asr/transcribe_speech.py \
model_path=<path_to_of_your_model>.nemo \
dataset_manifest=./test_decoded_processed.json \
output_filename=./test_with_predictions.json \
batch_size=8 \
cuda=1 \
amp=True
To run inference with NVIDIA's Kinyarwanda checkpoints `STT Rw Conformer-CTC Large <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_rw_conformer_ctc_large>`_ or `STT Rw Conformer-Transducer Large <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_rw_conformer_transducer_large>`_ use:
.. code-block:: bash
python ${NEMO_ROOT}/examples/asr/transcribe_speech.py \
pretrained_name="stt_rw_conformer_ctc_large" \
dataset_manifest=test_decoded_processed.json \
output_filename=./pred_ctc.json \
batch_size=8 \
cuda=1 \
amp=True
**Note:** If you want to transcribe new audios, you can pass a folder with audio files using `audio_dir` parameter instead of `dataset_manifest`.
After the inference is finished the `output_filename` is a `.json` manifest augmented with a new field `pred_text` containing the resulting transcript. Example:
.. code-block::
{"audio_filepath": "test/common_voice_rw_19835615.wav", "text": "kw'ibumoso", "up_votes": 2, "down_votes": 0, "age": NaN, "gender": NaN, "accents": NaN, "client_id": "66675a7003e6baa3e7d4af01bff8324ac3c5f15e7f8918180799dd2928227c791f19e2811f9ec5779a2b06dac1b7a97fa7740dcfe98646ea1b5e106250c260be", "duration": 3.672, "pred_text": "n'ibumoso"}
{"audio_filepath": "test/common_voice_rw_24795878.wav", "text": "ni ryari uheruka kurya urusenda", "up_votes": 2, "down_votes": 0, "age": NaN, "gender": NaN, "accents": NaN, "client_id": "90e0438947a75b6c0cf59a0444aee3b81a76c5f9459c4b22df2e14b4ce257aeacaed8ac6092bfcd75b8e831633d58a84287fd62190c21d70d75efe8d93ed74ed", "duration": 3.312, "pred_text": "ni ryari uheruka kurya urusenda"}
{"audio_filepath": "test/common_voice_rw_24256935.wav", "text": "umunani", "up_votes": 2, "down_votes": 0, "age": NaN, "gender": NaN, "accents": NaN, "client_id": "974d4876e99e7437183c20f9107053acc9e514379d448bcf00aaaabc0927f5380128af86d39650867fa80a82525110dfc40784a5371c989de1a5bdf531f6d943", "duration": 3.24, "pred_text": "umunani"}
Word Error Rate (WER) and Character Error Rate (CER)
####################################################
As soon as we have a manifest file with `text` and `pred_text` we can measure the quality of predictions of our model.
.. code-block:: bash
# Calculate WER
python ${NEMO_ROOT}/examples/asr/speech_to_text_eval.py \
dataset_manifest=test_with_predictions.json \
use_cer=False \
only_score_manifest=True
# Calculate CER
python ${NEMO_ROOT}/examples/asr/speech_to_text_eval.py \
dataset_manifest=test_with_predictions.json \
use_cer=True \
only_score_manifest=True
Evaluation of NVIDIA's Kinyarwanda checkpoints
##############################################
If you run inference and evaluation of NVIDIA's published Kinyarwanda models, you should get metrics like these:
+----------------------------------+-------+-------+
| Model | WER % | CER % |
+==================================+=======+=======+
| stt_rw_conformer_ctc_large | 18.22 | 5.45 |
+----------------------------------+-------+-------+
| stt_rw_conformer_trasducer_large | 16.19 | 5.7 |
+----------------------------------+-------+-------+
Error analysis
##############
Still, even WER of 16% is not as good as we usually get for other languages trained with NeMo toolkit, so we may want to look at the errors that the model makes to better understand what's the problem.
We can use :doc:`Speech Data Explorer <../../tools/speech_data_explorer>` to analyze the errors.
If we run
.. code-block:: bash
python ${NEMO_ROOT}/tools/speech_data_explorer/data_explorer.py <your manifest file>
it will start a local server, and provide a http address to open from the browser.
In the UI we can see the model predictions and their diff with the reference, and also we can listen to the corresponding audio. We also can sort the sentences by descending WER and look through the top of them.
The error analysis showed several problems concerning the Kinyarwanda dataset:
* Noisy multi-speaker records (e.g. common_voice_rw_19830859.wav)
* Bad quality of record (e.g. common_voice_rw_24452415.wav)
* Orthographic variability related to space/no space/apostrophe
* *kugira ngo / kugirango*
* *nkuko / nk'uko*
* *n iyo / n'iyo*
* Multiple orthographic variants for foreign words
* *telefoni / telephone*
* *film / filime*
* *isiraheli / israel*
* *radio / radiyo*
* *kongo / congo*
* l/r variability
* *abamalayika / abamarayika*
docs/source/asr/featured_community_checkpoints.rst
+49
View File
@@ -0,0 +1,49 @@
.. _featured-community-checkpoints:
Featured Community Checkpoints
==============================
Community fine-tunes built on NVIDIA NeMo ASR checkpoints and published on Hugging Face.
For NVIDIA-published checkpoints, see :doc:`./asr_checkpoints` and the `NVIDIA Hugging Face organization <https://huggingface.co/nvidia>`__.
.. note::
Community checkpoints are maintained by their authors, not by the NeMo team.
Use each model's Hugging Face model card and the framework project linked below for up-to-date setup and inference instructions.
.. list-table::
:header-rows: 1
:widths: 28 52 20
* - Checkpoint
- What's special
- Framework
* - `akera/parakeet-tdt-salt <https://huggingface.co/akera/parakeet-tdt-salt>`__
- SALT multilingual ASR for 10 East African languages. Hybrid TDT+CTC FastConformer (600M), fine-tuned from `parakeet-tdt-0.6b-v3 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3>`__.
- NeMo
* - `johannhartmann/parakeet_de_med <https://huggingface.co/johannhartmann/parakeet_de_med>`__
- German medical documentation ASR (PEFT). WER 11.73% → 3.28% on a 122-sample medical eval set.
- NeMo
* - `qenneth/parakeet-tdt-0.6b-v3-finetuned-for-ATC <https://huggingface.co/qenneth/parakeet-tdt-0.6b-v3-finetuned-for-ATC>`__
- ATC English ASR on `jacktol/ATC-ASR-Dataset <https://huggingface.co/datasets/jacktol/ATC-ASR-Dataset>`__. Test WER 5.99%.
- NeMo
* - `KasuleTrevor/parakeet-0.6b-cv-sw-5hr_v9 <https://huggingface.co/KasuleTrevor/parakeet-0.6b-cv-sw-5hr_v9>`__
- Swahili ASR fine-tune on ~5 hours of Common Voice data.
- NeMo
* - `NeurologyAI/neuro-parakeet-mlx <https://huggingface.co/NeurologyAI/neuro-parakeet-mlx>`__
- German medical/neurology ASR for Apple Silicon. WER 1.04% on the author's medical validation set.
- MLX
* - `cstr/parakeet-tdt-0.6b-v3-GGUF <https://huggingface.co/cstr/parakeet-tdt-0.6b-v3-GGUF>`__
- Quantised Parakeet TDT (Q4_K ~467 MB). 25 EU languages, word-level timestamps.
- GGUF (`CrispASR <https://github.com/CrispStrobe/CrispASR>`__)
* - `cstr/canary-1b-v2-GGUF <https://huggingface.co/cstr/canary-1b-v2-GGUF>`__
- Quantised Canary 1B (Q4_K ~673 MB). Multilingual ASR and speech translation.
- GGUF (`CrispASR <https://github.com/CrispStrobe/CrispASR>`__)
.. _submit-a-community-checkpoint:
Submit a Community Checkpoint
-----------------------------
To suggest a checkpoint for this page, open a `GitHub issue <https://github.com/NVIDIA-NeMo/NeMo/issues/new>`__ with the Hugging Face model link, NeMo base checkpoint, task, languages, evaluation results, and inference framework.
docs/source/asr/featured_models.rst
+111
View File
@@ -0,0 +1,111 @@
Featured Models
===============
NeMo's ASR collection supports several model architectures. This page covers the key model families and their capabilities.
For pretrained checkpoints, see :doc:`All Checkpoints <./asr_checkpoints>`.
For config file details, see :doc:`Configuration Files <./configs>`.
Parakeet
~~~~~~~~
Parakeet is a family of ASR models with a :ref:`FastConformer Encoder <Fast-Conformer>` and CTC, RNN-T, or TDT decoders.
* `Parakeet-TDT-0.6B V3 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3>`__ — 25 languages, PnC, blazing fast
* `Parakeet-TDT-0.6B V2 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2>`__ — English-only, PnC, blazing fast
* `Parakeet-TDT/CTC-110M <https://huggingface.co/nvidia/parakeet-tdt_ctc-110m>`__ — Edge deployment
* `Nemotron-3.5-ASR-Streaming <https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b>`__ — Real-time streaming, 40 languages
* `Multitalker-Parakeet <https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1>`__ — Multi-speaker streaming
Canary
~~~~~~
Canary models are encoder-decoder models with a :ref:`FastConformer Encoder <Fast-Conformer>` and Transformer Decoder :cite:`asr-models-vaswani2017aayn`.
They support ASR in 25 EU languages, speech translation (AST), and punctuation/capitalization (PnC).
* `Canary-1B V2 <https://huggingface.co/nvidia/canary-1b-v2>`__ — Flagship: 25 languages, PnC, timestamps
* `Canary-Qwen-2.5B <https://huggingface.co/nvidia/canary-qwen-2.5b>`__ — English only, PnC, highest accuracy
* `Canary-1B Flash <https://huggingface.co/nvidia/canary-1b-flash>`__ / `180M Flash <https://huggingface.co/nvidia/canary-180m-flash>`__ — Optimized for speed
Canary supports chunked and `streaming inference <https://github.com/NVIDIA-NeMo/NeMo/blob/main/tutorials/asr/Streaming_ASR_Pipelines.ipynb>`__.
.. _Conformer_model:
Conformer
---------
The Conformer :cite:`asr-models-gulati2020conformer` combines self-attention and convolution modules. NeMo supports CTC, Transducer, and HAT variants.
* **Conformer-CTC**: Non-autoregressive, uses :class:`~nemo.collections.asr.models.EncDecCTCModelBPE`
* **Conformer-Transducer**: Autoregressive, uses :class:`~nemo.collections.asr.models.EncDecRNNTBPEModel`
* **Conformer-HAT**: Separates labels and blank predictions for better external LM integration (`paper <https://arxiv.org/abs/2003.07705>`_)
.. _Conformer-CTC_model:
.. _Conformer-Transducer_model:
.. _Conformer-HAT_model:
Configs: ``examples/asr/conf/conformer/``
.. _Fast-Conformer:
Fast-Conformer
--------------
Fast Conformer has 8x depthwise convolutional subsampling and reduced kernel sizes, making it ~2.4x faster than standard Conformer with minimal quality loss.
Supports Longformer-style local attention for audio >1 hour.
Configs: ``examples/asr/conf/fastconformer/``
.. _cache-aware streaming conformer:
Cache-aware Streaming Conformer
-------------------------------
Streaming models trained with limited right context for real-time inference with caching to avoid duplicate computation. Supports three modes: fully causal, regular look-ahead, and chunk-aware look-ahead (recommended).
* `Tutorial notebook <https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Online_ASR_Microphone_Demo_Cache_Aware_Streaming.ipynb>`_
* Simulation script: ``examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py``
* Supports multiple look-aheads with ``att_context_size`` lists
Configs: ``examples/asr/conf/fastconformer/cache_aware_streaming/``
.. _RNNT-Prompt_model:
**With Prompt Conditioning (RNN-T only):** Cache-aware streaming RNN-T model with language-ID prompt conditioning for multilingual ASR via
:class:`~nemo.collections.asr.models.EncDecRNNTBPEModelWithPrompt`. The streaming inference
script accepts a ``target_lang`` flag to select the prompt at runtime
(see :ref:`RNN-T with Prompt Conditioning Configuration <RNNT-Prompt_model__Config>`).
Config: ``fastconformer_transducer_bpe_streaming_prompt.yaml``
Multitalker Streaming
---------------------
Streaming multi-speaker ASR based on cache-aware FastConformer with speaker kernel injection :cite:`asr-models-wang25y_interspeech`. Deploys one model instance per speaker for robust transcription of overlapped speech.
* `Model card <https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1>`__
* `Tutorial <https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Streaming_Multitalker_ASR.ipynb>`_
.. _Hybrid-Transducer_CTC_model:
Hybrid-Transducer-CTC
----------------------
Models with both RNN-T and CTC decoders trained jointly. Switch at inference time via ``asr_model.change_decoding_strategy(decoder_type='ctc' or 'rnnt')``.
* :class:`~nemo.collections.asr.models.EncDecHybridRNNTCTCBPEModel` (BPE) / :class:`~nemo.collections.asr.models.EncDecHybridRNNTCTCModel` (char)
* Configs: ``examples/asr/conf/fastconformer/hybrid_transducer_ctc/``
.. _Hybrid-Transducer-CTC-Prompt_model:
**With Prompt Conditioning:** Extends Hybrid models with learnable prompt embeddings for multilingual/multi-domain ASR via :class:`~nemo.collections.asr.models.EncDecHybridRNNTCTCBPEModelWithPrompt`. Config: ``fastconformer_hybrid_transducer_ctc_bpe_prompt.yaml``
References
----------
.. bibliography:: asr_all.bib
:style: plain
:labelprefix: ASR-MODELS
:keyprefix: asr-models-
docs/source/asr/fine_tuning.rst
+176
View File
@@ -0,0 +1,176 @@
.. _asr-fine-tuning:
===========
Fine-Tuning
===========
This page covers how to fine-tune pretrained ASR models in NeMo.
When to Fine-Tune
-----------------
Fine-tuning is recommended when:
* You have domain-specific data (medical, legal, call center, etc.) and want to improve accuracy on that domain.
* You need to adapt to a new accent, speaking style, or acoustic environment.
* You want to add support for a new language using a pretrained multilingual model.
If you have a large, diverse dataset and want to train from scratch, see :doc:`Configuration Files <./configs>` for full training setup.
Working with an agent?
----------------------
Check out our latest ``/nemo-speech-finetune-asr`` `agent skill <https://github.com/NVIDIA-NeMo/NeMo/tree/main/.claude/skills/nemo-speech-asr-finetune>`_.
Fine-Tuning Script
------------------
Use the ``speech_to_text_finetune.py`` script with the default config at
``examples/asr/conf/asr_finetune/speech_to_text_finetune.yaml``:
.. code-block:: bash
python examples/asr/speech_to_text_finetune.py \
--config-path=../conf/asr_finetune \
--config-name=speech_to_text_finetune \
init_from_pretrained_model="nvidia/parakeet-tdt-0.6b-v2" \
model.train_ds.manifest_filepath=/path/to/train_manifest.json \
model.validation_ds.manifest_filepath=/path/to/val_manifest.json \
trainer.devices=1 \
trainer.max_epochs=50
You must specify either ``init_from_pretrained_model`` (NGC/HuggingFace name) or ``init_from_nemo_model`` (local ``.nemo`` path) to load the pretrained weights.
Initialization Options
-----------------------
NeMo supports several ways to initialize a model for fine-tuning:
**From a pretrained model (NGC/HuggingFace):**
.. code-block:: yaml
init_from_pretrained_model: "nvidia/parakeet-tdt-0.6b-v2"
**From a local .nemo checkpoint:**
.. code-block:: yaml
init_from_nemo_model: "/path/to/checkpoint.nemo"
**Partial loading (selective layers):**
You can include or exclude specific model components using ``include`` and ``exclude`` lists:
.. code-block:: yaml
init_from_nemo_model: "/path/to/checkpoint.nemo"
init_from_nemo_model_include:
- encoder
- preprocessor
init_from_nemo_model_exclude:
- decoder
This is useful when changing the decoder architecture or tokenizer while keeping the pretrained encoder.
Tokenizer Changes
------------------
**Same tokenizer (same vocabulary):**
No special handling needed — fine-tune directly.
**New tokenizer (different vocabulary):**
When changing the tokenizer (e.g., for a new language or domain), you need to:
1. Provide the new tokenizer directory in the config.
2. Exclude the decoder/joint from initialization (for Transducer models) or exclude the final linear layer (for CTC models).
.. code-block:: yaml
model:
tokenizer:
dir: /path/to/new/tokenizer
type: bpe
init_from_nemo_model: "/path/to/pretrained.nemo"
init_from_nemo_model_exclude:
- decoder
- joint
**Enforcing a single language after fine-tuning:**
When fine-tuning a multilingual ``EncDecMultiTaskModel`` (e.g., Canary) on a single language, the model may still exhibit phonetic drift — switching languages mid-utterance at inference time. To enforce a specific language during decoding, explicitly set ``source_lang`` and ``target_lang`` to the same language:
.. code-block:: python
results = model.transcribe(
audio=["audio.wav"],
source_lang="de",
target_lang="de",
)
See :ref:`Enforcing a Single Language <asr-enforcing-single-language>` in the Inference documentation for more details.
Fine-Tuning with HuggingFace Datasets
---------------------------------------
NeMo supports loading datasets directly from HuggingFace:
.. note::
HuggingFace dataset loading is not currently supported with the Lhotse dataloader.
.. code-block:: bash
python examples/asr/speech_to_text_finetune_with_hf.py \
--config-path=<path to config directory> \
--config-name=<config name> \
model.train_ds.hf_data_cfg.path="mozilla-foundation/common_voice_11_0" \
model.train_ds.hf_data_cfg.name="en" \
model.train_ds.hf_data_cfg.split="train" \
model.validation_ds.hf_data_cfg.path="mozilla-foundation/common_voice_11_0" \
model.validation_ds.hf_data_cfg.name="en" \
model.validation_ds.hf_data_cfg.split="validation"
Key Configuration Parameters
-----------------------------
The most important parameters for fine-tuning:
.. list-table::
:header-rows: 1
:widths: 30 70
* - Parameter
- Description
* - ``trainer.max_epochs``
- Number of fine-tuning epochs (typically 50-100 for domain adaptation)
* - ``model.optim.lr``
- Learning rate (use lower than training from scratch, e.g., 1e-4 to 1e-5)
* - ``model.train_ds.manifest_filepath``
- Path to training manifest (NeMo JSON format)
* - ``model.train_ds.batch_size``
- Batch size per GPU
* - ``init_from_pretrained_model``
- NGC/HF model name to initialize from
* - ``init_from_nemo_model``
- Local .nemo file to initialize from
For the complete configuration reference, see :doc:`Configuration Files <./configs>`.
Tips
----
1. **Start with a low learning rate** — fine-tuning with too high a learning rate can destroy pretrained features. Typical fine-tuning LRs are 1e-4 to 1e-5. If your pretrained config uses the Noam (warmup + decay) scheduler, override it with a constant or cosine-annealing schedule to avoid the warmup phase resetting to a high LR.
2. **Use Lhotse dataloading** for efficient training with dynamic batching. See :doc:`Lhotse Dataloading </dataloaders>`.
3. **Use spec augmentation** during fine-tuning to improve robustness. See :ref:`Augmentation Configurations <asr-configs-augmentation-configurations>`.
4. **For multilingual fine-tuning**, use a multilingual tokenizer. NeMo supports two approaches: a **unified multilingual SentencePiece tokenizer** — a single BPE model trained on all target languages (as used by Canary v2/Flash), and an ``AggregateTokenizer`` that combines separate monolingual tokenizers with per-language routing (see :doc:`Configs <./configs>` for the ``agg`` tokenizer setup). For prompt-conditioned multilingual models, see the :ref:`Hybrid model with prompt conditioning <Hybrid-Transducer-CTC-Prompt_model__Config>` or the :ref:`RNN-T-only streaming variant <RNNT-Prompt_model__Config>`.
docs/source/asr/images/citrinet_vertical.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 135 KiB

Some files were not shown because too many files have changed in this diff Show More