chore: import upstream snapshot with attribution

docs/source/tools/asr_evaluator.rst

@@ -0,0 +1,6 @@
+ASR Evaluator
+========================
+
+ASR evaluator is a tool for thoroughly evaluating the performance of ASR models and other features such as Voice Activity Detection.
+
+See more details in: https://github.com/NVIDIA/NeMo/tree/stable/tools/asr_evaluator

docs/source/tools/comparison_tool.rst

@@ -0,0 +1,182 @@
+Comparison tool for ASR Models
+==============================
+
+The Comparison Tool (CT) allows to compare predictions of different ASR models at word accuracy and utterance level. 
+
++--------------------------------------------------------------------------------------------------------------------------+
+| **Comparison tool features:**                                                                                            |
++--------------------------------------------------------------------------------------------------------------------------+
+| navigation across dataset's vocabulary using an interactive datatable that supports sorting and filtering                |
++--------------------------------------------------------------------------------------------------------------------------+
+| interactive visualization of model's accuracy                                                                            |
++--------------------------------------------------------------------------------------------------------------------------+
+| visual comparison of predictions of different models                                                                     |
++--------------------------------------------------------------------------------------------------------------------------+
+| visual comparison of utterances by their WER/CER                                                                         |
++--------------------------------------------------------------------------------------------------------------------------+
+| listening selected utterance                                                                                             |
++--------------------------------------------------------------------------------------------------------------------------+
+
+Getting Started
+---------------
+The Comparison Tool is integrated in NeMo Speech Data Explorer (SDE) that could be found at `NeMo/tools/speech_data_explorer <https://github.com/NVIDIA/NeMo/tree/main/tools/speech_data_explorer>`__.
+
+Please install the SDE requirements:
+
+.. code-block:: bash
+
+    pip install -r tools/speech_data_explorer/requirements.txt
+
+Then run:
+
+.. code-block:: bash
+
+    python tools/speech_data_explorer/data_explorer.py -h
+
+    usage: data_explorer.py [-h] [--vocab VOCAB] [--port PORT] [--disable-caching-metrics] [--estimate-audio-metrics] [--debug] manifest
+
+    Speech Data Explorer
+
+    positional arguments:
+    manifest              path to JSON manifest file
+
+    optional arguments:
+    -h, --help            show this help message and exit
+    --vocab VOCAB         optional vocabulary to highlight OOV words
+    --port PORT           serving port for establishing connection
+    --disable-caching-metrics
+                            disable caching metrics for errors analysis
+    --estimate-audio-metrics, -a
+                            estimate frequency bandwidth and signal level of audio recordings
+    --debug, -d           enable debug mode
+    --audio-base-path A base path for the relative paths in manifest. It defaults to manifest path.
+    --names_compared, -nc names of the two fields that will be compared, example: pred_text_model1 pred_text_model2.
+    --show_statistics, -shst field name for which you want to see statistics (optional). Example: pred_text_model1.
+
+CT takes a JSON manifest file (that describes speech datasets in NeMo) as an input. It should contain the following fields:
+
+* `audio_filepath` (path to audio file)
+* `duration` (duration of the audio file in seconds)
+* `text` (reference transcript)
+* `pred_text_<model_1_name>`
+* `pred_text_<model_2_name>`
+
+SDE supports any extra custom fields in the JSON manifest. If the field is numeric, then SDE can visualize its distribution across utterances.
+
+JSON manifest has attribute `pred_text`, SDE interprets it as a predicted ASR transcript and computes error analysis metrics.
+If you want SDE to analyse another prediction field, then please use `--show_statistics` argument.
+
+User Interface
+--------------
+
+SDE has three pages if `--names_compared` argument is not empty:
+
+* `Statistics` (to display global statistics and aggregated error metrics)
+
+    .. image:: images/sde_base_stats.png
+        :align: center
+        :width: 800px
+        :alt: SDE Statistics
+        
+
+* `Samples` (to allow navigation across the entire dataset and exploration of individual utterances)
+
+    .. image:: images/sde_player.png
+        :align: center
+        :width: 800px
+        :alt: SDE Statistics
+
+* `Comparison tool` (to explore predictions at word level)
+
+    .. image:: images/scrsh_2.png
+        :align: center
+        :width: 800px
+        :alt: Comparison tool
+
+
+CT has an interactive datatable for dataset's vocabulary (that supports navigation, filtering, and sorting):
+
+
+* Data (that visualizes all dataset's words and adds each one's accuracy)
+
+    .. image:: images/scrsh_3.png
+        :align: center
+        :width: 800px
+        :alt: Data
+
+CT supports all operations, that present in SDE, and allows combination of filtering expressions with "or" and "and" operations
+
+* filtering (by entering a filtering expression in a cell below the header's cell)
+
+    .. image:: images/scrsh_4.png
+        :align: center
+        :width: 800px
+        :alt: Filtering
+
+
+Analysis of Speech Datasets
+---------------------------
+
+If there is a pre-trained ASR model, then the JSON manifest file can be extended with ASR predicted transcripts:
+
+.. code-block:: bash
+
+    python examples/asr/transcribe_speech.py pretrained_name=<ASR_MODEL_NAME> dataset_manifest=<JSON_FILENAME> append_pred=False pred_name_postfix=<model_name_1>
+    
+
+More information about transcribe_speech parameters is available in the code: `NeMo/examples/asr/transcribe_speech.py <https://github.com/NVIDIA/NeMo/blob/main/examples/asr/transcribe_speech.py>`__.
+.
+
+    .. image:: images/scrsh_2.png
+        :align: center
+        :width: 800px
+        :alt: fields
+
+Fields 1 and 2 are responsible for what will be displayed on the horizontal and vertical axes.
+
+Fields 3 and 4 allow you to convert any available numeric parameter into color and size, respectively.
+
+Fields 5 and 6 are responsible for point spacing. Some data points might have the same coordinates on both axes, in which case there will be an overlap, and in order to be able to explore each point, the option for their spreading was added.
+
+    .. image:: images/scrsh_5.png
+        :align: center
+        :width: 800px
+        :alt: dot spacing
+
+Point spacing works as follows: a small random value is added to all point coordinates, the value of which is limited by the "radius" parameter, which can be set manually.
+
+    .. image:: images/scrsh_9.png
+        :align: center
+        :width: 800px
+        :alt: Example
+
+In this case, all points lying above the diagonal have higher accuracy with the model displayed on the vertical axis, and all points below the diagonal were recognized better with the model displayed on the horizontal axis.
+
+Points marked with circles should be explored first.
+
+Words in the first quarter were well recognized by both models, and conversely, words in the third quarter were poorly recognized by both models.
+
+To compare models at utterance level, pick it at top dropdown field.
+
+At the next field you could choose metric: WER or CER
+
+    .. image:: images/scr_10.png
+        :align: center
+        :width: 800px
+        :alt: Switch mode
+		
+When an utterance level is selected, it is possible to click on a point on the graph, and the corresponding utterance will be automatically selected. 
+
+If audio files are available, there will be an option to listen to the audio recording and view its waveform. 
+
+    .. image:: images/scr_11.png
+        :align: center
+        :width: 800px
+        :alt: Audio player
+		
+In this mode, filtering is still available as well.
+
+**Limitations**
+
+To ensure efficient processing and avoid issues with memory limitations and slow performance, it is recommended to keep the manifests within the limits of 320 hours or around 170,000 utterances. 
+Exceeding these limits may result in both memory constraints and slower processing.

docs/source/tools/ctc_segmentation.rst

@@ -0,0 +1,20 @@
+Dataset Creation Tool Based on CTC-Segmentation
+===============================================
+
+This tool provides functionality to align long audio files with the corresponding transcripts and split them into shorter fragments
+that are suitable for an Automatic Speech Recognition (ASR) model training.
+
+More details could be found in `NeMo/tutorials/tools/CTC_Segmentation_Tutorial.ipynb <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/tools/CTC_Segmentation_Tutorial.ipynb>`__ (can be executed with `Google's Colab <https://colab.research.google.com/notebooks/intro.ipynb>`_).
+
+The tool is based on the `CTC-Segmentation <https://github.com/lumaku/ctc-segmentation>`__ package and
+`CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
+<https://arxiv.org/abs/2007.09127>`__ :cite:`tools-kurzinger2020ctc`
+
+
+References
+----------
+
+.. bibliography:: tools_all.bib
+    :style: plain
+    :labelprefix: TOOLS
+    :keyprefix: tools-

docs/source/tools/intro.rst

@@ -0,0 +1,24 @@
+:orphan:
+
+Speech AI Tools
+===============
+
+NeMo provides a set of tools useful for developing Automatic Speech Recognitions (ASR) and Text-to-Speech (TTS) synthesis models: \
+`https://github.com/NVIDIA/NeMo/tree/stable/tools <https://github.com/NVIDIA/NeMo/tree/stable/tools>`__ .
+
+.. toctree::
+   :maxdepth: 1
+
+   nemo_forced_aligner
+   ctc_segmentation
+   speech_data_explorer
+   comparison_tool
+   asr_evaluator
+
+
+There are also additional NeMo-related tools hosted in separate github repositories:
+
+.. toctree::
+   :maxdepth: 1
+
+   speech_data_processor

docs/source/tools/nemo_forced_aligner.rst

@@ -0,0 +1,173 @@
+NeMo Forced Aligner (NFA)
+=========================
+
+NFA is hosted here: https://github.com/NVIDIA/NeMo/tree/main/tools/nemo_forced_aligner.
+
+
+NFA is a tool for generating token-, word- and segment-level timestamps of speech in audio using NeMo's CTC-based Automatic Speech Recognition models. 
+You can provide your own reference text, or use ASR-generated transcription. 
+You can use NeMo's ASR Model checkpoints out of the box in 14+ languages (see :doc:`ASR Model Checkpoints </asr/asr_checkpoints>`), or train your own model.
+NFA can be used on long audio files of 1+ hours duration (subject to your hardware and the ASR model used).
+
+Demos & Tutorials
+-----------------
+
+* HuggingFace Space `demo <https://huggingface.co/spaces/erastorgueva-nv/NeMo-Forced-Aligner>`__ to quickly try out NFA in various languages.
+* NFA "how-to" notebook `tutorial <https://nvidia.github.io/NeMo/blogs/2023/2023-08-forced-alignment/>`__.
+* "How forced alignment works" NeMo blog `tutorial <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/tools/NeMo_Forced_Aligner_Tutorial.ipynb>`__.
+
+Quickstart
+----------
+
+1. Install NeMo with the ASR collection. See :ref:`installation`.
+2. Prepare a NeMo-style manifest containing the paths of audio files you would like to proces, and (optionally) their text.
+3. Run NFA's ``align.py`` script with the desired config, e.g.:
+
+.. code-block::
+
+    python <path_to_NeMo>/tools/nemo_forced_aligner/align.py \
+	    pretrained_name="stt_en_fastconformer_hybrid_large_pc" \
+	    manifest_filepath=<path to manifest of utterances you want to align> \
+	    output_dir=<path to where your output files will be saved>
+
+.. image:: https://github.com/NVIDIA/NeMo/releases/download/v1.20.0/nfa_run.png
+
+How do I use NeMo Forced Aligner?
+---------------------------------
+
+To use NFA, all you need to provide is a correct NeMo manifest (with ``"audio_filepath"`` and, optionally, ``"text"`` fields).
+
+Call the ``align.py`` script, specifying the parameters as follows:
+
+* ``pretrained_name``: string specifying the name of a CTC NeMo ASR model which will be automatically downloaded from NGC and used for generating the log-probs which we will use to do alignment. Any CTC model (e.g., Conformer CTC, FastConformer CTC) should work, in any language (only English has been tested so far). If ``model_path`` is specified, ``pretrained_name`` must not be specified.
+
+	Note: Currently NFA can only use CTC models, or Hybrid CTC-Transducer models (in CTC mode). Pure Transducer models cannot be used.
+
+* ``model_path``: string specifying the local filepath to a CTC NeMo ASR model which will be used to generate the log-probs which we will use to do alignment. If ``pretrained_name`` is specified, ``model_path`` must not be specified.
+
+	.. note:: Currently NFA can only use CTC models, or Hybrid CTC-Transducer models (in CTC mode). Pure Transducer models cannot be used.
+
+* ``manifest_filepath``: The path to the manifest of the data you want to align, containing ``'audio_filepath'`` and ``'text'`` fields. The audio filepaths need to be absolute paths.
+
+* ``output_dir``: The folder where to save the output files (e.g. CTM, ASS) containing the generated alignments and new JSON manifest containing paths to those CTM/ASS files. The CTM file will be called ``<output_dir>/ctm/{tokens,words,segments}/<utt_id>.ctm`` and each line in each file will start with ``<utt_id>``. By default, ``utt_id`` will be the stem of the audio_filepath. This can be changed by overriding ``audio_filepath_parts_in_utt_id``. The new JSON manifest will be at ``<output_dir>/<original manifest file name>_with_ctm_paths.json``. The ASS files will be at ``<output_dir>/ass/{tokens,words}/<utt_id>.ass``. You can adjust which files should be saved by adjusting the parameter ``save_output_file_formats``. 
+
+Optional parameters:
+^^^^^^^^^^^^^^^^^^^^
+
+* ``align_using_pred_text``: if True, will transcribe the audio using the ASR model (specified by ``pretrained_name`` or ``model_path``) and then use that transcription as the reference text for the forced alignment. The ``"pred_text"`` will be saved in the output JSON manifest at ``<output_dir>/{original manifest name}_with_ctm_paths.json``. To avoid over-writing other transcribed texts, if there are already ``"pred_text"`` entries in the original manifest, the program will exit without attempting to generate alignments.  (Default: False). 
+
+* ``transcribe_device``: The device that will be used for generating log-probs (i.e. transcribing). If None, NFA will set it to 'cuda' if it is available (otherwise will set it to 'cpu'). If specified ``transcribe_device`` needs to be a string that can be input to the ``torch.device()`` method. (Default: ``None``).
+
+* ``viterbi_device``: The device that will be used for doing Viterbi decoding. If None, NFA will set it to 'cuda' if it is available (otherwise will set it to 'cpu'). If specified ``transcribe_device`` needs to be a string that can be input to the ``torch.device()`` method.(Default: ``None``).
+
+* ``batch_size``: The batch_size that will be used for generating log-probs and doing Viterbi decoding. (Default: 1).
+
+* ``use_local_attention``: boolean flag specifying whether to try to use local attention for the ASR Model (will only work if the ASR Model is a Conformer model). If local attention is used, we will set the local attention context size to [64,64].
+
+* ``additional_segment_grouping_separator``: a list of strings used to separate the text into smaller segments. If set to ``None``, then the whole text will be treated as a single segment. (Default: ``['.', '?', '!', '...']``. Cannot be empty string or space (" "), as NFA will automatically produce word-level timestamps for substrings separated by spaces).
+
+	.. note:: Starting in NeMo 2.5.0, separators are preserved in segment text after splitting. if ``additional_segment_grouping_separator="['.', '?', '!', '...']"`` (as is the default), then the text ``"Hi, have you updated your NeMo? Yes. Sure!"`` will result in the following segments ``["Hi, have you updated your NeMo?", "Yes.", "Sure!"]``.
+
+* ``remove_blank_tokens_from_ctm``: a boolean denoting whether to remove <blank> tokens from token-level output CTMs. (Default: False). 
+
+* ``audio_filepath_parts_in_utt_id``: This specifies how many of the 'parts' of the audio_filepath we will use (starting from the final part of the audio_filepath) to determine the utt_id that will be used in the CTM files. (Default: 1, i.e. utt_id will be the stem of the basename of audio_filepath). Note also that any spaces that are present in the audio_filepath will be replaced with dashes, so as not to change the number of space-separated elements in the CTM files.
+
+* ``minimum_timestamp_duration``: a float indicating a minimum duration (in seconds) for timestamps in the CTM. If any line in the CTM has a duration lower than the ``minimum_timestamp_duration``, it will be enlarged from the middle outwards until it meets the minimum_timestamp_duration, or reaches the beginning or end of the audio file. Note that this may cause timestamps to overlap. (Default: 0, i.e. no modifications to predicted duration).
+
+* ``use_buffered_chunked_streaming``: a flag to indicate whether to do buffered chunk streaming. Notice only CTC models with ``per_feature`` preprocessor are supported. The below two params are needed if this option set to ``True``.
+
+* ``chunk_len_in_secs``: the chunk size for buffered chunked streaming inference. Default is 1.6 seconds.
+
+* ``total_buffer_in_secs``: the buffer size for buffered chunked streaming inference. Default is 4.0 seconds.
+
+* ``simulate_cache_aware_streaming``: a flag to indicate whether to use cache aware streaming to do get the logits for alignment. Default: ``False``.
+
+* ``save_output_file_formats``: list of file formats to use for saving the output. Default: ``["ctm", "ass"]`` (these are all the available ones currently).
+
+* ``ctm_file_config``: ``CTMFileConfig`` to specify the configuration of the output CTM files.
+
+* ``ass_file_config``: ``ASSFileConfig`` to specify the configuration of the output ASS files.
+
+Input manifest file format
+--------------------------
+By default, NFA needs to be provided with a 'manifest' file where each line specifies the absolute "audio_filepath" and "text" of each utterance that you wish to produce alignments for, like the format below:
+
+.. code-block::
+
+    {"audio_filepath": "/absolute/path/to/audio.wav", "text": "the transcription of the utterance"}
+
+You can omit the ``"text"`` field from the manifest if you specify ``align_using_pred_text=true``. In that case, any ``"text"`` fields in the manifest will be ignored: the ASR model at ``pretrained_name`` or ``model_path`` will be used to transcribe the audio and obtain ``"pred_text"``, which will be used as the reference text for the forced alignment process. The ``"pred_text"`` will also be saved in the output manifest JSON file at ``<output_dir>/<original manifest file name>_with_output_file_paths.json``. To remove the possibility of overwriting ``"pred_text"``, NFA will raise an error if ``align_using_pred_text=true`` and there are existing ``"pred_text"`` fields in the original manifest.
+
+	.. note:: NFA does not require ``"duration"`` fields in the manifest, and can align long audio files without running out of memory. The duration of audio file you can align will depend on the amount of memory on your machine. NFA will also produce better alignments the more accurate the reference text in ``"text"`` is.
+
+
+Output CTM file format
+----------------------
+
+For each utterance specified in a line of ``manifest_filepath``, several CTM files will be generated:
+
+* a CTM file containing token-level alignments at ``<output_dir>/ctm/tokens/<utt_id>.ctm``,
+* a CTM file containing word-level alignments at ``<output_dir>/ctm/words/<utt_id>.ctm``,
+* a CTM file containing segment-level alignments at ``<output_dir>/ctm/segments/<utt_id>.ctm``. If ``additional_segment_grouping_separator`` is specified, the segments will be parts of the text separated by ``additonal_segment_grouping_separator``. If it is not specified, the entire text will be treated as a single segment.
+
+Each CTM file will contain lines of the format:
+``<utt_id> 1 <start time in seconds> <duration in seconds> <text, ie token/word/segment>``.
+Note the second item in the line (the 'channel ID', which is required by the CTM file format) is always 1, as NFA operates on single channel audio.
+
+``CTMFileConfig`` parameters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``CTMFileConfig`` (which is passed into the main NFA config) has the following parameters:
+
+* ``remove_blank_tokens``: bool (default ``False``) to specify if the token-level CTM files should have the timestamps of the blank tokens removed.
+* ``minimum_timestamp_duration``: float (default ``0``) to specify the minimum duration that will be applied to all timestamps. If any line in the CTM has a duration lower than this, it will be enlarged from the middle outwards until it meets the ``minimum_timestamp_duration``, or reaches the beginning or end of the audio file. Note that using a non-zero value may cause timestamps to overlap.
+
+Output ASS file format
+----------------------
+
+NFA will produce the following ASS files, which you can use to generate subtitle videos:
+
+* ASS files with token-level highlighting will be at ``<output_dir>/ass/tokens/<utt_id>.ass,``
+* ASS files with word-level highlighting will be at ``<output_dir>/ass/words/<utt_id>.ass``.
+
+All words belonging to the same segment 'segments' will appear at the same time in the subtitles generated with the ASS files. If you find that your segments are not the right size, you can use set ``ass_file_config.resegment_text_to_fill_space=true`` and specify some number of ``ass_file_config.max_lines_per_segment``.
+
+``ASSFileConfig`` parameters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``ASSFileConfig`` (which is passed into the main NFA config) has the following parameters:
+
+* ``fontsize``: int (default value ``20``) which will be the fontsize of the text
+* ``vertical_alignment``: string (default value ``center``) to specify the vertical alignment of the text. Can be one of ``center``, ``top``, ``bottom``.
+* ``resegment_text_to_fill_space``: bool (default value ``False``). If ``True``, the text will be resegmented such that each segment will not take up more than (approximately) ``max_lines_per_segment`` when the ASS file is applied to a video.
+* ``max_lines_per_segment``: int (defaulst value ``2``) which specifies the number of lines per segment to display. This parameter is only used if ``resegment_text_to_fill_space`` is ``True``.
+* ``text_already_spoken_rgb``: List of 3 ints (default value is [49, 46, 61], which makes a dark gray). The RGB values of the color that will be used to highlight text that has already been spoken.
+* ``text_being_spoken_rgb``: List of 3 ints (default value is [57, 171, 9] which makes a dark green). The RGB values of the color that will be used to highlight text that is being spoken.
+* ``text_not_yet_spoken_rgb``: List of 3 ints (default value is [194, 193, 199] which makes a dark green). The RGB values of the color that will be used to highlight text that has not yet been spoken.
+
+Output JSON manifest file format
+--------------------------------
+
+A new manifest file will be saved at ``<output_dir>/<original manifest file name>_with_output_file_paths.json``. It will contain the same fields as the original manifest, and additionally:
+
+* ``"token_level_ctm_filepath"`` (if ``save_output_file_formats`` contains ``ctm``)
+* ``"word_level_ctm_filepath"`` (if ``save_output_file_formats`` contains ``ctm``)
+* ``"segment_level_ctm_filepath"`` (if ``save_output_file_formats`` contains ``ctm``)
+* ``"token_level_ass_filepath"`` (if ``save_output_file_formats`` contains ``ass``)
+* ``"word_level_ass_filepath"`` (if ``save_output_file_formats`` contains ``ass``)
+* ``"pred_text"`` (if ``align_using_pred_text=true``)
+
+
+How do I evaluate the alignment accuracy?
+-----------------------------------------
+
+Ideally you would have some 'true' CTM files to compare with your generated CTM files. With these you could obtain metrics such as the mean (absolute) errors between predicted starts/ends and the 'true' starts/ends of the segments.
+
+Alternatively (or additionally), you can visualize the quality of alignments using tools such as Gecko, which can play your audio file and display the predicted alignments at the same time. The Gecko tool requires you to upload an audio file and at least one CTM file. The Gecko tool can be accessed here: https://gong-io.github.io/gecko/. More information about the Gecko tool can be found on its Github page here: https://github.com/gong-io/gecko. 
+
+.. note:: 
+	The following may help improve your experience viewing the CTMs in Gecko:
+
+	* setting ``minimum_timestamp_duration`` to a larger number, as Gecko may not display some tokens/words/segments properly if their timestamps are too short.
+	* setting ``remove_blank_tokens_from_ctm=true`` if you are analyzing token-level CTMs, as it will make the Gecko visualization less cluttered.
+

docs/source/tools/speech_data_explorer.rst

@@ -0,0 +1,175 @@
+Speech Data Explorer
+====================
+
+Speech Data Explorer (SDE) is a `Dash <https://plotly.com/dash/>`__-based web application for interactive exploration and analysis of speech datasets.
+
++--------------------------------------------------------------------------------------------------------------------------+
+| **SDE Features:**                                                                                                        |
++--------------------------------------------------------------------------------------------------------------------------+
+| global dataset statistics [alphabet, vocabulary, duration-based histograms, number of hours, number of utterances, etc.] |
++--------------------------------------------------------------------------------------------------------------------------+
+| navigation across the dataset using an interactive datatable that supports sorting and filtering                         |
++--------------------------------------------------------------------------------------------------------------------------+
+| inspection of individual utterances [plotting waveforms, spectrograms, custom attributes, and playing audio]             |
++--------------------------------------------------------------------------------------------------------------------------+
+| error analysis [word error rate (WER), character error rate (CER), word match rate (WMR), word accuracy,                 |
+| display highlighted the difference between the reference text and ASR model prediction]                                  |
++--------------------------------------------------------------------------------------------------------------------------+
+| estimation of audio signal parameters [peak level, frequency bandwidth]                                                  |
++--------------------------------------------------------------------------------------------------------------------------+
+
+Getting Started
+---------------
+SDE could be found in `NeMo/tools/speech_data_explorer <https://github.com/NVIDIA/NeMo/tree/stable/tools/speech_data_explorer>`__.
+
+Please install the SDE requirements:
+
+.. code-block:: bash
+
+    pip install -r tools/speech_data_explorer/requirements.txt
+
+Then run:
+
+.. code-block:: bash
+
+    python tools/speech_data_explorer/data_explorer.py -h
+
+    usage: data_explorer.py [-h] [--vocab VOCAB] [--port PORT] [--disable-caching-metrics] [--estimate-audio-metrics] [--debug] manifest
+
+    Speech Data Explorer
+
+    positional arguments:
+    manifest              path to JSON manifest file
+
+    optional arguments:
+    -h, --help            show this help message and exit
+    --vocab VOCAB         optional vocabulary to highlight OOV words
+    --port PORT           serving port for establishing connection
+    --disable-caching-metrics
+                            disable caching metrics for errors analysis
+    --estimate-audio-metrics, -a
+                            estimate frequency bandwidth and signal level of audio recordings
+    --debug, -d           enable debug mode
+
+
+SDE takes as an input a JSON manifest file (that describes speech datasets in NeMo). It should contain the following fields:
+
+* `audio_filepath` (path to audio file)
+* `duration` (duration of the audio file in seconds)
+* `text` (reference transcript)
+
+SDE supports any extra custom fields in the JSON manifest. If the field is numeric, then SDE can visualize its distribution across utterances.
+
+If the JSON manifest has attribute `pred_text`, SDE interprets it as a predicted ASR transcript and computes error analysis metrics.
+The command line option ``--estimate-audio-metrics`` allows SDE to estimate the signal's peak level and frequency bandwidth for each utterance.
+By default, SDE caches all computed metrics to a pickle file. The caching can be disabled with ``--disable-caching-metrics`` option.
+
+User Interface
+--------------
+
+SDE application has two pages:
+
+* `Statistics` (to display global statistics and aggregated error metrics)
+
+    .. image:: images/sde_base_stats.png
+        :align: center
+        :width: 800px
+        :alt: SDE Statistics
+        
+
+* `Samples` (to allow navigation across the entire dataset and exploration of individual utterances)
+
+    .. image:: images/sde_player.png
+        :align: center
+        :width: 800px
+        :alt: SDE Statistics
+        
+
+Plotly Dash Datatable provides core SDE's interactive features (navigation, filtering, and sorting).
+SDE has two datatables:
+
+* Vocabulary (that shows all words from dataset's reference texts on `Statistics` page)
+
+    .. image:: images/sde_words.png
+        :align: center
+        :width: 800px
+        :alt: Vocabulary
+        
+
+* Data (that visualizes all dataset's utterances on `Samples` page)
+
+    .. image:: images/sde_utterances.png
+        :align: center
+        :width: 800px
+        :alt: Data
+        
+
+Every column of the DataTable has the following interactive features:
+
+* toggling off (by clicking on the `eye` icon in the column's header cell) or on (by clicking on the `Toggle Columns` button below the table)
+
+    .. image:: images/datatable_toggle.png
+        :align: center
+        :width: 800px
+        :alt: Toggling
+        
+
+* sorting (by clicking on small triangle icons in the column's header cell): unordered (two triangles point up and down), ascending (a triangle points up), descending (a triangle points down)
+
+    .. image:: images/datatable_sort.png
+        :align: center
+        :width: 800px
+        :alt: Sorting
+        
+
+* filtering (by entering a filtering expression in a cell below the header's cell): SDE supports ``<``, ``>``, ``<=``, ``>=``, ``=``, ``!=``, and ``contains`` operators; to match a specific substring, the quoted substring can be used as a filtering expression
+
+    .. image:: images/datatable_filter.png
+        :align: center
+        :width: 800px
+        :alt: Filtering
+        
+
+
+Analysis of Speech Datasets
+---------------------------
+
+In the simplest use case, SDE helps to explore a speech dataset interactively and get basic statistics.
+If there is no available pre-trained ASR model to get predicted transcripts, there are still available heuristic rules to spot potential issues in a dataset:
+
+1. Check dataset alphabet (it should contain only target characters)
+2. Check vocabulary for uncommon words (e.g., foreign words, typos). SDE can take an external vocabulary file passed with ``--vocab`` option. Then it is easy to filter out-of-vocabulary (OOV) words in the dataset and sort them by their number of occurrences (count).
+3. Check utterances with a high character rate. A high character rate might indicate that the utterance has more words in the reference transcript than the corresponding audio recording.
+
+If there is a pre-trained ASR model, then the JSON manifest file can be extended with ASR predicted transcripts:
+
+.. code-block:: bash
+
+    python examples/asr/transcribe_speech.py pretrained_name=<ASR_MODEL_NAME> dataset_manifest=<JSON_FILENAME>
+    
+After that it is worth to check words with zero accuracy. 
+
+    .. image:: images/sde_mls_words.png
+        :align: center
+        :width: 800px
+        :alt: MLS Words
+        
+
+And then look at high CER utterances.
+
+    .. image:: images/sde_mls_cer.png
+        :align: center
+        :width: 800px
+        :alt: MLS CER
+        
+
+Listening to the audio recording helps to validate the corresponding reference transcript.
+
+    .. image:: images/sde_mls_player.png
+        :align: center
+        :width: 800px
+        :alt: MLS Player
+        
+
+
+

docs/source/tools/speech_data_processor.rst

@@ -0,0 +1,10 @@
+Speech Data Processor
+=====================
+
+Speech Data Processor (SDP) is a toolkit to make it easy to:
+  1. write code to process a new dataset, minimizing the amount of boilerplate code required.
+  2. share the steps for processing a speech dataset.
+
+SDP is hosted here: https://github.com/NVIDIA/NeMo-speech-data-processor.
+
+To learn more about SDP, please check the [documentation](https://nvidia.github.io/NeMo-speech-data-processor/).

docs/source/tools/tools_all.bib

@@ -0,0 +1,8 @@
+@inproceedings{kurzinger2020ctc,
+  title={CTC-segmentation of large corpora for german end-to-end speech recognition},
+  author={K{\"u}rzinger, Ludwig and Winkelbauer, Dominik and Li, Lujun and Watzel, Tobias and Rigoll, Gerhard},
+  booktitle={International Conference on Speech and Computer},
+  pages={267--278},
+  year={2020},
+  organization={Springer}
+}