chore: import upstream snapshot with attribution

docs/source/asr/asr_customization/legacy_language_modeling_and_customization.rst

@@ -0,0 +1,297 @@
+.. _ngram_modeling:
+
+****************************
+N-gram Language Model Fusion
+****************************
+
+In this approach, an N-gram LM is trained on text data, then it is used in fusion with beam search decoding to find the
+best candidates. The beam search decoders in NeMo support language models trained with KenLM library (
+`https://github.com/kpu/kenlm <https://github.com/kpu/kenlm>`__).
+The beam search decoders and KenLM library are not installed by default in NeMo. 
+You need to install them to be able to use beam search decoding and N-gram LM.
+Please refer to `scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh>`__
+on how to install them. Alternatively, you can build Docker image
+`scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__ with all the necessary dependencies.
+
+Please, refer to :ref:`train-ngram-lm` for more details on how to train an N-gram LM using KenLM library.
+
+NeMo supports both character-based and BPE-based models for N-gram LMs. An N-gram LM can be used with beam search
+decoders on top of the ASR models to produce more accurate candidates. The beam search decoder would incorporate
+the scores produced by the N-gram LM into its score calculations as the following:
+
+.. code-block::
+
+    final_score = acoustic_score + beam_alpha*lm_score + beam_beta*seq_length
+
+where acoustic_score is the score predicted by the acoustic encoder and lm_score is the one estimated by the LM.
+The parameter 'beam_alpha' determines the weight given to the N-gram language model, while 'beam_beta' is a penalty term that accounts for sequence length in the scores. A larger 'beam_alpha' places more emphasis on the language model and less on the acoustic model. Negative values for 'beam_beta' penalize longer sequences, encouraging the decoder to prefer shorter predictions. Conversely, positive values for 'beam_beta' favor longer candidates.
+
+Evaluate by Beam Search Decoding and N-gram LM
+==============================================
+
+NeMo's beam search decoders are capable of using the KenLM's N-gram models to find the best candidates.
+The script to evaluate an ASR model with beam search decoding and N-gram models can be found at
+`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__.
+
+This script has a large number of possible argument overrides; therefore, it is recommended that you use ``python eval_beamsearch_ngram_ctc.py --help`` to see the full list of arguments.
+
+You can evaluate an ASR model using the following:
+
+.. code-block::
+
+    python eval_beamsearch_ngram_ctc.py nemo_model_file=<path to the .nemo file of the model> \
+           input_manifest=<path to the evaluation JSON manifest file \
+           kenlm_model_file=<path to the binary KenLM model> \
+           beam_width=[<list of the beam widths, separated with commas>] \
+           beam_alpha=[<list of the beam alphas, separated with commas>] \
+           beam_beta=[<list of the beam betas, separated with commas>] \
+           preds_output_folder=<optional folder to store the predictions> \
+           probs_cache_file=null \
+           decoding_mode=beamsearch_ngram \
+           decoding_strategy="<Beam library such as beam, pyctcdecode or flashlight>"
+
+It can evaluate a model in the following three modes by setting the argument ``--decoding_mode``:
+
+*  greedy: Just greedy decoding is done and no beam search decoding is performed.
+*  beamsearch: The beam search decoding is done, but without using the N-gram language model. Final results are equivalent to setting the weight of LM (beam_beta) to zero.
+*  beamsearch_ngram: The beam search decoding is done with N-gram LM.
+
+In ``beamsearch`` mode, the evaluation is performed using beam search decoding without any language model. The performance is reported in terms of Word Error Rate (WER) and Character Error Rate (CER). Moreover, when the best candidate is selected among the candidates, it is also reported as the best WER/CER. This can serve as an indicator of the quality of the predicted candidates.
+
+
+The script initially loads the ASR model and predicts the outputs of the model's encoder as log probabilities. This part is computed in batches on a device specified by --device, which can be either a CPU (`--device=cpu`) or a single GPU (`--device=cuda:0`).
+The batch size for this part is specified by ``--acoustic_batch_size``. Using the largest feasible batch size can speed up the calculation of log probabilities. Additionally, you can use `--use_amp` to accelerate the calculation and allow for larger --acoustic_batch_size values.
+Currently, multi-GPU support is not available for calculating log probabilities. However, using ``--probs_cache_file`` can help. This option stores the log probabilities produced by the model's encoder in a pickle file, allowing you to skip the first step in future runs.
+
+The following is the list of the important arguments for the evaluation script:
+
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| **Argument**                         | **Type** | **Default**      | **Description**                                                         |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| nemo_model_file                      | str      | Required         | The path of the `.nemo` file of the ASR model to extract the tokenizer. |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| input_manifest                       | str      | Required         | Path to the training file, it can be a text file or JSON manifest.      |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| kenlm_model_file                     | str      | Required         | The path to store the KenLM binary model file.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| preds_output_folder                  | str      | None             | The path to an optional folder to store the predictions.                |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| probs_cache_file                     | str      | None             | The cache file for storing the outputs of the model.                    |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| acoustic_batch_size                  | int      | 16               | The batch size to calculate log probabilities.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| use_amp                              | bool     | False            | Whether to use AMP if available to calculate log probabilities.         |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| device                               | str      | cuda             | The device to load the model onto to calculate log probabilities.       |
+|                                      |          |                  | It can `cpu`, `cuda`, `cuda:0`, `cuda:1`, ...                           |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| decoding_mode                        | str      | beamsearch_ngram | The decoding scheme to be used for evaluation.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_width                           | float    | Required         | List of the width or list of the widths of the beam search decoding.    |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_alpha                           | float    | Required         | List of the alpha parameter for the beam search decoding.               |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_beta                            | float    | Required         | List of the beta parameter for the beam search decoding.                |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_batch_size                      | int      | 128              | The batch size to be used for beam search decoding.                     |
+|                                      |          |                  | Larger batch size can be a little faster, but uses larger memory.       |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| decoding_strategy                    | str      | beam             | String argument for type of decoding strategy for the model.            |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| decoding                             | Dict     | BeamCTC          | Subdict of beam search configs. Values found via                        |
+|                                      | Config   | InferConfig      | python eval_beamsearch_ngram_ctc.py --help                              |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.do_lowercase         | bool     | ``False``        | Whether to make the training text all lower case.                       |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.punctuation_marks    | str      | ``""``           | String with punctuation marks to process. Example: ".\,?"               |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.rm_punctuation       |  bool    | ``False``        | Whether to remove punctuation marks from text.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.separate_punctuation | bool     | ``True``         | Whether to separate punctuation with the previous word by space.        |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+
+The width of the beam search (``--beam_width``) specifies the number of top candidates or predictions the beam search decoder will consider. Larger beam widths result in more accurate but slower predictions.
+
+.. note::
+
+    The ``eval_beamsearch_ngram_ctc.py`` script contains the entire subconfig used for CTC Beam Decoding.
+    Therefore it is possible to forward arguments for various beam search libraries such as ``flashlight``
+    and ``pyctcdecode`` via the ``decoding`` subconfig.
+
+To learn more about evaluating the ASR models with N-gram LM, refer to the tutorial here: Offline ASR Inference with Beam Search and External Language Model Rescoring
+`Offline ASR Inference with Beam Search and External Language Model Rescoring <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/Offline_ASR.ipynb>`_
+
+Beam Search Engines
+-------------------
+
+NeMo ASR CTC supports multiple beam search engines for decoding. The default engine is beam, which is the OpenSeq2Seq decoding library.
+
+OpenSeq2Seq (``beam``)
+~~~~~~~~~~~~~~~~~~~~~~
+
+CPU-based beam search engine that is quite efficient and supports char and subword models. It requires a character/subword
+KenLM model to be provided.
+
+The config for this decoding library is described above.
+
+Flashlight (``flashlight``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Flashlight is a C++ library for ASR decoding provided at `https://github.com/flashlight/flashlight <https://github.com/flashlight/flashlight>`_. It is a CPU- and CUDA-based beam search engine that is quite efficient and supports char and subword models. It requires an ARPA KenLM file.
+
+It supports several advanced features, such as lexicon-based decoding, lexicon-free decoding, beam pruning threshold, and more.
+
+.. code-block:: python
+
+    @dataclass
+    class FlashlightConfig:
+        lexicon_path: Optional[str] = None
+        boost_path: Optional[str] = None
+        beam_size_token: int = 16
+        beam_threshold: float = 20.0
+        unk_weight: float = -math.inf
+        sil_weight: float = 0.0
+
+.. code-block::
+
+    # Lexicon-based decoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="flashlight" \
+           decoding.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon' \
+           decoding.beam.flashlight_cfg.beam_size_token = 32 \
+           decoding.beam.flashlight_cfg.beam_threshold = 25.0
+
+    # Lexicon-free decoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="flashlight" \
+           decoding.beam.flashlight_cfg.beam_size_token = 32 \
+           decoding.beam.flashlight_cfg.beam_threshold = 25.0
+
+PyCTCDecode (``pyctcdecode``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+PyCTCDecode is a Python library for ASR decoding provided at `https://github.com/kensho-technologies/pyctcdecode <https://github.com/kensho-technologies/pyctcdecode>`_. It is a CPU-based beam search engine that is somewhat efficient for a pure Python library, and supports char and subword models. It requires a character/subword KenLM ARPA / BINARY model to be provided.
+
+
+It has advanced features, such as word boosting, which can be useful for transcript customization.
+
+.. code-block:: python
+
+   @dataclass
+    class PyCTCDecodeConfig:
+        beam_prune_logp: float = -10.0
+        token_min_logp: float = -5.0
+        prune_history: bool = False
+        hotwords: Optional[List[str]] = None
+        hotword_weight: float = 10.0
+
+.. code-block::
+
+    # PyCTCDecoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="pyctcdecode" \
+           decoding.beam.pyctcdecode_cfg.beam_prune_logp = -10. \
+           decoding.beam.pyctcdecode_cfg.token_min_logp = -5. \
+           decoding.beam.pyctcdecode_cfg.hotwords=[<List of str words>] \
+           decoding.beam.pyctcdecode_cfg.hotword_weight=10.0
+
+
+Hyperparameter Grid Search
+--------------------------
+
+Beam search decoding with N-gram LM has three main hyperparameters: `beam_width`, `beam_alpha`, and `beam_beta`.
+The accuracy of the model is dependent on the values of these parameters, specifically, beam_alpha and beam_beta. To perform grid search, you can specify a single value or a list of values for each of these parameters. In this case, it would perform the beam search decoding on all combinations of the three hyperparameters.
+For example, the following set of parameters would result in 212=4 beam search decodings:
+
+.. code-block::
+
+    python eval_beamsearch_ngram_ctc.py ... \
+                        beam_width=[64,128] \
+                        beam_alpha=[1.0] \
+                        beam_beta=[1.0,0.5]
+
+
+Beam Search ngram Decoding for Transducer Models (RNNT and HAT)
+===============================================================
+
+You can also find a similar script to evaluate an RNNT/HAT model with beam search decoding and N-gram models at:
+`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_transducer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_transducer.py>`_
+
+.. code-block::
+
+    python eval_beamsearch_ngram_transducer.py nemo_model_file=<path to the .nemo file of the model> \
+            input_manifest=<path to the evaluation JSON manifest file \
+            kenlm_model_file=<path to the binary KenLM model> \
+            beam_width=[<list of the beam widths, separated with commas>] \
+            beam_alpha=[<list of the beam alphas, separated with commas>] \
+            preds_output_folder=<optional folder to store the predictions> \
+            probs_cache_file=null \
+            decoding_strategy=<greedy_batch or maes decoding>
+            maes_prefix_alpha=[<list of the maes prefix alphas, separated with commas>] \
+            maes_expansion_gamma=[<list of the maes expansion gammas, separated with commas>] \
+            hat_subtract_ilm=<in case of HAT model: subtract internal LM or not (True/False)> \
+            hat_ilm_weight=[<in case of HAT model: list of the HAT internal LM weights, separated with commas>] \
+
+
+.. _wfst-ctc-decoding:
+
+WFST CTC decoding
+=================
+Weighted Finite-State Transducers (WFST) are finite-state machines with input and output symbols on each transition and some weight element of a semiring. WFSTs can act as N-gram LMs in a special type of LM-forced beam search, called WFST decoding.
+
+.. note::
+
+    More precisely, WFST decoding is more of a greedy N-depth search with LM.
+    Thus, it is asymptotically worse than conventional beam search decoding algorithms, but faster.
+
+**WARNING**  
+At the moment, NeMo supports WFST decoding only for CTC models and word-based LMs.
+
+To run WFST decoding in NeMo, one needs to provide a NeMo ASR model and either an ARPA LM or a WFST LM (advanced). An ARPA LM can be built from source text with KenLM as follows: ``<kenlm_bin_path>/lmplz -o <ngram_length> --arpa <out_arpa_path> --prune <ngram_prune>``.
+
+The script to evaluate an ASR model with WFST decoding and N-gram models can be found at
+`scripts/asr_language_modeling/ngram_lm/eval_wfst_decoding_ctc.py
+<https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_wfst_decoding_ctc.py>`__.
+
+This script has a large number of possible argument overrides, therefore it is advised to use ``python eval_wfst_decoding_ctc.py --help`` to see the full list of arguments.
+
+You may evaluate an ASR model as the following:
+
+.. code-block::
+
+    python eval_wfst_decoding_ctc.py nemo_model_file=<path to the .nemo file of the model> \
+           input_manifest=<path to the evaluation JSON manifest file> \
+           arpa_model_file=<path to the ARPA LM model> \
+           decoding_wfst_file=<path to the decoding WFST file> \
+           beam_width=[<list of the beam widths, separated with commas>] \
+           lm_weight=[<list of the LM weight multipliers, separated with commas>] \
+           open_vocabulary_decoding=<whether to use open vocabulary mode for WFST decoding> \
+           decoding_mode=<decoding mode, affects output. Usually "nbest"> \
+           decoding_search_type=<WFST decoding library. Usually "riva"> \
+           preds_output_folder=<optional folder to store the predictions> \
+           probs_cache_file=null
+
+.. note::
+
+    Since WFST decoding is LM-forced (the search goes over the WIDEST graph), only word sequences accepted by the WFST can appear in the decoding results.
+    To circumvent this restriction, one can pass ``open_vocabulary_decoding=true`` (experimental feature).
+
+
+Quick start example
+-------------------
+
+.. code-block::
+
+    wget -O - https://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz | \
+    gunzip -c | tr '[:upper:]' '[:lower:]' > 3-gram.pruned.1e-7.arpa && \
+    python eval_wfst_decoding_ctc.py nemo_model_file="stt_en_conformer_ctc_small_ls" \
+           input_manifest="<data_dir>/Librispeech/test_other.json" \
+           arpa_model_file="3-gram.pruned.1e-7.arpa" \
+           decoding_wfst_file="3-gram.pruned.1e-7.fst" \
+           beam_width=[8] \
+           lm_weight=[0.5,0.6,0.7,0.8,0.9]
+
+.. note::
+
+    Building a decoding WFST is a long process, so it is better to provide a ``decoding_wfst_file`` path even if you don't have it.
+    This way, the decoding WFST will be buffered to the specified file path and there will be no need to re-build it on the next run.

docs/source/asr/asr_customization/neural_rescoring.rst

@@ -0,0 +1,105 @@
+.. _neural_rescoring:
+
+****************
+Neural Rescoring
+****************
+
+When using the neural rescoring approach, a neural network is used to score candidates. A candidate is the text transcript predicted by the ASR model's decoder. The top K candidates produced by beam search decoding (with a beam width of K) are given to a neural language model for ranking. The language model assigns a score to each candidate, which is usually combined with the scores from beam search decoding to produce the final scores and rankings.
+
+Train Neural Rescorer
+=====================
+
+An example script to train such a language model with Transformer can be found at `examples/nlp/language_modeling/transformer_lm.py <https://github.com/NVIDIA/NeMo/blob/stable/examples/nlp/language_modeling/transformer_lm.py>`__.
+It trains a ``TransformerLMModel`` which can be used as a neural rescorer for an ASR system. For more information on language models training, see LLM/NLP documentation.
+
+
+You can also use a pretrained language model from the Hugging Face library, such as Transformer-XL and GPT, instead of training your model.
+Models like BERT and RoBERTa are not supported by this script because they are trained as Masked Language Models. As a result, they are not efficient or effective for scoring sentences out of the box.
+
+
+Evaluation
+==========
+
+Given a trained TransformerLMModel `.nemo` file or a pretrained HF model, the script available at
+`scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__
+can be used to re-score beams obtained with ASR model. You need the `.tsv` file containing the candidates produced
+by the acoustic model and the beam search decoding to use this script. The candidates can be the result of just the beam
+search decoding or the result of fusion with an N-gram LM. You can generate this file by specifying `--preds_output_folder` for
+`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__.
+
+The neural rescorer would rescore the beams/candidates by using two parameters of `rescorer_alpha` and `rescorer_beta`, as follows:
+
+.. code-block::
+
+    final_score = beam_search_score + rescorer_alpha*neural_rescorer_score + rescorer_beta*seq_length
+
+The parameter `rescorer_alpha` specifies the importance placed on the neural rescorer model, while `rescorer_beta` is a penalty term that accounts for sequence length in the scores. These parameters have similar effects to `beam_alpha` and `beam_beta` in the beam search decoder and N-gram language model.
+
+Use the following steps to evaluate a neural LM:
+
+#. Obtain `.tsv` file with beams and their corresponding scores. Scores can be from a regular beam search decoder or
+   in fusion with an N-gram LM scores. For a given beam size `beam_size` and a number of examples
+   for evaluation `num_eval_examples`, it should contain (`num_eval_examples` x `beam_size`) lines of
+   form `beam_candidate_text \t score`. This file can be generated by `scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__
+
+#. Rescore the candidates by `scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__.
+
+.. code-block::
+
+    python eval_neural_rescorer.py
+        --lm_model=[path to .nemo file of the LM or the name of a HF pretrained model]
+        --beams_file=[path to beams .tsv file]
+        --beam_size=[size of the beams]
+        --eval_manifest=[path to eval manifest .json file]
+        --batch_size=[batch size used for inference on the LM model]
+        --alpha=[the value for the parameter rescorer_alpha]
+        --beta=[the value for the parameter rescorer_beta]
+        --scores_output_file=[the optional path to store the rescored candidates]
+
+The candidates, along with their new scores, are stored at the file specified by `--scores_output_file`.
+
+The following is the list of the arguments for the evaluation script:
+
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| **Argument**        |**Type**| **Default**      | **Description**                                                         |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| lm_model            | str    | Required         | The path of the '.nemo' file of an ASR model, or the name of a          |
+|                     |        |                  | Hugging Face pretrained model like 'transfo-xl-wt103' or 'gpt2'.        |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| eval_manifest       | str    | Required         | Path to the evaluation manifest file (.json manifest file).             |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| beams_file          | str    | Required         | Path to beams file (.tsv) containing the candidates and their scores.   |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| beam_size           | int    | Required         | The width of the beams (number of candidates) generated by the decoder. |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| alpha               | float  | None             | The value for parameter rescorer_alpha                                  |
+|                     |        |                  | Not passing value would enable linear search for rescorer_alpha.        |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| beta                | float  | None             | The value for parameter rescorer_beta                                   |
+|                     |        |                  | Not passing value would enable linear search for rescorer_beta.         |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| batch_size          | int    | 16               | The batch size used to calculate the scores.                            |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| max_seq_length      | int    | 512              | Maximum sequence length (in tokens) for the input.                      |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| scores_output_file  | str    | None             | The optional file to store the rescored beams.                          |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| use_amp             | bool   | ``False``        | Whether to use AMP if available calculate the scores.                   |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| device              | str    | cuda             | The device to load LM model onto to calculate the scores                |
+|                     |        |                  | It can be 'cpu', 'cuda', 'cuda:0', 'cuda:1', ...                        |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+
+
+Hyperparameter Linear Search
+----------------------------
+
+The hyperparameter linear search script also supports linear search for parameters `alpha` and `beta`. If any of the two is not
+provided, a linear search is performed to find the best value for that parameter. When linear search is used, initially
+`beta` is set to zero and the best value for `alpha` is found, then `alpha` is fixed with
+that value and another linear search is done to find the best value for `beta`.
+If any of the of these two parameters is already specified, then search for that one is skipped. After each search for a
+parameter, the plot of WER% for different values of the parameter is also shown.
+
+It is recommended to first use the linear search for both parameters on a validation set by not providing any values for `--alpha` and `--beta`.
+Then check the WER curves and decide on the best values for each parameter. Finally, evaluate the best values on the test set.

docs/source/asr/asr_customization/ngpulm_language_modeling_and_customization.rst

@@ -0,0 +1,346 @@
+.. _ngpulm_ngram_modeling:
+
+***************************************************************
+NGPU-LM (GPU-based N-gram Language Model) Language Model Fusion
+***************************************************************
+
+ASR systems can achieve significantly improved accuracy by leveraging **external language model (LM) shallow fusion** during the decoding process. 
+This technique integrates knowledge from an external LM without requiring the ASR model itself to be retrained.
+
+**How Shallow Fusion Works:**
+
+During shallow fusion, the output probabilities generated by the ASR model are combined with those from a separate, external language model.
+The final transcription is then determined by selecting the word sequence that yields the highest combined score. 
+These external LMs are typically trained on vast text datasets, allowing them to capture the statistical patterns, syntactic structures, and contextual dependencies of language. 
+This enables them to predict more plausible word sequences, thereby correcting potential errors from the ASR model.
+
+**Domain Adaptation Benefits:**
+
+Shallow fusion is particularly valuable for **adapting ASR systems to new or specialized domains**.
+By training the external LM on domain-specific text-such as medical, legal, or technical documents-it learns the vocabulary of that field.
+This specialized knowledge guides the ASR decoding process towards more accurate and contextually relevant transcriptions.
+
+Traditionally, shallow fusion has been performed during **beam search decoding**, a method that explores multiple promising hypotheses to find the most likely transcription.
+
+
+NGPU-LM
+=======
+
+A widely used library for training traditional n-gram language models is KenLM. 
+While KenLM (https://github.com/kpu/kenlm) is known for its efficient CPU-based implementation, its reliance on the CPU can limit performance in high-throughput scenarios, especially when dealing with large-scale data.
+
+NGPU-LM on contrast is a GPU-accelerated implementation of a statistical n-gram language model. 
+It uses a **universal trie-based data structure**, which enables fast, batched queries. For full details, please refer to the paper [ngpulm]_.
+
+This enables shallow fusion during **greedy decoding**, creating a middle ground between standard greedy decoding and full beam search with a language model.  
+It preserves the speed and simplicity of greedy decoding while regaining much of the accuracy typically achieved with beam search with external LM fusion.
+While not as accurate as full beam search, greedy decoding with NGPU-LM fusion offers a compelling balance between speed and accuracy.
+
+NeMo provides efficient, fully GPU-based beam search implementations for all major ASR model types, 
+allowing **beam decoding to operate with real-time factors (RTFx) close to those of greedy decoding**.
+At a batch size of 32, the RTFx difference between beam and greedy decoding is only about 20%.
+These implementations incorporate NGPU-LM, enabling fast, fully GPU-based decoding and customization.  
+This enables users to customize decoding while maintaining reasonable speed, even in beam search mode.
+For full details, please refer to the [beamsearch]_.
+
+NGPU-LM fusion is supported for BPE-based ASR models (CTC, RNNT, TDT, AED) during both greedy and beam decoding.
+
+Train NGPU-LM
+=============
+
+NGPU-LM is built using `.ARPA` files generated by the KenLM library. You can train an n-gram LM using the following script:  
+`train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
+
+The generated `.ARPA` files can be directly used for GPU-based decoding.  
+However, for faster performance, it is recommended to convert the model to the `.nemo` format by setting the ``save_nemo`` flag to ``true``.
+
+.. code-block::
+
+    python train_kenlm.py nemo_model_file=<path to the .nemo file of the model> \
+                              train_paths=<list of paths to the training text or JSON manifest files> \
+                              kenlm_bin_path=<path to the bin folder of KenLM library> \
+                              kenlm_model_file=<path to store the binary KenLM model> \
+                              ngram_length=<order of N-gram model> \
+                              preserve_arpa=true \
+                              save_nemo=True
+
+For a complete list of arguments and usage details, refer to the :ref:`train-ngram-lm`.
+
+
+.. note::
+
+    It is recommended that you use 6 as the order of the N-gram model for BPE-based models. Higher orders may require re-compiling KenLM to support them.
+
+.. _ctc-decoding-with-ngpulm:
+
+Decoding with NGPU-LM
+=====================
+
+To run inference with NGPU-LM fusion, the ``ngram_lm_model`` and ``ngram_lm_alpha`` fields must be specified in the decoding configuration.
+
+.. note::
+
+    For CTC, RNNT, and TDT models, these fields should be set within the respective ``greedy`` or ``beam`` sub-configurations.  
+    For AED models running in greedy mode, set the beam size to 1 and specify these fields under the ``beam`` sub-configuration.
+
+Examples for different model types are provided below.
+
+
+
+CTC Decoding with NGPU-LM
+-------------------------
+
+**Greedy Search:**
+
+You can run NGPU-LM shallow fusion during greedy CTC decoding using the following command:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-ctc-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        ctc_decoding.greedy.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        ctc_decoding.greedy.ngram_lm_alpha=0.2 \
+        ctc_decoding.greedy.allow_cuda_graphs=True \
+        ctc_decoding.strategy="greedy_batch"
+
+**Beam Search:**
+
+During CTC beam search, each hypothesis is scored using the following formula:
+
+.. code-block::
+
+    final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length
+
+where:
+    - ``acoustic_score`` is the score predicted by the ASR.
+    - ``lm_score`` is the score predicted by the NGPU-LM LM.
+    - ``ngram_lm_alpha`` is the weight given to the language model.
+    - ``beam_beta`` is a penalty term that accounts for sequence length in the scores.
+
+For running fully batched GPU-based CTC decoding with NGPU-LM, you can use the following command:
+
+The following is the list of the adjustable arguments of batched CTC decoding algorithm ``beam_batch``:
+
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| **Argument**     | **Type**  | **Default** | **Description**                                                                                                                |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_lm_alpha   | float     | Required    | Weight factor applied to the language model scores.                                                                            |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_size        | int       | 4           | Beam size.                                                                                                                     |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_beta        | float     | 1           | Penalty applied to word insertions to control the trade-off between insertion and deletion errors during beam search decoding. |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_threshold   | float     | 20          | Threshold used to prune candidate hypotheses by comparing their scores to the best hypothesis.                                 |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-ctc-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        ctc_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        ctc_decoding.beam.ngram_lm_alpha=0.2 \
+        ctc_decoding.beam.beam_size=12 \
+        ctc_decoding.beam.beam_beta=1.0 \
+        ctc_decoding.strategy="beam_batch" \
+        ctc_decoding.beam.allow_cuda_graphs=True
+
+
+RNN-T/TDT decoding with NGPU-LM
+-------------------------------
+
+**Greedy Search:**
+
+You can run NGPU-LM shallow fusion during greedy RNN-T / TDT decoding using the following command:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-rnnt-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        rnnt_decoding.greedy.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        rnnt_decoding.greedy.ngram_lm_alpha=0.2 \
+        rnnt_decoding.greedy.allow_cuda_graphs=True \
+        rnnt_decoding.strategy="greedy_batch"
+
+.. note::
+
+    To run the inference with TDT model, you need to provide pretrained TDT model in ``pretrained_name`` field (for example ``nvidia/parakeet-tdt_ctc-1.1b`` ).
+
+
+**Beam Search:**
+
+During RNN-T / TDT beam search, each hypothesis is scored using the following formula:
+
+.. code-block::
+
+    final_score = acoustic_score + ngram_lm_alpha * lm_score
+
+where:
+    - ``acoustic_score`` is the score predicted by the ASR.
+    - ``lm_score`` is the score predicted by the NGPU-LM LM.
+    - ``ngram_lm_alpha`` is the weight given to the language model.
+
+Final hypotheses is chosen based on the normalized score ``final_score / seq_length``.
+
+
+*Blank Scoring in Transducer Models*
+
+Transducer models include a blank symbol (``∅``) for frame transitions, while LMs do not model blanks.  
+During shallow fusion, the LM is typically applied only to non-blank tokens:
+
+.. math::
+
+    \ln p_{\text{tot}}[k] =
+    \begin{cases}
+      \ln p[k] + \lambda \ln p_{\text{LM}}[k], & k \in V \\
+      \ln p[\emptyset], & k = \emptyset
+    \end{cases}
+
+This can lead to excessive blank predictions at higher LM weights, increasing deletion errors.  
+NeMo supports a blank-aware scoring method that adjusts LM contributions to better balance predictions:
+
+.. math::
+
+    \ln p_{\text{tot}}[k] =
+    \begin{cases}
+      \ln p[k] + \lambda \ln((1 - p[\emptyset]) \cdot p_{\text{LM}}[k]), & k \in V \\
+      (1 + \lambda) \ln p[\emptyset], & k = \emptyset
+    \end{cases}
+
+*Early vs. Late Pruning*
+
+In shallow fusion, LM and ASR scores can be combined at different stages:
+
+- **Early pruning:** ASR selects top hypotheses, then LM rescoring is applied. Efficient for small beams.
+- **Late pruning:** ASR and LM scores are combined before pruning. More accurate but requires full-vocab LM queries.
+
+For Transducer models, late pruning with the blank-aware scoring method generally yields better performance than the standard approach.
+
+*Beam Search Strategies:*
+
+In NeMo fully batched implementation of following strategies are supported:
+
+- **malsd_batch:** fully batched implemention of modified Alignment-Length Synchronous Decoding [alsd]_, supporting both RNNT and TDT models.
+- **maes_batch:** fully batched implemention of modified Adaptive Expansion Search [aes]_, supporting for only RNNT models. CudaGraphs are not supported.
+
+The following is the list of the adjustable arguments of batched CTC decoding algorithm ``beam_batch``:
+
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| **Argument**          | **Type**  | **Strategy**            | **Default**      | **Description**                                                                                                                |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_lm_alpha        | float     | malsd_batch, maes_batch | Required         | Weight factor applied to the language model scores.                                                                            |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_size             | int       | malsd_batch, maes_batch | 4                | Beam size.                                                                                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| pruning_mode          | str       | malsd_batch, maes_batch | late             | Mode for hypotheses pruning. Can be ``early`` or ``late``.                                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| blank_lm_score_mode   | str       | malsd_batch, maes_batch | lm_weighted_full | Mode for blank symbol scoring. Can be ``no_score`` or ``lm_weighted_full``                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| max_symbols_per_step  | int       | malsd_batch             | 10               | Max symbols to emit on each step to avoid infinite looping.                                                                    |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| maes_num_step         | int       | maes_batch              | 2                | Number of adaptive steps to take.                                                                                              |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| maes_expansion_beta   | float     | maes_batch              | 1.0              | Maximum number of prefix expansions allowed, in addition to the beam size.                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| maes_expansion_gamma  | float     | maes_batch              | 2.3              | Threshold used to prune candidate hypotheses by comparing their scores to the best hypothesis.                                 |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+
+You can run NGPU-LM shallow fusion during beam RNN-T / TDT decoding using the following command:
+
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-rnnt-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        rnnt_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        rnnt_decoding.beam.ngram_lm_alpha=0.2 \
+        rnnt_decoding.beam.beam_size=12 \
+        rnnt_decoding.beam.pruning_mode="late" \
+        rnnt_decoding.beam.blank_lm_score_mode="lm_weighted_full" \
+        rnnt_decoding.beam.allow_cuda_graphs=True \
+        rnnt_decoding.strategy="malsd_batch"
+
+.. note::
+
+    To run the inference with TDT model, you need to provide pretrained TDT model in ``pretrained_name`` field (for example ``nvidia/parakeet-tdt_ctc-1.1b`` ).
+
+AED Decoding with NGPU-LM
+-------------------------
+
+**Beam Search:**
+
+You can run NGPU-LM shallow fusion during greedy CTC decoding using the following command:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+    pretrained_name="nvidia/canary-1b" \
+    amp=false \
+    amp_dtype=bfloat16 \
+    matmul_precision=high \
+    compute_dtype=bfloat16 \
+    presort_manifest=true \
+    cuda=0 \
+    batch_size=32 \
+    dataset_manifest=<dataset_manifest>  \
+    multitask_decoding.beam.beam_size=4 \
+    multitask_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+    multitask_decoding.beam.ngram_lm_alpha=0.2 \
+    multitask_decoding.strategy="beam"
+
+.. note::
+
+    For greedy decoding with NGPU-LM, use beam search with beam_size=1.
+
+
+References
+==========
+
+.. [ngpulm] V. Bataev, A. Andrusenko, L. Grigoryan, A. Laptev, V. Lavrukhin, and B. Ginsburg.  
+   *NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding*.  
+   arXiv:2505.22857, 2025. Available at: https://arxiv.org/abs/2505.22857
+
+.. [beamsearch] L. Grigoryan, V. Bataev, A. Andrusenko, H. Xu, V. Lavrukhin, and B. Ginsburg.  
+   *Pushing the Limits of Beam Search Decoding for Transducer-based ASR Models*.  
+   arXiv:2506.00185, 2025. Available at: https://arxiv.org/abs/2506.00185
+
+.. [alsd] G. Saon, Z. Tüske, and K. Audhkhasi.  
+   *Alignment-Length Synchronous Decoding for RNN Transducer*.  
+   In: ICASSP 2020 – IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7804–7808, 2020.  
+   doi: https://doi.org/10.1109/ICASSP40776.2020.9053040
+
+.. [aes] J. Kim, Y. Lee, and E. Kim.  
+   *Accelerating RNN Transducer Inference via Adaptive Expansion Search*.  
+   IEEE Signal Processing Letters, vol. 27, pp. 2019–2023, 2020.  
+   doi: https://doi.org/10.1109/LSP.2020.3036335

docs/source/asr/asr_customization/ngram_utils.rst

@@ -0,0 +1,151 @@
+.. _ngram-utils:
+
+Scripts for building and merging N-gram Language Models
+=======================================================
+
+.. _train-ngram-lm:
+
+Train N-gram LM
+===============
+
+NeMo utilizes the KenLM library (`https://github.com/kpu/kenlm`) for building efficient n-gram language models.
+
+.. note::
+
+    KenLM is not installed by default in NeMo.  
+    Please see the installation instructions in the script:  
+    `scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh>`__.
+
+    Alternatively, you can build a Docker image with all required dependencies using:  
+    `scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__.
+
+The script for training an n-gram language model with KenLM is available here:  
+`scripts/asr_language_modeling/ngram_lm/train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
+
+This script supports training n-gram LMs on both character-level and BPE-level encodings, which are automatically detected from the model type. The resulting language models can then be used with beam search decoders integrated on top of ASR models.
+
+You can train an n-gram model using the following command:
+
+.. code-block::
+
+    python train_kenlm.py nemo_model_file=<path to the .nemo file of the model> \
+                              train_paths=<list of paths to the training text or JSON manifest files> \
+                              kenlm_bin_path=<path to the bin folder of KenLM library> \
+                              kenlm_model_file=<path to store the binary KenLM model> \
+                              ngram_length=<order of N-gram model> \
+                              preserve_arpa=true
+
+The `train_paths` parameter allows for various input types, such as a list of text files, JSON manifests, or directories, to be used as the training data.
+If the file's extension is anything other than `.json`, it assumes that data format is plain text. For plain text format, each line should contain one
+sample. For the JSON manifests, the file must contain JSON-formatted samples per each line like this:
+
+.. code-block::
+
+    {"audio_filepath": "/data_path/file1.wav", "text": "The transcript of the audio file."}
+
+This code extracts the `text` field from each line to create the training text file. After the N-gram model is trained, it is stored at the path specified by `kenlm_model_file`.
+
+The following is the list of the arguments for the training script:
+
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| **Argument**     | **Type**  | **Default** | **Description**                                                                                                                |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| nemo_model_file  | str       | Required    | The path to `.nemo` file of the ASR model, or name of a pretrained NeMo model to extract a tokenizer.                          |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| train_paths      | List[str] | Required    | List of training files or folders. Files can be a plain text file or ".json" manifest or ".json.gz".                           |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| kenlm_model_file | str       | Required    | The path to store the KenLM binary model file.                                                                                 |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| kenlm_bin_path   | str       | Required    | The path to the bin folder of KenLM. It is a folder named `bin` under where KenLM is installed.                                |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_length**   | int       | Required    | Specifies order of N-gram LM.                                                                                                  |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_prune      | List[int] | [0]         | List of thresholds to prune N-grams. Example: [0,0,1]. See Pruning section on the https://kheafield.com/code/kenlm/estimation  |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| cache_path       | str       | ``""``      | Cache path to save tokenized files.                                                                                            |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| preserve_arpa    | bool      | ``False``   | Whether to preserve the intermediate ARPA file after construction of the BIN file.                                             |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| verbose          | int       | 1           | Verbose level.                                                                                                                 |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| save_nemo        | bool      | ``False``   | Whether to save LM in .nemo format.                                                                                            |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+
+..note::
+It is recommended that you use 6 as the order of the N-gram model for BPE-based models. Higher orders may require re-compiling KenLM to support them.
+
+
+Combine N-gram Language Models
+==============================
+
+Before combining N-gram LMs, install the required OpenGrm NGram library using `scripts/installers/install_opengrm.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/install_opengrm.sh>`__.
+Alternatively, you can use Docker image `scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__ with all the necessary dependencies.
+
+Alternatively, you can use the Docker image at:
+`scripts/asr_language_modeling/ngram_lm/ngram_merge.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/ngram_merge.py>`__, which includes all the necessary dependencies.
+
+This script interpolates two ARPA N-gram language models and creates a KenLM binary file that can be used with the beam search decoders on top of ASR models.
+You can specify weights (`--alpha` and `--beta`) for each of the models (`--ngram_a` and `--ngram_b`) correspondingly: `alpha` * `ngram_a` + `beta` * `ngram_b`.
+This script supports both character level and BPE level encodings and models which are detected automatically from the type of the model.
+
+To combine two N-gram models, you can use the following command:
+
+.. code-block::
+
+    python ngram_merge.py  --kenlm_bin_path <path to the bin folder of KenLM library> \
+                    --ngram_bin_path  <path to the bin folder of OpenGrm Ngram library> \
+                    --arpa_a <path to the ARPA N-gram model file A> \
+                    --alpha <weight of N-gram model A> \
+                    --ar
+                    pa_b <path to the ARPA N-gram model file B> \
+                    --beta <weight of N-gram model B> \
+                    --out_path <path to folder to store the output files>
+
+
+
+If you provide `--test_file` and `--nemo_model_file`, This script supports both character-level and BPE-level encodings and models, which are detected automatically based on the type of the model.
+Note, the result of each step during the process is cached in the temporary file in the `--out_path`, to speed up further run.
+You can use the `--force` flag to discard the cache and recalculate everything from scratch.
+
+.. code-block::
+
+    python ngram_merge.py  --kenlm_bin_path <path to the bin folder of KenLM library> \
+                    --ngram_bin_path  <path to the bin folder of OpenGrm Ngram library> \
+                    --arpa_a <path to the ARPA N-gram model file A> \
+                    --alpha <weight of N-gram model A> \
+                    --arpa_b <path to the ARPA N-gram model file B> \
+                    --beta <weight of N-gram model B> \
+                    --out_path <path to folder to store the output files>
+                    --nemo_model_file <path to the .nemo file of the model> \
+                    --test_file <path to the test file> \
+                    --symbols <path to symbols (.syms) file> \
+                    --force <flag to recalculate and rewrite all cached files>
+
+
+The following is the list of the arguments for the opengrm script:
+
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| **Argument**         |**Type**| **Default**      | **Description**                                                                                                 |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| kenlm_bin_path       | str    | Required         | The path to the bin folder of KenLM library. It is a folder named `bin` under where KenLM is installed.         |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| ngram_bin_path       | str    | Required         | The path to the bin folder of OpenGrm Ngram. It is a folder named `bin` under where OpenGrm Ngram is installed. |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| arpa_a               | str    | Required         | Path to the ARPA N-gram model file A.                                                                           |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| alpha                | float  | Required         | Weight of N-gram model A.                                                                                       |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| arpa_b               | int    | Required         | Path to the ARPA N-gram model file B.                                                                           |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| beta                 | float  | Required         | Weight of N-gram model B.                                                                                       |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| out_path             | str    | Required         | Path for writing temporary and resulting files.                                                                 |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| test_file            | str    | None             | Path to test file to count perplexity if provided.                                                              |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| symbols              | str    | None             | Path to symbols (.syms) file. Could be calculated if it is not provided.                                        |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| nemo_model_file      | str    | None             | The path to '.nemo' file of the ASR model, or name of a pretrained NeMo model.                                  |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| force                | bool   | ``False``        | Whether to recompile and rewrite all files.                                                                     |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+

docs/source/asr/asr_customization/word_boosting.rst

@@ -0,0 +1,362 @@
+.. _word_boosting:
+
+****************************************************
+Word Boosting
+****************************************************
+
+.. _word_boosting_gpupb:
+
+GPU-PB
+========================
+
+GPU-PB is a GPU-accelerated Phrase-Boosting method supported for CTC, RNN-T/TDT, and AED (Canary) models based on NGPU-LM infrastructure.
+The method supports greedy and beam search decoding, including CUDA graphs mode. GPU-PB is compatible with NGPU-LM at the same decoding run.
+
+GPU-PB is applied only at the decoding step in shallow fusion mode. You do not need to retrain the ASR model.
+During greedy or beam search decoding, GPU-PB rescales ASR model scores with a boosting tree at the token level.
+The boosting tree is built from a context phrases list, which is provided by the user.
+
+**NOTE**: for ASR models that support capitalization by default (e.g., Canary or parakeet-tdt-0.6b-v2), you need to capitalize all the key phrases in advance (and capitalize the full word for abbreviations).
+You can use LLM for this task.
+
+More details about GPU-PB method can be found in the `original paper <https://arxiv.org/abs/2508.07014>`__.
+
+Usage
+-----
+We support three ways to pass the context phrases into the decoding script:
+
+1. Build a boosting tree for a specific ASR model (step 0.0) and use it for all the decoding evaluation by ``boosting_tree.model_path`` (step 1.1-3.1).
+2. Provide a file with context phrases ``boosting_tree.key_phrases_file`` - one phrase per line  (step 1.1-3.1).
+3. Provide a python list of context phrases ``boosting_tree.key_phrases_list`` (step 1.1-3.1).
+
+The use of the Phrase-Boosting tree is controlled by ``boosting_tree`` config (``BoostingTreeModelConfig``) for all the models.
+For prepared boosting tree use ``boosting_tree.model_path=${PATH_TO_BTREE}``.
+We recommend to provide the list of context phrases directly into ``speech_to_text_eval.py`` by ``boosting_tree.key_phrases_file=${KEY_WORDS_LIST}``.
+
+List of the most important parameters:
+
+*  ``strategy`` - The strategy to use for decoding depending on the model type (CTC - greedy_batch or beam_batch; RNN-T/TDT - greedy_batch or malsd_batch; AED - beam).
+*  ``model_path``, ``key_phrases_file``, ``key_phrases_list`` - The way to pass the context phrases into the decoding script.
+*  ``context_score`` - The score for each arc transition in the context graph (1.0 is recommended).
+*  ``depth_scaling`` - The scaling factor for the depth of the context graph (2.0 is recommended for CTC, RNN-T and TDT, 1.0 for Canary).
+*  ``boosting_tree_alpha`` - Weight of the GPU-PB boosting tree during shallow fusion decoding (tune it according to your data).
+
+**0.0. [Optional] Build the boosting tree for a specific ASR model:**
+
+.. code-block::
+
+    python scripts/asr_context_biasing/build_gpu_boosting_tree.py \
+            asr_model_path=${ASR_NEMO_MODEL_FILE} \
+            key_phrases_file=${CONTEXT_BIASING_LIST} \
+            save_to=${PATH_TO_SAVE_BTREE} \
+            context_score=${CONTEXT_SCORE} \
+            depth_scaling=${DEPTH_SCALING} \
+            use_triton=True
+
+**1.1. CTC greedy batch decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        ctc_decoding.strategy="greedy_batch" \
+        ctc_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        ctc_decoding.greedy.boosting_tree.context_score=1.0 \
+        ctc_decoding.greedy.boosting_tree.depth_scaling=2.0 \
+        ctc_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}
+
+
+**1.2. CTC beam batch decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        ctc_decoding.strategy="beam_batch" \
+        ctc_decoding.beam.beam_size=${BEAM_SIZE} \
+        ctc_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        ctc_decoding.beam.boosting_tree.context_score=1.0 \
+        ctc_decoding.beam.boosting_tree.depth_scaling=2.0 \
+        ctc_decoding.beam.boosting_tree_alpha=${BT_ALPHA}
+
+**2.1. RNN-T/TDT greedy batch decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        rnnt_decoding.strategy="greedy_batch" \
+        rnnt_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        rnnt_decoding.greedy.boosting_tree.context_score=1.0 \
+        rnnt_decoding.greedy.boosting_tree.depth_scaling=2.0 \
+        rnnt_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}
+
+**2.2. RNN-T/TDT beam (malsd_batch) decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        rnnt_decoding.strategy="malsd_batch" \
+        rnnt_decoding.beam.beam_size=${BEAM_SIZE} \
+        rnnt_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        rnnt_decoding.beam.boosting_tree.context_score=1.0 \
+        rnnt_decoding.beam.boosting_tree.depth_scaling=2.0 \
+        rnnt_decoding.beam.boosting_tree_alpha=${BT_ALPHA}
+
+**3.1. AED (Canary) greedy (beam_size=1) or beam (beam_size>1) decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        multitask_decoding.strategy="beam" \
+        multitask_decoding.beam.beam_size=${BEAM_SIZE} \
+        multitask_decoding.beam.boosting_tree.key_phrases_file=${CONTEXT_BIASING_LIST} \
+        multitask_decoding.beam.boosting_tree.context_score=1.0 \
+        multitask_decoding.beam.boosting_tree.depth_scaling=1.0 \
+        multitask_decoding.beam.boosting_tree_alpha=${BT_ALPHA} \
+        gt_lang_attr_name="target_lang" \
+        gt_text_attr_name="text"
+
+Results evaluation
+------------------
+
+You can compute the F-score for the list of context phrases directly from the decoding manifest.
+
+.. code-block::
+
+    python scripts/asr_context_biasing/compute_key_words_fscore.py \
+            --input_manifest=${DECODING_MANIFEST} \
+            --key_words_file=${CONTEXT_PHRASES_LIST}
+
+
+.. _word_boosting_per_stream:
+
+Per-Stream Phrase Boosting
+==========================
+
+Per-stream (per-utterance) phrase boosting extends GPU-PB to allow specifying different key phrases for each audio stream or utterance in a batch.
+This is useful when different utterances require different context biasing (e.g., different speaker names, product terms, or domain vocabulary per audio).
+
+Per-stream boosting is currently supported for **greedy label-looping decoding with Transducers (RNN-T, TDT)**, including cache-aware streaming models.
+
+Manifest-based Usage
+--------------------
+
+Specify per-utterance key phrases in your manifest using the ``biasing_request`` field:
+
+.. code-block:: json
+
+    {"audio_filepath": "/data/file1.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["one phrase"]}}}
+    {"audio_filepath": "/data/file2.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["other phrases", "and this one"]}}}
+
+Use the streaming inference script with ``use_per_stream_biasing=true``:
+
+.. code-block:: bash
+
+    python examples/asr/asr_streaming_inference/asr_streaming_infer.py \
+        --config-path="../conf/asr_streaming_inference/" \
+        --config-name=cache_aware_rnnt.yaml \
+        audio_file="<manifest_with_boosting_requests>" \
+        output_filename="result.jsonl" \
+        asr.model_name="nvidia/parakeet-rnnt-1.1b" \
+        asr.decoding.greedy.enable_per_stream_biasing=True
+
+Python API Usage
+----------------
+
+.. code-block:: python
+
+    from omegaconf import open_dict
+    from nemo.collections.asr.models import EncDecRNNTBPEModel
+    from nemo.collections.asr.parts.context_biasing.biasing_multi_model import BiasingRequestItemConfig
+    from nemo.collections.asr.parts.context_biasing.boosting_graph_batched import BoostingTreeModelConfig
+    from nemo.collections.asr.parts.utils.rnnt_utils import Hypothesis
+
+    asr_model = EncDecRNNTBPEModel.from_pretrained("nvidia/parakeet-rnnt-1.1b")
+    asr_model.to("cuda")
+
+    with open_dict(asr_model.cfg.decoding):
+        asr_model.cfg.decoding.strategy = "greedy_batch"
+        asr_model.cfg.decoding.greedy.loop_labels = True
+        asr_model.cfg.decoding.greedy.enable_per_stream_biasing = True
+    asr_model.change_decoding_strategy(asr_model.cfg.decoding)
+
+    biasing_requests = [
+        BiasingRequestItemConfig(
+            boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["one phrase"]),
+            boosting_model_alpha=2.0,
+        ),
+        None,  # no biasing for this utterance
+        BiasingRequestItemConfig(
+            boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["other phrases"]),
+            boosting_model_alpha=1.0,
+        ),
+    ]
+
+    results = asr_model.transcribe(
+        audio=["file1.wav", "file2.wav", "file3.wav"],
+        partial_hypothesis=[
+            Hypothesis.empty_with_biasing_cfg(biasing_cfg=req) if req else None
+            for req in biasing_requests
+        ],
+        return_hypotheses=True,
+    )
+
+Caching
+-------
+
+Building a boosting model from a phrase list has some overhead. NeMo provides caching mechanisms to speed up repeated use of the same phrases:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 15 60 25
+
+   * - Strategy
+     - Description
+     - Recommended For
+   * - Memory
+     - Set ``cache_key`` on ``BiasingRequestItemConfig`` to cache compiled models in memory by a string key.
+     - Repeated phrase sets
+   * - Disk
+     - Set ``model_path`` on ``BoostingTreeModelConfig`` to save/load compiled models from disk.
+     - Persistent caching
+   * - Decoder
+     - Set ``auto_manage_multi_model=False`` and manually manage models in the decoder's multi-model.
+     - Advanced use cases
+
+With memory caching, per-stream boosting achieves near-zero overhead compared to global (shared) boosting.
+
+
+.. _word_boosting_flashlight:
+
+Flashlight-based Word Boosting
+==============================
+
+
+The Flashlight decoder supports word boosting during CTC decoding using a KenLM binary and corresponding lexicon. Word boosting only works in lexicon-decoding mode and does not function in lexicon-free mode. It allows you to bias the decoder for certain words by manually increasing or decreasing the probability of emitting specific words. This can be very helpful if you have uncommon or industry-specific terms that you want to ensure are transcribed correctly.
+
+For more information, go to `word boosting <https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-customizing.html#word-boosting>`__
+
+To use word boosting in NeMo, create a simple tab-separated text file. Each line should contain a word to be boosted, followed by a tab, and then the boosted score for that word.
+
+For example:
+
+.. code-block::
+
+    nvidia	40
+    geforce	50
+    riva	80
+    turing	30
+    badword	-100
+
+Positive scores boost words higher in the LM decoding step so they show up more frequently, whereas negative scores
+squelch words so they show up less frequently. The recommended range for the boost score is +/- 20 to 100.
+
+The boost file handles both in-vocabulary words and OOV words just fine, so you can specify both IV and OOV words with corresponding scores.
+
+You can then pass this file to your Flashlight config object during decoding:
+
+.. code-block::
+
+    # Lexicon-based decoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="flashlight" \
+           decoding.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon' \
+           decoding.beam.flashlight_cfg.boost_path='/path/to/my_boost_file.boost' \
+           decoding.beam.flashlight_cfg.beam_size_token = 32 \
+           decoding.beam.flashlight_cfg.beam_threshold = 25.0
+
+.. _word_boosting_ctcws:
+
+CTC-WS: Context-biasing (Word Boosting) without External LM
+===========================================================
+
+NeMo toolkit supports a fast context-biasing method for CTC and Transducer (RNN-T) ASR models with CTC-based Word Spotter.
+The method involves decoding CTC log probabilities with a context graph built for words and phrases from the context-biasing list.
+The spotted context-biasing candidates (with their scores and time intervals) are compared by scores with words from the greedy CTC decoding results to improve recognition accuracy and prevent false accepts of context-biasing.
+
+A Hybrid Transducer-CTC model (a shared encoder trained together with CTC and Transducer output heads) enables the use of the CTC-WS method for the Transducer model.
+Context-biasing candidates obtained by CTC-WS are also filtered by the scores with greedy CTC predictions and then merged with greedy Transducer results.
+
+Scheme of the CTC-WS method:
+
+.. image:: https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/asset-post-v1.22.0-ctcws_scheme_1.png
+    :align: center
+    :alt: CTC-WS scheme
+    :width: 80%
+
+High-level overview of the context-biasing words replacement with CTC-WS method:
+
+.. image:: https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/asset-post-v1.22.0-ctcws_scheme_2.png
+    :align: center
+    :alt: CTC-WS high level overview
+    :width: 80%
+
+More details about CTC-WS context-biasing can be found in the `tutorial <https://github.com/NVIDIA/NeMo/tree/main/tutorials/asr/ASR_Context_Biasing.ipynb>`__.
+
+To use CTC-WS context-biasing, you need to create a context-biasing text file that contains words/phrases to be boosted, with its transcriptions (spellings) separated by underscore.
+Multiple transcriptions can be useful for abbreviations ("gpu" -> "g p u"), compound words ("nvlink" -> "nv link"), 
+or words with common mistakes in the case of our ASR model ("nvidia" -> "n video").
+
+Example of the context-biasing file:
+
+.. code-block::
+
+    nvidia_nvidia
+    omniverse_omniverse
+    gpu_gpu_g p u
+    dgx_dgx_d g x_d gx
+    nvlink_nvlink_nv link
+    ray tracing_ray tracing
+
+The main script for CTC-WS context-biasing in NeMo is: 
+
+.. code-block::
+
+    {NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py
+
+Context-biasing is managed by ``apply_context_biasing`` parameter [true or false].
+Other important context-biasing parameters are:
+
+*  ``beam_threshold`` - threshold for CTC-WS beam pruning.
+*  ``context_score`` - per token weight for context biasing.
+*  ``ctc_ali_token_weight`` - per token weight for CTC alignment (prevents false acceptances of context-biasing words).
+
+All the context-biasing parameters are selected according to the default values in the script.
+You can tune them according to your data and ASR model (list all the values in the [] separated by commas)
+for example: ``beam_threshold=[7.0,8.0,9.0]``, ``context_score=[3.0,4.0,5.0]``, ``ctc_ali_token_weight=[0.5,0.6,0.7]``.
+The script will run the recognition with all the combinations of the parameters and will select the best one based on WER value.
+
+.. code-block::
+
+    # Context-biasing with the CTC-WS method for CTC ASR model 
+    python {NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py \
+            nemo_model_file={ctc_model_name} \
+            input_manifest={test_nemo_manifest} \
+            preds_output_folder={exp_dir} \
+            decoder_type="ctc" \
+            acoustic_batch_size=64 \
+            apply_context_biasing=true \
+            context_file={cb_list_file_modified} \
+            beam_threshold=[7.0] \
+            context_score=[3.0] \
+            ctc_ali_token_weight=[0.5]
+
+To use Transducer head of the Hybrid Transducer-CTC model, you need to set ``decoder_type=rnnt``.