chore: import upstream snapshot with attribution

.claude/settings.json

@@ -0,0 +1,16 @@
+{
+  "hooks": {
+    "PostToolUse": [
+      {
+        "matcher": "Write|Edit",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "jq -r '.tool_input.file_path' | { read -r f; f=\"${f#\"$(git rev-parse --show-toplevel)/\"}\"; case \"$f\" in *.py) ;; *) exit 0;; esac; case \"$f\" in docs/*|external/*|examples/*) exit 0;; nemo/collections/speechlm2/*) ;; nemo/collections/*) exit 0;; esac; black --quiet \"$f\"; isort --quiet \"$f\"; } 2>/dev/null || true",
+            "timeout": 30
+          }
+        ]
+      }
+    ]
+  }
+}

.claude/skills/babysit-pr/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: babysit-pr
+description: Get a pull request to green CI. Diagnose and fix CI failures, push fixes, re-trigger CI via the "Run CICD" label, and repeat until all checks pass. Does not post comments — this is a local developer tool.
+---
+
+# babysit-pr
+
+Get a PR's CI to green. Nothing else — no review comments, no PR comments, no status summaries. This skill is run locally by developers in their own sandboxes.
+
+## Inputs
+
+The PR number is the primary input. It may come from:
+- An explicit argument: `/babysit-pr 567`
+- The conversation context: "check on PR #567"
+- A GitHub PR URL pasted into the chat
+
+If no PR number is clear, ask for it before proceeding.
+
+## Workflow
+
+### Step 1 — Get the full picture
+
+```bash
+gh pr view <PR_NUMBER> --repo NVIDIA-NeMo/NeMo
+gh pr checks <PR_NUMBER> --repo NVIDIA-NeMo/NeMo
+gh pr diff <PR_NUMBER> --repo NVIDIA-NeMo/NeMo
+```
+
+Determine the current state:
+
+| State                        | What to do                           |
+|------------------------------|--------------------------------------|
+| CI failing                   | Diagnose and fix (Step 2)            |
+| Merge conflicts              | Resolve (Step 3)                     |
+| Formatting check failing     | Wait — do NOT fix (see below)        |
+| CI green                     | Done, nothing to do                  |
+| CI pending                   | Wait for it to finish, then reassess |
+
+### Checks to ignore
+
+The **"Isort and Black Formatting"** workflow (`reformat_with_isort_and_black` job) auto-pushes formatting fixes. If that check is failing or pending:
+- Do NOT fix formatting yourself — the auto-formatter will push a commit shortly.
+- Wait for it to complete and for the new commit to land before assessing other failures, since the formatting push changes the HEAD SHA and may re-trigger CI.
+
+### Step 2 — Fix CI failures
+
+Check out the PR branch and inspect the failure logs:
+
+```bash
+gh pr checkout <PR_NUMBER> --repo NVIDIA-NeMo/NeMo
+gh run list --repo NVIDIA-NeMo/NeMo --branch <branch-name>
+gh run view <RUN_ID> --repo NVIDIA-NeMo/NeMo --log-failed
+```
+
+Before attempting a fix, check `git log` for recent commits. If you see a previous fix attempt that addressed the same failure and it is still failing, **stop and tell the user** — the issue needs human attention. Do not keep retrying the same fix.
+
+Otherwise, identify the root cause, fix the code, and push:
+
+```bash
+git add <changed files>
+git commit -s -m "<brief summary of fix>"
+git push
+```
+
+### Step 3 — Re-trigger CI
+
+After pushing a fix, add the "Run CICD" label to re-trigger the CI pipeline:
+
+```bash
+gh pr edit <PR_NUMBER> --repo NVIDIA-NeMo/NeMo --add-label "Run CICD"
+```
+
+The "CICD NeMo" workflow is triggered by this label and removes it automatically when done.
+
+Then wait for CI to complete and reassess. Go back to Step 1.
+
+### Step 4 — Resolve merge conflicts
+
+If the PR branch has fallen behind main and has conflicts:
+
+```bash
+git fetch origin main
+git rebase origin/main
+# Resolve conflicts — keep the PR's intent, adopt refactors from main
+git add <resolved files>
+git rebase --continue
+git push --force-with-lease
+```
+
+After rebasing, go back to Step 3 to re-trigger CI.
+
+## Rules
+
+- **Do NOT post comments on the PR.** This is a local tool — communicate with the user directly.
+- **Do NOT address review comments.** The goal is green CI, not review resolution.
+- **Do NOT fix formatting.** The `Isort and Black Formatting` action handles that automatically.
+- **Do NOT retry a failed fix.** If your previous commit didn't resolve the failure, tell the user.
+- **Do NOT create new PRs.** Push to the existing branch only.
+- **Do NOT attempt to merge the PR or push to main.**
+
+## What done looks like
+
+CI is green (or the only remaining failures are pre-existing / flaky and you've told the user about them). That's it.

.claude/skills/debug-training-logs/SKILL.md

@@ -0,0 +1,320 @@
+---
+name: debug-training-logs
+description: Debug distributed training failures (NeMo, Megatron, PyTorch) from worker stderr logs and optional AIStore daemon logs. Finds root cause across NCCL timeouts, data loading errors, and storage failures.
+disable-model-invocation: true
+allowed-tools: Bash Read Grep Glob Agent
+argument-hint: <path-to-logs-dir> [ais-logs-dir]
+---
+
+# Distributed Training Log Debugger
+
+You are debugging a distributed training job failure. The user will provide one or two directories:
+- **Worker logs** (stderr from SLURM/torchrun): `$ARGUMENTS[0]` (required)
+- **AIStore daemon logs** (proxy/target tarballs or extracted dirs): `$ARGUMENTS[1]` (optional)
+
+Your goal: find the **root cause**, not just the symptom. There are often cascading failures — one root cause triggers many downstream errors. Trace backwards from the final crash to the original trigger.
+
+## CRITICAL: Verification discipline
+
+**You MUST double-check every conclusion before presenting it to the user.** Distributed training failures have cascading effects that make it easy to mistake a symptom for the root cause. Follow these rules:
+
+1. **Check ALL ranks, not a sample.** Do not look at 5 ranks and assume the other 123 are the same. Use `sort -u` on the full output to find outliers. A single outlier rank can be the entire root cause.
+2. **Verify NCCL state for EVERY rank.** Extract `last enqueued work` and `last completed work` for all ranks and check for ANY rank that differs. A rank with `enqueued == completed` (no pending ops) is fundamentally different from ranks with `enqueued > completed` (ops pending) — it means that rank never entered the stuck collective.
+3. **Distinguish "First PG on this rank to signal dumping" from "Observed flight recorder dump signal from another rank".** The first is the initiator (its own watchdog fired). The second was notified (it may not even be in the collective). These are DIFFERENT failure modes.
+4. **Before stating a root cause, re-verify it against the raw data.** Re-read the actual log lines. Do not rely on earlier summaries you wrote — they may have been wrong.
+5. **If your analysis changes during the investigation, explicitly state what was wrong before and why.** Do not silently shift conclusions.
+
+## Phase 0: Obtain logs
+
+If the user has not provided worker logs, ask them to download and provide the SLURM error logs. The logs are typically on the compute nodes or in a shared filesystem. Suggest:
+
+```bash
+# From the user's machine, SCP the SLURM error logs:
+scp <cluster>:/path/to/slurm/logs/error-<JOBID>-*.out ./training_logs/
+
+# Or if the logs are on a shared filesystem accessible from a login node:
+mkdir -p ./training_logs
+cp /path/to/slurm/error-<JOBID>-*.out ./training_logs/
+```
+
+The user needs to provide ALL per-node error files (e.g., `error-JOBID-0.out` through `error-JOBID-N.out`) — not just one node. Without all files you cannot identify which specific rank caused the failure.
+
+## Phase 1: Triage worker logs
+
+### 1.1 Understand the job
+Read the first 80 lines of a few log files to determine:
+- Framework (NeMo, Megatron, PyTorch Lightning, DeepSpeed, etc.)
+- Scale (GPU count, node count, ranks per node)
+- What the job is doing (training, fine-tuning, inference)
+- Whether it resumed from a checkpoint
+
+### 1.2 Find the fatal errors
+Search ALL log files in parallel for these patterns (priority order):
+
+**Tier 1 — Process killers:**
+```
+NCCL.*timeout|Watchdog caught collective operation timeout
+taking the entire process down
+SIGTERM|SIGKILL|SIGABRT
+CUDA error|CUDA out of memory|OOM
+```
+
+**Tier 2 — Training loop crashes:**
+```
+RuntimeError|Exception.*Error
+AISBatchLoaderError|StopIteration
+Traceback \(most recent call last\)
+```
+
+**Tier 3 — Data loading / IO:**
+```
+Connection reset|Connection broken|Connection refused
+retrying [0-9]+/[0-9]+
+timed out|deadline exceeded
+broken pipe
+```
+
+### 1.3 Identify the initiator and the straggler
+For NCCL timeouts, you MUST check ALL ranks — do not sample. Extract the full NCCL state for every rank:
+
+```bash
+# Get watchdog state for all ranks that fired their own watchdog:
+grep "failure detected by watchdog" error-*.out | grep -o "Rank [0-9]*.*last enqueued work: [0-9]*, last completed work: [0-9]*" | sort -u
+
+# Get ranks that were NOTIFIED (did not fire their own watchdog):
+grep "Observed flight recorder dump signal from another rank" error-*.out | grep -o "Rank [0-9]*"
+
+# Count total unique ranks found vs expected:
+grep "failure detected by watchdog" error-*.out | grep -o "Rank [0-9]*" | sort -t' ' -k2 -n -u | wc -l
+```
+
+**The rank that "Observed" instead of "detected" is likely the straggler** — it had no pending NCCL operations because it never entered the collective. Check its `Last enqueued NCCL work` — if it equals `last completed NCCL work`, the rank was stuck OUTSIDE NCCL (in the training loop, data loading, etc.), not inside a collective.
+
+For NCCL BROADCAST/ALLREDUCE timeouts, calculate when the collective started:
+`start_time = timeout_time - timeout_ms` (usually 1800000ms = 30min)
+
+### 1.4 Count and classify
+- Count `Connection reset` across all files (total and per-file)
+- Check retry patterns: are retries always 1/N (recovering) or do they escalate to N/N (exhausted)?
+- Count unique error types and affected ranks
+- Look for `AISBatchLoaderError` or similar batch loader errors — these indicate AIStore returned fewer objects than requested
+
+### 1.5 Build the timeline
+Determine the **causal chain**: which error happened first, which are consequences.
+The pattern is usually:
+```
+Data loading error (root cause)
+  -> Some ranks crash out of training loop
+  -> Crashed ranks can't participate in NCCL collective
+  -> NCCL collective hangs for timeout period (usually 30min)
+  -> Watchdog kills all remaining ranks
+```
+
+### 1.6 Check NeMo-specific synchronization points
+
+NeMo has per-step collective operations that can cause rank desynchronization:
+
+- **PreemptionCallback** (`nemo/utils/callbacks/preemption.py`): Calls `torch.distributed.broadcast(interrupted, 0)` at the end of EVERY training step via `on_train_batch_end`. If one rank's training step takes >30 min longer than others, this broadcast times out.
+- **NeMoModelCheckpoint** (`nemo/utils/callbacks/nemo_model_checkpoint.py`): Multiple `trainer.strategy.broadcast()` calls during checkpoint save/load.
+- **DDP gradient all-reduce**: Automatic per-step synchronization during backward pass.
+- **`broadcast_buffers`** (DDP default=True): Broadcasts model buffers (e.g., batch norm stats) from rank 0 at each forward pass.
+
+If rank 0 is ahead of other ranks in NCCL SeqNum, check whether the gap matches the number of per-step collectives (PreemptionCallback broadcast + DDP allreduce + broadcast_buffers = ~3 ops per step).
+
+## Phase 1.7: Request AIStore logs if not provided
+
+If the analysis points to storage I/O issues (connection resets, data loading errors, timeouts) and the user did NOT provide AIS daemon logs, suggest downloading them. First check if the `ais` CLI is available (`which ais`). If it is, guide the user through:
+
+1. **Set the cluster endpoint** (ask the user for the endpoint URL):
+   ```bash
+   export AIS_ENDPOINT=https://<ais-cluster-endpoint>:<port>
+   ```
+
+2. **Set the auth token** (ask the user for the token value):
+   ```bash
+   export AIS_AUTHN_TOKEN=<token>
+   ```
+
+3. **Handle TLS** — skip cert verification or point to the CA cert:
+   ```bash
+   # Option A: skip verification
+   ais config cli set cluster.skip_verify_crt=true
+   # Option B: set CA cert
+   export AIS_SERVER_CRT=/path/to/ca.crt
+   ```
+
+4. **Download all cluster logs** to the current log directory:
+   ```bash
+   ais log get cluster <path-to-worker-logs-dir>/ais_logs
+   ```
+   This downloads TAR.GZ archives from all proxy and target nodes.
+
+5. Extract and analyze per Phase 2 below.
+
+If the `ais` CLI is not installed, it can be built from the AIStore repository (`cd cmd/cli && go install .`) or downloaded as a binary. Alternatively, ask the user to download the logs manually from the AIS cluster.
+
+## Phase 2: Debug AIStore logs (if provided)
+
+### 2.1 Extract and orient
+If tarballs (`.tar.gz`), extract them:
+```bash
+mkdir -p extracted && cd extracted
+for f in ../*.tar.gz; do
+  name=$(basename "$f" .tar.gz)
+  mkdir -p "$name" && tar xzf "$f" -C "$name"
+done
+```
+
+**AIS log file naming and time ranges:**
+
+AIS daemon logs follow the naming convention:
+```
+aistarget.ais-target-N.INFO.MMDD-HHMMSS.1    # target logs
+aisproxy.ais-proxy-N.INFO.MMDD-HHMMSS.1      # proxy logs
+```
+
+A single daemon (target or proxy) may have **multiple log files** — each file covers a specific time range:
+- The `MMDD-HHMMSS` in the filename is the **start time** of that file
+- The file covers from its start time **until the start time of the next file** for the same daemon
+- If there is no next file, it covers until the daemon stopped or the logs were collected
+- A new file is created when the daemon restarts (crash, upgrade, maintenance cycle)
+
+**To find the correct file for a failure window:**
+1. List all files for each daemon, sorted by the timestamp in the filename
+2. For each daemon, find the file whose start time is BEFORE the failure window AND whose next file's start time is AFTER the failure window (or there is no next file)
+3. A daemon may have been restarted DURING the failure window — check if any file starts within the window (this indicates a restart, which is itself significant)
+4. Log lines within a file only have TIME (HH:MM:SS), not dates. If a file spans midnight, the same time (e.g., "21:30") may appear twice — once for each day. Use surrounding context (stats counter values, known events) to disambiguate which day a log line belongs to.
+
+**Always check ALL files for a daemon, not just the latest.** The latest file may only cover minutes after a restart, while the failure evidence is in an earlier file.
+
+**Timezone verification:**
+AIS daemon logs and worker (SLURM/torchrun) logs may be on different machines in different timezones. NEVER assume they match. To verify:
+1. Look for AIS periodic timestamp markers: `common:NNN DD Mon YY HH:MM UTC =============` — these explicitly state the timezone (typically UTC).
+2. Look for absolute timestamps in worker logs: NeMo uses `YYYY-MM-DD HH:MM:SS`, SLURM uses `YYYY-MM-DDThh:mm:ss` — but neither includes timezone by default.
+3. **Cross-reference a known event** visible in both log sources. The best anchor is the job death: find the SLURM `CANCELLED` timestamp in worker logs, then find the exact moment `get.n` stops incrementing in AIS stats. If they align, the timezones match. If they're offset by a round number of hours, one system is in a different timezone.
+4. If timezones don't match, apply the offset consistently before correlating events.
+
+### 2.2 Check target stats during failure window
+AIStore targets emit periodic stats lines (every ~3 min). Extract key counters around the failure time:
+
+**Critical counters to track:**
+- `err.get.n` — GET errors (should be stable; spikes indicate problems)
+- `err.getbatch.n` — batch GET errors
+- `err.http.write.n` — broken HTTP responses to clients
+- `err.put.n`, `err.head.n`, `err.lst.n` — other operation errors
+- `get.n` vs `err.get.n` — compute error rate
+- `getbatch.n`, `getbatch.obj.n` — batch operation counts
+
+Compare values between consecutive stats lines to find the **delta** (new errors in that interval).
+
+### 2.3 Search for error-level messages
+```
+grep "^E " <logfile>          # Error messages
+grep "^W " <logfile>          # Warning messages
+```
+
+**Key error patterns in AIStore:**
+- `x-get-batch.*out-of-bounds index` — batch GET lost objects during inter-target streaming
+- `shared-dm.*terminated.*broken pipe` — inter-target data mover stream failure
+- `shared-dm: xid.*not found, dropping recv` — objects dropped because batch job already aborted
+- `resource pressure: load=critical` — target under disk/memory/CPU pressure
+- `lcache.*hk.*dsk=critical` — disk at critical level, housekeeping skipped
+- `gc:.*free mem` / `oom:` — memory pressure / forced GC
+
+### 2.4 Check proxy logs
+The proxy orchestrates batch GETs. Check proxy stats for:
+- `err.get.n` — should be near zero; high = proxy-level routing failures
+- `err.http.write.n` — proxy dropping client connections
+
+If proxy errors are stable but target errors are spiking, the problem is target-side (disk, memory, inter-target networking).
+
+### 2.5 Correlate the `x-get-batch` failure mechanism
+The typical AIStore batch GET failure chain:
+```
+1. Target under resource pressure (dsk=critical, mem=low)
+2. Inter-target shared-dm streams break (broken pipe)
+3. x-get-batch gets out-of-bounds index (recv'd len=0)
+4. Batch job aborted, subsequent objects dropped (xid not found)
+5. Client receives fewer objects than requested
+6. Lhotse/client batch loader raises error (iterator exhausted prematurely)
+```
+
+## Phase 3: Synthesize
+
+### 3.1 Write the report
+Structure your output as:
+
+**Job Details** — framework, scale, start time, data source
+
+**Timeline Table** — chronological events with timestamps, source file:line
+
+**Root Cause Chain** — numbered chain from trigger to final crash, with arrows
+
+**Key Files** — which log files contain the critical evidence
+
+**Recommendations** — actionable fixes (storage-side, client-side, training config)
+
+### 3.2 Classify the failure
+Common root cause categories:
+- **Storage I/O**: disk pressure, broken pipes, connection resets, batch object loss
+- **Network**: NCCL timeout without data errors, NIC failures, switch issues
+- **GPU/CUDA**: OOM, ECC errors, CUDA assertions
+- **Data**: corrupt/missing data files, manifest mismatches, schema errors
+- **Software**: version mismatches, config errors, OOM in Python process
+- **Infrastructure**: node failure, preemption, SLURM timeout
+- **Data loading stall**: single rank stuck in data loading (no timeout on read), blocking all other ranks at the next collective
+
+## Reference: AIStore error counter meanings
+
+| Counter | Meaning |
+|---|---|
+| `err.get.n` | Individual object GET failures on this target |
+| `err.getbatch.n` | Batch GET operation failures |
+| `err.http.write.n` | Failed HTTP writes to client (connection dropped) |
+| `err.append.n` | Append operation failures |
+| `err.put.n` | PUT operation failures |
+| `err.head.n` | HEAD (metadata) operation failures |
+| `err.lst.n` | List operation failures |
+| `err.kalive.n` | Keep-alive failures |
+| `getbatch.n` | Total batch GET operations |
+| `getbatch.obj.n` | Total objects served via batch GET |
+| `getbatch.throttle.ns` | Time spent throttling batch GETs |
+
+## Reference: NCCL timeout anatomy
+
+- `Watchdog caught collective operation timeout` — the NCCL watchdog detected a stuck collective
+- `SeqNum=N, OpType=BROADCAST/ALLREDUCE` — which collective and sequence number
+- `last enqueued work: N, last completed work: M` — work M completed, work M+1 is stuck
+- `Timeout(ms)=1800000` — 30-minute timeout (default)
+- `First PG on this rank to signal dumping` — THIS rank initiated the cascade
+- `Observed flight recorder dump signal from another rank` — this rank is reacting to another's timeout
+- `To avoid data inconsistency, we are taking the entire process down` — watchdog kills the process
+
+### Distinguishing data loading stalls from GPU/NCCL hangs
+
+The `last enqueued work` vs `last completed work` state is critical for determining whether the hang is in the CPU (data loading, training loop) or in the GPU (NCCL communication):
+
+- **If `enqueued == completed` (no pending ops)**: This rank has NO NCCL work in-flight. It is stuck in the CPU-side training loop (data loading, audio decoding, batch prep) and never entered the collective. **This is the straggler — the rank that caused the hang.**
+- **If `enqueued == completed + 1`**: The rank has submitted exactly one operation that hasn't completed. It entered the collective but can't complete because the straggler rank hasn't joined.
+- **If `enqueued > completed + 1`**: Multiple operations are queued — the CPU has progressed past the stuck point and submitted additional operations asynchronously (e.g., DDP gradient allreduce via hooks). Still waiting for the straggler.
+- **If rank 0 has higher `enqueued`/`completed` than others**: Rank 0 (often the broadcast root) completed its side of a collective (send) but receivers can't complete (receive) because the straggler hasn't joined.
+
+**The key pattern for a data loading stall:**
+- 1 rank: `enqueued == completed`, `active collectives: 0`, "Observed flight recorder dump signal from another rank" — **this is the straggler**
+- N-1 ranks: `enqueued > completed`, "failure detected by watchdog" — these are waiting for the straggler
+- Rank 0: may be further ahead if it's the broadcast root (can complete send without receivers)
+
+**The key pattern for a GPU fabric issue:**
+- ALL ranks: `enqueued > completed` (all entered the collective), all show "First PG on this rank to signal dumping" — no straggler, the collective itself is broken
+
+Compare `enqueued` counts across ALL ranks. Even ONE outlier changes the entire diagnosis.
+
+## Reference: NeMo/Lhotse data loading pitfalls
+
+Common data loading issues that cause single-rank stalls:
+
+- **No timeout on Lhotse URL audio reads**: `AudioSource._prepare_for_reading()` calls `f.read()` with no Lhotse-level timeout. A stalled download blocks the DataLoader worker indefinitely.
+- **`fault_tolerant=True` silently drops failed audio**: Failed audio files are skipped, reducing effective batch size per rank. Different ranks may have different failure rates depending on their shard assignments.
+- **`.m4a` files lose extension in BytesIO**: When audio is downloaded from URLs and wrapped in BytesIO, the file extension is lost. Lhotse's `CompositeAudioBackend` cannot use the fast-path for m4a (TorchaudioFFMPEGBackend) and falls through to expensive cascading backend trials.
+- **Connection reset on idle keep-alive**: AIStore closes idle HTTP connections after 30s (`DfltMaxIdleTimeout`). The Python SDK's urllib3 pool doesn't match this timeout, causing `Connection reset by peer` on stale pooled connections. These are caught and retried (always succeed on 1st retry) — they are noise, NOT a root cause.
+- **Each rank gets disjoint data shards**: Lhotse splits shards via `src[rank::world_size]`. One rank may get shards with more corrupt files, larger audio, or slower storage targets.

.claude/skills/fix-issue/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: fix-issue
+description: Fix a GitHub issue in NeMo Speech (NVIDIA-NeMo/NeMo). Read the issue, reproduce the bug with a failing test, implement the fix, and verify tests pass. Only opens a PR if the user explicitly asks for it.
+---
+
+# fix-issue
+
+Fix a GitHub issue in NeMo Speech from first principles: understand the bug, prove it with a test, and fix the root cause.
+
+The key discipline is **reproduce first, fix second**. Writing the failing test before touching the source code forces a precise understanding of what broken actually means, which keeps the fix focused and correct.
+
+## Inputs
+
+The issue number is the primary input. It may come from:
+- An explicit argument: `/fix-issue 1234`
+- The conversation context: "fix issue #1234"
+- A GitHub issue URL pasted into the chat
+
+If no issue number is clear, and the description provided is not good enough, ask for more details before proceeding.
+
+## Before starting
+
+Read the issue description carefully. Identify:
+- What is the reported failure (error message, wrong output, crash)?
+- Which collections in `nemo/collections` and components are affected?
+- Are there any repro steps already in the issue? If so, run them first.
+- Is there a linked PR or related issue that gives more context?
+
+## Workflow
+
+1. Read the issue: `gh issue view <ISSUE_NUMBER> --repo NVIDIA-NeMo/NeMo`
+2. Understand the bug — identify the relevant code
+3. Write a minimal reproduction test in `tests/` that demonstrates the failure
+4. Run the test to confirm it fails: `pytest <your_test_file> -v`
+5. Implement the fix in the source code
+6. Run the test again to confirm it now passes
+7. Run the broader test suite for the affected collection to check for regressions:
+   ```bash
+   pytest tests/collections/<collection>/ -m "not pleasefixme" -v --timeout=120
+   ```
+8. Tell the user the fix is ready and what was changed
+
+## Opening a PR
+
+Only create a branch and open a PR if the user explicitly asks for it. When they do:
+
+```bash
+git checkout -b fix/<ISSUE_NUMBER>-<short-description>
+git add <changed files>
+git commit -s -m "Fix <short-description> (closes #<ISSUE_NUMBER>)"
+git push origin fix/<ISSUE_NUMBER>-<short-description>
+gh pr create --repo NVIDIA-NeMo/NeMo \
+  --title "Fix <short-description>" \
+  --body "$(cat <<'EOF'
+# What does this PR do ?
+<one-line overview>
+
+# Changelog
+<line-by-line high-level changes>
+
+# Usage
+<usage example if behavior changed>
+
+Fixes #<ISSUE_NUMBER>
+EOF
+)"
+```
+
+Notes:
+- Add specific files by name — do not use `git add -A` or `git add .`.
+
+## Rules
+
+- **Reproduce first.** Do not attempt a fix without a failing test that demonstrates the bug.
+- **Do NOT open a PR unless the user asks.** The default is local-only: fix the code, verify it works, report back.
+- **Do NOT post comments on the GitHub issue.** Communicate with the user directly.
+- **Do NOT reformat files outside your changes.** The `Isort and Black Formatting` action handles formatting automatically.
+- **Never push to `main` directly.**
+- **Never attempt to merge the PR yourself.**

.claude/skills/nemo-speech-asr-finetune/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: nemo-speech-asr-finetune
+description: Guide NeMo Speech users through ASR fine-tuning with container setup and Lhotse training.
+---
+
+# NeMo Speech ASR Fine-Tuning
+
+Use this skill when a user wants to fine-tune a NeMo Speech ASR model, choose a checkpoint, adapt a tokenizer,
+configure Lhotse dataloading, train, average checkpoints, or evaluate a fine-tuned ASR `.nemo` checkpoint.
+Also use it for post-run refinement planning after fine-tuning.
+
+Default posture:
+
+- Use the NeMo container unless the user explicitly asks for local execution.
+- Prefer Lhotse for train and validation dataloaders.
+- Use `trainer.max_steps`, not `trainer.max_epochs`.
+- Use `val_wer` as the checkpoint monitor for validation.
+- By default, evaluate WER without capitalization and punctuation effects. Change that only when the user explicitly
+  asks for raw/cased/punctuated scoring.
+- Report final quality from standalone evaluation, not only in-training validation logs.
+
+## Staged Workflow
+
+Load only the reference file needed for the current stage:
+
+1. Setup and checkpoint selection: read `references/setup-checkpoints.md`.
+2. Data prep, transcript-style preflight, Lhotse, bucketing, validation dataloader, and blends: read
+   `references/data-lhotse.md`.
+3. Architecture detection, tokenizer changes, and AED/Canary multitask metrics: read
+   `references/architecture-tokenizer-metrics.md`.
+4. Training, checkpoint averaging, and evaluation: read `references/training-evaluation.md` and, when reporting WER,
+   `references/evaluation-style-contract.md`.
+5. Post-run refinement, error analysis, curriculum, and general-vs-domain evaluation: read
+   `references/refinement-iteration.md`.
+
+If the user explicitly asks for parallel/sub-agent work, split the work by these same stages. Keep each agent scoped to
+one stage and have the main agent integrate the final command/config.
+
+## Core Commands
+
+Generic fine-tuning uses `examples/asr/speech_to_text_finetune.py`. For architecture-specific recipes, route to:
+
+- CTC: `examples/asr/asr_ctc/speech_to_text_ctc_bpe.py`
+- RNNT: `examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py`
+- Hybrid RNNT/CTC or TDT/CTC: `examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe.py`
+- AED/Canary: `examples/asr/speech_multitask/speech_to_text_aed.py`
+
+Always check the current repo docs before giving version-sensitive claims:
+
+- `README.md`
+- `docs/source/asr/fine_tuning.rst`
+- `docs/source/asr/datasets.rst`
+- `docs/source/dataloaders.rst`
+- `docs/source/asr/featured_models.rst`
+- `docs/source/asr/asr_checkpoints.rst`
+- `nemo/collections/common/data/lhotse/dataloader.py`
+
+## Non-Negotiable Pitfalls
+
+- When changing Lhotse batch modes, explicitly null conflicting options. For OOMptimizer profiles, set
+  `batch_size=null`, `batch_duration=null`, and `quadratic_duration=null` when adding `bucket_batch_size`.
+- Set `model.validation_ds.use_lhotse=true`, but prefer static validation `batch_size` with bucketing disabled.
+- Do not use fused loss/WER or tune `fused_batch_size` for RNNT/TDT fine-tuning guidance from this skill.
+- Run the first OOMptimizer pass with default CLI settings; lower `--memory-fraction` only after a real training OOM.
+- Run preflight checks before long jobs: disk space, free GPUs, manifest validity, and duration/text sanity.
+- Before any fine-tuning, audit transcript style within and across all fine-tuning/validation/test sources. Do not
+  train on mixed casing, punctuation, inverse-text-normalization, or symbol conventions; choose and fix one target style
+  first, and compare it with the original checkpoint's prediction style when applicable.
+- For small domain adaptation, start with a lower LR than large-data fine-tuning; do not blindly use `1e-4`.
+- Do not train a tokenizer on validation or test transcripts.
+- Do not ignore silent Lhotse filtering from `min_duration`, `max_duration`, `min_tps`, and `max_tps`.
+- Do not use `amp=true` for inference/evaluation; use `amp=false compute_dtype=bfloat16`.
+- Unless the user asks otherwise, report the default WER with capitalization and punctuation removed, and record any raw
+  WER separately when it helps diagnose transcript-style mismatch.
+- For AED/Canary, configure `multitask_metrics_cfg` so ASR and translation/task-specific samples are evaluated with
+  the right constrained metrics.
+- If checkpoint averaging is used, evaluate the averaged checkpoint and keep it only if it beats the best individual
+  checkpoint.

.claude/skills/nemo-speech-asr-finetune/assets/experiment-ledger-template.md

@@ -0,0 +1,92 @@
+# ASR Fine-Tuning Experiment Ledger
+
+## Goal
+
+- User goal:
+- Target checkpoint:
+- Architecture:
+- Success metric:
+- Optional guardrail metrics:
+
+## Data
+
+- Train manifests or input config:
+- Validation manifest:
+- Test manifest:
+- Data sources and weights:
+- Tarred/non-tarred:
+- Manifest sharding:
+- Transcript style policy:
+- Style transform artifact:
+- Original checkpoint output style:
+- Style mismatch decision:
+
+## Preflight
+
+- Disk space:
+- GPUs:
+- Container/image:
+- Manifest validation:
+- Duration distribution:
+- Text/token distribution:
+- Duration/token filters:
+- Examples/hours filtered:
+
+## Lhotse And OOMptimizer
+
+- Lhotse train:
+- Lhotse validation:
+- Bucketing mode:
+- Duration bins:
+- Bucket batch sizes:
+- Static batch size:
+- `bucket_buffer_size`:
+- `shuffle_buffer_size`:
+- `seed`:
+- `shard_seed`:
+- OOMptimizer settings:
+- Training pilot utilization:
+- CPU memory notes:
+
+## Training
+
+- Init checkpoint:
+- Script/config:
+- Precision:
+- `sync_batchnorm`:
+- `max_steps`:
+- `limit_train_batches`:
+- `val_check_interval`:
+- LR:
+- Scheduler:
+- Warmup:
+- Min LR:
+- Save top K:
+- Command/log path:
+
+## Evaluation
+
+| Model | Artifact | Prediction Manifest | Raw WER | Default WER | CER | Notes |
+| --- | --- | --- | --- | --- | --- | --- |
+| baseline |  |  |  |  |  |  |
+| final `.nemo` |  |  |  |  |  |  |
+| best single |  |  |  |  |  |  |
+| averaged |  |  |  |  |  |  |
+
+Default WER uses capitalization and punctuation removal unless the user requested a different metric.
+
+## Error Analysis
+
+- Raw vs default WER gap:
+- Worst sources/domains:
+- Worst categories:
+- Label/audio defects:
+- Decoding findings:
+
+## Decision
+
+- Keep artifact:
+- Drop artifacts:
+- Next intervention:
+- Reason:
+- If validation/test influenced data or weights, blind holdout plan:

.claude/skills/nemo-speech-asr-finetune/references/architecture-tokenizer-metrics.md

@@ -0,0 +1,153 @@
+# Stage 3: Architecture, Tokenizer, And AED Metrics
+
+## Architecture Detection
+
+Inspect the model config before choosing scripts and overrides:
+
+```python
+from nemo.collections.asr.models import ASRModel
+
+cfg = ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v3", return_config=True)
+print(cfg.target)
+print(cfg.get("decoder", None))
+print(cfg.get("joint", None))
+print(cfg.get("loss", None))
+print(cfg.get("decoding", None))
+```
+
+Classify:
+
+- CTC: `EncDecCTC*`, `ConvASRDecoder`, no RNNT-style `joint`.
+- RNNT: `EncDecRNNT*` with `decoder` and `joint`.
+- TDT: RNNT-family config with `loss.loss_name: tdt`, `decoding.model_type: tdt`, durations, or extra duration
+  outputs.
+- Hybrid RNNT/CTC or TDT/CTC: `EncDecHybridRNNTCTC*`, `aux_ctc`, `ctc_decoder`.
+- AED/Canary: `EncDecMultiTaskModel`, Transformer decoder, `prompt_format`.
+
+Use `examples/asr/speech_to_text_finetune.py` for compatible-architecture fine-tuning. For architecture-specific
+recipes:
+
+- CTC: `examples/asr/asr_ctc/speech_to_text_ctc_bpe.py`
+- RNNT: `examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py`
+- Hybrid RNNT/CTC or TDT/CTC: `examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe.py`
+- AED/Canary: `examples/asr/speech_multitask/speech_to_text_aed.py`
+
+Reference configs to inspect before writing overrides:
+
+- CTC: `examples/asr/conf/fastconformer/fast-conformer_ctc_bpe.yaml`
+- RNNT: `examples/asr/conf/fastconformer/fast-conformer_transducer_bpe.yaml`
+- TDT: `examples/asr/conf/conformer/tdt/conformer_tdt_bpe.yaml`
+- Hybrid TDT/CTC: `examples/asr/conf/fastconformer/hybrid_transducer_ctc/fastconformer_hybrid_tdt_ctc_bpe.yaml`
+- AED/Canary: `examples/asr/conf/speech_multitask/fast-conformer_aed.yaml`
+
+## Tokenizer Decisions
+
+Keep the pretrained tokenizer when language/script, casing, punctuation, and symbols match. Replace or extend it when
+the target language/script is new, important symbols are missing, normalization changes substantially, the run is
+multilingual/code-switching, or Canary/AED prompt/special tokens change.
+
+Train from training text only:
+
+```bash
+python scripts/tokenizers/process_asr_text_tokenizer.py \
+  --manifest=/data/train.json \
+  --data_root=/data/tokenizers/my_tokenizer \
+  --vocab_size=1024 \
+  --tokenizer=spe \
+  --spe_type=unigram \
+  --log
+```
+
+Replace in generic fine-tuning:
+
+```bash
+model.tokenizer.update_tokenizer=true \
+model.tokenizer.dir=/data/tokenizers/my_tokenizer/tokenizer_spe_unigram_v1024 \
+model.tokenizer.type=bpe
+```
+
+Changing tokenizer size usually reinitializes decoder-side parameters such as CTC projection or RNNT/TDT
+decoder/joint pieces. Use conservative LR and validate early.
+
+Aggregate tokenizer:
+
+```yaml
+tokenizer:
+  type: agg
+  langs:
+    en:
+      dir: /data/tokenizers/en/tokenizer_spe_unigram_v1024
+      type: bpe
+    es:
+      dir: /data/tokenizers/es/tokenizer_spe_unigram_v1024
+      type: bpe
+```
+
+Standard aggregate ASR configs expect a manifest language field such as `lang`. AED/Canary configs use their own
+prompt and language fields; follow `examples/asr/conf/speech_multitask/fast-conformer_aed.yaml`.
+
+## Architecture-Specific Knobs
+
+CTC:
+
+- Main tokenizer-sensitive module is the decoder projection.
+- Useful when alignments or non-autoregressive decoding matter.
+- Long transcripts can violate CTC length constraints; subword tokenization helps reduce target length.
+
+RNNT:
+
+- Disable fused loss/WER for this skill: `model.joint.fuse_loss_wer=false`.
+- Do not tune `fused_batch_size`; use Lhotse bucketing plus OOMptimizer-generated `bucket_batch_size`.
+- `model.compute_eval_loss=false` is common when validation samples are long and WER is the main metric.
+- Use CUDA graphs for inference/evaluation when supported.
+
+TDT:
+
+- Preserve `loss.loss_name=tdt`, duration settings, extra outputs, and `decoding.model_type=tdt`.
+- Disable fused loss/WER and use Lhotse bucketing plus OOMptimizer.
+- Use CUDA graphs for inference/evaluation when supported.
+
+Hybrid:
+
+- Check `model.aux_ctc.ctc_loss_weight`; reference configs often use `0.3`.
+- Evaluate both decoder paths when relevant with `decoder_type=ctc` and `decoder_type=rnnt`.
+
+AED/Canary:
+
+- Use `examples/asr/speech_multitask/speech_to_text_aed.py`.
+- Preserve `prompt_format` and expected manifest fields.
+- Prefer 2D Lhotse buckets plus OOMptimizer.
+
+## AED/Canary Multitask Metrics
+
+`EncDecMultiTaskModel` reads `model.multitask_metrics_cfg` and constructs `MultiTaskMetric`
+(`nemo/collections/asr/metrics/multitask.py`). Metric constraints are evaluated against each Lhotse cut's `custom`
+dict, including manifest fields and `input_cfg.tags`.
+
+Reference config:
+
+```yaml
+model:
+  multitask_metrics_cfg:
+    log_predictions: true
+    metrics:
+      wer:
+        _target_: nemo.collections.asr.metrics.WER
+        constraint: ".source_lang==.target_lang"
+      bleu:
+        _target_: nemo.collections.asr.metrics.BLEU
+        constraint: ".source_lang!=.target_lang"
+        bleu_tokenizer: 13a
+        check_cuts_for_bleu_tokenizers: false
+```
+
+Use constraints to route ASR samples to WER and translation samples to BLEU. Add dataset/task/domain metadata through
+manifest fields or `input_cfg.tags`, then reference it in constraints such as `.domain==target` or
+`.task==asr and .source_lang==.target_lang`.
+
+Current implementation supports only one instance of each metric class in a single `multitask_metrics_cfg`. For
+multiple WER slices by language/domain, prefer separate validation manifests/dataloaders or extend metric aggregation
+rather than defining duplicate WER metrics.
+
+For AED validation data, set `use_lhotse: true`, `use_bucketing: false`, static `batch_size`, `text_field: "text"`,
+and `lang_field: "target_lang"`.

.claude/skills/nemo-speech-asr-finetune/references/data-lhotse.md

@@ -0,0 +1,255 @@
+# Stage 2: Data, Lhotse, Bucketing, And Blends
+
+Use Lhotse by default for training and validation. For validation, prefer Lhotse with static `batch_size` and no
+bucketing; dynamic validation batches can rarely starve DDP ranks.
+
+## Manifest Preparation
+
+Standard ASR JSONL:
+
+```json
+{"audio_filepath": "/data/audio/sample.wav", "text": "transcript text", "duration": 3.42}
+```
+
+Use separate train, validation, and test manifests. Prefer 16 kHz mono audio unless the model card/config says
+otherwise.
+
+## Transcript Style Preflight
+
+Do this before sharding, tokenizer changes, OOMptimizer, or fine-tuning. Treat mixed transcript style as an experiment
+setup failure.
+
+For every train, validation, and test manifest, and for each source inside a blend, audit a representative sample of
+reference transcripts. At minimum, report:
+
+- Fraction with uppercase letters.
+- Fraction with punctuation or symbols, including sentence punctuation, quotes, dashes, and brackets.
+- Number, abbreviation, currency, unit, and special-symbol conventions.
+- Unicode normalization and script/language consistency.
+- A few examples before and after the intended text transform.
+
+The target style must be consistent within and across all fine-tuning sources, and the validation/test style must match
+the metric being optimized. Do not rely on blend weights to average away style conflicts. If one source is cased and
+punctuated and another is lowercase/no-punctuation, fix the manifests or split the task before launching training.
+
+Also check the original checkpoint's prediction style when applicable:
+
+- First read the model card/config/docs for punctuation, capitalization, ITN, language, and prompt behavior.
+- If the checkpoint can already transcribe the target language or a close proxy, run a small baseline transcription on
+  representative fine-tuning audio and compute the same style rates on predictions.
+- If the checkpoint is being moved to a new language/script and baseline predictions are unreliable, do not infer style
+  from bad transcripts; record the checkpoint style as documented or unknown.
+
+If all fine-tuning data is internally consistent but differs from the original checkpoint's prediction style, that can
+be acceptable. Flag it to the user because the model may shift output style and raw WER may move even when recognition
+improves. In autonomous runs, consider adapting the fine-tuning transcripts to the checkpoint's output style when that
+style is required by deployment or evaluation. If that requires adding missing punctuation/case/ITN labels, treat the
+new labels as generated data: validate them and keep the raw manifests.
+
+Choose one target style and materialize new manifests before training:
+
+- Normalized ASR style: lowercase if appropriate for the language, remove punctuation/symbols that are not part of the
+  target transcript, normalize whitespace and Unicode, and score with the same normalization.
+- Cased/punctuated/ITN style: use data that already has reliable labels in that style, or restore missing style labels
+  with a validated post-processing pipeline. Do not mix unstyled labels into this target style without relabeling.
+- Prompted AED/Canary style: use task/language/punctuation prompts only when the selected script and metrics support
+  separating those behaviors; otherwise still enforce one transcript style per fine-tuning objective.
+
+Preserve raw manifests. Write derived manifests with a clear suffix such as `_style_normalized.json`, record the exact
+transform, and include the style audit in the run ledger.
+
+Shard training manifests even when not tarred. For resume-heavy fine-tuning with an unsharded non-tarred manifest,
+each restart can begin iterating from the start again. Split into shards like `manifest_0.json` ... `manifest_N.json`
+with roughly 200 utterances per shard. This is less important for one-shot runs that will not be interrupted.
+
+Strongly prefer tarred data for slow filesystems and object stores. If data is small or there is abundant fast local
+SSD, non-tarred audio is fine, but still shard the manifest. Do not use tarred datasets for validation/test unless the
+target script and docs explicitly support the behavior you need.
+
+Before training, inspect duration and token-per-second distributions. Set `min_duration`, `max_duration`, `min_tps`,
+and `max_tps` so they do not silently filter out a large or important part of the fine-tuning set.
+
+If OOMptimizer cannot fit batch size 1 for an extreme duration bucket, inspect the duration tail before changing the
+model or precision. When only a tiny outlier fraction is affected, set `max_duration` to cap that tail, record the
+number and hours filtered, and rerun OOMptimizer with the capped final bucket. Do not silently cap substantial or
+domain-critical long-form data.
+
+When duration-tail capping is used, include the chosen cap as the final duration bucket passed to OOMptimizer. This
+keeps the last bucket aligned with the longest sample that training will actually admit.
+
+Run the first OOMptimizer pass with its default CLI settings. Do not lower `--memory-fraction`, disable DDP simulation,
+change dtype, or adjust the search threshold unless there is a concrete reason. The default profile is intended to be
+aggressive enough for high GPU utilization while reserving non-training-loop memory. If the resulting real training run
+OOMs, then rerun OOMptimizer with a lower `--memory-fraction` and document why.
+
+Validate the generated profile with a short multi-GPU pilot and sample GPU utilization during training steps, not just
+startup, checkpointing, or validation. OOMptimizer sizes worst-case synthetic batches at the bucket upper bounds; real
+training memory can be noticeably lower when the sampler draws shorter cuts or smaller buckets. Focus on peak training
+memory and sustained SM utilization over enough steps to exercise several buckets.
+
+Sample utilization while training is inside the train loop:
+
+```bash
+nvidia-smi --query-gpu=index,utilization.gpu,utilization.memory,memory.used,memory.total \
+  --format=csv,noheader,nounits
+```
+
+Do not judge utilization from checkpoint save, validation, export, dataloader prefill, or process shutdown windows.
+Record peak memory and sustained SM utilization in the run ledger. If memory remains much lower than expected, first
+confirm the run is using the OOMptimizer `bucket_batch_size` profile and that conflicting batch settings are null.
+
+For non-tarred data, Lhotse may tokenize samples during sampling when `pretokenize=true`. This enables token-per-second
+filtering and 2D token bucketing, but tokenization happens in the main training process and can slow training with large
+tokenizers. If the run does not use token-per-second filters or 2D bucketing, consider `pretokenize=false`.
+
+## Batch Mode Compatibility
+
+When changing Lhotse batch settings, explicitly null conflicting options. `dataloader.py` accepts these batch sizing
+modes:
+
+- Static batch size: set `batch_size=<N>`, `use_bucketing=false`, and set `batch_duration=null`,
+  `quadratic_duration=null`, `bucket_duration_bins=null`, `bucket_batch_size=null`, `batch_tokens=null`.
+- OOMptimizer profile: set `bucket_duration_bins=[...]` and `bucket_batch_size=[...]`, and set `batch_size=null`,
+  `batch_duration=null`, `quadratic_duration=null`, `batch_tokens=null`.
+- Heuristic dynamic duration batching: set `batch_duration=<seconds>`, optionally `quadratic_duration=<seconds>`, and
+  set `bucket_batch_size=null`, `batch_tokens=null`. Prefer OOMptimizer instead.
+
+`bucket_batch_size` requires `bucket_duration_bins`. Setting either auto-enables `use_bucketing=true`, but set it
+explicitly for clarity.
+
+## Training Lhotse Defaults
+
+Use integer `val_check_interval`, `limit_train_batches` as pseudo-epoch length, and `max_steps` as the real duration.
+Do not use `trainer.max_epochs`.
+
+```bash
+++model.train_ds.use_lhotse=true \
+++model.train_ds.use_bucketing=true \
+++model.train_ds.batch_size=null \
+++model.train_ds.batch_duration=null \
+++model.train_ds.quadratic_duration=null \
+++model.train_ds.bucket_duration_bins='[...]' \
+++model.train_ds.bucket_batch_size='[...]' \
+++model.train_ds.bucket_buffer_size=10000 \
+++model.train_ds.shuffle_buffer_size=10 \
+++trainer.use_distributed_sampler=false \
++trainer.limit_train_batches=1000 \
+trainer.val_check_interval=1000 \
+trainer.max_steps=<steps>
+```
+
+Validation should use Lhotse but static batches:
+
+```bash
+++model.validation_ds.use_lhotse=true \
+++model.validation_ds.use_bucketing=false \
+model.validation_ds.batch_size=8 \
+++model.validation_ds.batch_duration=null \
+++model.validation_ds.quadratic_duration=null \
+++model.validation_ds.bucket_duration_bins=null \
+++model.validation_ds.bucket_batch_size=null \
+model.validation_ds.shuffle=false
+```
+
+Use Hydra `++` for dataloader keys that are not declared in the selected YAML. This commonly applies to Lhotse-specific
+validation keys such as `use_lhotse`, `use_bucketing`, `batch_duration`, `bucket_duration_bins`, and sometimes
+`min_duration`/`max_duration`.
+
+## Bucketing Policy
+
+- For CTC/RNNT/TDT, prefer 1D duration bucketing.
+- For AED/Canary, prefer 2D bucketing over duration and token count.
+- Strongly prefer OOMptimizer-generated `bucket_batch_size` over manually tuning `batch_duration` and
+  `quadratic_duration`.
+- If all utterances are fixed length, disable bucketing and use static `batch_size`.
+- For very small fine-tuning datasets around 100 hours, especially on one GPU, consider disabling bucketing and using
+  fully random static `batch_size`.
+
+1D bins and OOMptimizer:
+
+```bash
+python scripts/speech_recognition/estimate_duration_bins.py -b 30 /data/train_input_cfg.yaml
+python scripts/speech_recognition/oomptimizer.py \
+  --pretrained-name nvidia/parakeet-tdt-0.6b-v2 \
+  --buckets '[2.0,3.1,5.6,8.4,12.0,18.0,30.0]'
+```
+
+If `max_duration` was capped to 30.0, make sure 30.0 is present as the final bucket.
+
+2D AED/Canary bins and OOMptimizer:
+
+```bash
+python scripts/speech_recognition/estimate_duration_bins_2d.py \
+  --prompt-format canary \
+  --prompt "[{'role':'user','slots':{'source_lang':'en','target_lang':'en','pnc':'yes'}}]" \
+  --tokenizer /data/tokenizers/spl_tokens/tokenizer.model /data/tokenizers/en/tokenizer.model \
+  --langs spl_tokens en \
+  --buckets 30 \
+  --sub-buckets 2 \
+  /data/train_input_cfg.yaml
+
+python scripts/speech_recognition/oomptimizer.py \
+  --config-path examples/asr/conf/speech_multitask/fast-conformer_aed.yaml \
+  --module-name nemo.collections.asr.models.EncDecMultiTaskModel \
+  --buckets '[[3.9,30],[3.9,48],[5.0,37]]'
+```
+
+Nested `bucket_duration_bins` automatically activate 2D bucketing.
+
+## Buffers, Memory, And RNG
+
+- `bucket_buffer_size=10000` is a good default.
+- With bucketing enabled, set `shuffle_buffer_size=10`; it is added on top of the bucket buffer.
+- With bucketing disabled, set `shuffle_buffer_size=1000` to `10000`, but do not rely on it as the only source of
+  randomness. Shard the data and blend datasets when applicable.
+- With tarred data, larger buffers and more workers directly increase CPU memory pressure. Segfaults, CPU OOM, or
+  unexplained dataloader errors often indicate CPU memory pressure.
+- `seed` controls base dataloading RNGs.
+- `shard_seed` controls sharded/tarred randomization. Default `shard_seed="trng"` gives different non-reproducible
+  orders per run, rank, and worker, but is simpler to manage. Use `shard_seed="randomized"` with a managed `seed` when deterministic
+  dataloading is needed. For 100% determinism, disable `concurrent_bucketing=false` at the cost of longer tarred data bucket prefill.
+
+## Data Blends
+
+Use Lhotse `input_cfg` for mixed datasets rather than concatenating manifests blindly.
+
+Before running helper scripts such as duration-bin estimation, OOMptimizer, or data-weight estimation, validate the
+input YAML shape they expect in the current checkout. Some scripts consume a list-form input config directly, while
+training configs may wrap the same list under `input_cfg:`. Run a tiny dry run or inspect the script help before a long
+OOMptimizer pass.
+
+For a large generic dataset plus a smaller domain dataset, either manually upweight the domain data or estimate weights
+then apply temperature reweighting. Lower temperature oversamples smaller datasets; `1.0` is neutral.
+
+For small-domain adaptation, target-domain real data should usually dominate synthetic or heavily augmented data.
+Synthetic data can help coverage, but it can also dilute the real-domain signal. Start conservatively, e.g. keep
+synthetic sources below roughly 30% total weight, evaluate ablations, and only increase synthetic weight when standalone
+domain WER improves. This applies especially when the synthetic data is generated from TTS, noisy augmentation, or
+generic prompts rather than real target-domain audio.
+
+```bash
+python scripts/speech_recognition/estimate_data_weights.py \
+  generic.yaml domain.yaml blended.yaml \
+  --temperature 0.5 \
+  --strategy num_hours
+```
+
+Example:
+
+```yaml
+input_cfg:
+  - type: nemo
+    manifest_filepath: /data/generic/manifest__OP_0..512_CL_.json
+    weight: 0.7
+    tags:
+      domain: generic
+  - type: nemo
+    manifest_filepath: /data/domain/manifest__OP_0..128_CL_.json
+    weight: 0.3
+    tags:
+      domain: target
+```
+
+For tarred data, use `type: nemo_tarred` with matching `manifest_filepath` and `tarred_audio_filepath`. Avoid mixing
+tarred and non-tarred inputs in one Lhotse multi-source setup unless the current dataloader docs say the selected mode
+supports it.

.claude/skills/nemo-speech-asr-finetune/references/evaluation-style-contract.md

@@ -0,0 +1,86 @@
+# Evaluation Style Contract
+
+Use this reference whenever reporting WER/CER for an ASR fine-tune.
+
+## Default Metric
+
+Unless the user explicitly requests a different metric, evaluate WER with capitalization and punctuation removed. This
+default keeps model quality comparisons focused on lexical recognition instead of punctuation/casing style drift.
+
+Still preserve raw prediction manifests. Raw WER is useful as a diagnostic, especially when the fine-tuning labels use
+a different style than the original checkpoint, but it is not the default success metric for this skill.
+
+Change the default only when the user asks for raw/cased/punctuated scoring. If the target product requires
+punctuation, casing, ITN, or symbolic formatting to be part of the ASR metric, treat that as an explicit user
+requirement and record the changed metric contract.
+
+## Two-Step Evaluation
+
+First transcribe with stable decoding, no AMP, and bfloat16 compute:
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best.nemo \
+  dataset_manifest=/data/test.json \
+  output_filename=/exp/asr-ft/test_predictions.json \
+  batch_size=32 \
+  amp=false \
+  compute_dtype=bfloat16 \
+  matmul_precision=high
+```
+
+For RNNT/TDT, enable CUDA graphs when supported:
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best.nemo \
+  dataset_manifest=/data/test.json \
+  output_filename=/exp/asr-ft/test_predictions.json \
+  batch_size=32 \
+  amp=false \
+  compute_dtype=bfloat16 \
+  matmul_precision=high \
+  rnnt_decoding.strategy=greedy_batch \
+  rnnt_decoding.greedy.use_cuda_graph_decoder=true
+```
+
+Then score the saved predictions with the default text-processing contract:
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  dataset_manifest=/exp/asr-ft/test_predictions.json \
+  only_score_manifest=true \
+  text_processing.do_lowercase=true \
+  text_processing.rm_punctuation=true \
+  use_cer=false
+```
+
+The WER printed by the first transcription command is raw unless the same `text_processing` overrides were passed to
+that command. Treat the score-only command above as the default metric.
+
+If the user requested raw scoring, run score-only without `text_processing.do_lowercase` and
+`text_processing.rm_punctuation`, and label the result as raw WER.
+
+## Punctuation Coverage
+
+`text_processing.rm_punctuation=true` removes the punctuation marks configured in
+`text_processing.punctuation_marks`. If predictions contain punctuation or symbols outside that list, either:
+
+- Override `text_processing.punctuation_marks` with the full set relevant to the target style.
+- Or create a derived scored manifest where both `text` and `pred_text` are transformed by the same documented
+  normalizer, then score that manifest.
+
+Use the second option when the target style removes broad Unicode punctuation/symbol categories, normalizes whitespace,
+or applies language-specific text normalization that NeMo's built-in punctuation helper does not express.
+
+## Reporting
+
+For each model variant, report:
+
+- Default WER/CER from score-only evaluation with capitalization and punctuation removed.
+- Raw WER/CER only when it helps explain style mismatch or the user requested it.
+- The exact prediction manifest and text-processing settings.
+- Whether the target references were raw labels or derived style-normalized labels.
+
+If raw WER and default WER differ substantially, treat that as a transcript-style finding before changing training
+hyperparameters. Do not claim an acoustic/model regression from raw WER alone when the default normalized WER improved.

.claude/skills/nemo-speech-asr-finetune/references/refinement-iteration.md

@@ -0,0 +1,132 @@
+# Stage 5: Refinement And Iteration
+
+Use this stage after the first fine-tuning run has a standalone evaluation result. The goal is to decide what to change
+next from evidence, not from guesswork.
+
+## Evaluation Matrix
+
+Always compare:
+
+- Baseline pretrained model on the domain validation/test set.
+- Fine-tuned model on the same domain validation/test set.
+
+Add a general or out-of-domain guardrail set only when it matches the user's goal, such as preserving same-language
+general ASR quality, broad production behavior, or a known existing benchmark. If the user is intentionally changing
+language, task, script, domain, or deployment scope, do not assume a generic guardrail is meaningful; pick validation
+sets that reflect the desired behavior.
+
+Report standalone `speech_to_text_eval.py` WER for each row. In-training `val_wer` is only for checkpoint selection.
+For small-domain adaptation, avoid optimizing only one tiny target set when the model must still work broadly; track the
+smallest goal-relevant guardrail set that would reveal unacceptable regressions.
+
+Example tracking table:
+
+| Run | Checkpoint | Target WER | Optional Guardrail WER | Notes |
+| --- | --- | --- | --- | --- |
+| baseline | pretrained |  |  | no fine-tune |
+| ft-001 | best single |  |  | first pass |
+| ft-001-avg | averaged |  |  | keep only if better |
+
+## Error Analysis Loop
+
+1. Transcribe the held-out domain set with the current best checkpoint.
+2. Compute per-utterance WER/CER and sort from worst to best.
+3. Before treating raw WER as acoustic/model quality, rescore with the chosen transcript normalization and compare
+   raw vs the default capitalization/punctuation-normalized WER. This is mandatory. A large gap usually means the
+   Stage 2 transcript-style preflight or the evaluation-style contract is incomplete.
+4. Categorize errors by actionable patterns: numbers, named entities, abbreviations, rare domain words, commands,
+   readbacks, punctuation/capitalization, language/task tags, accents, noise, long utterances, and clipping/VAD issues.
+5. Count errors by category and inspect representative audio. Separate model errors from label/audio defects.
+6. Choose one or two interventions for the next run. Avoid changing LR, data mix, tokenizer, and decoding all at once.
+
+Do not train on validation or test transcripts. If error analysis uses a public or user-visible test set to guide new
+data generation, create a new blind holdout before claiming final quality.
+
+## Intervention Choices
+
+Prefer the least invasive intervention that matches the error pattern:
+
+- Data issue: fix labels/audio, remove broken samples, adjust `min_duration`, `max_duration`, `min_tps`, or `max_tps`.
+- Rare vocabulary or entities: add more real examples if available; otherwise add carefully reviewed synthetic examples.
+- Overfitting or regression: lower LR, reduce `max_steps`, stop at an earlier checkpoint, or increase a generic/guardrail
+  blend only when preserving that behavior is part of the user's objective.
+- Domain underfitting: raise target-domain real-data weight, add targeted data, or run a lower-LR domain-focus phase.
+- Decoding issue: compare decoder options, prompts, punctuation/capitalization settings, or CTC/RNNT head for hybrids.
+- Tokenization issue: revisit tokenizer only when transcript language/domain coverage cannot be represented well by the
+  existing tokenizer.
+
+## Source-Weighted Rebalancing
+
+Use source-weighted rebalancing when standalone evaluation shows that a few train sources, domains, or acoustic styles
+are underfitting and those sources are important to the user's goal. Do not use it to chase a public test set without a
+new blind holdout.
+
+Recipe:
+
+1. Ensure every sample has a stable source/domain tag, either in manifest metadata such as `source_dataset` or in
+   Lhotse `input_cfg.tags`.
+2. Run standalone evaluation and compute source-wise WER using the default evaluation-style contract.
+3. Decide whether weights should reflect user priority, source-wise validation errors, dataset hours, or a blend of
+   those signals.
+4. Apply a floor so low-error sources are not removed entirely, and cap extreme weight jumps unless the user explicitly
+   wants domain-only adaptation.
+5. Write a new Lhotse `input_cfg` with normalized weights. Prefer changing weights over duplicating manifests.
+6. Re-run duration-bin estimation and OOMptimizer, or at least verify the previous profile is still valid when the
+   source union is unchanged.
+7. Fine-tune from the current best checkpoint with a lower LR refinement phase, commonly `1e-5` or `5e-6`.
+8. Compare the previous best, the rebalanced final checkpoint, the best individual checkpoint, and any averaged model
+   with the same standalone scoring contract.
+
+When using `estimate_data_weights.py`, validate the YAML shape expected by the current script before running it. If the
+new weights were influenced by validation/test performance, record that in the ledger. For final claims, evaluate on a
+holdout that did not influence the reweighting decision.
+
+## Targeted Synthetic Data
+
+Synthetic data is most useful when it fills a measured gap. Generate small, targeted batches for the worst categories
+instead of flooding the run with generic synthetic audio.
+
+Recommendations:
+
+- Keep synthetic text TTS-friendly: expand symbols and ambiguous abbreviations when needed, and avoid text forms that a
+  synthesizer will read incorrectly.
+- Match target-domain acoustics only when the target deployment needs them; generic noise can hurt.
+- Filter synthetic audio with ASR or manual spot checks before adding it to training.
+- Add synthetic data as a separately weighted Lhotse input source so it can be ablated.
+- For small-domain adaptation, keep real target-domain audio dominant unless standalone WER proves otherwise.
+
+## Run Ledger
+
+Use `assets/experiment-ledger-template.md` as the starting template for experiment notes.
+
+Maintain a compact run ledger with:
+
+- Data sources and blend weights.
+- Transcript style policy, any manifest transform, and whether it differs from the original checkpoint output style.
+- LR, `max_steps`, warmup, precision, and batch profile.
+- Duration/token filters and number of examples filtered.
+- Final `.nemo`, best individual checkpoint-derived artifact, averaged artifact, and the keep/drop decision.
+- Raw WER/CER and default capitalization/punctuation-normalized WER/CER when both are informative.
+- Target-set WER and any goal-relevant guardrail WER.
+- GPU utilization evidence from training steps, not validation/checkpoint/export windows.
+- Decision for the next run.
+
+## Late-Stage Curriculum Pattern
+
+Use curriculum-style staged fine-tuning only after simpler refinements stop improving standalone WER. Try data cleanup,
+filter fixes, blend-weight changes, targeted data additions, decoding choices, and tokenizer decisions first.
+
+For small-data domain adaptation, use short lower-LR phases instead of one long aggressive run:
+
+1. Foundation phase: preserve required broad behavior with a mix of source and target-domain data when that is a goal.
+2. Domain-focus phase: increase real target-domain and targeted data weight, lower LR.
+3. Final refinement phase: lower LR again and evaluate carefully; this phase can regress.
+
+Typical starting points:
+
+- First small-domain phase: `model.optim.lr=3e-5`.
+- Follow-up domain-focus phase: `model.optim.lr=1e-5`.
+- Final refinement phase: `model.optim.lr=5e-6` or lower.
+
+Use `trainer.max_steps` for each phase, checkpoint by `val_wer`, and run standalone evaluation after each phase. Keep
+the best phase, not necessarily the last phase.

.claude/skills/nemo-speech-asr-finetune/references/setup-checkpoints.md

@@ -0,0 +1,98 @@
+# Stage 1: Setup And Checkpoint Selection
+
+Default to the NeMo release container. Use local execution only when the user explicitly asks for it.
+
+Check `README.md`, `docker/Dockerfile.speech`, and the NGC catalog before quoting container tags. The container avoids
+most local CUDA, torch, audio backend, `sox`, `ffmpeg`, and `libsndfile` issues.
+
+When running repo scripts from a mounted source checkout inside a container, ensure Python imports the mounted checkout,
+not an unrelated package already installed in the image:
+
+```bash
+export PYTHONPATH=/workspace/NeMo
+python -c "import nemo.collections.asr as asr; print(asr.__file__)"
+```
+
+If the source checkout must be installed in the container, prefer `pip install -e '.[asr]'`.
+
+For Hugging Face models or datasets that require authentication, verify access before launching long jobs. Containers
+often do not include `huggingface-cli`; mounting a populated `HF_HOME` or token/cache directory is usually enough:
+
+```bash
+docker run -e HF_HOME=/tmp/hf_home -v /host/hf_home:/tmp/hf_home ...
+```
+
+For explicit local setup, verify:
+
+```bash
+python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
+python -c "import nemo.collections.asr as asr; print(asr.__file__)"
+nvidia-smi
+```
+
+For a source checkout local install, prefer:
+
+```bash
+pip install -e '.[asr]'
+```
+
+If CUDA, torch, audio backend, or dependency issues appear, steer back to the container.
+
+## Preflight Checks
+
+Before launching a long fine-tune, spend a few minutes on cheap failure checks:
+
+- Confirm the intended NeMo checkout is imported from inside the container.
+- Confirm each training/validation manifest exists, has non-empty `text`, valid `audio_filepath`, and usable
+  `duration` values.
+- Run the transcript-style preflight in `data-lhotse.md` before training. Mixed casing, punctuation, number, or symbol
+  conventions across fine-tuning sources are an experiment setup issue, not a tuning detail.
+- Inspect duration and token-per-second distributions before setting filters; Lhotse filtering is silent.
+- Check GPU state with `nvidia-smi` and stop stale training processes before sizing batches.
+- Check free disk space for experiment logs and checkpoints. Large `.nemo` exports, `save_top_k`, checkpoint averaging,
+  and EMA can produce tens of GB per run.
+- Confirm `HF_HOME`/tokens and any object-store credentials are visible inside the container before launching jobs that
+  will download checkpoints or data.
+
+Useful quick checks:
+
+```bash
+nvidia-smi
+df -h /exp /tmp /data
+python - /data/train.json <<'PY'
+import json, pathlib, statistics, sys
+manifest = pathlib.Path(sys.argv[1])
+durations, missing_audio, empty_text = [], 0, 0
+for line in manifest.open():
+    item = json.loads(line)
+    if not pathlib.Path(item["audio_filepath"]).exists():
+        missing_audio += 1
+    if not item.get("text", "").strip():
+        empty_text += 1
+    if "duration" in item:
+        durations.append(float(item["duration"]))
+print({"items": len(durations), "missing_audio": missing_audio, "empty_text": empty_text})
+if durations:
+    print({"hours": sum(durations) / 3600, "min": min(durations), "p50": statistics.median(durations), "max": max(durations)})
+PY
+```
+
+## Checkpoint Candidates
+
+Use the user's constraints first: language, streaming vs offline, latency, memory budget, punctuation/capitalization,
+translation, speaker overlap, CTC alignments, and deployment target.
+
+Good candidates from local docs:
+
+- `nvidia/parakeet-tdt-0.6b-v3` or `nvidia/parakeet-tdt-0.6b-v2`: general Parakeet/TDT fine-tuning. Check the
+  model card for exact language and punctuation support.
+- `nvidia/parakeet-tdt_ctc-110m`: compact hybrid candidate.
+- `nvidia/parakeet-ctc-0.6b` or `nvidia/parakeet-ctc-1.1b`: CTC when simple non-autoregressive decoding, alignments,
+  or vocabulary replacement matter.
+- `nvidia/nemotron-speech-streaming-en-0.6b`: real-time English streaming.
+- `nvidia/multitalker-parakeet-streaming-0.6b-v1`: streaming multi-speaker input.
+- `nvidia/canary-1b-v2`: AED/Canary for multilingual ASR, speech translation, and prompt-controlled behavior.
+- `nvidia/canary-1b-flash` or `nvidia/canary-180m-flash`: faster Canary-family models.
+
+Always check current `README.md`, `docs/source/asr/featured_models.rst`, `docs/source/asr/asr_checkpoints.rst`, and
+the model card or HF collection before finalizing the checkpoint list.

.claude/skills/nemo-speech-asr-finetune/references/training-evaluation.md

@@ -0,0 +1,216 @@
+# Stage 4: Training, Checkpoint Averaging, And Evaluation
+
+## Optimizer And Trainer
+
+Use `trainer.max_steps`, not `trainer.max_epochs`. Use a cosine LR schedule with the same `max_steps` as the trainer.
+Choose the initial LR by data size and risk:
+
+- For large or mixed fine-tuning runs, start with `model.optim.lr=1e-4`, then tune.
+- For small domain adaptation, especially below roughly 20 hours of high-value target-domain audio, start around
+  `3e-5` and watch early validation closely.
+- For later refinement phases or unstable/diverging runs, try `1e-5` or lower.
+
+Set warmup to 1-2% of `trainer.max_steps`.
+
+```bash
++init_from_pretrained_model=<hf-or-ngc-name> \
+trainer.max_steps=50000 \
++trainer.limit_train_batches=1000 \
+trainer.val_check_interval=1000 \
+model.optim.lr=1e-4 \
+model.optim.sched.name=CosineAnnealing \
+++model.optim.sched.max_steps=50000 \
+model.optim.sched.warmup_steps=500 \
+model.optim.sched.min_lr=5e-6
+```
+
+`examples/asr/speech_to_text_finetune.py` supports `init_from_pretrained_model`, but some fine-tune YAMLs do not
+declare that key. Use `+init_from_pretrained_model=...` when Hydra says the key is not in struct. Use `+` or `++` for
+other trainer/model keys that are valid for the script but absent from the selected YAML; do not remove the plus just
+because a similar key exists in another config.
+
+Prefer `trainer.precision=bf16-true` for memory savings and larger batch sizes, especially for datasets over 1k
+hours. If it diverges or has stability issues, fall back to a more stable precision mode. Prefer `bf16-true` over
+`bf16-mixed` as the first bfloat16 option.
+
+Monitor `val_wer` for all validation runs and checkpoint selection:
+
+```bash
+exp_manager.checkpoint_callback_params.monitor=val_wer \
+exp_manager.checkpoint_callback_params.mode=min \
+exp_manager.checkpoint_callback_params.save_top_k=5 \
+exp_manager.checkpoint_callback_params.always_save_nemo=true
+```
+
+## Multi-GPU Launch Troubleshooting
+
+For routine single-node runs, start with the normal Lightning launch (`trainer.devices=<num_gpus>`). If a container run
+hangs immediately after NCCL registration or one rank restores the model on the wrong GPU before DDP takes ownership,
+use `torchrun --nproc_per_node=<num_gpus>` with the normal script entry point. Keep `trainer.devices=<num_gpus>` and
+set `trainer.use_distributed_sampler=false` for Lhotse.
+
+Container `torchrun` template:
+
+```bash
+docker run --rm --gpus all --ipc=host \
+  --ulimit memlock=-1 --ulimit stack=67108864 \
+  -e PYTHONPATH=/workspace/NeMo \
+  -v /path/to/NeMo:/workspace/NeMo \
+  -v /data:/data \
+  nemo-speech:<tag> bash -lc '
+    cd /workspace/NeMo &&
+    torchrun --nproc_per_node=2 examples/asr/speech_to_text_finetune.py \
+      +init_from_pretrained_model=nvidia/parakeet-tdt-0.6b-v3 \
+      model.train_ds.manifest_filepath=/data/train.json \
+      model.validation_ds.manifest_filepath=/data/val.json \
+      ++model.train_ds.use_lhotse=true \
+      ++model.validation_ds.use_lhotse=true \
+      +trainer.use_distributed_sampler=false \
+      trainer.devices=2 \
+      trainer.max_steps=10000
+  '
+```
+
+Keep `trainer.num_nodes` for real multi-node jobs only. For single-node container fine-tuning, avoid manual
+one-visible-GPU-per-process launch patterns unless the user explicitly asks to debug that environment.
+
+On systems with NCCL P2P or CUDA memory allocator issues, retry with conservative distributed environment settings
+after confirming the job is stuck in communication setup:
+
+```bash
+export NCCL_CUMEM_ENABLE=0
+export NCCL_P2P_DISABLE=1
+```
+
+Keep `trainer.sync_batchnorm` the same as it was set in the original model's config unless the user explicitly asks to
+change it.
+
+## Checkpoint Averaging
+
+At the end of a run, optionally average the N best checkpoints saved by `save_top_k=N`. Then evaluate the averaged
+model and keep it only if it beats the best individual checkpoint on validation/test.
+
+The simple `.nemo` averaging utility averages all non-`-last.ckpt` checkpoints in the same folder as the `.nemo` file,
+so control N with `save_top_k`:
+
+```bash
+python scripts/checkpoint_averaging/checkpoint_averaging.py \
+  /exp/asr-ft/checkpoints/best.nemo
+```
+
+This produces `*-averaged.nemo`. If model class loading fails, use the script's `--class_path` or `--import_fname_list`
+options. Some checkpoint averaging scripts are marked deprecated in the repo; verify they still work in the current
+checkout before relying on them.
+
+With PyTorch 2.6+, the deprecated averaging utility may fail because `torch.load` defaults to `weights_only=True`.
+Only for checkpoints you trust, rerun with:
+
+```bash
+TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 python scripts/checkpoint_averaging/checkpoint_averaging.py \
+  /exp/asr-ft/checkpoints/best.nemo
+```
+
+Do not assume the latest `.nemo` is the best validation checkpoint. `always_save_nemo=true` can overwrite the `.nemo`
+artifact on later saves. Check the `.ckpt` filenames and validation logs, then evaluate the exported `.nemo`, the best
+checkpoint-derived artifact, and any averaged model before deciding what to keep.
+
+Decision rule: evaluate the final exported `.nemo`, the best individual validation checkpoint-derived artifact, and the
+averaged artifact with the same standalone command and scoring contract. Keep the artifact with the best standalone
+default WER. If averaging is worse, record it and drop the averaged artifact.
+
+Always run the same standalone evaluation command for the baseline model, the final exported `.nemo`, the best
+checkpoint-derived artifact, and any averaged model. In-training `val_wer` is useful for checkpoint selection, but it is
+not the final number to report. Standalone WER is the fair comparison because it holds decoding options, text
+processing, precision, and output scoring constant.
+
+Evaluate both:
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best.nemo \
+  dataset_manifest=/data/val.json \
+  output_filename=/exp/asr-ft/val_best.json \
+  batch_size=32 \
+  amp=false \
+  compute_dtype=bfloat16
+
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best-averaged.nemo \
+  dataset_manifest=/data/val.json \
+  output_filename=/exp/asr-ft/val_avg.json \
+  batch_size=32 \
+  amp=false \
+  compute_dtype=bfloat16
+```
+
+## Evaluation
+
+Do not use AMP for inference/evaluation. Use `compute_dtype=bfloat16` and `amp=false`. Report standalone
+`speech_to_text_eval.py` results for every model variant being compared; do not report only trainer logs.
+
+Use `evaluation-style-contract.md` for scoring. By default, report WER with capitalization and punctuation removed via
+score-only evaluation of the saved prediction manifest. Report raw WER separately only when the user requests it or it
+helps diagnose transcript-style mismatch.
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best.nemo \
+  dataset_manifest=/data/test.json \
+  output_filename=/exp/asr-ft/test_predictions.json \
+  batch_size=32 \
+  amp=false \
+  compute_dtype=bfloat16 \
+  matmul_precision=high \
+  use_cer=False
+```
+
+For RNNT/TDT evaluation, enable CUDA graphs when supported:
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best.nemo \
+  dataset_manifest=/data/test.json \
+  output_filename=/exp/asr-ft/test_predictions.json \
+  batch_size=32 \
+  amp=false \
+  compute_dtype=bfloat16 \
+  matmul_precision=high \
+  rnnt_decoding.strategy=greedy_batch \
+  rnnt_decoding.greedy.use_cuda_graph_decoder=true
+```
+
+For a hybrid model, compare decoder choices:
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best.nemo \
+  dataset_manifest=/data/test.json \
+  decoder_type=ctc \
+  output_filename=/exp/asr-ft/test_predictions_ctc.json \
+  amp=false \
+  compute_dtype=bfloat16
+
+python examples/asr/speech_to_text_eval.py \
+  model_path=/exp/asr-ft/checkpoints/best.nemo \
+  dataset_manifest=/data/test.json \
+  decoder_type=rnnt \
+  output_filename=/exp/asr-ft/test_predictions_rnnt.json \
+  amp=false \
+  compute_dtype=bfloat16 \
+  rnnt_decoding.strategy=greedy_batch \
+  rnnt_decoding.greedy.use_cuda_graph_decoder=true
+```
+
+If predictions already exist:
+
+```bash
+python examples/asr/speech_to_text_eval.py \
+  dataset_manifest=/exp/asr-ft/test_predictions.json \
+  only_score_manifest=True \
+  text_processing.do_lowercase=true \
+  text_processing.rm_punctuation=true \
+  use_cer=False
+```
+
+Use `examples/asr/transcribe_speech.py` for direct offline transcription and streaming or chunked inference scripts for
+streaming models.

.claude/skills/verify/SKILL.md

@@ -0,0 +1,28 @@
+---
+name: verify
+description: Run style checks and tests on changed files to verify code quality before committing.
+---
+
+Run verification on the current changes:
+
+1. **Find changed files**:
+   ```bash
+   git diff --name-only HEAD
+   ```
+   Also include any files you've been editing in this session.
+
+2. **Style check** on each changed Python file or its parent directory:
+   ```bash
+   python setup.py style --scope <path>
+   ```
+   If issues are found, fix them with `--fix` and report what changed.
+
+3. **Run relevant tests** based on which collection was modified:
+   - `nemo/collections/asr/` → `pytest tests/collections/asr --download -m "not pleasefixme" -v --timeout=300`
+   - `nemo/collections/tts/` → `pytest tests/collections/tts --download -m "not pleasefixme" -v --timeout=300`
+   - `nemo/collections/audio/` → `pytest tests/collections/audio --download -m "not pleasefixme" -v --timeout=300`
+   - `nemo/collections/speechlm2/` → `pytest tests/collections/speechlm2 -m "not pleasefixme" -v --timeout=300`
+   - `nemo/collections/common/` → `pytest tests/collections/common -m "not pleasefixme" -v --timeout=300`
+   - `nemo/core/` → `pytest tests/core -m "not pleasefixme" -v --timeout=300`
+
+4. **Report results**: summarize passes and failures. For failures, show relevant error output and suggest fixes.

.codex/skills

@@ -0,0 +1 @@
+../.claude/skills

.coveragerc

@@ -0,0 +1,40 @@
+[run]
+concurrency = thread,multiprocessing
+omit =
+    /tmp/*
+    /home/TestData/*
+    /workspace/Megatron-LM/*
+    nemo/collections/multimodal/*
+    nemo/collections/multimodal_autoregressive/*
+    nemo/collections/vision/*
+    nemo/collections/diffusion/*
+    nemo/collections/nlp/*
+
+    nemo/collections/asr/*
+    nemo/collections/speechlm/*
+    nemo/collections/tts/*
+
+    # omit from audio
+    nemo/collections/audio/data/data_simulation.py
+    nemo/collections/audio/metrics/squim.py
+    nemo/collections/audio/losses/maxine/*
+    nemo/collections/audio/models/maxine/*
+    nemo/collections/audio/parts/utils/maxine.py
+
+    nemo/core/*
+    nemo/collections/common/*
+
+    /workspace/config-3.12.py
+    /workspace/config-3.py
+    /workspace/config.py
+    /workspace/_remote_module_non_scriptable*
+
+[report]
+omit =
+    /workspace/_remote_module_non_scriptable*
+
+[paths]
+source =
+    nemo/
+    /home/runner/work/NeMo/NeMo/nemo
+    /workspace/nemo

.cursor/skills

@@ -0,0 +1 @@
+../.claude/skills

.dockerignore

@@ -0,0 +1,19 @@
+__pycache__
+*.pyc
+*.pyo
+*.pyd
+.Python
+env
+pip-log.txt
+pip-delete-this-directory.txt
+.tox
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+*.log
+.git
+**/*.nemo
+**/*.ckpt

.flake8

@@ -0,0 +1,9 @@
+[flake8]
+max-line-length = 119
+select = 
+    F541, # f-string without any placeholders
+    F841, # local variable 'x' is assigned to but never used
+    F401, # 'x' imported but unused
+    E741, # ambiguous variable name 'l'
+    F821, # undefined name 'x'
+    E266, # too many leading '#' for block comment

.flake8.other

@@ -0,0 +1,9 @@
+[flake8]
+max-line-length = 119
+select = 
+    F541, # f-string without any placeholders
+    F841, # local variable 'x' is assigned to but never used
+    F401, # 'x' imported but unused
+    E741, # ambiguous variable name 'l'
+    F821, # undefined name 'x'
+    E266, # too many leading '#' for block comment

.flake8.speech

@@ -0,0 +1,9 @@
+[flake8]
+max-line-length = 119
+select = 
+    F541, # f-string without any placeholders
+    F841, # local variable 'x' is assigned to but never used
+    F401, # 'x' imported but unused
+    E741, # ambiguous variable name 'l'
+    F821, # undefined name 'x'
+    E266, # too many leading '#' for block comment

.github/CODEOWNERS

@@ -0,0 +1,6 @@
+.github/ @nvidia-nemo/automation
+docker/ @nvidia-nemo/automation
+requirements/ @nvidia-nemo/automation
+pyproject.toml @nvidia-nemo/automation
+setup.py @nvidia-nemo/automation
+uv.lock @nvidia-nemo/automation

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,42 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+
+A clear and concise description of what the bug is.
+
+**Steps/Code to reproduce bug**
+
+Please list *minimal* steps or code snippet for us to be able to reproduce the bug.
+
+A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 
+
+
+**Expected behavior**
+
+A clear and concise description of what you expected to happen.
+
+**Environment overview (please complete the following information)**
+
+ - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
+ - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
+ - If method of install is [Docker], provide `docker pull` & `docker run` commands used
+
+**Environment details**
+
+If NVIDIA docker image is used you don't need to specify these.
+Otherwise, please provide:
+- OS version
+- PyTorch version
+- Python version
+
+**Additional context**
+
+Add any other context about the problem here.
+Example: GPU model

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,2 @@
+blank_issues_enabled: false
+

.github/ISSUE_TEMPLATE/dev_container_bug_report.md

@@ -0,0 +1,35 @@
+---
+container pulled on date: mm/dd/yyyy
+name: Dev container - Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+
+A clear and concise description of what the bug is.
+
+**Steps/Code to reproduce bug**
+
+Please list *minimal* steps or code snippet for us to be able to reproduce the bug.
+
+A  helpful guide on on how to craft a minimal bug report  http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports. 
+
+
+**Expected behavior**
+
+A clear and concise description of what you expected to happen.
+
+**Environment overview (please complete the following information)**
+
+ - Environment location: Docker
+ - Method of install: Please specify exact commands you used to install.
+ - If method of install is [Docker], provide `docker pull` & `docker run` commands used
+
+**Additional context**
+
+Add any other context about the problem here.
+Example: GPU model

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,25 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: feature request
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+
+A clear and concise description of what you want to happen.
+Provide a code snippet on how new APIs/changes would be used by others.
+
+**Describe alternatives you've considered**
+
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+
+Add any other context or screenshots about the feature request here.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,57 @@
+> [!IMPORTANT]  
+> The `Update branch` button must only be pressed in very rare occassions.
+> An outdated branch is never blocking the merge of a PR.
+> Please reach out to the automation team before pressing that button.
+
+# What does this PR do ?
+
+Add a one line overview of what this PR aims to accomplish.
+
+**Collection**: [Note which collection this PR will affect]
+
+# Changelog
+
+- Add specific line by line info of high level changes in this PR.
+
+# Usage
+
+- You can potentially add a usage example below
+
+```python
+# Add a code snippet demonstrating how to use this 
+```
+
+# GitHub Actions CI
+
+The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
+
+The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
+To re-run CI remove and add the label again.
+To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
+
+# Before your PR is "Ready for review"
+
+**Pre checks**:
+
+- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
+- [ ] Did you write any new necessary tests?
+- [ ] Did you add or update any necessary documentation?
+- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
+  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
+  
+**PR Type**:
+
+- [ ] New Feature
+- [ ] Bugfix
+- [ ] Documentation
+
+If you haven't finished some of the above items you can still open "Draft" PR.
+
+## Who can review?
+
+Anyone in the community is free to review the PR once the checks have passed.
+[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
+
+# Additional Information
+
+- Related to # (issue)

.github/actions/cancel-workflow/action.yml

@@ -0,0 +1,25 @@
+name: Cancel Workflow
+description: >
+  Cancels the current workflow run, i.e. all jobs. Useful if you want to cancel the rest of the workflow when one job
+  fails. Note that this will cause the workflow to appear cancelled, not failed.
+
+# Cancelling the workflow in a post-script (like this:
+# https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#runspost; can also be done with
+# this action: https://github.com/webiny/action-post-run, see Git history of this file) wouldn't help the status, it
+# would still be cancelled. It actually indeed is, but it would be nicer to set it to failed, but there seems to be no
+# way to do this.
+
+runs:
+  using: "composite"
+  steps:
+    - name: Cancel Workflow
+      # # Fork PRs won't have a token with write access to Actions, thus won't be able to cancel the workflow.
+      # if: github.event.pull_request == '' || github.event.pull_request.head.repo.fork == false
+      shell: bash
+      run: |
+        curl --verbose \
+          -X POST \
+          -H "Accept: application/vnd.github+json" \
+          -H "Authorization: Bearer ${{ github.token }}" \
+          -H "X-GitHub-Api-Version: 2022-11-28" \
+          https://api.github.com/repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/cancel

.github/actions/test-template/action.yml

@@ -0,0 +1,234 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: "Test Template"
+description: "Template for running NeMo tests in a containerized environment"
+
+inputs:
+  runner:
+    description: "Runner to use for test"
+    required: true
+  timeout:
+    description: "Max runtime of test in minutes"
+    required: false
+    default: "10"
+  script:
+    description: "Test script to execute"
+    required: true
+  after_script:
+    description: "Script to run after main test"
+    required: false
+    default: ":"
+  is_optional:
+    description: "Failure will cancel all other tests if set to true"
+    required: false
+    default: "false"
+  is_unit_test:
+    description: "Upload coverage as unit test"
+    required: false
+    default: "false"
+  tests_to_run:
+    description: "Tests to run"
+    required: false
+    default: '["all"]'
+  image:
+    description: "Full image URL with tag (e.g., 766267172432.dkr.ecr.us-east-1.amazonaws.com/nemo-speech:nemo_container-12345)"
+    required: true
+  test-data-path:
+    description: "Host path to mount as /home/TestData inside the container"
+    required: false
+    default: /mnt/datadrive/TestData
+  cpu-only:
+    description: "Run tests on CPU only"
+    required: false
+    default: "false"
+  test_dir:
+    description: "Directory under tests/ containing the test scripts"
+    required: false
+    default: "functional_tests"
+runs:
+  using: "composite"
+  steps:
+    - name: Noop
+      shell: bash
+      run: |
+        chmod -R u+rwX ${{ github.run_id }}
+        echo "noop"
+
+    - name: Docker system cleanup
+      shell: bash
+      run: |
+        docker system prune -af --filter "until=24h" --filter "label!=nemo.pr_number=${{ github.event.pull_request.number || 0 }}" --force || true
+
+    - name: Docker pull image
+      shell: bash
+      run: |
+        docker pull ${{ inputs.image }}
+
+    - name: Clean repos
+      shell: bash
+      run: |
+
+    - name: Create UUID
+      id: uuid
+      shell: bash
+      run: |
+        echo "id=$(uuidgen)" >> "$GITHUB_OUTPUT"
+
+    - name: Checkout NeMo
+      uses: actions/checkout@v6
+      env:
+        DIR: ${{ github.run_id }}
+      with:
+        path: ${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo
+
+    - name: Start container
+      shell: bash
+      env:
+        DIR: ${{ github.run_id }}
+      run: |
+        mkdir -p $DIR
+
+        # Map of runner names to GPU device configurations
+        declare -A GPU_CONFIGS=(
+          ["myVm-01"]="0,1"
+          ["myVm-02"]="2,3"
+          ["myVm-03"]="4,5"
+          ["myVm-04"]="6,7"
+        )
+
+        ARG=("")
+        if [[ "${{ inputs.cpu-only }}" == "false" ]]; then
+          ARG=("--runtime=nvidia --gpus all")
+        fi
+
+        cmd=$(cat <<RUN_TEST_EOF
+        #!/bin/bash
+        docker container rm -f nemo_container_${{ github.run_id }}_${{ inputs.runner }} || true
+        docker run \
+          --rm \
+          -d \
+          --name nemo_container_${{ github.run_id }}_${{ inputs.runner }} ${ARG[@]} \
+          --shm-size=64g \
+          --env TRANSFORMERS_OFFLINE=0 \
+          --env NEMO_HOME="/home/TestData/nemo_home" \
+          --env HYDRA_FULL_ERROR=1 \
+          --env HF_HOME=/home/TestData/HF_HOME \
+          --env RUN_ID=${{ github.run_id }} \
+          --volume $(pwd)/${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo:/workspace \
+          --volume ${{ inputs.test-data-path }}:/home/TestData ${{ inputs.image }} \
+          bash -c "sleep $(( ${{ inputs.timeout }} * 60 + 60 ))"
+        RUN_TEST_EOF
+        )
+
+        echo "$cmd" | tee "$DIR/retry_job.sh"
+        bash $DIR/retry_job.sh
+
+    - name: Create run-script
+      id: create
+      env:
+        DIR: ${{ github.run_id }}
+      shell: bash
+      run: |
+        COVERAGE_PREFIX=$([[ "${{ inputs.is_unit_test }}" == "true" ]] && echo "unit-test" || echo "e2e")
+        echo "coverage-prefix=$COVERAGE_PREFIX" | tee -a "$GITHUB_OUTPUT"
+
+        mkdir -p $DIR
+        rm $DIR/.coverage || true
+        rm $DIR/err.log || true
+
+        cmd=$(cat <<RUN_TEST_EOF
+        #!/bin/bash
+
+        (
+          set -e
+
+          docker exec -t nemo_container_${{ github.run_id }}_${{ inputs.runner }} bash -c '\
+            bash tests/${{ inputs.test_dir }}/${{ inputs.script }}.sh && \
+            echo "Finished successfully." || echo "Did not finish."'
+        ) 2>&1 | tee $DIR/err.log
+
+        RUN_TEST_EOF
+        )
+
+        echo "timeout_in_seconds=$(( ${{ inputs.timeout }} * 60 ))" | tee -a "$GITHUB_OUTPUT"
+        echo "$cmd" | tee "$DIR/job.sh"
+
+    - name: Run main script
+      uses: nick-fields/retry@v3
+      with:
+        timeout_seconds: ${{ steps.create.outputs.timeout_in_seconds }}
+        max_attempts: 3
+        shell: bash
+        retry_on: timeout
+        command: /bin/bash ${{ github.run_id }}/job.sh
+        on_retry_command: /bin/bash ${{ github.run_id }}/retry_job.sh
+
+    - name: Check result
+      id: check
+      shell: bash
+      env:
+        DIR: ${{ github.run_id }}
+      run: |
+        cat $DIR/err.log
+
+        log=$(tail -c 2000 $DIR/err.log |  base64 -w 0)
+        echo "log=$log" >> "$GITHUB_OUTPUT"
+
+        potential_infra_failure=$(cat $DIR/err.log | grep -Eqiw "device" && echo true || echo false)
+        echo "potential_infra_failure=$potential_infra_failure" >> "$GITHUB_OUTPUT"
+
+        docker exec nemo_container_${{ github.run_id }}_${{ inputs.runner }} coverage combine
+        docker exec nemo_container_${{ github.run_id }}_${{ inputs.runner }} coverage xml
+        docker cp nemo_container_${{ github.run_id }}_${{ inputs.runner }}:/workspace/.coverage $DIR/.coverage
+        docker cp nemo_container_${{ github.run_id }}_${{ inputs.runner }}:/workspace/coverage.xml $DIR/coverage.xml
+
+        coverage_report=coverage-${{ steps.create.outputs.coverage-prefix }}-${{ github.run_id }}-$(python3 -c "import uuid; print(uuid.uuid4())")
+        echo "coverage_report=$coverage_report" >> "$GITHUB_OUTPUT"
+
+        IS_SUCCESS=$(tail -n 1 $DIR/err.log | grep -q "Finished successfully." && echo "true" || echo "false")
+
+        if [[ "$IS_SUCCESS" == "false" && "${{ inputs.is_optional }}" == "true" ]]; then
+          echo "::warning:: Test failed, but displayed as successful because it is marked as optional."
+          IS_SUCCESS=true
+        fi
+
+        if [[ "$IS_SUCCESS" == "false" ]]; then
+          echo Test did not finish successfully.
+          exit 1
+        fi
+
+        exit $EXIT_CODE
+
+    - name: Test coverage
+      shell: bash -x -e -u -o pipefail {0}
+      run: |
+        docker exec -t nemo_container_${{ github.run_id }}_${{ inputs.runner }} coverage report -i
+
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v6
+      if: ${{ steps.check.outputs.coverage_report != 'none' }}
+      with:
+        name: ${{ steps.check.outputs.coverage_report }}
+        path: |
+          ${{ github.run_id }}/coverage.xml
+          ${{ github.run_id }}/.coverage
+        include-hidden-files: true
+
+    - name: Container shutdown
+      if: always()
+      shell: bash
+      run: |
+        docker exec nemo_container_${{ github.run_id }}_${{ inputs.runner }} bash -c "chown -R $(id -u):$(id -g) /workspace"
+        rm -rf $(pwd)/${{ github.run_id }}/${{steps.uuid.outputs.id }} || true
+        docker container rm -f nemo_container_${{ github.run_id }}_${{ inputs.runner }} || true

.github/copy-pr-bot.yaml

@@ -0,0 +1,3 @@
+enabled: true
+auto_sync_draft: false
+auto_sync_ready: true

.github/labeler.yml

@@ -0,0 +1,41 @@
+ASR:
+- nemo/collections/asr/**/*
+- examples/asr/**/*
+- tutorials/asr/**/*
+- docs/source/asr/**/*
+- tests/collections/asr/**
+
+Speaker Tasks:
+- examples/speaker_tasks/**/*
+- tutorials/speaker_tasks/**/*
+
+TTS:
+- nemo/collections/tts/**/*
+- nemo/collections/common/tokenizers/text_to_speech/**
+- examples/tts/**/*
+- tutorials/tts/**/*
+- docs/source/tts/**/*
+- scripts/dataset_processing/tts/**
+- scripts/tts_dataset_files/**
+- tests/collections/tts/**
+- tests/collections/common/tokenizers/text_to_speech/**
+
+Audio:
+- nemo/collections/audio/**/*
+- examples/audio/**/*
+- tutorials/audio/**/*
+- docs/source/audio/**/*
+- tests/collections/audio/**
+
+core:
+- nemo/core/**/*
+- tests/core/**
+
+common:
+- nemo/collections/common/**/*
+
+CI:
+- .github/**/*
+- Jenkinsfile
+- Dockerfile
+- ci.groovy

.github/scripts/notify.py

@@ -0,0 +1,79 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+
+import requests
+from github import Github
+
+
+def send_slack_notification():
+    # Get environment variables
+    gh_token = os.environ.get('GH_TOKEN')
+    webhook_url = os.environ.get('SLACK_WEBHOOK')
+    repository = os.environ.get('REPOSITORY')
+    run_id = os.environ.get('RUN_ID')
+    server_url = os.environ.get('SERVER_URL', 'https://github.com')
+    pr_number = int(os.environ.get('PR_NUMBER', 0))
+    branch_name = os.environ.get('BRANCH_NAME')
+
+    # Get failure info from GitHub API
+    gh = Github(gh_token)
+    repo = gh.get_repo(repository)
+
+    # Get failed jobs
+    failed_jobs = [job.name for job in repo.get_workflow_run(int(run_id)).jobs() if job.conclusion == 'failure']
+
+    if pr_number != 0:
+        pr = repo.get_pull(pr_number)
+
+        title = f"*<{server_url}/{repository}/pull/{pr_number}|PR#{pr_number}>: {pr.title.replace('`', '')}*"
+        author = f"<{server_url}/{pr.user.login}|{pr.user.login}>"
+        branch = f"<{server_url}/{pr.head.repo.full_name}/tree/{pr.head.ref}|{pr.head.ref}>"
+    else:
+        title = f"*Run on <{server_url}/{repository}/tree/{branch_name}|{branch_name}>*"
+        author = "No author"
+        branch = f"<{server_url}/{repository}/tree/{branch_name}|{branch_name}>"
+
+    blocks = [
+        {
+            "type": "section",
+            "text": {
+                "type": "mrkdwn",
+                "text": (
+                    f"{title}\n"
+                    f"• Author: {author}\n"
+                    f"• Branch: {branch}\n"
+                    f"• Pipeline: <{server_url}/{repository}/actions/runs/{run_id}|View Run>\n"
+                    f"• Failed Jobs:\n"
+                    + "\n".join(
+                        [
+                            f"    • <{server_url}/{repository}/actions/runs/{run_id}|{job.split('/')[-1]}>"
+                            for job in failed_jobs
+                            if job.split('/')[-1] != 'Nemo_CICD_Test'
+                        ]
+                    )
+                ),
+            },
+        }
+    ]
+
+    print({"blocks": blocks})
+
+    # Send to Slack
+    response = requests.post(webhook_url, json={"blocks": blocks})
+    response.raise_for_status()
+
+
+if __name__ == "__main__":
+    send_slack_notification()

.github/workflows/_build_container.yml

@@ -0,0 +1,111 @@
+name: ~Build container template
+on:
+  workflow_call:
+    inputs:
+      image-name:
+        required: true
+        type: string
+        description: "The name of the image to build"
+      dockerfile:
+        required: true
+        type: string
+      runner:
+        required: false
+        default: nemo-ci-aws-gpu-x2
+        type: string
+        description: "The runner to use for the build"
+      registry:
+        required: false
+        default: 766267172432.dkr.ecr.us-east-1.amazonaws.com
+        type: string
+        description: "Container registry"
+    outputs:
+      image:
+        description: "Full image URL (registry/nemo-speech:image-name-run_id)"
+        value: ${{ jobs.build.outputs.image }}
+
+permissions:
+  id-token: write
+  contents: read
+
+jobs:
+  pre-flight:
+    runs-on: ubuntu-latest
+    outputs:
+      build_args: ${{ steps.manifest.outputs.BUILD_ARGS }}
+      cache-from: ${{ steps.cache_from.outputs.LAST_PRS }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Get last merged PR
+        id: cache_from
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          LAST_PRS=$(gh api graphql -f query='
+            query {
+              repository(owner: "NVIDIA", name: "NeMo") {
+                pullRequests(states: MERGED, first: 100, orderBy: {field: UPDATED_AT, direction: DESC}) {
+                  nodes {
+                    number
+                  }
+                }
+              }
+            }' | jq -r '.data.repository.pullRequests.nodes[].number' | while read -r number; do
+              echo "type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-$number,mode=max"
+            done)
+
+          echo "LAST_PRS<<EOF" | tee -a $GITHUB_OUTPUT
+          echo "$LAST_PRS" | tee -a $GITHUB_OUTPUT
+          echo "EOF" | tee -a $GITHUB_OUTPUT
+
+  build:
+    runs-on: ${{ inputs.runner }}
+    needs: [pre-flight]
+    outputs:
+      image: ${{ steps.image-tag.outputs.image }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Compute image tag
+        id: image-tag
+        run: |
+          echo "image=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-${{ github.run_id }}" | tee -a $GITHUB_OUTPUT
+
+      - name: Compute cache keys
+        id: cache-keys
+        run: |
+          PR_NUMBER="${{ github.event.pull_request.number }}"
+          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
+            KEY="main"
+          elif [[ -n "$PR_NUMBER" ]]; then
+            KEY="$PR_NUMBER"
+          else
+            KEY=$(echo "${{ github.ref_name }}" | tr '/' '-' | tr -cd '[:alnum:]._-')
+          fi
+          echo "cache-to=type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-${KEY},mode=max" | tee -a $GITHUB_OUTPUT
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build and push
+        uses: docker/build-push-action@v5
+        with:
+          file: ${{ inputs.dockerfile }}
+          push: true
+          context: .
+          build-args: |
+            IMAGE_LABEL=nemo-core
+            NEMO_TAG=${{ github.sha }}
+            NEMO_REPO=https://github.com/NVIDIA/NeMo
+            PR_NUMBER=${{ github.event.pull_request.number || 0 }}
+          cache-from: |
+            type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-main,mode=max
+            type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-${{ github.event.pull_request.number || 0 }},mode=max
+            ${{ needs.pre-flight.outputs.cache-from }}
+          cache-to: ${{ steps.cache-keys.outputs.cache-to }}
+          tags: |
+            ${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-${{ github.run_id }}
+            ${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-${{ github.sha }}

.github/workflows/_bump_mcore_tag.yml

@@ -0,0 +1,56 @@
+name: ~Bump Megatron Tag template
+on:
+  workflow_call:
+    inputs:
+      nemo-target-branch:
+        required: true
+        type: string
+        description: "The target branch to bump"
+      mcore-target-branch:
+        required: true
+        type: string
+        description: "The target branch to bump"
+    secrets:
+      PAT:
+        required: true
+
+jobs:
+  update-branch:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          ref: ${{ inputs.nemo-target-branch }}
+
+      - name: Set Git config
+        run: |
+          git config --local user.email "actions@github.com"
+          git config --local user.name "Github Actions"
+      - name: Merge weekly-bump-${{ inputs.nemo-target-branch }} back to base branch
+        env:
+          SOURCE_BRANCH: weekly-bump-${{ inputs.nemo-target-branch }}
+          TARGET_BRANCH: ${{ inputs.nemo-target-branch }}
+        run: |
+          if git ls-remote --exit-code origin $SOURCE_BRANCH; then
+            git fetch --unshallow
+            git checkout $SOURCE_BRANCH
+            git pull
+            git merge --no-ff $TARGET_BRANCH -m "chore: Auto-merge $TARGET_BRANCH into $SOURCE_BRANCH"
+          else
+            git checkout -b $SOURCE_BRANCH $TARGET_BRANCH
+          fi
+          git push -u origin $SOURCE_BRANCH
+
+  mcore:
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_bump_yamlfile.yml@v0.27.1
+    needs: [update-branch]
+    with:
+      source-repository: NVIDIA/Megatron-LM
+      source-ref: ${{ inputs.mcore-target-branch }}
+      yaml-path: '."vcs-dependencies"."megatron-lm".ref'
+      file: requirements/manifest.json
+      base-branch: weekly-bump-${{ inputs.nemo-target-branch }}
+      cicd-labels: Run CICD,no-fail-fast
+      pr-reviewers: ${{ inputs.pr-reviewers }}
+    secrets:
+      PAT: ${{ secrets.PAT }}

.github/workflows/cherry-pick-release-commit.yml

@@ -0,0 +1,14 @@
+name: Create PR to main with cherry-pick from release
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  cherry-pick:
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cherry_pick.yml@v0.63.0
+    secrets:
+      PAT: ${{ secrets.PAT }}
+      SLACK_WEBHOOK_ADMIN: ${{ secrets.SLACK_WEBHOOK_ADMIN }}
+      SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}

.github/workflows/cicd-approve-test-queue.yml

@@ -0,0 +1,245 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Approve Test Queue
+
+on:
+  schedule:
+    - cron: '*/5 * * * *'  # Runs every 5 minutes
+  workflow_dispatch:  # Allows manual triggering
+
+jobs:
+  approve-queue:
+    runs-on: ubuntu-latest
+    environment: main
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install requests
+
+      - name: Approve waiting deployments
+        env:
+          GITHUB_TOKEN: ${{ secrets.PAT }}
+          MAX_CONCURRENCY: ${{ vars.MAX_CONCURRENCY || 1 }}
+        run: |
+          python - <<EOF
+          import os
+          import requests
+
+
+          # GitHub API configuration
+          GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
+          REPO = os.environ["GITHUB_REPOSITORY"]
+          MAX_CONCURRENCY = int(os.environ["MAX_CONCURRENCY"])
+          API_BASE = f"https://api.github.com/repos/{REPO}"
+
+          # Headers for GitHub API
+          headers = {
+              "Authorization": f"token {GITHUB_TOKEN}",
+              "Accept": "application/vnd.github.v3+json",
+              "X-GitHub-Api-Version": "2022-11-28",
+          }
+
+          def make_request(endpoint, method="GET", data=None, allow_no_pending_deployments=False):
+              """Make a request to the GitHub API with error handling."""
+              url = f"{API_BASE}/{endpoint}"
+              try:
+                  if method == "GET":
+                      response = requests.get(url, headers=headers)
+                  else:
+                      response = requests.post(url, headers=headers, json=data)
+                  response.raise_for_status()
+                  response_json = response.json()
+                  if hasattr(response, "links") and "actions/runs?status" in endpoint:
+                      response_json["next"] = response.links.get("next", {}).get("url")
+
+                  return response_json
+              except requests.exceptions.HTTPError as e:
+                  if (
+                      allow_no_pending_deployments
+                      and e.response is not None
+                      and e.response.status_code == 422
+                  ):
+                      try:
+                          response_json = e.response.json()
+                      except ValueError:
+                          response_json = {}
+                      if "No pending deployment requests to approve or reject" in str(response_json.get("errors", "")):
+                          print(f"No pending deployment requests remain for {endpoint}; skipping")
+                          return {"skipped": True, "reason": "no_pending_deployments"}
+
+                  print(f"Error making request to {endpoint}: {str(e)}")
+                  if e.response is not None:
+                      print(f"Response: {e.response.text}")
+                  return None
+              except requests.exceptions.RequestException as e:
+                  print(f"Error making request to {endpoint}: {str(e)}")
+                  if e.response is not None:
+                      print(f"Response: {e.response.text}")
+                  return None
+
+
+          def get_workflow_runs(status):
+              """Get all workflow runs for a given status."""
+              all_results = []
+              endpoint = f"actions/runs?status={status}"
+              while endpoint:
+                  response = make_request(endpoint)
+                  if not response:
+                      break
+
+                  all_results.extend(response.get("workflow_runs", []))
+                  endpoint = None
+                  next_url = response.get("next")
+                  if next_url:
+                      endpoint = f"actions/runs?{next_url.split('?')[1]}"
+
+              return all_results
+
+
+          def filter_cicd_runs(workflow_runs):
+              """Keep only CICD workflow runs."""
+              return [run for run in workflow_runs if run.get("name") == "CICD NeMo"]
+
+
+          def print_workflow_run_details(label, workflow_runs):
+              """Print the runs that are counted against concurrency."""
+              if not workflow_runs:
+                  print(f"{label}: none")
+                  return
+
+              print(f"{label}:")
+              for run in workflow_runs:
+                  print(
+                      "  "
+                      f"id={run.get('id')} "
+                      f"status={run.get('status')} "
+                      f"branch={run.get('head_branch')} "
+                      f"title={run.get('display_title')}"
+                  )
+
+
+          # Get current running and queued workflows
+          print("Fetching workflow runs...")
+          queued_workflow_runs = filter_cicd_runs(get_workflow_runs("queued"))
+          in_progress_workflow_runs = filter_cicd_runs(get_workflow_runs("in_progress"))
+          print_workflow_run_details("Queued CICD workflows counted against concurrency", queued_workflow_runs)
+          print_workflow_run_details("Running CICD workflows counted against concurrency", in_progress_workflow_runs)
+
+          # Count running and queued workflows
+          queued_workflows = len(queued_workflow_runs)
+          in_progress_workflows = len(in_progress_workflow_runs)
+
+          total_workflows = queued_workflows + in_progress_workflows
+          print(f"Current queued workflows: {queued_workflows}")
+          print(f"Current running workflows: {in_progress_workflows}")
+          print(f"Total workflows: {total_workflows}")
+          print(f"Max concurrency: {MAX_CONCURRENCY}")
+
+          if total_workflows >= MAX_CONCURRENCY:
+              print("Maximum concurrency reached, no new approvals will be made")
+              exit(0)
+
+          # Get waiting CI workflows for test environment
+          print("Fetching deployments...")
+          pending_workflows = filter_cicd_runs(get_workflow_runs("waiting"))
+
+          # Sort deployments by creation date (oldest first)
+          print("Sorting workflows...")
+          pending_workflows = sorted(pending_workflows, key=lambda x: x.get("created_at", ""))
+
+          # Process each deployment
+          print("Processing ...")
+          for workflow in pending_workflows:
+              if total_workflows >= MAX_CONCURRENCY:
+                  print("Maximum concurrency reached, stopping approvals")
+                  break
+
+              workflow_id = workflow.get("id")
+              workflow_name = workflow.get("display_title") or workflow.get("name") or "<unknown>"
+              if not workflow_id:
+                  print(f"Skipping workflow without a run id: {workflow_name}")
+                  continue
+              print(f"Approving workflow {workflow_name} with Run Id: {workflow_id}")
+
+              deployment_url = f"actions/runs/{workflow_id}/pending_deployments"
+              pending_deployments = make_request(deployment_url)
+              if not pending_deployments:
+                  print(f"No pending deployments found for workflow {workflow_name}; skipping")
+                  continue
+
+              environment_ids = []
+              environment_names = []
+              for deployment in pending_deployments:
+                  environment = deployment.get("environment") or {}
+                  environment_id = environment.get("id")
+                  environment_name = environment.get("name") or "<unknown>"
+                  if not environment_id:
+                      print(f"Skipping deployment without an environment id for workflow {workflow_name}")
+                      continue
+
+                  environment_ids.append(environment_id)
+                  environment_names.append(environment_name)
+
+              if not environment_ids:
+                  print(f"No pending deployments with environment ids found for workflow {workflow_name}")
+                  exit(1)
+
+              # Approve the deployment
+              status_data = {
+                  "environment_ids": environment_ids,
+                  "state": "approved",
+                  "comment": "Automatically approved by queue manager"
+              }
+              result = make_request(
+                  deployment_url,
+                  method="POST",
+                  data=status_data,
+                  allow_no_pending_deployments=True,
+              )
+
+              if result is None:
+                  print(f"Failed to approve environments {environment_names} for workflow {workflow_name}")
+                  exit(1)
+              if isinstance(result, dict) and result.get("skipped"):
+                  continue
+
+              total_workflows += 1
+
+          EOF
+  notify:
+    if: failure()
+    runs-on: ubuntu-latest
+    needs: [approve-queue]
+    steps:
+      - name: Notify
+        env:
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_WEBHOOK_ADMIN: <!subteam^${{ secrets.SLACK_WEBHOOK_ADMIN }}>
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+        run: |
+          curl -X POST \
+            -H 'Content-type: application/json' \
+            --data "{\"text\":\":robot_joy: <https://github.com/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}|Test-queue-approval-bot workflow> failed. Please review manually.\n\ncc ${SLACK_WEBHOOK_ADMIN}\"}" \
+            $SLACK_WEBHOOK

.github/workflows/cicd-main-speech.yml

@@ -0,0 +1,578 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: NeMo E2E Speech Tests
+on:
+  workflow_call:
+    inputs:
+      test_to_run:
+        required: true
+        type: string
+      image-name:
+        required: false
+        default: nemo_container_speech
+        type: string
+      runner:
+        required: false
+        default: nemo-ci-aws-gpu-x2
+        type: string
+        description: "Runner to use for GPU jobs"
+      run_nightly:
+        required: false
+        default: 'false'
+        type: string
+        description: "Set to 'true' to run the nightly e2e suite"
+
+jobs:
+  build:
+    uses: ./.github/workflows/_build_container.yml
+    with:
+      image-name: ${{ inputs.image-name }}
+      dockerfile: docker/Dockerfile
+      runner: ${{ inputs.runner }}
+
+  unit-tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - script: L0_Unit_Tests_GPU_ASR_1
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_GPU_ASR_2
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_GPU_ASR_3
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_GPU_ASR_4
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_GPU_ASR_5
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_GPU_ASR_6
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_CPU_ASR_1
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_ASR_2
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_ASR_3
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_ASR_4
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_ASR_5
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_ASR_6
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_GPU_TTS
+            runner: ${{ inputs.runner }}
+            timeout: 30
+          - script: L0_Unit_Tests_CPU_TTS
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+          - script: L0_Unit_Tests_GPU_Audio_1
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_GPU_Audio_2
+            runner: ${{ inputs.runner }}
+            timeout: 15
+          - script: L0_Unit_Tests_CPU_Audio
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 20
+          - script: L0_Unit_Tests_GPU_SpeechLM2
+            runner: ${{ inputs.runner }}
+            timeout: 20
+          - script: L0_Unit_Tests_CPU_SpeechLM2_1
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_SpeechLM2_2
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_SpeechLM2_3
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+          - script: L0_Unit_Tests_CPU_SpeechLM2_4
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+            timeout: 12
+    needs: [build]
+    runs-on: ${{ matrix.runner }}
+    name: ${{ matrix.script }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+      - name: main
+        uses: ./.github/actions/test-template
+        with:
+          runner: ${{ runner.name }}
+          script: ${{ matrix.script }}
+          is_unit_test: true
+          tests_to_run: ${{ inputs.test_to_run }}
+          image: ${{ needs.build.outputs.image }}
+          test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
+          timeout: ${{ matrix.timeout || 10 }}
+          cpu-only: ${{ matrix.cpu-only || false }}
+          is_optional: ${{ matrix.is-optional || false }}
+
+  e2e-tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ${{ inputs.runner }}
+            script: ASR_dev_run_Speech_to_Text
+          - runner: ${{ inputs.runner }}
+            script: Optional_ASR_dev_run_Speech_To_Text_Finetuning
+            is-optional: true
+          - runner: ${{ inputs.runner }}
+            script: Optional_ASR_dev_run_Speech_To_Text_HF_Finetuning
+            is-optional: true
+          - runner: ${{ inputs.runner }}
+            script: ASR_dev_run_Speech_to_Text_WPE_-_Conformer
+          - runner: ${{ inputs.runner }}
+            script: ASR_dev_run_Speech_to_Text_Hybrid_RNNT_CTC_Prompt
+          - runner: ${{ inputs.runner }}
+            script: ASR_dev_run_Speech_to_Text_RNNT_Prompt
+          - runner: ${{ inputs.runner }}
+            script: L2_ASR_Multi-dataloader_dev_run_Speech_to_Text_multi-dataloader
+          - runner: ${{ inputs.runner }}
+            script: L2_ASR_Multi-dataloader_dev_run_Speech_to_Label_multi-dataloader
+          - runner: ${{ inputs.runner }}
+            script: L2_ASR_Adapters_Linear_Adapters
+          - runner: ${{ inputs.runner }}
+            script: L2_ASR_Adapters_RelPos_MHA_Adapters
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_to_Text_EMA
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_to_Text_AED
+          - runner: ${{ inputs.runner }}
+            script: L2_Speaker_dev_run_Speech_to_Label
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Estimate_Duration_Bins
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Estimate_Token_Bins
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Batch_Size_OOMptimizer
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Distributed_Batch_Size_OOMptimizer
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: Optional_L2_Speech_Batch_Size_OOMptimizer_Canary
+            is-optional: true
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Speech_to_Text_Transcribe
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Speech_to_Text_Streaming_Infer
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Speech_to_Text_Cache_Aware_Infer
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Speech_to_Text_Cache_Aware_with_Prompt_Infer
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Streaming_Inference
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Speech_to_Text_Inference_Boost_GT
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Speech_to_Text_Transcribe_Boost_GT
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Canary_Transcribe_Full_Manifest
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Canary_Transcribe_With_Prompt
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Canary_Transcribe_Audio_Dir
+          - runner: ${{ inputs.runner }}
+            script: L2_Speech_Transcription_Canary_Streaming_Full_Manifest
+          - runner: ${{ inputs.runner }}
+            script: L2_Longform_Speech_Transcription_Canary_Chunked_Infer_from_Audio_Dir
+          - runner: ${{ inputs.runner }}
+            script: L2_Longform_Speech_Transcription_with_TimeStamps_Canary_Chunked_Infer_from_Audio_Dir
+          - runner: ${{ inputs.runner }}
+            script: L2_Longform_Speech_Transcription_with_TimeStamps_Canary_Chunked_Infer_from_Manifest
+          - runner: ${{ inputs.runner }}
+            script: Speech_Checkpoints_tests
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_Speaker_dev_run_Speaker_Recognition
+          - runner: ${{ inputs.runner }}
+            script: L2_Speaker_dev_run_EndtoEnd_Speaker_Diarization_Sortformer
+          - runner: ${{ inputs.runner }}
+            script: L2_Speaker_dev_run_EndtoEnd_Diarizer_Inference
+          - runner: ${{ inputs.runner }}
+            script: L2_Speaker_dev_run_Speaker_Diarization_with_ASR_Inference
+          - runner: ${{ inputs.runner }}
+            script: L2_Speaker_dev_run_Clustering_Diarizer_Inference
+          - runner: ${{ inputs.runner }}
+            script: L2_Speaker_dev_run_Multispeaker_ASR_Data_Simulation
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_1_FastPitch
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_1_Hifigan
+          - runner: ${{ inputs.runner }}
+            script: L2_G2P_Models_G2P_Conformer_training_evaluation_and_inference
+          - runner: ${{ inputs.runner }}
+            script: SPEECHLM_HF_Training_DuplexS2S
+          - runner: ${{ inputs.runner }}
+            script: SPEECHLM_HF_Training_DuplexS2SSpeechDecoder
+          - runner: ${{ inputs.runner }}
+            script: SPEECHLM_HF_Training_SALM
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_Magpietts_DecoderContext
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_Magpietts_MultiEncoder
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_Magpietts_MoE
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_Magpietts_OnlinePO
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_InferEvaluate_Magpietts_ZeroShot
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_InferEvaluate_Magpietts_SeenSpeakers
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_InferEvaluatelongform_Magpietts_ZeroShot
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_InferEvaluate_Magpietts_MoE_ZeroShot
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_InferEvaluatelongform_Magpietts_MoE_ZeroShot
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_InferEvaluate_Magpietts_FrameStacking
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_Magpietts_OnlineCFGDistillation
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_Magpietts_MoE_OnlineCFGDistillation
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_Magpietts_FrameStacking_OnlineCFGDistillation
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_EasyMagpietts_Qwen
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_EasyMagpietts_Nemotron
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_Fast_dev_runs_EasyMagpietts_OnlinePO
+            timeout: 20
+          - runner: ${{ inputs.runner }}
+            script: L2_TTS_InferEvaluate_EasyMagpietts
+            timeout: 20
+    needs: [build, unit-tests]
+    runs-on: ${{ matrix.runner }}
+    name: ${{ matrix.is-optional && 'PLEASEFIXME_' || '' }}${{ matrix.script }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+      - name: main
+        uses: ./.github/actions/test-template
+        with:
+          runner: ${{ runner.name }}
+          script: ${{ matrix.script }}
+          tests_to_run: ${{ inputs.test_to_run }}
+          image: ${{ needs.build.outputs.image }}
+          test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
+          timeout: ${{ matrix.timeout || 10 }}
+          is_optional: ${{ matrix.is-optional || false }}
+
+  e2e-nightly:
+    if: ${{ github.event_name == 'schedule' || inputs.run_nightly == 'true' }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_de_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_es_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_it_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_ua_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_pl_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_hr_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_be_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_fr_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_ru_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_nl_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_fa_fastconformer_hybrid_large
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_ka_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_kk_ru_fastconformer_hybrid_large
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_uz_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_ar_fastconformer_hybrid_large_pc_v1_0
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_hy_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_medium_streaming_80ms_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_medium_streaming_80ms
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_pt_fastconformer_hybrid_large_pc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_es_fastconformer_hybrid_large_pc_nc
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_ar_fastconformer_hybrid_large_pcd_v1_0
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_large_streaming_multi
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_large
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_large
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_xlarge
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_xlarge
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_xxlarge
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_xxlarge
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__stt_en_fastconformer_tdt_large
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_stt_en_fastconformer_hybrid_large_streaming_1040ms
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_stt_multilingual_fastconformer_hybrid_large_pc_blend_eu
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_rnnt_1_1b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_ctc_1_1b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_rnnt_0_6b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_ctc_0_6b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_tdt_1_1b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_tdt_ctc_1_1b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_tdt_ctc_0_6b_ja
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_tdt_ctc_110m
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_tdt_0_6b_v2
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_rnnt_110m_da_dk
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_tdt_0_6b_v3
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_ctc_0_6b_Vietnamese
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__canary_1b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__canary_1b_flash
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__canary_180m_flash
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__canary_1b_v2
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__parakeet_realtime_eou_120m_v1
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__multitalker_parakeet_streaming_0_6b_v1
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__nemotron_speech_streaming_en_0_6b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__canary_qwen_2_5b
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__diar_sortformer_4spk_v1
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__diar_streaming_sortformer_4spk_v2
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__diar_streaming_sortformer_4spk_v2_1
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_titanet_large
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__speakerverification_en_titanet_large
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__ssl_en_nest_large_v1_0
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__ssl_en_nest_xlarge_v1_0
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_vad_multilingual_marblenet
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_vad_multilingual_frame_marblenet
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__Frame_VAD_Multilingual_MarbleNet_v2_0
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__se_den_sb_16k_small
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__se_der_sb_16k_small
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__sr_ssl_flowmatching_16k_430m
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_mel_codec_44khz_medium
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_mel_codec_22khz_fullband_medium
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__low_frame_rate_speech_codec_22khz
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__audio_codec_22khz
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__audio_codec_44khz
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__mel_codec_22khz
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__mel_codec_44khz
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_1_78kbps_12_5fps
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_1_89kbps_21_5fps
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_0_6kbps_12_5fps
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__tts_en_fastpitch
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__tts_hifigan
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_nvidia__magpie_tts_multilingual_357m
+            timeout: 15
+          - runner: ${{ inputs.runner }}
+            script: L2_Model_Support_tts_en_e2e_fastspeech2hifigan
+            timeout: 15
+    needs: [build, unit-tests]
+    runs-on: ${{ matrix.runner }}
+    name: ${{ matrix.script }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+      - name: main
+        uses: ./.github/actions/test-template
+        with:
+          runner: ${{ runner.name }}
+          script: ${{ matrix.script }}
+          tests_to_run: ${{ inputs.test_to_run }}
+          image: ${{ needs.build.outputs.image }}
+          test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
+          timeout: ${{ matrix.timeout || 10 }}
+          test_dir: e2e_nightly

.github/workflows/cicd-main-unit-tests.yml

@@ -0,0 +1,126 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: NeMo Unit Tests
+on:
+  workflow_call:
+    inputs:
+      test_to_run:
+        required: true
+        type: string
+      runner:
+        required: false
+        default: nemo-ci-aws-gpu-x2
+        type: string
+        description: "Runner to use for GPU jobs"
+      container-image:
+        required: true
+        type: string
+        description: "Full image URL with tag for the NeMo container"
+
+jobs:
+  collections-common-tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - script: L0_Unit_Tests_GPU_Common
+            runner: ${{ inputs.runner }}
+          - script: L0_Unit_Tests_CPU_Common
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+    runs-on: ${{ matrix.runner }}
+    name: ${{ matrix.script }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+      - name: main
+        uses: ./.github/actions/test-template
+        with:
+          runner: ${{ runner.name }}
+          script: ${{ matrix.script }}
+          is_unit_test: true
+          tests_to_run: ${{ inputs.test_to_run }}
+          image: ${{ inputs.container-image }}
+          test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
+          cpu-only: ${{ matrix.cpu-only || false }}
+
+  core-tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - script: L0_Unit_Tests_GPU_Core
+            runner: ${{ inputs.runner }}
+          - script: L0_Unit_Tests_CPU_Core
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+          - script: L0_Unit_Tests_GPU_Hydra
+            runner: ${{ inputs.runner }}
+          - script: L0_Unit_Tests_CPU_Hydra
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+    runs-on: ${{ matrix.runner }}
+    name: ${{ matrix.script }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+      - name: main
+        uses: ./.github/actions/test-template
+        with:
+          runner: ${{ runner.name }}
+          script: ${{ matrix.script }}
+          is_unit_test: true
+          tests_to_run: ${{ inputs.test_to_run }}
+          image: ${{ inputs.container-image }}
+          test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
+          cpu-only: ${{ matrix.cpu-only || false }}
+
+  other-tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - script: L0_Unit_Tests_GPU_Others
+            runner: ${{ inputs.runner }}
+          - script: L0_Unit_Tests_CPU_Others
+            runner: ${{ inputs.runner }}
+            cpu-only: true
+    runs-on: ${{ matrix.runner }}
+    name: ${{ matrix.script }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+      - name: main
+        uses: ./.github/actions/test-template
+        with:
+          runner: ${{ runner.name }}
+          script: ${{ matrix.script }}
+          is_unit_test: true
+          tests_to_run: ${{ inputs.test_to_run }}
+          image: ${{ inputs.container-image }}
+          test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
+          cpu-only: ${{ matrix.cpu-only || false }}
+          is_optional: ${{ matrix.is-optional || false }}

.github/workflows/cicd-main.yml

@@ -0,0 +1,416 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: CICD NeMo
+on:
+  schedule:
+    - cron: 0 0 * * *
+  push:
+    branches:
+      - main
+      - r**
+      - weekly-bump*
+      - "pull-request/[0-9]+"
+  workflow_dispatch:
+    inputs:
+      test_to_run:
+        required: false
+        default: all
+        type: string
+        description: Comma-separated list of tests to run. Use "all" to run the full test suite.
+
+concurrency:
+  # group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.event.pull_request.number || github.ref }}-${{ github.event_name }}
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}
+  cancel-in-progress: true
+
+jobs:
+  pre-flight:
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cicd_preflight.yml@v0.89.0
+    with:
+      default_runner_prefix: ${{ vars.DEFAULT_RUNNER_PREFIX }}
+      non_nvidia_runner_prefix: ${{ vars.NON_NVIDIA_RUNNER_PREFIX }}
+      default_test_data_path: ${{ vars.DEFAULT_TEST_DATA_PATH }}
+      non_nvidia_test_data_path: ${{ vars.NON_NVIDIA_TEST_DATA_PATH }}
+      default_registry: ${{ vars.DEFAULT_CONTAINER_REGISTRY }}
+      non_nvidia_registry: ${{ vars.NON_NVIDIA_CONTAINER_REGISTRY }}
+      sso_users_filename: ${{ vars.SSO_USERS_FILENAME }}
+    secrets:
+      NVIDIA_MANAGEMENT_ORG_PAT: ${{ secrets.NVIDIA_MANAGEMENT_ORG_PAT }}
+
+  configure:
+    runs-on: ubuntu-latest
+    needs: [pre-flight]
+    outputs:
+      test_to_run: ${{ steps.test_to_run.outputs.main }}
+      is_ci_workload: ${{ steps.is_ci_workload.outputs.main }}
+      no_fail_fast: ${{ steps.no_fail_fast.outputs.main }}
+      components_to_run: ${{ steps.components_to_run.outputs.main }}
+      run_nightly: ${{ steps.pr_labels.outputs.run_nightly }}
+      skip_docs: ${{ steps.pr_labels.outputs.skip_docs }}
+      skip_linting: ${{ steps.pr_labels.outputs.skip_linting }}
+    env:
+      TESTS_TO_RUN: ${{ inputs.test_to_run }}
+      EVENT_NAME: ${{ github.event_name }}
+    steps:
+      - name: Checkout branch
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Select components to run
+        id: components_to_run
+        run: |
+          echo "main=[\"speech\"]" | tee -a "$GITHUB_OUTPUT"
+
+      - name: Select tests to run
+        id: test_to_run
+        run: |
+          # For manual dispatch, use the provided input
+          if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
+            TESTS_TO_RUN=$TESTS_TO_RUN
+
+          # For push (PR branches via copy-pr-bot, main, release) and schedule, run all tests
+          elif [[ "$EVENT_NAME" == "push" || "$EVENT_NAME" == "schedule" ]]; then
+            TESTS_TO_RUN=all
+
+          else
+            echo "Unsupported event_name $EVENT_NAME provided".
+            exit 1
+          fi
+
+          parsed_string=$(echo "$TESTS_TO_RUN" | jq -c --raw-input 'split(",")')
+          echo "main=${parsed_string}" | tee -a "$GITHUB_OUTPUT"
+
+      - name: Check if this is a CI workload
+        shell: bash
+        id: is_ci_workload
+        run: |
+          branch_name=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}
+
+          if [[ "$branch_name" =~ ^bump-ci-container || "$EVENT_NAME" == "schedule" ]]; then
+            is_ci_workload=true
+            echo "main=true" | tee -a "$GITHUB_OUTPUT"
+          else
+            is_ci_workload=false
+          fi
+
+          echo "main=$is_ci_workload" | tee -a "$GITHUB_OUTPUT"
+
+      - name: Fetch PR labels
+        id: pr_labels
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+        run: |
+          REF="${GITHUB_REF#refs/heads/}"
+          if [[ "$REF" =~ ^pull-request/([0-9]+)$ ]]; then
+            PR_NUMBER="${BASH_REMATCH[1]}"
+            LABELS=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json labels -q '[.labels[].name]')
+          else
+            LABELS='[]'
+          fi
+          echo "run_nightly=$(echo "$LABELS" | jq 'any(. == "Run e2e nightly")')" | tee -a "$GITHUB_OUTPUT"
+          echo "skip_docs=$(echo "$LABELS" | jq 'any(. == "skip-docs")')" | tee -a "$GITHUB_OUTPUT"
+          echo "skip_linting=$(echo "$LABELS" | jq 'any(. == "skip-linting")')" | tee -a "$GITHUB_OUTPUT"
+          echo "no_fail_fast=$(echo "$LABELS" | jq 'any(. == "no-fail-fast")')" | tee -a "$GITHUB_OUTPUT"
+
+      - name: Check if no-fail-fast is set
+        shell: bash
+        id: no_fail_fast
+        env:
+          HAS_FAIL_FAST_LABEL: ${{ steps.pr_labels.outputs.no_fail_fast }}
+        run: |
+          if [[ "$HAS_FAIL_FAST_LABEL" == "true" || "$EVENT_NAME" == "schedule" ]]; then
+            no_fail_fast=true
+          else
+            no_fail_fast=false
+          fi
+
+          echo "main=$no_fail_fast" | tee -a "$GITHUB_OUTPUT"
+
+  code-linting:
+    if: needs.configure.outputs.test_to_run != '[]'
+    needs: [configure]
+    uses: ./.github/workflows/code-linting.yml
+    with:
+      skip_docs: ${{ needs.configure.outputs.skip_docs }}
+      skip_linting: ${{ needs.configure.outputs.skip_linting }}
+
+  cicd-wait-in-queue:
+    needs: [configure, code-linting]
+    runs-on: ubuntu-latest
+    environment: test
+    if: |
+      needs.configure.outputs.test_to_run != '[]'
+      && needs.configure.outputs.components_to_run != '[]'
+      && needs.configure.outputs.is_ci_workload == 'false'
+    steps:
+      - name: Running CI tests
+        run: |
+          echo "Running CI tests"
+
+  cicd-test-container-build:
+    uses: ./.github/workflows/_build_container.yml
+    needs: [pre-flight, configure, code-linting, cicd-wait-in-queue]
+    if: |
+      needs.configure.outputs.test_to_run != '[]'
+      && needs.configure.outputs.components_to_run != '[]'
+      && (
+        success()
+        || (
+          needs.cicd-wait-in-queue.result == 'skipped'
+          && needs.configure.outputs.is_ci_workload == 'true'
+        )
+      )
+      && !cancelled()
+    with:
+      image-name: nemo_container
+      dockerfile: docker/Dockerfile
+      runner: ${{ needs.pre-flight.outputs.runner_prefix }}
+      registry: ${{ needs.pre-flight.outputs.registry }}
+
+  cicd-import-tests:
+    if: |
+      needs.configure.outputs.test_to_run != '[]'
+      && needs.configure.outputs.components_to_run != '[]'
+      && (
+        success()
+        || (
+          needs.cicd-wait-in-queue.result == 'skipped'
+          && needs.configure.outputs.is_ci_workload == 'true'
+        )
+      )
+      && !cancelled()
+    needs: [cicd-test-container-build, configure, pre-flight]
+    runs-on: ${{ needs.pre-flight.outputs.runner_prefix }}
+    steps:
+      - name: Create UUID
+        id: uuid
+        run: |
+          echo "id=$(uuidgen)" >> "$GITHUB_OUTPUT"
+
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo
+
+      - name: Run some checks
+        run: |
+          docker run \
+              --rm \
+            --device=/dev/nvidia0 \
+            --gpus all \
+            --shm-size=8g \
+            --volume $(pwd)/${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo:/workspace \
+            --env TRANSFORMERS_OFFLINE=0 \
+            --env HYDRA_FULL_ERROR=1 --env PYTHONUNBUFFERED=1 ${{ needs.cicd-test-container-build.outputs.image }} bash -c '\
+            # PyTorch Lightning version
+            python -c "import lightning.pytorch; print(lightning.pytorch.__version__)"
+
+            # Basic Import Checks
+            python tests/core_ptl/check_imports.py --domain asr
+            python tests/core_ptl/check_imports.py --domain tts
+          '
+
+  L0_Setup_Test_Data_And_Models:
+    needs: [configure, cicd-test-container-build, cicd-wait-in-queue, pre-flight]
+    runs-on: ${{ needs.pre-flight.outputs.runner_prefix }}
+    if: |
+      needs.configure.outputs.test_to_run != '[]'
+      && needs.configure.outputs.components_to_run != '[]'
+      && (
+        success()
+        || (
+          needs.cicd-wait-in-queue.result == 'skipped'
+          && needs.configure.outputs.is_ci_workload == 'true'
+        )
+      )
+      && !cancelled()
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+      - name: Checkout NeMo
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+
+      - name: main
+        uses: ./.github/actions/test-template
+        with:
+          runner: ${{ runner.name }}
+          script: L0_Setup_Test_Data_And_Models
+          tests_to_run: '["L0_Setup_Test_Data_And_Models"]'
+          image: ${{ needs.cicd-test-container-build.outputs.image }}
+          test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
+
+  cicd-main-unit-tests:
+    needs: [pre-flight, configure, cicd-test-container-build]
+    uses: ./.github/workflows/cicd-main-unit-tests.yml
+    if: |
+      needs.configure.outputs.test_to_run != '[]'
+      && needs.configure.outputs.components_to_run != '[]'
+      && (
+        success()
+        || (
+          needs.cicd-wait-in-queue.result == 'skipped'
+          && needs.configure.outputs.is_ci_workload == 'true'
+        )
+      )
+      && !cancelled()
+    with:
+      test_to_run: ${{ needs.configure.outputs.test_to_run }}
+      runner: ${{ needs.pre-flight.outputs.runner_prefix }}
+      container-image: ${{ needs.cicd-test-container-build.outputs.image }}
+
+  cicd-main-speech:
+    needs: [pre-flight, configure, cicd-test-container-build, cicd-main-unit-tests]
+    uses: ./.github/workflows/cicd-main-speech.yml
+    if: |
+      (
+        needs.configure.outputs.test_to_run != '[]'
+        && (
+          contains(fromJson(needs.configure.outputs.components_to_run), 'speech')
+        )
+      )
+      && (
+        success()
+        || (
+          needs.cicd-wait-in-queue.result == 'skipped'
+          && needs.configure.outputs.is_ci_workload == 'true'
+        )
+      )
+      && !cancelled()
+    with:
+      test_to_run: ${{ needs.configure.outputs.test_to_run }}
+      runner: ${{ needs.pre-flight.outputs.runner_prefix }}
+      run_nightly: ${{ needs.configure.outputs.run_nightly }}
+
+  Nemo_CICD_Test:
+    needs:
+      - configure
+      - cicd-test-container-build
+      - cicd-import-tests
+      - L0_Setup_Test_Data_And_Models
+      - cicd-main-unit-tests
+      - cicd-main-speech
+    if: always()
+    runs-on: ubuntu-latest
+    permissions: write-all
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Get workflow result
+        id: result
+        env:
+          GH_TOKEN: ${{ github.token }}
+          RUN_ID: ${{ github.run_id }}
+        run: |
+          # Get workflow run details and check job conclusions
+          LATEST_ATTEMPT=$(gh run view $RUN_ID --json jobs -q '[.jobs[] | select(.conclusion != null) | .conclusion] | last')
+          NUM_FAILED=$(gh run view $RUN_ID --json jobs -q '[.jobs[] | select(.conclusion == "failure") | .name] | length')
+          NUM_CANCELLED=$(gh run view $RUN_ID --json jobs -q '[.jobs[] | select(.conclusion == "cancelled") | .name] | length')
+
+          if [[ $NUM_FAILED -eq 0 && $NUM_CANCELLED -eq 0 ]]; then
+            RESULT="success"
+          elif [[ $NUM_CANCELLED -gt 0 ]]; then
+            RESULT="cancelled"
+          else
+            RESULT="failure"
+          fi
+
+          # Output the final status
+          echo "code=$RESULT" | tee -a $GITHUB_OUTPUT
+
+      - name: Pipeline successful, add PR comment
+        if: |
+          steps.result.outputs.code == 'success'
+          && github.event_name == 'push'
+          && startsWith(github.ref, 'refs/heads/pull-request/')
+          && env.SLACK_WEBHOOK != ''
+        env:
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          REPOSITORY: ${{ github.repository }}
+          RUN_ID: ${{ github.run_id }}
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          PR_NUMBER="${GITHUB_REF#refs/heads/pull-request/}"
+          PR_AUTHOR=$(gh pr view "$PR_NUMBER" --repo "$REPOSITORY" --json author -q '.author.login')
+          gh pr comment "$PR_NUMBER" --repo "$REPOSITORY" --body "[🤖]: Hi @${PR_AUTHOR} 👋,
+
+          We wanted to let you know that a [CICD pipeline](https://github.com/${REPOSITORY}/actions/runs/${RUN_ID}) for this PR just finished successfully.
+
+          So it might be time to merge this PR or get some approvals."
+
+      - name: Exit
+        if: ${{ always() }}
+        env:
+          RESULT: ${{ steps.result.outputs.code }}
+        run: |
+          if [ $RESULT == "success" ]; then
+            exit 0
+          else
+            exit 1
+          fi
+
+  Coverage:
+    runs-on: ubuntu-latest
+    needs: [configure, Nemo_CICD_Test]
+    if: |
+      needs.configure.outputs.test_to_run != '[]'
+      && needs.configure.outputs.components_to_run != '[]'
+      && (
+        success()
+        || needs.Nemo_CICD_Test.result == 'success'
+      )
+      && !cancelled()
+    strategy:
+      matrix:
+        flag: [unit-test, e2e]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Download coverage reports of current branch
+        uses: actions/download-artifact@v7
+        with:
+          pattern: coverage-${{ matrix.flag }}-*
+
+      - name: Get total coverage of current branch
+        shell: bash -x -e -u -o pipefail {0}
+        if: always()
+        run: |
+          pip install coverage
+
+          ls -al .
+          ls -al coverage-*/
+          coverage combine --keep $(ls coverage-*/.coverage)
+          coverage report -i
+          rm -rf coverage-*
+          ls -al
+
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v5
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          verbose: true
+          flags: ${{ matrix.flag }}
+
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v6
+        with:
+          name: coverage-${{ matrix.flag }}-aggregated
+          path: |
+            .coverage
+          include-hidden-files: true

.github/workflows/claude-answer.yml

@@ -0,0 +1,70 @@
+name: Claude Answer Issue
+
+on:
+  issue_comment:
+    types: [created]
+
+permissions:
+  contents: read
+  issues: write
+  id-token: write
+
+jobs:
+  authorize:
+    if: >-
+      !github.event.issue.pull_request &&
+      contains(github.event.comment.body, '/claude answer')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check team membership
+        uses: actions/github-script@v8
+        with:
+          github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
+          script: |
+            const username = context.payload.comment.user.login;
+            try {
+              const res = await github.rest.teams.getMembershipForUserInOrg({
+                org: 'NVIDIA-NeMo',
+                team_slug: 'speech_team',
+                username,
+              });
+              if (res.data.state !== 'active') {
+                core.setFailed(`${username} is not an active member of NVIDIA Speech Team`);
+              }
+            } catch (e) {
+              core.setFailed(`${username} is not a member of NVIDIA Speech Team`);
+            }
+
+  acknowledge:
+    needs: authorize
+    runs-on: ubuntu-latest
+    steps:
+      - name: Add eyes reaction to comment
+        uses: actions/github-script@v8
+        with:
+          script: |
+            await github.rest.reactions.createForIssueComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              comment_id: context.payload.comment.id,
+              content: 'eyes'
+            });
+
+  claude-answer:
+    needs: acknowledge
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: anthropics/claude-code-action@v1
+        env:
+          ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
+          CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
+          DISABLE_PROMPT_CACHING: "1"
+        with:
+          prompt: |
+            You are a helpful assistant for the NeMo repository.
+            Answer the user's question based on the issue description, comments, and the codebase.
+            Be concise and provide code references where relevant.
+            Do NOT make any code changes or create PRs — only answer the question.
+          anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
+          claude_args: --model "${{ vars.CLAUDE_MODEL }}"

.github/workflows/claude-babysit-pr.yml

@@ -0,0 +1,648 @@
+# PR Babysitter — half-autonomous CI fix loop, speech_team-only.
+#
+# Master switch: the "Has Babysitter" label. Adding it activates, removing
+# it deactivates and cancels any in-flight work.
+#
+# Authorization model:
+#   - The entire workflow is restricted to the NVIDIA-NeMo/speech_team team.
+#     Every job that mutates state starts with a preflight that verifies the
+#     acting user (label sender / comment author / review author) is an active
+#     speech_team member. Non-members cause the job to post a refusal comment
+#     and revert the triggering label, leaving no misleading state behind.
+#   - The workflow never runs for PRs from forks. Every PR-scoped job
+#     fork-guards on head.repo.full_name == github.repository.
+#
+# Lifecycle (all participants must be speech_team members):
+#   1. "Has Babysitter" added  -> activate posts a takeover comment and adds
+#                                 "Run CICD" to start the pipeline.
+#   2. CI runs (CICD NeMo workflow).
+#   3. If all checks pass      -> babysitter stays silent, done.
+#      If a CI workflow fails  -> workflow_run completion event triggers
+#                                 investigate-and-propose. The Isort+Black
+#                                 workflow isn't in the trigger list — it
+#                                 auto-pushes its own fixes.
+#   4. investigate-and-propose: Claude investigates root cause, posts a *plan
+#      comment* on the PR (tagged with `<!-- babysit-plan -->`), and adds
+#      "Agent Plan Awaiting Approval". It does NOT push code.
+#   5. The PR author (who must also be speech_team) approves by replying with a
+#      comment that mentions `@claude` affirmatively (e.g. `@claude go ahead`).
+#      evaluate-comment-approval classifies the reply via an LLM and, on APPROVE,
+#      swaps "Agent Plan Awaiting Approval" for "Agent Plan Approved".
+#   6. "Agent Plan Approved" added -> execute-fix verifies the sender is
+#      speech_team, verifies a bot-authored plan comment exists, then reads the
+#      plan, pushes the fix, re-adds "Run CICD", and goes back to #2.
+#
+# Termination:
+#   - "Has Babysitter" removed at any time -> deactivate cancels in-progress
+#     work via a shared concurrency group and clears plan-state labels.
+#   - Claude can't investigate or can't execute -> ping-author-on-failure pings
+#     the author and clears labels.
+#
+# Additionally, speech_team members can @claude in a PR review comment on a
+# babysitter-enabled PR to ask questions or request changes.
+#
+# Required secrets: NVIDIA_INFERENCE_URL, NVIDIA_INFERENCE_KEY, ORG_TEAM_READ_TOKEN, NEMO_RELABEL_TOKEN
+
+name: Claude Code — PR Babysitter
+
+on:
+  pull_request:
+    types: [labeled, unlabeled]
+  pull_request_review_comment:
+    types: [created]
+  pull_request_review:
+    types: [submitted]
+  # `workflow_run` rather than `check_run`: check-runs produced by GHA jobs
+  # authenticated with GITHUB_TOKEN do not fire `check_run` events on
+  # downstream workflows (GitHub's recursion guard), so the babysitter never
+  # saw real CI failures. `workflow_run` bypasses that restriction.
+  # Omitted intentionally: "Isort and Black Formatting" (auto-pushes fixes),
+  # this workflow itself, and labeler/relabel bots.
+  workflow_run:
+    workflows:
+      - "CICD NeMo"
+      - "PyLint and flake8 linting"
+      - "Build, test, and publish a PyPi wheel (to testpypi)."
+      - "Check __init__ files"
+      - "Copyright check"
+      - "CI-Install-Check"
+      - "CodeQL"
+      - "Secrets detector"
+    types: [completed]
+  issue_comment:
+    types: [created]
+
+permissions:
+  contents: write
+  pull-requests: write
+  issues: write
+  id-token: write
+
+jobs:
+  # ---- Activation: one-time setup when "Has Babysitter" is added ----
+
+  activate:
+    if: >-
+      github.event_name == 'pull_request' &&
+      github.event.action == 'labeled' &&
+      github.event.label.name == 'Has Babysitter' &&
+      github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    concurrency:
+      group: babysit-activate-${{ github.event.pull_request.number }}
+      cancel-in-progress: true
+    steps:
+      # Preflight: only speech_team members may enable the babysitter. If a
+      # non-member adds the label, revert it and explain.
+      - name: Verify label sender is speech_team
+        id: authz
+        uses: actions/github-script@v8
+        with:
+          github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
+          script: |
+            const username = context.payload.sender.login;
+            try {
+              const res = await github.rest.teams.getMembershipForUserInOrg({
+                org: 'NVIDIA-NeMo',
+                team_slug: 'speech_team',
+                username,
+              });
+              if (res.data.state !== 'active') {
+                core.setOutput('authorized', 'false');
+                return;
+              }
+              core.setOutput('authorized', 'true');
+            } catch (e) {
+              core.setOutput('authorized', 'false');
+            }
+
+      - name: Checkout for GH CLI
+        uses: actions/checkout@v6
+
+      - name: Refuse activation from non-speech_team sender
+        if: steps.authz.outputs.authorized != 'true'
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          SENDER: ${{ github.event.sender.login }}
+        run: |
+          gh pr comment "$PR_NUMBER" --body \
+            "@${SENDER} the PR Babysitter is restricted to the NVIDIA-NeMo \`speech_team\`. Removing the \`Has Babysitter\` label."
+          gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter"
+          exit 0
+
+      - name: Post takeover comment
+        if: steps.authz.outputs.authorized == 'true'
+        uses: actions/github-script@v8
+        with:
+          script: |
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.payload.pull_request.number,
+              body: "I'll monitor this PR until CI is green. I'll post a plan for any fix and wait for your approval before pushing anything. Ping me by removing 'Has Babysitter' to cancel."
+            });
+
+      - name: Trigger CI
+        if: steps.authz.outputs.authorized == 'true'
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: gh pr edit "$PR_NUMBER" --add-label "Run CICD"
+
+  # ---- @claude mentions in PR reviews (speech_team only) ----
+
+  authorize-review:
+    if: >-
+      github.event.pull_request.head.repo.full_name == github.repository &&
+      ((github.event_name == 'pull_request_review_comment' &&
+        contains(github.event.comment.body, '@claude') &&
+        contains(github.event.pull_request.labels.*.name, 'Has Babysitter')) ||
+       (github.event_name == 'pull_request_review' &&
+        contains(github.event.review.body, '@claude') &&
+        contains(github.event.pull_request.labels.*.name, 'Has Babysitter')))
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check team membership
+        uses: actions/github-script@v8
+        with:
+          github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
+          script: |
+            const username =
+              context.payload.comment?.user?.login ||
+              context.payload.review?.user?.login;
+            try {
+              const res = await github.rest.teams.getMembershipForUserInOrg({
+                org: 'NVIDIA-NeMo',
+                team_slug: 'speech_team',
+                username,
+              });
+              if (res.data.state !== 'active') {
+                core.setFailed(`${username} is not an active member of NVIDIA Speech Team`);
+              }
+            } catch (e) {
+              core.setFailed(`${username} is not a member of NVIDIA Speech Team`);
+            }
+
+  acknowledge-review:
+    needs: authorize-review
+    runs-on: ubuntu-latest
+    steps:
+      - name: Add eyes reaction
+        uses: actions/github-script@v8
+        with:
+          script: |
+            if (context.payload.comment) {
+              await github.rest.reactions.createForPullRequestReviewComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                comment_id: context.payload.comment.id,
+                content: 'eyes'
+              });
+            }
+
+  respond-to-review:
+    needs: acknowledge-review
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - uses: anthropics/claude-code-action@v1
+        env:
+          ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
+          CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
+          DISABLE_PROMPT_CACHING: "1"
+        with:
+          anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
+          # No prompt — Claude reads the @claude mention and responds in context
+          claude_args: "--max-turns 15 --model ${{ vars.CLAUDE_MODEL }}"
+
+  # ---- CI failure: investigate, propose a plan, DO NOT push ----
+
+  check-label-for-ci:
+    # Fires on CI workflow completion. The `workflow_run.pull_requests` array is
+    # populated for same-repo PRs whose head SHA matches the completed run; for
+    # fork PRs it is empty, which doubles as an implicit fork-guard here.
+    if: >-
+      github.event_name == 'workflow_run' &&
+      github.event.workflow_run.conclusion == 'failure' &&
+      github.event.workflow_run.pull_requests[0] != null
+    runs-on: ubuntu-latest
+    outputs:
+      pr_number: ${{ steps.lookup.outputs.pr_number }}
+      should_propose: ${{ steps.lookup.outputs.should_propose }}
+      head_ref: ${{ steps.lookup.outputs.head_ref }}
+      head_sha: ${{ steps.lookup.outputs.head_sha }}
+      author_login: ${{ steps.lookup.outputs.author_login }}
+    steps:
+      - name: Look up PR and assess state
+        id: lookup
+        uses: actions/github-script@v8
+        with:
+          script: |
+            const prNumber = context.payload.workflow_run.pull_requests[0].number;
+            const { data: pr } = await github.rest.pulls.get({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: prNumber,
+            });
+            const labels = pr.labels.map(l => l.name);
+            const hasBabysitter = labels.includes('Has Babysitter');
+            const hasPending = labels.includes('Agent Plan Awaiting Approval');
+            const hasApproved = labels.includes('Agent Plan Approved');
+            const failedSha = context.payload.workflow_run.head_sha;
+            const isStale = pr.head.sha !== failedSha;
+            // Fork-guard: the babysitter never runs on PRs from forks.
+            const isFork = pr.head.repo.full_name !== `${context.repo.owner}/${context.repo.repo}`;
+            // Propose a new plan only if: babysitter is on, no plan is already
+            // in flight, the failure isn't stale, and the PR is not from a fork.
+            const shouldPropose = hasBabysitter && !hasPending && !hasApproved && !isStale && !isFork;
+            core.setOutput('pr_number', prNumber);
+            core.setOutput('should_propose', shouldPropose ? 'true' : 'false');
+            core.setOutput('head_ref', pr.head.ref);
+            core.setOutput('head_sha', pr.head.sha);
+            core.setOutput('author_login', pr.user.login);
+            if (!shouldPropose) {
+              core.info(`Skipping investigate-and-propose: babysitter=${hasBabysitter}, pending=${hasPending}, approved=${hasApproved}, stale=${isStale}, fork=${isFork}`);
+            }
+
+  investigate-and-propose:
+    needs: check-label-for-ci
+    if: needs.check-label-for-ci.outputs.should_propose == 'true'
+    runs-on: ubuntu-latest
+    concurrency:
+      group: babysit-fix-${{ needs.check-label-for-ci.outputs.pr_number }}
+      cancel-in-progress: true
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          ref: ${{ needs.check-label-for-ci.outputs.head_ref }}
+
+      - uses: anthropics/claude-code-action@v1
+        env:
+          ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
+          CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
+          DISABLE_PROMPT_CACHING: "1"
+        with:
+          anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
+          prompt: |
+            The CI workflow "${{ github.event.workflow_run.name }}" just failed
+            on PR #${{ needs.check-label-for-ci.outputs.pr_number }}.
+
+            You are in HALF-AUTONOMOUS mode. You MUST NOT edit files, commit,
+            or push code in this job. Investigation and planning only.
+
+            Instructions:
+            1. Inspect the failure logs for ALL currently failing CI checks on
+               this PR, not just the one named above.
+            2. Check `git log` for recent commits — if a prior babysitter plan
+               already tried the same approach and it's still failing, call
+               that out in your plan so the author can reconsider.
+            3. Identify the root cause and draft a concrete fix plan covering:
+               - What's broken and why
+               - Which files/lines you'd change
+               - The minimal diff you'd apply
+            4. Post a SINGLE PR comment with this exact structure:
+
+               **CI Fix Plan** — awaiting approval from @${{ needs.check-label-for-ci.outputs.author_login }}
+
+               <your analysis and proposed change>
+
+               ---
+               To approve, **reply with an affirmative comment that mentions `@claude`**
+               (e.g., `@claude go ahead`). Approval is restricted to the PR author.
+               Remove the `Has Babysitter` label to cancel.
+
+               <!-- babysit-plan -->
+
+               The `<!-- babysit-plan -->` marker MUST be the last line — later
+               jobs use it to find the approved plan.
+            5. If you conclude the issue cannot be fixed from this PR alone,
+               DO NOT include the `<!-- babysit-plan -->` marker. Instead post
+               a comment that mentions @${{ needs.check-label-for-ci.outputs.author_login }}
+               describing what you found and asking for help.
+          claude_args: "--max-turns 10 --model ${{ vars.CLAUDE_MODEL }}"
+
+      - name: Mark plan awaiting approval (or stop if Claude gave up)
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ needs.check-label-for-ci.outputs.pr_number }}
+          REPO: ${{ github.repository }}
+        run: |
+          # Detect whether Claude actually posted a plan comment.
+          PLAN_COUNT=$(gh api "repos/$REPO/issues/$PR_NUMBER/comments" \
+            --jq '[.[] | select(.body | contains("<!-- babysit-plan -->"))] | length')
+          if [ "$PLAN_COUNT" -gt 0 ]; then
+            echo "Plan comment posted; adding 'Agent Plan Awaiting Approval'"
+            gh pr edit "$PR_NUMBER" --add-label "Agent Plan Awaiting Approval"
+          else
+            echo "No plan posted; Claude decided it couldn't fix. Disabling babysitter."
+            gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter"
+          fi
+
+  # ---- Approval Path A: affirmative comment from PR author mentioning @claude ----
+
+  evaluate-comment-approval:
+    if: >-
+      github.event_name == 'issue_comment' &&
+      github.event.action == 'created' &&
+      github.event.issue.pull_request != null &&
+      contains(github.event.issue.labels.*.name, 'Has Babysitter') &&
+      contains(github.event.issue.labels.*.name, 'Agent Plan Awaiting Approval') &&
+      github.event.comment.user.type != 'Bot' &&
+      github.event.comment.user.login == github.event.issue.user.login &&
+      contains(github.event.comment.body, '@claude')
+    runs-on: ubuntu-latest
+    concurrency:
+      group: babysit-fix-${{ github.event.issue.number }}
+      cancel-in-progress: false
+    steps:
+      # Preflight: the commenter is the PR author (enforced by `if:`) AND must
+      # be an active speech_team member. Also fork-guard by fetching the PR.
+      - name: Verify commenter is speech_team and PR is not a fork
+        id: authz
+        uses: actions/github-script@v8
+        with:
+          github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
+          script: |
+            const prNumber = context.payload.issue.number;
+            const { data: pr } = await github.rest.pulls.get({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: prNumber,
+            });
+            const isFork = pr.head.repo.full_name !== `${context.repo.owner}/${context.repo.repo}`;
+            if (isFork) {
+              core.info(`PR #${prNumber} is a fork; babysitter does not operate on forks.`);
+              core.setOutput('authorized', 'false');
+              core.setOutput('reason', 'fork');
+              return;
+            }
+            const username = context.payload.comment.user.login;
+            try {
+              const res = await github.rest.teams.getMembershipForUserInOrg({
+                org: 'NVIDIA-NeMo',
+                team_slug: 'speech_team',
+                username,
+              });
+              if (res.data.state !== 'active') {
+                core.setOutput('authorized', 'false');
+                core.setOutput('reason', 'not-speech-team');
+                return;
+              }
+              core.setOutput('authorized', 'true');
+            } catch (e) {
+              core.setOutput('authorized', 'false');
+              core.setOutput('reason', 'not-speech-team');
+            }
+
+      - name: Checkout for GH CLI
+        uses: actions/checkout@v6
+
+      - name: Refuse non-speech_team approval
+        if: steps.authz.outputs.authorized != 'true' && steps.authz.outputs.reason == 'not-speech-team'
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUMBER: ${{ github.event.issue.number }}
+          SENDER: ${{ github.event.comment.user.login }}
+        run: |
+          gh pr comment "$PR_NUMBER" --body \
+            "@${SENDER} only NVIDIA-NeMo \`speech_team\` members can approve the babysitter's plan. Ignoring this reply."
+
+      - uses: anthropics/claude-code-action@v1
+        if: steps.authz.outputs.authorized == 'true'
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ github.event.issue.number }}
+          ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
+          CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
+          DISABLE_PROMPT_CACHING: "1"
+        with:
+          anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
+          prompt: |
+            Decide whether the PR author has approved the fix plan, then act.
+
+            Context:
+            - PR #${{ github.event.issue.number }} has label "Agent Plan Awaiting Approval".
+            - The PR author (@${{ github.event.issue.user.login }}) just posted a reply.
+            - The approved plan is a previous PR comment containing `<!-- babysit-plan -->`.
+
+            Steps:
+            1. Read the latest `<!-- babysit-plan -->` comment on the PR for the plan.
+            2. Read the PR author's reply (use `gh api repos/${{ github.repository }}/issues/comments/${{ github.event.comment.id }}`).
+            3. Classify the reply as exactly one of:
+               - APPROVE: the author clearly agrees to proceed with the plan.
+               - REJECT: the author clearly disagrees or wants changes.
+               - NEITHER: the reply is a question, side remark, or ambiguous.
+            4. Take action:
+               - APPROVE: run exactly
+                 `gh pr edit $PR_NUMBER --remove-label "Agent Plan Awaiting Approval" --add-label "Agent Plan Approved"`
+                 and do nothing else.
+               - REJECT: post a short PR comment acknowledging the rejection
+                 and suggesting they either remove `Has Babysitter` to cancel
+                 or reply with a new direction. Do NOT change labels.
+               - NEITHER: post a short PR comment asking for an explicit
+                 approve/reject. Do NOT change labels.
+            5. Do not push any code. Do not edit files in the repository.
+          claude_args: "--max-turns 8 --model ${{ vars.CLAUDE_MODEL }}"
+
+  # ---- Execute the approved plan (the only job allowed to push code) ----
+
+  execute-fix:
+    if: >-
+      github.event_name == 'pull_request' &&
+      github.event.action == 'labeled' &&
+      github.event.label.name == 'Agent Plan Approved' &&
+      contains(github.event.pull_request.labels.*.name, 'Has Babysitter') &&
+      github.event.pull_request.head.repo.full_name == github.repository
+    runs-on: ubuntu-latest
+    concurrency:
+      group: babysit-fix-${{ github.event.pull_request.number }}
+      cancel-in-progress: true
+    steps:
+      # Preflight 1: the sender (who added `Agent Plan Approved`) must be
+      # speech_team. Defense-in-depth against a non-member adding the label.
+      - name: Verify label sender is speech_team
+        id: authz
+        uses: actions/github-script@v8
+        with:
+          github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
+          script: |
+            const username = context.payload.sender.login;
+            try {
+              const res = await github.rest.teams.getMembershipForUserInOrg({
+                org: 'NVIDIA-NeMo',
+                team_slug: 'speech_team',
+                username,
+              });
+              core.setOutput('authorized', res.data.state === 'active' ? 'true' : 'false');
+            } catch (e) {
+              core.setOutput('authorized', 'false');
+            }
+
+      # Preflight 2: there must exist a bot-authored `<!-- babysit-plan -->`
+      # comment on this PR. Prevents execution when someone adds the approved
+      # label manually without an investigator-posted plan.
+      - name: Verify a bot-authored plan comment exists
+        id: plan
+        if: steps.authz.outputs.authorized == 'true'
+        uses: actions/github-script@v8
+        with:
+          script: |
+            const comments = await github.paginate(github.rest.issues.listComments, {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.payload.pull_request.number,
+              per_page: 100,
+            });
+            const planComment = [...comments].reverse().find(c =>
+              c.body.includes('<!-- babysit-plan -->') && c.user && c.user.type === 'Bot'
+            );
+            core.setOutput('found', planComment ? 'true' : 'false');
+
+      - name: Checkout for GH CLI (refusal paths)
+        if: steps.authz.outputs.authorized != 'true' || steps.plan.outputs.found != 'true'
+        uses: actions/checkout@v6
+
+      - name: Refuse execution from non-speech_team sender
+        if: steps.authz.outputs.authorized != 'true'
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          SENDER: ${{ github.event.sender.login }}
+        run: |
+          gh pr comment "$PR_NUMBER" --body \
+            "@${SENDER} only NVIDIA-NeMo \`speech_team\` members can approve the babysitter's plan. Removing the \`Agent Plan Approved\` label."
+          gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
+          exit 0
+
+      - name: Refuse execution without a plan comment
+        if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found != 'true'
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          gh pr comment "$PR_NUMBER" --body \
+            "No approved plan found for execution. Remove \`Agent Plan Approved\` and wait for the investigator to post a plan first."
+          gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
+          exit 0
+
+      - uses: actions/checkout@v6
+        if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
+        with:
+          ref: ${{ github.event.pull_request.head.ref }}
+
+      - name: Record starting SHA
+        id: start
+        if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
+        run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+
+      - uses: anthropics/claude-code-action@v1
+        if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
+        env:
+          ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
+          CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
+          DISABLE_PROMPT_CACHING: "1"
+        with:
+          anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
+          prompt: |
+            The PR author has approved your previously posted fix plan. Execute it now.
+
+            Instructions:
+            1. Read the PR comment thread and find the most recent comment
+               containing the marker `<!-- babysit-plan -->`. That is the
+               approved plan.
+            2. Implement exactly what the plan described. Do not expand scope.
+               Do not refactor code outside the plan.
+            3. Commit with DCO sign-off and push to the same branch.
+            4. If the plan is no longer applicable (e.g., merge conflicts, the
+               code has moved on, the reproduction no longer triggers the bug):
+               post a PR comment mentioning @${{ github.event.pull_request.user.login }}
+               explaining what changed and DO NOT push.
+          claude_args: "--max-turns 10 --model ${{ vars.CLAUDE_MODEL }}"
+
+      - name: Re-trigger CI or stop loop
+        if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          ORIGINAL_SHA: ${{ steps.start.outputs.sha }}
+        run: |
+          # Clear "Agent Plan Approved" so the next iteration can re-use it.
+          gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
+          NEW_SHA=$(gh pr view "$PR_NUMBER" --json headRefOid -q '.headRefOid')
+          if [ "$NEW_SHA" != "$ORIGINAL_SHA" ]; then
+            echo "Fix pushed (SHA changed); re-triggering CI"
+            gh pr edit "$PR_NUMBER" --add-label "Run CICD"
+          else
+            echo "No fix pushed; disabling babysitter"
+            gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter"
+          fi
+
+  # ---- Safety net: ping author and clear state if investigate-and-propose crashes ----
+
+  ping-author-on-failure:
+    needs: [check-label-for-ci, investigate-and-propose]
+    if: failure() && needs.check-label-for-ci.outputs.should_propose == 'true'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Ping PR author
+        uses: actions/github-script@v8
+        env:
+          PR_NUMBER: ${{ needs.check-label-for-ci.outputs.pr_number }}
+          CHECK_NAME: ${{ github.event.workflow_run.name }}
+          AUTHOR_LOGIN: ${{ needs.check-label-for-ci.outputs.author_login }}
+        with:
+          script: |
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: parseInt(process.env.PR_NUMBER, 10),
+              body: `I wasn't able to investigate the CI failure in \`${process.env.CHECK_NAME}\`. @${process.env.AUTHOR_LOGIN}, could you take a look?`
+            });
+
+      - name: Checkout for GH CLI
+        uses: actions/checkout@v6
+
+      - name: Clear all babysitter labels
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ needs.check-label-for-ci.outputs.pr_number }}
+        run: |
+          gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter" || true
+          gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Awaiting Approval" || true
+          gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
+
+  # ---- Deactivation: cancel in-progress fixes and clear state ----
+
+  deactivate:
+    if: >-
+      github.event_name == 'pull_request' &&
+      github.event.action == 'unlabeled' &&
+      github.event.label.name == 'Has Babysitter'
+    runs-on: ubuntu-latest
+    concurrency:
+      group: babysit-fix-${{ github.event.pull_request.number }}
+      cancel-in-progress: true
+    steps:
+      - name: Post deactivation comment
+        uses: actions/github-script@v8
+        with:
+          script: |
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.payload.pull_request.number,
+              body: "Babysitter deactivated for this PR."
+            });
+
+      - name: Checkout for GH CLI
+        uses: actions/checkout@v6
+
+      - name: Clear plan-state labels
+        env:
+          GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Awaiting Approval" || true
+          gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true

.github/workflows/claude-fix.yml

@@ -0,0 +1,119 @@
+name: Claude Fix Issue
+
+on:
+  issue_comment:
+    types: [created]
+
+permissions:
+  contents: write
+  pull-requests: write
+  issues: write
+  id-token: write
+
+jobs:
+  authorize:
+    if: >-
+      !github.event.issue.pull_request &&
+      contains(github.event.comment.body, '/claude fix')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check team membership
+        uses: actions/github-script@v8
+        with:
+          github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
+          script: |
+            const username = context.payload.comment.user.login;
+            try {
+              const res = await github.rest.teams.getMembershipForUserInOrg({
+                org: 'NVIDIA-NeMo',
+                team_slug: 'speech_team',
+                username,
+              });
+              if (res.data.state !== 'active') {
+                core.setFailed(`${username} is not an active member of NVIDIA Speech Team`);
+              }
+            } catch (e) {
+              core.setFailed(`${username} is not a member of NVIDIA Speech Team`);
+            }
+
+  acknowledge:
+    needs: authorize
+    runs-on: ubuntu-latest
+    steps:
+      - name: Add eyes reaction to comment
+        uses: actions/github-script@v8
+        with:
+          script: |
+            await github.rest.reactions.createForIssueComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              comment_id: context.payload.comment.id,
+              content: 'eyes'
+            });
+
+  claude-fix:
+    needs: acknowledge
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: anthropics/claude-code-action@v1
+        id: claude
+        env:
+          ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
+          CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
+          DISABLE_PROMPT_CACHING: "1"
+        with:
+          prompt: |
+            You are a developer working on the NeMo repository.
+            Implement a focused fix for the issue based on the issue description and comments.
+
+            Requirements:
+            - Always use `git commit -s` to sign off all commits (DCO requirement).
+            - Prioritize correctness and minimal scope; avoid unrelated refactors.
+            - Reproduce or reason about the failure first, then implement the smallest robust fix.
+            - If required, add or update tests for the changed behavior. If tests are not feasible, explain why.
+            - If required, update related docs or comments when behavior or usage changes.
+
+            PR expectations:
+            - Create a new branch and open a pull request with your changes.
+            - Include a concise summary of root cause and fix based on the following PR template:
+              ```
+              # What does this PR do ?
+              Add a one line overview of what this PR aims to accomplish.
+
+              # Changelog
+              Add specific line by line info of high level changes in this PR.
+
+              # Usage
+              Add a usage example of the changed functionality.
+              
+              Related to #${{ github.event.issue.number }}
+
+              This PR is created by Claude. 
+              ```
+
+            
+          anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
+          claude_args: --model "${{ vars.CLAUDE_MODEL }}"
+
+      - name: Label PR with agent-contribution
+        if: steps.claude.outputs.branch_name
+        uses: actions/github-script@v8
+        with:
+          script: |
+            const prs = await github.rest.pulls.list({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              head: `${context.repo.owner}:${process.env.BRANCH_NAME}`,
+              state: 'open'
+            });
+            if (prs.data.length > 0) {
+              await github.rest.issues.addLabels({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: prs.data[0].number,
+                labels: ['agent-contribution']
+              });
+            }
+        env:
+          BRANCH_NAME: ${{ steps.claude.outputs.branch_name }}

.github/workflows/claude-review.yml

@@ -0,0 +1,60 @@
+name: Claude Code Review
+
+on:
+  issue_comment:
+    types: [created]
+
+permissions:
+  contents: read
+  pull-requests: write
+  issues: write
+  id-token: write
+
+jobs:
+  acknowledge:
+    if: >-
+      github.event.issue.pull_request &&
+      contains(github.event.comment.body, '/claude review')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Add eyes reaction to comment
+        uses: actions/github-script@v8
+        with:
+          script: |
+            await github.rest.reactions.createForIssueComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              comment_id: context.payload.comment.id,
+              content: 'eyes'
+            });
+
+  claude-review:
+    needs: acknowledge
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_claude_review.yml@v1.7.0
+    with:
+      model: ${{ vars.CLAUDE_MODEL }}
+      prompt: |
+        You are doing a light code review. Keep it concise and actionable.
+
+        Focus ONLY on:
+        - Critical bugs or logic errors
+        - Typos in code, comments, or strings
+        - Missing or insufficient test coverage for changed code
+        - Outdated or inaccurate documentation affected by the changes
+
+        Do NOT comment on:
+        - Style preferences or formatting
+        - Minor naming suggestions
+        - Architectural opinions or refactoring ideas
+        - Performance unless there is a clear, measurable issue
+
+        Provide feedback using inline comments for specific code suggestions.
+        Use top-level comments for general observations.
+
+        IMPORTANT: Do NOT approve the pull request. Only leave comments.
+
+        It's perfectly acceptable to not have anything to comment on.
+        If you do not have anything to comment on, post "LGTM".
+    secrets:
+      NVIDIA_INFERENCE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
+      NVIDIA_INFERENCE_KEY: ${{ secrets.NVIDIA_INFERENCE_KEY }}

.github/workflows/close-inactive-issue-pr.yml

@@ -0,0 +1,25 @@
+name: Stale-Close-Inactive-Issues-PRs
+on:
+  schedule:
+    - cron: "30 1 * * *"
+
+jobs:
+  close-issues:
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+    steps:
+      - uses: actions/stale@v6
+        with:
+          operations-per-run: 100
+          days-before-issue-stale: 30
+          days-before-issue-close: 7 
+          stale-issue-label: "stale"
+          stale-issue-message: "This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days."
+          close-issue-message: "This issue was closed because it has been inactive for 7 days since being marked as stale."
+          days-before-pr-stale: 14 
+          days-before-pr-close: 7 
+          stale-pr-message: "This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days."
+          close-pr-message: "This PR was closed because it has been inactive for 7 days since being marked as stale."
+          repo-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/code-formatting.yml

@@ -0,0 +1,50 @@
+name: Isort and Black Formatting Check
+# Check that changed files are formatted with black and isort.
+# Fails if any file needs reformatting — run `pre-commit run --all-files` locally to fix.
+
+on:
+  pull_request:
+    paths:
+      - "**.py"
+    types: [opened, synchronize, reopened]
+
+defaults:
+  run:
+    shell: bash -x -e -u -o pipefail {0}
+
+jobs:
+  check_isort_and_black:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - name: Checkout branch
+        uses: actions/checkout@v6
+
+      - name: Get changed files
+        id: changed-files
+        uses: step-security/changed-files@v45.0.1
+        with:
+          files: |
+            **.py
+
+      - name: Setup Python env
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.10"
+
+      - name: black
+        uses: psf/black@stable
+        if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
+        with:
+          options: "--check --verbose"
+          # check only changed files (pass explicitly the files)
+          src: "${{ steps.changed-files.outputs.all_changed_files }}"
+          version: "~= 24.3"
+
+      - name: isort
+        uses: isort/isort-action@v1
+        if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
+        with:
+          isort-version: "5.13.2"
+          configuration: "--check-only --diff"

.github/workflows/code-init-file-checker.yml

@@ -0,0 +1,23 @@
+name: Check __init__ files
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+
+jobs:
+  check-init-files:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.11"
+
+      - name: Install init-file-checker
+        run: pip install init-file-checker
+
+      - name: Run init-file-checker
+        run: init-file-checker nemo/

.github/workflows/code-linting.yml

@@ -0,0 +1,178 @@
+name: PyLint and flake8 linting
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, labeled, unlabeled]
+  workflow_call:
+    inputs:
+      skip_docs:
+        required: false
+        default: 'false'
+        type: string
+      skip_linting:
+        required: false
+        default: 'false'
+        type: string
+
+jobs:
+  linting:
+    name: "Domain: ${{ matrix.domain }}"
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        domain: [speech, other]
+    env:
+      DOMAIN: ${{ matrix.domain }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Get PR info
+        id: get-pr-info
+        if: startsWith(github.ref, 'refs/heads/pull-request/')
+        uses: nv-gha-runners/get-pr-info@main
+
+      - name: Select filter
+        id: filter
+        run: |
+          if [[ "$DOMAIN" == "speech" ]]; then
+            FILTER=$(jq -crn '[
+              "nemo/collections/common/data/lhotse/*.py",
+              "nemo/collections/asr/**/*.py",
+              "nemo/collections/tts/**/*.py",
+              "nemo/collections/audio/**/*.py",
+              "nemo/collections/multimodal/speech_llm/**/*.py",
+              "nemo/collections/speechlm/**/*.py",
+              "nemo/collections/speechlm2/**/*.py"
+            ] | join(",")')
+
+          else
+            FILTER=$(jq -crn '[
+              "nemo/**/*.py",
+              "!nemo/collections/common/data/lhotse/*.py",
+              "!nemo/collections/asr/**/*.py",
+              "!nemo/collections/tts/**/*.py",
+              "!nemo/collections/audio/**/*.py",
+              "!nemo/collections/multimodal/speech_llm/**/*.py",
+              "!nemo/collections/speechlm/**/*.py",
+              "!nemo/collections/speechlm2/**/*.py",
+              "!nemo/export/**/*.py"
+            ] | join(",")')
+          fi
+
+          echo "main=$FILTER" | tee -a "$GITHUB_OUTPUT"
+
+      - name: Get changed files
+        id: changed-files
+        uses: step-security/changed-files@v45.0.1
+        with:
+          sha: ${{ github.sha }}
+          base_sha: ${{ fromJSON(steps.get-pr-info.outputs.pr-info || '{}').base.sha || github.event.pull_request.base.sha }}
+          files: ${{ steps.filter.outputs.main }}
+          files_separator: ","
+          separator: " "
+
+      - name: Run PyLint
+        id: pylint
+        env:
+          CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
+          SKIP_DOCS: ${{ inputs.skip_docs == 'true' || contains(github.event.pull_request.labels.*.name, 'skip-docs') }}
+          SKIP_LINTING: ${{ inputs.skip_linting == 'true' || contains(github.event.pull_request.labels.*.name, 'skip-linting') }}
+        run: |
+          if [[ -z "$CHANGED_FILES" ]]; then
+            echo Nothing to lint.
+            echo "exit-code=0" | tee -a "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          if [[ $SKIP_DOCS == true ]]; then
+            ADDITIONAL_PYLINT_ARGS="--disable=C0115,C0116"
+          else
+            ADDITIONAL_PYLINT_ARGS=""
+          fi
+
+          if [[ $SKIP_LINTING == true ]]; then
+            ADDITIONAL_PYLINT_ARGS="--exit-zero"
+          fi
+
+          pip install pylint
+          set +e
+          pylint $ADDITIONAL_PYLINT_ARGS --output "pylintrc.$DOMAIN.txt" --rcfile ".pylintrc.$DOMAIN" ${CHANGED_FILES[@]}
+          echo "exit-code=$?" | tee -a "$GITHUB_OUTPUT"
+
+      - name: Run flake8
+        id: flake8
+        env:
+          CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
+          SKIP_LINTING: ${{ inputs.skip_linting == 'true' || contains(github.event.pull_request.labels.*.name, 'skip-linting') }}
+        run: |
+          if [[ -z "$CHANGED_FILES" ]]; then
+            echo Nothing to lint.
+            echo "exit-code=0" | tee -a "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          if [[ $SKIP_LINTING == true ]]; then
+            ADDITIONAL_FLAKE8_ARGS="--exit-zero"
+          else
+            ADDITIONAL_FLAKE8_ARGS=""
+          fi
+
+          pip install flake8
+          set +e
+          flake8 $ADDITIONAL_FLAKE8_ARGS --output "flake8.$DOMAIN.txt" --config ".flake8.$DOMAIN" ${CHANGED_FILES[@]}
+          echo "exit-code=$?" | tee -a "$GITHUB_OUTPUT"
+
+      - name: Summary
+        env:
+          PYLINT: ${{ steps.pylint.outputs.exit-code == 0 }}
+          FLAKE8: ${{ steps.flake8.outputs.exit-code == 0 }}
+        run: |
+
+          if [[ "$PYLINT" != "true" ]]; then
+            echo "Pylint output:" | tee -a $GITHUB_STEP_SUMMARY
+
+            echo '```' | tee -a $GITHUB_STEP_SUMMARY
+            cat pylintrc.$DOMAIN.txt | tee -a $GITHUB_STEP_SUMMARY
+            echo '```' | tee -a $GITHUB_STEP_SUMMARY
+          fi
+
+          if [[ "$FLAKE8" != "true" ]]; then
+            echo "Flake8 output:" | tee -a $GITHUB_STEP_SUMMARY
+
+            echo '```' | tee -a $GITHUB_STEP_SUMMARY
+            cat flake8.$DOMAIN.txt | tee -a $GITHUB_STEP_SUMMARY
+            echo '```' | tee -a $GITHUB_STEP_SUMMARY
+          fi
+
+          if [[ "$PYLINT" != "true" ||  "$FLAKE8" != "true" ]]; then
+            echo "The following directories got scanned:" | tee -a $GITHUB_STEP_SUMMARY
+
+            echo '```' | tee -a $GITHUB_STEP_SUMMARY
+            echo ${{ steps.filter.outputs.main }} | tee -a $GITHUB_STEP_SUMMARY
+            echo '```' | tee -a $GITHUB_STEP_SUMMARY
+
+            exit 1
+          fi
+
+  Nemo_Linting_Test:
+    needs: linting
+    runs-on: ubuntu-latest
+    if: always()
+    steps:
+      - name: Main
+        env:
+          RESULTS: ${{ toJson(needs.linting) }}
+        run: |
+          RESULT=$(echo "$RESULTS" | jq -r '.result')
+
+          if [[ "$RESULT" == "success" ]]; then
+            echo "All passed."
+            exit 0
+          else
+            echo "Some linting domains failed."
+            exit 1
+          fi

.github/workflows/codeql.yml

@@ -0,0 +1,75 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ "main", "[rv][0-9]*",  "gh-pages-src" ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ "main" ]
+  schedule:
+    - cron: '19 1 * * 4'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'python' ]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
+        # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v6
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v2
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+
+        # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
+        queries: security-and-quality  # security-extended,
+        config-file: ./.github/workflows/config/codeql.yml
+
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, Go, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v2
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
+
+    #   If the Autobuild fails above, remove it and uncomment the following three lines.
+    #   modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
+
+    # - run: |
+    #   echo "Run, Build Application using script"
+    #   ./location_of_script_within_repo/buildscript.sh
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v2
+      with:
+        category: "/language:${{matrix.language}}"

.github/workflows/community-bot.yml

@@ -0,0 +1,15 @@
+name: Community Bot
+
+on:
+  issues:
+    types: [opened, edited, reopened, closed, deleted]
+  issue_comment:
+    types: [created, edited, deleted]
+
+jobs:
+  community-bot:
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_community_bot.yml@v0.62.0
+    with:
+      community_project_id: ${{ vars.COMMUNITY_PROJECT_ID }}
+    secrets:
+      GH_TOKEN: ${{ secrets.PAT }}

.github/workflows/config/changelog-config.json

@@ -0,0 +1,205 @@
+{
+    "categories": [
+        {
+            "title": "## ASR\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "asr"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## TTS\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "tts"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## NLP / NMT\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "nlp",
+                "nmt",
+                "megatron"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## Text Normalization / Inverse Text Normalization\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "tn",
+                "itn"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## NeMo Tools\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "tools"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## Export\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "export"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## Documentation\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "docs"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## Bugfixes\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "bug"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        },
+        {
+            "title": "</details>\n\n## Cherrypick\n\n<details><summary>Changelog</summary>",
+            "labels": [
+                "cherry-pick"
+            ],
+            "exclude_labels": [
+                "cherry-pick"
+            ]
+        }
+    ],
+    "ignore_labels": [
+        "ignore"
+    ],
+    "sort": "ASC",
+    "template": "\n${{CHANGELOG}}</details>\n\n## Uncategorized:\n\n<details><summary>Changelog</summary>\n\n${{UNCATEGORIZED}}\n</details>\n",
+    "pr_template": "- ${{TITLE}} by @${{AUTHOR}} :: PR: #${{NUMBER}}",
+    "empty_template": "${{OWNER}}\n${{REPO}}\n${{FROM_TAG}}\n${{TO_TAG}}",
+    "label_extractor": [
+        {
+            "pattern": "(.*tts.*)|(.*g2p.*)",
+            "target": "tts",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*asr.*)|(.*ctc.*)|(.*rnnt.*)|(.*transducer.*)|(.*dali.*)|(.*k2.*)",
+            "target": "asr",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*nlp.*)|(.*punctuation.*)|(.*capitalization.*)|(.*entity.*)|(.*glue.*)|(.*entity.*)|(.*retrieval.*)|(.*entity.*)|(.*intent.*)|(.*slot.*)|(.*entity.*)|(.*language.*)|(.*qa.*)|(.*token class.*)|(.*text class.*)",
+            "target": "nlp",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*nmt.*)|(.*bignlp.*)|(.*megatron.*)|(.*machine.*)|(.*translation.*)|(.*gpt.*)",
+            "target": "nmt",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*tn.*)|(.*itn.*)|(.*text norm.*)",
+            "target": "tn",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*sde.*)|(.*ctc segment.*)",
+            "target": "tools",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*trt.*)|(.*onnx.*)|(.*export.*)",
+            "target": "export",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*\\[x\\] Documentation.*)",
+            "target": "docs",
+            "flags": "gmu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*\\[x\\] Bugfix.*)|(.*patch.*)",
+            "target": "bug",
+            "flags": "gmu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        },
+        {
+            "pattern": "(.*cherry-pick.*)|(.*cherrypick.*)",
+            "target": "cherrypick",
+            "flags": "gimu",
+            "on_property": [
+                "title",
+                "body"
+            ]
+        }
+    ],
+    "duplicate_filter": {
+        "pattern": ".+",
+        "on_property": "title",
+        "method": "match"
+    },
+    "transformers": [
+        {
+            "pattern": "^cp:\\s*`(.+?)`\\s*into\\s*\\S+",
+            "target": "$1"
+        }
+    ],
+    "max_tags_to_fetch": 100,
+    "max_pull_requests": 500,
+    "max_back_track_time_days": 365,
+    "exclude_merge_branches": [],
+    "tag_resolver": {
+        "method": "semver"
+    }
+}

.github/workflows/config/codeql.yml

@@ -0,0 +1,9 @@
+name: "CodeQL config"
+
+paths:
+  - nemo/
+  - tests/
+  - tools/
+  - scripts/
+  - examples/
+  - .github/

.github/workflows/copyright-check.yml

@@ -0,0 +1,22 @@
+# Copyright (c) 2020-2021, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Copyright check
+
+on:
+  pull_request:
+
+jobs:
+  copyright-check:
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_copyright_check.yml@v0.2.0

.github/workflows/install-test.yml

@@ -0,0 +1,283 @@
+name: CI-Install-Check
+
+on:
+  pull_request:
+    paths:
+      - "**"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test-installs-macos:
+    name: ${{ matrix.os }}-py${{ matrix.python }}-${{ matrix.installer }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [macos-latest]
+        python: ["3.10", "3.11", "3.12"]
+        installer: ["pip-install", "nemo-install"]
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+
+      - name: Check disk space before cleanup
+        run: df -h
+
+      - name: Free up disk space
+        run: |
+          # Remove unnecessary files on macOS
+          sudo rm -rf /usr/local/lib/android || true
+          sudo rm -rf /usr/local/.ghcup || true
+          sudo rm -rf /usr/local/lib/node_modules || true
+          brew cleanup || true
+          # Clear pip cache
+          pip cache purge || true
+
+      - name: Check disk space after cleanup
+        run: df -h
+
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "${{ matrix.python }}"
+
+      - name: Install NeMo
+        env:
+          INSTALLER: ${{ matrix.installer }}
+          NEMO_TAG: ${{ github.sha }}
+          NEMO_REPO: ${{ github.server_url }}/${{ github.repository }}
+        run: |
+          if [[ "$INSTALLER" == "pip-install" ]]; then
+            pip install --no-cache-dir -U pip
+            pip install --no-cache-dir ".[all]"
+          else
+            export NEMO_TAG
+            export NEMO_REPO
+            export INSTALL_DIR=$(pwd)
+
+            pip install --no-cache-dir ".[all]"
+          fi
+
+      - name: Check disk space after installation
+        run: df -h
+
+      - name: Run import checks
+        run: |
+          # Run import checks
+          for collection in "asr" "tts" "lightning" "core"; do
+            python tests/core_ptl/check_imports.py --domain "$collection"
+          done
+
+  test-installs-linux-amd:
+    name: ubuntu-22.04-amd-py${{ matrix.python }}-${{ matrix.installer }}
+    runs-on: ubuntu-22.04
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.10", "3.11", "3.12"]
+        installer: ["pip-install", "nemo-install"]
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+
+      - name: Check disk space before cleanup
+        run: df -h
+
+      - name: Free up disk space
+        run: |
+          # Remove unnecessary packages and files on Ubuntu
+          sudo apt-get clean
+          sudo rm -rf /usr/local/lib/android || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf /usr/local/.ghcup || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/az || true
+          # Clear pip and npm caches
+          pip cache purge || true
+          sudo npm cache clean --force || true
+
+      - name: Check disk space after cleanup
+        run: df -h
+
+      - name: Install Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Install NeMo
+        env:
+          INSTALLER: ${{ matrix.installer }}
+        run: |
+          if [ "$INSTALLER" = "pip-install" ]; then
+            pip install --no-cache-dir --upgrade pip
+            pip install --no-cache-dir ".[all]"
+          else
+            export INSTALL_DIR=$(pwd)
+            pip install --no-cache-dir ".[all]"
+          fi
+
+      - name: Check disk space after installation
+        run: df -h
+
+      - name: Run import checks
+        run: |
+          # Run import checks
+          for collection in "asr" "tts" "lightning" "core"; do
+            python tests/core_ptl/check_imports.py --domain "$collection"
+          done
+
+  test-asr-install-linux-amd:
+    name: ubuntu-22.04-amd-py${{ matrix.python }}-asr
+    runs-on: ubuntu-22.04
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.10", "3.11", "3.12"]
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+
+      - name: Check disk space before cleanup
+        run: df -h
+
+      - name: Free up disk space
+        run: |
+          # Remove unnecessary packages and files on Ubuntu
+          sudo apt-get clean
+          sudo rm -rf /usr/local/lib/android || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf /usr/local/.ghcup || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/az || true
+          # Clear pip and npm caches
+          pip cache purge || true
+          sudo npm cache clean --force || true
+
+      - name: Check disk space after cleanup
+        run: df -h
+
+      - name: Install Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Install NeMo
+        run: |
+          pip install --no-cache-dir --upgrade pip
+          pip install --no-cache-dir ".[asr]"
+
+      - name: Check disk space after installation
+        run: df -h
+
+      - name: Run import checks
+        run: |
+          # Run import checks
+          python tests/core_ptl/check_imports.py --domain asr
+
+  test-installs-linux-arm:
+    name: ubuntu-22.04-arm-py${{ matrix.python }}-${{ matrix.installer }}
+    runs-on: ubuntu-22.04-arm
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.10", "3.11", "3.12"]
+        installer: ["pip-install", "nemo-install"]
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+
+      - name: Check disk space before cleanup
+        run: df -h
+
+      - name: Free up disk space
+        run: |
+          # Remove unnecessary packages and files on Ubuntu ARM
+          sudo apt-get clean
+          sudo rm -rf /usr/local/lib/android || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf /usr/local/.ghcup || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/az || true
+          # Clear pip and npm caches
+          pip cache purge || true
+          sudo npm cache clean --force || true
+
+      - name: Check disk space after cleanup
+        run: df -h
+
+      - name: Install Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Install NeMo
+        env:
+          INSTALLER: ${{ matrix.installer }}
+        run: |
+          if [ "$INSTALLER" = "pip-install" ]; then
+            pip install --no-cache-dir --upgrade pip
+            pip install --no-cache-dir ".[all]"
+          else
+            export INSTALL_DIR=$(pwd)
+            pip install --no-cache-dir ".[all]"
+          fi
+
+      - name: Check disk space after installation
+        run: df -h
+
+      - name: Run import checks
+        run: |
+          # Run import checks
+          for collection in "asr" "tts" "lightning" "core"; do
+            python tests/core_ptl/check_imports.py --domain "$collection"
+          done
+
+  test-asr-installs-linux-arm:
+    name: ubuntu-22.04-arm-py${{ matrix.python }}-asr
+    runs-on: ubuntu-22.04-arm
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ["3.10", "3.11", "3.12"]
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+
+      - name: Check disk space before cleanup
+        run: df -h
+
+      - name: Free up disk space
+        run: |
+          # Remove unnecessary packages and files on Ubuntu ARM
+          sudo apt-get clean
+          sudo rm -rf /usr/local/lib/android || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf /usr/local/.ghcup || true
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /opt/az || true
+          # Clear pip and npm caches
+          pip cache purge || true
+          sudo npm cache clean --force || true
+
+      - name: Check disk space after cleanup
+        run: df -h
+
+      - name: Install Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Install NeMo
+        run: |
+          pip install --no-cache-dir --upgrade pip
+          pip install --no-cache-dir ".[asr]"
+
+      - name: Check disk space after installation
+        run: df -h
+
+      - name: Run import checks
+        run: |
+          # Run import checks
+          python tests/core_ptl/check_imports.py --domain asr

.github/workflows/labeler.yml

@@ -0,0 +1,14 @@
+name: "Pull Request Labeler"
+on:
+- pull_request_target
+
+jobs:
+  triage:
+    permissions:
+      contents: read
+      pull-requests: write
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/labeler@v4
+      with:
+        repo-token: "${{ secrets.GITHUB_TOKEN }}"

.github/workflows/mcore-tag-bump-bot.yml

@@ -0,0 +1,62 @@
+# Regularly updates the CI container
+name: Megatron Tag Bump Bot
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: 0 0 * * *
+
+jobs:
+  get-release-branch-names:
+    runs-on: ubuntu-latest
+    outputs:
+      mcore: ${{ steps.get-branch.outputs.mcore_release_branch }}
+      nemo: ${{ steps.get-branch.outputs.nemo_release_branch }}
+    steps:
+      - name: Get release branch names
+        id: get-branch
+        run: |
+          latest_branch=$(git ls-remote --heads https://github.com/NVIDIA/Megatron-LM.git 'refs/heads/core_r*' | 
+            grep -o 'core_r[0-9]\+\.[0-9]\+\.[0-9]\+' | 
+            sort -V | 
+            tail -n1)
+          echo "mcore_release_branch=$latest_branch" >> $GITHUB_OUTPUT
+
+          latest_branch=$(git ls-remote --heads https://github.com/NVIDIA/NeMo.git 'refs/heads/r*' | 
+            grep -o 'r[0-9]\+\.[0-9]\+\.[0-9]\+' | 
+            sort -V | 
+            tail -n1)
+          echo "nemo_release_branch=$latest_branch" >> $GITHUB_OUTPUT
+
+  bump-tags:
+    needs: [get-release-branch-names]
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - nemo-target-branch: ${{ needs.get-release-branch-names.outputs.nemo }}
+            mcore-target-branch: ${{ needs.get-release-branch-names.outputs.mcore }}
+          - nemo-target-branch: main
+            mcore-target-branch: main
+    uses: ./.github/workflows/_bump_mcore_tag.yml
+    with:
+      nemo-target-branch: ${{ matrix.nemo-target-branch }}
+      mcore-target-branch: ${{ matrix.mcore-target-branch }}
+    secrets:
+      PAT: ${{ secrets.PAT }}
+
+  notify:
+    if: failure()
+    runs-on: ubuntu-latest
+    needs: [bump-tags]
+    steps:
+      - name: Notify
+        env:
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_WEBHOOK_ADMIN: <!subteam^${{ secrets.SLACK_WEBHOOK_ADMIN }}>
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+        run: |
+          curl -X POST \
+            -H 'Content-type: application/json' \
+            --data "{\"text\":\":robot_joy: <https://github.com/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}|Mcore-bump-bot workflow> failed. Please fix manually.\n\ncc ${SLACK_WEBHOOK_ADMIN}\"}" \
+            $SLACK_WEBHOOK

.github/workflows/monitor-single-vm.yml

@@ -0,0 +1,54 @@
+name: ~shut down a single VM
+
+on:
+  workflow_call:
+    inputs:
+      vm:
+        type: string
+        description: Name of VM
+        required: true
+      n_gpus:
+        type: string
+        description: Number of GPUs this VM has
+        required: true
+
+jobs:
+  check-status-and-maybe-shutdown:
+    environment: main
+    runs-on: ${{ inputs.vm }}
+    outputs:
+      status: ${{ steps.status.outputs.main }}
+    steps:
+      - name: Check status
+        id: status
+        run: |
+          docker run --rm --runtime=nvidia --gpus ${{ inputs.n_gpus }} ubuntu nvidia-smi
+
+          NUM_GPUS=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
+
+          if [[ $NUM_GPUS -ne ${{ inputs.n_gpus }} ]]; then
+            echo "Issues with GPU detected, will take this runner offline."
+            echo "main=degraded" >> "$GITHUB_OUTPUT"
+          else
+            echo "main=healthy" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Send Slack message & Disconnect runner from GitHub
+        if: ${{ steps.status.outputs.main == 'degraded' || failure() }}
+        run: |
+          MESSAGE='{
+            "blocks": [
+              {
+                "type": "section",
+                "text": {
+                  "type": "mrkdwn",
+                  "text": ":alert: VM bot 🤖: Hey <!subteam^${{ secrets.SLACK_WEBHOOK_ADMIN }}>: VM `${{ inputs.vm }}` is having not the best day of their life, maybe bring them an apple or so."
+                }
+              }
+            ]
+          }'
+
+          curl -X POST -H "Content-type: application/json" --data "$MESSAGE" ${{ secrets.SLACK_WEBHOOK }}
+
+          cd /home/azureuser/actions-runner
+          echo ${{ secrets.VM_KEY }} | sudo -S ./svc.sh stop

.github/workflows/monitor-vms.yml

@@ -0,0 +1,54 @@
+# Regularly updates the CI container
+name: Reboots VMs in a controlled way
+on:
+  schedule:
+    - cron: 0/15 * * * *
+  workflow_dispatch:
+
+jobs:
+  pre-flight:
+    runs-on: ubuntu-latest
+    if: github.repository_owner == 'NVIDIA'
+    outputs:
+      list-of-vms: ${{ steps.main.outputs.main }}
+    environment: main
+    steps:
+      - name: Get list of VMs
+        id: main
+        env:
+          GITHUB_TOKEN: ${{ secrets.PAT }}
+        run: |
+          RUNNERS=$(curl -L \
+            -H "Accept: application/vnd.github+json" \
+            -H "Authorization: Bearer $GITHUB_TOKEN" \
+            -H "X-GitHub-Api-Version: 2022-11-28" \
+            https://api.github.com/repos/NVIDIA/NeMo/actions/runners)
+
+          MATRIX=$(echo $RUNNERS \
+            | jq -c '[
+                .runners[]
+                | select(.status == "online")
+                | select(.name | contains("cpu") | not)
+                | {
+                  "vm": .name,
+                  "n_gpus": [
+                    .labels[]
+                    | select(.name | endswith("gpu")) | .name
+                  ][0][:1]
+                }
+              ]
+            '
+          )
+          echo main=$MATRIX | tee -a "$GITHUB_OUTPUT"
+
+  maintenance:
+    needs: pre-flight
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJSON(needs.pre-flight.outputs.list-of-vms )}}
+    uses: ./.github/workflows/monitor-single-vm.yml
+    with:
+      vm: ${{ matrix.vm }}
+      n_gpus: ${{ matrix.n_gpus }}
+    secrets: inherit  # pragma: allowlist secret

.github/workflows/release-docs.yml

@@ -0,0 +1,123 @@
+# Copyright (c) 2026, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: Release docs
+on:
+  workflow_dispatch:
+    inputs:
+      dry-run:
+        description: Whether to run the workflow in dry-run mode
+        required: true
+        type: boolean
+        default: true
+      publish-as-latest:
+        description: Publish as Latest stable version.
+        required: false
+        type: boolean
+        default: true
+      docs-version-override:
+        description: Docs version if commit is not tagged
+        required: false
+        type: string
+        default: ""
+      update-version-picker:
+        description: Update version picker.
+        required: false
+        type: boolean
+        default: true
+      notify-emails:
+        description: Email addresses to send the notification to. Format as "me@me.com,you@you.com".
+        required: false
+        type: string
+      github-ref:
+        description: Github ref to checkout
+        required: false
+        type: string
+        default: ""
+  workflow_call:
+    inputs:
+      dry-run:
+        description: Whether to run the workflow in dry-run mode
+        required: false
+        type: boolean
+        default: true
+      publish-as-latest:
+        description: Publish as Latest stable version.
+        required: false
+        type: boolean
+        default: true
+      docs-version-override:
+        description: Docs version if commit is not tagged
+        required: false
+        type: string
+        default: ""
+      update-version-picker:
+        description: Update version picker.
+        required: false
+        type: boolean
+        default: true
+      notify-emails:
+        description: Email addresses to send the notification to. Format as "me@me.com,you@you.com".
+        required: false
+        type: string
+      github-ref:
+        description: Github ref to checkout
+        required: false
+        type: string
+        default: ""
+
+jobs:
+  build-docs:
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_build_docs.yml@v0.83.0
+    with:
+      ref: ${{ inputs.github-ref }}
+      docs-directory: docs/source
+      sync-all: true
+      no-extras: "--no-extra cu12 --no-extra compiled --no-extra compiled-a100"
+
+  publish-docs:
+    runs-on: ubuntu-latest
+    needs: [build-docs]
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          repository: NVIDIA-NeMo/FW-CI-templates
+          ref: v0.74.0
+          path: FW-CI-templates
+
+      - uses: ./FW-CI-templates/.github/actions/publish-docs
+        # This workflow runs either on main, or on a version tag. Any other git ref will lead
+        # to an error.
+        # If its on main, it will publish to "latest" directory in Akamai.
+        # If its on a versioned tag, it will extract the version number from the tag (strip `v` prefix)
+        # and publish to the versioned directory in Akamai.
+        with:
+          dry-run: ${{ inputs.dry-run }}
+          artifacts-name: docs-html
+          artifacts-path: _build/html
+          emails-csv: ${{ inputs.notify-emails && format('{0},{1}', vars.docs_release_emails, inputs.notify-emails) || vars.docs_release_emails }}
+          overwrite-latest-on-tag: ${{ inputs.publish-as-latest }}
+          docs-version-override: ${{ inputs.docs-version-override }}
+          update-version-picker: ${{ inputs.update-version-picker }}
+          run-on-version-tag-only: ${{ github.ref_name != 'main' }}
+          request-name: nemo-speech-publish-docs-${{ github.run_id }}
+          aws-region: ${{ vars.DOCS_AWS_REGION }}
+          aws-role-to-assume: ${{ secrets.AWS_ASSUME_ROLE_ARN }}
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          akamai-host: ${{ secrets.AKAMAI_HOST }}
+          akamai-client-token: ${{ secrets.AKAMAI_CLIENT_TOKEN }}
+          akamai-client-secret: ${{ secrets.AKAMAI_CLIENT_SECRET }}
+          akamai-access-token: ${{ secrets.AKAMAI_ACCESS_TOKEN }}
+          s3-target-root: ${{ secrets.S3_BUCKET_NAME }}
+          s3-target-path: nemo/speech

.github/workflows/release-freeze.yml

@@ -0,0 +1,80 @@
+name: "Code freeze"
+
+on:
+  workflow_dispatch:
+    inputs:
+      type_of_release:
+        type: choice
+        description: Type of release
+        options:
+          - major
+          - minor
+      freeze-commit:
+        type: string
+        description: Commit SHA to use for cut-off
+        required: false
+        default: main
+      mcore_version:
+        description: "Version of MCore to use (must be a valid git ref)"
+        required: true
+        type: string
+      dry-run:
+        type: boolean
+        description: Dry-run of code-freeze
+        required: false
+        default: true
+
+jobs:
+  code-freeze:
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_code_freeze.yml@v0.86.0
+    with:
+      library-name: NeMo-Toolkit
+      python-package: nemo
+      release-type: ${{ inputs.type_of_release }}
+      freeze-commit: ${{ inputs.freeze-commit }}
+      dry-run: ${{ inputs.dry-run }}
+      use-pat: true
+    secrets:
+      SLACK_WEBHOOK: ${{ secrets.SLACK_RELEASE_ENDPOINT }}
+      SLACK_WEBHOOK_ADMIN: ${{ secrets.SLACK_WEBHOOK_ADMIN }}
+      PAT: ${{ secrets.PAT }}
+
+  freeze-tags:
+    runs-on: ubuntu-latest
+    needs: [code-freeze]
+    environment: main
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          path: ${{ github.run_id }}
+          token: ${{ secrets.PAT }}
+          fetch-depth: 0
+          fetch-tags: true
+          ref: ${{ inputs.dry-run == true && inputs.freeze-commit || needs.code-freeze.outputs.release-branch }}
+
+      - name: Pin branch name in Notebooks
+        run: |
+          cd ${{ github.run_id }}
+          find tutorials -type f -name "*.ipynb" -exec sed -i "s/BRANCH = 'main'/BRANCH = '${{ needs.code-freeze.outputs.release-branch }}'/g" {} +
+
+      - name: Show status
+        run: |
+          cd ${{ github.run_id }}
+          git status
+
+      - name: Create PR
+        uses: peter-evans/create-pull-request@v6
+        id: create-pull-request
+        if: ${{ inputs.dry-run != true }}
+        with:
+          path: ${{ github.run_id }}
+          base: ${{ needs.code-freeze.outputs.release-branch }}
+          branch: ci/freeze-tags-${{ needs.code-freeze.outputs.release-branch }}
+          title: "Freeze tags in in `${{ needs.code-freeze.outputs.release-branch }}`"
+          body: |
+            🚀 PR to freeze tags in `${{ needs.code-freeze.outputs.release-branch }}`.
+
+          commit-message: "[🤠]: Howdy folks, let's release NeMo `${{ needs.code-freeze.outputs.release-branch }}` !"
+          signoff: true
+          assignees: okoenig

.github/workflows/release-nightly-docs.yml

@@ -0,0 +1,29 @@
+# Copyright (c) 2026, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Release Nightly Docs
+
+on:
+  schedule:
+    - cron: "0 10 * * *"
+
+jobs:
+  call-release-docs:
+    uses: ./.github/workflows/release-docs.yml
+    with:
+      dry-run: false
+      publish-as-latest: false
+      docs-version-override: "nightly"
+      update-version-picker: false
+    secrets: inherit

.github/workflows/release.yml

@@ -0,0 +1,127 @@
+# Copyright (c) 2020-2026, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: "Build, validate, and release Neural Modules"
+
+on:
+  push:
+    branches:
+      - main
+      - "r**"
+      - "pull-request/**"
+      - "deploy-release/*"
+  workflow_dispatch:
+    inputs:
+      release-ref:
+        description: Ref (SHA or branch name) to release
+        required: true
+        type: string
+      version-bump-branch:
+        description: Branch for version bump
+        required: true
+        type: string
+      dry-run:
+        description: Compute the release but do not publish wheel, GH release, or docs.
+        required: true
+        default: true
+        type: boolean
+      create-gh-release:
+        description: Create a GitHub release
+        required: true
+        default: true
+        type: boolean
+      generate-changelog:
+        description: Generate changelog
+        required: false
+        default: true
+        type: boolean
+      publish-docs:
+        description: Publish docs
+        required: false
+        default: true
+        type: boolean
+      gh-release-from-tag:
+        description: Tag of previous release for changelog builder
+        required: false
+        type: string
+        default: ""
+
+defaults:
+  run:
+    shell: bash -x -e -u -o pipefail {0}
+
+permissions:
+  id-token: write
+  contents: write
+  pull-requests: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}-${{ github.event_name }}
+  cancel-in-progress: ${{ github.event_name == 'push' }}
+
+jobs:
+  pre-flight:
+    if: github.event_name != 'workflow_dispatch'
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cicd_preflight.yml@v0.94.1
+
+  release:
+    needs: [pre-flight]
+    if: |
+      !cancelled() && !failure()
+      && !(needs.pre-flight.outputs.docs_only == 'true'
+           || needs.pre-flight.outputs.is_deployment_workflow == 'true')
+    uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_release_library.yml@v1.4.3
+    with:
+      release-ref: ${{ inputs.release-ref || github.sha }}
+      python-package: nemo
+      python-version: "3.10"
+      library-name: Neural Modules
+      validate-only: ${{ github.event_name != 'workflow_dispatch' }}
+      dry-run: ${{ inputs.dry-run || false }}
+      wheel-content-ignore: "W002,W004,W005,W009"
+      skip-check-manifest: true
+      version-bump-branch: ${{ inputs.version-bump-branch || github.ref_name }}
+      create-gh-release: ${{ inputs.create-gh-release || true }}
+      app-id: ${{ vars.BOT_ID }}
+      gh-release-use-changelog-builder: ${{ inputs.generate-changelog || false }}
+      publish-docs: ${{ inputs.publish-docs || true }}
+      docs-target-path: nemo
+      docs-directory: docs/source
+      docs-requirements-file: requirements/requirements_docs.txt
+      docs-sync-all: true
+      docs-no-extras: "--no-extra cu12 --no-extra compiled --no-extra compiled-a100"
+      publish-as-latest: true
+      run-on-version-tag-only: ${{ github.ref_name != 'main' }}
+      gh-release-from-tag: ${{ inputs.gh-release-from-tag || '' }}
+      restrict-to-admins: true
+    secrets: inherit  # pragma: allowlist secret
+
+  release-summary:
+    needs: [pre-flight, release]
+    if: ${{ !cancelled() }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Result
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          FAILED_JOBS=$(gh run view $GITHUB_RUN_ID --repo ${{ github.repository }} --json jobs --jq '[.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "action_required")] | length')
+
+          if [ "${FAILED_JOBS:-0}" -eq 0 ]; then
+              echo "✅ All previous jobs completed successfully"
+              exit 0
+          else
+              echo "❌ Found $FAILED_JOBS failed job(s)"
+              gh run view $GITHUB_RUN_ID --repo ${{ github.repository }} --json jobs --jq '.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "action_required") | .name'
+              exit 1
+          fi

.github/workflows/secrets-detector.yml

@@ -0,0 +1,43 @@
+# Copyright (c) 2020-2021, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: Secrets detector
+
+on:
+  pull_request_target:
+    branches:
+      - 'main'
+
+jobs:
+  main:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+          token: ${{ secrets.NEMO_REFORMAT_TOKEN }}
+
+      - name: Install secrets detector
+        run: pip install detect-secrets
+
+      - name: Run on change-set
+        run: |
+          git diff --name-only --diff-filter=d --merge-base origin/main -z | xargs -0 detect-secrets-hook --disable-plugin HexHighEntropyString --baseline .secrets.baseline
+      
+      - uses: EndBug/add-and-commit@v9
+        # Commit changes. Nothing is committed if no changes.
+        if: always()
+        with:
+          message: Update baseline
+          commit: --signoff

.gitignore

@@ -0,0 +1,197 @@
+# log and data files
+*.model
+*.pkl
+#*.ipynb
+output
+output_2048
+result
+*.pt
+tests/data/asr
+.DS_Store
+bert.pt.json
+work
+runs
+fastspeech_output
+.hydra
+.bash_history.local
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+**.pyc
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.idea
+.Python
+wandb
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+#parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/build
+docs/source/_build
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# Override Jupyter in Github Language states for more accurate estimate of repo code.
+# Reference: https://github.com/github/linguist/blob/master/docs/overrides.md#generated-code
+*.ipynb linguist-generated
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don’t work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# VSCode project settins
+.vscode/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+/docs/html
+/docs/docs_zh/zh
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# Emacs backup files
+*~
+
+cifar-10-batches-py
+*.tar.gz
+
+# Test data.
+tests/.data
+tests/data
+
+# outputs folder
+examples/*/outputs
+examples/*/NeMo_experiments
+examples/*/nemo_experiments
+examples/*/.hydra
+examples/*/wandb
+examples/*/data
+wandb
+dump.py
+
+docs/sources/source/test_build/
+
+# Checkpoints, config files and temporary files created in tutorials.
+examples/neural_graphs/*.chkpt
+examples/neural_graphs/*.yml
+
+.hydra/
+nemo_experiments/
+
+slurm*.out
+
+# Onelogger
+onelogger.*
+
+# voice agent
+node_modules/
+.vite/
+bot_server.*
+audio_logs/
+eval_results/
+
+# agents
+.claude/settings.local.json

.pre-commit-config.yaml

@@ -0,0 +1,45 @@
+# Copyright (c) 2022, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+default_language_version:
+  python: python3
+
+ci:
+  autofix_prs: true
+  autoupdate_commit_msg: '[pre-commit.ci] pre-commit suggestions'
+  autoupdate_schedule: quarterly
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.3.0
+    hooks:
+      - id: check-yaml
+      - id: check-case-conflict
+      - id: detect-private-key
+      - id: check-added-large-files
+        args: ['--maxkb=1000']
+      - id: requirements-txt-fixer
+
+  - repo: https://github.com/PyCQA/isort
+    rev: 5.13.2
+    hooks:
+      - id: isort
+        name: Format imports
+        exclude: docs/
+
+  # Using this mirror lets us use mypyc-compiled black, which is about 2x faster
+  - repo: https://github.com/psf/black-pre-commit-mirror
+    rev: 24.10.0
+    hooks:
+      - id: black

.pylintrc

@@ -0,0 +1,5 @@
+[MAIN]
+ignore-paths=tests
+max-line-length=119
+ignore-patterns=.*\.ipynb$
+recursive=y

.pylintrc.other

@@ -0,0 +1,12 @@
+[MAIN]
+ignore-paths=tests
+max-line-length=119
+
+[MESSAGES CONTROL]
+disable=all
+
+enable=C0115,C0116,W0611,C0301
+# C0115: missing-class-docstring
+# C0116: missing-function-docstring
+# W0611: unused-import
+# C0301: line-too-long

.pylintrc.speech

@@ -0,0 +1,9 @@
+[MAIN]
+ignore-paths=tests
+max-line-length=119
+
+[MESSAGES CONTROL]
+disable=all
+
+enable=W0611
+# W0611: unused-import

.readthedocs.yml

@@ -0,0 +1,35 @@
+# =============================================================================
+# Copyright (c) 2020 NVIDIA. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# =============================================================================
+
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required field.
+version: 2
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
+# Build documentation in the docs/ directory with Sphinx.
+sphinx:
+  configuration: docs/source/conf.py
+
+# Set the version of Python and requirements required to build your docs
+python:
+  install:
+    - requirements: requirements/requirements_docs.txt

AGENTS.md

@@ -0,0 +1 @@
+CLAUDE.md

CITATION.cff

@@ -0,0 +1,41 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+title: "NeMo: a toolkit for Conversational AI and Large Language Models"
+url: https://nvidia.github.io/NeMo/
+repository-code: https://github.com/NVIDIA/NeMo
+authors:
+  - family-names: Harper
+    given-names: Eric
+  - family-names: Majumdar
+    given-names: Somshubra
+  - family-names: Kuchaiev
+    given-names: Oleksii
+  - family-names: Jason
+    given-names: Li
+  - family-names: Zhang
+    given-names: Yang
+  - family-names: Bakhturina
+    given-names: Evelina
+  - family-names: Noroozi 
+    given-names: Vahid
+  - family-names: Subramanian
+    given-names: Sandeep
+  - family-names: Nithin
+    given-names: Koluguri
+  - family-names: Jocelyn
+    given-names: Huang
+  - family-names: Jia
+    given-names: Fei
+  - family-names: Balam
+    given-names: Jagadeesh
+  - family-names: Yang
+    given-names: Xuesong
+  - family-names: Livne
+    given-names: Micha
+  - family-names: Dong
+    given-names: Yi
+  - family-names: Naren
+    given-names: Sean
+  - family-names: Ginsburg
+    given-names: Boris
+

CLAUDE.md

@@ -0,0 +1,108 @@
+# CLAUDE.md / AGENTS.md
+
+This file provides guidance when working with code in this repository.
+
+## Project Overview
+
+NeMo Speech — toolkit for training/deploying speech models (ASR, TTS, Speech LLM). Active collections: `asr`, `tts`, `audio`, `speechlm2`, `common`. No Megatron / Megatron Core / Transformer Engine — parallelism is PyTorch-native (DDP, FSDP2, TP/SP via DTensor).
+
+## Build & Install
+
+See the canonical installation guide — [`docs/source/starthere/install.rst`](docs/source/starthere/install.rst) (published at https://docs.nvidia.com/nemo/speech/nightly/) — for the uv, pip (bring-your-own Python/PyTorch/CUDA), Docker, and optional `compiled` (SpeechLM2/Automodel) install paths.
+
+Dev quickstart: `uv sync --extra all --extra cu13` (Python 3.12+, PyTorch 2.7+; `test`/`docs` are `--group`s, not extras).
+
+## Code Style
+
+- **Line length: 119** (not default 88) — consistent across black, isort, flake8
+- Black with `skip_string_normalization = true`
+- isort with `profile = black`
+- Check: `isort --check <path> && black --check <path>` or `isort --check . && black --check .`
+- Fix: `isort <path> && black <path>` or `isort . && black .`
+- Jupyter Notebooks are excluded from automatic black reformatting (see `extend-exclude`), but can be still reformatted when passed directly. Do not reformat notebooks outside your changes.
+
+## Testing
+
+```bash
+pytest tests/collections/asr -m "not pleasefixme" -v     # ASR tests, skip broken
+pytest tests/collections/tts -m unit -v                  # TTS unit tests
+pytest -k "test_name" tests/                             # Single test by name
+```
+
+Markers: `unit`, `integration`, `system`, `pleasefixme` (broken — skip), `skipduringci`.
+
+## CI & PRs
+
+- NVIDIA developers: feature branches off `main`; community: fork-based workflow
+- CI triggered by adding **"Run CICD"** label to the PR
+- E2E nightly tests: only when really needed. Add both **"Run e2e nightly"** and **"Run CICD"** labels
+- `skip-linting` / `skip-docs` labels bypass those checks
+- Formatting CI auto-commits black/isort fixes back to the PR branch
+- CI: GitHub Actions in `.github/workflows/`
+
+## Documentation
+
+Sphinx-based docs live in `docs/source/`. Build with:
+
+```bash
+uv sync --locked --group docs                        # one-time setup (matches CI)
+uv run make -C docs clean html                       # full rebuild
+uv run make -C docs html                             # incremental rebuild
+```
+
+Output goes to `docs/build/html/`. Open `docs/build/html/index.html` to preview locally.
+
+Other useful targets: `make -C docs linkcheck` (verify external links), `make -C docs doctest` (run embedded doctests).
+
+## Training & Inference
+
+Entry-point scripts live under `examples/<collection>/`.
+
+All scripts follow the same Hydra pattern — a `@hydra_runner` decorator points to a YAML config in a nearby `conf/` directory:
+
+```python
+@hydra_runner(config_path="conf", config_name="fast-conformer_transducer_bpe")
+def main(cfg):
+    trainer = pl.Trainer(**resolve_trainer_cfg(cfg.trainer))
+    exp_manager(trainer, cfg.get("exp_manager", None))
+    model = EncDecRNNTBPEModel(cfg=cfg.model, trainer=trainer)
+    trainer.fit(model)
+```
+
+Override any config value from the CLI with Hydra syntax: `python script.py model.optim.lr=1e-4 trainer.max_epochs=50`. Browse configs with `ls examples/<collection>/conf/` to see which models and variants are supported.
+
+## Handy Scripts
+
+Utility scripts live under `scripts/`. Key subdirectories: `speech_recognition/`, `speechlm2/`, `speaker_tasks/`, `tokenizers/`, `dataset_processing/`, `asr_language_modeling/`. Browse with `ls scripts/`.
+
+Four frequently used data/training helpers:
+
+- **`scripts/speech_recognition/estimate_duration_bins.py`** — estimate Lhotse dynamic-bucketing duration bins from a manifest or YAML input config. Usage: `python scripts/speech_recognition/estimate_duration_bins.py <input> -b 30 -n 100000`
+- **`scripts/speech_recognition/oomptimizer.py`** — find the largest batch size per bucket that fits in GPU memory. Usage: `python scripts/speech_recognition/oomptimizer.py --pretrained-name nvidia/canary-1b` or point to a config with `--config-path`.
+- **`scripts/speech_recognition/estimate_data_weights.py`** — compute per-dataset sampling weights from YAML input configs, with optional temperature re-weighting. Usage: `python scripts/speech_recognition/estimate_data_weights.py input.yaml output.yaml -t 0.5`
+- **`scripts/speech_recognition/convert_to_tarred_audio_dataset.py`** — shard audio+manifest into tar files. Usage: `python scripts/speech_recognition/convert_to_tarred_audio_dataset.py --manifest_path=m.json --target_dir=./tar --num_shards=512 --max_duration=60.0`
+
+## Architecture
+
+- **Hydra + OmegaConf** for all config management (YAML configs)
+- **PyTorch Lightning** for training orchestration
+- **Lhotse** (>=1.32.2) for audio data loading
+- Collections are semi-isolated domains sharing `nemo.core` and `nemo.collections.common`
+
+## Subdirectory Instructions
+
+Module-specific instructions can be added as `CLAUDE.md` or `AGENTS.md` files in subdirectories.
+
+## Issue Reproduction
+
+When fixing a bug, always:
+1. First reproduce the issue with a minimal test case
+2. Add the reproduction as a unit test
+3. Then fix the issue
+4. Verify the test passes
+
+## Forbidden Operations
+
+- Never push directly to `main`
+- Never modify `.github/workflows/` without explicit instruction
+- Never delete test files without explicit instruction

CONTRIBUTING.md

@@ -0,0 +1,104 @@
+# Contributions are welcome!
+
+We do all of NeMo's development in the open. Contributions from NeMo community are welcome.
+
+
+# Pull Requests (PR) Guidelines
+
+**Send your PRs to the `main` branch**
+
+1) Make sure your PR does one thing. Have a clear answer to "What does this PR do?".
+2) Read General Principles and style guide below
+3) Make sure you sign your commits. E.g. use ``git commit -s`` when before your commit
+4) Make sure relevant unittests finish successfully before sending PR
+5) Send your PR and request a review
+
+## Unit tests
+Quick tests (locally, while developing)
+```
+pytest
+# If you don't have NVIDIA GPU do:
+# pytest -m "not pleasefixme" --cpu path/to/relevant_tests
+```
+Full tests, including pre-trained model downloads
+```
+pytest -m "not pleasefixme" --with_downloads  path/to/relevant_tests
+```
+
+Replace `path/to/relevant_tests` with the test directory to run such as `tests/collections/asr`. Check the test scripts in `tests/functional_tests`
+that begin with `L0_Unit_Tests_` for the specific test configuration used by different parts of the unit test suite. Different suites may expect
+different environment variables to be set.
+
+## Running the Github CI
+
+CI is powered by [copy-pr-bot](https://github.com/apps/copy-pr-bot), which mirrors each PR to a `pull-request/<number>` branch and triggers the workflow on push.
+
+**CI runs automatically** when all of the following are true:
+- Every commit in the PR is [GPG-signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)
+- Every committer is a member of the NVIDIA-NeMo GitHub org (or listed as an `additional_trustee`)
+- The PR has no more than 249 commits
+
+If any of those conditions are not met, copy-pr-bot posts a comment and skips branch creation — CI will not run. A maintainer (anyone with write access or greater) can trigger it manually by commenting:
+```
+/ok to test <sha>
+```
+where `<sha>` is the full or abbreviated SHA of the PR's HEAD commit. The bot also accepts `/okay to test` and `/ok-to-test`.
+
+To re-run CI after a new push, the bot will sync automatically if the PR is still trusted. Otherwise comment `/ok to test <sha>` again with the new HEAD SHA.
+
+The CI test suites are selectively ran based on the files that are changed. In some cases, no tests may be ran such as when docs are updated.
+
+Lint checks using flake8 and pylint are ran on the code based on the files that were changed. Please resolve any lint errors. It is possible but discouraged
+to ignore the lint errors by adding the "skip-linting" label to the PR.
+
+To run the nightly e2e test suite on a PR, add the "Run e2e nightly" label. Labels are read once by the `pre-flight` job at the start of each run, so the label must be present before CI starts. If CI is already running when you add the label, cancel the active workflow run and re-trigger by pushing a new commit.
+
+## Whom should you ask for review:
+Please tag @nithinraok for NeMo core and ASR related PRs and @blisc for TTS related PRs.
+
+Note that some people may self-assign to review your PR - in which case, please wait for them to add a review.
+
+Your  pull requests must pass all checks and peer-review before they can be merged.
+
+# General principles
+1. **User-oriented**: make it easy for end users, even at the cost of writing more code in the background
+1. **Robust**: make it hard for users to make mistakes.
+1. **Well-tested**: please add simple, fast unittests. Consider adding CI tests for end-to-end functionality.
+1. **Reusable**: for every piece of code, think about how it can be reused in the future and make it easy to be reused.
+1. **Readable**: code should be easier to read.
+1. **Legal**: if you copy even one line of code from the Internet, make sure that the code allows the license that NeMo supports. Give credit and link back to the code.
+1. **Sensible**: code should make sense. If you think a piece of code might be confusing, write comments.
+
+## Class naming conventions
+* No “I”, “Interface”, “NM” nor “NeMo” pre/postfixes anywhere
+* Core interfaces have simple names: Typing, Cloud, Serialization, FileIO*
+* Core classes have the simplest names ever: NeuralModule, Model, Graph, Dataset, Loss, Module*
+* Abstract classes in the Model hierarchy have Model postfix
+* A config class for MyModel should be called MyModelConfig
+* Leaf Neural Module classes have simple names without any postfixes (e.g. AudioPreprocess)
+* Leaf Datasets have Dataset postfix (e.g. AudioToSpeechLabelDataset)
+* Leaf Losses have Loss postfix (e.g. CTCLoss)
+* Leaf Models do not have any postfix, just name (e.g. QuartzNet)
+
+## Python style
+We use ``black`` as our style guide. To check whether your code will pass style check (from the NeMo's repo folder) run:
+``python setup.py style --scope path/to/changed/files`` and if it does not pass run ``python setup.py style --scope path/to/changed/files --fix``.
+
+1. Include docstrings for every class and method exposed to the user.
+1. Use Python 3 type hints for every class and method exposed to the user.
+1. Avoid wild import: ``from X import *`` unless in ``X.py``, ``__all__`` is defined.
+1. Minimize the use of ``**kwargs``.
+1. ``RaiseError`` is preferred to ``assert``. Write: ```if X: raise Error``` instead of ```assert X```.
+1. Classes are preferred to standalone methods.
+1. Methods should be atomic. A method shouldn't be longer than 75 lines, e.g. can be fit into the computer screen without scrolling.
+1. If a method has arguments that don't fit into one line, each argument should be in its own line for readability.
+1. Add ``__init__.py`` for every folder.
+1. F-strings are prefered to formatted strings.
+1. Loggers are preferred to print. In NeMo, you can use logger from ``from nemo.utils import logging``
+1. Private functions (functions start with ``_``) shouldn't be called outside its host file.
+1. If a comment lasts multiple lines, use ``'''`` instead of ``#``.
+
+# Collections
+Collection is a logical grouping of related Neural Modules. It is a grouping of modules that share a domain area or semantics.
+When contributing module to a collection, please make sure it belongs to that category.
+If you would like to start a new one and contribute back to the platform, you are very welcome to do so.

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

MANIFEST.in

@@ -0,0 +1 @@
+include requirements/*

README.md

@@ -0,0 +1,128 @@
+[![Project Status: Active -- The project has reached a stable, usable state and is being actively developed.](http://www.repostatus.org/badges/latest/active.svg)](http://www.repostatus.org/#active)
+[![Documentation](https://readthedocs.com/projects/nvidia-nemo/badge/?version=main)](https://docs.nvidia.com/nemo/speech/nightly/)
+[![CodeQL](https://github.com/nvidia/nemo/actions/workflows/codeql.yml/badge.svg?branch=main&event=push)](https://github.com/nvidia/nemo/actions/workflows/codeql.yml)
+[![NeMo core license and license for collections in this repo](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](https://github.com/NVIDIA/NeMo/blob/master/LICENSE)
+[![Release version](https://badge.fury.io/py/nemo-toolkit.svg)](https://badge.fury.io/py/nemo-toolkit)
+[![Python version](https://img.shields.io/pypi/pyversions/nemo-toolkit.svg)](https://badge.fury.io/py/nemo-toolkit)
+[![PyPi total downloads](https://static.pepy.tech/personalized-badge/nemo-toolkit?period=total&units=international_system&left_color=grey&right_color=brightgreen&left_text=downloads)](https://pepy.tech/project/nemo-toolkit)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+# **NVIDIA NeMo Speech**
+Checkout our [HuggingFace🤗 collection](https://huggingface.co/collections/nvidia/nemotron-speech) for the latest open
+weight checkpoints and demos!
+
+## Updates
+
+> The first release of NeMo Speech after NeMo repository split is scheduled for June 2026, as the repo undergoes transformation.
+> For the latest stable released version, please use [the 26.02 NGC container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo?version=26.02).
+
+- 2026-06: [Nemotron-3.5-ASR-Streaming-0.6B](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b) has been released with 40 languages supported, controllable latency 80ms-1s, and 240-2400 1xH100 concurrent streams. Built on cache-aware Fastconformer architecture.
+- 2026-04: [Parakeet-unified-en-0.6b](https://huggingface.co/nvidia/parakeet-unified-en-0.6b) has been released with high-quality offline and streaming (with a minimum latency of 160ms) inference in one model for English language with punctuation and capitalization support. 
+- 2026-03: [Nemotron 3 VoiceChat](https://build.nvidia.com/nvidia/nemotron-voicechat/modelcard) is now released in Early Access. Built on the Nemotron Nano v2 LLM backbone with Nemotron speech and TTS decoder, VoiceChat delivers full-duplex, natural, interruptible conversations with low latency. Try out [the demo](https://build.nvidia.com/nvidia/nemotron-voicechat) and apply for [early access](https://developer.nvidia.com/nemotron-voicechat-early-access).
+- 2026-03: [Nemotron-Speech-Streaming v2603](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b) has been
+    updated. It has been trained on a larger and more diverse corpus, resulting in lower WER across all latency modes.
+    Try out [the demo](https://huggingface.co/spaces/nvidia/nemotron-speech-streaming-en-0.6b) and check out
+    [the NIM](https://build.nvidia.com/nvidia/nemotron-asr-streaming).
+- 2026-03: [MagpieTTS v2602](https://huggingface.co/nvidia/magpie_tts_multilingual_357m) has been released with support
+    for 9 languages(En, Es, De, Fr, Vi, It, Zh, Hi, Ja). Try out
+    [the demo](https://huggingface.co/nvidia/magpie_tts_multilingual_357m) and check out
+    [the NIM](https://build.nvidia.com/nvidia/magpie-tts-multilingual).
+- 2026-01: Nemotron-Speech-Streaming was released: One checkpoint that enables users to pick their optimal point
+    on the latency-accuracy Pareto curve!
+- 2026-01: MagpieTTS was released.
+- 2026: This repo has pivoted to focus on audio, speech, and multimodal LLM. For the last NeMo release with support for more
+    modalities, see [v2.7.0](https://github.com/NVIDIA-NeMo/NeMo/releases/tag/v2.7.0)
+- 2025-08: [Parakeet V3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) and
+    [Canary V2](https://huggingface.co/nvidia/canary-1b-v2) have been released with speech recognition and translation
+    support for 25 European languages.
+- 2025-06: [Canary-Qwen-2.5B](https://huggingface.co/nvidia/canary-qwen-2.5b) has been released with record-setting
+    5.63% WER on English Open ASR Leaderboard.
+
+## Introduction
+
+NVIDIA NeMo Speech is built for researchers and PyTorch developers working on Speech models including Automatic Speech
+Recognition (ASR), Text to Speech (TTS), and Speech LLMs. It is designed to help you efficiently create, customize, and
+deploy new AI models by leveraging existing code and pre-trained model checkpoints.
+
+For technical documentation, please see the
+[NeMo Framework User Guide](https://docs.nvidia.com/nemo/speech/nightly/).
+
+## Requirements
+
+NeMo Speech works with the **Python, PyTorch, and CUDA versions of your choosing**:
+
+- Python 3.12 or above
+- PyTorch 2.7 or above (CPU, CUDA, etc. — your choice)
+- NVIDIA GPU + CUDA (required for training; recommended for inference)
+
+If you already have a Python/PyTorch/CUDA stack that satisfies those minimums, NeMo Speech installs on top of it **without replacing it**, so your existing PyTorch build is kept (see the install options below). The versions pinned in `uv.lock` and shipped in the official container — Python 3.13, PyTorch 2.12, CUDA 12.6/13.2 — are simply the combination we actively test and support. They make setup turnkey and reproducible, but they are **not** a hard requirement.
+
+As of [Pytorch 2.6](https://docs.pytorch.org/docs/stable/notes/serialization.html#torch-load-with-weights-only-true),
+`torch.load` defaults to using `weights_only=True`. Some model checkpoints may require using `weights_only=False`.
+In this case, you can set the env var `TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1` before running code that uses `torch.load`.
+However, this should only be done with trusted files. Loading files from untrusted sources with more than weights only
+can have the risk of arbitrary code execution.
+
+## Developer Documentation
+
+| Version | Status                                                                                                                                                              | Description                                                                                                                    |
+| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| Latest  | [![Documentation Status](https://readthedocs.com/projects/nvidia-nemo/badge/?version=main)](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/)     | [Documentation of the latest (i.e. main) branch.](https://docs.nvidia.com/nemo/speech/nightly/)          |
+| Stable  | [![Documentation Status](https://readthedocs.com/projects/nvidia-nemo/badge/?version=stable)](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/) | Documentation of the stable (i.e. most recent release) - To be added |
+
+## Install NeMo Speech
+
+The recommended way to install NeMo Speech is from source with [uv](https://docs.astral.sh/uv/), which reproduces our actively-tested stack from the committed `uv.lock`. If you need different Python/PyTorch/CUDA versions, NeMo also installs over your existing environment via pip — see the [pip fallback](#from-pypi-with-pip-fallback--bring-your-own-versions) below.
+
+### From source with uv (recommended)
+
+```bash
+git clone https://github.com/NVIDIA-NeMo/NeMo.git
+cd NeMo
+uv sync --extra all --extra cu13     # CUDA 13.x (recommended) — use --extra cu12 for CUDA 12.x
+```
+
+This installs our supported stack (Python 3.13, PyTorch 2.12, CUDA 13.2) into `.venv/` with NeMo editable. Add `--group test` for the test suite or `--group docs` to build the docs; run tools via `uv run <cmd>` or activate with `source .venv/bin/activate`. On Linux, `cu12` and `cu13` are mutually exclusive — pass exactly one (`cu13` is the default). For the **exact** container baseline, add `--locked --python 3.13` (the path the Dockerfile and CI use).
+
+> **SpeechLM2 / Automodel:** the Automodel backend runs **without** any compiled dependencies. It can *optionally* benefit from dedicated accelerated backends (Transformer Engine, FlashAttention, Mamba, grouped-GEMM/MoE, DeepEP) for better performance — these source-built kernels come from the `compiled` (Hopper/Blackwell) or `compiled-a100` (A100) extras, built by `docker/Dockerfile` (`GPU_TARGET=h100plus` / `a100`). See the [installation guide](https://docs.nvidia.com/nemo/speech/nightly/) for the full list and build details.
+
+### Docker (turnkey, our supported stack)
+
+> **NGC container:** _Coming soon — the pull command for the prebuilt NeMo Speech container image will be published here._
+
+To build the container from source (CUDA 13 / H100+ by default):
+
+```bash
+git clone https://github.com/NVIDIA-NeMo/NeMo.git
+cd NeMo
+docker buildx build -f docker/Dockerfile -t nemo-speech .          # CUDA 13 / H100+ (default)
+docker run --rm -it --gpus all -v "$PWD:/workspace" nemo-speech bash
+```
+
+For A100, set `GPU_TARGET=a100` — A100 works with **both CUDA 12 and CUDA 13** (CUDA 13, the default base image, is recommended; the CUDA 12 base is a convenience). See the header of [`docker/Dockerfile`](docker/Dockerfile) for all build arguments (`BASE_IMAGE`, `GPU_TARGET`).
+
+### From PyPI with pip (fallback — bring your own versions)
+
+Prefer your own Python/PyTorch/CUDA? Install your PyTorch first (any version ≥ 2.7 for your CPU/CUDA/etc. target — see the [PyTorch install matrix](https://pytorch.org/get-started/locally/)), then add NeMo and it **keeps your build**. `uv pip` (uv's fast, pip-compatible installer) works like `pip`:
+
+```bash
+uv pip install 'nemo-toolkit[asr,tts]'   # or plain: pip install 'nemo-toolkit[asr,tts]'
+```
+
+> ⚠️ Do **not** use `uv sync --locked` for a bring-your-own stack — it applies `uv.lock` and replaces your Python/PyTorch/CUDA with the supported baseline. Use `uv pip`/`pip` here; reserve `uv sync --locked` for reproducing our stack.
+
+To instead pull *our* pinned PyTorch build, add the CUDA extra and the matching wheel index (pip/uv pip do not read uv's project index config, so `--extra-index-url` is required):
+
+```bash
+pip install 'nemo-toolkit[asr,tts,cu13]' --extra-index-url https://download.pytorch.org/whl/cu132   # CUDA 13.x
+pip install 'nemo-toolkit[asr,tts,cu12]' --extra-index-url https://download.pytorch.org/whl/cu126   # CUDA 12.x
+```
+
+## Contribute to NeMo
+
+We welcome community contributions! Please refer to
+[CONTRIBUTING.md](https://github.com/NVIDIA-NeMo/NeMo/blob/main/CONTRIBUTING.md) for the process.
+
+## Licenses
+
+NeMo is licensed under the [Apache License 2.0](https://github.com/NVIDIA/NeMo?tab=Apache-2.0-1-ov-file).

README.wehub.md

@@ -0,0 +1,7 @@
+# WeHub 来源说明
+
+- 原始项目：`NVIDIA-NeMo/Speech`
+- 原始仓库：https://github.com/NVIDIA-NeMo/Speech
+- 导入方式：上游默认分支的最新快照
+- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
+- 本文件仅用于记录来源，不代表 WeHub 是原项目作者

SECURITY.md

@@ -0,0 +1,25 @@
+## Security
+
+NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization.
+
+If you need to report a security issue, please use the appropriate contact points outlined below. **Please do not report security vulnerabilities through GitHub.** If a potential security issue is inadvertently reported via a public issue or pull request, NVIDIA maintainers may limit public discussion and redirect the reporter to the appropriate private disclosure channels.
+
+## Reporting Potential Security Vulnerability in an NVIDIA Product
+
+To report a potential security vulnerability in any NVIDIA product:
+
+- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html)
+- E-Mail: psirt@nvidia.com
+  - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key)
+  - Please include the following information:
+    - Product/Driver name and version/branch that contains the vulnerability
+  - Type of vulnerability (code execution, denial of service, buffer overflow, etc.)
+    - Instructions to reproduce the vulnerability
+    - Proof-of-concept or exploit code
+    - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability
+
+While NVIDIA currently does not have a bug bounty program, we do offer acknowledgement when an externally reported security issue is addressed under our coordinated vulnerability disclosure policy. Please visit our [Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) policies page for more information.
+
+## NVIDIA Product Security
+
+For all security-related concerns, please visit NVIDIA's Product Security portal at https://www.nvidia.com/en-us/security

codecov.yml

@@ -0,0 +1,23 @@
+comment: false
+coverage:
+  status:
+    project: false
+    patch:
+      default:
+        target: 80%
+        threshold: 0%
+        base: auto
+        if_ci_failed: error
+        paths:
+          - "!nemo/collections/asr"
+          - "!nemo/collections/audio"
+          - "!nemo/collections/speechlm"
+          - "!nemo/collections/tts"
+          - "!nemo/core"
+          - "!nemo/collections/common"
+          - "!nemo/collections/multimodal_autoregressive"
+          - "!nemo/collections/diffusion"
+          - "!nemo/collections/nlp"
+
+fixes:
+  - "/workspace/::"

docker/Dockerfile

@@ -0,0 +1,294 @@
+# syntax=docker/dockerfile:1-labs
+
+# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This CI Dockerfile supports CUDA 13 and CUDA 12 from a single BASE_IMAGE
+# build arg. The default is the recommended CUDA 13 image:
+#   nvcr.io/nvidia/cuda-dl-base:26.04-cuda13.2-devel-ubuntu24.04
+# The current recommended CUDA 12 image is:
+#   nvcr.io/nvidia/cuda-dl-base:25.06-cuda12.9-devel-ubuntu24.04
+#
+# The build derives CUDA_FLAVOR internally from BASE_IMAGE by matching
+# "cuda13" or "cuda12" in the image tag. That flavor selects the matching uv
+# extra (cu13 or cu12) and CUDA Python package include path. If BASE_IMAGE does
+# not contain either token, the build fails early.
+#
+# Example CUDA 13 H100+ build:
+#   docker buildx build -f docker/Dockerfile \
+#     --build-arg GPU_TARGET=h100plus .
+# Example CUDA 12 A100 build:
+#   docker buildx build -f docker/Dockerfile \
+#     --build-arg BASE_IMAGE=nvcr.io/nvidia/cuda-dl-base:25.06-cuda12.9-devel-ubuntu24.04 \
+#     --build-arg GPU_TARGET=a100 .
+#
+# GPU_TARGET controls compiled Automodel dependency tuning. "h100plus" builds
+# for SM90/SM100/SM120 and includes H100+ features such as DeepEP and
+# flash-attn-4. "a100" builds only SM80 and uses an A100-specific DeepEP patch
+# to avoid unsupported newer-GPU/NVSHMEM build paths. This keeps CI images
+# smaller and avoids compiling kernels for architectures a target image cannot
+# use.
+ARG BASE_IMAGE=nvcr.io/nvidia/cuda-dl-base:26.04-cuda13.2-devel-ubuntu24.04
+FROM ${BASE_IMAGE} AS base-image
+ARG BASE_IMAGE
+ARG UV_VERSION=0.11.27
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV TRANSFORMERS_OFFLINE=0
+ENV HYDRA_FULL_ERROR=1
+ENV PYTHONUNBUFFERED=1
+ENV UV_PYTHON=3.13
+ENV UV_PROJECT_ENVIRONMENT=/opt/venv
+ENV UV_LINK_MODE=copy
+ENV UV_NO_CACHE=1
+ENV UV_NO_PROGRESS=1
+ENV VIRTUAL_ENV=/opt/venv
+ENV PATH="/root/.local/bin:${VIRTUAL_ENV}/bin:${PATH}"
+
+RUN <<"EOF" bash -ex
+apt-get update
+apt-get install -y \
+    curl \
+    git
+apt-get clean
+EOF
+# NOTE: removed libsndfile1. Soundfile ships with its own version of libsndfile1
+
+RUN <<"EOF" bash -ex
+curl -LsSf "https://astral.sh/uv/${UV_VERSION}/install.sh" | sh
+uv --version
+EOF
+
+RUN <<"EOF" bash -euxo pipefail
+case "${BASE_IMAGE}" in
+    *cuda12*) cuda_flavor=cu12 ;;
+    *cuda13*) cuda_flavor=cu13 ;;
+    *)
+        echo "Cannot derive CUDA flavor from BASE_IMAGE='${BASE_IMAGE}'. Expected image tag containing 'cuda12' or 'cuda13'."
+        exit 1
+        ;;
+esac
+cuda_major_minor="$(sed -n 's/.*cuda\([0-9][0-9]*\.[0-9][0-9]*\).*/\1/p' <<<"${BASE_IMAGE}")"
+if [[ -z "${cuda_major_minor}" ]]; then
+    echo "Cannot derive CUDA major.minor from BASE_IMAGE='${BASE_IMAGE}'. Expected image tag containing e.g. 'cuda12.9' or 'cuda13.2'."
+    exit 1
+fi
+cat >/usr/local/bin/nemo-cuda-flavor <<SCRIPT
+#!/usr/bin/env bash
+set -euo pipefail
+echo "${cuda_flavor}"
+SCRIPT
+cat >/usr/local/bin/nemo-install-cuda-python <<SCRIPT
+#!/usr/bin/env bash
+set -euo pipefail
+cuda_major_minor="${cuda_major_minor}"
+cuda_major="\${cuda_major_minor%%.*}"
+cuda_minor="\${cuda_major_minor#*.}"
+cuda_next_minor="\$((cuda_minor + 1))"
+uv pip install \
+    "cuda-bindings>=\${cuda_major_minor}.0,<\${cuda_major}.\${cuda_next_minor}" \
+    "cuda-python>=\${cuda_major_minor}.0,<\${cuda_major}.\${cuda_next_minor}"
+SCRIPT
+chmod +x /usr/local/bin/nemo-cuda-flavor /usr/local/bin/nemo-install-cuda-python
+echo "Derived CUDA flavor: ${cuda_flavor}"
+echo "Derived CUDA Python major.minor: ${cuda_major_minor}"
+EOF
+
+WORKDIR /workspace
+COPY pyproject.toml uv.lock /workspace/
+COPY nemo/__init__.py nemo/package_info.py /workspace/nemo/
+RUN <<"EOF" bash -ex
+cuda_flavor="$(nemo-cuda-flavor)"
+uv sync --link-mode copy --locked --extra all --extra "${cuda_flavor}" --group test
+nemo-install-cuda-python
+EOF
+
+RUN <<"EOF" bash -ex
+# Container-only runtime utilities. Keep these out of pyproject.toml so they do
+# not become NeMo package dependencies.
+uv pip install \
+    dill \
+    orjson
+
+case "$(nemo-cuda-flavor)" in
+    cu12) torchcodec_index=https://download.pytorch.org/whl/cu126 ;;
+    cu13) torchcodec_index=https://download.pytorch.org/whl/cu132 ;;
+esac
+uv pip install --index-url "${torchcodec_index}" torchcodec
+EOF
+
+FROM base-image AS automodel-deps
+ARG GPU_TARGET=h100plus
+ARG DEEPEP_REPO=https://github.com/deepseek-ai/DeepEP.git
+ARG DEEPEP_REF=v1.2.1
+ARG FLASH_ATTN_MAX_JOBS=8
+ARG FLASH_ATTN_NVCC_THREADS=1
+
+ENV PIP_NO_CACHE_DIR=1
+
+COPY external/patches/deepep_v1_a100_disable_nvshmem.patch /tmp/patches/
+RUN <<"EOF" bash -euxo pipefail
+apt-get update
+apt-get install -y \
+    build-essential \
+    ca-certificates \
+    cmake \
+    ninja-build \
+    patch
+apt-get clean
+rm -rf /var/lib/apt/lists/*
+EOF
+
+RUN <<"EOF" bash -euxo pipefail
+cat >/usr/local/bin/automodel-build-env <<'SCRIPT'
+case "${GPU_TARGET}" in
+    h100plus)
+        export DEEPEP_TORCH_CUDA_ARCH_LIST="9.0+PTX"
+        export TORCH_CUDA_ARCH_LIST="9.0;10.0;12.0"
+        export NVTE_CUDA_ARCHS="90;100;120"
+        export FLASH_ATTN_CUDA_ARCHS="90;100;120"
+        export CMAKE_CUDA_ARCHITECTURES="90;100;120"
+        ;;
+    a100)
+        export DEEPEP_TORCH_CUDA_ARCH_LIST="8.0"
+        export TORCH_CUDA_ARCH_LIST="8.0"
+        export NVTE_CUDA_ARCHS="80"
+        export FLASH_ATTN_CUDA_ARCHS="80"
+        export CMAKE_CUDA_ARCHITECTURES="80"
+        export DISABLE_SM90_FEATURES=1
+        ;;
+    *)
+        echo "Unsupported GPU_TARGET='${GPU_TARGET}'. Expected 'h100plus' or 'a100'."
+        exit 1
+        ;;
+esac
+
+CUDA_FLAVOR="$(nemo-cuda-flavor)"
+AUTOMODEL_CCCL_INCLUDES="/usr/local/cuda/include/cccl"
+PYTHON_CCCL_INCLUDE="${VIRTUAL_ENV}/lib/python${UV_PYTHON}/site-packages/nvidia/${CUDA_FLAVOR}/include/cccl"
+if [[ -d "${PYTHON_CCCL_INCLUDE}" ]]; then
+    AUTOMODEL_CCCL_INCLUDES="${AUTOMODEL_CCCL_INCLUDES}:${PYTHON_CCCL_INCLUDE}"
+fi
+export CPATH="${AUTOMODEL_CCCL_INCLUDES}${CPATH:+:${CPATH}}"
+export CPLUS_INCLUDE_PATH="${AUTOMODEL_CCCL_INCLUDES}${CPLUS_INCLUDE_PATH:+:${CPLUS_INCLUDE_PATH}}"
+export NVTE_FRAMEWORK=pytorch
+export MAX_JOBS="${FLASH_ATTN_MAX_JOBS}"
+export NVCC_THREADS="${FLASH_ATTN_NVCC_THREADS}"
+SCRIPT
+mkdir -p /opt/automodel-src
+EOF
+
+COPY external/patches/flash_attn_4_sm120_disable_tma_o.patch /tmp/patches/
+RUN <<"EOF" bash -euxo pipefail
+source /usr/local/bin/automodel-build-env
+uv pip install --upgrade \
+    cmake \
+    ninja \
+    nvidia-mathdx \
+    packaging \
+    pip \
+    psutil \
+    pybind11 \
+    setuptools \
+    setuptools-scm \
+    wheel
+
+automodel_extra=compiled
+if [[ "${GPU_TARGET}" == "h100plus" ]]; then
+    automodel_extra=compiled
+elif [[ "${GPU_TARGET}" == "a100" ]]; then
+    automodel_extra=compiled-a100
+fi
+cuda_flavor="$(nemo-cuda-flavor)"
+uv sync \
+    --inexact \
+    --link-mode copy \
+    --locked \
+    --extra all \
+    --extra "${cuda_flavor}" \
+    --extra "${automodel_extra}" \
+    --group test
+nemo-install-cuda-python
+
+if [[ "${GPU_TARGET}" == "a100" ]]; then
+    git clone "${DEEPEP_REPO}" /opt/automodel-src/DeepEP
+    git -C /opt/automodel-src/DeepEP checkout -f "${DEEPEP_REF}"
+    patch -d /opt/automodel-src/DeepEP -p1 < /tmp/patches/deepep_v1_a100_disable_nvshmem.patch
+    TORCH_CUDA_ARCH_LIST="${DEEPEP_TORCH_CUDA_ARCH_LIST}" \
+        uv pip install \
+        --no-build-isolation \
+        --no-deps \
+        /opt/automodel-src/DeepEP
+fi
+
+if [[ "${GPU_TARGET}" == "h100plus" ]]; then
+    # flash-attn-4 requires apache-tvm-ffi 0.1.11, while mamba-ssm
+    # currently constrains the solved environment to apache-tvm-ffi<=0.1.9.
+    cutlass_packages=(
+        "nvidia-cutlass-dsl==4.5.2"
+        "nvidia-cutlass-dsl-libs-base==4.5.2"
+    )
+    if [[ "$(nemo-cuda-flavor)" == "cu13" ]]; then
+        cutlass_packages+=("nvidia-cutlass-dsl-libs-cu13==4.5.2")
+    fi
+    uv pip install \
+        --no-deps \
+        "apache-tvm-ffi==0.1.11" \
+        "${cutlass_packages[@]}" \
+        "quack-kernels==0.5.0" \
+        "torch-c-dlpack-ext==0.1.5"
+
+    mkdir -p /opt/automodel-src/flash-attn-4 /tmp/flash-attn-4
+    python -m pip download \
+        --no-binary=flash-attn-4 \
+        --no-deps \
+        --dest /tmp/flash-attn-4 \
+        "flash-attn-4==4.0.0b15"
+    tar -xf /tmp/flash-attn-4/flash_attn_4-*.tar.gz -C /opt/automodel-src/flash-attn-4 --strip-components=1
+    patch -d /opt/automodel-src/flash-attn-4 -p1 < /tmp/patches/flash_attn_4_sm120_disable_tma_o.patch
+
+    MAX_JOBS="${FLASH_ATTN_MAX_JOBS}" \
+    NVCC_THREADS="${FLASH_ATTN_NVCC_THREADS}" \
+    uv pip install \
+        --no-build-isolation \
+        --no-deps \
+        /opt/automodel-src/flash-attn-4
+fi
+EOF
+
+FROM base-image
+COPY --from=automodel-deps /opt/venv /opt/venv
+
+ENV NVIDIA_BUILD_ID=${NVIDIA_BUILD_ID:-<unknown>}
+LABEL com.nvidia.build.id="${NVIDIA_BUILD_ID}"
+ARG NVIDIA_BUILD_REF
+LABEL com.nvidia.build.ref="${NVIDIA_BUILD_REF}"
+ARG RC_DATE=00.00
+ARG TARGETARCH
+
+ARG INSTALL_FFMPEG=false
+RUN <<"EOF" bash -ex
+if [ "${INSTALL_FFMPEG}" = "true" ]; then
+    apt-get update
+    apt-get install -y ffmpeg
+    apt-get clean
+    rm -rf /var/lib/apt/lists/*
+fi
+EOF
+
+COPY nemo /workspace/nemo
+
+# NOTICES.txt file points to where the OSS source code is archived
+RUN echo "This distribution includes open source which is archived at the following URL: https://opensource.nvidia.com/oss/teams/nvidia/nemo/${RC_DATE}:linux-${TARGETARCH}/index.html" > NOTICES.txt && \
+    echo "For further inquiries or assistance, contact us at oss-requests@nvidia.com" >> NOTICES.txt

docs/Makefile

@@ -0,0 +1,216 @@
+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+PAPER         =
+BUILDDIR      = build
+
+# User-friendly check for sphinx-build
+ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
+$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
+endif
+
+# Internal variables.
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
+
+.PHONY: help
+help:
+	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  html       to make standalone HTML files"
+	@echo "  dirhtml    to make HTML files named index.html in directories"
+	@echo "  singlehtml to make a single large HTML file"
+	@echo "  pickle     to make pickle files"
+	@echo "  json       to make JSON files"
+	@echo "  htmlhelp   to make HTML files and a HTML help project"
+	@echo "  qthelp     to make HTML files and a qthelp project"
+	@echo "  applehelp  to make an Apple Help Book"
+	@echo "  devhelp    to make HTML files and a Devhelp project"
+	@echo "  epub       to make an epub"
+	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
+	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
+	@echo "  text       to make text files"
+	@echo "  man        to make manual pages"
+	@echo "  texinfo    to make Texinfo files"
+	@echo "  info       to make Texinfo files and run them through makeinfo"
+	@echo "  gettext    to make PO message catalogs"
+	@echo "  changes    to make an overview of all changed/added/deprecated items"
+	@echo "  xml        to make Docutils-native XML files"
+	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
+	@echo "  linkcheck  to check all external links for integrity"
+	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
+	@echo "  coverage   to run coverage check of the documentation (if enabled)"
+
+.PHONY: clean
+clean:
+	rm -rf $(BUILDDIR)/*
+
+.PHONY: html
+html:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+.PHONY: dirhtml
+dirhtml:
+	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+.PHONY: singlehtml
+singlehtml:
+	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+	@echo
+	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+.PHONY: pickle
+pickle:
+	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+	@echo
+	@echo "Build finished; now you can process the pickle files."
+
+.PHONY: json
+json:
+	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+	@echo
+	@echo "Build finished; now you can process the JSON files."
+
+.PHONY: htmlhelp
+htmlhelp:
+	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+	@echo
+	@echo "Build finished; now you can run HTML Help Workshop with the" \
+	      ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+.PHONY: qthelp
+qthelp:
+	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+	@echo
+	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
+	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/OpenSeq2Seq.qhcp"
+	@echo "To view the help file:"
+	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/OpenSeq2Seq.qhc"
+
+.PHONY: applehelp
+applehelp:
+	$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
+	@echo
+	@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
+	@echo "N.B. You won't be able to view it unless you put it in" \
+	      "~/Library/Documentation/Help or install it in your application" \
+	      "bundle."
+
+.PHONY: devhelp
+devhelp:
+	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+	@echo
+	@echo "Build finished."
+	@echo "To view the help file:"
+	@echo "# mkdir -p $$HOME/.local/share/devhelp/OpenSeq2Seq"
+	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/OpenSeq2Seq"
+	@echo "# devhelp"
+
+.PHONY: epub
+epub:
+	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+	@echo
+	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+.PHONY: latex
+latex:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo
+	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+	@echo "Run \`make' in that directory to run these through (pdf)latex" \
+	      "(use \`make latexpdf' here to do that automatically)."
+
+.PHONY: latexpdf
+latexpdf:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through pdflatex..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+.PHONY: latexpdfja
+latexpdfja:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through platex and dvipdfmx..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+.PHONY: text
+text:
+	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+	@echo
+	@echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+.PHONY: man
+man:
+	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+	@echo
+	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+.PHONY: texinfo
+texinfo:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo
+	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+	@echo "Run \`make' in that directory to run these through makeinfo" \
+	      "(use \`make info' here to do that automatically)."
+
+.PHONY: info
+info:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo "Running Texinfo files through makeinfo..."
+	make -C $(BUILDDIR)/texinfo info
+	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+.PHONY: gettext
+gettext:
+	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+	@echo
+	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+.PHONY: changes
+changes:
+	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+	@echo
+	@echo "The overview file is in $(BUILDDIR)/changes."
+
+.PHONY: linkcheck
+linkcheck:
+	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	@echo
+	@echo "Link check complete; look for any errors in the above output " \
+	      "or in $(BUILDDIR)/linkcheck/output.txt."
+
+.PHONY: doctest
+doctest:
+	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+	@echo "Testing of doctests in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/doctest/output.txt."
+
+.PHONY: coverage
+coverage:
+	$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
+	@echo "Testing of coverage in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/coverage/python.txt."
+
+.PHONY: xml
+xml:
+	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
+	@echo
+	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
+
+.PHONY: pseudoxml
+pseudoxml:
+	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
+	@echo
+	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

docs/README.md

@@ -0,0 +1,61 @@
+# Documentation Process for NeMo
+
+## Building the Documentation
+
+1. Install the documentation dependencies into the locked `uv` environment:
+
+   ```console
+   $ uv sync --locked --group docs
+   ```
+
+1. Build the documentation:
+
+   ```console
+   $ uv run make -C docs html
+   ```
+
+## Checking for Broken Links
+
+1. Build the documentation, as described in the preceding section, but use the following command:
+
+   ```shell
+   uv run make -C docs clean linkcheck
+   ```
+
+1. Run the link-checking script:
+
+   ```shell
+   ./docs/check_for_broken_links.sh
+   ```
+
+If there are no broken links, then the script exits with `0`.
+
+If the script produces any output, cut and paste the `uri` value into your browser to confirm
+that the link is broken.
+
+```json
+{
+  "filename": "nlp/text_normalization/nn_text_normalization.rst",
+  "lineno": 247,
+  "status": "broken",
+  "code": 0,
+  "uri": "https://research.fb.com/wp-content/uploads/2019/03/Neural-Models-of-Text-Normalization-for-Speech-Applications.pdf",
+  "info": "400 Client Error: Bad Request for url: https://research.facebook.com/wp-content/uploads/2019/03/Neural-Models-of-Text-Normalization-for-Speech-Applications.pdf"
+}
+```
+
+If the link is OK, and this is the case with many URLs that reference GitHub repository file headings,
+then cut and paste the JSON output and add it to `docs/false_positives.json`.
+Run the script again to confirm that the URL is no longer reported as a broken link.
+
+There may be false positives due to Sphinx not being able to detect links from built html files.
+Instead of adding those to the `docs/false_positives.json` file, it would be best to rewrite the
+reference using a [:ref:](https://www.sphinx-doc.org/en/master/usage/referencing.html#role-ref).
+
+For example, instead of writing `Modules <../api.html#modules>` to link to the modules section of
+a `api.rst` file, write it as ``:ref:`Modules <asr-api-modules>` ``. And in the `api.rst` file, add
+this label before the section being linked to:
+
+```
+.. _asr-api-modules:
+```

docs/check_for_broken_links.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+DOCS_DIR=$(dirname "${BASH_SOURCE[0]}")
+FALSE_POSITIVES_JSON="${DOCS_DIR}/false_positives.json"
+NEEDS_REVIEW_JSON="${DOCS_DIR}/links_needing_review.json"
+LINKCHECK_JSON="${DOCS_DIR}/build/linkcheck/output.json"
+
+function check_environment {
+  local err=0
+  if ! [ -x "$(command -v jq)" ]; then
+    >&2 echo "jq is required but is not found."
+    ((err++))
+  fi
+  if [ ! -f "${FALSE_POSITIVES_JSON}" ]; then
+    >&2 echo "A JSON file with false positives is required: ${FALSE_POSITIVES_JSON}"
+    ((err++))
+  fi
+  if [ ! -f "${LINKCHECK_JSON}" ]; then
+    >&2 echo "Did not find linkcheck output JSON file: ${LINKCHECK_JSON}."
+    >&2 echo "Run Sphinx with the linkcheck arg: make -C docs clean linkcheck"
+    ((err++))
+  fi
+  if [ "${err}" -gt 0 ]; then
+    exit 2
+  fi
+}
+
+function check_links {
+  local err=0
+  broken=$(jq -s 'map(select(.status=="broken"))' "$LINKCHECK_JSON")
+  count=$(echo "${broken}" | jq 'length')
+  for i in $(seq 0 $(($count - 1)))
+  do
+    entry=$(echo "${broken}" | jq ".[${i}]")
+    link=$(echo "${entry}" | jq -r '.uri')
+    [ -n "${DEBUG}" ] && {
+     echo >&2 "Checking for false positive: ${link}"
+    }
+    local false_positive_resp; false_positive_resp=$(jq --arg check "${link}" -s 'any(.uri == $check)' < "${FALSE_POSITIVES_JSON}")
+    local needs_review_resp; needs_review_resp=$(jq --arg check "${link}" -s 'any(.uri == $check)' < "${NEEDS_REVIEW_JSON}")
+    # "false" indicates that the URL did not match any of the URIs in the false positive file.
+    if [[ "false" = "${false_positive_resp}" && "false" = "${needs_review_resp}" ]]; then
+      ((err++))
+      echo "${entry}"
+    fi
+  done
+  exit "${err}"
+}
+
+check_environment
+check_links

docs/combined.json

@@ -0,0 +1 @@
+[]

docs/source/_static/css/custom.css

@@ -0,0 +1,381 @@
+@import url("theme.css");
+
+body {
+	font-size: 100%;
+	font-family: 'NVIDIA Sans', sans-serif;
+}
+
+
+/* Width of template */
+
+.wy-nav-content {
+	max-width: 1200px !important;
+}
+
+
+
+/* Standard Text Formatting */
+
+h1 {
+	color: #76b900;
+	text-align: center;
+	/* background-color: #ffffff; */
+}
+
+h2 {
+	color: #ffffff;
+	/* background-color: #ffffff; */
+	/* #76b900 */
+	Padding: 5px;
+}
+
+h3 {
+	padding-top: 0px;
+	border-top: solid 3px #000000;
+	/* #76b900 */
+	border-bottom: solid 3px #000000;
+	/* #76b900 */
+}
+
+p {
+	margin-bottom: 24px;
+}
+
+/* Link Colors */
+/*
+a {
+    color: #76b900;
+}
+/*
+
+/*
+a:visited {
+	color: #218219;
+}
+*/
+
+.container-xl {
+	margin-right: unset;
+	margin-left: unset;
+}
+
+section {
+	overflow-x: auto;
+}
+
+/* ----------------------------------------------TABLES--------------------------------------- */
+section table {
+	overflow-x: auto;
+	display: block;
+}
+
+table {
+	font-size: small;
+}
+
+/* Table head Color */
+thead td {
+	background-color: #333333 !important;
+}
+
+.row-odd p {
+	/*padding-bottom: 0px;*/
+	/*margin-bottom: 0px;*/
+}
+
+/* even rows*/
+
+.row-even tr {
+	background-color: #e5f1e6 !important;
+}
+
+/* odd rows*/
+
+
+.wy-table-responsive table tr {
+	background-color: #ffffff !important;
+}
+
+
+
+.wy-table-responsive table td {
+	white-space: normal;
+}
+
+
+/* Removes bottom margin in tables*/
+
+.rst-content .line-block {
+	margin-bottom: 0px;
+}
+
+.wy-table-responsive {
+	overflow: visible !important;
+}
+
+/* reduces the size of text in multiline table columns. */
+
+.rst-content table.docutils td {
+	font-size: 80%;
+}
+
+.rst-content dl:not(.docutils) dt {
+
+	background-color: inherit;
+	color: #000000;
+	border-top: solid 0px #000000;
+
+}
+
+.rst-content dl:not(.docutils) dt:before {
+	color: #333333;
+}
+
+.rst-content .line-block {
+	margin-bottom: 0px;
+}
+
+.wy-side-nav-search,
+.wy-nav-top {
+	background-color: #000000;
+	padding: 0;
+}
+
+.wy-side-nav-search img {
+	padding: 0px;
+	padding: 0px 0px;
+	margin-bottom: 0;
+}
+
+.wy-side-nav-search input[type=text] {
+	border-radius: 0px;
+}
+
+
+.wy-menu-vertical p.caption {
+	color: #76b900;
+}
+
+
+.wy-side-nav-search>a img.logo,
+.wy-side-nav-search .wy-dropdown>a img.logo {
+	margin: 0px 0px 0px 0px;
+}
+
+.wy-nav-content {
+	margin: 0;
+	min-height: 100%;
+	height: 100%;
+	background: #ffffff;
+}
+
+/* NVIDIA green hover on table rows */
+table tbody tr:hover,
+table.table tbody tr:hover,
+table.docutils tbody tr:hover,
+.pst-scrollable-table-container table tbody tr:hover,
+.rst-content table tbody tr:hover {
+	background-color: rgba(118, 185, 0, 0.15) !important;
+}
+
+/* List (numbered, bulleted) padding Fix */
+
+
+.wy-plain-list-decimal li {
+	margin-top: -6px;
+	margin-bottom: -6px;
+}
+
+.rst-content .section ol.loweralpha {
+	margin-top: -6px;
+	margin-bottom: 12px;
+}
+
+.wy-plain-list-disc,
+.rst-content .toctree-wrapper ul,
+article ul {
+	margin-top: 0px !important;
+	margin-bottom: 12px;
+}
+
+/* Alert Boxes */
+/* Background color of Alert Box Title */
+
+.rst-content .section ul {
+	margin-top: -12px;
+	margin-bottom: 16px;
+}
+
+.wy-alert.wy-alert-info .wy-alert-title,
+.rst-content .note .wy-alert-title,
+.rst-content .wy-alert-info.attention .wy-alert-title,
+.rst-content .wy-alert-info.caution .wy-alert-title,
+.rst-content .wy-alert-info.danger .wy-alert-title,
+.rst-content .wy-alert-info.error .wy-alert-title,
+.rst-content .wy-alert-info.hint .wy-alert-title,
+.rst-content .wy-alert-info.important .wy-alert-title,
+.rst-content .wy-alert-info.tip .wy-alert-title,
+.rst-content .wy-alert-info.warning .wy-alert-title,
+.rst-content .seealso .wy-alert-title,
+.rst-content .wy-alert-info.admonition-todo .wy-alert-title,
+.rst-content .wy-alert-info.admonition .wy-alert-title,
+.wy-alert.wy-alert-info .rst-content .admonition-title,
+.rst-content .wy-alert.wy-alert-info .admonition-title,
+.rst-content .note .admonition-title,
+.rst-content .wy-alert-info.attention .admonition-title,
+.rst-content .wy-alert-info.caution .admonition-title,
+.rst-content .wy-alert-info.danger .admonition-title,
+.rst-content .wy-alert-info.error .admonition-title,
+.rst-content .wy-alert-info.hint .admonition-title,
+.rst-content .wy-alert-info.important .admonition-title,
+.rst-content .wy-alert-info.tip .admonition-title,
+.rst-content .wy-alert-info.warning .admonition-title,
+.rst-content .seealso .admonition-title,
+.rst-content .wy-alert-info.admonition-todo .admonition-title,
+.rst-content .wy-alert-info.admonition .admonition-title {
+	background: #76b900;
+}
+
+/* Background and Font Color of Alert Box Main Body*/
+.wy-alert.wy-alert-info,
+.rst-content .note,
+.rst-content .wy-alert-info.attention,
+.rst-content .wy-alert-info.caution,
+.rst-content .wy-alert-info.danger,
+.rst-content .wy-alert-info.error,
+.rst-content .wy-alert-info.hint,
+.rst-content .wy-alert-info.important,
+.rst-content .wy-alert-info.tip,
+.rst-content .wy-alert-info.warning,
+.rst-content .seealso,
+.rst-content .wy-alert-info.admonition-todo,
+.rst-content .wy-alert-info.admonition {
+	background: #333333;
+	color: #999999;
+}
+
+.section {
+	margin-top: 50px;
+}
+
+/* Logo */
+.navbar-brand-box {
+	background-color: #ffffff;
+}
+
+/* ---------------------------------------------- Media Queries --------------------------------------- */
+@media (min-width: 1200px) {
+	.container-xl {
+		max-width: 100%;
+	}
+}
+
+@media (min-width: none) {
+	body {
+		font-size: 18px;
+	}
+
+	#site-navigation nav ul.nav {
+		font-size: 18px;
+	}
+
+	#site-navigation nav.bd-links p {
+		font-size: 18px;
+	}
+
+	#site-navigation {
+		width: 350px;
+	}
+
+	.toc-h2 {
+		font-size: 18px;
+	}
+
+	.toc-h3 {
+		font-size: 1rem;
+	}
+
+	.toc-h4 {
+		font-size: 0.85rem;
+	}
+
+	.header-article .bd-toc {
+		font-size: 18px;
+	}
+
+	#main-content>div {
+		margin-left: 10%;
+		margin-right: 10%;
+	}
+}
+
+/* ---------------------------------------------- NVIDIA Sans --------------------------------------- */
+
+:root {
+	--md-text-font: "NVIDIA Sans";
+	/* --md-code-font: "NVIDIA Sans"; */
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/5/2/52891dda673228d54e5d57bf1e4a3880d4b22405.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/e/0/e090b7dda7a582522c7f9045c6ce949cce60134f.woff) format("woff");
+	font-weight: 300;
+	font-style: normal;
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/a/1/a107baabcbf6b241099122336bce7429bcfd377a.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/3/a/3a6060a4e3bce70e5552ba0de8af4b22c6cf9144.woff) format("woff");
+	font-weight: 300;
+	font-style: italic;
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/9/9/9920d2b172b01d92fc9c1c0e521dcf45b59c47c3.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/6/c/6c7d947928a7e4ef3e80ed409bef6c243f2148cb.woff) format("woff");
+	font-weight: 400;
+	font-style: normal;
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/e/8/e8e63fe1244372cd942d957f44a5616a1eba0644.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/0/f/0f1fb2af0283ab09d36e7097bb07d895c3228f12.woff) format("woff");
+	font-weight: 400;
+	font-style: italic;
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/7/9/79d3c513a9cd72c59f65354f39f89ca52dc17dd2.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/2/5/2581ac533f5d01f4985d8a7245b0766b4630ced8.woff) format("woff");
+	font-weight: 500;
+	font-style: normal;
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/3/9/39d9ef1ee9770dd503f19bb2ace2fdb4eff3bb50.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/7/b/7bb5d5e2e71b2e13c8098b2e67c0a0ed9258e6c7.woff) format("woff");
+	font-weight: 500;
+	font-style: italic;
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/0/5/05276a55a43eb3f74981ec1e93252727afcd9d16.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/9/c/9cfec7ed941b06564aa4d5ca14610e81542d070f.woff) format("woff");
+	font-weight: 700;
+	font-style: normal;
+}
+
+@font-face {
+	font-family: "NVIDIA Sans";
+	src: url(https://aws1.discourse-cdn.com/nvidia/original/3X/a/e/aebd14d09ba56f541e1b8735fb051e33710f9ae7.woff2) format("woff2"),
+		url(https://aws1.discourse-cdn.com/nvidia/original/3X/e/d/edbdabef43acc5c12e84a94baaa5542c9404cfeb.woff) format("woff");
+	font-weight: 700;
+	font-style: italic;
+}

docs/source/_static/js/pk_scripts.js

@@ -0,0 +1,19 @@
+document.addEventListener("DOMContentLoaded", function () {
+    var params = window.location.search.substring(1).split("&").reduce(function (params, param) {
+        if (!param) {
+            return params;
+        }
+
+        var values = param.split("=");
+        var name = values[0];
+        var value = values[1];
+        params[name] = value;
+        return params;
+    }, {});
+
+    var form = document.getElementById("feedback-form");
+    for (var name in params) {
+        var input = form.querySelector("[name=" + name + "]");
+        input.value = params[name];
+    }
+});

docs/source/apis.rst

@@ -0,0 +1,48 @@
+:orphan:
+
+=========
+NeMo APIs
+=========
+
+You can learn more about the underlying principles of the NeMo codebase in this section.
+
+The `NeMo Toolkit codebase <https://github.com/NVIDIA/NeMo>`__ is composed of a `core <https://github.com/NVIDIA/NeMo/tree/main/nemo/core>`__ section which contains the main building blocks of the framework, and various `collections <https://github.com/NVIDIA/NeMo/tree/main/nemo/collections>`__ which help you
+build specialized AI models.
+
+You can learn more about aspects of the NeMo "core" by following the links below:
+
+.. toctree::
+   :maxdepth: 1
+   :name: core
+   :titlesonly:
+
+   core/core
+   core/neural_modules
+   core/exp_manager
+   core/neural_types
+   core/adapters/intro
+
+You can learn more about aspects of the NeMo APIs by following the links below:
+
+.. toctree::
+   :maxdepth: 1
+   :name: API
+   :titlesonly:
+
+   core/api
+   common/intro
+   asr/api
+   tts/api
+   audio/api
+
+
+Alternatively, you can jump straight to the documentation for the individual collections:
+
+* :doc:`Automatic Speech Recognition (ASR) <../asr/intro>`
+
+* :doc:`Text-to-Speech (TTS) <../tts/intro>`
+
+* :doc:`Audio Processing <../audio/intro>`
+
+* :doc:`SpeechLM2 <../speechlm2/intro>`
+

docs/source/asr/all_chkpt.rst

@@ -0,0 +1,402 @@
+:orphan:
+
+All Checkpoints
+===============
+English
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_en.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+German
+^^^^^^
+.. csv-table::
+   :file: data/benchmark_de.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Spanish
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_es.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+French
+^^^^^^
+.. csv-table::
+   :file: data/benchmark_fr.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Arabic
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_ar.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+------------------------------
+
+Russian
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_ru.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Portuguese
+^^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_pt.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Belarusian
+^^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_be.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Japanese
+^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_jp.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Armenian
+^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_hy.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Georgian
+^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_ka.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Kazakh
+^^^^^^
+.. csv-table::
+   :file: data/benchmark_kz.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Persian
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_fa.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Uzbek
+^^^^^
+.. csv-table::
+   :file: data/benchmark_uz.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Ukrainian
+^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_ua.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Polish
+^^^^^^
+.. csv-table::
+   :file: data/benchmark_pl.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Italian
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_it.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+
+Croatian
+^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_hr.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Esperanto
+^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_eo.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Kabyle
+^^^^^^
+.. csv-table::
+   :file: data/benchmark_kab.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Dutch
+^^^^^
+.. csv-table::
+   :file: data/benchmark_nl.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Catalan
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_ca.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Hindi
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_hi.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Marathi
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_mr.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Mandarin
+^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_zh.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Czech
+^^^^^
+.. csv-table::
+   :file: data/benchmark_cs.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Bulgarian
+^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_bg.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Danish
+^^^^^^
+.. csv-table::
+   :file: data/benchmark_da.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Estonian
+^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_et.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Finnish
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_fi.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Greek
+^^^^^
+.. csv-table::
+   :file: data/benchmark_el.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Hungarian
+^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_hu.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Latvian
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_lv.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Lithuanian
+^^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_lt.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Maltese
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_mt.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Romanian
+^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_ro.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Slovak
+^^^^^^
+.. csv-table::
+   :file: data/benchmark_sk.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Slovenian
+^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_sl.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Swedish
+^^^^^^^
+.. csv-table::
+   :file: data/benchmark_sv.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1
+
+-----------------------------
+
+Kinyarwanda
+^^^^^^^^^^^
+.. csv-table::
+   :file: data/benchmark_rw.csv
+   :align: left
+   :widths: 50,50
+   :header-rows: 1

docs/source/asr/api.rst

@@ -0,0 +1,367 @@
+NeMo ASR API
+============
+
+
+Model Classes
+-------------
+
+.. autoclass:: nemo.collections.asr.models.EncDecCTCModel
+    :show-inheritance:
+    :members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecCTCModelBPE
+    :show-inheritance:
+    :members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecRNNTModel
+    :show-inheritance:
+    :members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecRNNTBPEModel
+    :show-inheritance:
+    :members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecRNNTBPEModelWithPrompt
+    :show-inheritance:
+    :members: transcribe, set_inference_prompt, initialize_prompt_feature, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecHybridRNNTCTCBPEModelWithPrompt
+    :show-inheritance:
+    :members: transcribe, set_inference_prompt, initialize_prompt_feature, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecMultiTalkerRNNTBPEModel
+    :show-inheritance:
+    :members: transcribe, change_vocabulary, setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecClassificationModel
+    :show-inheritance:
+    :members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. autoclass:: nemo.collections.asr.models.EncDecSpeakerLabelModel
+    :show-inheritance:
+    :members: setup_training_data, setup_optimization, setup_validation_data, setup_test_data, register_artifact
+
+
+.. _asr-api-modules:
+
+Modules
+-------
+
+.. autoclass:: nemo.collections.asr.modules.ConvASREncoder
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.modules.ConvASRDecoder
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.modules.ConvASRDecoderClassification
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.modules.SpeakerDecoder
+    :show-inheritance:
+    :members:
+
+.. _conformer-encoder-api:
+
+.. autoclass:: nemo.collections.asr.modules.ConformerEncoder
+    :show-inheritance:
+    :members:
+
+
+.. _rnn-encoder-api:
+
+.. autoclass:: nemo.collections.asr.modules.RNNEncoder
+    :show-inheritance:
+    :members:
+
+.. _rnnt-decoder-api:
+
+.. autoclass:: nemo.collections.asr.modules.RNNTDecoder
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.modules.StatelessTransducerDecoder
+    :show-inheritance:
+    :members:
+
+.. _rnnt-joint-api:
+
+.. autoclass:: nemo.collections.asr.modules.RNNTJoint
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.modules.SampledRNNTJoint
+    :show-inheritance:
+    :members:
+
+
+Mixins
+------
+
+.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRBPEMixin
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.mixins.mixins.ASRModuleMixin
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.mixins.transcription.TranscriptionMixin
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.mixins.transcription.TranscribeConfig
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.mixins.interctc_mixin.InterCTCMixin
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.mixins.multitalker_asr_mixins.SpeakerKernelMixin
+    :show-inheritance:
+    :members: 
+
+.. _asr-api-datasets:
+
+Datasets
+--------
+
+Character Encoding Datasets
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToCharDataset
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset
+    :show-inheritance:
+    :members:
+
+
+Text-to-Text Datasets
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.data.text_to_text.TextToTextDataset
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.data.text_to_text.TextToTextIterableDataset
+    :show-inheritance:
+    :members:
+
+
+Speaker-Tagged Datasets for Multitalker ASR models
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.data.audio_to_text_lhotse_speaker.LhotseSpeechToTextSpkBpeDataset
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.data.audio_to_diar_label_lhotse.LhotseAudioToSpeechE2ESpkDiarDataset
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.data.data_simulation.MultiSpeakerSimulator
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.data.data_simulation.RIRMultiSpeakerSimulator
+    :show-inheritance:
+    :members:
+
+Subword Encoding Datasets
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.data.audio_to_text.AudioToBPEDataset
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset
+    :show-inheritance:
+    :members:
+
+.. _asr-audio-preprocessors:
+
+Audio Preprocessors
+-------------------
+
+.. autoclass:: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.modules.AudioToMFCCPreprocessor
+    :show-inheritance:
+    :members:
+
+.. _asr-api-audio-augmentors:
+
+Audio Augmentors
+----------------
+
+.. autoclass:: nemo.collections.asr.modules.SpectrogramAugmentation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.modules.CropOrPadSpectrogramAugmentation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.SpeedPerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TimeStretchPerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.GainPerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ImpulsePerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.ShiftPerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.NoisePerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.WhiteNoisePerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.RirAndNoisePerturbation
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.preprocessing.perturb.TranscodePerturbation
+    :show-inheritance:
+    :members:
+
+Miscellaneous Classes
+---------------------
+
+CTC Decoding
+~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.parts.submodules.ctc_decoding.CTCDecoding
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.ctc_decoding.CTCBPEDecoding
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.ctc_greedy_decoding.GreedyCTCInfer
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.ctc_beam_decoding.BeamCTCInfer
+    :show-inheritance:
+    :members:
+
+RNNT Decoding
+~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_decoding.RNNTDecoding
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_decoding.RNNTBPEDecoding
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyRNNTInfer
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedRNNTInfer
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_beam_decoding.BeamRNNTInfer
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_beam_decoding.BeamBatchedRNNTInfer
+    :show-inheritance:
+    :members:
+
+TDT Decoding
+~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyTDTInfer
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.rnnt_greedy_decoding.GreedyBatchedTDTInfer
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.tdt_beam_decoding.BeamTDTInfer
+    :show-inheritance:
+    :members:
+
+.. autoclass:: nemo.collections.asr.parts.submodules.tdt_beam_decoding.BeamBatchedTDTInfer
+    :show-inheritance:
+    :members:
+
+Hypotheses
+~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.Hypothesis
+    :show-inheritance:
+    :no-members:
+
+.. autoclass:: nemo.collections.asr.parts.utils.rnnt_utils.NBestHypotheses
+    :show-inheritance:
+    :no-members:
+
+Adapter Networks
+~~~~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.MultiHeadAttentionAdapter
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+
+.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.RelPositionMultiHeadAttentionAdapter
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+
+.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.PositionalEncodingAdapter
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+
+.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.RelPositionalEncodingAdapter
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+
+
+Adapter Strategies
+~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: nemo.collections.asr.parts.submodules.adapters.multi_head_attention_adapter_module.MHAResidualAddAdapterStrategy
+    :show-inheritance:
+    :members:
+    :member-order: bysource
+    :undoc-members: adapter_module_names
+

docs/source/asr/asr_checkpoints.rst

@@ -0,0 +1,396 @@
+.. _asr-checkpoints-list:
+
+=======================
+ASR Model Checkpoints
+=======================
+
+This page lists all supported ASR model checkpoints released by NVIDIA NeMo.
+Benchmark scores for each model can be found on its `HuggingFace model card <https://huggingface.co/nvidia>`__.
+
+For community fine-tunes built on these checkpoints, see :doc:`Featured Community Checkpoints <./featured_community_checkpoints>`.
+
+Glossary
+--------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Term
+     - Definition
+   * - **ASR**
+     - Automatic Speech Recognition — transcribing speech to text
+   * - **AST**
+     - Automatic Speech Translation — translating speech to text from one language to another
+   * - **AED**
+     - Attention Encoder-Decoder — autoregressive decoder using cross-attention (Canary family)
+   * - **CTC**
+     - Connectionist Temporal Classification — non-autoregressive decoder
+   * - **RNN-T**
+     - Recurrent Neural Network Transducer — autoregressive streaming-friendly decoder
+   * - **TDT**
+     - Token-and-Duration Transducer — extends RNN-T with duration prediction for faster inference
+   * - **Hybrid**
+     - Joint RNN-T + CTC model — both decoders trained together, either usable at inference
+   * - **PnC**
+     - Punctuation and Capitalization in the output
+   * - **SALM**
+     - Speech Augmented Language Model — combines a speech encoder with a large language model
+   * - **Streaming**
+     - Real-time / cache-aware inference capability
+   * - **EU4**
+     - English, German, Spanish, French
+   * - **EU25**
+     - English, German, Spanish, French, Italian, Polish, Portuguese, Dutch, Russian, Ukrainian, Belarusian, Croatian, Czech, Bulgarian, Danish, Estonian, Finnish, Greek, Hungarian, Latvian, Lithuanian, Maltese, Romanian, Slovak, Slovenian, Swedish
+
+Canary Models (AED)
+-------------------
+
+Multi-task encoder-decoder models supporting ASR, AST, PnC, and timestamps across multiple languages.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Model
+     - Decoder
+     - Capabilities
+     - Language
+     - Size
+   * - `canary-1b-v2 <https://huggingface.co/nvidia/canary-1b-v2>`__
+     - AED
+     - ASR, AST, PnC, timestamps
+     - EU25
+     - 1B
+   * - `canary-qwen-2.5b <https://huggingface.co/nvidia/canary-qwen-2.5b>`__
+     - SALM
+     - ASR, AST, PnC, timestamps
+     - EU25
+     - 2.5B
+   * - `canary-1b-flash <https://huggingface.co/nvidia/canary-1b-flash>`__
+     - AED
+     - ASR, AST, PnC, timestamps, fast
+     - EU4
+     - 1B
+   * - `canary-180m-flash <https://huggingface.co/nvidia/canary-180m-flash>`__
+     - AED
+     - ASR, AST, PnC, timestamps, fast
+     - EU4
+     - 180M
+   * - `canary-1b <https://huggingface.co/nvidia/canary-1b>`__
+     - AED
+     - ASR, AST, PnC
+     - EU4
+     - 1B
+
+
+Parakeet Models
+-----------------
+
+High-accuracy ASR models built on the FastConformer encoder architecture.
+Parakeet, Nemotron Speech, and the ``stt_*_fastconformer_*`` models below all share the same underlying FastConformer encoder;
+the different names reflect release branding, not architectural differences.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Model
+     - Decoder
+     - Capabilities
+     - Language
+     - Size
+   * - `parakeet-tdt-0.6b-v3 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3>`__
+     - TDT
+     - ASR, PnC, timestamps
+     - English
+     - 0.6B
+   * - `parakeet-tdt-0.6b-v2 <https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2>`__
+     - TDT
+     - ASR, PnC, timestamps
+     - English
+     - 0.6B
+   * - `parakeet-tdt-1.1b <https://huggingface.co/nvidia/parakeet-tdt-1.1b>`__
+     - TDT
+     - ASR, timestamps
+     - English
+     - 1.1B
+   * - `parakeet-tdt_ctc-1.1b <https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b>`__
+     - Hybrid TDT+CTC
+     - ASR, timestamps
+     - English
+     - 1.1B
+   * - `parakeet-tdt_ctc-0.6b-ja <https://huggingface.co/nvidia/parakeet-tdt_ctc-0.6b-ja>`__
+     - Hybrid TDT+CTC
+     - ASR, timestamps
+     - Japanese
+     - 0.6B
+   * - `parakeet-tdt_ctc-110m <https://huggingface.co/nvidia/parakeet-tdt_ctc-110m>`__
+     - Hybrid TDT+CTC
+     - ASR, timestamps
+     - English
+     - 110M
+   * - `parakeet-rnnt-1.1b <https://huggingface.co/nvidia/parakeet-rnnt-1.1b>`__
+     - RNN-T
+     - ASR, timestamps
+     - English
+     - 1.1B
+   * - `parakeet-rnnt-0.6b <https://huggingface.co/nvidia/parakeet-rnnt-0.6b>`__
+     - RNN-T
+     - ASR, timestamps
+     - English
+     - 0.6B
+   * - `parakeet-ctc-1.1b <https://huggingface.co/nvidia/parakeet-ctc-1.1b>`__
+     - CTC
+     - ASR
+     - English
+     - 1.1B
+   * - `parakeet-ctc-0.6b <https://huggingface.co/nvidia/parakeet-ctc-0.6b>`__
+     - CTC
+     - ASR
+     - English
+     - 0.6B
+   * - `parakeet-ctc-0.6b-Vietnamese <https://huggingface.co/nvidia/parakeet-ctc-0.6b-Vietnamese>`__
+     - CTC
+     - ASR
+     - Vietnamese
+     - 0.6B
+   * - `parakeet-rnnt-110m-da-dk <https://huggingface.co/nvidia/parakeet-rnnt-110m-da-dk>`__
+     - RNN-T
+     - ASR
+     - Danish
+     - 110M
+
+
+Streaming Models
+-----------------
+
+Cache-aware models for real-time / low-latency inference.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Model
+     - Decoder
+     - Capabilities
+     - Language
+     - Size
+   * - `nemotron-3.5-asr-streaming-0.6b <https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b>`__
+     - Hybrid
+     - ASR, streaming
+     - 40 languages
+     - 0.6B
+   * - `multitalker-parakeet-streaming-0.6b-v1 <https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1>`__
+     - RNN-T
+     - ASR, multitalker, streaming
+     - English
+     - 0.6B
+   * - `parakeet_realtime_eou_120m-v1 <https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1>`__
+     - RNN-T
+     - ASR, end-of-utterance, streaming
+     - English
+     - 120M
+   * - `stt_en_fastconformer_hybrid_large_streaming_multi <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_streaming_multi>`__
+     - Hybrid
+     - ASR, streaming, multiple look-aheads
+     - English
+     - Large
+   * - `stt_en_fastconformer_hybrid_medium_streaming_80ms_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms_pc>`__
+     - Hybrid
+     - ASR, PnC, streaming
+     - English
+     - Medium
+   * - `stt_en_fastconformer_hybrid_medium_streaming_80ms <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms>`__
+     - Hybrid
+     - ASR, streaming
+     - English
+     - Medium
+   * - `stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc>`__
+     - Hybrid
+     - ASR, PnC, streaming
+     - Georgian
+     - Large
+   * - `stt_en_fastconformer_hybrid_large_streaming_1040ms <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_en_fastconformer_hybrid_large_streaming_1040ms>`__
+     - Hybrid
+     - ASR, streaming
+     - English
+     - Large
+
+
+FastConformer English Models (Non-Streaming)
+----------------------------------------------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Model
+     - Decoder
+     - Capabilities
+     - Language
+     - Size
+   * - `stt_en_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_en_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - English
+     - Large
+   * - `stt_en_fastconformer_ctc_large <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_large>`__
+     - CTC
+     - ASR
+     - English
+     - Large
+   * - `stt_en_fastconformer_ctc_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xlarge>`__
+     - CTC
+     - ASR
+     - English
+     - XLarge
+   * - `stt_en_fastconformer_ctc_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_ctc_xxlarge>`__
+     - CTC
+     - ASR
+     - English
+     - XXLarge
+   * - `stt_en_fastconformer_transducer_large <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_large>`__
+     - RNN-T
+     - ASR
+     - English
+     - Large
+   * - `stt_en_fastconformer_transducer_xlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xlarge>`__
+     - RNN-T
+     - ASR
+     - English
+     - XLarge
+   * - `stt_en_fastconformer_transducer_xxlarge <https://huggingface.co/nvidia/stt_en_fastconformer_transducer_xxlarge>`__
+     - RNN-T
+     - ASR
+     - English
+     - XXLarge
+   * - `stt_en_fastconformer_tdt_large <https://huggingface.co/nvidia/stt_en_fastconformer_tdt_large>`__
+     - TDT
+     - ASR
+     - English
+     - Large
+
+
+FastConformer Multilingual Models
+----------------------------------
+
+Single-language FastConformer Hybrid models. Models with ``_pc`` suffix support punctuation and capitalization.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Model
+     - Decoder
+     - Capabilities
+     - Language
+     - Size
+   * - `stt_multilingual_fastconformer_hybrid_large_pc_blend_eu <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/stt_multilingual_fastconformer_hybrid_large_pc_blend_eu>`__
+     - Hybrid
+     - ASR, PnC
+     - Multilingual EU
+     - Large
+   * - `stt_de_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_de_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - German
+     - Large
+   * - `stt_es_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Spanish
+     - Large
+   * - `stt_es_fastconformer_hybrid_large_pc_nc <https://huggingface.co/nvidia/stt_es_fastconformer_hybrid_large_pc_nc>`__
+     - Hybrid
+     - ASR, Punctuation only
+     - Spanish
+     - Large
+   * - `stt_fr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_fr_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - French
+     - Large
+   * - `stt_it_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_it_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Italian
+     - Large
+   * - `stt_ru_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ru_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Russian
+     - Large
+   * - `stt_ua_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ua_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Ukrainian
+     - Large
+   * - `stt_pl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pl_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Polish
+     - Large
+   * - `stt_hr_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hr_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Croatian
+     - Large
+   * - `stt_be_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_be_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Belarusian
+     - Large
+   * - `stt_nl_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_nl_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Dutch
+     - Large
+   * - `stt_pt_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_pt_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Portuguese
+     - Large
+   * - `stt_fa_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large>`__
+     - Hybrid
+     - ASR
+     - Farsi
+     - Large
+   * - `stt_ka_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Georgian
+     - Large
+   * - `stt_hy_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_hy_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Armenian
+     - Large
+   * - `stt_ar_fastconformer_hybrid_large_pc_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0>`__
+     - Hybrid
+     - ASR, PnC
+     - Arabic
+     - Large
+   * - `stt_ar_fastconformer_hybrid_large_pcd_v1.0 <https://huggingface.co/nvidia/stt_ar_fastconformer_hybrid_large_pcd_v1.0>`__
+     - Hybrid
+     - ASR, PnC (diacritized)
+     - Arabic
+     - Large
+   * - `stt_uz_fastconformer_hybrid_large_pc <https://huggingface.co/nvidia/stt_uz_fastconformer_hybrid_large_pc>`__
+     - Hybrid
+     - ASR, PnC
+     - Uzbek
+     - Large
+   * - `stt_kk_ru_fastconformer_hybrid_large <https://huggingface.co/nvidia/stt_kk_ru_fastconformer_hybrid_large>`__
+     - Hybrid
+     - ASR
+     - Kazakh + Russian
+     - Large
+
+
+Loading Models
+--------------
+
+All models (except SALM — see :doc:`SpeechLM2 </speechlm2/intro>`) can be loaded via the ``from_pretrained()`` API:
+
+.. code-block:: python
+
+    import nemo.collections.asr as nemo_asr
+
+    model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")
+
+

docs/source/asr/asr_customization/legacy_language_modeling_and_customization.rst

@@ -0,0 +1,297 @@
+.. _ngram_modeling:
+
+****************************
+N-gram Language Model Fusion
+****************************
+
+In this approach, an N-gram LM is trained on text data, then it is used in fusion with beam search decoding to find the
+best candidates. The beam search decoders in NeMo support language models trained with KenLM library (
+`https://github.com/kpu/kenlm <https://github.com/kpu/kenlm>`__).
+The beam search decoders and KenLM library are not installed by default in NeMo. 
+You need to install them to be able to use beam search decoding and N-gram LM.
+Please refer to `scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh>`__
+on how to install them. Alternatively, you can build Docker image
+`scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__ with all the necessary dependencies.
+
+Please, refer to :ref:`train-ngram-lm` for more details on how to train an N-gram LM using KenLM library.
+
+NeMo supports both character-based and BPE-based models for N-gram LMs. An N-gram LM can be used with beam search
+decoders on top of the ASR models to produce more accurate candidates. The beam search decoder would incorporate
+the scores produced by the N-gram LM into its score calculations as the following:
+
+.. code-block::
+
+    final_score = acoustic_score + beam_alpha*lm_score + beam_beta*seq_length
+
+where acoustic_score is the score predicted by the acoustic encoder and lm_score is the one estimated by the LM.
+The parameter 'beam_alpha' determines the weight given to the N-gram language model, while 'beam_beta' is a penalty term that accounts for sequence length in the scores. A larger 'beam_alpha' places more emphasis on the language model and less on the acoustic model. Negative values for 'beam_beta' penalize longer sequences, encouraging the decoder to prefer shorter predictions. Conversely, positive values for 'beam_beta' favor longer candidates.
+
+Evaluate by Beam Search Decoding and N-gram LM
+==============================================
+
+NeMo's beam search decoders are capable of using the KenLM's N-gram models to find the best candidates.
+The script to evaluate an ASR model with beam search decoding and N-gram models can be found at
+`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__.
+
+This script has a large number of possible argument overrides; therefore, it is recommended that you use ``python eval_beamsearch_ngram_ctc.py --help`` to see the full list of arguments.
+
+You can evaluate an ASR model using the following:
+
+.. code-block::
+
+    python eval_beamsearch_ngram_ctc.py nemo_model_file=<path to the .nemo file of the model> \
+           input_manifest=<path to the evaluation JSON manifest file \
+           kenlm_model_file=<path to the binary KenLM model> \
+           beam_width=[<list of the beam widths, separated with commas>] \
+           beam_alpha=[<list of the beam alphas, separated with commas>] \
+           beam_beta=[<list of the beam betas, separated with commas>] \
+           preds_output_folder=<optional folder to store the predictions> \
+           probs_cache_file=null \
+           decoding_mode=beamsearch_ngram \
+           decoding_strategy="<Beam library such as beam, pyctcdecode or flashlight>"
+
+It can evaluate a model in the following three modes by setting the argument ``--decoding_mode``:
+
+*  greedy: Just greedy decoding is done and no beam search decoding is performed.
+*  beamsearch: The beam search decoding is done, but without using the N-gram language model. Final results are equivalent to setting the weight of LM (beam_beta) to zero.
+*  beamsearch_ngram: The beam search decoding is done with N-gram LM.
+
+In ``beamsearch`` mode, the evaluation is performed using beam search decoding without any language model. The performance is reported in terms of Word Error Rate (WER) and Character Error Rate (CER). Moreover, when the best candidate is selected among the candidates, it is also reported as the best WER/CER. This can serve as an indicator of the quality of the predicted candidates.
+
+
+The script initially loads the ASR model and predicts the outputs of the model's encoder as log probabilities. This part is computed in batches on a device specified by --device, which can be either a CPU (`--device=cpu`) or a single GPU (`--device=cuda:0`).
+The batch size for this part is specified by ``--acoustic_batch_size``. Using the largest feasible batch size can speed up the calculation of log probabilities. Additionally, you can use `--use_amp` to accelerate the calculation and allow for larger --acoustic_batch_size values.
+Currently, multi-GPU support is not available for calculating log probabilities. However, using ``--probs_cache_file`` can help. This option stores the log probabilities produced by the model's encoder in a pickle file, allowing you to skip the first step in future runs.
+
+The following is the list of the important arguments for the evaluation script:
+
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| **Argument**                         | **Type** | **Default**      | **Description**                                                         |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| nemo_model_file                      | str      | Required         | The path of the `.nemo` file of the ASR model to extract the tokenizer. |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| input_manifest                       | str      | Required         | Path to the training file, it can be a text file or JSON manifest.      |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| kenlm_model_file                     | str      | Required         | The path to store the KenLM binary model file.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| preds_output_folder                  | str      | None             | The path to an optional folder to store the predictions.                |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| probs_cache_file                     | str      | None             | The cache file for storing the outputs of the model.                    |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| acoustic_batch_size                  | int      | 16               | The batch size to calculate log probabilities.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| use_amp                              | bool     | False            | Whether to use AMP if available to calculate log probabilities.         |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| device                               | str      | cuda             | The device to load the model onto to calculate log probabilities.       |
+|                                      |          |                  | It can `cpu`, `cuda`, `cuda:0`, `cuda:1`, ...                           |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| decoding_mode                        | str      | beamsearch_ngram | The decoding scheme to be used for evaluation.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_width                           | float    | Required         | List of the width or list of the widths of the beam search decoding.    |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_alpha                           | float    | Required         | List of the alpha parameter for the beam search decoding.               |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_beta                            | float    | Required         | List of the beta parameter for the beam search decoding.                |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| beam_batch_size                      | int      | 128              | The batch size to be used for beam search decoding.                     |
+|                                      |          |                  | Larger batch size can be a little faster, but uses larger memory.       |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| decoding_strategy                    | str      | beam             | String argument for type of decoding strategy for the model.            |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| decoding                             | Dict     | BeamCTC          | Subdict of beam search configs. Values found via                        |
+|                                      | Config   | InferConfig      | python eval_beamsearch_ngram_ctc.py --help                              |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.do_lowercase         | bool     | ``False``        | Whether to make the training text all lower case.                       |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.punctuation_marks    | str      | ``""``           | String with punctuation marks to process. Example: ".\,?"               |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.rm_punctuation       |  bool    | ``False``        | Whether to remove punctuation marks from text.                          |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+| text_processing.separate_punctuation | bool     | ``True``         | Whether to separate punctuation with the previous word by space.        |
++--------------------------------------+----------+------------------+-------------------------------------------------------------------------+
+
+The width of the beam search (``--beam_width``) specifies the number of top candidates or predictions the beam search decoder will consider. Larger beam widths result in more accurate but slower predictions.
+
+.. note::
+
+    The ``eval_beamsearch_ngram_ctc.py`` script contains the entire subconfig used for CTC Beam Decoding.
+    Therefore it is possible to forward arguments for various beam search libraries such as ``flashlight``
+    and ``pyctcdecode`` via the ``decoding`` subconfig.
+
+To learn more about evaluating the ASR models with N-gram LM, refer to the tutorial here: Offline ASR Inference with Beam Search and External Language Model Rescoring
+`Offline ASR Inference with Beam Search and External Language Model Rescoring <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/Offline_ASR.ipynb>`_
+
+Beam Search Engines
+-------------------
+
+NeMo ASR CTC supports multiple beam search engines for decoding. The default engine is beam, which is the OpenSeq2Seq decoding library.
+
+OpenSeq2Seq (``beam``)
+~~~~~~~~~~~~~~~~~~~~~~
+
+CPU-based beam search engine that is quite efficient and supports char and subword models. It requires a character/subword
+KenLM model to be provided.
+
+The config for this decoding library is described above.
+
+Flashlight (``flashlight``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Flashlight is a C++ library for ASR decoding provided at `https://github.com/flashlight/flashlight <https://github.com/flashlight/flashlight>`_. It is a CPU- and CUDA-based beam search engine that is quite efficient and supports char and subword models. It requires an ARPA KenLM file.
+
+It supports several advanced features, such as lexicon-based decoding, lexicon-free decoding, beam pruning threshold, and more.
+
+.. code-block:: python
+
+    @dataclass
+    class FlashlightConfig:
+        lexicon_path: Optional[str] = None
+        boost_path: Optional[str] = None
+        beam_size_token: int = 16
+        beam_threshold: float = 20.0
+        unk_weight: float = -math.inf
+        sil_weight: float = 0.0
+
+.. code-block::
+
+    # Lexicon-based decoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="flashlight" \
+           decoding.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon' \
+           decoding.beam.flashlight_cfg.beam_size_token = 32 \
+           decoding.beam.flashlight_cfg.beam_threshold = 25.0
+
+    # Lexicon-free decoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="flashlight" \
+           decoding.beam.flashlight_cfg.beam_size_token = 32 \
+           decoding.beam.flashlight_cfg.beam_threshold = 25.0
+
+PyCTCDecode (``pyctcdecode``)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+PyCTCDecode is a Python library for ASR decoding provided at `https://github.com/kensho-technologies/pyctcdecode <https://github.com/kensho-technologies/pyctcdecode>`_. It is a CPU-based beam search engine that is somewhat efficient for a pure Python library, and supports char and subword models. It requires a character/subword KenLM ARPA / BINARY model to be provided.
+
+
+It has advanced features, such as word boosting, which can be useful for transcript customization.
+
+.. code-block:: python
+
+   @dataclass
+    class PyCTCDecodeConfig:
+        beam_prune_logp: float = -10.0
+        token_min_logp: float = -5.0
+        prune_history: bool = False
+        hotwords: Optional[List[str]] = None
+        hotword_weight: float = 10.0
+
+.. code-block::
+
+    # PyCTCDecoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="pyctcdecode" \
+           decoding.beam.pyctcdecode_cfg.beam_prune_logp = -10. \
+           decoding.beam.pyctcdecode_cfg.token_min_logp = -5. \
+           decoding.beam.pyctcdecode_cfg.hotwords=[<List of str words>] \
+           decoding.beam.pyctcdecode_cfg.hotword_weight=10.0
+
+
+Hyperparameter Grid Search
+--------------------------
+
+Beam search decoding with N-gram LM has three main hyperparameters: `beam_width`, `beam_alpha`, and `beam_beta`.
+The accuracy of the model is dependent on the values of these parameters, specifically, beam_alpha and beam_beta. To perform grid search, you can specify a single value or a list of values for each of these parameters. In this case, it would perform the beam search decoding on all combinations of the three hyperparameters.
+For example, the following set of parameters would result in 212=4 beam search decodings:
+
+.. code-block::
+
+    python eval_beamsearch_ngram_ctc.py ... \
+                        beam_width=[64,128] \
+                        beam_alpha=[1.0] \
+                        beam_beta=[1.0,0.5]
+
+
+Beam Search ngram Decoding for Transducer Models (RNNT and HAT)
+===============================================================
+
+You can also find a similar script to evaluate an RNNT/HAT model with beam search decoding and N-gram models at:
+`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_transducer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_transducer.py>`_
+
+.. code-block::
+
+    python eval_beamsearch_ngram_transducer.py nemo_model_file=<path to the .nemo file of the model> \
+            input_manifest=<path to the evaluation JSON manifest file \
+            kenlm_model_file=<path to the binary KenLM model> \
+            beam_width=[<list of the beam widths, separated with commas>] \
+            beam_alpha=[<list of the beam alphas, separated with commas>] \
+            preds_output_folder=<optional folder to store the predictions> \
+            probs_cache_file=null \
+            decoding_strategy=<greedy_batch or maes decoding>
+            maes_prefix_alpha=[<list of the maes prefix alphas, separated with commas>] \
+            maes_expansion_gamma=[<list of the maes expansion gammas, separated with commas>] \
+            hat_subtract_ilm=<in case of HAT model: subtract internal LM or not (True/False)> \
+            hat_ilm_weight=[<in case of HAT model: list of the HAT internal LM weights, separated with commas>] \
+
+
+.. _wfst-ctc-decoding:
+
+WFST CTC decoding
+=================
+Weighted Finite-State Transducers (WFST) are finite-state machines with input and output symbols on each transition and some weight element of a semiring. WFSTs can act as N-gram LMs in a special type of LM-forced beam search, called WFST decoding.
+
+.. note::
+
+    More precisely, WFST decoding is more of a greedy N-depth search with LM.
+    Thus, it is asymptotically worse than conventional beam search decoding algorithms, but faster.
+
+**WARNING**  
+At the moment, NeMo supports WFST decoding only for CTC models and word-based LMs.
+
+To run WFST decoding in NeMo, one needs to provide a NeMo ASR model and either an ARPA LM or a WFST LM (advanced). An ARPA LM can be built from source text with KenLM as follows: ``<kenlm_bin_path>/lmplz -o <ngram_length> --arpa <out_arpa_path> --prune <ngram_prune>``.
+
+The script to evaluate an ASR model with WFST decoding and N-gram models can be found at
+`scripts/asr_language_modeling/ngram_lm/eval_wfst_decoding_ctc.py
+<https://github.com/NVIDIA/NeMo/blob/main/scripts/asr_language_modeling/ngram_lm/eval_wfst_decoding_ctc.py>`__.
+
+This script has a large number of possible argument overrides, therefore it is advised to use ``python eval_wfst_decoding_ctc.py --help`` to see the full list of arguments.
+
+You may evaluate an ASR model as the following:
+
+.. code-block::
+
+    python eval_wfst_decoding_ctc.py nemo_model_file=<path to the .nemo file of the model> \
+           input_manifest=<path to the evaluation JSON manifest file> \
+           arpa_model_file=<path to the ARPA LM model> \
+           decoding_wfst_file=<path to the decoding WFST file> \
+           beam_width=[<list of the beam widths, separated with commas>] \
+           lm_weight=[<list of the LM weight multipliers, separated with commas>] \
+           open_vocabulary_decoding=<whether to use open vocabulary mode for WFST decoding> \
+           decoding_mode=<decoding mode, affects output. Usually "nbest"> \
+           decoding_search_type=<WFST decoding library. Usually "riva"> \
+           preds_output_folder=<optional folder to store the predictions> \
+           probs_cache_file=null
+
+.. note::
+
+    Since WFST decoding is LM-forced (the search goes over the WIDEST graph), only word sequences accepted by the WFST can appear in the decoding results.
+    To circumvent this restriction, one can pass ``open_vocabulary_decoding=true`` (experimental feature).
+
+
+Quick start example
+-------------------
+
+.. code-block::
+
+    wget -O - https://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz | \
+    gunzip -c | tr '[:upper:]' '[:lower:]' > 3-gram.pruned.1e-7.arpa && \
+    python eval_wfst_decoding_ctc.py nemo_model_file="stt_en_conformer_ctc_small_ls" \
+           input_manifest="<data_dir>/Librispeech/test_other.json" \
+           arpa_model_file="3-gram.pruned.1e-7.arpa" \
+           decoding_wfst_file="3-gram.pruned.1e-7.fst" \
+           beam_width=[8] \
+           lm_weight=[0.5,0.6,0.7,0.8,0.9]
+
+.. note::
+
+    Building a decoding WFST is a long process, so it is better to provide a ``decoding_wfst_file`` path even if you don't have it.
+    This way, the decoding WFST will be buffered to the specified file path and there will be no need to re-build it on the next run.

docs/source/asr/asr_customization/neural_rescoring.rst

@@ -0,0 +1,105 @@
+.. _neural_rescoring:
+
+****************
+Neural Rescoring
+****************
+
+When using the neural rescoring approach, a neural network is used to score candidates. A candidate is the text transcript predicted by the ASR model's decoder. The top K candidates produced by beam search decoding (with a beam width of K) are given to a neural language model for ranking. The language model assigns a score to each candidate, which is usually combined with the scores from beam search decoding to produce the final scores and rankings.
+
+Train Neural Rescorer
+=====================
+
+An example script to train such a language model with Transformer can be found at `examples/nlp/language_modeling/transformer_lm.py <https://github.com/NVIDIA/NeMo/blob/stable/examples/nlp/language_modeling/transformer_lm.py>`__.
+It trains a ``TransformerLMModel`` which can be used as a neural rescorer for an ASR system. For more information on language models training, see LLM/NLP documentation.
+
+
+You can also use a pretrained language model from the Hugging Face library, such as Transformer-XL and GPT, instead of training your model.
+Models like BERT and RoBERTa are not supported by this script because they are trained as Masked Language Models. As a result, they are not efficient or effective for scoring sentences out of the box.
+
+
+Evaluation
+==========
+
+Given a trained TransformerLMModel `.nemo` file or a pretrained HF model, the script available at
+`scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__
+can be used to re-score beams obtained with ASR model. You need the `.tsv` file containing the candidates produced
+by the acoustic model and the beam search decoding to use this script. The candidates can be the result of just the beam
+search decoding or the result of fusion with an N-gram LM. You can generate this file by specifying `--preds_output_folder` for
+`scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__.
+
+The neural rescorer would rescore the beams/candidates by using two parameters of `rescorer_alpha` and `rescorer_beta`, as follows:
+
+.. code-block::
+
+    final_score = beam_search_score + rescorer_alpha*neural_rescorer_score + rescorer_beta*seq_length
+
+The parameter `rescorer_alpha` specifies the importance placed on the neural rescorer model, while `rescorer_beta` is a penalty term that accounts for sequence length in the scores. These parameters have similar effects to `beam_alpha` and `beam_beta` in the beam search decoder and N-gram language model.
+
+Use the following steps to evaluate a neural LM:
+
+#. Obtain `.tsv` file with beams and their corresponding scores. Scores can be from a regular beam search decoder or
+   in fusion with an N-gram LM scores. For a given beam size `beam_size` and a number of examples
+   for evaluation `num_eval_examples`, it should contain (`num_eval_examples` x `beam_size`) lines of
+   form `beam_candidate_text \t score`. This file can be generated by `scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram_ctc.py>`__
+
+#. Rescore the candidates by `scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/neural_rescorer/eval_neural_rescorer.py>`__.
+
+.. code-block::
+
+    python eval_neural_rescorer.py
+        --lm_model=[path to .nemo file of the LM or the name of a HF pretrained model]
+        --beams_file=[path to beams .tsv file]
+        --beam_size=[size of the beams]
+        --eval_manifest=[path to eval manifest .json file]
+        --batch_size=[batch size used for inference on the LM model]
+        --alpha=[the value for the parameter rescorer_alpha]
+        --beta=[the value for the parameter rescorer_beta]
+        --scores_output_file=[the optional path to store the rescored candidates]
+
+The candidates, along with their new scores, are stored at the file specified by `--scores_output_file`.
+
+The following is the list of the arguments for the evaluation script:
+
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| **Argument**        |**Type**| **Default**      | **Description**                                                         |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| lm_model            | str    | Required         | The path of the '.nemo' file of an ASR model, or the name of a          |
+|                     |        |                  | Hugging Face pretrained model like 'transfo-xl-wt103' or 'gpt2'.        |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| eval_manifest       | str    | Required         | Path to the evaluation manifest file (.json manifest file).             |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| beams_file          | str    | Required         | Path to beams file (.tsv) containing the candidates and their scores.   |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| beam_size           | int    | Required         | The width of the beams (number of candidates) generated by the decoder. |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| alpha               | float  | None             | The value for parameter rescorer_alpha                                  |
+|                     |        |                  | Not passing value would enable linear search for rescorer_alpha.        |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| beta                | float  | None             | The value for parameter rescorer_beta                                   |
+|                     |        |                  | Not passing value would enable linear search for rescorer_beta.         |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| batch_size          | int    | 16               | The batch size used to calculate the scores.                            |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| max_seq_length      | int    | 512              | Maximum sequence length (in tokens) for the input.                      |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| scores_output_file  | str    | None             | The optional file to store the rescored beams.                          |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| use_amp             | bool   | ``False``        | Whether to use AMP if available calculate the scores.                   |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+| device              | str    | cuda             | The device to load LM model onto to calculate the scores                |
+|                     |        |                  | It can be 'cpu', 'cuda', 'cuda:0', 'cuda:1', ...                        |
++---------------------+--------+------------------+-------------------------------------------------------------------------+
+
+
+Hyperparameter Linear Search
+----------------------------
+
+The hyperparameter linear search script also supports linear search for parameters `alpha` and `beta`. If any of the two is not
+provided, a linear search is performed to find the best value for that parameter. When linear search is used, initially
+`beta` is set to zero and the best value for `alpha` is found, then `alpha` is fixed with
+that value and another linear search is done to find the best value for `beta`.
+If any of the of these two parameters is already specified, then search for that one is skipped. After each search for a
+parameter, the plot of WER% for different values of the parameter is also shown.
+
+It is recommended to first use the linear search for both parameters on a validation set by not providing any values for `--alpha` and `--beta`.
+Then check the WER curves and decide on the best values for each parameter. Finally, evaluate the best values on the test set.

docs/source/asr/asr_customization/ngpulm_language_modeling_and_customization.rst

@@ -0,0 +1,346 @@
+.. _ngpulm_ngram_modeling:
+
+***************************************************************
+NGPU-LM (GPU-based N-gram Language Model) Language Model Fusion
+***************************************************************
+
+ASR systems can achieve significantly improved accuracy by leveraging **external language model (LM) shallow fusion** during the decoding process. 
+This technique integrates knowledge from an external LM without requiring the ASR model itself to be retrained.
+
+**How Shallow Fusion Works:**
+
+During shallow fusion, the output probabilities generated by the ASR model are combined with those from a separate, external language model.
+The final transcription is then determined by selecting the word sequence that yields the highest combined score. 
+These external LMs are typically trained on vast text datasets, allowing them to capture the statistical patterns, syntactic structures, and contextual dependencies of language. 
+This enables them to predict more plausible word sequences, thereby correcting potential errors from the ASR model.
+
+**Domain Adaptation Benefits:**
+
+Shallow fusion is particularly valuable for **adapting ASR systems to new or specialized domains**.
+By training the external LM on domain-specific text-such as medical, legal, or technical documents-it learns the vocabulary of that field.
+This specialized knowledge guides the ASR decoding process towards more accurate and contextually relevant transcriptions.
+
+Traditionally, shallow fusion has been performed during **beam search decoding**, a method that explores multiple promising hypotheses to find the most likely transcription.
+
+
+NGPU-LM
+=======
+
+A widely used library for training traditional n-gram language models is KenLM. 
+While KenLM (https://github.com/kpu/kenlm) is known for its efficient CPU-based implementation, its reliance on the CPU can limit performance in high-throughput scenarios, especially when dealing with large-scale data.
+
+NGPU-LM on contrast is a GPU-accelerated implementation of a statistical n-gram language model. 
+It uses a **universal trie-based data structure**, which enables fast, batched queries. For full details, please refer to the paper [ngpulm]_.
+
+This enables shallow fusion during **greedy decoding**, creating a middle ground between standard greedy decoding and full beam search with a language model.  
+It preserves the speed and simplicity of greedy decoding while regaining much of the accuracy typically achieved with beam search with external LM fusion.
+While not as accurate as full beam search, greedy decoding with NGPU-LM fusion offers a compelling balance between speed and accuracy.
+
+NeMo provides efficient, fully GPU-based beam search implementations for all major ASR model types, 
+allowing **beam decoding to operate with real-time factors (RTFx) close to those of greedy decoding**.
+At a batch size of 32, the RTFx difference between beam and greedy decoding is only about 20%.
+These implementations incorporate NGPU-LM, enabling fast, fully GPU-based decoding and customization.  
+This enables users to customize decoding while maintaining reasonable speed, even in beam search mode.
+For full details, please refer to the [beamsearch]_.
+
+NGPU-LM fusion is supported for BPE-based ASR models (CTC, RNNT, TDT, AED) during both greedy and beam decoding.
+
+Train NGPU-LM
+=============
+
+NGPU-LM is built using `.ARPA` files generated by the KenLM library. You can train an n-gram LM using the following script:  
+`train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
+
+The generated `.ARPA` files can be directly used for GPU-based decoding.  
+However, for faster performance, it is recommended to convert the model to the `.nemo` format by setting the ``save_nemo`` flag to ``true``.
+
+.. code-block::
+
+    python train_kenlm.py nemo_model_file=<path to the .nemo file of the model> \
+                              train_paths=<list of paths to the training text or JSON manifest files> \
+                              kenlm_bin_path=<path to the bin folder of KenLM library> \
+                              kenlm_model_file=<path to store the binary KenLM model> \
+                              ngram_length=<order of N-gram model> \
+                              preserve_arpa=true \
+                              save_nemo=True
+
+For a complete list of arguments and usage details, refer to the :ref:`train-ngram-lm`.
+
+
+.. note::
+
+    It is recommended that you use 6 as the order of the N-gram model for BPE-based models. Higher orders may require re-compiling KenLM to support them.
+
+.. _ctc-decoding-with-ngpulm:
+
+Decoding with NGPU-LM
+=====================
+
+To run inference with NGPU-LM fusion, the ``ngram_lm_model`` and ``ngram_lm_alpha`` fields must be specified in the decoding configuration.
+
+.. note::
+
+    For CTC, RNNT, and TDT models, these fields should be set within the respective ``greedy`` or ``beam`` sub-configurations.  
+    For AED models running in greedy mode, set the beam size to 1 and specify these fields under the ``beam`` sub-configuration.
+
+Examples for different model types are provided below.
+
+
+
+CTC Decoding with NGPU-LM
+-------------------------
+
+**Greedy Search:**
+
+You can run NGPU-LM shallow fusion during greedy CTC decoding using the following command:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-ctc-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        ctc_decoding.greedy.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        ctc_decoding.greedy.ngram_lm_alpha=0.2 \
+        ctc_decoding.greedy.allow_cuda_graphs=True \
+        ctc_decoding.strategy="greedy_batch"
+
+**Beam Search:**
+
+During CTC beam search, each hypothesis is scored using the following formula:
+
+.. code-block::
+
+    final_score = acoustic_score + ngram_lm_alpha * lm_score + beam_beta * seq_length
+
+where:
+    - ``acoustic_score`` is the score predicted by the ASR.
+    - ``lm_score`` is the score predicted by the NGPU-LM LM.
+    - ``ngram_lm_alpha`` is the weight given to the language model.
+    - ``beam_beta`` is a penalty term that accounts for sequence length in the scores.
+
+For running fully batched GPU-based CTC decoding with NGPU-LM, you can use the following command:
+
+The following is the list of the adjustable arguments of batched CTC decoding algorithm ``beam_batch``:
+
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| **Argument**     | **Type**  | **Default** | **Description**                                                                                                                |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_lm_alpha   | float     | Required    | Weight factor applied to the language model scores.                                                                            |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_size        | int       | 4           | Beam size.                                                                                                                     |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_beta        | float     | 1           | Penalty applied to word insertions to control the trade-off between insertion and deletion errors during beam search decoding. |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_threshold   | float     | 20          | Threshold used to prune candidate hypotheses by comparing their scores to the best hypothesis.                                 |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-ctc-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        ctc_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        ctc_decoding.beam.ngram_lm_alpha=0.2 \
+        ctc_decoding.beam.beam_size=12 \
+        ctc_decoding.beam.beam_beta=1.0 \
+        ctc_decoding.strategy="beam_batch" \
+        ctc_decoding.beam.allow_cuda_graphs=True
+
+
+RNN-T/TDT decoding with NGPU-LM
+-------------------------------
+
+**Greedy Search:**
+
+You can run NGPU-LM shallow fusion during greedy RNN-T / TDT decoding using the following command:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-rnnt-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        rnnt_decoding.greedy.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        rnnt_decoding.greedy.ngram_lm_alpha=0.2 \
+        rnnt_decoding.greedy.allow_cuda_graphs=True \
+        rnnt_decoding.strategy="greedy_batch"
+
+.. note::
+
+    To run the inference with TDT model, you need to provide pretrained TDT model in ``pretrained_name`` field (for example ``nvidia/parakeet-tdt_ctc-1.1b`` ).
+
+
+**Beam Search:**
+
+During RNN-T / TDT beam search, each hypothesis is scored using the following formula:
+
+.. code-block::
+
+    final_score = acoustic_score + ngram_lm_alpha * lm_score
+
+where:
+    - ``acoustic_score`` is the score predicted by the ASR.
+    - ``lm_score`` is the score predicted by the NGPU-LM LM.
+    - ``ngram_lm_alpha`` is the weight given to the language model.
+
+Final hypotheses is chosen based on the normalized score ``final_score / seq_length``.
+
+
+*Blank Scoring in Transducer Models*
+
+Transducer models include a blank symbol (``∅``) for frame transitions, while LMs do not model blanks.  
+During shallow fusion, the LM is typically applied only to non-blank tokens:
+
+.. math::
+
+    \ln p_{\text{tot}}[k] =
+    \begin{cases}
+      \ln p[k] + \lambda \ln p_{\text{LM}}[k], & k \in V \\
+      \ln p[\emptyset], & k = \emptyset
+    \end{cases}
+
+This can lead to excessive blank predictions at higher LM weights, increasing deletion errors.  
+NeMo supports a blank-aware scoring method that adjusts LM contributions to better balance predictions:
+
+.. math::
+
+    \ln p_{\text{tot}}[k] =
+    \begin{cases}
+      \ln p[k] + \lambda \ln((1 - p[\emptyset]) \cdot p_{\text{LM}}[k]), & k \in V \\
+      (1 + \lambda) \ln p[\emptyset], & k = \emptyset
+    \end{cases}
+
+*Early vs. Late Pruning*
+
+In shallow fusion, LM and ASR scores can be combined at different stages:
+
+- **Early pruning:** ASR selects top hypotheses, then LM rescoring is applied. Efficient for small beams.
+- **Late pruning:** ASR and LM scores are combined before pruning. More accurate but requires full-vocab LM queries.
+
+For Transducer models, late pruning with the blank-aware scoring method generally yields better performance than the standard approach.
+
+*Beam Search Strategies:*
+
+In NeMo fully batched implementation of following strategies are supported:
+
+- **malsd_batch:** fully batched implemention of modified Alignment-Length Synchronous Decoding [alsd]_, supporting both RNNT and TDT models.
+- **maes_batch:** fully batched implemention of modified Adaptive Expansion Search [aes]_, supporting for only RNNT models. CudaGraphs are not supported.
+
+The following is the list of the adjustable arguments of batched CTC decoding algorithm ``beam_batch``:
+
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| **Argument**          | **Type**  | **Strategy**            | **Default**      | **Description**                                                                                                                |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_lm_alpha        | float     | malsd_batch, maes_batch | Required         | Weight factor applied to the language model scores.                                                                            |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| beam_size             | int       | malsd_batch, maes_batch | 4                | Beam size.                                                                                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| pruning_mode          | str       | malsd_batch, maes_batch | late             | Mode for hypotheses pruning. Can be ``early`` or ``late``.                                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| blank_lm_score_mode   | str       | malsd_batch, maes_batch | lm_weighted_full | Mode for blank symbol scoring. Can be ``no_score`` or ``lm_weighted_full``                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| max_symbols_per_step  | int       | malsd_batch             | 10               | Max symbols to emit on each step to avoid infinite looping.                                                                    |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| maes_num_step         | int       | maes_batch              | 2                | Number of adaptive steps to take.                                                                                              |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| maes_expansion_beta   | float     | maes_batch              | 1.0              | Maximum number of prefix expansions allowed, in addition to the beam size.                                                     |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+| maes_expansion_gamma  | float     | maes_batch              | 2.3              | Threshold used to prune candidate hypotheses by comparing their scores to the best hypothesis.                                 |
++-----------------------+-----------+-------------------------+------------------+--------------------------------------------------------------------------------------------------------------------------------+
+
+You can run NGPU-LM shallow fusion during beam RNN-T / TDT decoding using the following command:
+
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+        pretrained_name=nvidia/parakeet-rnnt-1.1b \
+        amp=false \
+        amp_dtype=bfloat16 \
+        matmul_precision=high \
+        compute_dtype=bfloat16 \
+        presort_manifest=true \
+        cuda=0 \
+        batch_size=32 \
+        dataset_manifest=<path to the evaluation JSON manifest file> \
+        rnnt_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+        rnnt_decoding.beam.ngram_lm_alpha=0.2 \
+        rnnt_decoding.beam.beam_size=12 \
+        rnnt_decoding.beam.pruning_mode="late" \
+        rnnt_decoding.beam.blank_lm_score_mode="lm_weighted_full" \
+        rnnt_decoding.beam.allow_cuda_graphs=True \
+        rnnt_decoding.strategy="malsd_batch"
+
+.. note::
+
+    To run the inference with TDT model, you need to provide pretrained TDT model in ``pretrained_name`` field (for example ``nvidia/parakeet-tdt_ctc-1.1b`` ).
+
+AED Decoding with NGPU-LM
+-------------------------
+
+**Beam Search:**
+
+You can run NGPU-LM shallow fusion during greedy CTC decoding using the following command:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+    pretrained_name="nvidia/canary-1b" \
+    amp=false \
+    amp_dtype=bfloat16 \
+    matmul_precision=high \
+    compute_dtype=bfloat16 \
+    presort_manifest=true \
+    cuda=0 \
+    batch_size=32 \
+    dataset_manifest=<dataset_manifest>  \
+    multitask_decoding.beam.beam_size=4 \
+    multitask_decoding.beam.ngram_lm_model=<path to the .nemo/.ARPA file of the NGPU-LM model> \
+    multitask_decoding.beam.ngram_lm_alpha=0.2 \
+    multitask_decoding.strategy="beam"
+
+.. note::
+
+    For greedy decoding with NGPU-LM, use beam search with beam_size=1.
+
+
+References
+==========
+
+.. [ngpulm] V. Bataev, A. Andrusenko, L. Grigoryan, A. Laptev, V. Lavrukhin, and B. Ginsburg.  
+   *NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding*.  
+   arXiv:2505.22857, 2025. Available at: https://arxiv.org/abs/2505.22857
+
+.. [beamsearch] L. Grigoryan, V. Bataev, A. Andrusenko, H. Xu, V. Lavrukhin, and B. Ginsburg.  
+   *Pushing the Limits of Beam Search Decoding for Transducer-based ASR Models*.  
+   arXiv:2506.00185, 2025. Available at: https://arxiv.org/abs/2506.00185
+
+.. [alsd] G. Saon, Z. Tüske, and K. Audhkhasi.  
+   *Alignment-Length Synchronous Decoding for RNN Transducer*.  
+   In: ICASSP 2020 – IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7804–7808, 2020.  
+   doi: https://doi.org/10.1109/ICASSP40776.2020.9053040
+
+.. [aes] J. Kim, Y. Lee, and E. Kim.  
+   *Accelerating RNN Transducer Inference via Adaptive Expansion Search*.  
+   IEEE Signal Processing Letters, vol. 27, pp. 2019–2023, 2020.  
+   doi: https://doi.org/10.1109/LSP.2020.3036335

docs/source/asr/asr_customization/ngram_utils.rst

@@ -0,0 +1,151 @@
+.. _ngram-utils:
+
+Scripts for building and merging N-gram Language Models
+=======================================================
+
+.. _train-ngram-lm:
+
+Train N-gram LM
+===============
+
+NeMo utilizes the KenLM library (`https://github.com/kpu/kenlm`) for building efficient n-gram language models.
+
+.. note::
+
+    KenLM is not installed by default in NeMo.  
+    Please see the installation instructions in the script:  
+    `scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/install_beamsearch_decoders.sh>`__.
+
+    Alternatively, you can build a Docker image with all required dependencies using:  
+    `scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__.
+
+The script for training an n-gram language model with KenLM is available here:  
+`scripts/asr_language_modeling/ngram_lm/train_kenlm.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/train_kenlm.py>`__.
+
+This script supports training n-gram LMs on both character-level and BPE-level encodings, which are automatically detected from the model type. The resulting language models can then be used with beam search decoders integrated on top of ASR models.
+
+You can train an n-gram model using the following command:
+
+.. code-block::
+
+    python train_kenlm.py nemo_model_file=<path to the .nemo file of the model> \
+                              train_paths=<list of paths to the training text or JSON manifest files> \
+                              kenlm_bin_path=<path to the bin folder of KenLM library> \
+                              kenlm_model_file=<path to store the binary KenLM model> \
+                              ngram_length=<order of N-gram model> \
+                              preserve_arpa=true
+
+The `train_paths` parameter allows for various input types, such as a list of text files, JSON manifests, or directories, to be used as the training data.
+If the file's extension is anything other than `.json`, it assumes that data format is plain text. For plain text format, each line should contain one
+sample. For the JSON manifests, the file must contain JSON-formatted samples per each line like this:
+
+.. code-block::
+
+    {"audio_filepath": "/data_path/file1.wav", "text": "The transcript of the audio file."}
+
+This code extracts the `text` field from each line to create the training text file. After the N-gram model is trained, it is stored at the path specified by `kenlm_model_file`.
+
+The following is the list of the arguments for the training script:
+
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| **Argument**     | **Type**  | **Default** | **Description**                                                                                                                |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| nemo_model_file  | str       | Required    | The path to `.nemo` file of the ASR model, or name of a pretrained NeMo model to extract a tokenizer.                          |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| train_paths      | List[str] | Required    | List of training files or folders. Files can be a plain text file or ".json" manifest or ".json.gz".                           |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| kenlm_model_file | str       | Required    | The path to store the KenLM binary model file.                                                                                 |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| kenlm_bin_path   | str       | Required    | The path to the bin folder of KenLM. It is a folder named `bin` under where KenLM is installed.                                |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_length**   | int       | Required    | Specifies order of N-gram LM.                                                                                                  |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| ngram_prune      | List[int] | [0]         | List of thresholds to prune N-grams. Example: [0,0,1]. See Pruning section on the https://kheafield.com/code/kenlm/estimation  |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| cache_path       | str       | ``""``      | Cache path to save tokenized files.                                                                                            |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| preserve_arpa    | bool      | ``False``   | Whether to preserve the intermediate ARPA file after construction of the BIN file.                                             |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| verbose          | int       | 1           | Verbose level.                                                                                                                 |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+| save_nemo        | bool      | ``False``   | Whether to save LM in .nemo format.                                                                                            |
++------------------+-----------+-------------+--------------------------------------------------------------------------------------------------------------------------------+
+
+..note::
+It is recommended that you use 6 as the order of the N-gram model for BPE-based models. Higher orders may require re-compiling KenLM to support them.
+
+
+Combine N-gram Language Models
+==============================
+
+Before combining N-gram LMs, install the required OpenGrm NGram library using `scripts/installers/install_opengrm.sh <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/install_opengrm.sh>`__.
+Alternatively, you can use Docker image `scripts/installers/Dockerfile.ngramtools <https://github.com/NVIDIA/NeMo/blob/stable/scripts/installers/Dockerfile.ngramtools>`__ with all the necessary dependencies.
+
+Alternatively, you can use the Docker image at:
+`scripts/asr_language_modeling/ngram_lm/ngram_merge.py <https://github.com/NVIDIA/NeMo/blob/stable/scripts/asr_language_modeling/ngram_lm/ngram_merge.py>`__, which includes all the necessary dependencies.
+
+This script interpolates two ARPA N-gram language models and creates a KenLM binary file that can be used with the beam search decoders on top of ASR models.
+You can specify weights (`--alpha` and `--beta`) for each of the models (`--ngram_a` and `--ngram_b`) correspondingly: `alpha` * `ngram_a` + `beta` * `ngram_b`.
+This script supports both character level and BPE level encodings and models which are detected automatically from the type of the model.
+
+To combine two N-gram models, you can use the following command:
+
+.. code-block::
+
+    python ngram_merge.py  --kenlm_bin_path <path to the bin folder of KenLM library> \
+                    --ngram_bin_path  <path to the bin folder of OpenGrm Ngram library> \
+                    --arpa_a <path to the ARPA N-gram model file A> \
+                    --alpha <weight of N-gram model A> \
+                    --ar
+                    pa_b <path to the ARPA N-gram model file B> \
+                    --beta <weight of N-gram model B> \
+                    --out_path <path to folder to store the output files>
+
+
+
+If you provide `--test_file` and `--nemo_model_file`, This script supports both character-level and BPE-level encodings and models, which are detected automatically based on the type of the model.
+Note, the result of each step during the process is cached in the temporary file in the `--out_path`, to speed up further run.
+You can use the `--force` flag to discard the cache and recalculate everything from scratch.
+
+.. code-block::
+
+    python ngram_merge.py  --kenlm_bin_path <path to the bin folder of KenLM library> \
+                    --ngram_bin_path  <path to the bin folder of OpenGrm Ngram library> \
+                    --arpa_a <path to the ARPA N-gram model file A> \
+                    --alpha <weight of N-gram model A> \
+                    --arpa_b <path to the ARPA N-gram model file B> \
+                    --beta <weight of N-gram model B> \
+                    --out_path <path to folder to store the output files>
+                    --nemo_model_file <path to the .nemo file of the model> \
+                    --test_file <path to the test file> \
+                    --symbols <path to symbols (.syms) file> \
+                    --force <flag to recalculate and rewrite all cached files>
+
+
+The following is the list of the arguments for the opengrm script:
+
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| **Argument**         |**Type**| **Default**      | **Description**                                                                                                 |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| kenlm_bin_path       | str    | Required         | The path to the bin folder of KenLM library. It is a folder named `bin` under where KenLM is installed.         |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| ngram_bin_path       | str    | Required         | The path to the bin folder of OpenGrm Ngram. It is a folder named `bin` under where OpenGrm Ngram is installed. |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| arpa_a               | str    | Required         | Path to the ARPA N-gram model file A.                                                                           |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| alpha                | float  | Required         | Weight of N-gram model A.                                                                                       |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| arpa_b               | int    | Required         | Path to the ARPA N-gram model file B.                                                                           |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| beta                 | float  | Required         | Weight of N-gram model B.                                                                                       |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| out_path             | str    | Required         | Path for writing temporary and resulting files.                                                                 |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| test_file            | str    | None             | Path to test file to count perplexity if provided.                                                              |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| symbols              | str    | None             | Path to symbols (.syms) file. Could be calculated if it is not provided.                                        |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| nemo_model_file      | str    | None             | The path to '.nemo' file of the ASR model, or name of a pretrained NeMo model.                                  |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| force                | bool   | ``False``        | Whether to recompile and rewrite all files.                                                                     |
++----------------------+--------+------------------+-----------------------------------------------------------------------------------------------------------------+

docs/source/asr/asr_customization/word_boosting.rst

@@ -0,0 +1,362 @@
+.. _word_boosting:
+
+****************************************************
+Word Boosting
+****************************************************
+
+.. _word_boosting_gpupb:
+
+GPU-PB
+========================
+
+GPU-PB is a GPU-accelerated Phrase-Boosting method supported for CTC, RNN-T/TDT, and AED (Canary) models based on NGPU-LM infrastructure.
+The method supports greedy and beam search decoding, including CUDA graphs mode. GPU-PB is compatible with NGPU-LM at the same decoding run.
+
+GPU-PB is applied only at the decoding step in shallow fusion mode. You do not need to retrain the ASR model.
+During greedy or beam search decoding, GPU-PB rescales ASR model scores with a boosting tree at the token level.
+The boosting tree is built from a context phrases list, which is provided by the user.
+
+**NOTE**: for ASR models that support capitalization by default (e.g., Canary or parakeet-tdt-0.6b-v2), you need to capitalize all the key phrases in advance (and capitalize the full word for abbreviations).
+You can use LLM for this task.
+
+More details about GPU-PB method can be found in the `original paper <https://arxiv.org/abs/2508.07014>`__.
+
+Usage
+-----
+We support three ways to pass the context phrases into the decoding script:
+
+1. Build a boosting tree for a specific ASR model (step 0.0) and use it for all the decoding evaluation by ``boosting_tree.model_path`` (step 1.1-3.1).
+2. Provide a file with context phrases ``boosting_tree.key_phrases_file`` - one phrase per line  (step 1.1-3.1).
+3. Provide a python list of context phrases ``boosting_tree.key_phrases_list`` (step 1.1-3.1).
+
+The use of the Phrase-Boosting tree is controlled by ``boosting_tree`` config (``BoostingTreeModelConfig``) for all the models.
+For prepared boosting tree use ``boosting_tree.model_path=${PATH_TO_BTREE}``.
+We recommend to provide the list of context phrases directly into ``speech_to_text_eval.py`` by ``boosting_tree.key_phrases_file=${KEY_WORDS_LIST}``.
+
+List of the most important parameters:
+
+*  ``strategy`` - The strategy to use for decoding depending on the model type (CTC - greedy_batch or beam_batch; RNN-T/TDT - greedy_batch or malsd_batch; AED - beam).
+*  ``model_path``, ``key_phrases_file``, ``key_phrases_list`` - The way to pass the context phrases into the decoding script.
+*  ``context_score`` - The score for each arc transition in the context graph (1.0 is recommended).
+*  ``depth_scaling`` - The scaling factor for the depth of the context graph (2.0 is recommended for CTC, RNN-T and TDT, 1.0 for Canary).
+*  ``boosting_tree_alpha`` - Weight of the GPU-PB boosting tree during shallow fusion decoding (tune it according to your data).
+
+**0.0. [Optional] Build the boosting tree for a specific ASR model:**
+
+.. code-block::
+
+    python scripts/asr_context_biasing/build_gpu_boosting_tree.py \
+            asr_model_path=${ASR_NEMO_MODEL_FILE} \
+            key_phrases_file=${CONTEXT_BIASING_LIST} \
+            save_to=${PATH_TO_SAVE_BTREE} \
+            context_score=${CONTEXT_SCORE} \
+            depth_scaling=${DEPTH_SCALING} \
+            use_triton=True
+
+**1.1. CTC greedy batch decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        ctc_decoding.strategy="greedy_batch" \
+        ctc_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        ctc_decoding.greedy.boosting_tree.context_score=1.0 \
+        ctc_decoding.greedy.boosting_tree.depth_scaling=2.0 \
+        ctc_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}
+
+
+**1.2. CTC beam batch decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        ctc_decoding.strategy="beam_batch" \
+        ctc_decoding.beam.beam_size=${BEAM_SIZE} \
+        ctc_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        ctc_decoding.beam.boosting_tree.context_score=1.0 \
+        ctc_decoding.beam.boosting_tree.depth_scaling=2.0 \
+        ctc_decoding.beam.boosting_tree_alpha=${BT_ALPHA}
+
+**2.1. RNN-T/TDT greedy batch decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        rnnt_decoding.strategy="greedy_batch" \
+        rnnt_decoding.greedy.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        rnnt_decoding.greedy.boosting_tree.context_score=1.0 \
+        rnnt_decoding.greedy.boosting_tree.depth_scaling=2.0 \
+        rnnt_decoding.greedy.boosting_tree_alpha=${BT_ALPHA}
+
+**2.2. RNN-T/TDT beam (malsd_batch) decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        rnnt_decoding.strategy="malsd_batch" \
+        rnnt_decoding.beam.beam_size=${BEAM_SIZE} \
+        rnnt_decoding.beam.boosting_tree.key_phrases_file=${KEY_WORDS_LIST} \
+        rnnt_decoding.beam.boosting_tree.context_score=1.0 \
+        rnnt_decoding.beam.boosting_tree.depth_scaling=2.0 \
+        rnnt_decoding.beam.boosting_tree_alpha=${BT_ALPHA}
+
+**3.1. AED (Canary) greedy (beam_size=1) or beam (beam_size>1) decoding:**
+
+.. code-block::
+
+    python examples/asr/speech_to_text_eval.py \
+        model_path=${MODEL_NAME} \
+        dataset_manifest=${EVAL_MANIFEST} \
+        batch_size=${BATCH_SIZE} \
+        output_filename=${OUT_MANIFEST} \
+        multitask_decoding.strategy="beam" \
+        multitask_decoding.beam.beam_size=${BEAM_SIZE} \
+        multitask_decoding.beam.boosting_tree.key_phrases_file=${CONTEXT_BIASING_LIST} \
+        multitask_decoding.beam.boosting_tree.context_score=1.0 \
+        multitask_decoding.beam.boosting_tree.depth_scaling=1.0 \
+        multitask_decoding.beam.boosting_tree_alpha=${BT_ALPHA} \
+        gt_lang_attr_name="target_lang" \
+        gt_text_attr_name="text"
+
+Results evaluation
+------------------
+
+You can compute the F-score for the list of context phrases directly from the decoding manifest.
+
+.. code-block::
+
+    python scripts/asr_context_biasing/compute_key_words_fscore.py \
+            --input_manifest=${DECODING_MANIFEST} \
+            --key_words_file=${CONTEXT_PHRASES_LIST}
+
+
+.. _word_boosting_per_stream:
+
+Per-Stream Phrase Boosting
+==========================
+
+Per-stream (per-utterance) phrase boosting extends GPU-PB to allow specifying different key phrases for each audio stream or utterance in a batch.
+This is useful when different utterances require different context biasing (e.g., different speaker names, product terms, or domain vocabulary per audio).
+
+Per-stream boosting is currently supported for **greedy label-looping decoding with Transducers (RNN-T, TDT)**, including cache-aware streaming models.
+
+Manifest-based Usage
+--------------------
+
+Specify per-utterance key phrases in your manifest using the ``biasing_request`` field:
+
+.. code-block:: json
+
+    {"audio_filepath": "/data/file1.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["one phrase"]}}}
+    {"audio_filepath": "/data/file2.wav", "text": "ground truth", "biasing_request": {"boosting_model_cfg": {"key_phrases_list": ["other phrases", "and this one"]}}}
+
+Use the streaming inference script with ``use_per_stream_biasing=true``:
+
+.. code-block:: bash
+
+    python examples/asr/asr_streaming_inference/asr_streaming_infer.py \
+        --config-path="../conf/asr_streaming_inference/" \
+        --config-name=cache_aware_rnnt.yaml \
+        audio_file="<manifest_with_boosting_requests>" \
+        output_filename="result.jsonl" \
+        asr.model_name="nvidia/parakeet-rnnt-1.1b" \
+        asr.decoding.greedy.enable_per_stream_biasing=True
+
+Python API Usage
+----------------
+
+.. code-block:: python
+
+    from omegaconf import open_dict
+    from nemo.collections.asr.models import EncDecRNNTBPEModel
+    from nemo.collections.asr.parts.context_biasing.biasing_multi_model import BiasingRequestItemConfig
+    from nemo.collections.asr.parts.context_biasing.boosting_graph_batched import BoostingTreeModelConfig
+    from nemo.collections.asr.parts.utils.rnnt_utils import Hypothesis
+
+    asr_model = EncDecRNNTBPEModel.from_pretrained("nvidia/parakeet-rnnt-1.1b")
+    asr_model.to("cuda")
+
+    with open_dict(asr_model.cfg.decoding):
+        asr_model.cfg.decoding.strategy = "greedy_batch"
+        asr_model.cfg.decoding.greedy.loop_labels = True
+        asr_model.cfg.decoding.greedy.enable_per_stream_biasing = True
+    asr_model.change_decoding_strategy(asr_model.cfg.decoding)
+
+    biasing_requests = [
+        BiasingRequestItemConfig(
+            boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["one phrase"]),
+            boosting_model_alpha=2.0,
+        ),
+        None,  # no biasing for this utterance
+        BiasingRequestItemConfig(
+            boosting_model_cfg=BoostingTreeModelConfig(key_phrases_list=["other phrases"]),
+            boosting_model_alpha=1.0,
+        ),
+    ]
+
+    results = asr_model.transcribe(
+        audio=["file1.wav", "file2.wav", "file3.wav"],
+        partial_hypothesis=[
+            Hypothesis.empty_with_biasing_cfg(biasing_cfg=req) if req else None
+            for req in biasing_requests
+        ],
+        return_hypotheses=True,
+    )
+
+Caching
+-------
+
+Building a boosting model from a phrase list has some overhead. NeMo provides caching mechanisms to speed up repeated use of the same phrases:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 15 60 25
+
+   * - Strategy
+     - Description
+     - Recommended For
+   * - Memory
+     - Set ``cache_key`` on ``BiasingRequestItemConfig`` to cache compiled models in memory by a string key.
+     - Repeated phrase sets
+   * - Disk
+     - Set ``model_path`` on ``BoostingTreeModelConfig`` to save/load compiled models from disk.
+     - Persistent caching
+   * - Decoder
+     - Set ``auto_manage_multi_model=False`` and manually manage models in the decoder's multi-model.
+     - Advanced use cases
+
+With memory caching, per-stream boosting achieves near-zero overhead compared to global (shared) boosting.
+
+
+.. _word_boosting_flashlight:
+
+Flashlight-based Word Boosting
+==============================
+
+
+The Flashlight decoder supports word boosting during CTC decoding using a KenLM binary and corresponding lexicon. Word boosting only works in lexicon-decoding mode and does not function in lexicon-free mode. It allows you to bias the decoder for certain words by manually increasing or decreasing the probability of emitting specific words. This can be very helpful if you have uncommon or industry-specific terms that you want to ensure are transcribed correctly.
+
+For more information, go to `word boosting <https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-customizing.html#word-boosting>`__
+
+To use word boosting in NeMo, create a simple tab-separated text file. Each line should contain a word to be boosted, followed by a tab, and then the boosted score for that word.
+
+For example:
+
+.. code-block::
+
+    nvidia	40
+    geforce	50
+    riva	80
+    turing	30
+    badword	-100
+
+Positive scores boost words higher in the LM decoding step so they show up more frequently, whereas negative scores
+squelch words so they show up less frequently. The recommended range for the boost score is +/- 20 to 100.
+
+The boost file handles both in-vocabulary words and OOV words just fine, so you can specify both IV and OOV words with corresponding scores.
+
+You can then pass this file to your Flashlight config object during decoding:
+
+.. code-block::
+
+    # Lexicon-based decoding
+    python eval_beamsearch_ngram_ctc.py ... \
+           decoding_strategy="flashlight" \
+           decoding.beam.flashlight_cfg.lexicon_path='/path/to/lexicon.lexicon' \
+           decoding.beam.flashlight_cfg.boost_path='/path/to/my_boost_file.boost' \
+           decoding.beam.flashlight_cfg.beam_size_token = 32 \
+           decoding.beam.flashlight_cfg.beam_threshold = 25.0
+
+.. _word_boosting_ctcws:
+
+CTC-WS: Context-biasing (Word Boosting) without External LM
+===========================================================
+
+NeMo toolkit supports a fast context-biasing method for CTC and Transducer (RNN-T) ASR models with CTC-based Word Spotter.
+The method involves decoding CTC log probabilities with a context graph built for words and phrases from the context-biasing list.
+The spotted context-biasing candidates (with their scores and time intervals) are compared by scores with words from the greedy CTC decoding results to improve recognition accuracy and prevent false accepts of context-biasing.
+
+A Hybrid Transducer-CTC model (a shared encoder trained together with CTC and Transducer output heads) enables the use of the CTC-WS method for the Transducer model.
+Context-biasing candidates obtained by CTC-WS are also filtered by the scores with greedy CTC predictions and then merged with greedy Transducer results.
+
+Scheme of the CTC-WS method:
+
+.. image:: https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/asset-post-v1.22.0-ctcws_scheme_1.png
+    :align: center
+    :alt: CTC-WS scheme
+    :width: 80%
+
+High-level overview of the context-biasing words replacement with CTC-WS method:
+
+.. image:: https://github.com/NVIDIA/NeMo/releases/download/v1.22.0/asset-post-v1.22.0-ctcws_scheme_2.png
+    :align: center
+    :alt: CTC-WS high level overview
+    :width: 80%
+
+More details about CTC-WS context-biasing can be found in the `tutorial <https://github.com/NVIDIA/NeMo/tree/main/tutorials/asr/ASR_Context_Biasing.ipynb>`__.
+
+To use CTC-WS context-biasing, you need to create a context-biasing text file that contains words/phrases to be boosted, with its transcriptions (spellings) separated by underscore.
+Multiple transcriptions can be useful for abbreviations ("gpu" -> "g p u"), compound words ("nvlink" -> "nv link"), 
+or words with common mistakes in the case of our ASR model ("nvidia" -> "n video").
+
+Example of the context-biasing file:
+
+.. code-block::
+
+    nvidia_nvidia
+    omniverse_omniverse
+    gpu_gpu_g p u
+    dgx_dgx_d g x_d gx
+    nvlink_nvlink_nv link
+    ray tracing_ray tracing
+
+The main script for CTC-WS context-biasing in NeMo is: 
+
+.. code-block::
+
+    {NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py
+
+Context-biasing is managed by ``apply_context_biasing`` parameter [true or false].
+Other important context-biasing parameters are:
+
+*  ``beam_threshold`` - threshold for CTC-WS beam pruning.
+*  ``context_score`` - per token weight for context biasing.
+*  ``ctc_ali_token_weight`` - per token weight for CTC alignment (prevents false acceptances of context-biasing words).
+
+All the context-biasing parameters are selected according to the default values in the script.
+You can tune them according to your data and ASR model (list all the values in the [] separated by commas)
+for example: ``beam_threshold=[7.0,8.0,9.0]``, ``context_score=[3.0,4.0,5.0]``, ``ctc_ali_token_weight=[0.5,0.6,0.7]``.
+The script will run the recognition with all the combinations of the parameters and will select the best one based on WER value.
+
+.. code-block::
+
+    # Context-biasing with the CTC-WS method for CTC ASR model 
+    python {NEMO_DIR_PATH}/scripts/asr_context_biasing/eval_greedy_decoding_with_context_biasing.py \
+            nemo_model_file={ctc_model_name} \
+            input_manifest={test_nemo_manifest} \
+            preds_output_folder={exp_dir} \
+            decoder_type="ctc" \
+            acoustic_batch_size=64 \
+            apply_context_biasing=true \
+            context_file={cb_list_file_modified} \
+            beam_threshold=[7.0] \
+            context_score=[3.0] \
+            ctc_ali_token_weight=[0.5]
+
+To use Transducer head of the Hybrid Transducer-CTC model, you need to set ``decoder_type=rnnt``.

docs/source/asr/asr_language_modeling_and_customization.rst

@@ -0,0 +1,213 @@
+.. _asr_language_modeling_and_customization:
+
+#######################################
+ASR Language Modeling and Customization
+#######################################
+
+NeMo supports decoding-time customization techniques such as *language modeling* and *word boosting*,
+which improve transcription accuracy by incorporating external knowledge or domain-specific vocabulary—without retraining the model.
+
+
+Decoder Types
+-------------
+
+NeMo ASR models use different decoder architectures. The table below summarizes them:
+
+.. list-table::
+   :header-rows: 1
+
+   * - Decoder
+     - Type
+     - Description
+     - Models
+   * - **CTC**
+     - Non-autoregressive
+     - Connectionist Temporal Classification. Fast inference, supports LM fusion and word boosting.
+     - Parakeet-CTC, FastConformer-CTC
+   * - **RNN-T**
+     - Autoregressive
+     - Recurrent Neural Network Transducer. Strong accuracy, streaming-friendly.
+     - Parakeet-RNNT, FastConformer-Transducer
+   * - **TDT**
+     - Autoregressive
+     - Token-and-Duration Transducer. Extends RNN-T with duration prediction for better timestamps.
+     - Parakeet-TDT
+   * - **AED**
+     - Autoregressive
+     - Attention Encoder-Decoder. Multi-task capable (ASR + AST), prompt-based language control.
+     - Canary-1B, Canary-1B-V2, Canary-1B-Flash
+   * - **Hybrid**
+     - Both
+     - Joint RNN-T + CTC training. Use either decoder at inference time.
+     - FastConformer Hybrid models
+
+
+Language Modeling
+-----------------
+
+In NeMo two approaches of external language modeling are supported:
+
+- **Language Model Fusion:** 
+    Language model (LM) fusion integrates scores from an external statistical n-gram model into the ASR decoder.
+    This helps guide decoding toward more likely word sequences based on text corpora.
+
+    NeMo provides two approaches for language model shallow fusion with ASR systems:
+
+    **1. NGPU-LM (Recommended for Production)**
+        GPU-accelerated LM fusion for all major model types: CTC, RNN-T, TDT, and AED models.
+
+        - Customization during both greedy and beam decoding.
+
+        - Fast beam decoding for all major model types, offering only 20% RTFx difference between beam and greedy decoding.
+
+        - Integration with NGPU-LM GPU-based ngram LM.
+
+        For details, please refer to :ref:`ngpulm_ngram_modeling`
+
+    **2. KenLM (Traditional CPU-based)**
+        CPU-based LM fusion using the KenLM library.
+        
+        .. note::
+
+            These approaches, especially beam decoding, can be extremely slow and are retained in the repository primarily for backward compatibility.
+            If possible, we recommend using NGPU-LM for improved performance.
+
+        For details, please refer to :ref:`ngram_modeling`
+
+- **Neural Rescoring:** 
+    When using the neural rescoring approach, a neural network is used to score candidates. A candidate is the text transcript predicted by the ASR model’s decoder. 
+    The top K candidates produced by beam search decoding (with a beam width of K) are given to a neural language model for ranking.
+    The language model assigns a score to each candidate, which is usually combined with the scores from beam search decoding to produce the final scores and rankings.
+
+    For details, please refer to :ref:`neural_rescoring`.
+
+
+Word Boosting
+-------------
+
+Word boosting increases the likelihood of specific words or phrases during decoding by applying a positive bias, helping the model better recognize names,
+uncommon terms, and custom vocabulary.
+
+- :ref:`word_boosting_gpupb` (preferred): GPU-accelerated phrase-boosting for CTC, RNN-T/TDT, and AED (Canary) models supporting greedy and beam search decoding.
+
+- :ref:`word_boosting_flashlight`: Word-boosting method for CTC models with external n-gram LM.
+
+- :ref:`word_boosting_ctcws`: Word-boosting method for hybrid (Transducer-CTC) models without LM.
+
+For details, please refer to: :ref:`word_boosting`.
+
+
+LM Training
+-----------
+
+NeMo provides tools for training n-gram language models that can be used for language model fusion or word-boosting.
+For details, please refer to: :ref:`ngram-utils`.
+
+
+CUDA Graphs
+-----------
+
+CUDA graphs accelerate decoding by capturing and replaying GPU operations, eliminating kernel launch overhead.
+Support varies by decoder strategy:
+
+.. list-table::
+   :header-rows: 1
+
+   * - Strategy
+     - Config Parameter
+     - Default
+     - Notes
+   * - ``greedy_batch`` (RNN-T, TDT)
+     - ``use_cuda_graph_decoder``
+     - ``true``
+     - Requires ``loop_labels=True`` and ``blank_as_pad=True``
+   * - ``maes_batch``, ``malsd_batch`` (beam)
+     - ``allow_cuda_graphs``
+     - ``true``
+     - Batched beam search strategies
+   * - Non-batched ``greedy`` / ``beam``
+     - N/A
+     - N/A
+     - Not supported; standard decoding used
+
+To disable CUDA graphs (e.g. for debugging or when preserving alignments with frame-looping):
+
+**Via Python (at runtime):**
+
+.. code-block:: python
+
+    model.disable_cuda_graphs()
+
+**Greedy decoding** — use ``use_cuda_graph_decoder=true/false``:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+       pretrained_name="nvidia/parakeet-rnnt-1.1b" \
+       dataset_manifest=<dataset_manifest> \
+       batch_size=32 \
+       output_filename=decoded.jsonl \
+       rnnt_decoding.strategy="greedy_batch" \
+       rnnt_decoding.greedy.use_cuda_graph_decoder=true
+
+**Beam decoding** — use ``allow_cuda_graphs=true/false``:
+
+.. code-block:: bash
+
+    python examples/asr/speech_to_text_eval.py \
+       pretrained_name="nvidia/parakeet-rnnt-1.1b" \
+       dataset_manifest=<dataset_manifest> \
+       batch_size=32 \
+       output_filename=decoded.jsonl \
+       rnnt_decoding.strategy="malsd_batch" \
+       rnnt_decoding.beam.max_symbols_per_step=10 \
+       rnnt_decoding.beam.beam_size=12 \
+       rnnt_decoding.beam.allow_cuda_graphs=true
+
+When unsupported, NeMo falls back to standard decoding automatically.
+
+
+Confidence Estimation
+---------------------
+
+NeMo supports per-frame, per-token, and per-word confidence scores during decoding.
+Confidence estimation helps applications decide when to trust ASR output and when to request human review.
+
+.. code-block:: yaml
+
+  decoding:
+    confidence_cfg:
+      preserve_frame_confidence: false
+      preserve_token_confidence: false
+      preserve_word_confidence: false
+      exclude_blank: true
+      aggregation: "mean"       # mean, min, max, prod
+      method_cfg:
+        name: "entropy"         # max_prob or entropy
+        entropy_type: "tsallis" # gibbs, tsallis, renyi
+        alpha: 0.33
+        entropy_norm: "exp"     # lin or exp
+
+**Confidence methods:**
+
+* ``max_prob``: Maximum token probability as confidence. Simple and fast.
+* ``entropy``: Normalized entropy of the log-likelihood vector (default). Entropy types:
+
+  - ``gibbs``: Standard Gibbs entropy
+  - ``tsallis``: Tsallis entropy (default, recommended)
+  - ``renyi``: Renyi entropy
+
+**Aggregation** combines frame-level scores into token/word scores: ``mean``, ``min``, ``max``, or ``prod``.
+
+For TDT models, set ``tdt_include_duration_confidence: true`` to include duration prediction confidence.
+
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   asr_customization/ngpulm_language_modeling_and_customization
+   asr_customization/neural_rescoring
+   asr_customization/legacy_language_modeling_and_customization
+   asr_customization/ngram_utils
+   asr_customization/word_boosting

docs/source/asr/configs.rst

@@ -0,0 +1,925 @@
+NeMo ASR Configuration Files
+============================
+
+This section describes the NeMo configuration file setup that is specific to models in the ASR collection. For general information
+about how to set up and run experiments that is common to all NeMo models (e.g. Experiment Manager and PyTorch Lightning trainer
+parameters), see the :doc:`../core/core` section.
+
+The model section of the NeMo ASR configuration files generally requires information about the dataset(s) being used, the preprocessor
+for audio files, parameters for any augmentation being performed, as well as the model architecture specification. The sections on
+this page cover each of these in more detail.
+
+Example configuration files for all of the NeMo ASR scripts can be found in the
+`config directory of the examples <https://github.com/NVIDIA/NeMo/tree/stable/examples/asr/conf>`_.
+
+.. _asr-configs-dataset-configuration:
+
+Dataset Configuration
+---------------------
+
+Training, validation, and test parameters are specified using the ``train_ds``, ``validation_ds``, and
+``test_ds`` sections in the configuration file, respectively. Depending on the task, there may be arguments specifying the sample rate
+of the audio files, the vocabulary of the dataset (for character prediction), whether or not to shuffle the dataset, and so on. You may
+also decide to leave fields such as the ``manifest_filepath`` blank, to be specified via the command-line at runtime.
+
+Any initialization parameter that is accepted for the Dataset class used in the experiment can be set in the config file.
+Refer to the :ref:`Datasets <asr-api-datasets>` section of the API for a list of Datasets and their respective parameters.
+
+An example ASR train and validation configuration should look similar to the following:
+
+.. code-block:: yaml
+
+  # Specified at the beginning of the config file
+  labels: &labels [" ", "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m",
+           "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z", "'"]
+
+  model:
+    train_ds:
+      manifest_filepath: ???
+      sample_rate: 16000
+      labels: *labels   # Uses the labels above
+      batch_size: 32
+      trim_silence: True
+      max_duration: 16.7
+      shuffle: True
+      num_workers: 8
+      pin_memory: true
+      # tarred datasets
+      is_tarred: false # If set to true, uses the tarred version of the Dataset
+      tarred_audio_filepaths: null     # Not used if is_tarred is false
+      shuffle_n: 2048                  # Not used if is_tarred is false
+      # bucketing params
+      bucketing_strategy: "synced_randomized"
+      bucketing_batch_size: null
+      bucketing_weights: null
+
+    validation_ds:
+      manifest_filepath: ???
+      sample_rate: 16000
+      labels: *labels   # Uses the labels above
+      batch_size: 32
+      shuffle: False    # No need to shuffle the validation data
+      num_workers: 8
+      pin_memory: true
+
+There are two ways to test/validate on more than one manifest:
+
+- Specify a list in the `manifest_filepath` field. Results will be reported for each, the first one being used for overall loss / WER (specify `val_dl_idx` if you wish to change that). In this case, all manifests will share configuration parameters.
+- Use the ds_item key and pass a list of config objects to it. This allows you to use differently configured datasets for validation, e.g.
+
+.. code-block:: yaml
+
+  model:
+    validation_ds:
+      ds_item:
+      - name: dataset1
+        manifest_filepath: ???
+        # Config parameters for dataset1
+        ...
+      - name: dataset2
+        manifest_filepath: ???
+        # Config parameters for dataset2
+        ...
+
+By default, dataloaders are set up when the model is instantiated. However, dataloader setup can be deferred to
+model's `setup()` method by setting ``defer_setup`` in the configuration.
+
+For example, training data setup can be deferred as follows:
+
+.. code-block:: yaml
+
+  model:
+    train_ds:
+      # Configure training data as usual
+      ...
+      # Defer train dataloader setup from `__init__` to `setup`
+      defer_setup: true
+
+
+.. _asr-configs-metric-configuration:
+
+
+.. _asr-configs-preprocessor-configuration:
+
+Preprocessor Configuration
+--------------------------
+
+If you are loading audio files for your experiment, you will likely want to use a preprocessor to convert from the
+raw audio signal to features (e.g. mel-spectrogram or MFCC). The ``preprocessor`` section of the config specifies the audio
+preprocessor to be used via the ``_target_`` field, as well as any initialization parameters for that preprocessor.
+
+An example of specifying a preprocessor is as follows:
+
+.. code-block:: yaml
+
+  model:
+    ...
+    preprocessor:
+      # _target_ is the audio preprocessor module you want to use
+      _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
+      normalize: "per_feature"
+      window_size: 0.02
+      ...
+      # Other parameters for the preprocessor
+
+Refer to the :ref:`Audio Preprocessors <asr-audio-preprocessors>` API section for the preprocessor options, expected arguments,
+and defaults.
+
+.. _asr-configs-augmentation-configurations:
+
+Augmentation Configurations
+---------------------------
+
+There are a few on-the-fly spectrogram augmentation options for NeMo ASR, which can be specified by the
+configuration file using a ``spec_augment`` section.
+
+For example, there are options for `Cutout <https://arxiv.org/abs/1708.04552>`_ and
+`SpecAugment <https://arxiv.org/abs/1904.08779>`_ available via the ``SpectrogramAugmentation`` module.
+
+The following example sets up both ``Cutout`` (via the ``rect_*`` parameters) and ``SpecAugment`` (via the ``freq_*``
+and ``time_*`` parameters).
+
+.. code-block:: yaml
+
+  model:
+    ...
+    spec_augment:
+      _target_: nemo.collections.asr.modules.SpectrogramAugmentation
+      # Cutout parameters
+      rect_masks: 5   # Number of rectangles to cut from any given spectrogram
+      rect_freq: 50   # Max cut of size 50 along the frequency dimension
+      rect_time: 120  # Max cut of size 120 along the time dimension
+      # SpecAugment parameters
+      freq_masks: 2   # Cut two frequency bands
+      freq_width: 15  # ... of width 15 at maximum
+      time_masks: 5    # Cut out 10 time bands
+      time_width: 25  # ... of width 25 at maximum
+
+You can use any combination of ``Cutout``, frequency/time ``SpecAugment``, or neither of them.
+
+You can also add audio augmentation pipelines via an ``augmentor`` section in ``train_ds``.
+
+.. caution::
+   The ``augmentor`` pipeline is not supported by the Lhotse dataloader, which provides its own set of augmentation options.
+   See :doc:`Lhotse Dataloading </dataloaders>` for details.
+
+Augmentors are applied on-the-fly to audio data in the data layer. The following example
+adds white noise (probability 0.5, level between -50 dB and -10 dB) and room impulse response
+augmentation (probability 0.3, from a manifest of impulse responses):
+
+.. code-block:: yaml
+
+  model:
+    ...
+    train_ds:
+    ...
+        augmentor:
+            white_noise:
+                prob: 0.5
+                min_level: -50
+                max_level: -10
+            impulse:
+                prob: 0.3
+                manifest_path: /path/to/impulse_manifest.json
+
+Refer to the :ref:`Audio Augmentors <asr-api-audio-augmentors>` API section for more details.
+
+
+Metric Configurations
+---------------------
+
+NeMo ASR models supports WER and BLEU metric logging during training and validation. All metrics are based on the TorchMetrics backend, allowing for distributed training without additional code.
+
+Word Error Rate (WER)
+~~~~~~~~~~~~~~~~~~~~~
+
+WER is the default metric for all ASR models and measures transcription accuracy at the word or character level.
+
+.. code-block:: yaml
+
+  model:
+    use_cer: false                  # Set to true for Character Error Rate instead (default: false)
+    log_prediction: true            # Whether to log a sample prediction during training (default: true)
+    batch_dim_index: 0              # Index of batch dimension in prediction tensors output. Set to 1 for RNNT models.
+
+BLEU Score
+~~~~~~~~~~
+
+BLEU score can be used for ASR models to evaluate translation quality. NeMo's BLEU implementation is based on SacreBLEU for standardized, reproducible scoring:
+
+.. code-block:: yaml
+
+  model:
+    bleu_tokenizer: "13a"        # SacreBLEU tokenizer type (see below). (default: "13a")
+    n_gram: 4                    # Maximum n-gram order for BLEU calculation. (default: 4)
+    lowercase: false             # Whether to lowercase before computing BLEU. (default: False)
+    weights: null                # Optional custom weights for n-gram orders. (default: null)
+    smooth: false                # Whether to apply smoothing to BLEU calculation. (default: False)
+    check_cuts_for_bleu_tokenizers: false  # Enable per-sample tokenizer selection. (See below for more details.) (default: False)
+    log_prediction: true         # Whether to log sample predictions. (default: True)
+    batch_dim_index: 0           # Index of batch dimension in prediction tensors output. Set to 1 for RNNT models. (default: 0)
+
+BLEU score relies on TorchMetrics' SacreBLEU implementation and supports all SacreBLEU tokenization options. Valid strings may be passed to ``bleu_tokenizer`` parameter to configure base tokenizer behavior during BLEU calculation. Available options are:
+
+* ``"13a"`` - Default WMT tokenizer (mteval-v13a script compatible)
+* ``"none"`` - No tokenization applied
+* ``"intl"`` - International tokenization (mteval-v14 script compatible)  
+* ``"char"`` - Character-level tokenization (language-agnostic)
+* ``"zh"`` - Chinese tokenization (separates Chinese characters, uses 13a for non-Chinese)
+* ``"ja-mecab"`` - Japanese tokenization using MeCab morphological analyzer
+* ``"ko-mecab"`` - Korean tokenization using MeCab-ko morphological analyzer
+* ``"flores101"`` / ``"flores200"`` - SentencePiece models from Flores datasets
+
+**Note** Due to their unique orthographies, it is highly recommended to use ``zh``, ``ja-mecab``, or ``ko-mecab`` tokenizers for Chinese, Japanese, and Korean target evaluations, respectively. For more information on SacreBLEU tokenizers, please refer to the `SacreBLEU documentation <https://github.com/mjpost/sacrebleu>`__.
+
+**Dynamic Tokenizer Selection**
+
+In multilingual training scenarios, it is somtimes desireable to configure the BLEU tokenizer per sample to avoid sub-optimal parsing (e.g. tokenizing Chinese characters as English words). This can be toggled with ``check_cuts_for_bleu_tokenizers: true``. When enabled with Lhotse dataloading, BLEU will check individual ``cuts`` in a batch's Lhotse ``CutSet`` for the ``bleu_tokenizer`` attribute. If found, the tokenizer will be used for that sample. If not, the default ``bleu_tokenizer`` from config will be used.
+
+MultiTask Metrics
+~~~~~~~~~~~~~~~~~
+
+Multiple metrics can be configured simultaneously using a ``MultiTaskMetric`` config. This is done by specifying in the config each desired metric as a DictConfig entry with a custom key name and ``_target_`` path, along with desired properties. All properties specified within a metric config will be passed only to the metric class. All properties specified at the top level of the config will be inherited by all submetrics.
+
+.. code-block:: yaml
+
+  model:
+    multitask_metrics_config:
+      log_prediction: true
+      metrics:
+        wer:
+          _target_: nemo.collections.asr.metrics.wer.WER
+          use_cer: true
+          constraint: ".task==transcribe"  # Only apply WER to transcription samples
+        bleu:
+          _target_: nemo.collections.asr.metrics.bleu.BLEU
+          bleu_tokenizer: flores101
+          lowercase: true
+          check_cuts_for_bleu_tokenizers: true
+          constraint: ".task==translate"   # Only apply BLEU to translation samples
+
+**Metric Constraints**
+
+Each metric within ``MultiTaskMetric`` can be configured with an optional boolean ``constraint`` pattern that filters batch samples before metric computation. This allows validation to be limited to only applicable samples in a batch (e.g. only apply WER to transcription samples, only apply BLEU to translation samples). Constraint patterns match against property keywords in the batch's Lhotse CutSet.
+
+.. code-block:: yaml
+
+  model:
+    multitask_metrics_config:
+      metrics:
+        pnc_wer:
+          _target_: nemo.collections.asr.metrics.wer.WER
+          constraint: ".task==transcribe and .pnc==true"
+
+        multilingual_bleu:
+          _target_: nemo.collections.asr.metrics.bleu.BLEU
+          constraint: "(.source_lang!=.target_lang) or .task==translate"
+
+**Note:** MultiTaskMetric is currently only supported for AED multitask models.
+
+
+Tokenizer Configurations
+------------------------
+
+Some models utilize sub-word encoding via an external tokenizer instead of explicitly defining their vocabulary.
+
+For such models, a ``tokenizer`` section is added  to the model config. ASR models currently support two types of
+custom tokenizers:
+
+- Google Sentencepiece tokenizers (tokenizer type of ``bpe`` in the config)
+- HuggingFace WordPiece tokenizers (tokenizer type of ``wpe`` in the config)
+- Aggregate tokenizers ((tokenizer type of ``agg`` in the config), see below)
+
+In order to build custom tokenizers, refer to the ``ASR_with_Subword_Tokenization`` notebook available in the
+ASR tutorials directory.
+
+The following example sets up a ``SentencePiece Tokenizer`` at a path specified by the user:
+
+.. code-block:: yaml
+
+  model:
+    ...
+    tokenizer:
+      dir: "<path to the directory that contains the custom tokenizer files>"
+      type: "bpe"  # can be "bpe" or "wpe"
+
+The Aggregate (``agg``) tokenizer feature makes it possible to combine tokenizers in order to train multilingual
+models. The config file would look like this:
+
+.. code-block:: yaml
+
+  model:
+    ...
+    tokenizer:
+      type: "agg"  # aggregate tokenizer
+      langs:
+        en:
+          dir: "<path to the directory that contains the tokenizer files>"
+          type: "bpe"  # can be "bpe" or "wpe"
+        es:
+          dir: "<path to the directory that contains the tokenizer files>"
+          type: "bpe"  # can be "bpe" or "wpe"
+
+In the above config file, each language is associated with its own pre-trained tokenizer, which gets assigned
+a token id range in the order the tokenizers are listed. To train a multilingual model, one needs to populate the
+``lang`` field in the manifest file, allowing the routing of each sample to the correct tokenizer. At inference time,
+the routing is done based on the inferred token id range.
+
+For models which utilize sub-word tokenization, we share the decoder module (``ConvASRDecoder``) with character tokenization models.
+All parameters are shared, but for models which utilize sub-word encoding, there are minor differences when setting up the config. For
+such models, the tokenizer is utilized to fill in the missing information when the model is constructed automatically.
+
+For example, a decoder config corresponding to a sub-word tokenization model should look similar to the following:
+
+.. code-block:: yaml
+
+  model:
+    ...
+    decoder:
+      _target_: nemo.collections.asr.modules.ConvASRDecoder
+      feat_in: *enc_final
+      num_classes: -1  # filled with vocabulary size from tokenizer at runtime
+      vocabulary: []  # filled with vocabulary from tokenizer at runtime
+
+
+On-the-fly Code Switching
+-------------------------
+
+Nemo supports creating code-switched synthetic utterances on-the-fly during training/validation/testing. This allows you to create ASR models which
+support intra-utterance code switching. If you have Nemo formatted audio data on disk (either JSON manifests or tarred audio data), you
+can easily mix as many of these audio sources together as desired by adding some extra parameters to your `train_ds`, `validation_ds`, and `test_ds`.
+
+Please note that this allows you to mix any kind of audio sources together to create synthetic utterances which sample from all sources. The most
+common use case for this is blending different languages together to create a multilingual code-switched model, but you can also blend
+together different audio sources from the same languages (or language families), to create noise robust data, or mix fast and slow speech from the
+same language.
+
+For multilingual code-switched models, we recommend using AggTokenizer for your Tokenizer if mixing different languages.
+
+The following example shows how to mix 3 different languages: English (en), German (de), and Japanese (ja) added to the `train_ds` model block, however
+you can add similar logic to your `validation_ds` and `test_ds` blocks for on-the-fly code-switched validation and test data too. This example mixes
+together 3 languages, but you can use as many as you want. However, be advised that the more languages you add, the higher your `min_duration` and `max_duration`
+need to be set to ensure all languages are sampled into each synthetic utterance, and setting these hyperparameters higher will use more VRAM per mini-batch during
+training and evaluation.
+
+.. code-block:: yaml
+
+  model:
+    train_ds:
+      manifest_filepath: [/path/to/EN/tarred_manifest.json, /path/to/DE/tarred_manifest.json, /path/to/JA/tarred_manifest.json]
+      tarred_audio_filepaths: ['/path/to/EN/tars/audio__OP_0..511_CL_.tar', '/path/to/DE/tars/audio__OP_0..1023_CL_.tar', '/path/to/JA/tars/audio__OP_0..2047_CL_.tar']
+      is_code_switched: true
+      is_tarred: true
+      shuffle: true
+        code_switched:              # add this block for code-switching
+          min_duration: 12          # the minimum number of seconds for each synthetic code-switched utterance
+          max_duration: 20          # the maximum number of seconds for each synthetic code-switched utterance
+          min_monolingual: 0.3      # the minimum percentage of utterances which will be pure monolingual (0.3 = 30%)
+          probs: [0.25, 0.5, 0.25]  # the probability to sample each language (matches order of `language` above) if not provided, assumes uniform distribution
+          force_monochannel: true   # if your source data is multi-channel, then setting this to True will force the synthetic utterances to be mono-channel
+          sampling_scales: 0.75     # allows you to down/up sample individual languages. Can set this as an array for individual languages, or a scalar for all languages
+          seed: 123                 # add a seed for replicability in future runs (highly useful for `validation_ds` and `test_ds`)
+
+
+Model Architecture Configurations
+---------------------------------
+
+Each configuration file should describe the model architecture being used for the experiment. Models in the NeMo ASR collection need
+an ``encoder`` section and a ``decoder`` section, with the ``_target_`` field specifying the module to use for each.
+
+Here is the list of the parameters in the model section which are shared among most of the ASR models:
+
++-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
+| **Parameter**           | **Datatype**     | **Description**                                                                                               | **Supported Values**            |
++=========================+==================+===============================================================================================================+=================================+
+| :code:`log_prediction`  | bool             | Whether a random sample should be printed in the output at each step, along with its predicted transcript.    |                                 |
++-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
+| :code:`ctc_reduction`   | string           | Specifies the reduction type of CTC loss. Defaults to ``mean_batch`` which would take the average over the    | :code:`none`,                   |
+|                         |                  | batch after taking the average over the length of each sample.                                                | :code:`mean_batch`              |
+|                         |                  |                                                                                                               | :code:`mean`, :code:`sum`       |
++-------------------------+------------------+---------------------------------------------------------------------------------------------------------------+---------------------------------+
+
+For more information about the ASR models, refer to the :doc:`Featured Models <./featured_models>` section.
+
+
+.. _asr-configs-conformer-ctc:
+
+Conformer-CTC
+~~~~~~~~~~~~~
+
+The config files for Conformer-CTC model contain character-based encoding and sub-word encoding at
+``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_char.yaml`` and ``<NeMo_git_root>/examples/asr/conf/conformer/conformer_ctc_bpe.yaml``
+respectively. Some components of the configs of :ref:`Conformer-CTC <Conformer-CTC_model>` include the following datasets:
+
+* ``train_ds``, ``validation_ds``, and ``test_ds``
+* opimizer (``optim``)
+* augmentation (``spec_augment``)
+* ``decoder``
+* ``trainer``
+* ``exp_manager``
+
+There should be a tokenizer section where you can
+specify the tokenizer if you want to use sub-word encoding instead of character-based encoding.
+
+
+The encoder section includes the details about the Conformer-CTC encoder architecture. You may find more information in the
+config files and also :ref:`nemo.collections.asr.modules.ConformerEncoder <conformer-encoder-api>`.
+
+
+Conformer-Transducer
+~~~~~~~~~~~~~~~~~~~~
+
+Please refer to the model page of :ref:`Conformer-Transducer <Conformer-Transducer_model>` for more information on this model.
+
+
+Transducer Configurations
+-------------------------
+
+All CTC-based ASR model configs can be modified to support Transducer loss training. Below, we discuss the modifications required in the config to enable Transducer training. All modifications are made to the ``model`` config.
+
+Model Defaults
+~~~~~~~~~~~~~~
+
+It is a subsection to the model config representing the default values shared across the entire model represented as ``model.model_defaults``.
+
+There are three values that are primary components of a transducer model. They are :
+
+* ``enc_hidden``: The hidden dimension of the final layer of the Encoder network.
+* ``pred_hidden``: The hidden dimension of the final layer of the Prediction network.
+* ``joint_hidden``: The hidden dimension of the intermediate layer of the Joint network.
+
+One can access these values inside the config by using OmegaConf interpolation as follows :
+
+.. code-block:: yaml
+
+    model:
+      ...
+      model_defaults:
+        enc_hidden: 256
+        pred_hidden: 256
+        joint_hidden: 256
+      ...
+      decoder:
+        ...
+        prednet:
+          pred_hidden: ${model.model_defaults.pred_hidden}
+
+Acoustic Encoder Model
+~~~~~~~~~~~~~~~~~~~~~~
+
+The transducer model is comprised of three models combined. One of these models is the Acoustic (encoder) model. We should be able to drop in any CTC Acoustic model config into this section of the transducer config.
+
+The only condition that needs to be met is that **the final layer of the acoustic model must have the hidden dimension defined in ``model_defaults.enc_hidden``**.
+
+Decoder / Prediction Model
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Prediction model is generally an autoregressive, causal model that consumes text tokens and returns embeddings that will be used by the Joint model. The base config for an LSTM based Prediction network can be found in the ``decoder`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
+
+**This config can be copy-pasted into any custom transducer model with no modification.**
+
+Let us discuss some of the important arguments:
+
+* ``blank_as_pad``: In ordinary transducer models, the embedding matrix does not acknowledge the ``Transducer Blank`` token (similar to CTC Blank). However, this causes the autoregressive loop to be more complicated and less efficient. Instead, this flag which is set by default, will add the ``Transducer Blank`` token to the embedding matrix - and use it as a pad value (zeros tensor). This enables more efficient inference without harming training. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
+
+* ``prednet.pred_hidden``: The hidden dimension of the LSTM and the output dimension of the Prediction network.
+
+.. code-block:: yaml
+
+  decoder:
+    _target_: nemo.collections.asr.modules.RNNTDecoder
+    normalization_mode: null
+    random_state_sampling: false
+    blank_as_pad: true
+
+    prednet:
+      pred_hidden: ${model.model_defaults.pred_hidden}
+      pred_rnn_layers: 1
+      t_max: null
+      dropout: 0.0
+
+Joint Model
+~~~~~~~~~~~
+
+The Joint model is a simple feed-forward Multi-Layer Perceptron network. This MLP accepts the output of the Acoustic and Prediction models and computes a joint probability distribution over the entire vocabulary space. The base config for the Joint network can be found in the ``joint`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
+
+**This config can be copy-pasted into any custom transducer model with no modification.**
+
+The Joint model config has several essential components which we discuss below :
+
+* ``log_softmax``: Due to the cost of computing softmax on such large tensors, the Numba CUDA implementation of RNNT loss will implicitly compute the log softmax when called (so its inputs should be logits). The CPU version of the loss doesn't face such memory issues so it requires log-probabilities instead. Since the behaviour is different for CPU-GPU, the ``None`` value will automatically switch behaviour dependent on whether the input tensor is on a CPU or GPU device.
+
+* ``preserve_memory``: This flag will call ``torch.cuda.empty_cache()`` at certain critical sections when computing the Joint tensor. While this operation might allow us to preserve some memory, the empty_cache() operation is tremendously slow and will slow down training by an order of magnitude or more. It is available to use but not recommended.
+
+* ``fuse_loss_wer``: This flag performs "batch splitting" and then "fused loss + metric" calculation. It will be discussed in detail in the next tutorial that will train a Transducer model.
+
+* ``fused_batch_size``: When the above flag is set to True, the model will have two distinct "batch sizes". The batch size provided in the three data loader configs (``model.*_ds.batch_size``) will now be the ``Acoustic model`` batch size, whereas the ``fused_batch_size`` will be the batch size of the ``Prediction model``, the ``Joint model``, the ``transducer loss`` module and the ``decoding`` module.
+
+* ``jointnet.joint_hidden``: The hidden intermediate dimension of the joint network.
+
+.. code-block:: yaml
+
+  joint:
+    _target_: nemo.collections.asr.modules.RNNTJoint
+    log_softmax: null  # sets it according to cpu/gpu device
+
+    # fused mode
+    fuse_loss_wer: false
+    fused_batch_size: 16
+
+    jointnet:
+      joint_hidden: ${model.model_defaults.joint_hidden}
+      activation: "relu"
+      dropout: 0.0
+
+Sampled Softmax Joint Model
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are some situations where a large vocabulary with a Transducer model - such as for multilingual models with a large
+number of languages. In this setting, we need to consider the cost of memory of training Transducer networks which does
+not allow large vocabulary.
+
+For such cases, one can instead utilize the ``SampledRNNTJoint`` module instead of the usual ``RNNTJoint`` module, in order
+to compute the loss using a sampled subset of the vocabulary rather than the full vocabulary file.
+
+It adds only one additional parameter :
+
+* ``n_samples``: Specifies the minimum number of tokens to sample from the vocabulary space,
+  excluding the RNNT blank token. If a given value is larger than the entire vocabulary size,
+  then the full vocabulary will be used.
+
+The only difference in config required is to replace ``nemo.collections.asr.modules.RNNTJoint`` with ``nemo.collections.asr.modules.SampledRNNTJoint``
+
+.. code-block:: yaml
+
+  joint:
+    _target_: nemo.collections.asr.modules.SampledRNNTJoint
+    n_samples: 500
+    ...  # All other arguments from RNNTJoint can be used after this.
+
+
+Effect of Batch Splitting / Fused Batch step
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following information below explain why memory is an issue when training Transducer models and how NeMo tackles the issue with its Fused Batch step. The material can be read for a thorough understanding, otherwise, it can be skipped. You can also follow these steps in the "ASR_with_Transducers" tutorial.
+
+**Diving deeper into the memory costs of Transducer Joint**
+
+One of the significant limitations of Transducers is the exorbitant memory cost of computing the Joint module. The Joint module is comprised of two steps.
+
+1) Projecting the Acoustic and Transcription feature dimensions to some standard hidden dimension (specified by model.model_defaults.joint_hidden)
+
+2) Projecting this intermediate hidden dimension to the final vocabulary space to obtain the transcription.
+
+Take the following example.
+
+BS=32 ; T (after 2x stride) = 800, U (with character encoding) = 400-450 tokens, Vocabulary size V = 28 (26 alphabet chars, space and apostrophe). Let the hidden dimension of the Joint model be 640 (Most Google Transducer papers use hidden dimension of 640).
+
+* Memory (Hidden, gb) = 32 x 800 x 450 x 640 x 4 = 29.49 gigabytes (4 bytes per float).
+
+* Memory (Joint, gb) = 32 x 800 x 450 x 28 x 4 = 1.290 gigabytes (4 bytes per float)
+
+**NOTE**: This is just for the forward pass! We need to double this memory to store gradients! This much memory is also just for the Joint model **alone**. Far more memory is required for the Prediction model as well as the large Acoustic model itself and its gradients!
+
+Even with mixed precision, that's ~30 GB of GPU RAM for just 1 part of the network + its gradients.
+
+Effect of Fused Batch Step
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The fundamental problem is that the joint tensor grows in size when ``[T x U]`` grows in size. This growth in memory cost is due to many reasons - either by model construction (downsampling) or the choice of dataset preprocessing (character tokenization vs. sub-word tokenization).
+
+Another dimension that NeMo can control is **batch**. Due to how we batch our samples, small and large samples all get clumped together into a single batch. So even though the individual samples are not all as long as the maximum length of T and U in that batch, when a batch of such samples is constructed, it will consume a significant amount of memory for the sake of compute efficiency.
+
+So as is always the case - **trade-off compute speed for memory savings**.
+
+The fused operation goes as follows :
+
+1) Forward the entire acoustic model in a single pass. (Use global batch size here for acoustic model - found in ``model.*_ds.batch_size``)
+
+2) Split the Acoustic Model's logits by ``fused_batch_size`` and loop over these sub-batches.
+
+3) Construct a sub-batch of same ``fused_batch_size`` for the Prediction model. Now the target sequence length is U_sub-batch < U.
+
+4) Feed this U_sub-batch into the Joint model, along with a sub-batch from the Acoustic model (with T_sub-batch < T). Remember, we only have to slice off a part of the acoustic model here since we have the full batch of samples (B, T, D) from the acoustic model.
+
+5) Performing steps (3) and (4) yields T_sub-batch and U_sub-batch. Perform sub-batch joint step - costing an intermediate (B, T_sub-batch, U_sub-batch, V) in memory.
+
+6) Compute loss on sub-batch and preserve in a list to be later concatenated.
+
+7) Compute sub-batch metrics (such as Character / Word Error Rate) using the above Joint tensor and sub-batch of ground truth labels. Preserve the scores to be averaged across the entire batch later.
+
+8) Delete the sub-batch joint matrix (B, T_sub-batch, U_sub-batch, V). Only gradients from .backward() are preserved now in the computation graph.
+
+9) Repeat steps (3) - (8) until all sub-batches are consumed.
+
+10) Cleanup step. Compute full batch WER and log. Concatenate loss list and pass to PTL to compute the equivalent of the original (full batch) Joint step. Delete ancillary objects necessary for sub-batching.
+
+Transducer Decoding
+~~~~~~~~~~~~~~~~~~~
+
+Models which have been trained with CTC can transcribe text simply by performing a regular argmax over the output of their decoder. For transducer-based models, the three networks must operate in a synchronized manner in order to transcribe the acoustic features. The base config for the Transducer decoding step can be found in the ``decoding`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
+
+**This config can be copy-pasted into any custom transducer model with no modification.**
+
+The most important component at the top level is the ``strategy``. It can take one of many values:
+
+* ``greedy``: This is sample-level greedy decoding. It is generally exceptionally slow as each sample in the batch will be decoded independently. For publications, this should be used alongside batch size of 1 for exact results.
+
+* ``greedy_batch``: This is the general default and should nearly match the ``greedy`` decoding scores (if the acoustic features are not affected by feature mixing in batch mode). Even for small batch sizes, this strategy is significantly faster than ``greedy``.
+
+* ``beam``: Runs beam search with the implicit language model of the Prediction model. It will generally be quite slow, and might need some tuning of the beam size to get better transcriptions.
+
+* ``tsd``: Time synchronous decoding. Please refer to the paper: `Alignment-Length Synchronous Decoding for RNN Transducer <https://ieeexplore.ieee.org/document/9053040>`_ for details on the algorithm implemented. Time synchronous decoding (TSD) execution time grows by the factor T * max_symmetric_expansions. For longer sequences, T is greater and can therefore take a long time for beams to obtain good results. TSD also requires more memory to execute.
+
+* ``alsd``: Alignment-length synchronous decoding. Please refer to the paper: `Alignment-Length Synchronous Decoding for RNN Transducer <https://ieeexplore.ieee.org/document/9053040>`_ for details on the algorithm implemented. Alignment-length synchronous decoding (ALSD) execution time is faster than TSD, with a growth factor of T + U_max, where U_max is the maximum target length expected during execution. Generally, T + U_max < T * max_symmetric_expansions. However, ALSD beams are non-unique. Therefore it is required to use larger beam sizes to achieve the same (or close to the same) decoding accuracy as TSD. For a given decoding accuracy, it is possible to attain faster decoding via ALSD than TSD.
+
+* ``maes``: Modified Adaptive Expansion Search Decoding. Please refer to the paper `Accelerating RNN Transducer Inference via Adaptive Expansion Search <https://ieeexplore.ieee.org/document/9250505>`_. Modified Adaptive Synchronous Decoding (mAES) execution time is adaptive w.r.t the number of expansions (for tokens) required per timestep. The number of expansions can usually be constrained to 1 or 2, and in most cases 2 is sufficient. This beam search technique can possibly obtain superior WER while sacrificing some evaluation time.
+
+.. code-block:: yaml
+
+  decoding:
+    strategy: "greedy_batch"
+
+    # preserve decoding alignments
+    preserve_alignments: false
+
+    # Overrides the fused batch size after training.
+    # Setting it to -1 will process whole batch at once when combined with `greedy_batch` decoding strategy
+    fused_batch_size: -1
+
+    # greedy strategy config
+    greedy:
+      max_symbols: 10
+
+    # beam strategy config
+    beam:
+      beam_size: 2
+      score_norm: true
+      softmax_temperature: 1.0  # scale the logits by some temperature prior to softmax
+      tsd_max_sym_exp: 10  # for Time Synchronous Decoding, int > 0
+      alsd_max_target_len: 5.0  # for Alignment-Length Synchronous Decoding, float > 1.0
+      maes_num_steps: 2  # for modified Adaptive Expansion Search, int > 0
+      maes_prefix_alpha: 1  # for modified Adaptive Expansion Search, int > 0
+      maes_expansion_beta: 2  # for modified Adaptive Expansion Search, int >= 0
+      maes_expansion_gamma: 2.3  # for modified Adaptive Expansion Search, float >= 0
+
+Transducer Loss
+~~~~~~~~~~~~~~~
+
+This section configures the type of Transducer loss itself, along with possible sub-sections. By default, an optimized implementation of Transducer loss will be used which depends on Numba for CUDA acceleration. The base config for the Transducer loss section can be found in the ``loss`` section of Transducer architectures. For further information refer to the ``Intro to Transducers`` tutorial in the ASR tutorial section.
+
+**This config can be copy-pasted into any custom transducer model with no modification.**
+
+The loss config is based on a resolver pattern and can be used as follows:
+
+1) ``loss_name``: ``default`` is generally a good option. Will select one of the available resolved losses and match the kwargs from a sub-configs passed via explicit ``{loss_name}_kwargs`` sub-config.
+
+2) ``{loss_name}_kwargs``: This sub-config is passed to the resolved loss above and can be used to configure the resolved loss.
+
+
+.. code-block:: yaml
+
+  loss:
+    loss_name: "default"
+    warprnnt_numba_kwargs:
+      fastemit_lambda: 0.0
+
+FastEmit Regularization
+^^^^^^^^^^^^^^^^^^^^^^^
+
+FastEmit Regularization is supported for the default Numba based WarpRNNT loss. Recently proposed regularization approach - `FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization <https://arxiv.org/abs/2010.11148>`_ allows us near-direct control over the latency of transducer models.
+
+Refer to the above paper for results and recommendations of ``fastemit_lambda``.
+
+For decoding customization (confidence scores, CUDA graphs, language models, word boosting), see :doc:`ASR Language Modeling and Customization <./asr_language_modeling_and_customization>`.
+
+
+InterCTC Config
+---------------
+
+All CTC-based models also support `InterCTC loss <https://arxiv.org/abs/2102.03216>`_. To use it, you need to specify
+2 parameters as in example below
+
+.. code-block:: yaml
+
+   model:
+      # ...
+      interctc:
+        loss_weights: [0.3]
+        apply_at_layers: [8]
+
+which can be used to reproduce the default setup from the paper (assuming the total number of layers is 18).
+You can also specify multiple CTC losses from different layers, e.g., to get 2 losses from layers 3 and 8 with
+weights 0.1 and 0.3, specify:
+
+.. code-block:: yaml
+
+   model:
+      # ...
+      interctc:
+        loss_weights: [0.1, 0.3]
+        apply_at_layers: [3, 8]
+
+Note that the final-layer CTC loss weight is automatically computed to normalize
+all weight to 1 (0.6 in the example above).
+
+
+Stochastic Depth Config
+-----------------------
+
+`Stochastic Depth <https://arxiv.org/abs/2102.03216>`_ is a useful technique for regularizing ASR model training.
+Currently it's only supported for :ref:`nemo.collections.asr.modules.ConformerEncoder <conformer-encoder-api>`. To
+use it, specify the following parameters in the encoder config file to reproduce the default setup from the paper:
+
+.. code-block:: yaml
+
+   model:
+      # ...
+      encoder:
+        # ...
+        stochastic_depth_drop_prob: 0.3
+        stochastic_depth_mode: linear  # linear or uniform
+        stochastic_depth_start_layer: 1
+
+See :ref:`documentation of ConformerEncoder <conformer-encoder-api>` for more details. Note that stochastic depth
+is supported for both CTC and Transducer model variations (or any other kind of model/loss that's using
+conformer as encoder).
+
+
+.. _Hybrid-Transducer-CTC-Prompt_model__Config:
+
+Hybrid-Transducer-CTC with Prompt Conditioning Configuration
+------------------------------------------------------------
+
+The :ref:`Hybrid-Transducer-CTC model with prompt conditioning <Hybrid-Transducer-CTC-Prompt_model>` 
+(``EncDecHybridRNNTCTCBPEModelWithPrompt``) extends the base hybrid model to support prompt-based multilingual ASR/AST.
+
+**Key Configuration Parameters:**
+
+The model introduces several prompt-specific configuration parameters in the ``model_defaults`` section:
+
+.. code-block:: yaml
+
+  model:
+    model_defaults:
+      # Prompt Feature Configuration
+      initialize_prompt_feature: true  # Enable prompt conditioning
+      num_prompts: 128                 # Number of supported prompt categories
+      prompt_dictionary: {             # Mapping from identifiers to prompt indices
+        # Language prompts (0-99)
+        'en-US': 0,
+        'de-DE': 1,
+        'fr-FR': 2,
+        'es-ES': 3,
+        # Task/domain prompts (100-127)
+        'pnc': 100,                    # Punctuation mode
+        'no_pnc': 101,                 # No punctuation mode
+      }
+
+**Dataset Configuration:**
+
+The model requires training data with prompt annotations when using Lhotse datasets:
+
+.. code-block:: yaml
+
+  model:
+    train_ds:
+      use_lhotse: true
+      initialize_prompt_feature: true
+      prompt_field: "target_lang"     # Field name for prompt extraction
+      prompt_dictionary: ${model.model_defaults.prompt_dictionary}
+      num_prompts: ${model.model_defaults.num_prompts}
+      
+    validation_ds:
+      use_lhotse: true
+      initialize_prompt_feature: true
+      prompt_field: "target_lang"
+      prompt_dictionary: ${model.model_defaults.prompt_dictionary}
+      num_prompts: ${model.model_defaults.num_prompts}
+
+**Manifest Format:**
+
+Training manifests should include prompt information:
+
+.. code-block:: json
+
+  {
+    "audio_filepath": "/path/to/audio.wav",
+    "text": "transcription text",
+    "duration": 10.5,
+    "target_lang": "en-US"
+  }
+
+**Example Configuration:**
+
+A complete example configuration can be found at:
+``<NeMo_git_root>/examples/asr/conf/fastconformer/hybrid_transducer_ctc/fastconformer_hybrid_transducer_ctc_bpe_prompt.yaml``
+
+**Training Command:**
+
+.. code-block:: bash
+
+  python <NeMo_git_root>/examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe_prompt.py \
+      --config-path=<NeMo_git_root>/examples/asr/conf/fastconformer/hybrid_transducer_ctc/ \
+      --config-name=fastconformer_hybrid_transducer_ctc_bpe_prompt.yaml \
+      model.train_ds.manifest_filepath=<path_to_train_manifest> \
+      model.validation_ds.manifest_filepath=<path_to_val_manifest> \
+      model.tokenizer.dir=<path_to_tokenizer> \
+      model.test_ds.manifest_filepath=<path_to_test_manifest>
+
+
+.. _RNNT-Prompt_model__Config:
+
+RNN-T with Prompt Conditioning Configuration
+--------------------------------------------
+
+The :ref:`RNN-T model with prompt conditioning <RNNT-Prompt_model>`
+(``EncDecRNNTBPEModelWithPrompt``) is the RNN-T-only counterpart of the hybrid prompt model
+(no auxiliary CTC head). It targets cache-aware streaming multilingual ASR using the same
+one-hot language-ID prompt concatenation as the hybrid variant.
+
+**Key Configuration Parameters:**
+
+The prompt-specific parameters live in the ``model_defaults`` section, mirroring the hybrid
+variant:
+
+.. code-block:: yaml
+
+  model:
+    model_defaults:
+      # Prompt Feature Configuration
+      initialize_prompt_feature: true  # Enable prompt conditioning
+      num_prompts: 128                 # Number of supported prompt categories
+      prompt_dictionary: {             # Mapping from identifiers to prompt indices
+        'en-US': 0,
+        'de-DE': 1,
+        'fr-FR': 2,
+        'es-ES': 3,
+        # ... additional language codes ...
+        'auto': 127,                   # Per-sample dynamic language (read from manifest)
+      }
+
+**Dataset Configuration:**
+
+The model uses the same index-based Lhotse dataset
+(``LhotseSpeechToTextBpeDatasetWithPromptIndex``) as the hybrid model:
+
+.. code-block:: yaml
+
+  model:
+    train_ds:
+      use_lhotse: true
+      initialize_prompt_feature: true
+      prompt_field: "target_lang"     # Field name for per-sample prompt extraction
+      prompt_dictionary: ${model.model_defaults.prompt_dictionary}
+      num_prompts: ${model.model_defaults.num_prompts}
+
+    validation_ds:
+      use_lhotse: true
+      initialize_prompt_feature: true
+      prompt_field: "target_lang"
+      prompt_dictionary: ${model.model_defaults.prompt_dictionary}
+      num_prompts: ${model.model_defaults.num_prompts}
+
+**Manifest Format:**
+
+Identical to the hybrid model — each entry needs a ``target_lang`` field:
+
+.. code-block:: json
+
+  {
+    "audio_filepath": "/path/to/audio.wav",
+    "text": "transcription text",
+    "duration": 10.5,
+    "target_lang": "en-US"
+  }
+
+**Example Configuration:**
+
+A cache-aware streaming RNN-T prompt config ships at:
+``<NeMo_git_root>/examples/asr/conf/fastconformer/cache_aware_streaming/fastconformer_transducer_bpe_streaming_prompt.yaml``
+
+**Training Command:**
+
+.. code-block:: bash
+
+  python <NeMo_git_root>/examples/asr/asr_transducer/speech_to_text_rnnt_bpe_prompt.py \
+      --config-path=<NeMo_git_root>/examples/asr/conf/fastconformer/cache_aware_streaming/ \
+      --config-name=fastconformer_transducer_bpe_streaming_prompt.yaml \
+      model.train_ds.manifest_filepath=<path_to_train_manifest> \
+      model.validation_ds.manifest_filepath=<path_to_val_manifest> \
+      model.tokenizer.dir=<path_to_tokenizer> \
+      model.test_ds.manifest_filepath=<path_to_test_manifest>
+
+**Streaming Inference:**
+
+The standard cache-aware streaming inference script accepts ``target_lang`` (and the optional
+``strip_lang_tags`` / ``lang_tag_pattern`` flags) for prompt-conditioned models:
+
+.. code-block:: bash
+
+  python <NeMo_git_root>/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
+      model_path=<path_to_nemo_checkpoint> \
+      dataset_manifest=<path_to_manifest> \
+      target_lang=<en-US|auto|...> \
+      strip_lang_tags=true

docs/source/asr/data/asrlm_results.csv

@@ -0,0 +1,2 @@
+Model Name,Model Base Class,Model Card
+asrlm_en_transformer_large_ls,TransformerLMModel,"https://ngc.nvidia.com/catalog/models/nvidia:nemo:asrlm_en_transformer_large_ls"