wehub-resource-sync
828 lines
66 KiB
Markdown
828 lines
66 KiB
Markdown
# Voicebox Project Status & Roadmap
|
||
|
||
> Last updated: 2026-07-02 | Current version: **v0.5.0** | 402 open issues | 88 open PRs | 1.3M downloads · 34.8k stars
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [Architecture Overview](#architecture-overview)
|
||
2. [Current State](#current-state)
|
||
3. [Open PRs — Triage & Analysis](#open-prs--triage--analysis)
|
||
4. [Open Issues — Categorized](#open-issues--categorized)
|
||
5. [Existing Plan Documents — Status](#existing-plan-documents--status)
|
||
6. [New Model Integration — Landscape](#new-model-integration--landscape)
|
||
7. [Architectural Bottlenecks](#architectural-bottlenecks)
|
||
8. [Recommended Priorities](#recommended-priorities)
|
||
|
||
---
|
||
|
||
## Architecture Overview
|
||
|
||
**Tauri shell (Rust)** hosts a **React frontend** (`app/`) that talks over HTTP on `localhost:17493` to a **FastAPI backend** (`backend/`).
|
||
|
||
The backend exposes:
|
||
|
||
- **`TTSBackend` Protocol** with seven concrete engine implementations:
|
||
- Qwen3-TTS (PyTorch or MLX depending on platform)
|
||
- Qwen CustomVoice (predefined speakers with instruct)
|
||
- LuxTTS (fast, CPU-friendly)
|
||
- Chatterbox Multilingual (23 languages)
|
||
- Chatterbox Turbo (English, paralinguistic tags)
|
||
- TADA (1B English, 3B multilingual via HumeAI)
|
||
- Kokoro 82M (pre-built voices, CPU realtime)
|
||
- **`STTBackend` Protocol** for Whisper (PyTorch or MLX-Whisper)
|
||
- **Profiles / History / Stories** services for persistence and timeline editing
|
||
|
||
### Key Files
|
||
|
||
| Layer | File | Purpose |
|
||
|-------|------|---------|
|
||
| Backend entry | `backend/main.py` | FastAPI app, all API routes (~2850 lines) |
|
||
| TTS protocol | `backend/backends/__init__.py:32-101` | `TTSBackend` Protocol definition |
|
||
| Model registry | `backend/backends/__init__.py:17-29,153-366` | `ModelConfig` dataclass + registry helpers |
|
||
| TTS factory | `backend/backends/__init__.py:382-426` | Thread-safe engine registry (double-checked locking) |
|
||
| PyTorch TTS | `backend/backends/pytorch_backend.py` | Qwen3-TTS via `qwen_tts` package |
|
||
| MLX TTS | `backend/backends/mlx_backend.py` | Qwen3-TTS via `mlx_audio.tts` |
|
||
| LuxTTS | `backend/backends/luxtts_backend.py` | LuxTTS — fast, CPU-friendly |
|
||
| Chatterbox MTL | `backend/backends/chatterbox_backend.py` | Chatterbox Multilingual — 23 languages |
|
||
| Chatterbox Turbo | `backend/backends/chatterbox_turbo_backend.py` | Chatterbox Turbo — English, paralinguistic tags |
|
||
| TADA | `backend/backends/hume_backend.py` | HumeAI TADA — 1B English + 3B Multilingual |
|
||
| Kokoro | `backend/backends/kokoro_backend.py` | Kokoro 82M — CPU realtime, pre-built voices |
|
||
| Qwen CustomVoice | `backend/backends/qwen_custom_voice_backend.py` | Qwen CustomVoice — predefined speakers with instruct |
|
||
| Platform detect | `backend/platform_detect.py` | Apple Silicon → MLX, else → PyTorch |
|
||
| API types | `backend/models.py` | Pydantic request/response models |
|
||
| HF progress | `backend/utils/hf_progress.py` | HFProgressTracker (tqdm patching for download progress) |
|
||
| Audio utils | `backend/utils/audio.py` | `trim_tts_output()`, normalize, load/save audio |
|
||
| Frontend API | `app/src/lib/api/client.ts` | Hand-written fetch wrapper |
|
||
| Frontend types | `app/src/lib/api/types.ts` | TypeScript API types |
|
||
| Engine selector | `app/src/components/Generation/EngineModelSelector.tsx` | Shared engine/model dropdown |
|
||
| Generation form | `app/src/components/Generation/GenerationForm.tsx` | TTS generation UI |
|
||
| Floating gen box | `app/src/components/Generation/FloatingGenerateBox.tsx` | Compact generation UI |
|
||
| Model manager | `app/src/components/ServerSettings/ModelManagement.tsx` | Model download/status/progress UI |
|
||
| GPU acceleration | `app/src/components/ServerSettings/GpuAcceleration.tsx` | CUDA backend swap UI |
|
||
| Gen form hook | `app/src/lib/hooks/useGenerationForm.ts` | Form validation + submission |
|
||
| Language constants | `app/src/lib/constants/languages.ts` | Per-engine language maps |
|
||
|
||
### How TTS Generation Works (Current Flow)
|
||
|
||
```
|
||
POST /generate
|
||
1. Look up voice profile from DB
|
||
2. Resolve engine from request (qwen | qwen_custom_voice | luxtts | chatterbox | chatterbox_turbo | tada | kokoro)
|
||
3. Get backend: get_tts_backend_for_engine(engine) # thread-safe singleton per engine
|
||
4. Check model cache → if missing, trigger background download, return HTTP 202
|
||
5. Load model (lazy): tts_backend.load_model(model_size)
|
||
6. Create voice prompt: profiles.create_voice_prompt_for_profile(engine=engine)
|
||
→ tts_backend.create_voice_prompt(audio_path, reference_text)
|
||
7. Generate: tts_backend.generate(text, voice_prompt, language, seed, instruct)
|
||
8. Post-process: trim_tts_output() for Chatterbox engines
|
||
9. Save WAV → data/generations/{id}.wav
|
||
10. Insert history record in SQLite
|
||
11. Return GenerationResponse
|
||
```
|
||
|
||
---
|
||
|
||
## Current State
|
||
|
||
### Since v0.5.0 — Two-Month Pulse (2026-04-25 → 2026-06-27)
|
||
|
||
**The repo went quiet while demand kept climbing.** 0.5.0 (the Capture release) shipped 2026-04-25. In the two months since, only **2 PRs merged** (#544 the release itself, #550 a remote-URL fix) while **150 new issues** were opened and the open-PR queue more than tripled to **88**. The community kept contributing — translations, new engines, GPU fixes — but nothing's been reviewed or merged. This is a review-and-merge backlog, not a build backlog.
|
||
|
||
| Metric | At v0.4.1 (2026-04-18) | Now (2026-06-27) | Δ |
|
||
|--------|------------------------|------------------|---|
|
||
| Open issues | 232 | 402 | +170 |
|
||
| Open PRs | 12 | 88 | +76 |
|
||
| GitHub stars | ~28k | 34.8k | +~7k |
|
||
| Downloads | — | 1.3M | — |
|
||
|
||
**What the two months actually produced (all unmerged):**
|
||
- **A flood of community translations** — pt-BR, de-DE, Russian (+docs), Arabic+RTL, Spanish (+docs site), French, Cantonese. ~12 i18n PRs sitting on the i18next foundation that landed in 0.5.0.
|
||
- **GPU coverage PRs** — AMD ROCm on Windows (#538), Intel XPU (#539), DirectML for Intel iGPU (#674), Blackwell diagnostic (#653), MLX threading fix (#789).
|
||
- **A 17-PR hardening dump** from one contributor (@neuron-tech-ai, all 2026-05-14): CI pipeline, Biome, OpenAI-compatible `/v1/audio/speech`, SQLite WAL + indexes, LIKE-injection / upload-limit / N+1 fixes, platform gating. High value, entirely unreviewed.
|
||
- **New engine PRs** — MiniMax cloud, MOSS-TTS-Nano, Fun-CosyVoice3, Parakeet STT.
|
||
- **Two giant Linux PRs** — vendored `tao` patch for the Wayland startup panic (#748), and an SRT2Voice workflow (#673).
|
||
|
||
**0.5.0 regressions worth triaging first:**
|
||
- macOS Apple Silicon — all TTS models crash the server on load (#606, #615), MLX falls back to CPU on M4/M5 (#706, #650).
|
||
- Capture cutoffs at 30s for imported audio (#609, #626); paste broken in 0.5.0 (#762).
|
||
- MCP rough edges — dotted tool names violate Claude Desktop's name pattern (#790), audio scrambled over MCP (#780).
|
||
- Refinement silently translates non-English transcripts to English (#603).
|
||
|
||
**Funding model — `$VOICEBOX` token (#806):** the two-month gap was a solo-dev decision about long-term sustainability, not neglect or a compromise. `$VOICEBOX` (Solana) is the **official, dev-controlled** token and the chosen revenue path — donations/sponsors didn't cover full-time work. The app stays 100% free, open-source, local-first, no subscriptions. Dev supply is being bought back and burned (done twice), liquidity locked. It has funded ~2–3 months of full-time work, so the cadence resumes this week. The #806 thread was a community concern, addressed transparently and resolved amicably; keep an eye out for actual impersonator/community tokens, which are a separate thing.
|
||
|
||
**Other trust/security signals:** macOS malware-flag reports continue (#369); a DNS-rebinding / Host-header exposure on the local API+MCP server was reported with fixes attached (#778).
|
||
|
||
---
|
||
|
||
### What's Shipped (v0.5.0 — the Capture release)
|
||
|
||
Shipped 2026-04-25 (PR #544). Voicebox went from a voice-cloning studio to a full voice studio — dictation in, agent speech out, a local LLM in the middle.
|
||
|
||
- **Dictation** — global hotkey capture (push-to-talk + toggle chords), on-screen pill with live state, auto-paste into the focused field with clipboard save/restore, chord-picker UI. Scoped Accessibility permission (transcripts still land if paste is denied).
|
||
- **MCP server** at `http://127.0.0.1:17493/mcp` — `voicebox.speak` / `.transcribe` / `.list_captures` / `.list_profiles`. Streamable HTTP primary transport, stdio sidecar shim, per-client voice binding via `X-Voicebox-Client-Id`. Speaking pill always shows agent-initiated output.
|
||
- **Personality** — voice profiles carry an optional ≤2000-char persona. Compose (shuffle an in-character line) and Speak-in-character (rewrite input before TTS), both on a local Qwen3 LLM that doubles as the refinement model.
|
||
- **Refinement** — on-device Qwen3 strips fillers, fixes punctuation, optional self-correction rewrites; Whisper hallucination-loop stripping at a 6-token threshold; per-capture flag snapshots; model picker (0.6B / 1.7B / 4B).
|
||
- **`POST /speak` REST wrapper** and **i18next foundation** (English + zh-CN) also landed.
|
||
|
||
### What's Shipped (v0.4.x)
|
||
|
||
**New since v0.3.0:**
|
||
- Kokoro 82M TTS engine + voice profile type system (PR #325)
|
||
- Qwen CustomVoice preset engine — predefined speakers with instruct support (PR #328)
|
||
- Intel Arc (XPU) GPU support (PR #320)
|
||
- Blackwell GPU (sm_120) CUDA support (PR #401)
|
||
- Generation cancellation flow (PR #444)
|
||
- Frontend quality gates + TypeScript hardening (PR #418)
|
||
- macOS Intel (x86_64) PyTorch compatibility (PR #416)
|
||
- Frozen-binary import fixes for Kokoro / Chatterbox Multilingual / scipy / transformers (PR #438)
|
||
- Linux PipeWire/PulseAudio monitor detection (PR #457)
|
||
- Server survives GUI close on Windows (PR #402)
|
||
- GPU arch compatibility warning on startup (catches unsupported PyTorch builds)
|
||
- cpal Stream playback reliability (PR #405), clip-splitting stability (PR #403)
|
||
- torch.from_numpy crash with numpy 2.x in frozen binary (PR #361)
|
||
- Async CUDA download lock (PR #428), NUMBA_CACHE_DIR env var (PR #425)
|
||
- "Clear failed" history button (PR #412)
|
||
- External server GUI startup + data refresh (PR #319)
|
||
- Force offline mode for cached Qwen/Whisper models (PR #318)
|
||
- macOS 11 ScreenCaptureKit launch crash fix (PR #424)
|
||
|
||
**Core TTS (cumulative):**
|
||
- Qwen3-TTS voice cloning (1.7B and 0.6B models, MLX + PyTorch)
|
||
- Qwen CustomVoice (preset speakers, instruct)
|
||
- LuxTTS — fast, CPU-friendly English TTS (PR #254)
|
||
- Chatterbox Multilingual — 23 languages including Hebrew (PR #257)
|
||
- Chatterbox Turbo — paralinguistic tags, low latency English (PR #258)
|
||
- HumeAI TADA — 1B English + 3B Multilingual (PR #296)
|
||
- Kokoro 82M — CPU-realtime, 8 languages, Apache 2.0 (PR #325)
|
||
- Multi-engine architecture with thread-safe backend registry (PR #254)
|
||
- Chunked TTS generation — engine-agnostic, removes ~500 char limit (PR #266)
|
||
- Async generation queue (PR #269)
|
||
- Post-processing audio effects system (PR #271)
|
||
- Voice profile type system (preset vs cloned, engine compatibility gating)
|
||
- Centralized `ModelConfig` registry — no per-engine dispatch maps
|
||
- Shared `EngineModelSelector` component
|
||
|
||
**Infrastructure (cumulative):**
|
||
- CUDA backend swap via binary download (PR #252), cu128 upgrade (PR #316), Blackwell/sm_120 (PR #401)
|
||
- CUDA backend split into independently versioned server + libs archives (PR #298)
|
||
- Intel Arc XPU support (PR #320)
|
||
- Docker + web deployment (PR #161)
|
||
- Backend refactor: modular architecture, style guide, tooling (PR #285)
|
||
- Settings overhaul: routed sub-tabs, server logs, changelog, about page (PR #294)
|
||
- Windows support: CUDA detection, cross-platform justfile, server lifecycle (PR #272, #402)
|
||
- Linux audio capture via pactl monitor detection (PR #457)
|
||
- macOS Intel x86_64 compatibility (PR #416)
|
||
- Voice profiles with multi-sample support
|
||
- Stories editor (multi-track DAW timeline)
|
||
- Whisper transcription (base, small, medium, large, turbo variants)
|
||
- Model management UI with inline download progress + folder migration (PR #268)
|
||
- Download cancel/clear UI with error panel (PR #238)
|
||
- Generation history with caching and cancellation (PR #444)
|
||
- Streaming generation endpoint (MLX only)
|
||
- Audio player freeze fix + UX improvements (PR #293)
|
||
- CORS restriction to known local origins (PR #88)
|
||
|
||
### Abandoned / Backlogged Integrations
|
||
|
||
| Model | PR / Branch | Reason |
|
||
|-------|-------------|--------|
|
||
| **CosyVoice2/3** | PR #311 | Output quality too poor. Heavy deps, no PyPI, needed 5+ shims. PR should be closed. |
|
||
| **VoxCPM 1.5 / VoxCPM2** | `voicebox-new-models` research (2026-04-18) | **Backlogged.** See detailed analysis below. |
|
||
|
||
#### VoxCPM — Evaluation Notes (2026-04-18)
|
||
|
||
**Project:** [OpenBMB/VoxCPM](https://github.com/OpenBMB/VoxCPM) — tokenizer-free TTS, 2B params (VoxCPM2), end-to-end diffusion autoregressive architecture, 30 languages, 48 kHz output, Apache 2.0, `pip install voxcpm`.
|
||
|
||
**Why it looked interesting:**
|
||
- Clean PyPI install (`pip install voxcpm`)
|
||
- Apache 2.0 — commercially safe
|
||
- Voice cloning via `reference_wav_path` with optional `prompt_wav_path` + `prompt_text` for "ultimate" cloning
|
||
- Streaming API via `generate_streaming()`
|
||
- Zero-shot cloning + style control via parenthetical prefixes in text (`(slightly faster, cheerful tone)...`)
|
||
- Relatively high-quality output per demos
|
||
|
||
**Why we backlogged it:**
|
||
- **Effectively CUDA-only.** README states `CUDA ≥ 12.0` as hard requirement. Source code's `from_pretrained(device=None|"auto")` claims "preferring CUDA, then MPS, then CPU," but in practice:
|
||
- **MPS (Apple Silicon) broken upstream** — OpenBMB/VoxCPM issues #232 (`NotImplementedError: Output channels > 65536 not supported at the MPS device`) and #248 (`IndexError` on M3 Mac) are both open with no resolution.
|
||
- **CPU unsupported in the Python package** — issue #256 shows `voxcpm --device cpu` rejected with `unrecognized arguments`. The only CPU path is the third-party **VoxCPM.cpp** GGML engine, which is a separate ecosystem project, not `pip install voxcpm`.
|
||
- **macOS source install fails** — issue #233 open with no resolution.
|
||
- Would require CUDA-only gating in UI (new `requires_cuda` flag on `ModelConfig`, lock icon + "Requires NVIDIA GPU" in `ModelManagement.tsx` / `EngineModelSelector.tsx`) plus a hard error at `load_model()` as safety net. Doable but adds first-class platform gating that doesn't exist for any other engine today.
|
||
- Voicebox's user base skews Apple Silicon (MLX is a primary backend). Shipping a CUDA-only model sets a precedent worth a separate scoping discussion (see issues #419 engine sprawl, #420 platform tiers, PR #465).
|
||
|
||
**What would change the decision:**
|
||
- Upstream fixes MPS crashes (watch issues #232, #248).
|
||
- We define an "experimental / CUDA-only" engine tier as part of issue #419 / PR #465, and decide it's acceptable to ship engines that are hidden on non-NVIDIA platforms.
|
||
- VoxCPM.cpp matures into a viable CPU path we can wrap (currently separate project, C++/GGML, unclear ergonomics).
|
||
|
||
**Integration shape if we revive it:** Zero-shot cloning maps naturally to the Chatterbox-style backend (store `ref_audio` + `ref_text` paths in the voice prompt dict, process at generate time). Est. ~250 lines for `voxcpm_backend.py` + one `ModelConfig` entry + engine registration in `backends/__init__.py`. Frontend UI gating is the bigger lift.
|
||
|
||
### Funded Roadmap (2026-H2)
|
||
|
||
`$VOICEBOX` funded ~2–3 months of full-time work; cadence resumes the week of 2026-06-27. Direction committed publicly in #806:
|
||
|
||
| Item | Notes |
|
||
|------|-------|
|
||
| **Resume merge/release cadence** | Clear the 88-PR backlog, regular commits + releases — this is the immediate focus (see Tier 1) |
|
||
| **Mobile companion app** | New surface; already drawing issues (#773 iPhone logout) |
|
||
| **Encrypted cloud backup/sync** | For voice profiles + generations — first cloud feature; stays opt-in, local-first remains default |
|
||
| **More TTS models** | Engine candidates in the Landscape section below; community PRs #507/#766/#777 in queue |
|
||
| **Better GPU support** | Blackwell/sm_120, ROCm, DirectML, Intel — incl. paying testers for hardware the dev lacks |
|
||
| **Bug fixes** | 0.5.0 regression cluster first (macOS load crash, capture cutoffs, MCP, refinement) |
|
||
|
||
### What's In-Flight
|
||
|
||
| Feature | Branch/PR | Status |
|
||
|---------|-----------|--------|
|
||
| Platform support tiers | PR #465, issue #420 | Defining tier-1 (supported) vs tier-2 (community) platforms |
|
||
| Engine sprawl cleanup | issue #419 | First-class vs experimental TTS backends distinction |
|
||
| Frontend tech-debt burn-down | issue #421 | Biome + a11y debt before gating CI |
|
||
| Docker registry auto-publish | PR #463, issue #453 | ghcr.io image on tag push |
|
||
| New model research | `voicebox-new-models` branch | Evaluating Fish Speech, XTTS-v2, Pocket TTS, VibeVoice, Fish Audio S2, index-tts2. **2026-06-27 sweep** added dots.tts, LongCat-AudioDiT, SoproTTS, NeuTTS, Nemotron/Cohere STT — see Landscape → New Candidate Sweep |
|
||
|
||
### TTS Engine Comparison
|
||
|
||
| Engine | Model Name | Profile Type | Languages | Size | Key Features | Instruct Support |
|
||
|--------|-----------|--------------|-----------|------|-------------|-----------------|
|
||
| Qwen3-TTS 1.7B | `qwen-tts-1.7B` | Cloned | 10 (zh, en, ja, ko, de, fr, ru, pt, es, it) | ~3.5 GB | Highest quality, voice cloning | None (Base model has no instruct path) |
|
||
| Qwen3-TTS 0.6B | `qwen-tts-0.6B` | Cloned | 10 | ~1.2 GB | Lighter, faster | None |
|
||
| Qwen CustomVoice 1.7B | `qwen-custom-voice-1.7B` | Preset | 10 | ~3.5 GB | Predefined speakers, instruct support | **Yes** |
|
||
| Qwen CustomVoice 0.6B | `qwen-custom-voice-0.6B` | Preset | 10 | ~1.2 GB | Predefined speakers, instruct support | **Yes** |
|
||
| LuxTTS | `luxtts` | Cloned | English | ~300 MB | CPU-friendly, 48 kHz, fast | None |
|
||
| Chatterbox | `chatterbox-tts` | Cloned | 23 (incl. Hebrew, Arabic, Hindi, etc.) | ~3.2 GB | Zero-shot cloning, multilingual | Partial — `exaggeration` float (0-1) |
|
||
| Chatterbox Turbo | `chatterbox-turbo` | Cloned | English | ~1.5 GB | Paralinguistic tags ([laugh], [cough]), 350M params, low latency | Partial — inline tags only |
|
||
| TADA 1B | `tada-1b` | Cloned | English | ~4 GB | HumeAI speech-language model, 700s+ coherent audio | None |
|
||
| TADA 3B Multilingual | `tada-3b-ml` | Cloned | 10 (en, ar, zh, de, es, fr, it, ja, pl, pt) | ~8 GB | Multilingual, text-acoustic dual alignment | None |
|
||
| Kokoro 82M | `kokoro` | Preset | 8 (en, es, fr, hi, it, pt, ja, zh) | ~350 MB | 82M params, CPU realtime, Apache 2.0, pre-built voices | None |
|
||
|
||
### Multi-Engine Architecture (Shipped)
|
||
|
||
- **Thread-safe backend registry** (`_tts_backends` dict + `_tts_backends_lock`) with double-checked locking
|
||
- **Per-engine backend instances** — each engine gets its own singleton, loaded lazily
|
||
- **Engine field on GenerationRequest** — frontend sends `engine: 'qwen' | 'qwen_custom_voice' | 'luxtts' | 'chatterbox' | 'chatterbox_turbo' | 'tada' | 'kokoro'`
|
||
- **Per-engine language filtering** — `ENGINE_LANGUAGES` map in frontend, backend regex accepts all languages
|
||
- **Per-engine voice prompts** — `create_voice_prompt_for_profile()` dispatches to the correct backend
|
||
- **Profile type system** — preset vs cloned profiles, UI grays out incompatible engines and auto-switches on selection
|
||
- **Trim post-processing** — `trim_tts_output()` for Chatterbox engines (cuts trailing silence/hallucination)
|
||
|
||
### Known Limitations
|
||
|
||
- **HF XET progress**: Large files downloaded via `hf-xet` (HuggingFace's new transfer backend) report `n=0` in tqdm updates. Progress bars may appear stuck for large `.safetensors` files even though the download is proceeding. This is a known upstream limitation.
|
||
- **Chatterbox Turbo upstream token bug**: `from_pretrained()` passes `token=os.getenv("HF_TOKEN") or True` which fails without a stored HF token. Our backend works around this by calling `snapshot_download(token=None)` + `from_local()`.
|
||
- **chatterbox-tts must install with `--no-deps`**: It pins `numpy<1.26`, `torch==2.6.0`, `transformers==4.46.3` — all incompatible with our stack (Python 3.12, torch 2.10, transformers 4.57.3). Sub-deps listed explicitly in `requirements.txt`.
|
||
- **Instruct parameter partially shipped** (#224, #303): Qwen CustomVoice (PR #328) now provides real instruct support via predefined speakers. Other backends still silently drop the instruct field — the UI exposes the field broadly but most engines ignore it. The floating generate box was patched to restore instruct for CustomVoice (commit `106aec4`).
|
||
- **Streaming generation** only works for Qwen on MLX. Other engines use the non-streaming `/generate` endpoint.
|
||
- **dicta-onnx** (Hebrew diacritization) not included — upstream Chatterbox bug requires `model_path` arg but calls `Dicta()` with none. Hebrew works fine without it.
|
||
- **Blackwell (RTX 50-series) CUDA**: cu128 + sm_120 kernel support shipped (PR #401, #316), but users still report `cudaErrorNoKernelImageForDevice` (#417, #400, #396, #395, #390, #362) — likely a stale CUDA binary on upgraded installs. Needs a follow-up diagnostic / forced re-download path.
|
||
- **Long text 50k character limit** (#464, #365, #354): Still hit on GPU despite chunking (PR #266). Chunking reliability needs another pass.
|
||
- **ROCm on RDNA 3/4** (#469): `HSA_OVERRIDE_GFX_VERSION` is hardcoded and harms newer cards.
|
||
- **`flash-attn is not installed` warning on every platform (cosmetic, common user complaint)**: Our transformer-based engines (Chatterbox / Qwen) emit `Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.` on every startup, on every platform — we don't pin `flash-attn` in requirements because installing it is fragile and version-sensitive. Fallback is PyTorch SDPA, which is near-FA2 throughput on Ampere+ and is what actually runs. **Per-platform reality:** (a) **macOS/Apple Silicon** — FlashAttention is CUDA-only, irrelevant here; MLX has its own attention kernels. (b) **Linux** — `pip install flash-attn --no-build-isolation` works but takes 20+ min to compile. (c) **Windows** — no official support (Dao-AILab README still says only "Might work"; source builds routinely fail on recent CUDA/MSVC, issues #1715, #1828, #2395). Windows users can install community prebuilt wheels from `kingbri1/flash-attention` or `bdashore3/flash-attention` (latest v2.8.3, Aug 2025; `win_amd64` wheels for CUDA 12.4/12.8, Torch 2.6–2.9, Python 3.10–3.13) matching their exact CUDA/Torch/Python, or use WSL2. **Native-Windows alternatives worth considering as a build-time swap:** SageAttention (thu-ml, Apache 2.0, claims 2–5× over FA2) and xformers (official Windows wheels). **Action for us:** troubleshooting doc now covers it (see `docs/content/docs/overview/troubleshooting.mdx`), and we should optionally suppress the warning via `logging.getLogger(...).setLevel(ERROR)` at backend import since the fallback is functionally fine.
|
||
- **WebAudio playback dies after audio-session interruption** (#41, plus an internal repro where the app is backgrounded long enough): WaveSurfer's `AudioContext` gets suspended by macOS — either because another app grabs the audio output, or because the WKWebView throttles when backgrounded. `play()` resolves and `timeupdate` can still fire, but no audio reaches the output. Only app restart fixes it. **Things already tried that didn't work:** (a) swapping WaveSurfer backend away from WebAudio — introduced more bugs, not an option; (b) remount hook on the player — doesn't help because a freshly-created `AudioContext` is born suspended and only resumes on a user gesture. PR #293 was a prior partial fix that doesn't cover this path. **Next thing to try** (not yet attempted — confirmed via grep of `AudioPlayer.tsx`): call `wavesurfer.getMediaElement().getGainNode().context.resume()` on the play button click (the click itself is a valid user gesture), plus a `visibilitychange` + `statechange` listener as belt-and-suspenders. The `ctx.resume()` pattern already exists in the codebase at `useStoryPlayback.ts:52` — just not wired into the main player.
|
||
|
||
---
|
||
|
||
## Open PRs — Triage & Analysis
|
||
|
||
**88 open PRs, only 2 merged since 0.5.0.** The queue is the single biggest lever right now — a lot of finished community work is waiting on review. Clustered below by theme. Counts are approximate; a PR can span clusters.
|
||
|
||
### Merged since 0.5.0
|
||
|
||
| PR | Title | Merged |
|
||
|----|-------|--------|
|
||
| **#550** | Fix web API URL for remote access | 2026-04-25 |
|
||
| **#544** | feat: 0.5.0 Capture release — dictation, MCP, personalities | 2026-04-25 |
|
||
|
||
### i18n / translations (~12) — easy wins, unblock a large user segment
|
||
|
||
The i18next + zh-CN foundation shipped in 0.5.0; these stack on it. Triage as a batch.
|
||
|
||
| PR | Locale / scope |
|
||
|----|----------------|
|
||
| #528 | pt-BR translation |
|
||
| #571 | de-DE translation |
|
||
| #599 / #600 / #601 | Russian — app, landing routes, docs |
|
||
| #569 | Arabic + RTL layout fixes |
|
||
| #798 / #799 / #801 | Spanish — locale, README/CONTRIBUTING/SECURITY, docs site (see #800 for approach alignment) |
|
||
| #802 | French translation |
|
||
| #776 | Cantonese language option |
|
||
| #688 | Compose follows the selected language |
|
||
|
||
### New engines / models (~7)
|
||
|
||
| PR | Engine | Notes |
|
||
|----|--------|-------|
|
||
| #507 | MOSS-TTS-Nano (0.1B, 20 langs, CPU realtime) | Matches our cross-platform criteria — top engine candidate |
|
||
| #331 / #430 | MiniMax Cloud TTS | Two PRs, same provider — **dedupe**. External-API direction. |
|
||
| #777 | Fun-CosyVoice3 (draft) | We abandoned CosyVoice2/3 once on quality — re-evaluate output before reviving |
|
||
| #766 | Parakeet as STT model | Whisper alternative |
|
||
| #563 | 4-bit quantized Qwen + Russian abbreviations | Smaller/faster Qwen |
|
||
| #225 | Custom HuggingFace voice models | Long-lived; needs rework for multi-engine arch |
|
||
| #195 | Per-profile LoRA fine-tuning (draft, +6.2k) | Complex, 15 endpoints — addresses #185/#224 demand |
|
||
|
||
### GPU / hardware (~9)
|
||
|
||
| PR | Scope |
|
||
|----|-------|
|
||
| #538 | Native AMD ROCm on Windows (+2.8k, resolves #531) |
|
||
| #539 | Optional IPEX + native Intel XPU detection for PyTorch 2.9+ |
|
||
| #674 | DirectML for Intel iGPU (Iris/UHD/Arc) — pairs with demand in #676, #759 |
|
||
| #653 | Blackwell GPU arch-mismatch diagnostic — directly targets the sm_120 cluster |
|
||
| #789 | Run MLX load+inference on one thread (fixes #699) — likely fixes the M-series load crashes |
|
||
| #560 / #561 | Linux NVIDIA auto-detect + Linux CUDA backend build |
|
||
| #736 / #785 / #769 | ROCm `HSA_OVERRIDE_GFX_VERSION` cleanup (resolves #469) |
|
||
| #770 | Fix CUDA downloads on unsupported platforms |
|
||
|
||
### Capture / transcription / refinement (~6) — fixes 0.5.0 regressions
|
||
|
||
| PR | Fix |
|
||
|----|-----|
|
||
| #602 | Re-encode uploaded audio as PCM WAV before Whisper — likely fixes the 30s import cutoff (#609/#626) |
|
||
| #616 | Enable long-form Whisper on the PyTorch path |
|
||
| #629 | Preserve source language in refinement (fixes #603 English translation) |
|
||
| #637 | MCP/REST generations incorrectly trigger autoplay |
|
||
| #712 | Personality LLM respects the selected refinement model |
|
||
| #796 | Capture preview placement setting (#698) |
|
||
|
||
### Long-form / stories / streaming (~5)
|
||
|
||
| PR | Scope |
|
||
|----|-------|
|
||
| #154 | Audiobook tab with chunked generation (predates shipped chunking — reconcile) |
|
||
| #673 | SRT2Voice workflow (+14k) — large, subtitle-driven generation |
|
||
| #787 | m4b/mp3 story export with auto chapter markers |
|
||
| #642 | `stream=true` immediate-audio mode for `GET /tts` |
|
||
| #804 | Stream MLX TTS audio chunks (draft) |
|
||
|
||
### Linux / Wayland (~5)
|
||
|
||
| PR | Scope |
|
||
|----|-------|
|
||
| #748 | Vendor-patch tao 0.34.8 for Wayland startup panic (+39k — large vendored diff, verify approach) |
|
||
| #624 | Linux tauri schema + build deps (+6.2k) |
|
||
| #747 | Avoid abort when hiding the dictate pill |
|
||
| #622 / #768 | Linux audio monitor selection / thread-unsafe `PULSE_SOURCE` |
|
||
| #677 | HF cache permissions + startup fallback |
|
||
|
||
### Hardening / CI / perf — the @neuron-tech-ai batch (all 2026-05-14, ~17 PRs)
|
||
|
||
One contributor opened a large, coherent quality suite in a single day. Review as a group; #662 is the headline.
|
||
|
||
| PR | Scope |
|
||
|----|-------|
|
||
| #662 | LIKE injection, upload-size enforcement, N+1 queries, SSE reconnect, memory leaks, model display names (+6.8k) |
|
||
| #654 | CI pipeline + pre-commit hooks + Biome config + test suite |
|
||
| #656 | OpenAI-compatible `/v1/audio/speech` + `/v1/models` (addresses #10) |
|
||
| #657 | Platform gating on `ModelConfig` + UI (addresses bottleneck #6 / issue #419) |
|
||
| #666 / #667 | DB indexes on hot FKs + SQLite WAL + busy timeout |
|
||
| #659 / #660 / #661 / #663 / #665 / #668 | datetime.utcnow→UTC, MediaRecorder crash + sample reorder, fail-fast on missing model, batch story counts, Metal warmup, drop debug logs |
|
||
| #652 / #655 / #658 / #664 | AGENTS.md, docs GitHub Pages, avatar size limit, non-fatal actool |
|
||
|
||
### Build / dev tooling / docker (~6)
|
||
|
||
#764 uv for backend env · #632 docker GPU build + cache + fastmcp (+7.9k) · #630 ROCm docker overlay · #463 ghcr.io auto-publish · #543 / #681 setup-script fixes · #584 docker permission fix
|
||
|
||
### Smaller fixes worth grabbing
|
||
|
||
#786 remove 50k char limit (#464) · #621 broken-pipe crashes on model load · #743 harden mac generation status + MLX threading · #788 missing male Mandarin Kokoro voices · #794 build mcp shim on Windows · #527 Chatterbox exaggeration + CFG sliders · #253 48kHz speech tokenizer
|
||
|
||
### Stale / low-signal — close or request changes
|
||
|
||
#91 (draft, Feb, CoreAudio, +6.2k unrebased) · #649 ("fix this errors") · #623 / #782 (badges / package tweaks) · #311-style abandoned engines — verify before merging anything older than ~April against the 0.5.0 codebase.
|
||
|
||
---
|
||
|
||
## Open Issues — Categorized
|
||
|
||
**402 open, +150 in the two months since 0.5.0.** Demand snapshot from a keyword sweep over all open titles (buckets overlap):
|
||
|
||
| Theme | ~Open | Signal |
|
||
|-------|-------|--------|
|
||
| New model / engine requests | ~79 | Largest category. Voxtral, OmniVoice, VibeVoice, VoxCPM2, CosyVoice3, Dramabox, Parakeet, GGUF, ONNX/Piper export |
|
||
| CUDA / GPU / Blackwell | ~53 | Still the #1 *bug* driver — sm_120 "no kernel image", ROCm, DirectML, Intel Arc, VRAM/load times |
|
||
| Model download / server startup | ~42 | Stuck downloads, "server process ended unexpectedly", `loading_model` hangs |
|
||
| Capture / dictation / transcribe | ~31 | New surface from 0.5.0 — 30s cutoffs, paste, mic permission, refinement translation |
|
||
| Language / locale requests | ~27 | Bengali, Ukrainian, Filipino, Indonesian, Cantonese, zh-TW; plus UI localization |
|
||
| Fine-tune / clone quality | ~22 | #185 (top-engagement issue), accent leakage, "finetunes not working" |
|
||
| Long-form / chunking / export | ~16 | Pause control, speed control, audiobook export, >50k chars |
|
||
| Linux / Wayland | ~11 | Build failures, Wayland panics, CUDA-on-Linux packaging |
|
||
| MCP / agent / API | ~9 | Dotted tool names (#790), scrambled audio (#780), OpenAI compat (#10) |
|
||
| Security / trust | ~4 | DNS-rebinding (#778), malware flag (#369); funding via official $VOICEBOX token (#806) |
|
||
|
||
**Highest-engagement open issues:** #185 Fine-tune instructions (32c) · #98 Connecting to Download (16c) · #301 CUDA generation failure (18c) · #20 Model download failed (13c) · #364 Voxtral-TTS FR (11r) · #341 Arch Linux build · #513 server startup failed (12c) · #138 ONNX/Piper export (9r) · #10 OpenAI API compat.
|
||
|
||
### New since 0.5.0 — clusters to triage first
|
||
|
||
- **macOS Apple Silicon load crashes (regression):** #606, #615 — all TTS models crash the server on load; #706, #650 — MLX falls back to CPU / 7-min VRAM load on M4/M5. PR #789 (single-thread MLX, fixes #699) and #743 are the candidate fixes. **Highest priority — breaks the primary platform.**
|
||
- **Capture cutoffs & paste:** #609, #626 — transcription stops at 30s for imported audio (PR #602 re-encodes WAV); #762 — paste broken in 0.5.0; #698, #577 — capture/output folder locations.
|
||
- **MCP integration:** #790 — dotted tool names violate Claude Desktop's `^[a-zA-Z0-9_-]{1,64}$`; #780 — audio scrambled over MCP; #728 — CUDA re-downloads on cold start.
|
||
- **Refinement:** #603 — silently translates non-English transcripts to English (PR #629 preserves source language).
|
||
- **GPU expansion requests:** #676 DirectML (AMD/Intel), #759 Intel Arc, #684 RTX 5060 Ti CUDA 13, #774 CUDA 11.x for older cards, #767 Linux CUDA installs Windows `.exe`.
|
||
- **New engines/langs:** #791 OmniVoice, #633 VoxCPM2, #690 Dramabox, #638 Bengali, #754 zh-TW, #761 Filipino.
|
||
- **Open plugin interface (#771):** request for a community engine/provider plugin API — ties into engine-sprawl (#419) and platform-gating work.
|
||
- **Trust/security:** #806 — `$VOICEBOX` is the official dev-backed funding token (concern raised and resolved on-thread; see funding note above); #778 DNS-rebinding/Host-header exposure on local API+MCP (fixes attached) — genuine security item; #369 macOS malware flag (ongoing).
|
||
|
||
### GPU / Hardware Detection — still the top category
|
||
|
||
**RTX 50-series (Blackwell / sm_120) cluster — NEW:** #417, #400, #396, #395, #390, #362 all report `cudaErrorNoKernelImageForDevice` / "no kernel image available." sm_120 support shipped in PR #401 + cu128 in PR #316, but users on upgraded installs still hit it — likely stale CUDA binary. Needs a diagnostic that detects binary/GPU-arch mismatch and prompts re-download.
|
||
|
||
**AMD / ROCm — NEW:** #469 `HSA_OVERRIDE_GFX_VERSION` is hardcoded and breaks RDNA 3/4 cards. #313 DirectML on AMD Ryzen AI Max+ 395 not working.
|
||
|
||
**Intel Arc:** PR #320 shipped XPU support — may resolve #119.
|
||
|
||
**General GPU-not-detected (older):** #368, #310, #330, #324, #326, #355 (multi-GPU / eGPU).
|
||
|
||
**Fix path:** CUDA backend swap (PR #252) + cu128 (PR #316) + sm_120 (PR #401) + GPU-arch warning (`73170d0`) are all in. Remaining work is diagnostics + re-download prompts for users whose binary predates the kernel updates.
|
||
|
||
### Model Downloads
|
||
|
||
Still reported. Users get stuck downloads, can't resume, offline mode edge cases.
|
||
|
||
**Key issues:** #475 (MAC CustomVoice install error), #449 (infinite loading macOS), #445 (can't download CustomVoice), #462 (Qwen requires internet even when loaded — regression from #150), #434 (infinite retry loop offline — PR #443 open), #432 (storage location change hangs when empty — partly fixed by PR #439/#433), #348 (TADA 3B Multilingual download fails), #336 (TADA model not listed in app), #275 (`No module named 'chatterbox'` on download), #304 (whisper-base feature extractor load error), #287 (macOS ARM `check_model_inputs` ImportError on new version), #181, #180.
|
||
|
||
**Fix path:** PR #443 addresses infinite offline retry. CustomVoice-specific download failures (#475, #445) need triage — likely related to frozen-binary import fixes in PR #438. TADA cluster (#336, #348) and macOS ARM import regressions (#287, #275, #304) need a dedicated triage pass.
|
||
|
||
**Qwen 0.6B-downloads-1.7B reports:** **#485** (2026-04-19), **#423** (macOS M1), **#329**. Originally a stale-fallback bug: `mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16` wasn't published when MLX support shipped, so the 0.6B slot was aliased to the 1.7B repo. The 0.6B bf16 conversion is live now and both `backend/backends/mlx_backend.py` and `backend/backends/__init__.py` point at their correct repos. Qwen CustomVoice is unaffected — it runs via PyTorch on all platforms, both sizes always have dedicated repos.
|
||
|
||
### Language Requests (ongoing)
|
||
|
||
Strong demand: Hungarian (#479), Indonesian (#458, #247), Thai (#455), Bangla (#454), Arabic (#379), Persian (#162), IndicF5 (#339 — Indian languages), Ukrainian (#109), Chinese UI (#392, #261).
|
||
|
||
**Fix path:** Chatterbox Multilingual (PR #257) covers Arabic, Danish, German, Greek, Finnish, Hebrew, Hindi, Dutch, Norwegian, Polish, Swedish, Swahili, Turkish. Still missing: Hungarian, Indonesian, Thai, Bangla, Ukrainian. Issue #411 offers a PR for UI i18n foundation.
|
||
|
||
### New Model Requests (growing)
|
||
|
||
| Issue | Model Requested |
|
||
|-------|----------------|
|
||
| #478 | CosyVoice3 (we tried & abandoned CosyVoice2/3 — see #311) |
|
||
| #407, #347 | RVC-style voice-to-voice / seed voice conversion (STS) |
|
||
| #385 | Fish Audio S2 |
|
||
| #380 | OmniVoice |
|
||
| #370 | index-tts2 |
|
||
| #364 | Voxtral-TTS |
|
||
| #335 | Faster-Qwen-TTS |
|
||
| #346 | Multi-model batch request |
|
||
| #381 | Microsoft MAI models |
|
||
| #339 | IndicF5 |
|
||
| #226 | GGUF support |
|
||
| #172 | VibeVoice |
|
||
| #138 | Export to ONNX/Piper format |
|
||
| #132 | LavaSR (transcription) |
|
||
| #147 | Facebook Omnilingual ASR |
|
||
| #338 | Default voices |
|
||
|
||
The multi-engine architecture makes integration straightforward — see [`content/docs/developer/tts-engines.mdx`](content/docs/developer/tts-engines.mdx). Platform-specific gating (e.g. VoxCPM CUDA-only) doesn't exist yet and would need design.
|
||
|
||
### Platform Scope & Quality Debt — NEW category
|
||
|
||
Awareness issues filed this cycle — ties into engine sprawl and platform tier work.
|
||
|
||
- **#419** — Engine sprawl: define first-class vs experimental TTS backends
|
||
- **#420** — Formalize tier-1 vs tier-2 platform support targets (PR #465 open)
|
||
- **#421** — Track & burn down frontend Biome + a11y debt before gating CI
|
||
- **#422** — Code-split web build (main bundle > 1 MB)
|
||
|
||
### Long-Form / Chunking
|
||
|
||
Still reported despite chunking + queue being merged.
|
||
|
||
**Key issues:** #464 (50k char limit on GPU despite 16 GB VRAM — v0.4.0), #365 (FR: >50k chars), #363 (smart chunking to prevent robotic artifacts), #354 (50k limit v0.3.0).
|
||
|
||
**Fix path:** Chunking (#266) and queue (#269) shipped. Remaining work is raising/removing the 50k guard and tuning chunk boundaries for prosody.
|
||
|
||
### Feature Requests (ongoing)
|
||
|
||
Notable:
|
||
- **#480** — Noise removal on uploaded recordings
|
||
- **#448** — API for non-Qwen models (external integrations)
|
||
- **#427** — Task status control
|
||
- **#407, #347** — Voice-to-voice / audio-to-audio conversion
|
||
- **#387** — Location of downloaded generated voices
|
||
- **#383** — Concatenate partial reference audio into generated audio
|
||
- **#382** — Lightning.ai support
|
||
- **#376** — Remote mode
|
||
- **#353** — Audio transcoding
|
||
- **#317** — Voice pitch control
|
||
- **#189** — "Auto" language option
|
||
- **#173** — Vocal intonation/inflection control
|
||
- **#165, #270** — Audiobook mode (PR #154 open)
|
||
- **#242** — Seed value pinning
|
||
- **#228** — Always use 0.6B option
|
||
- **#235** — Finetuned Qwen3-TTS tokenizer (PR #253 open)
|
||
- **#144** — Copy text to clipboard
|
||
|
||
### Housekeeping / Triage Needed
|
||
|
||
| Issue | Reason |
|
||
|-------|--------|
|
||
| **#431**, **#408** | Spam — Chinese "free Claude API" promos. Close. |
|
||
| **#398** ("Excelente") | Non-issue. Close. |
|
||
| **#357** | Informational — project featured in Awesome MLX. Close after acknowledgement. |
|
||
| **#374**, **#377** | Version-release questions, no bug. Close. |
|
||
| **#306** ("voice model"), **#389** ("New model"), **#473** ("New functionality") | Title-only issues, no content. Request details or close. |
|
||
| **#309** | Uninstall/cleanup question. Answer and close. |
|
||
| **#241** | "How to use in Colab" — support question, not a bug. |
|
||
| **#423** / **#485** / **#329** | Stale MLX fallback to 1.7B repo — fixed; 0.6B bf16 conversion now live on `mlx-community`, registry points at correct repo on both backends. |
|
||
| **#336** / **#348** | TADA download/registration cluster — triage together. |
|
||
| **#287** / **#275** / **#304** | macOS ARM import regressions on new version — likely one root cause. |
|
||
| **#292**, **#349** | Possibly already fixed by merged PRs (#321/#412 and #345). Verify + close. |
|
||
|
||
**~70 older issues (pre-#170) not individually categorized above.** Most are long-tail support questions or duplicates of problems now addressed by the multi-engine / model-registry work. A dedicated backlog-sweep pass is overdue.
|
||
|
||
### Bugs (ongoing)
|
||
|
||
| Category | Issues |
|
||
|----------|--------|
|
||
| Generation failures | #476, #467, #452, #459 (voice clone fetch error), #468 (tada-1b marked error), #437, #300, #301, #282 |
|
||
| Audio quality | #456 (clipping errors v0.4.0), #436 (emotion labels), #333 (pitch/echo), #307 (by-model breakdown), #340 (all generations say "www...") |
|
||
| Transcription | #371 (fails every time), #291 (extract transcription from generated audio) |
|
||
| Effects / presets | #349 ("Failed to save" when creating effects presets — possibly fixed by merged #345) |
|
||
| File ops | #477 (spacy_pkuseg dict missing on frozen Windows build), #472 (storage location change), #283 (allow longer files for voice creation + in-app trim), #350 (failed to add sample) |
|
||
| History | #292 (can't delete failed generations — possibly fixed by merged #321/#412) |
|
||
| Windows | #466 (install problem), #375 (WinError 5 access denied), #273 (port 8000 conflict), #201 (model doesn't stay loaded) |
|
||
| Linux | #471 (thread-safe PULSE_SOURCE), #413 (Arch build), #409 (Kubuntu build), #351, #341 |
|
||
| macOS | #441 (older macOS), #369 (malware flag), #334 (microphone permission), #287 (`check_model_inputs` ImportError — regression), #171 (ARM64 binary won't open) |
|
||
| Profile/UI | #360 (Kokoro profile hides others — partly addressed by auto-switch), #299 (drag-drop on Win11), #329 (size selector state bug), #393 (stuck loading screen after reinstall to new dir) |
|
||
| Integrations | #397 (SAMMI-bot 422 Unprocessable Entity) |
|
||
| Audio playback / session | **#41** (macOS: Voicebox goes silent after another app takes audio output; restart restores it) — see deep-dive below |
|
||
| Database | #174 (sqlite3 IntegrityError) |
|
||
|
||
---
|
||
|
||
## Existing Plan Documents — Status
|
||
|
||
| Document | Target Version | Status | Relevance |
|
||
|----------|---------------|--------|-----------|
|
||
| `TTS_PROVIDER_ARCHITECTURE.md` | v0.1.13 | **Partially superseded** by multi-engine arch + CUDA swap | Core concepts implemented differently than planned |
|
||
| `CUDA_BACKEND_SWAP.md` | — | **Shipped** (PR #252) | CUDA binary download + backend restart |
|
||
| `CUDA_BACKEND_SWAP_FINAL.md` | — | **Shipped** (PR #252) | Final implementation plan |
|
||
| `EXTERNAL_PROVIDERS.md` | v0.2.0 | **Not started** | Remote server support |
|
||
| `MLX_AUDIO.md` | — | **Shipped** | MLX backend is live |
|
||
| `DOCKER_DEPLOYMENT.md` | v0.2.0 | **Shipped** (PR #161) | Docker + web deployment |
|
||
| `OPENAI_SUPPORT.md` | v0.2.0 | **Not started** | OpenAI-compatible API layer |
|
||
| `PR33_CUDA_PROVIDER_REVIEW.md` | — | **Reference** | Analysis of the original provider approach |
|
||
|
||
---
|
||
|
||
## New Model Integration — Landscape
|
||
|
||
### Status Snapshot (2026-04-18)
|
||
|
||
| Model | Cloning | Speed | Sample Rate | Languages | VRAM | Instruct | Cross-platform? | Status |
|
||
|-------|---------|-------|-------------|-----------|------|----------|-----------------|--------|
|
||
| **Qwen3-TTS** | 10s zero-shot | Medium | 24 kHz | 10 | Medium | None | MLX + PyTorch | **Shipped** |
|
||
| **Qwen CustomVoice** | Preset speakers | Medium | 24 kHz | 10 | Medium | **Yes** | PyTorch | **Shipped** (PR #328) |
|
||
| **LuxTTS** | 3s zero-shot | 150x RT, CPU ok | 48 kHz | English | <1 GB | None | All | **Shipped** (PR #254) |
|
||
| **Chatterbox MTL** | 5s zero-shot | Medium | 24 kHz | 23 | Medium | Partial — `exaggeration` | CPU/CUDA | **Shipped** (PR #257) |
|
||
| **Chatterbox Turbo** | 5s zero-shot | Fast | 24 kHz | English | Low | Partial — inline tags | CPU/CUDA | **Shipped** (PR #258) |
|
||
| **HumeAI TADA 1B/3B** | Zero-shot | 5x faster than LLM-TTS | 24 kHz | EN (1B), 10 (3B) | Medium | Partial — prosody | PyTorch | **Shipped** (PR #296) |
|
||
| **Kokoro-82M** | Preset voices | CPU realtime | 24 kHz | 8 | Tiny (82M) | None | All | **Shipped** (PR #325) |
|
||
| ~~**CosyVoice2-0.5B**~~ | 3-10s zero-shot | Very fast | 24 kHz | Multilingual | Low | **Yes** | — | **Abandoned** (PR #311) — poor output quality |
|
||
| ~~**VoxCPM2**~~ | Zero-shot | ~0.15 RTF streaming | 48 kHz | 30 | Medium | Partial — parenthetical style | **CUDA-only in practice** | **Backlogged** (2026-04-18) — see notes above |
|
||
| **Fish Speech** | 10-30s few-shot | Real-time | 24-44 kHz | 50+ | Medium | **Yes** — word-level inline | All | Candidate — license TBD |
|
||
| **Fish Audio S2** | — | — | — | — | — | — | — | Candidate (#385) |
|
||
| **XTTS-v2** | 6s zero-shot | Mid-GPU | 24 kHz | 17+ | Medium | Partial — style transfer from ref | All | Candidate — CPML license likely blocker |
|
||
| **Pocket TTS** (Kyutai) | Zero-shot + streaming | >1x RT on CPU | — | English + several European (FR/DE/PT/IT/ES added by Feb 2026) | ~100M | None | CPU-first | Candidate — MIT |
|
||
| **MOSS-TTS-Nano** | Zero-shot | **Realtime on 4 CPU cores** | 48 kHz stereo | 20 | 0.1B | Partial — MOSS-VoiceGenerator companion does text-to-voice design | All (ONNX CPU path dropped 2026-04-17) | **Top candidate** — Apache 2.0, released 2026-04-13, streaming |
|
||
| **VibeVoice** (Microsoft) | — | — | — | Multi-speaker long-form (up to 90 min, 4 speakers) | 1.5B | — | — | Candidate (#172) — Stories-editor fit |
|
||
| **index-tts2** | — | — | — | — | — | — | — | Candidate (#370) |
|
||
| **Voxtral TTS** (Mistral) | Zero-shot (short clips) + 20 preset voices | Single-GPU | — | — | 4B (`Voxtral-4B-TTS-2603`) | Presets + cloning | CUDA (16 GB+ VRAM) | Candidate (#364) — frontier quality claim, open-weight |
|
||
| **Dia / Dia2** | — | — | — | — | — | — | — | Watch — emotion-forward, but "rough edges" / artifacts per April reviews |
|
||
| **IndicF5** | — | — | — | Indian languages | — | — | — | Candidate (#339) — fills Indic gap |
|
||
| **MiniMax Cloud TTS** | — | Cloud | — | — | N/A (API) | — | N/A | Community PR #430, #331 — new direction (external API) |
|
||
| **OmniVoice** | — | — | — | — | — | — | — | Candidate (#380) |
|
||
| **RVC voice conversion** | N/A (STS) | — | — | — | — | N/A | All | New modality, not TTS (#407, #347) |
|
||
|
||
**Watch list:** MioTTS-2.6B (fast LLM-based EN/JP, vLLM compatible), Oolel-Voices (Soynade Research, expressive modular control), Faster-Qwen-TTS (#335), Orpheus / Sesame CSM (on-device fine-tuning discussions), Fish Audio S2 Pro / Fish Speech V1.5 (benchmark leader but research/non-commercial license — same blocker as Fish Speech).
|
||
|
||
**Deep-research pass (2026-04-18):** MOSS-TTS-Nano identified as the freshest high-alignment candidate — verified via [OpenMOSS/MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS) README (0.1B params, Apache 2.0, 48 kHz stereo, 4-core CPU realtime, streaming, released 2026-04-13). Dedicated repo: [OpenMOSS/MOSS-TTS-Nano](https://github.com/OpenMOSS/MOSS-TTS-Nano). Voxtral TTS verified on HF as `mistralai/Voxtral-4B-TTS-2603`.
|
||
|
||
#### Active Evaluation Criteria (learned from cycle)
|
||
|
||
1. **Cross-platform first.** MLX is a primary backend for our Apple Silicon user base. CUDA-only models require platform gating that doesn't exist yet — shipping one sets a precedent (see VoxCPM notes, issues #419/#420).
|
||
2. **PyPI + Apache/MIT licensing preferred.** Heavy deps, git-only installs, and `--no-deps` workarounds are expensive to maintain (Chatterbox taught us this).
|
||
3. **Output quality is non-negotiable.** CosyVoice was abandoned despite the best instruct API.
|
||
4. **Instruct support fills a real gap** (#173, #224, #303). Qwen CustomVoice partially addresses it with preset speakers; zero-shot clone-with-instruct is still unmet.
|
||
5. **Long-form + streaming are user-requested** (#363, #365, #464). Candidates with native streaming (Pocket TTS, Fish Speech) get extra weight.
|
||
|
||
### New Candidate Sweep (2026-06-27)
|
||
|
||
A follow-up deep-research pass, filtered against everything already tracked — the shipped engines plus MOSS-TTS-Nano, Pocket TTS, IndicF5, VibeVoice, Voxtral, Fish/Fish Audio, XTTS-v2, index-tts2, VoxCPM2, OmniVoice, MioTTS, Oolel, Faster-Qwen, Orpheus/Sesame, MiniMax, RVC, Parakeet, Qwen3-ASR, Moshi, GLM-4-Voice, Qwen2.5-Omni — kept only where a **newer sibling/variant** changes the evaluation. Same criteria as the 04-18 cycle: cross-platform, PyPI/clean packaging, permissive license, quality, instruct/style control, long-form, streaming.
|
||
|
||
**Top new TTS candidates**
|
||
|
||
| Candidate | Add as | Why it matters | Caveat |
|
||
|-----------|--------|----------------|--------|
|
||
| **[dots.tts](https://github.com/rednote-hilab/dots.tts)** (soar / mf) | **Top new TTS candidate** | 2B fully-continuous end-to-end autoregressive TTS, 48 kHz AudioVAE output, zero-shot cloning via prompt audio/text, Apache-2.0 code+checkpoints, MeanFlow-distilled variant for low latency. Freshest "serious clone engine" not yet on the roadmap. | Git-source install with constraints, not clean PyPI. Needs Windows/macOS packaging + VRAM/CPU smoke test; probably experimental until platform gating exists. |
|
||
| **[MOSS-TTS family](https://github.com/OpenMOSS/MOSS-TTS)** / v1.5 / Local-Transformer-v1.5 | **Upgrade the MOSS-Nano entry into a MOSS family epic** | We track only Nano, but MOSS now spans MOSS-TTS, TTSD (long multi-speaker dialogue), VoiceGenerator (text-prompt voice design), TTS-Realtime, SoundEffect. v1.5 adds broader languages, long-reference cloning, pause control, 48 kHz stereo, MLX/vLLM support, Apache-2.0. | Full 4B/8B variants aren't the lightweight Nano win. Treat as several engines/features, not one checkbox. |
|
||
| **[LongCat-AudioDiT](https://arxiv.org/html/2603.29339v1)** | **High-priority Apple Silicon candidate** | 3.5B non-autoregressive diffusion TTS in waveform latent space, zero-shot cloning, already has an MLX conversion usable via `mlx_audio` — unusually aligned with our Apple Silicon base. | zh/en only, not realtime. Quality play, not low-latency agent speech. |
|
||
| **[SoproTTS](https://github.com/samuel-vitorino/sopro)** | **Lightweight CPU/streaming cloned TTS** | 135M zero-shot cloning, `pip install -U sopro`, streaming + non-streaming APIs, 3–12s reference, claimed 250 ms TTFA / 0.05 RTF on M3 CPU. Strong local-first/low-maintenance fit. | English-focused, self-described as inconsistent — quality-test before promoting past experimental. |
|
||
| **[NeuTTS Air / Nano](https://github.com/neuphonic/neutts)** | **GGUF/on-device cloned TTS** | On-device instant cloning, GGUF-ready, ~3s reference, laptop/phone/Pi targets. Air is Apache-2.0. | Needs a GGUF/llama.cpp-style wrapper, not a normal PyTorch backend. Nano has a separate NeuTTS Open License — split needs review. |
|
||
| **[X-Voice](https://github.com/sunnyxrxrx/X-Voice)** | **Small multilingual clone** | 0.4B multilingual zero-shot cloning, 30 languages, IPA-style unified rep, claims no prompt-transcript requirement — targets a real cloning-UX pain point. | Verify license, packaging, production-readiness of weights/code. |
|
||
| **[FireRedTTS-2](https://huggingface.co/FireRedTeam/FireRedTTS2)** | **Stories / podcast / multi-speaker** | Apache-2.0 long-form streaming, 3-min / 4-speaker dialogue, cross-lingual code-switching cloning, low first-packet latency. | Stories-editor engine more than a general default. Needs platform/VRAM testing. |
|
||
| **[Maya1](https://huggingface.co/maya-research/maya1)** | **Expressive English voice-design** | 3B Apache-2.0, voice design, streaming, emotion/style tags, vLLM-compatible, 24 kHz, single-GPU. Good "voice personalities" / game-dialogue fit. | English-only, 16 GB+ VRAM — platform gating required. |
|
||
|
||
**MOSS is now a family, not one checkbox.** The single `MOSS-TTS-Nano` row above should become an epic: keep Nano as the CPU-friendly model, and track v1.5 / Local-Transformer-v1.5, Realtime, TTSD, VoiceGenerator, and SoundEffect as siblings under it.
|
||
|
||
**STT / capture candidates** (feed the planned streaming-transcription roadmap)
|
||
|
||
| Candidate | Add as | Why it matters | Caveat |
|
||
|-----------|--------|----------------|--------|
|
||
| **[Nemotron 3.5 ASR Streaming 0.6B](https://huggingface.co/mlx-community/nemotron-3.5-asr-streaming-0.6b)** | **Top new STT candidate** | Cache-aware streaming FastConformer-RNNT, 40 language-locales, punctuation/caps, language-ID conditioning, MLX conversion path — strongest fit for planned streaming transcription. | NVIDIA-origin; verify license + non-CUDA (MLX/CPU) performance. |
|
||
| **[Cohere Transcribe 03-2026](https://huggingface.co/blog/CohereLabs/cohere-transcribe-03-2026-release)** | **High-quality offline STT** | 2B Apache-2.0, 14 languages, ONNX/INT8 exports across CPU / Apple Silicon / GPU. Cleanest-looking offline `/transcribe` + captures candidate. | Less clearly a streaming dictation model than Nemotron. |
|
||
| **[ARK-ASR 3B / 0.6B](https://huggingface.co/AutoArk-AI/ARK-ASR-3B)** | **Multilingual STT watch** | New family, broad European/Asian coverage, strong leaderboard claims, INT8 ONNX for edge. | Very new; likely `trust_remote_code`. Validate stability first. |
|
||
| **[IBM Granite Speech 4.1 2B / NAR](https://huggingface.co/ibm-granite/granite-speech-4.1-2b)** | **ASR + speech translation** | Compact multilingual ASR + bidirectional speech translation (en/fr/de/es/pt/ja); NAR variant for latency-sensitive work. | More compelling if we expand into translation, not just dictation. |
|
||
|
||
**Watch-list / blocked** (license or platform work must land first): LEMAS-TTS, Supertonic 3, KugelAudio, GLM-TTS, KittenTTS, TinyTTS (preset/on-device, not cloning); Sarashina2.2, Higgs Audio v3, T5Gemma-TTS, Step-Audio-EditX, MisoTTS (non-commercial terms or CUDA-heavy); MegaTTS3 (incomplete WaveVAE encoder distribution); PFluxTTS, LongCat-Next (paper-only / too broad). **Low-hanging Qwen-family variants:** `Qwen3-TTS-VoiceDesign` (fills text-to-voice-design with minimal churn) and ZipVoice/ZipVoice-Dialog (only if it brings zh-en/dialogue behavior our shipped LuxTTS doesn't already expose).
|
||
|
||
**Roadmap patch from this sweep** (reflected in Tier 3 below):
|
||
1. Replace the `MOSS-TTS-Nano` checkbox with a **MOSS-TTS family** epic (Nano tracked separately as the CPU model).
|
||
2. New Tier-3 TTS candidates, in order: **dots.tts → LongCat-AudioDiT → SoproTTS → NeuTTS → X-Voice → FireRedTTS-2 → Maya1**.
|
||
3. New STT expansion candidates, in order: **Nemotron 3.5 → Cohere Transcribe → ARK-ASR → Granite Speech**.
|
||
4. Keep Sarashina2.2, Higgs v3, T5Gemma, Step-Audio-EditX, MisoTTS, MegaTTS3, PFluxTTS blocked/watch-only.
|
||
5. **Do platform gating (bottleneck #6 / `ModelConfig.requires`) before shipping GPU-only engines** — Maya1, Step-Audio-EditX, MisoTTS, and probably dots.tts stay experimental until it exists.
|
||
|
||
### Adding a New Engine (Now Straightforward)
|
||
|
||
With the model config registry and shared `EngineModelSelector` component, adding a new TTS engine requires:
|
||
|
||
1. **Create `backend/backends/<engine>_backend.py`** — implement `TTSBackend` protocol (~200-300 lines)
|
||
2. **Register in `backend/backends/__init__.py`** — add `ModelConfig` entry + `TTS_ENGINES` entry + factory elif
|
||
3. **Update `backend/models.py`** — add engine name to regex
|
||
4. **Update frontend** — add to engine union type, `EngineModelSelector` options, form schema, language map, profile type gating (icons/labels ~9 files per grep of `kokoro`)
|
||
|
||
`main.py` requires **zero changes** — the registry handles all dispatch automatically.
|
||
|
||
**Platform gating doesn't exist yet.** If we add a CUDA-only model (e.g. VoxCPM), we need a new `requires_cuda` (or more generally `requires: list[device]`) flag on `ModelConfig`, plumbed through `/models` API and surfaced in `ModelManagement.tsx` and `EngineModelSelector.tsx` as a lock icon + "Requires NVIDIA GPU" state. Backend should hard-error at `load_model()` as a safety net.
|
||
|
||
Total effort: **~1 day** for a well-documented model with a PyPI package, cross-platform. **~2 days** if platform gating is required. See [`content/docs/developer/tts-engines.mdx`](content/docs/developer/tts-engines.mdx) for the full guide.
|
||
|
||
---
|
||
|
||
## Architectural Bottlenecks
|
||
|
||
### ~~1. Single Backend Singleton~~ — RESOLVED
|
||
|
||
The singleton TTS backend was replaced with a thread-safe per-engine registry in PR #254. Multiple engines can now be loaded simultaneously.
|
||
|
||
### ~~2. `main.py` Dispatch Point Duplication~~ — RESOLVED
|
||
|
||
Previously, each engine required updates to 6+ hardcoded dispatch maps across `main.py` (~320 lines of if/elif chains). A model config registry in `backend/backends/__init__.py` now centralizes all model metadata (`ModelConfig` dataclass) with helper functions (`load_engine_model()`, `check_model_loaded()`, `engine_needs_trim()`, etc.). Adding a new engine requires zero changes to `main.py`.
|
||
|
||
### ~~3. Model Config is Scattered~~ — RESOLVED
|
||
|
||
Model identifiers, HF repo IDs, display names, and engine metadata are now consolidated in the `ModelConfig` registry. Backend-aware branching (e.g. MLX vs PyTorch Qwen repo IDs) happens inside the registry. Frontend model options are centralized in `EngineModelSelector.tsx`.
|
||
|
||
### 4. Voice Prompt Cache Assumes PyTorch Tensors
|
||
|
||
`backend/utils/cache.py` uses `torch.save()` / `torch.load()`. LuxTTS, Chatterbox, and Kokoro backends work around this by storing reference audio paths (or preset voice IDs) instead of tensors in their voice prompt dicts. Not ideal but functional.
|
||
|
||
### 5. ~~Frontend Assumes Qwen Model Sizes~~ — RESOLVED
|
||
|
||
The generation form now uses a flat model dropdown with engine-based routing. Per-engine language filtering is in place. Model size is only sent for Qwen / Qwen CustomVoice.
|
||
|
||
### 6. No Platform Gating on Models — NEW
|
||
|
||
`ModelConfig` has no way to express hardware requirements. Every engine is shown to every user, regardless of whether it'll actually load. Users on non-CUDA platforms discover failure at load time (or not at all — some fall back silently to CPU and never complete). Blocks shipping CUDA-only engines (VoxCPM) and would improve the Intel Arc / ROCm / CPU-only UX today. See `ModelConfig` TODO: add `requires: list[Literal["cuda", "mps", "xpu", "cpu", "rocm"]]` or equivalent, plumb through `/models` API, render in `ModelManagement.tsx` + `EngineModelSelector.tsx`.
|
||
|
||
### 7. Engine Sprawl — NEW
|
||
|
||
Seven TTS engines shipped, more candidates queued. Issue #419 asks for a first-class vs experimental distinction. Related: issue #420 asks for formalized platform support tiers. Combined, these would let us ship more engines more confidently with clearer expectations for users.
|
||
|
||
---
|
||
|
||
## Recommended Priorities
|
||
|
||
### Tier 1 — Ship Now (the next release is mostly a merge-and-fix pass)
|
||
|
||
The two-month gap means the highest-leverage work isn't new code — it's reviewing the 88-PR queue and shipping the 0.5.0 regression fixes that are already written.
|
||
|
||
| Priority | PR/Item | Impact | Effort |
|
||
|----------|---------|--------|--------|
|
||
| 1 | **macOS Apple Silicon load crash** (#606, #615, #706, #650) — review/merge PR #789 (single-thread MLX) + #743 | Breaks the primary platform on 0.5.0 | Low (PRs exist) |
|
||
| 2 | **Capture 30s import cutoff** (#609, #626) — review PR #602; paste-broken #762 | Core 0.5.0 feature degraded | Low–Medium |
|
||
| 3 | **Refinement translates to English** (#603) — merge PR #629 | Silent data loss for non-English users | Low |
|
||
| 4 | **MCP dotted tool names** (#790) — breaks Claude Desktop; scrambled audio #780 | Flagship integration broken for some clients | Low–Medium |
|
||
| 5 | **Blackwell / sm_120 diagnostic** — review PR #653; stale-binary re-download path | Largest GPU bug cluster | Medium |
|
||
| 6 | **Drain the i18n batch** (#528, #571, #599–601, #569, #798–801, #802, #776) | ~12 finished PRs, large user segment | Low (review-bound) |
|
||
| 7 | **Review the @neuron-tech-ai hardening batch** — start with #662, #657 (platform gating), #656 (OpenAI API), #654 (CI) | Security + perf + bottleneck #6 in one sweep | Medium (review-bound) |
|
||
| 8 | **Remove 50k char limit** (#464) — merge PR #786; tune chunk boundaries | Long-standing regression | Low |
|
||
| 9 | Housekeeping — dedupe MiniMax #331/#430, re-evaluate CosyVoice #777, close spam/empty issues (#805, #775) | Triage hygiene | Low |
|
||
|
||
### Tier 2 — Feature Work
|
||
|
||
| Priority | Item | Impact | Effort |
|
||
|----------|------|--------|--------|
|
||
| 1 | **Engine tier system** (#419) — first-class vs experimental, platform gating in `ModelConfig` | Unblocks CUDA-only engines (VoxCPM, etc.) and frontend polish | Medium |
|
||
| 2 | **Frontend tech-debt burn-down** (#421) + code-split (#422) | Before gating CI on Biome | Medium |
|
||
| 3 | **#154** — Audiobook tab | Long-form users. Chunking + queue shipped. | Medium |
|
||
| 4 | **UI i18n** (#411 PR offer, #392, #261) | Chinese UI + general localization | Medium |
|
||
| 5 | **#225** — Custom HuggingFace models | User-supplied models. Needs rework. | High |
|
||
| 6 | OpenAI-compatible API (plan doc exists) — see also #448 (API for non-Qwen) | Low effort once API is stable | Low |
|
||
| 7 | LoRA fine-tuning (PR #195) | Complex, needs rework for multi-engine | Very High |
|
||
| 8 | Streaming for non-MLX engines | Currently MLX-only | Medium |
|
||
| 9 | Voice-to-voice / RVC (#407, #347) | New modality — different arch shape | High |
|
||
|
||
### Tier 3 — Future Engines (cross-platform preferred)
|
||
|
||
Committed ordering (04-18 cycle), then the 2026-06-27 sweep additions. See Landscape → New Candidate Sweep for full rationale.
|
||
|
||
| Priority | Item | Notes |
|
||
|----------|------|-------|
|
||
| 1 | **MOSS-TTS family** (was MOSS-TTS-Nano) | Nano first: 0.1B, Apache 2.0, 4-core CPU realtime, 48 kHz stereo, streaming, 20 langs. Best alignment with our criteria. Then track v1.5 / Realtime / TTSD / VoiceGenerator / SoundEffect as siblings under one epic. |
|
||
| 2 | **Pocket TTS** (Kyutai) | CPU-first 100M model. MIT. Fills streaming gap without CUDA dependency. Several European langs added by Feb 2026. |
|
||
| 3 | **IndicF5** | Fills Indian-language gap (#339). Closes many language-request issues. |
|
||
| 4 | **VibeVoice** (Microsoft, #172) | 1.5B, long-form multi-speaker (up to 90 min, 4 speakers). Strong Stories-editor fit. |
|
||
| 5 | **Voxtral TTS** (Mistral, #364) | 4B presets+cloning. Frontier quality claim, but 16 GB+ VRAM — would need the platform-tier work first. |
|
||
| 6 | **Fish Speech / Fish Audio S2** | 50+ langs, word-level instruct. **License clarification first.** (#385) |
|
||
| 7 | **XTTS-v2** | 17+ langs, mature pip. CPML likely kills commercial use — verify. |
|
||
| 8 | **index-tts2** (#370) | Unvetted. |
|
||
| — | ~~**VoxCPM2**~~ | **Backlogged** — CUDA-only upstream. Revisit when tier system ships or MPS bugs are fixed upstream. |
|
||
| — | *New (06-27 sweep), in order* → | |
|
||
| 9 | **dots.tts** | 2B end-to-end AR, 48 kHz, Apache-2.0 + fast MeanFlow variant. Top new candidate. Git-source install — smoke-test packaging + VRAM; likely experimental until platform gating exists. |
|
||
| 10 | **LongCat-AudioDiT** | 3.5B diffusion, has an MLX/`mlx_audio` path — best Apple Silicon fit. zh/en only, not realtime. |
|
||
| 11 | **SoproTTS** | 135M, `pip install sopro`, streaming, ~250 ms TTFA / 0.05 RTF on M3 CPU. Quality-test first. |
|
||
| 12 | **NeuTTS Air/Nano** | On-device GGUF cloning, ~3s reference. Needs a GGUF wrapper; Air is Apache-2.0, Nano license split needs review. |
|
||
| 13 | **X-Voice** | 0.4B, 30 langs, no prompt-transcript required. Verify license/packaging. |
|
||
| 14 | **FireRedTTS-2** | Apache-2.0 long-form multi-speaker/podcast streaming. Stories-editor engine; needs VRAM testing. |
|
||
| 15 | **Maya1** | 3B Apache-2.0 expressive voice-design, emotion tags. English-only, 16 GB+ VRAM — gate behind platform tiers. |
|
||
|
||
### Tier 3b — STT / Capture Candidates (06-27 sweep)
|
||
|
||
Feeds the planned streaming-transcription roadmap; Whisper alternatives.
|
||
|
||
| Priority | Item | Notes |
|
||
|----------|------|-------|
|
||
| 1 | **Nemotron 3.5 ASR Streaming 0.6B** | Cache-aware streaming FastConformer-RNNT, 40 locales, MLX path. Strongest streaming-dictation fit. Verify license + non-CUDA perf. |
|
||
| 2 | **Cohere Transcribe 03-2026** | 2B Apache-2.0, 14 langs, ONNX/INT8 across CPU/Apple Silicon/GPU. Cleanest offline `/transcribe` candidate. |
|
||
| 3 | **ARK-ASR 3B / 0.6B** | Broad multilingual, INT8 ONNX for edge. Very new; likely `trust_remote_code` — validate stability. |
|
||
| 4 | **IBM Granite Speech 4.1 2B / NAR** | ASR + speech translation (en/fr/de/es/pt/ja). Compelling if we expand into translation. |
|
||
|
||
### ~~Previously Prioritized — Now Done~~
|
||
|
||
- ~~Kokoro 82M — finish integration~~ **Shipped** (PR #325)
|
||
- ~~Qwen CustomVoice~~ **Shipped** (PR #328)
|
||
- ~~Intel Arc (XPU) support~~ **Shipped** (PR #320)
|
||
- ~~Blackwell CUDA~~ **Shipped** (PR #401, follow-up work open)
|
||
- ~~Generation cancellation~~ **Shipped** (PR #444)
|
||
- ~~macOS Intel x86_64~~ **Shipped** (PR #416)
|
||
|
||
---
|
||
|
||
## Branch Inventory
|
||
|
||
| Branch | PR | Status | Notes |
|
||
|--------|-----|--------|-------|
|
||
| `voicebox-new-models` | — | **Active** | New model research (Fish Speech, Pocket TTS, VibeVoice, etc.); VoxCPM evaluated & backlogged |
|
||
| `fix/kokoro-pyinstaller-source-files` | — | Active | Kokoro frozen-build source bundling (parent of `voicebox-new-models`) |
|
||
| `feat/cosyvoice-engine` | #311 | Open — closing | CosyVoice2/3 — abandoned, poor quality |
|
||
| `feat/kokoro` | #325 | **Merged** | Kokoro 82M + voice profile type system |
|
||
| `feat/qwen-custom-voice` | #328 | **Merged** | Qwen CustomVoice preset engine |
|
||
| `feat/chatterbox-turbo` | #258 | **Merged** | Chatterbox Turbo + per-engine languages |
|
||
| `feat/chatterbox` | #257 | **Merged** | Chatterbox Multilingual |
|
||
| `feat/luxtts` | #254 | **Merged** | LuxTTS + multi-engine arch |
|
||
|
||
---
|
||
|
||
## Quick Reference: API Endpoints
|
||
|
||
<details>
|
||
<summary>All current endpoints</summary>
|
||
|
||
| Endpoint | Method | Purpose |
|
||
|----------|--------|---------|
|
||
| `/health` | GET | Health check, model/GPU status |
|
||
| `/profiles` | POST, GET | Create/list voice profiles |
|
||
| `/profiles/{id}` | GET, PUT, DELETE | Profile CRUD |
|
||
| `/profiles/{id}/samples` | POST, GET | Add/list voice samples |
|
||
| `/profiles/{id}/avatar` | POST, GET, DELETE | Avatar management |
|
||
| `/profiles/{id}/export` | GET | Export profile as ZIP |
|
||
| `/profiles/import` | POST | Import profile from ZIP |
|
||
| `/generate` | POST | Generate speech (engine param selects TTS backend) |
|
||
| `/generate/stream` | POST | Stream speech (MLX only) |
|
||
| `/history` | GET | List generation history |
|
||
| `/history/{id}` | GET, DELETE | Get/delete generation |
|
||
| `/history/{id}/export` | GET | Export generation ZIP |
|
||
| `/history/{id}/export-audio` | GET | Export audio only |
|
||
| `/transcribe` | POST | Transcribe audio (Whisper) |
|
||
| `/models/status` | GET | All model statuses (Qwen, LuxTTS, Chatterbox, Chatterbox Turbo, TADA, Whisper) |
|
||
| `/models/download` | POST | Trigger model download |
|
||
| `/models/download/cancel` | POST | Cancel/dismiss download |
|
||
| `/models/{name}` | DELETE | Delete downloaded model |
|
||
| `/models/load` | POST | Load model into memory |
|
||
| `/models/unload` | POST | Unload model |
|
||
| `/models/progress/{name}` | GET | SSE download progress |
|
||
| `/tasks/active` | GET | Active downloads/generations (with inline progress) |
|
||
| `/stories` | POST, GET | Create/list stories |
|
||
| `/stories/{id}` | GET, PUT, DELETE | Story CRUD |
|
||
| `/stories/{id}/items` | POST, GET | Story items CRUD |
|
||
| `/stories/{id}/export` | GET | Export story audio |
|
||
| `/channels` | POST, GET | Audio channel CRUD |
|
||
| `/channels/{id}` | PUT, DELETE | Channel update/delete |
|
||
| `/cache/clear` | POST | Clear voice prompt cache |
|
||
| `/server/cuda/status` | GET | CUDA binary availability |
|
||
| `/server/cuda/download` | POST | Download CUDA binary |
|
||
| `/server/cuda/switch` | POST | Switch to CUDA backend |
|
||
|
||
</details>
|