chore: import upstream snapshot with attribution
regression / regression-shards (style-16-prod style-9-prod style-17-prod iframe-render-compat variables-prod mp4-h265-sdr, shard-4) (push) Has been cancelled
regression / regression-shards (style-4-prod style-11-prod style-2-prod animejs-adapter typegpu-adapter parallel-capture-regression, shard-5) (push) Has been cancelled
regression / regression-shards (style-7-prod style-8-prod style-10-prod css-spinner-render-compat webm-transparency mp4-h264-sdr webm-vp9, shard-3) (push) Has been cancelled
regression / regression-shards (sub-composition-video style-18-prod raf-ball-render-compat font-variant-numeric sub-comp-t0 sub-comp-id-selector, shard-7) (push) Has been cancelled
Windows render verification / Detect changes (push) Has been cancelled
Windows render verification / Preflight (lint + format) (push) Has been cancelled
Windows render verification / Render on windows-latest (push) Has been cancelled
Windows render verification / Tests on windows-latest (push) Has been cancelled
CI / Detect changes (push) Has been cancelled
CI / Build (push) Has been cancelled
CI / Lint (push) Has been cancelled
CI / Fallow audit (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / Typecheck (push) Has been cancelled
CI / Test (push) Has been cancelled
CI / Producer: integration tests (push) Has been cancelled
CI / Producer: unit tests (push) Has been cancelled
CI / File size check (push) Has been cancelled
CI / Test: skills (push) Has been cancelled
CI / Skills: manifest in sync (push) Has been cancelled
CI / CLI: npx shim (macos-latest) (push) Has been cancelled
CI / CLI: npx shim (ubuntu-latest) (push) Has been cancelled
CI / CLI: npx shim (windows-latest) (push) Has been cancelled
CI / SDK: unit + contract + smoke (push) Has been cancelled
CI / Test: runtime contract (push) Has been cancelled
CI / Studio: load smoke (push) Has been cancelled
CI / Smoke: global install (push) Has been cancelled
CI / CLI smoke (required) (push) Has been cancelled
CI / Semantic PR title (push) Has been cancelled
Player perf / Detect changes (push) Has been cancelled
Player perf / Preflight (lint + format) (push) Has been cancelled
Player perf / player-perf (push) Has been cancelled
Player perf / Perf: drift (push) Has been cancelled
Player perf / Perf: fps (push) Has been cancelled
Player perf / Perf: parity (push) Has been cancelled
Player perf / Perf: scrub (push) Has been cancelled
Player perf / Perf: load (push) Has been cancelled
preview-regression / Detect changes (push) Has been cancelled
preview-regression / Preflight (lint + format) (push) Has been cancelled
preview-regression / Preview parity (push) Has been cancelled
preview-regression / preview-regression (push) Has been cancelled
regression / regression (push) Has been cancelled
regression / Detect changes (push) Has been cancelled
regression / Preflight (lint + format) (push) Has been cancelled
regression / regression-shards (hdr-regression style-5-prod style-3-prod mov-prores, shard-1) (push) Has been cancelled
regression / regression-shards (overlay-montage-prod style-12-prod chat missing-host-comp-id png-sequence portrait-edge-bleed, shard-6) (push) Has been cancelled
regression / regression-shards (style-13-prod style-6-prod vignelli-stacking gsap-letters-render-compat audio-mux-parity, shard-8) (push) Has been cancelled
regression / regression-shards (style-15-prod hdr-hlg-regression style-1-prod many-cuts vfr-screen-recording render-symlinked-assets, shard-2) (push) Has been cancelled
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Docs / Validate docs (push) Has been cancelled
Sync skills to ClawHub / Publish changed skills (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 12:58:35 +08:00
commit 85453da49f
4031 changed files with 710987 additions and 0 deletions
+1
View File
@@ -0,0 +1 @@
eval-report.html
+415
View File
@@ -0,0 +1,415 @@
---
name: media-use
description: Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Keeps search noise on disk, hands the agent one path or block. Use for any audio, image, icon, logo, voiceover, caption, color-grading, or media-asset need.
---
# media-use
The media OS for HyperFrames: resolve · generate · operate · remember, every media type, one skill, zero context noise.
## Setup — install heygen first (free-usage path)
```bash
curl -fsSL https://static.heygen.ai/cli/install.sh | bash
heygen update # free usage needs the OAuth-capable CLI (v0.3.0+)
heygen auth login --oauth # OAuth = free subscription credits; --api-key bills API credits
```
This unlocks the FREE path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar videos. Sign in with `--oauth` — the free allowance rides on the OAuth session (an API key bills API credits instead). **media-use requires heygen >= v0.3.0 uniformly** (the OAuth free-usage path needs it), so `--doctor` nudges older CLIs to update even for API-key-only use. Before resolving anything, verify setup with:
```bash
node <SKILL_DIR>/scripts/resolve.mjs --doctor
```
## What it owns (the gaps HyperFrames leaves)
HyperFrames owns media _playback_; media-use owns everything else. Each row is enforced by `scripts/lib/coverage.test.mjs` so the claim can't rot.
| HyperFrames gap | media-use owns it via |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| Audio-only, no image/icon | `resolve --type image\|icon` (heygen asset search) |
| No third-party brand logos | `resolve --type logo` (svgl → simple-icons → GitHub org avatar → domain favicon) |
| No voice / audio generation | `resolve --type voice` (HeyGen TTS free-usage path; optional local Kokoro) + the audio engine (`audio/scripts/audio.mjs`) |
| Scattered/duplicated audio engine | one consolidated engine under `audio/` (hyperframes-media retired) |
| No agent media-ops (cut/reframe/transform) | `references/operations.md` + `resolve --from` to register outputs |
| No transcript-driven cutting | `scripts/transcript-cut.mjs` compiles word-timestamp edits into cut lists |
| No auto-duck / publish loudness | `scripts/audio-duck.mjs` + `references/operations.md` loudnorm/sidechain recipes |
| No cross-project memory | global content-addressed cache + auto-promote (`~/.media`) |
| No color-grade authoring | `resolve --type grade` emits a paste-ready `data-color-grading` block; `resolve --type lut` freezes validated `.cube` files |
| No image generation | RAM-graded local mflux (FLUX) via `scripts/lib/mflux-provider.mjs`, codex `image_gen` upsell (`scripts/lib/codex-provider.mjs`) |
| No video generation | HeyGen avatar video free-usage path; optional spec-gated local LTX (`videogen` in `scripts/lib/local-models.mjs`) |
| Weak local-model defaults | HeyGen free-usage path via the `heygen` CLI; local open-source tools only as opt-in alternatives (`scripts/lib/local-run.mjs`) |
## When to use
Call `resolve` whenever a composition needs media: background music, sound effects, images, icons, brand logos, voice, a color grade, or a LUT. For voiceover / TTS, music, SFX, and caption timing, use the **audio engine** (below); background removal is delegated to the `hyperframes` CLI; transcription defaults to Parakeet (better than whisper.cpp: 6.05% vs 7.44% WER, 5-10x faster) via `scripts/transcribe.mjs`, with whisper.cpp auto-fallback (see `references/operations.md`). For cutting / reframing / transforming existing media, see `references/operations.md`. media-use searches the HeyGen catalog first for media files, resolves official logos through the logo cascade, uses local deterministic color grading for `grade`/`lut`, freezes the best match locally when a file is needed, registers it in a manifest, and hands the agent one line; all search noise stays on disk.
## Be proactive — run a media opportunity pass
The human usually can't tell which media would lift the piece. You can. When you build or review a composition, do **one** grounded scan and then **ask once** — don't silently add, and don't nag per asset.
Surface an opportunity only when a concrete signal is present:
| Signal detected | Offer |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
| On-screen text / a script with no voiceover | TTS voiceover (audio engine) |
| Emoji or a `<div>` styled as an icon | resolve real `icon`s |
| Image that is a placeholder, tiny, or upscaled-looking | a better `image` (and/or upscale — see `references/operations.md`) |
| Hard scene cuts / transitions with no sound | transition `sfx` |
| A piece over ~10s with no music bed | `bgm` |
| Footage that reads under/over-exposed or color-cast | a corrective `grade` (analyze with `grade --for`, preview with `hyperframes grade-compare`) |
Rules that keep this a help, not nagware:
- **Grounded, not generic.** No signal → no suggestion. Never open with "want better images?".
- **Opinionated + concrete.** Propose the specific fix ("add a VO from your script, swap 3 emoji for real icons, replace the 400×400 hero, whooshes on the 4 cuts"), with defaults chosen — the human just approves **all / some / none**.
- **Once per project.** One consolidated ask, top few highest-value items. Respect "leave it" and don't re-raise.
- **Surface, never silently mutate.** Color grades especially: propose and preview, never auto-apply — a gray-world "correction" ruins an intentional sunset or neon look.
## Resolve
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type <type> --intent "<description>" --project <dir>
```
Returns one line: `resolved <id> → <path> (<type>, <metadata>)`
### Types
| Type | What it finds | Provider / cascade |
| ------- | -------------------------------- | ------------------------------------------------------------ |
| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) |
| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog |
| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) |
| `icon` | Icons, symbols | HeyGen asset search (type=icon) |
| `logo` | Official brand marks | svgl → simple-icons → GitHub org avatar → domain favicon |
| `voice` | TTS voiceover | HeyGen TTS free-usage path; optional local Kokoro |
| `grade` | HyperFrames color-grading blocks | Core preset → look index params/CDN LUT → deterministic cube |
| `lut` | Reusable `.cube` LUT files | Look index params/CDN LUT → deterministic cube |
### Examples
```bash
# Background music
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --project .
# → resolved bgm_001 → .media/audio/bgm/bgm_001.mp3 (bgm, 25s)
# Sound effect
node <SKILL_DIR>/scripts/resolve.mjs --type sfx --intent "whoosh" --project .
# → resolved sfx_001 → .media/audio/sfx/sfx_001.mp3 (sfx, 0.57s)
# Image
node <SKILL_DIR>/scripts/resolve.mjs --type image --intent "gradient tech background" --project .
# → resolved image_001 → .media/images/image_001.jpg (image)
# Icon
node <SKILL_DIR>/scripts/resolve.mjs --type icon --intent "rocket" --project .
# → resolved icon_001 → .media/images/icon_001.png (icon, transparent)
# Brand logo (official mark — never redrawn by hand)
node <SKILL_DIR>/scripts/resolve.mjs --type logo --entity linkedin --intent "LinkedIn logo" --project .
# → resolved logo_001 → .media/images/logo_001.svg (logo, official mark)
# Color grade block
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm daylight" --project . --json
# → {"ok":true,"preset":"warm-daylight","grading":{"preset":"warm-daylight","intensity":1},...}
# LUT file
node <SKILL_DIR>/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project .
# → resolved lut_001 → .media/luts/lut_001.cube (lut)
```
### Flags
| Flag | Description |
| --------------- | ------------------------------------------------------------------------------------ |
| `--type, -t` | Media type: bgm, sfx, image, icon, logo, voice, grade, lut |
| `--intent, -i` | What you need (natural language) |
| `--entity, -e` | Entity name for cache matching (optional) |
| `--project, -p` | Project directory (default: .) |
| `--candidates` | List reusable assets (project + global cache) for `--type`; no download, no mutation |
| `--reuse <sha>` | Import a specific global-cache asset (by content sha/prefix, from `--candidates`) |
| `--from` | Freeze a local file or direct public URL (ingest) |
| `--for` | Analyze a local image/video and add measured adjust suggestions (`grade` only) |
| `--local-only` | Offline: skip every network provider (cache + local only) |
| `--provider` | Force one generator (e.g. `codex`, `mflux`, `kokoro`, `heygen`) |
| `--adopt` | Bulk-import existing assets/ into manifest |
| `--doctor` | Check local CLI dependencies; no manifest changes |
| `--stats` | Print local usage stats from `.media/` and `~/.media`; no manifest changes |
| `--days N` | Limit `--stats` to timestamped records/misses from the last N days |
| `--json` | Output JSON instead of one-line result |
## Reuse before you resolve
Before resolving bgm/sfx/image/icon/logo/grade/lut, **check what already exists and reuse it when it fits.** media-use does not semantically match for you — you are the judge. It surfaces candidates; you decide.
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --candidates --project .
# [project] upbeat tech launch (25s, heygen.audio.sounds)
# .media/audio/bgm/bgm_001.wav
# [global] energetic tech intro (22s, heygen.audio.sounds)
# --reuse 06e052c075fd2b80
```
Read the list and judge semantic fit yourself — "upbeat tech launch" ≈ "energetic tech intro" is a call only you can make from the descriptions. Then:
- **A project candidate fits** → just reference its path in your composition. Nothing else to run.
- **A global candidate fits** → `resolve --type bgm --reuse <sha>` copies it into this project (self-contained render) and records it.
- **Nothing fits** → resolve fresh (`--type ... --intent ...`).
**Trust guardrail — when unsure, resolve fresh.** A redundant download is cheap; shipping the wrong asset is not. Judge fit from description + prompt + type + duration/dims. For **brand/entity** assets, reuse a _global_ candidate only when the entity matches exactly — the global cache aggregates every project you have worked on, so a `--candidates` list can surface another client's brand mark and its prompt text. Never reuse a cross-project brand asset on a loose match.
The deterministic floor still runs automatically: an identical (case/whitespace-insensitive) repeat auto-reuses with no `--candidates` step. `--candidates` is only for the semantic layer above that floor — and a fuzzy match is **never** auto-applied; reuse is always your explicit call. On a resolve that misses the floor and is about to fetch, media-use prints a one-line stderr hint when similar cached assets exist, pointing you back here.
## Color grading
Use `grade` when you need the actual HyperFrames `data-color-grading` value to paste onto an `<img>` or `<video>`. Core presets and params-backed library looks resolve locally; future CDN-backed library looks require network unless already frozen:
**Never `cat`/read a `.cube` file into context.** A 3D LUT is ~size^3 lines of raw numbers (33^3 ≈ 36k lines at the default size). It bloats context and carries zero human/agent-legible signal. To understand or choose a LUT, use `hyperframes grade-compare` to see it rendered, or `cube-validate.mjs` for a one-line `{ok,size}` check. Read `.media/index.md` or `luts/index.json` for the description. Never read the LUT body itself.
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm daylight" --project . --json
```
Preset-first output uses the core runtime vocabulary and does not freeze a file:
```json
{
"preset": "warm-daylight",
"intensity": 1
}
```
Paste it as an attribute value after JSON string escaping:
```html
<video
class="clip"
src="./media/scene.mp4"
data-color-grading='{"preset":"warm-daylight","intensity":1}'
></video>
```
Looks beyond the preset vocabulary freeze a validated `.cube` under `.media/luts/` and return a block that references it:
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "teal orange blockbuster" --project . --json
```
```json
{
"intensity": 1,
"lut": { "src": ".media/luts/grade_001.cube", "intensity": 0.85 }
}
```
Use `lut` when you only need the reusable `.cube` file:
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project .
```
For a describable technical look, author an explicit parametric LUT with `--params`:
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type lut --params '{"contrast":0.2,"temperature":-0.3}' --project .
node <SKILL_DIR>/scripts/resolve.mjs --type grade --params '{"exposure":0.2}' --project . --json
```
For a LUT generated by your own script, ingest it with `--from`; media-use validates it before registration and rejects invalid or oversized cubes:
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type lut --from custom.cube --project .
```
Parametric math (`buildCube`) cannot reproduce real film stocks or emulsion looks. Use a CDN-backed scanned `.cube` entry or ingest a real scanned `.cube` for those.
For visual selection, list reusable looks with `resolve --type grade --candidates`, write the promising entries to a `grades.json`, run `hyperframes grade-compare --for <frame> --grades grades.json`, then commit the winner with `resolve -t grade` as the final `data-color-grading` block.
Smart grade is `grade --for <media>`. It runs local `ffmpeg`/`ffprobe` signalstats, merges a bounded `adjust` suggestion into the returned block, and prints the measured evidence to stderr. Stdout remains valid JSON under `--json`; the suggestion is a starting point for the agent to tune, not an automatic neutralization of intentional color.
```bash
node <SKILL_DIR>/scripts/resolve.mjs --type grade --intent "warm cinematic" --for ./frame.png --project . --json
```
Library looks live in `luts/index.json`. Each entry keeps `id`, `description`, `tags`, and `intensity`, then supplies either compact `params` for on-demand `buildCube(params)` generation or a direct CDN `url` for future scanned `.cube` files. Do not commit generated `.cube` bodies; resolve validates generated or downloaded cubes as it freezes them under `.media/luts/`.
```bash
node skills/media-use/scripts/resolve.mjs --type lut --intent "teal orange blockbuster" --project . --json
node skills/media-use/scripts/lib/cube-validate.mjs .media/luts/lut_001.cube
```
## Providers
media-use holds no keys; every external tool owns its auth. Generation is
centered on the HeyGen CLI free-usage path. Install and authenticate `heygen`
before resolving bgm/sfx/image/icon/voice/avatar-video. Local tools are opt-in
alternatives where they exist: mflux for image, Kokoro for voice, Parakeet for
transcription, and LTX for local video generation. `resolve` spec-checks
AVAILABLE RAM for those local ladders (`describeModelLadder`); the agent can
see the ladder and override.
| Type | Provider / path |
| --------- | -------------------------------------------------------------------------------- |
| bgm/sfx | heygen catalog free-usage path |
| image | heygen search free-usage path; optional local mflux; codex `image_gen` upsell |
| voice | heygen tts free-usage path; optional local **Kokoro** (free, on-device) |
| icon | heygen asset search free-usage path |
| logo | svgl, then simple-icons, then GitHub org avatar, then domain favicon (all free) |
| grade/lut | local core-preset map, params/CDN look index, deterministic `buildCube` fallback |
| video | heygen avatar video free-usage path; optional local LTX (`videogen` ladder) |
Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free,
private, offline once cached). The `codex` CLI remains the ChatGPT-sub image
upsell. Cost rule (X4): the agent confirms before an agent-initiated paid call;
a user-requested one just runs.
To force a specific generator (e.g. a user says "make this image with codex"),
pass `--provider codex`: it pins resolution to that provider and skips the
free-usage default. See `references/operations.md` for the RAM ladders and
provider recipes.
`--local-only` skips every network provider, including the free HeyGen ones,
leaving the project + global cache and any installed local provider. For
HeyGen-only types, that means no fresh resolve.
## How it works
`resolve` runs an automatic floor, then falls through to fetching:
1. Check project `.media/manifest.jsonl` for a prompt match (case- and whitespace-insensitive) — auto-reuse
2. Scan existing `assets/` directory for unregistered files that share a word with the need
3. Check global cache `~/.media/` for a reusable asset matched on the same normalized prompt — auto-reuse
4. Search via provider (HeyGen audio catalog, HeyGen asset search), or resolve color locally
5. Freeze file to `.media/<type>/`, register in manifest, regenerate `index.md`, auto-promote to `~/.media/`
Steps 1 and 3 are the **deterministic floor**: they only auto-reuse an exact-normalized match, never a fuzzy one. Semantic reuse ("close enough") is the agent's explicit call via [Reuse before you resolve](#reuse-before-you-resolve) — it never happens automatically. The agent gets back **one line**; candidates, scores, provenance stay on disk.
## Adopt existing projects
Most HyperFrames projects already have assets in `assets/`. media-use adopts them:
```bash
node <SKILL_DIR>/scripts/resolve.mjs --adopt --project .
# → adopted 9 assets from assets/
# bgm_001 → assets/bgm/mango-fizz.mp3 (bgm, 146.6s)
# image_001 → assets/images/avatar.jpg (image, 400×400)
```
`ffprobe` extracts real duration and dimensions. During resolve, unregistered files in `assets/` matching the intent are adopted on the fly.
## Reading the inventory
After resolve or adopt, read `.media/index.md` for the full inventory:
```
# .media · 4 assets
id type dur dims path description
bgm_001 bgm 25s - .media/audio/bgm/bgm_001.mp3 upbeat tech launch
sfx_001 sfx 0.6s - .media/audio/sfx/sfx_001.mp3 whoosh
image_001 image - 1920×1080 .media/images/image_001.jpg gradient tech background
icon_001 icon - 200×200 .media/images/icon_001.png rocket
```
## Cross-project reuse
Assets are cached automatically on resolve. Every resolved/ingested asset is auto-promoted to the global cache at `~/.media/`, so subsequent resolves for the same (or near-identical) prompt, in any project, hit the cache with no re-download and no provider call.
For a _semantically_ similar (not identical) need in another project, the exact-match floor won't fire — use [Reuse before you resolve](#reuse-before-you-resolve): `--candidates` lists the global assets, and `--reuse <sha>` imports the one you pick. This is how a track resolved in one project gets reused in the next when the wording differs.
## Usage stats
Use `resolve --stats` for a local, shareable report over the current project's `.media/` manifest, the global `~/.media/` cache, and local resolve misses. Human output is compact; add `--json` for a single machine-readable object, and `--days N` to window timestamped records.
```bash
node <SKILL_DIR>/scripts/resolve.mjs --stats --project . --days 7
# media-use stats
# total resolves: 12
# misses: 2
# hit rate: 86%
```
## Files
- `.media/manifest.jsonl`: machine SSOT, one JSON record per line
- `.media/index.md`: agent-readable table (id, type, dur, dims, path, description)
- `~/.media/`: global cross-project reuse cache (content-addressed, SHA-256)
- `~/.media/misses.jsonl`: local-only resolve misses, including intent text for `--stats`
## Audio engine: voiceover, music, SFX, captions, transcription
For a full audio pass (TTS voiceover + background music + sound effects in one
shot), use the shared engine at `audio/scripts/audio.mjs`. It takes a neutral
`audio_request.json` and writes `audio_meta.json` plus assets under
`.media/audio/{voice,bgm,sfx}`:
```bash
node <SKILL_DIR>/audio/scripts/audio.mjs --request ./audio_request.json --out ./audio_meta.json
```
- **Request** `{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`: `id` joins each line back to your model; `bgm.mode` = `retrieve | generate | none` (omit for auto). `--only tts,bgm,sfx` runs a subset and merges into an existing `--out`.
- **Output** `audio_meta.json` (id-keyed): `voices[].{path,duration_s,words[]}` (word timestamps for captions), `sfx[]`, `bgm`, `total_duration_s`.
- **HeyGen free-usage path**: HeyGen CLI auth unlocks TTS plus music/SFX retrieval. Local/provider-specific generators are explicit alternatives where installed; run `node <SKILL_DIR>/scripts/resolve.mjs --doctor` before assuming retrieval or TTS will work.
- If BGM took the generate path (`bgm_pending: true`), run `audio/scripts/wait-bgm.mjs` before final render.
Single-shot helpers: `audio/scripts/heygen-tts.mjs` (one voice file). Transcription / background removal / captions use the `hyperframes` CLI (`transcribe`, `remove-background`), see the per-topic guides in `audio/references/` (`tts.md`, `bgm.md`, `sfx.md`, `transcribe.md`, `remove-background.md`, `captions/`).
## Operating on media (cut, reframe, transform)
media-use resolves + remembers; for **operating** on assets see
`references/operations.md`: local-tool recipes (ffmpeg trim/reframe/montage,
auto-editor, scenedetect) and the local-vs-HeyGen transform table (background
removal, upscale, lipsync, translate). Run the tool, then register the output
with `resolve --from <output> --type <type>` so it joins the ledger + global
cache.
## CLI tools used (what to run, and how to enable each)
`resolve` auto-cascades; each provider shells one CLI. HeyGen is the
free-usage path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar
video, so those capabilities need `heygen` installed and authenticated. Local
tools are OPT-IN alternatives where they exist; install one to unlock its free,
private, on-device path instead of or ahead of HeyGen for that type. Only
`ffmpeg`/`ffprobe` are strictly required for the tool to run at all.
| Tool | Serves | Install |
| ------------------ | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `ffmpeg`/`ffprobe` | adopt probing, smart-grade signalstats, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
| `heygen` | catalog (bgm/sfx/image/icon) + TTS (voice) + avatar video — the free-usage path | `curl -fsSL https://static.heygen.ai/cli/install.sh \| bash` then `heygen auth login --oauth` (needs >= v0.3.0) |
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
| `parakeet-mlx` | local transcription (default ASR, best) | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
| `npx hyperframes` | Kokoro TTS (voice), whisper.cpp (transcribe fallback), remove-background | bundled with the hyperframes CLI |
The RAM-graded local-model shortlist + exact per-tier install/invoke lives in
`scripts/lib/local-models.mjs` (the agent can read `describeModelLadder(cap, specs)`
to see which model fits this machine). Without a tool on PATH, its provider
prints a one-line diagnostic to stderr and resolve falls through where another
provider exists (e.g. no `mflux` -> codex image upsell; no `parakeet-mlx` -> whisper.cpp).
`heygen asset search` is a pre-launch command hidden from `heygen --help`, but it
runs; providers tag requests with the allowlisted `X-HeyGen-Client-Source` header
(v0.3.0+).
## Telemetry
`resolve` and the edit tools (transcribe / transcript-cut / audio-duck) send an
anonymous usage event to PostHog (`scripts/lib/telemetry.mjs`), so we can see
which capabilities are actually used. It records only the media TYPE, the
resolution SOURCE, and the winning PROVIDER: never the intent text, file names,
or paths, and `$ip:null` so no IP is stored. Best-effort and non-blocking (a
resolve never waits on or fails from telemetry).
Opt out with `DO_NOT_TRACK=1` or `HYPERFRAMES_NO_TELEMETRY=1` (also off in CI and
dev). Same public PostHog project key and opt-outs as the `hyperframes` CLI.
## Privacy
media-use uses the same shared install id as the `hyperframes` CLI/studio
(`~/.hyperframes/config.json`). When you are signed in to HeyGen, usage is
linked to your account email, or username when email is unavailable, matching
the CLI behavior. The events stay coarse: media type, source, provider, and
small counts only; intent text and paths stay local. Disable telemetry with
`HYPERFRAMES_NO_TELEMETRY=1` or `DO_NOT_TRACK=1`.
@@ -0,0 +1,35 @@
# SFX Credits
All sound effects in this directory are sourced from [Pixabay](https://pixabay.com/sound-effects/) and used under the [Pixabay Content License](https://pixabay.com/service/license-summary/).
The Pixabay license allows free use for commercial and non-commercial purposes without attribution, but attribution is appreciated and given here for transparency.
## Files
The following `.mp3` files are bundled with this skill:
- `chime.mp3`
- `click.mp3` / `click-soft.mp3`
- `error.mp3`
- `glitch-1.mp3` / `glitch-2.mp3` / `glitch-3.mp3`
- `impact-bass-1.mp3` / `impact-bass-2.mp3`
- `key-press.mp3`
- `notification.mp3`
- `ping.mp3`
- `pop.mp3`
- `riser.mp3`
- `sparkle.mp3`
- `typing.mp3`
- `whoosh.mp3` / `whoosh-short.mp3` / `whoosh-cinematic.mp3`
See `manifest.json` for per-file metadata (duration, energy character, recommended use).
## License
All files are distributed under the [Pixabay Content License](https://pixabay.com/service/license-summary/), which permits:
- Commercial and non-commercial use
- Modification and remixing
- Redistribution as part of derivative works (such as videos rendered with HyperFrames)
without any attribution requirement.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,97 @@
{
"chime": {
"file": "chime.mp3",
"duration": 2.5,
"description": "Soft melodic chime — gentle positive beat: success/confirmation or a lighthearted transition. Sync to the visual moment."
},
"click-soft": {
"file": "click-soft.mp3",
"duration": 0.37,
"description": "Quiet short click — low-key UI tap / soft selection. Short accent, sync exactly to the on-screen action."
},
"click": {
"file": "click.mp3",
"duration": 0.37,
"description": "Crisp UI click — button press, toggle, selection. Short accent, sync exactly to the on-screen action."
},
"error": {
"file": "error.mp3",
"duration": 1.62,
"description": "Negative / error tone — failure state, a 'wrong' beat, or a glitchy interruption."
},
"glitch-1": {
"file": "glitch-1.mp3",
"duration": 2.64,
"description": "Punchy digital glitch — hard-cut accent or sudden reveal. Trigger on the hit; let the decay bleed into the next shot (J-cut)."
},
"glitch-2": {
"file": "glitch-2.mp3",
"duration": 3.5,
"description": "Harsh, longer glitch — chaotic / jarring transition or a distorted reveal."
},
"glitch-3": {
"file": "glitch-3.mp3",
"duration": 3.1,
"description": "Low-key glitch texture — subtle digital shift, minimal transition that sits under other audio."
},
"impact-bass-1": {
"file": "impact-bass-1.mp3",
"duration": 2.12,
"description": "Bass impact hit — logo/hero snap, headline slam. Trigger on the visual landing; decay carries into the next shot (J-cut)."
},
"impact-bass-2": {
"file": "impact-bass-2.mp3",
"duration": 2.59,
"description": "Bass impact with a short swell — brief anticipation then a deep hit. Place so the peak lands on the reveal."
},
"key-press": {
"file": "key-press.mp3",
"duration": 0.4,
"description": "Single key press — one keystroke / terminal-input beat. Short accent, sync to the typed character."
},
"notification": {
"file": "notification.mp3",
"duration": 2.46,
"description": "Notification chime — alert, message-in, toast/badge appears. Sync to the element entering."
},
"ping": {
"file": "ping.mp3",
"duration": 1.32,
"description": "Sharp electronic ping — punchy accent on a key reveal or data point. Sync to the beat."
},
"pop": {
"file": "pop.mp3",
"duration": 0.72,
"description": "Quick pop — element appear/spawn, chip/tag/badge in. Small precise accent, sync to the pop-in."
},
"riser": {
"file": "riser.mp3",
"duration": 10.03,
"description": "Long cinematic riser (~10s build, peak at the end). Trigger at (climax_time 10.03s) so it crests exactly on the reveal."
},
"sparkle": {
"file": "sparkle.mp3",
"duration": 1.8,
"description": "Bright sparkle / shimmer — magical reveal or 'shine' highlight on a hero element. Sync to the highlight."
},
"typing": {
"file": "typing.mp3",
"duration": 1.5,
"description": "Typing burst (~1.5s of keys) — keyboard / code typing reveal, text-being-typed beat. Start as the text begins typing."
},
"whoosh-cinematic": {
"file": "whoosh-cinematic.mp3",
"duration": 5.54,
"description": "Cinematic whoosh build (~5.5s) — sweeping scene transition. Align so the swell peaks on the cut."
},
"whoosh-short": {
"file": "whoosh-short.mp3",
"duration": 0.57,
"description": "Short whoosh — quick swipe/slide accent, fast element move, snappy transition. Sync to the motion."
},
"whoosh": {
"file": "whoosh.mp3",
"duration": 0.57,
"description": "Punchy whoosh/impact — fast reveal or hard transition accent. Sync to the motion."
}
}
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+72
View File
@@ -0,0 +1,72 @@
# Background music (BGM)
One music bed per composition, produced by the shared audio engine (`scripts/audio.mjs``scripts/lib/bgm.mjs`). Two routes, chosen by the engine's one switch — whether a HeyGen credential is present:
- **HeyGen retrieval — the default when credentialed.** Search HeyGen's music catalog by mood, download the top track. No generation; same `~/.heygen` / `$HEYGEN_API_KEY` credential as TTS.
- **Local generation (Lyria → MusicGen) — the fallback when there is no credential** (or when asked for explicitly). Generate a WAV from a mood prompt. There is **no `npx hyperframes bgm` command**; the engine spawns `scripts/lyria-recipe.py` or an inline MusicGen script directly.
> **Run the Preflight first — no credential is not a green light to silently generate locally.** Before generating, complete the sign-in **Preflight** (see `../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen's music library, or continue offline with local generation). This applies to a one-off "generate a BGM" request just as much as inside a full workflow.
## Driving it from the request
`audio_request.json``bgm: { mode?, query?, prompt? }`:
- **`mode`** — `retrieve | generate | none`. Omit for **auto** (retrieve when credentialed, else generate). An **explicit** `retrieve` is strict: no credential ⇒ skip, never a detached generate (so a caller with no `wait-bgm` step, e.g. product-launch, can't get a pending job it won't await).
- **`query`** — the mood, used for retrieval and as a fallback prompt seed (e.g. a storyboard's `music:` field, falling back to `message``arc``"calm cinematic underscore"`).
- **`prompt`** — an explicit full prompt for generation; omit and the engine infers one (see Mood inference). Optional `blob` / `archetype` / `arc` feed that inference.
## HeyGen retrieval (default)
`searchSounds(query, "music", { limit: 5 })``GET /audio/sounds?query=<mood>&type=music&limit=5`. Take the top result (ranked by `score`), download its presigned `audio_url``assets/bgm/track.mp3`. Synchronous. No match → skip (BGM is optional; never fail the render over it). Cue written to `audio_meta.json`:
```jsonc
{
"path": "assets/bgm/track.mp3",
"volume": 0.12,
"mode": "retrieve",
"query": "calm cinematic underscore",
"duration_s": 42.0,
}
```
`volume` comes from the engine's `bgmDefaultVolume()`: `BGM_BED_VOLUME` (currently `0.12` ≈ -18 dB — a bed under the voice) under narration, `BGM_SILENT_VOLUME` (currently `0.9`) for a silent film (no voice). Tune those constants in `scripts/lib/bgm.mjs`, not call sites. An explicit `volume` in `audio_meta.json` always overrides this default. `bgm_pending` is `false` — the file is on disk when the engine returns.
## Local generation (fallback) — Lyria → MusicGen
Spawned **detached** so voice work isn't blocked; `audio_meta.bgm_pending: true` and `bgm_pid` / `bgm_log` are set until it finishes. **Run `scripts/wait-bgm.mjs` before assembling** — it polls the output file / process / log, detects crashes, and writes `bgm_status.json` (`status: ready | failed | timeout | disabled`). A failed/absent track is simply omitted; it never blocks voice/SFX.
| Order | Provider | Env / deps | Speed | Quality |
| ----- | ------------------------------------ | ------------------------------------------------------------------------------------- | --------------------------------------- | --------------------------- |
| 1 | Google Lyria RealTime | `$GEMINI_API_KEY` or `$GOOGLE_API_KEY` + `google-genai` (auto-installed on demand) | Real-time stream (≈ requested duration) | Production-grade |
| 2 | MusicGen (`facebook/musicgen-small`) | Python `transformers + torch + soundfile + numpy` (~300 MB first run; auto-installed) | Slow on CPU; fast on Apple MPS / CUDA | Decent; prompt-only control |
Output → `assets/bgm/track.wav`, target = total voice duration. MusicGen generates **one** seed clip (≤2830s, under the decoder's positional limit) then crossfade-loops it up to the target (or trims down if shorter), avoiding per-segment seams. Backend selection is by what can actually **run**: Lyria only when `import google.genai` succeeds, else MusicGen; if neither can be made to run, BGM is skipped (voice + SFX still render).
## Mood inference (the generate prompt)
`inferBgmPrompt()` in `scripts/lib/bgm.mjs`: an explicit `prompt` wins; otherwise industry-keyword **base** → narrative-**archetype** shape → emotional-**arc** tiebreaker.
| Match in `blob` / `query` | Base prompt | BPM |
| ------------------------------------------------------ | --------------------------------------------------------------------------- | --- |
| `crypto / nft / web3 / defi / token / blockchain` | atmospheric electronic, deep bass, futuristic synths, restrained percussion | 100 |
| `finance / fintech / bank / payment / invest / wealth` | calm cinematic, soft strings, subtle piano, restrained percussion | 92 |
| `creative / agency / design / studio / art / brand` | playful electronic, warm pads, light percussion | 115 |
| _(default: SaaS / tech / platform)_ | uplifting corporate tech, bright modern piano with synth pads | 108 |
Archetype then reshapes the arc — PAS → "MINOR to MAJOR" build; BAB / future-pacing → aspirational rising; feature-cascade → +10 BPM driving; demo-loop → 8 BPM minimal. The emotional arc breaks remaining ties (tension→relief, excitement, trust/reassurance).
## Lyria knobs (direct recipe use)
The engine bakes BPM / scale into the **prompt text** (via the inference above) and passes only `--output` / `--duration` / `--prompt` to the recipe. If you invoke `scripts/lyria-recipe.py` directly you can also set: `--bpm` (90110 calm, 110130 energetic), `--brightness` (01, ≥0.7 promotional), `--density` (01, higher = fuller), `--scale` (`MAJOR` / `MINOR` / `PENTATONIC` / …), `--negative-prompt` (styles to exclude). MusicGen ignores all of these — put the mood in the prompt.
## Failure modes
| Failure | Behavior |
| --------------------------------------------- | ---------------------------------------------------------------------------------------- |
| No music match (retrieve) | `bgm: null`, anomaly logged. Render proceeds without BGM. |
| Explicit `retrieve`, no credential | Skipped (no silent generate fallback). Use `mode: generate` or omit `mode` for auto. |
| Neither Lyria nor MusicGen can run (generate) | `bgm` disabled with a `pip install …` hint. Voice + SFX still render. |
| Generate still rendering at assemble time | `bgm_pending: true`; `wait-bgm.mjs` waits/checks and writes `bgm_status.json` first. |
| Generate crashed | `wait-bgm.mjs``bgm_status.json { status: "failed" }`; the `<audio>` track is omitted. |
BGM failure never blocks a render.
@@ -0,0 +1,159 @@
# Captions
Before authoring: confirm the transcript came from the right Whisper model. CLI default `small.en` silently translates non-English audio — see [`../transcribe.md`](../transcribe.md) → "Language Rule" and [`transcript-handling.md`](transcript-handling.md) for the mandatory quality check.
Analyze spoken content to determine caption style. If user specifies a style, use that. Otherwise, detect tone from the transcript.
## Transcript Source
```json
[
{ "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
{ "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
]
```
`id` (`w0`, `w1`, …) is the stable reference for per-word overrides and is added by `hyperframes transcribe`. It's optional for backwards compatibility with hand-authored transcripts. See [`../transcribe.md`](../transcribe.md) → "Output Shape" for how this is produced, and [`transcript-handling.md`](transcript-handling.md) for cleanup before consumption.
## Style Detection (When No Style Specified)
Read the full transcript before choosing. Four dimensions:
**1. Visual feel** — corporate→clean; energetic→bold; storytelling→elegant; technical→precise; social→playful.
**2. Color palette** — dark+bright for energy; muted for professional; high contrast for clarity; one accent color.
**3. Font mood** — heavy/condensed for impact; clean sans for modern; rounded for friendly; serif for elegance.
**4. Animation character** — scale-pop for punchy; gentle fade for calm; word-by-word for emphasis; typewriter for technical.
## Per-Word Styling
Scan for words deserving distinct treatment:
- **Brand/product names** — larger size, unique color
- **ALL CAPS** — scale boost, flash, accent color
- **Numbers/statistics** — bold weight, accent color
- **Emotional keywords** — exaggerated animation (overshoot, bounce)
- **Call-to-action** — highlight, underline, color pop
- **Marker highlight** — for beyond-color emphasis (highlight sweep, circle, burst, scribble, sketchout), see `hyperframes-animation/rules/css-marker-patterns.md`.
## Script-to-Style Mapping
| Tone | Font mood | Animation | Color | Size |
| ------------ | ------------------------ | ---------------------------------- | --------------------------- | ------- |
| Hype/launch | Heavy condensed, 800-900 | Scale-pop, back.out(1.7), 0.1-0.2s | Bright on dark | 72-96px |
| Corporate | Clean sans, 600-700 | Fade+slide, power3.out, 0.3s | White/neutral, muted accent | 56-72px |
| Tutorial | Mono/clean sans, 500-600 | Typewriter/fade, 0.4-0.5s | High contrast, minimal | 48-64px |
| Storytelling | Serif/elegant, 400-500 | Slow fade, power2.out, 0.5-0.6s | Warm muted tones | 44-56px |
| Social | Rounded sans, 700-800 | Bounce, elastic.out, word-by-word | Playful, colored pills | 56-80px |
## Word Grouping
- **High energy:** 2-3 words. Quick turnover.
- **Conversational:** 3-5 words. Natural phrases.
- **Measured/calm:** 4-6 words. Longer groups.
Break on sentence boundaries, 150ms+ pauses, or max word count.
## Positioning
- **Landscape (1920x1080):** Bottom 80-120px, centered
- **Portrait (1080x1920):** Lower middle ~600-700px from bottom, centered
- Never cover the subject's face
- `position: absolute` — never relative
- One caption group visible at a time
## Text Overflow Prevention
Use `window.__hyperframes.fitTextFontSize()`:
```js
var result = window.__hyperframes.fitTextFontSize(group.text.toUpperCase(), {
fontFamily: "Outfit",
fontWeight: 900,
maxWidth: 1600,
});
el.style.fontSize = result.fontSize + "px";
```
Options: `maxWidth` (1600 landscape, 900 portrait), `baseFontSize` (78), `minFontSize` (42), `fontWeight`, `fontFamily`, `step` (2).
CSS safety nets: `max-width` on container, `overflow: visible` (**not** `hidden` — hidden clips scaled emphasis words and glow effects), `position: absolute`, explicit `height`. When per-word styling uses `scale > 1.0`, compute `maxWidth = safeWidth / maxScale` to leave headroom.
**Container pattern:** Full-width absolute container, centered. Do **not** use `left: 50%; transform: translateX(-50%)` — causes clipping at composition edges.
## Caption Exit Guarantee
Every group **must** have a hard kill after exit animation:
```js
tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, group.end - 0.12);
// `tl.set` is an instant flip, not a tween — safe to set `visibility` here (core's "no animating
// visibility" rule applies to tweens, which can't smoothly interpolate non-numeric values anyway).
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
```
Self-lint after building timeline — place **before** `window.__timelines[id] = tl` so it runs at composition init:
```js
GROUPS.forEach(function (group, gi) {
var el = document.getElementById("cg-" + gi);
if (!el) return;
tl.seek(group.end + 0.01);
var computed = window.getComputedStyle(el);
if (computed.opacity !== "0" && computed.visibility !== "hidden") {
console.warn(
"[caption-lint] group " + gi + " still visible at t=" + (group.end + 0.01).toFixed(2) + "s",
);
}
});
tl.seek(0);
```
## Pre-Built Caption Components
Before building caption styles from scratch, check the registry — 15 ready-to-use caption components cover the most common styles. Install with `npx hyperframes add <name>` and wire as a sub-composition via `data-composition-src` (see `hyperframes-registry`).
```bash
npx hyperframes catalog --tag caption-style # list all caption components
npx hyperframes add caption-highlight # install a specific one
```
| Style | Component | Best for |
| ------------------------- | ---------------------------- | ---------------------------- |
| TikTok-style highlight | `caption-highlight` | Social, high-energy |
| Karaoke pill | `caption-pill-karaoke` | Music, lyric videos |
| Cinematic editorial | `caption-editorial-emphasis` | Documentary, storytelling |
| Glitch / cyber | `caption-glitch-rgb` | Tech, gaming |
| Full-screen slam | `caption-kinetic-slam` | Hype, announcements |
| Neon glow | `caption-neon-glow` | Night, club, neon aesthetics |
| Neon accent (multi-color) | `caption-neon-accent` | Colorful, playful |
| Wipe reveal | `caption-clip-wipe` | Clean, modern |
| Gradient fill | `caption-gradient-fill` | Vibrant, eye-catching |
| Matrix decode | `caption-matrix-decode` | Sci-fi, tech reveals |
| Emoji pop | `caption-emoji-pop` | Social, casual |
| Parallax layers | `caption-parallax-layers` | Depth, cinematic |
| Particle burst | `caption-particle-burst` | Celebration, impact keywords |
| Lava texture | `caption-texture` | Bold, dramatic |
| Weight shift | `caption-weight-shift` | Elegant, typographic |
Related: `caption-blend-difference` (tagged `text` / `blend-mode`, not `caption-style`, so it won't appear under the filter above) auto-inverts text against any background via `mix-blend-mode: difference` — useful when the background is busy or unpredictable.
Browse all with previews: [hyperframes.heygen.com/catalog](https://hyperframes.heygen.com/catalog)
Caption components ship with transparent backgrounds — they're pure overlays. If the underlying video is bright or busy, add a contrast layer (e.g. a semi-transparent dark div) in the host composition beneath the caption sub-composition, not inside the component itself.
## Further References
- [`motion.md`](motion.md) — karaoke, marker effects, audio-reactive modulation, scatter exits.
- [`transcript-handling.md`](transcript-handling.md) — input formats, quality checks, cleaning, external API fallback.
- `hyperframes-animation/rules/css-marker-patterns.md` — marker highlighting (deterministic, fully seekable).
## Constraints
- Deterministic. No `Math.random()`, no `Date.now()`.
- Sync to transcript timestamps.
- One group visible at a time.
- Every group must have a hard `tl.set` kill at `group.end`.
- Fonts: the compiler auto-embeds only its **built-in mapped set** (Inter, Roboto, Montserrat, …) — for those, just declare `font-family` in CSS. Any **other** font (a brand/custom font like `TT Norms Pro`, or a non-Latin CJK/Devanagari family) is **not** auto-supplied: it needs an `@font-face` pointing at a real `.woff2` shipped with the project, or the text silently falls back to a generic font in the render. Don't assume a `font-family` you can see locally will render — the render machine is a clean headless Chrome with no installed fonts.
@@ -0,0 +1,87 @@
# Dynamic Caption Techniques
You are here because SKILL.md told you to read this file before writing animation code. Pick your technique combination from the table below based on the energy level you detected from the transcript, then implement using standard GSAP patterns.
## Technique Selection by Energy
| Energy level | Highlight | Exit | Cycle pattern |
| ------------ | ------------------------------------- | ------------------- | ----------------------------------------- |
| High | Karaoke with accent glow + scale pop | Scatter or drop | Alternate highlight styles every 2 groups |
| Medium-high | Karaoke with color pop | Scatter or collapse | Alternate every 3 groups |
| Medium | Karaoke (subtle, white only) | Fade + slide | Alternate every 3 groups |
| Medium-low | Karaoke (minimal scale change) | Fade | Single style, vary ease per group |
| Low | Karaoke (warm tones, slow transition) | Collapse | Alternate every 4 groups |
**All energy levels use karaoke highlight as the baseline.** The difference is intensity — high energy gets accent color + glow + 15% scale pop on active words, low energy gets a gentle white shift with 3% scale.
**Emphasis words always break the pattern.** When a word is flagged as emphasis (emotional keyword, ALL CAPS, brand name), give it a stronger animation than surrounding words (larger scale, accent color, overshoot ease). This creates contrast.
**Marker highlight modes add a visual layer on top of karaoke.** For emphasis words that need more than color/scale, add a marker-style effect: highlight sweep, circle, burst, scribble, or sketchout. See `hyperframes-animation/rules/css-marker-patterns.md` for implementation details. Match mode to energy: burst for hype, circle for key terms, highlight for standard, scribble for subtle.
## Audio-Reactive Captions (Mandatory for Music)
**If the source audio is music (vocals over instrumentation, beats, any musical content), you MUST extract audio data and add audio-reactive animations.** This is not optional — music without audio reactivity looks disconnected. Even low-energy ballads get subtle bass pulse and treble glow.
No special wiring is needed. The group loop already iterates over every caption group to build entrance, karaoke, and exit tweens. At that point, read the audio data for each group's time range and use it to modulate the group's animation intensity with regular GSAP tweens.
```js
// Load audio data inline (same pattern as TRANSCRIPT)
var AUDIO = JSON.parse(audioDataJson); // { fps, totalFrames, frames: [{ bands: [...] }] }
GROUPS.forEach(function (group, gi) {
var groupEl = document.getElementById("cg-" + gi);
if (!groupEl) return;
// Read peak energy for this group's time range
var startFrame = Math.floor(group.start * AUDIO.fps);
var endFrame = Math.min(Math.floor(group.end * AUDIO.fps), AUDIO.totalFrames - 1);
var peakBass = 0;
var peakTreble = 0;
for (var f = startFrame; f <= endFrame; f++) {
var frame = AUDIO.frames[f];
if (!frame) continue;
peakBass = Math.max(peakBass, frame.bands[0] || 0, frame.bands[1] || 0);
peakTreble = Math.max(peakTreble, frame.bands[6] || 0, frame.bands[7] || 0);
}
// Modulate entrance — louder groups enter bigger and glowier
tl.to(
groupEl,
{
scale: 1 + peakBass * 0.06,
textShadow:
"0 0 " + Math.round(peakTreble * 12) + "px rgba(255,255,255," + peakTreble * 0.4 + ")",
duration: 0.3,
ease: "power2.out",
},
group.start,
);
// Reset at exit so audio-driven values don't persist
tl.set(groupEl, { scale: 1, textShadow: "none" }, group.end - 0.15);
});
```
This shapes the animation at build time, not playback time — no per-frame callbacks, no `tl.call()` loops, no async fetch timing issues. Loud groups come in with more weight and glow; quiet groups come in soft. The audio data modulates _how much_, the content determines _what_.
Keep audio reactivity subtle — 3-6% scale variation and soft glow. Heavy pulsing makes text unreadable.
To generate the audio data file:
```bash
python3 skills/hyperframes-creative/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json
```
## Combining Techniques
Don't use the same highlight animation on every group — cycle through styles using the group index. Don't combine multiple competing animations on the same word at the same timestamp. Vary techniques across groups to match the content's pace changes.
**Marker highlight effects** layer well with karaoke — use karaoke for the word-by-word reveal, then add a marker effect on emphasis words only. For example: karaoke highlights each word in white, but brand names get a yellow highlight sweep and stats get a red circle. Cycle marker modes across groups for visual variety.
## Runtime Tools
Caption motion uses standard HyperFrames runtime APIs. Use the canonical sources:
- **GSAP timeline + tween syntax** — `hyperframes-animation/adapters/gsap.md` (eases, position parameter, performance)
- **`window.__hyperframes.fitTextFontSize` / `pretext`** — `hyperframes-core/references/determinism-rules.md` → Layout Contract (overflow prevention, per-frame text measurement)
- **Audio data extraction** — generate via `python3 skills/hyperframes-creative/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json`, then load inline as shown in "Audio-Reactive Captions" above
@@ -0,0 +1,97 @@
# Transcript Guide
For the `transcribe` CLI invocation, the `.en`-translates-non-English rule, and whisper model selection, see [`../transcribe.md`](../transcribe.md). This file covers what to do with the resulting transcript when authoring captions: input formats, mandatory quality checks, cleaning code, external-API fallbacks.
## Supported Input Formats
The CLI auto-detects and normalizes these formats:
| Format | Extension | Source | Word-level? |
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
## Transcript Quality Check (Mandatory)
After every transcription, **read the transcript and check for quality issues before proceeding.** Bad transcripts produce nonsensical captions. Never skip this step.
### What to look for
| Signal | Example | Cause |
| ---------------------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
| Music note tokens (`♪`, ``) | `{ "text": "♪" }` or `{ "text": "" }` | Whisper detected music, not speech |
| Garbled / nonsense words | "Do a chin", "Get so gay", "huh" | Model misheard lyrics or background noise |
| Long gaps with no words | 20+ seconds of only `♪` tokens | Instrumental section — expected, but high ratio means speech is being missed |
| Repeated filler | Many "huh", "uh", "oh" entries | Model is hallucinating on music |
| Very short word spans | Words with `end - start < 0.05` | Unreliable timestamp alignment |
### Automatic retry rules
**If more than 20% of entries are `♪`/`` tokens, or the transcript contains obvious nonsense words, the transcription failed.** Do not proceed with the bad transcript. Instead:
1. **Retry with `medium.en`** if the original used `small.en` or smaller:
```bash
npx hyperframes transcribe audio.mp3 --model medium.en
```
2. **If `medium.en` also fails** (still >20% music tokens or garbled), tell the user the audio is too noisy for local transcription and suggest:
- Providing lyrics manually as an SRT/VTT file
- Using an external API (OpenAI or Groq Whisper — see below)
3. **Always clean the transcript** before building captions — filter out ``/`` tokens and entries where `text` is a single non-word character. Only real words should reach the caption composition.
### Cleaning a transcript
After transcription (even with a good model), strip non-word entries:
```js
var raw = JSON.parse(transcriptJson);
var words = raw.filter(function (w) {
if (!w.text || w.text.trim().length === 0) return false;
if (/^[♪\u266a\u266b\u266c\u266d\u266e\u266f]+$/.test(w.text)) return false;
if (/^(huh|uh|um|ah|oh)$/i.test(w.text) && w.end - w.start < 0.1) return false;
return true;
});
```
For model-selection guidance by content type, see [`../transcribe.md`](../transcribe.md) → "Picking a model by content type".
## Using External Transcription APIs
For the best accuracy, use an external API and import the result:
**OpenAI Whisper API** (recommended for quality):
```bash
# Generate with word timestamps, then import
curl https://api.openai.com/v1/audio/transcriptions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F file=@audio.mp3 -F model=whisper-1 \
-F response_format=verbose_json \
-F "timestamp_granularities[]=word" \
-o transcript-openai.json
npx hyperframes transcribe transcript-openai.json
```
**Groq Whisper API** (fast, free tier available):
```bash
curl https://api.groq.com/openai/v1/audio/transcriptions \
-H "Authorization: Bearer $GROQ_API_KEY" \
-F file=@audio.mp3 -F model=whisper-large-v3 \
-F response_format=verbose_json \
-F "timestamp_granularities[]=word" \
-o transcript-groq.json
npx hyperframes transcribe transcript-groq.json
```
## If No Transcript Exists
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files.
2. If none found, run [`../transcribe.md`](../transcribe.md) — pick the starting model from "Picking a model by content type" there.
3. Run the quality check above. If it fails, retry with a larger model or fall back to manual lyrics / external API.
@@ -0,0 +1,143 @@
# Background Removal
Make a transparent overlay (typical: a talking head over an arbitrary scene). Uses `u2net_human_seg` (MIT).
```bash
npx hyperframes remove-background subject.mp4 -o transparent.webm # default: VP9 + alpha
npx hyperframes remove-background subject.mp4 -o transparent.mov # ProRes 4444 (editing)
npx hyperframes remove-background portrait.jpg -o cutout.png # single-image cutout
npx hyperframes remove-background subject.mp4 -o subject.webm \
--background-output plate.webm # both layers, one pass
npx hyperframes remove-background subject.mp4 -o transparent.webm --device cpu
npx hyperframes remove-background --info # detected providers
```
## Output Format
- **`.webm` (VP9 alpha)** — default. Plug straight into `<video>` for Chrome-native transparent playback (~1 MB / 4s @ 1080p).
- **`.mov` (ProRes 4444)** — round-trip in editors (Premiere / Resolve / DaVinci). ~50 MB / 4s.
- **`.png`** — single-image cutout.
## Quality (`--quality`)
Controls VP9 encoder CRF only — segmentation quality is fixed. Higher quality keeps the cutout's RGB closer to the source MP4 (important when overlaying the cutout on its own source).
| Preset | CRF | When |
| ---------- | --- | --------------------------------------------- |
| `fast` | 30 | Iterating, smaller files, looser color match |
| `balanced` | 18 | **Default**; visually identical for most uses |
| `best` | 12 | Master / final delivery, tightest color match |
## Device (`--device`)
`auto` (default) picks CoreML on Apple Silicon, CUDA when available, otherwise CPU. Force with `--device cpu | coreml | cuda`. CUDA requires `HYPERFRAMES_CUDA=1` plus a GPU-enabled `onnxruntime-node` build. Use `--info` to inspect detected providers without rendering.
## Compositing patterns — pick the right one
The cutout WebM is a **re-encoded copy** of the source MP4's RGB. What sits behind it matters.
| Pattern | Behind the cutout | Result |
| -------------------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Cutout over a different scene** (most common) | Static image, gradient, unrelated video | Looks great. Single RGB source for the subject. |
| **Cutout over its own source mp4** (text-behind-subject) | Same mp4 the cutout came from | At `balanced` doubling is barely visible; at `fast` you'll see color shift / edge halo. Use `best` for masters. |
| **Cutout over a different take of the same person** | Footage of the same subject | **Two overlapping people. Don't do this.** |
## Text-behind-subject pattern (two non-obvious rules)
Putting a headline behind a presenter cutout:
```html
<video
src="presenter.mp4"
id="bg"
data-start="0"
data-duration="6"
data-track-index="0"
muted
playsinline
></video>
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
<div class="cutout-wrap" style="position:absolute; inset:0; z-index:3; opacity:0">
<video
src="presenter.webm"
data-start="0"
data-duration="6"
data-track-index="1"
muted
playsinline
></video>
</div>
```
```js
// Flip the wrapper's opacity at the cut, NOT the video's
tl.set(".cutout-wrap", { opacity: 1 }, 3.3);
```
Two rules that are easy to miss:
1. **Wrap the cutout `<video>` in a non-timed `<div>` and animate the wrapper's opacity, not the video element's.** The framework forces `opacity: 1` on active clips (any element with `data-start` / `data-duration`), so animating the video's opacity directly is silently overridden. The wrapper has no `data-*` attributes, so it's owned by your CSS / GSAP.
2. **Both videos use `data-start="0"` and `data-media-start="0"`** so the framework decodes them in sync from t=0. Late-mounting the cutout (`data-start=3.3`) introduces a seek + warm-up that lands a frame off the base mp4 — visible as one frame of misalignment at the cut.
## Layer separation (`--background-output`)
Emits a **second** transparent video alongside the cutout: same source RGB, alpha is `255 - mask` instead of `mask`. The cutout has the subject opaque; the plate has the surroundings opaque (with a transparent hole where the subject was). Use it when text / graphics need to live **between** the two layers.
| File | Alpha is… | Use it for |
| -------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| `-o subject.webm` | mask — subject opaque, background transparent | Foreground layer (top) |
| `--background-output plate.webm` | inverse mask — surroundings opaque, subject transparent | Bottom layer; place text / graphics between this and the subject |
Both share the same `--quality` and run from a single inference pass — only encode cost roughly doubles. Only valid for video inputs with `.webm` / `.mov` outputs.
**Hole-cut, not inpainted.** The subject region in `plate.webm` is fully transparent — composite something opaque under it to fill the hole.
**Single test for whether `--background-output` is the right tool:** _will anything ever be visible through the subject's silhouette where the subject used to be?_ If no, you don't need the plate — `subject.webm` alone over a different background is enough.
### Use case → right tool
| Use case | Right tool |
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Text/graphics between the cutout and the plate (this command's reason for existing) | **Hole-cut** (`--background-output`) |
| Subject onto an unrelated scene | Just `subject.webm`; ignore the plate |
| Show the room _without_ the person, alone over no other content | **Clean plate** — needs an inpainter (LaMa, ProPainter, E2FGVI). Not this command. |
| Replace the subject with a different subject | **Clean plate** — same as above |
### Canonical 3-layer template (plate + content + cutout)
Ship just the two transparent layers and let arbitrary content live between them — no original mp4 needed:
```html
<!-- z=1 plate: surroundings opaque, subject silhouette transparent -->
<video
src="plate.webm"
data-start="0"
data-duration="6"
data-track-index="0"
muted
playsinline
></video>
<!-- z=2 your content lives between the layers -->
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
<!-- z=3 cutout floats the subject back on top -->
<div class="cutout-wrap" style="position:absolute; inset:0; z-index:3">
<video
src="subject.webm"
data-start="0"
data-duration="6"
data-track-index="1"
muted
playsinline
></video>
</div>
```
Functionally equivalent to the text-behind-subject pattern above, but doesn't require shipping the original mp4 — the plate replaces it. Use this when delivering just the two transparent layers as a reusable asset.
## When `remove-background` is NOT the right tool
If a user asks for "the room **without** the person, displayed standalone" (no subject anywhere, no compositing on top), `--background-output` is wrong — its plate has a transparent hole, not a filled-in clean plate. They need an **inpainter**: LaMa, ProPainter, or E2FGVI. Tell them this command can't do it.
@@ -0,0 +1,29 @@
# Requirements & Caches
## Credential & key priority
Run `npx hyperframes auth status` to see what's configured and which engines a workflow will use (see the skill's **Preflight** section). Keys resolve in this order — **first match wins**:
| Provider | Resolution order (first non-empty wins) | Local deps when used |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| **HeyGen** (TTS + BGM/SFX retrieval) | `$HEYGEN_API_KEY``$HYPERFRAMES_API_KEY``~/.heygen/credentials` (shared with heygen-cli; `$HEYGEN_CONFIG_DIR` overrides the dir; written by `hyperframes auth login`) | none (REST) |
| **ElevenLabs** (TTS fallback) | `$ELEVENLABS_API_KEY` | `pip install elevenlabs` |
| **Lyria** (BGM fallback) | `$GEMINI_API_KEY``$GOOGLE_API_KEY` | `pip install google-genai` |
| **Kokoro** (TTS, no key) | always — final voice fallback | `pip install kokoro-onnx soundfile` |
| **MusicGen** (BGM, no key) | always — final music fallback | `pip install transformers torch soundfile numpy` |
`hyperframes auth login` (browser OAuth) is the recommended setup: one sign-in, every project, no per-repo `.env`. An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both are tagged with `X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance for HeyGen TTS (10 min/month); API keys follow the normal API billing path. With no HeyGen credential, voice/BGM run fully locally (Kokoro / MusicGen) — `hyperframes auth status` and `hyperframes doctor` both report whether those local deps are installed.
## Model caches & system dependencies
Each command downloads its own model on first run and caches it under `~/.cache/hyperframes/`:
- **TTS (HeyGen)** — no local deps; needs a HeyGen credential + `ffmpeg` on PATH (to transcode the mp3 response to `.wav`). Credential resolves like the CLI: `$HEYGEN_API_KEY``$HYPERFRAMES_API_KEY``~/.heygen/credentials` (shared with heygen-cli; run `npx hyperframes auth login`). An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both include `X-HeyGen-Source: cli` so the backend can apply CLI OAuth free usage.
- **TTS (ElevenLabs)** — same as HeyGen: API key + `ffmpeg`.
- **TTS (Kokoro)** — Kokoro-82M (~311 MB) + voices (~27 MB) in `tts/`. Requires Python 3.8+ with `kokoro-onnx` and `soundfile` (`pip install kokoro-onnx soundfile`). Non-English text also needs `espeak-ng` system-wide.
- **BGM (Lyria)** — needs `$GEMINI_API_KEY` or `$GOOGLE_API_KEY` + `pip install google-genai`. No local model cache.
- **BGM (MusicGen)** — `pip install transformers torch soundfile`. `facebook/musicgen-small` (~300 MB) cached under `~/.cache/huggingface/` on first run.
- **Transcribe** — Whisper model size depending on choice (75 MB 3.1 GB) in `whisper/`. Bundles `whisper.cpp`.
- **Remove-background** — `u2net_human_seg` (~168 MB ONNX) in `background-removal/models/`. Peak inference RAM ~1.5 GB.
Run `npx hyperframes doctor` if a command fails because of a missing dependency.
+42
View File
@@ -0,0 +1,42 @@
# Sound effects (SFX)
Named sound effects, produced by the shared audio engine (`scripts/audio.mjs``scripts/lib/sfx.mjs`). **Provider-gated** by the engine's one switch — whether a HeyGen credential is present, decided once (not per cue):
- **HeyGen credential present → retrieve every cue** from HeyGen's audio library (`/v3/audio/sounds`, `type=sound_effects`, `min_score=0.4`). Search-and-download, **not** generation. The bundled library is NOT consulted.
- **No credential → the bundled 21-file library** (`assets/sfx/` + `manifest.json`): match each cue name, copy the matched file into the project. Offline, deterministic, free.
There is no `npx hyperframes sfx` command. SFX is never generated — it is retrieved (online) or taken from the bundled library (offline).
## Cues — request → meta
Each line names the effects it wants: `lines[].sfx: ["whoosh", "ui click"]`. The engine flattens these into cues, resolves them per the switch, dedupes identical `(id, name)` pairs (the same effect named twice downloads/copies once), and writes `audio_meta.sfx[]`:
```jsonc
{
"id": "3", // joins the cue to the caller's model (frame / scene / segment)
"name": "whoosh",
"file": "assets/sfx/whoosh.mp3", // downloaded or copied, relative to project root
"source": "heygen" | "local", // which route resolved it
"offset_s": 0, // delay from the line's start
"duration_s": 0.57,
"volume": 0.35 // SFX sit UNDER voice + BGM
}
```
A cue that matches nothing is **skipped** (recorded as an anomaly); SFX never blocks a render.
## HeyGen retrieval (credentialed)
`searchSounds(name, "sound_effects", { limit: 3, minScore: 0.4 })` → top hit → `assets/sfx/<slug>.mp3`. Results are ranked by `score` (each carries a presigned `audio_url`, `duration`, `description`). The floor is **0.4** because good SFX hits score ~0.50.67 — below the API's default `0.7`, which would silently drop most named cues (only whoosh/swoosh-family clears 0.7). `duration_s` comes from the result (else 1.0). Name effects concretely (`glass shatter`, not `dramatic sound`); a vague query returns a poor match.
## Bundled library (no credential)
21 curated files in `assets/sfx/`, indexed by `manifest.json``{ file, duration, description }` per key (e.g. `whoosh`, `pop`, `click`, `chime`, `riser`, `impact-bass-1`, `glitch-1`, `typing`, …). A cue name resolves by **manifest key, file basename, or slug**, so `whoosh`, `whoosh.mp3`, or `"ui click"` (→ slug) all match. Matched files are copied into the project's `assets/sfx/`; `duration_s` comes from the manifest, so timing is known **offline** — e.g. `riser` is 10.03s, so trigger it at `climax 10.03s`. The manifest's `description` field carries placement hints per effect; read `assets/sfx/manifest.json` for the full set and usage.
## Rules
- **Volume ~0.35.** SFX must sit under narration and BGM, not fight them.
- **No match → skip, don't fail.** A missing effect logs an anomaly and moves on; never a render blocker.
- **Retrieval (credentialed) or bundled library (offline) — never generation.** You search HeyGen by text, or match a name against the 21-file manifest.
- **One asset per distinct name.** Reuse across lines is deduped to a single download/copy, many cues.
- **The switch is global, not per cue.** With a credential, retrieval handles even the long tail (effects not in the 21); without one, only the 21 bundled names resolve.
@@ -0,0 +1,52 @@
# Transcription
Create normalized word-level timestamps. **Always specify `--model` explicitly** — the CLI default is `small.en`, which silently translates non-English audio into English.
```bash
npx hyperframes transcribe audio.mp3 --model small.en # known English
npx hyperframes transcribe video.mp4 --model small --language es # known Spanish
npx hyperframes transcribe audio.mp3 --model small # unknown language (auto-detect)
npx hyperframes transcribe subtitles.srt # import existing
npx hyperframes transcribe subtitles.vtt
npx hyperframes transcribe openai-response.json
```
## Language Rule (Non-Negotiable)
`.en` models (`tiny.en` / `base.en` / `small.en` / `medium.en`) **translate** non-English audio into English. This silently destroys the original language.
1. **Known English**`--model small.en` (or `medium.en` for music / noisy audio)
2. **Known non-English**`--model small --language <iso-code>` (no `.en` suffix)
3. **Unknown language**`--model small` (whisper auto-detects)
**CLI default is `small.en`** — do not rely on it; always pass `--model` to make the choice explicit. `--language` also filters out non-target-language segments from mixed-language audio.
## Model Sizes
| Model | Size | Speed | When |
| ---------- | ------ | -------- | ------------------------------------- |
| `tiny` | 75 MB | Fastest | Quick previews, smoke tests |
| `base` | 142 MB | Fast | Short clips, clear audio |
| `small` | 466 MB | Moderate | Default for most multilingual content |
| `medium` | 1.5 GB | Slow | Music with vocals, noisy audio |
| `large-v3` | 3.1 GB | Slowest | Production quality |
### Picking a model by content type
1. Speech over silence / light background → `small.en`
2. Speech over music, or music with vocals → start with `medium.en`
3. Produced music track (vocals + full instrumentation) → start with `medium.en`; expect to need manual lyrics or an external API ([`captions/transcript-handling.md`](captions/transcript-handling.md) → "Using External Transcription APIs")
4. Multilingual → `medium` or `large-v3` (no `.en` suffix), pair with `--language`
## Output Shape
Compositions consume a flat array of word objects. The `id` (`w0`, `w1`, …) is added during normalization for stable references in caption overrides; optional for backwards compatibility.
```json
[
{ "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
{ "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
]
```
For mandatory caption-quality checks, retry rules, and the OpenAI/Groq Whisper API import path, see `captions/transcript-handling.md`.
@@ -0,0 +1,24 @@
# TTS → Captions
When no recorded voiceover exists, generate one and obtain word-level caption timing. Two paths depending on which TTS provider is in use:
## Path A — HeyGen (single call, no Whisper)
HeyGen returns word timestamps in the same response as the audio. Pass `--words` and you're done:
```bash
npx hyperframes tts script.txt --provider heygen --output narration.wav --words narration.words.json
```
`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.
## Path B — ElevenLabs / Kokoro (TTS → Whisper)
These providers don't return word data. Generate the audio, then transcribe:
```bash
npx hyperframes tts script.txt --voice af_heart --output narration.wav
npx hyperframes transcribe narration.wav --model small.en # voice af_heart is American English
```
Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `small --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
+138
View File
@@ -0,0 +1,138 @@
# Text To Speech
`npx hyperframes tts` auto-detects a provider from env vars; explicit override via `--provider`.
> **Run the Preflight first — no credential is not a green light to silently use the local voice.** Before generating a voiceover, complete the sign-in **Preflight** (see `../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen voices, or continue offline with local Kokoro). This applies to a one-off "generate a voiceover" request just as much as inside a full workflow.
## Provider chain
| Order | Provider | Env trigger | Voice IDs | Word timestamps | Audio format |
| ----- | ----------------- | ------------------------------------------- | ------------------------------------------- | ----------------------------------------- | -------------------- |
| 1 | HeyGen (Starfish) | `$HEYGEN_API_KEY` / `~/.heygen/credentials` | UUIDs from `GET /v3/voices?engine=starfish` | **Yes** (`word_timestamps[]` in response) | mp3 → wav via ffmpeg |
| 2 | ElevenLabs | `$ELEVENLABS_API_KEY` | UUIDs from elevenlabs.io dashboard | No | mp3 → wav via ffmpeg |
| 3 | Kokoro-82M | always (local fallback) | `am_michael`, `af_heart`, … (54 voices) | No | wav direct |
```bash
# Auto-detect (HeyGen if key set, else ElevenLabs, else Kokoro)
npx hyperframes tts "Welcome to HyperFrames" -o narration.wav
# Pin the provider explicitly
npx hyperframes tts "Hello" --provider kokoro
npx hyperframes tts "Hello" --provider heygen --voice <heygen-uuid>
npx hyperframes tts "Hello" --provider elevenlabs --voice 21m00Tcm4TlvDq8ikWAM
# HeyGen path: capture word timestamps in one call (skips a Whisper pass)
npx hyperframes tts "Hi there" --words narration.words.json
```
## Self-contained HeyGen (no CLI) — `scripts/heygen-tts.mjs`
The published `hyperframes tts` CLI synthesizes locally with Kokoro only. When you
want HeyGen specifically — best quality **plus** word timestamps in one call — use
the skill's bundled script, which calls the HeyGen v3 REST API directly and needs
no CLI provider plumbing:
The script resolves a HeyGen credential the same way the CLI does — first source
wins: `$HEYGEN_API_KEY``$HYPERFRAMES_API_KEY` → a project `.env` (auto-loaded,
walks up ≤5 dirs) → `~/.heygen/credentials` (shared with heygen-cli;
`$HEYGEN_CONFIG_DIR` overrides the dir). An OAuth login is sent as
`Authorization: Bearer`; an API key as `X-Api-Key`; both include
`X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance
(10 min/month) before paid usage; API keys follow normal API billing. If the
only credential is an expired OAuth token it stops with a hint to run
`npx hyperframes auth refresh`.
```bash
# Only needed if you haven't run `npx hyperframes auth login`:
export HEYGEN_API_KEY=... # or put it in a project .env
# Synthesize + capture word timestamps in one call (skips a Whisper pass)
node skills/media-use/audio/scripts/heygen-tts.mjs \
"Welcome to HyperFrames." -o narration.wav --words narration.words.json
node skills/media-use/audio/scripts/heygen-tts.mjs ./script.txt -o narration.wav
node skills/media-use/audio/scripts/heygen-tts.mjs --list # public starfish voices
```
- **Voice:** `--voice <id>` must be a **starfish** voice_id (`--list`, or `GET /v3/voices?engine=starfish`). v2-catalog ids are rejected with HTTP 400. Omit `--voice` (English) and it defaults to **Marcia** (`05f19352e8f74b0392a8f411eba40de1`, a fixed default so the choice is deterministic). Non-English with no `--voice` falls back to the first matching catalog voice.
- **Output:** `.wav` → transcoded to 44.1k mono via ffmpeg; `.mp3` → raw bytes (no ffmpeg needed).
- **Words:** `--words <path>` writes the flat `[{id,text,start,end}]` shape below, drop-in for the captions pipeline. HeyGen's `<start>`/`<end>` boundary sentinels are filtered out and ids are re-contiguous.
- **Non-English:** `--lang <code>` (anything but `en`) is sent as the request `language`.
## When to use which provider
| Goal | Use |
| --------------------------------------------------------- | --------------------------------------------------- |
| Best voice quality + word timestamps in one call | **HeyGen** |
| Drop-in cloud TTS, big voice catalog | **ElevenLabs** |
| Offline, no API key, fast iteration | **Kokoro** |
| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …) |
## ffmpeg requirement
HeyGen + ElevenLabs return mp3. The CLI transcodes to wav when `--output` ends in `.wav` (the default and what downstream `ffprobe` + Whisper expect). If you'd rather skip the transcode, pass `-o file.mp3`. Without `ffmpeg` on PATH, `.wav` output from the cloud providers fails — install ffmpeg or use `.mp3`.
## Voice selection (Kokoro)
Default `af_heart`. Curated picks:
| Content type | Voice |
| ----------------- | ---------------------- |
| Product demo | `af_heart`, `af_nova` |
| Tutorial / how-to | `am_adam`, `bf_emma` |
| Marketing / promo | `af_sky`, `am_michael` |
| Documentation | `bf_emma`, `bm_george` |
| Casual / social | `af_heart`, `af_sky` |
Run `npx hyperframes tts --list` for the bundled set.
## Multilingual (Kokoro voice prefix → language)
The first letter of a Kokoro voice ID picks the phonemizer language; `--lang` overrides auto-detection.
| Prefix | Language |
| ------ | -------------------- |
| `a` | American English |
| `b` | British English |
| `e` | Spanish |
| `f` | French |
| `h` | Hindi |
| `i` | Italian |
| `j` | Japanese |
| `p` | Brazilian Portuguese |
| `z` | Mandarin |
```bash
npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora --provider kokoro
npx hyperframes tts "Today is a nice day" --voice af_heart --provider kokoro
```
Valid `--lang` codes (only needed to override the voice's auto-detected language): `en-us`, `en-gb`, `es`, `fr-fr`, `hi`, `it`, `pt-br`, `ja`, `zh`.
Non-English phonemization requires `espeak-ng` system-wide (`brew install espeak-ng` / `apt-get install espeak-ng`).
## Speed
- `0.7-0.8` — tutorial, complex content, accessibility
- `1.0` — natural pace (default)
- `1.1-1.2` — intros, transitions, upbeat content
- `1.5+` — rarely appropriate, test carefully
Honored by Kokoro + HeyGen; ElevenLabs ignores `--speed` (use voice settings on their dashboard).
## Long scripts
Past a few paragraphs, write the text to a `.txt` file and pass the path. Inputs over ~5 minutes of speech may benefit from splitting into segments.
## HeyGen word-timestamp shape
When `--words <path>` is passed to a HeyGen call, the file is written in the same flat shape `transcribe` produces — drop-in compatible with the captions pipeline:
```json
[
{ "id": "w0", "text": "Hi", "start": 0.0, "end": 0.21 },
{ "id": "w1", "text": "there", "start": 0.22, "end": 0.55 }
]
```
For ElevenLabs / Kokoro, run `npx hyperframes transcribe narration.wav --model small.en` to get the same shape.
+293
View File
@@ -0,0 +1,293 @@
#!/usr/bin/env node
// audio.mjs — the shared HyperFrames audio engine. ONE implementation of TTS +
// BGM + SFX for every video workflow (product-launch, general-video, pr-to-video,
// …). Workflows do NOT vendor a copy: they write a neutral `audio_request.json`
// (a tiny per-workflow adapter maps their storyboard/scenes into it) and call:
//
// node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json
//
// The three capabilities degrade on ONE switch — whether HeyGen is configured
// (credential present, NOT the CLI). This mirrors the table in ../SKILL.md:
//
// TTS : HeyGen REST → ElevenLabs → Kokoro (CLI)
// BGM : HeyGen retrieve → (no credential) Lyria/MusicGen generate
// SFX : HeyGen retrieve → (no credential) bundled 19-file library
//
// ── audio_request.json (input) ────────────────────────────────────────────────
// {
// "provider": "auto", // auto|heygen|elevenlabs|kokoro (override: --provider)
// "lang": "en", "speed": 1.0,
// "lines": [ // one TTS unit each; id joins back to the caller's model
// { "id": "01", "text": "...", "sfx": ["whoosh", "ui click"] }
// ],
// "bgm": { "mode": "retrieve", // retrieve|generate|none (override: --bgm-mode / --no-bgm)
// "query": "calm cinematic underscore", // mood for retrieval
// "prompt": null, // full prompt for generation (else inferred)
// "blob": "...", "archetype": "...", "arc": "..." } // optional mood-inference hints
// }
//
// ── audio_meta.json (output, id-keyed) ───────────────────────────────────────
// { tts_provider, voice_id,
// bgm: { path, volume, mode, query?, duration_s? } | null,
// bgm_pending, bgm_provider, bgm_pid, bgm_log, bgm_mode, bgm_target_duration_s, …,
// voices: [ { id, path, duration_s, words: [{id,text,start,end}] } ],
// sfx: [ { id, name, file, source, offset_s, duration_s, volume } ],
// total_duration_s }
//
// --only tts,bgm,sfx runs a subset and MERGES into an existing --out (so a
// workflow can do TTS+BGM early, then SFX later once cues exist). When BGM uses
// the generate path it is spawned detached (bgm_pending:true) — run wait-bgm.mjs
// before assembling.
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { heygenAuthHeaders, heygenCredential, loadEnvFromDir } from "./lib/heygen.mjs";
import {
ffprobeDuration,
pickProvider,
resolveVoiceId,
synthesizeOne,
transcribeWav,
withWordIds,
} from "./lib/tts.mjs";
import { generateBgmDetached, inferBgmPrompt, retrieveBgm } from "./lib/bgm.mjs";
import { resolveSfx } from "./lib/sfx.mjs";
import { mapWithConcurrency } from "./lib/concurrency.mjs";
const HERE = dirname(fileURLToPath(import.meta.url));
const argv = process.argv.slice(2);
const flag = (name, def) => {
const i = argv.indexOf(`--${name}`);
return i >= 0 && i + 1 < argv.length ? argv[i + 1] : def;
};
const has = (name) => argv.includes(`--${name}`);
const die = (m) => {
console.error(`✗ audio engine: ${m}`);
process.exit(1);
};
const r3 = (x) => Number(x.toFixed(3));
// Two independent reports of an unbounded Promise.all over TTS lines
// overwhelming a machine: one OOM'd 12/13 concurrent Kokoro TTS +
// whisper-transcribe lines on a resource-constrained laptop, the other saw
// 7/8 lines fail on first run (concurrent cold-start model loads) and pass on
// retry once the model was cached. Kokoro/Whisper each load their own local
// model per subprocess, so firing every line at once multiplies that cost by
// the line count. mapWithConcurrency caps how many run at once — still
// parallel, just bounded.
const ttsConcurrency = Math.max(1, Number(process.env.HYPERFRAMES_TTS_CONCURRENCY) || 4);
const hyperframesDir = resolve(flag("hyperframes", "."));
const requestPath = resolve(flag("request", join(hyperframesDir, "audio_request.json")));
const outPath = resolve(flag("out", join(hyperframesDir, "audio_meta.json")));
const sfxLibDir = resolve(flag("sfx-lib", join(HERE, "..", "assets", "sfx")));
const lyriaRecipe = resolve(flag("lyria-recipe", join(HERE, "lyria-recipe.py")));
const onlyArg = flag("only", "tts,bgm,sfx");
const only = new Set(
onlyArg
.split(",")
.map((s) => s.trim())
.filter(Boolean),
);
const providerOverride = flag("provider", null);
const bgmModeOverride = flag("bgm-mode", null);
const noBgm = has("no-bgm");
const voiceOverride = flag("voice", null);
const speedOverride = flag("speed", null);
const langOverride = flag("lang", null);
const seedSeconds = Number(flag("seed-seconds", "28")) || 28;
if (!existsSync(requestPath)) die(`audio_request.json not found at ${requestPath}`);
let request;
try {
request = JSON.parse(readFileSync(requestPath, "utf8"));
} catch (e) {
die(`audio_request.json parse: ${e.message}`);
}
const lines = Array.isArray(request.lines) ? request.lines : [];
const lang = langOverride || request.lang || "en";
const speed = Number(speedOverride ?? request.speed ?? 1.0) || 1.0;
// ── env + HeyGen availability (the single switch) ─────────────────────────────
loadEnvFromDir(hyperframesDir);
const heygenOK = heygenCredential() !== null;
const headers = heygenOK ? heygenAuthHeaders() : null;
// ── merge base: preserve sections not selected by --only ──────────────────────
const prev = existsSync(outPath) ? JSON.parse(readFileSync(outPath, "utf8")) : {};
const anomalies = [];
// ── TTS ───────────────────────────────────────────────────────────────────────
let voices = prev.voices ?? [];
let ttsProvider = prev.tts_provider ?? null;
let voiceId = prev.voice_id ?? null;
if (only.has("tts") && lines.length) {
try {
ttsProvider = pickProvider(
providerOverride || (request.provider === "auto" ? null : request.provider),
);
} catch (e) {
die(e.message);
}
voiceId = await resolveVoiceId({
provider: ttsProvider,
userVoice: voiceOverride || request.voice,
lang,
});
console.error(`· tts: ${ttsProvider} · voice ${voiceId} · ${lines.length} line(s)`);
const synthLine = async (line) => {
const id = String(line.id);
const text = String(line.text ?? "").trim();
if (!text) {
anomalies.push(`line ${id}: empty text — skipped`);
return null;
}
const rel = `assets/voice/${id}.wav`;
const abs = join(hyperframesDir, rel);
const { ok, words, error } = await synthesizeOne({
provider: ttsProvider,
text,
voiceId,
lang,
speed,
wavAbs: abs,
hyperframesDir,
});
if (!ok) {
anomalies.push(`line ${id}: TTS failed — omitted${error ? ` (${error})` : ""}`);
return null;
}
let wordArr = words; // heygen: native; else transcribe
if (!wordArr) wordArr = await transcribeWav({ wavRel: rel, lang, hyperframesDir });
const dur = ffprobeDuration(abs);
if (!isFinite(dur) || dur <= 0) {
anomalies.push(`line ${id}: bad voice duration — omitted`);
return null;
}
return { id, path: rel, duration_s: r3(dur), words: withWordIds(wordArr) };
};
const results = await mapWithConcurrency(lines, ttsConcurrency, synthLine);
voices = results.filter(Boolean);
for (const v of voices)
console.error(` voice ${v.id}: ${v.path} (${v.duration_s}s, ${v.words.length} words)`);
}
const hasVoice = voices.length > 0;
const totalDuration = r3(voices.reduce((a, v) => a + (v.duration_s || 0), 0));
// ── BGM ─────────────────────────────────────────────────────────────────────
let bgm = prev.bgm ?? null;
const bgmFields = {
bgm_pending: prev.bgm_pending ?? false,
bgm_provider: prev.bgm_provider ?? null,
bgm_pid: prev.bgm_pid ?? null,
bgm_log: prev.bgm_log ?? null,
bgm_mode: prev.bgm_mode ?? null,
bgm_target_duration_s: prev.bgm_target_duration_s ?? null,
bgm_seed_duration_s: prev.bgm_seed_duration_s ?? null,
bgm_loop_count: prev.bgm_loop_count ?? null,
};
if (only.has("bgm")) {
bgm = null;
Object.keys(bgmFields).forEach((k) => (bgmFields[k] = k === "bgm_pending" ? false : null));
// Mode resolution. An EXPLICIT mode (flag or request.bgm.mode) is strict:
// "retrieve" means retrieve-or-nothing — it never silently starts a detached
// generate (a caller with no wait-bgm step, e.g. product-launch, must not get
// a pending job it can't await). Only the UNSET/auto default picks generate
// when HeyGen is absent.
const explicitMode = bgmModeOverride || request.bgm?.mode || null;
let mode = noBgm ? "none" : explicitMode || (heygenOK ? "retrieve" : "generate");
if (mode === "retrieve" && !heygenOK) {
anomalies.push(
"bgm: retrieve requires a HeyGen credential — skipped (no generate fallback for an explicit retrieve)",
);
mode = "none";
}
if (mode === "none") {
console.error(`· bgm: disabled`);
} else if (mode === "retrieve") {
try {
bgm = await retrieveBgm({ query: request.bgm?.query, headers, hyperframesDir, hasVoice });
if (bgm) {
bgmFields.bgm_provider = "heygen";
bgmFields.bgm_mode = "retrieve";
console.error(` bgm: ${bgm.path} (retrieve "${bgm.query}")`);
} else {
anomalies.push(`bgm: no music match for "${request.bgm?.query ?? ""}" — skipped`);
}
} catch (e) {
anomalies.push(`bgm retrieve failed: ${e.message} — skipped`);
}
} else {
// generate
const prompt = inferBgmPrompt({
userPrompt: request.bgm?.prompt,
blob: request.bgm?.blob || request.bgm?.query,
archetype: request.bgm?.archetype,
arc: request.bgm?.arc,
});
const gen = generateBgmDetached({
prompt,
durationS: totalDuration || 30,
hyperframesDir,
lyriaRecipe: existsSync(lyriaRecipe) ? lyriaRecipe : null,
seedSeconds,
hasVoice,
});
if (gen.disabled) {
anomalies.push(`bgm: ${gen.reason}`);
} else {
bgm = { path: gen.path, volume: gen.volume, mode: gen.mode, duration_s: null };
bgmFields.bgm_pending = true;
bgmFields.bgm_provider = gen.provider;
bgmFields.bgm_pid = gen.pid;
bgmFields.bgm_log = gen.log;
bgmFields.bgm_mode = gen.mode;
bgmFields.bgm_target_duration_s = gen.target_duration_s ?? null;
bgmFields.bgm_seed_duration_s = gen.seed_duration_s ?? null;
bgmFields.bgm_loop_count = gen.loop_count ?? null;
console.error(` bgm: launched ${gen.provider} (detached, pid ${gen.pid}) → ${gen.path}`);
}
}
}
// ── SFX ─────────────────────────────────────────────────────────────────────
let sfx = prev.sfx ?? [];
if (only.has("sfx")) {
const cues = lines.flatMap((l) =>
(Array.isArray(l.sfx) ? l.sfx : [])
.map((name) => ({ id: String(l.id), name: String(name).trim() }))
.filter((c) => c.name),
);
const res = await resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir });
sfx = res.sfx;
anomalies.push(...res.anomalies);
console.error(
`· sfx: ${sfx.length} cue(s) resolved (${heygenOK ? "heygen retrieval" : "bundled library"})`,
);
}
// ── write audio_meta.json ─────────────────────────────────────────────────────
const meta = {
tts_provider: ttsProvider,
voice_id: voiceId,
bgm,
...bgmFields,
voices,
sfx,
total_duration_s: totalDuration,
};
mkdirSync(dirname(outPath), { recursive: true });
writeFileSync(outPath, JSON.stringify(meta, null, 2));
console.log(`✓ audio engine → ${outPath}`);
console.log(` heygen: ${heygenOK ? "yes" : "no"} · ran: ${[...only].join(",")}`);
console.log(
` voices: ${voices.length} · bgm: ${bgm ? `${bgmFields.bgm_provider}${bgmFields.bgm_pending ? " (pending)" : ""}` : "none"} · sfx: ${sfx.length}`,
);
console.log(` total voice duration: ${totalDuration}s`);
if (anomalies.length) {
console.log(`\nanomalies (non-fatal):`);
for (const a of anomalies) console.log(` - ${a}`);
}
@@ -0,0 +1,49 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { mkdtempSync, rmSync, existsSync } from "node:fs";
import { join, dirname } from "node:path";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
import { resolveSfx } from "./lib/sfx.mjs";
// Proves the relocated engine (skills/media-use/audio/) still resolves its
// bundled SFX library from the moved location — the path most likely to break
// on a subtree move. Offline (heygenOK:false), no network.
const HERE = dirname(fileURLToPath(import.meta.url));
const sfxLibDir = join(HERE, "..", "assets", "sfx"); // same offset the engine uses
test("bundled SFX library resolves from the relocated path", async () => {
assert.ok(existsSync(join(sfxLibDir, "manifest.json")), "moved manifest is present");
const dir = mkdtempSync(join(tmpdir(), "mu-audio-"));
try {
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "1", name: "whoosh" }],
heygenOK: false,
hyperframesDir: dir,
sfxLibDir,
});
assert.equal(sfx.length, 1, `expected 1 resolved cue, got anomalies: ${anomalies.join("; ")}`);
assert.equal(sfx[0].source, "local");
assert.match(sfx[0].file, /assets\/sfx\//);
assert.ok(existsSync(join(dir, sfx[0].file)), "matched SFX copied into the project");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("an unknown cue is reported, not fatal", async () => {
const dir = mkdtempSync(join(tmpdir(), "mu-audio-"));
try {
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "1", name: "definitely-not-a-real-sfx" }],
heygenOK: false,
hyperframesDir: dir,
sfxLibDir,
});
assert.equal(sfx.length, 0);
assert.ok(anomalies.some((a) => /not in bundled library/.test(a)));
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env node
// Self-contained HeyGen TTS — single text in → one wav (+ optional words JSON)
// out. A thin CLI over lib/tts.mjs (the same code the audio engine uses), so the
// HeyGen REST call, starfish voice pick, mp3→wav transcode, and word-timestamp
// filtering live in exactly one place. Bypasses the `hyperframes` CLI, which in
// the published build is Kokoro-only.
//
// Usage:
// node heygen-tts.mjs "Text to speak" -o narration.wav [--words narration.words.json]
// node heygen-tts.mjs ./script.txt -o narration.wav --words narration.words.json
// node heygen-tts.mjs "Bonjour" -o fr.wav --lang fr --voice <id>
// node heygen-tts.mjs --list # list starfish voices and exit
//
// Flags: -o/--output (.wav → ffmpeg transcode; .mp3 → raw bytes), --words,
// --voice (starfish id), --speed, --lang, --list.
// Requires: $HEYGEN_API_KEY / OAuth ~/.heygen credentials and ffmpeg for .wav output.
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { heygenAuthHeaders, heygenJSON, loadEnvFromDir } from "./lib/heygen.mjs";
import { ffprobeDuration, resolveVoiceId, synthesizeOne, withWordIds } from "./lib/tts.mjs";
const argv = process.argv.slice(2);
function flag(name, def) {
const i = argv.indexOf(`--${name}`);
if (i < 0) return def;
if (i + 1 >= argv.length) return true;
const v = argv[i + 1];
return v.startsWith("--") ? true : v;
}
const die = (m) => {
console.error(`✗ heygen-tts: ${m}`);
process.exit(1);
};
// First arg that isn't a flag or the -o value is the text / .txt path.
const positional = (() => {
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a.startsWith("--")) {
const next = argv[i + 1];
if (next && !next.startsWith("--")) i++;
continue;
}
if (a === "-o") {
i++;
continue;
}
return a;
}
return null;
})();
const output = resolve(
(typeof flag("output") === "string" && flag("output")) ||
(argv.includes("-o") && argv[argv.indexOf("-o") + 1]) ||
"narration.wav",
);
const wordsPath = typeof flag("words") === "string" ? resolve(flag("words")) : null;
const userVoice = typeof flag("voice") === "string" ? flag("voice") : null;
const speedRaw = typeof flag("speed") === "string" ? Number(flag("speed")) : 1.0;
const speed = isFinite(speedRaw) && speedRaw > 0 ? speedRaw : 1.0;
const lang = typeof flag("lang") === "string" ? flag("lang") : "en";
const listOnly = flag("list") === true;
loadEnvFromDir(process.cwd());
let authHeaders;
try {
authHeaders = heygenAuthHeaders();
} catch (e) {
die(e.message);
}
// ---------- --list ----------
if (listOnly) {
const payload = await heygenJSON(`/voices?engine=starfish&type=public&limit=50`, {
headers: authHeaders,
});
for (const v of payload.data ?? payload.voices ?? []) {
console.log(`${v.voice_id}\t${v.name}\t${v.language ?? ""}`);
}
process.exit(0);
}
// ---------- resolve text + voice ----------
if (!positional) die("no text given. Pass a string or a .txt path, or use --list.");
const text =
positional.endsWith(".txt") && existsSync(resolve(positional))
? readFileSync(resolve(positional), "utf8").trim()
: positional;
if (!text) die("input text is empty");
const voiceId = await resolveVoiceId({ provider: "heygen", userVoice, lang });
if (!userVoice) console.error(`· using voice ${voiceId}`);
// ---------- synthesize (shared engine code) ----------
const { ok, words } = await synthesizeOne({
provider: "heygen",
text,
voiceId,
lang,
speed,
wavAbs: output,
hyperframesDir: process.cwd(),
});
if (!ok) die("synthesis failed (HeyGen request/transcode error)");
let wordCount = 0;
if (wordsPath) {
if (words && words.length) {
mkdirSync(dirname(wordsPath), { recursive: true });
writeFileSync(wordsPath, JSON.stringify(withWordIds(words), null, 2));
wordCount = words.length;
} else {
console.error("⚠ no word_timestamps in response — run `hyperframes transcribe` instead");
}
}
const dur = ffprobeDuration(output);
const durStr = isFinite(dur) ? ` (${dur.toFixed(2)}s)` : "";
console.log(`${output}${durStr}${wordCount ? ` · ${wordsPath} (${wordCount} words)` : ""}`);
+257
View File
@@ -0,0 +1,257 @@
// bgm.mjs — background music for the media audio engine. Two routes, gated the
// same way as TTS/SFX:
//
// retrieve (default when HeyGen is configured) — search HeyGen's music library
// by mood, download the top track. Synchronous. assets/bgm/track.mp3.
// generate (the alternative; the automatic choice when HeyGen is absent) —
// Lyria (cloud, $GEMINI_API_KEY/$GOOGLE_API_KEY + google-genai) preferred,
// else local MusicGen (facebook/musicgen-small via transformers). Spawned
// DETACHED so the engine can return while audio renders; the caller marks
// bgm_pending and runs wait-bgm.mjs before assembling. assets/bgm/track.wav.
//
// Missing/failed BGM never blocks a render.
import { spawn, spawnSync } from "node:child_process";
import { existsSync, mkdirSync, openSync, closeSync } from "node:fs";
import { join } from "node:path";
import { downloadTo, searchSounds } from "./heygen.mjs";
import { pythonInvocation } from "./python.mjs";
const r3 = (x) => Number(x.toFixed(3));
const lyriaKey = () => process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY || "";
// Default BGM level. Under narration music is a bed that must stay under the
// voice — 0.12 linear ≈ -18 dB. A silent film (no voice) has no voice to duck
// beneath, so BGM sits forward at 0.9. Callers may override per composition.
export const BGM_BED_VOLUME = 0.12;
export const BGM_SILENT_VOLUME = 0.9;
export const bgmDefaultVolume = (hasVoice) => (hasVoice ? BGM_BED_VOLUME : BGM_SILENT_VOLUME);
const BGM_PY_DEPS = ["transformers", "torch", "soundfile", "numpy"];
const BGM_PY_PROBE =
"import transformers, soundfile, torch, numpy; from transformers import MusicgenForConditionalGeneration";
const LYRIA_PY_DEPS = ["google-genai", "python-dotenv"];
const LYRIA_PY_PROBE = "import google.genai";
function pyOk(probe) {
const { cmd, args } = pythonInvocation(["-c", probe]);
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
}
// `python -m pip`, not a bare `pip` binary: a Homebrew/system Python often
// exposes only `python3`/`pip3` on PATH, so a plain `pip` spawn silently
// no-ops (ENOENT) and the documented "auto-installed on demand" path never
// actually installs. `-m pip` also guarantees the packages land in the SAME
// interpreter pyOk() probes — a bare `pip`/`pip3` could resolve to a
// different Python installation than `python3` if more than one is on PATH.
function pipInstall(deps) {
const { cmd, args } = pythonInvocation(["-m", "pip", "install", "-q", ...deps]);
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
}
// ── retrieval (HeyGen music library) ──────────────────────────────────────────
export async function retrieveBgm({ query, headers, hyperframesDir, hasVoice }) {
const q = query || "calm cinematic underscore";
const results = await searchSounds(q, "music", headers, { limit: 5 });
if (!results.length) return null;
const top = results[0];
const rel = "assets/bgm/track.mp3";
await downloadTo(top.audio_url, join(hyperframesDir, rel));
return {
path: rel,
volume: bgmDefaultVolume(hasVoice),
query: q,
mode: "retrieve",
duration_s: typeof top.duration === "number" ? r3(top.duration) : null,
};
}
// ── mood inference (for the generate path's prompt) ──────────────────────────
// Industry base → archetype shape → emotional-arc tiebreaker. Exported so a
// workflow adapter can build a rich prompt from its own narrative metadata; the
// engine also calls it when generate has only a plain mood query.
export function inferBgmPrompt({ blob = "", archetype = "", arc = "", userPrompt = "" } = {}) {
if (userPrompt) return userPrompt;
const b = String(blob).toLowerCase();
let base;
let bpm;
if (/\b(crypto|nft|web3|defi|token|blockchain|exchange|wallet|dao)\b/.test(b)) {
base = "atmospheric electronic, deep bass, futuristic synths, restrained percussion";
bpm = 100;
} else if (/\b(finance|fintech|bank|payment|invest|wealth|insurance|treasury)\b/.test(b)) {
base = "calm cinematic, soft strings, subtle piano, restrained percussion";
bpm = 92;
} else if (/\b(creative|agency|design|studio|art|brand|marketing|content)\b/.test(b)) {
base = "playful electronic, warm pads, light percussion";
bpm = 115;
} else {
base = "uplifting corporate tech, bright modern piano with synth pads";
bpm = 108;
}
const at = String(archetype).toLowerCase();
const ar = String(arc).toLowerCase();
if (/\bpas\b|pain.agitate|pain.+solve/.test(at))
return `${base}, starts with subtle tension then builds to resolution, BPM ${bpm}, transitions from MINOR to MAJOR`;
if (/\bbab\b|before.after|future.pac|vision/.test(at))
return `${base}, cinematic and aspirational, steady build with rising energy, BPM ${bpm}, MAJOR`;
if (/cascade|feature.benefit/.test(at))
return `${base}, energetic and driving, consistent momentum, BPM ${Math.min(bpm + 10, 128)}, MAJOR`;
if (/demo.loop|question.+answer/.test(at))
return `${base}, clean and focused, minimal arrangement, BPM ${Math.max(bpm - 8, 88)}`;
if (/frustrat|anxiety|overwhelm|tension/.test(ar) && /relief|excite|triumph/.test(ar))
return `${base}, builds from understated tension to uplifting resolution, BPM ${bpm}, MINOR to MAJOR`;
if (/excit|awe|power|triumph/.test(ar))
return `${base}, energetic and confident, BPM ${bpm}, MAJOR`;
if (/trust|ease|clarity|reassur/.test(ar))
return `${base}, warm and reassuring, BPM ${Math.max(bpm - 5, 85)}`;
return `${base}, BPM ${bpm}, MAJOR`;
}
// ── generation (Lyria → MusicGen, detached) ──────────────────────────────────
// Returns a bgmMeta the caller folds into audio_meta:
// { path, mode, volume, provider, pid, log, target_duration_s, seed_duration_s,
// loop_count, pending:true } on success, or { disabled:true, reason }.
export function generateBgmDetached({
prompt,
durationS,
hyperframesDir,
lyriaRecipe,
seedSeconds = 28,
hasVoice,
}) {
const rel = "assets/bgm/track.wav";
const abs = join(hyperframesDir, rel);
mkdirSync(join(hyperframesDir, "assets", "bgm"), { recursive: true });
const log = join(hyperframesDir, "assets", "bgm", `bgm-${Date.now()}.log`);
const targetS = Math.max(1, durationS);
const baseMeta = { path: rel, mode: null, volume: bgmDefaultVolume(hasVoice), pending: true };
const lyriaConfigured = !!lyriaKey() && !!lyriaRecipe && existsSync(lyriaRecipe);
// Make a backend runnable: prefer Lyria when configured (install google-genai
// on demand), else ensure local MusicGen deps. Installs are synchronous here —
// generation itself is detached, so the engine still returns promptly.
if (lyriaConfigured && !pyOk(LYRIA_PY_PROBE)) pipInstall(LYRIA_PY_DEPS);
const useLyria = lyriaConfigured && pyOk(LYRIA_PY_PROBE);
if (!useLyria && !pyOk(BGM_PY_PROBE)) pipInstall(BGM_PY_DEPS);
const fd = openSync(log, "w");
if (useLyria) {
const { cmd, args } = pythonInvocation([
lyriaRecipe,
"--output",
abs,
"--duration",
String(targetS),
"--prompt",
prompt,
]);
const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
proc.unref();
closeSync(fd);
return {
...baseMeta,
mode: "detached-single",
provider: "lyria",
pid: proc.pid,
log,
target_duration_s: r3(targetS),
};
}
if (pyOk(BGM_PY_PROBE)) {
const seedS = Math.min(Math.max(seedSeconds, 10), 30);
const loops = targetS > seedS ? Math.ceil(targetS / seedS) : 1;
const script = musicgenScript({ prompt, abs, targetS, seedS });
const { cmd, args } = pythonInvocation(["-c", script]);
const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
proc.unref();
closeSync(fd);
return {
...baseMeta,
mode: targetS > seedS ? "detached-seed-loop" : "detached-seed-trim",
provider: "musicgen",
pid: proc.pid,
log,
target_duration_s: r3(targetS),
seed_duration_s: seedS,
loop_count: loops,
};
}
closeSync(fd);
return {
disabled: true,
reason: lyriaConfigured
? `Lyria configured but google-genai uninstallable, and local MusicGen unavailable (pip install ${BGM_PY_DEPS.join(" ")})`
: `no Lyria key/recipe and local MusicGen deps unavailable (pip install ${BGM_PY_DEPS.join(" ")})`,
};
}
// Inline MusicGen: generate ONE seed clip (≤30s to stay under the decoder's
// positional limit), then trim it down or crossfade-loop it up to the target.
function musicgenScript({ prompt, abs, targetS, seedS }) {
return `
import math, os, sys, traceback
from pathlib import Path
import numpy as np
import soundfile as sf
from transformers import MusicgenForConditionalGeneration, AutoProcessor
prompt = ${JSON.stringify(prompt)}
out_path = ${JSON.stringify(abs)}
target_s = float(${targetS.toFixed(3)})
seed_s = float(${seedS.toFixed(3)})
token_rate = 50
crossfade_s = 0.3
def apply_fade(arr, sr, fade_in_s=0.08, fade_out_s=0.5):
n_in = min(int(round(fade_in_s * sr)), arr.shape[0] // 2)
n_out = min(int(round(fade_out_s * sr)), arr.shape[0] // 2)
if n_in > 1: arr[:n_in] *= np.linspace(0.0, 1.0, n_in, dtype="float32")
if n_out > 1: arr[-n_out:] *= np.linspace(1.0, 0.0, n_out, dtype="float32")
return arr
def loop_crossfade(seed, target_len, xf):
if seed.shape[0] >= target_len: return seed[:target_len]
xf = min(xf, seed.shape[0] // 2)
if xf < 1:
reps = int(math.ceil(target_len / seed.shape[0]))
return np.tile(seed, reps)[:target_len]
t = np.linspace(0.0, 1.0, xf, dtype="float32")
fade_out = np.cos(t * (math.pi / 2)); fade_in = np.sin(t * (math.pi / 2))
out = seed.copy()
while out.shape[0] < target_len:
tail = out[-xf:] * fade_out; head = seed[:xf] * fade_in
out = np.concatenate([out[:-xf], tail + head, seed[xf:]])
return out[:target_len]
try:
Path(os.path.dirname(out_path)).mkdir(parents=True, exist_ok=True)
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
model.eval()
sr = int(model.config.audio_encoder.sampling_rate)
gen_s = min(seed_s, target_s)
tokens = max(1, int(math.ceil(gen_s * token_rate)))
print(f"[musicgen] seed dur={gen_s:.2f}s tokens={tokens}", flush=True)
inputs = processor(text=[prompt], padding=True, return_tensors="pt")
audio = model.generate(**inputs, max_new_tokens=tokens)
seed = audio[0, 0].detach().cpu().numpy().astype("float32")
peak = float(np.max(np.abs(seed)))
if peak > 1e-6: seed = seed * (0.89 / peak)
want = max(1, int(round(target_s * sr)))
if seed.shape[0] >= want:
final = seed[:want].copy()
else:
final = loop_crossfade(seed, want, int(round(crossfade_s * sr)))
if final.shape[0] < want: final = np.pad(final, (0, want - final.shape[0]))
else: final = final[:want]
final = apply_fade(final, sr)
peak = float(np.max(np.abs(final)))
if peak > 1.0: final = final / peak
sf.write(out_path, final, sr)
print(f"[musicgen] wrote {out_path} samples={final.shape[0]} sr={sr}", flush=True)
except Exception:
traceback.print_exc(); sys.exit(1)
`;
}
@@ -0,0 +1,30 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { BGM_BED_VOLUME, BGM_SILENT_VOLUME, bgmDefaultVolume } from "./bgm.mjs";
// Regression: narrated pipelines used to ship BGM at 0.8 (≈ -2 dB), ~16 dB
// hotter than a music bed under a voice should be. The default under narration
// must be a proper bed (≈ -18 dB); a silent film keeps the louder default.
const dbfs = (linear) => 20 * Math.log10(linear);
test("BGM under narration is a bed near -18 dB", () => {
assert.equal(bgmDefaultVolume(true), BGM_BED_VOLUME);
assert.equal(BGM_BED_VOLUME, 0.12);
const db = dbfs(BGM_BED_VOLUME);
assert.ok(db < -17 && db > -19, `bed should be ≈ -18 dB, got ${db.toFixed(1)} dB`);
});
test("a silent film (no voice) keeps BGM forward", () => {
assert.equal(bgmDefaultVolume(false), BGM_SILENT_VOLUME);
assert.equal(BGM_SILENT_VOLUME, 0.9);
});
test("the narrated default is well below the voice (≈ 0 dBFS)", () => {
// Voice sits at data-volume="1" (0 dBFS); the bed must be ~16+ dB under it.
const separation = dbfs(1) - dbfs(bgmDefaultVolume(true));
assert.ok(
separation >= 16,
`bed should sit ≥16 dB under the voice, got ${separation.toFixed(1)} dB`,
);
});
@@ -0,0 +1,14 @@
// mapWithConcurrency — run `fn` over `items` with at most `limit` in flight at
// once. Preserves input order in the result array regardless of completion order.
export async function mapWithConcurrency(items, limit, fn) {
const results = new Array(items.length);
let next = 0;
async function worker() {
while (next < items.length) {
const i = next++;
results[i] = await fn(items[i], i);
}
}
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
return results;
}
@@ -0,0 +1,41 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mapWithConcurrency } from "./concurrency.mjs";
// Regression: audio.mjs used a bare Promise.all(lines.map(synthLine)) to
// synthesize every TTS line at once, spawning one Kokoro/whisper model load
// per line concurrently. Two independent reports of this overwhelming a
// machine (OOM, and cold-start contention causing spurious failures).
// mapWithConcurrency is the extracted cap; test it in isolation since
// audio.mjs itself is a script (runs CLI/exit side effects on import).
test("processes every item and preserves input order regardless of completion order", async () => {
const order = [5, 1, 3, 2, 4];
const results = await mapWithConcurrency(order, 2, async (n) => {
await new Promise((r) => setTimeout(r, n));
return n * 10;
});
assert.deepEqual(results, [50, 10, 30, 20, 40]);
});
test("never runs more than `limit` at once", async () => {
let inFlight = 0;
let maxInFlight = 0;
const items = Array.from({ length: 10 }, (_, i) => i);
await mapWithConcurrency(items, 3, async () => {
inFlight++;
maxInFlight = Math.max(maxInFlight, inFlight);
await new Promise((r) => setTimeout(r, 5));
inFlight--;
});
assert.equal(maxInFlight, 3);
});
test("limit larger than the item count runs everything without hanging", async () => {
const results = await mapWithConcurrency([1, 2], 10, async (n) => n * 2);
assert.deepEqual(results, [2, 4]);
});
test("empty input resolves to an empty array", async () => {
const results = await mapWithConcurrency([], 4, async (n) => n);
assert.deepEqual(results, []);
});
@@ -0,0 +1,148 @@
// heygen.mjs — vendored HeyGen REST helpers (auth + transport) for the audio
// pipeline. The credential resolver matches the hyperframes CLI auth: first
// usable source wins — $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY → a nearby .env → ~/.heygen/
// credentials (oauth → Bearer, else api_key → X-Api-Key; $HEYGEN_CONFIG_DIR
// overrides the dir). Vendored so the skill ships standalone. Pure node.
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { homedir } from "node:os";
import { dirname, join, resolve } from "node:path";
export const HEYGEN_BASE = "https://api.heygen.com/v3";
export const HEYGEN_CLI_SOURCE_HEADERS = { "X-HeyGen-Source": "cli" };
// Walk up ≤5 dirs from startDir; load the first .env (shell env always wins).
export function loadEnvFromDir(startDir) {
let dir = resolve(startDir);
for (let i = 0; i < 5; i++) {
const envPath = join(dir, ".env");
if (existsSync(envPath)) {
for (const raw of readFileSync(envPath, "utf8").split("\n")) {
let line = raw.trim();
if (!line || line.startsWith("#")) continue;
if (line.startsWith("export ")) line = line.slice(7).trim();
const eq = line.indexOf("=");
if (eq < 1) continue;
const key = line.slice(0, eq).trim();
let val = line.slice(eq + 1).trim();
if (val.startsWith('"') || val.startsWith("'")) {
const q = val[0];
const end = val.indexOf(q, 1);
val = end > 0 ? val.slice(1, end) : val.slice(1);
}
if (!(key in process.env)) process.env[key] = val;
}
return;
}
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
}
}
// → { headers } | { expired: true } | null. Never throws.
export function heygenCredential() {
const envKey = process.env.HEYGEN_API_KEY || process.env.HYPERFRAMES_API_KEY;
if (envKey) return { headers: { "X-Api-Key": envKey } };
const file = join(process.env.HEYGEN_CONFIG_DIR || join(homedir(), ".heygen"), "credentials");
if (!existsSync(file)) return null;
const raw = readFileSync(file, "utf8").trim();
if (!raw) return null;
if (!raw.startsWith("{")) return { headers: { "X-Api-Key": raw } };
// A malformed credentials file (partial write / wrong shape) must degrade to
// "no credential", not crash the engine at startup — this function never throws.
let cred;
try {
cred = JSON.parse(raw);
} catch {
return null;
}
const oauth = cred.oauth;
if (oauth?.access_token) {
const expired = oauth.expires_at && new Date(oauth.expires_at).getTime() - 60_000 < Date.now();
if (!expired) return { headers: { Authorization: `Bearer ${oauth.access_token}` } };
if (!cred.api_key) return { expired: true };
}
if (cred.api_key) return { headers: { "X-Api-Key": cred.api_key } };
return null;
}
// → "oauth" | "api_key" | null. Same oauth-vs-api-key check heygenAuthHeaders()
// makes internally, exposed on its own so callers that only need to *tag* the
// auth path (telemetry) don't have to parse headers back apart. Never throws:
// no credential (or an expired one) is just `null`, same as a fresh resolve
// with nothing to tag.
export function heygenAuthMethod() {
const cred = heygenCredential();
if (!cred?.headers) return null;
return "Authorization" in cred.headers ? "oauth" : "api_key";
}
// → auth headers object, or throw with a fix hint.
export function heygenAuthHeaders() {
const cred = heygenCredential();
if (cred?.headers) {
// Only tag OAuth (Bearer) traffic as cli-source — the backend uses it to
// grant the free allowance for OAuth requests and ignores it for API-key
// (X-Api-Key) traffic, where it's dead metadata.
const isOauth = "Authorization" in cred.headers;
return isOauth ? { ...cred.headers, ...HEYGEN_CLI_SOURCE_HEADERS } : { ...cred.headers };
}
if (cred?.expired)
throw new Error(
"HeyGen OAuth token expired — run `npx hyperframes auth refresh` (or `npx hyperframes auth login`)",
);
throw new Error(
"no HeyGen credentials — set $HEYGEN_API_KEY, or run `npx hyperframes auth login` (writes ~/.heygen/credentials)",
);
}
// Authed JSON request against the v3 API; throws on a non-OK status.
export async function heygenJSON(path, { method = "GET", headers = {}, body } = {}) {
const opts = { method, headers: { ...headers } };
if (body !== undefined) {
opts.headers["Content-Type"] = "application/json";
opts.body = JSON.stringify(body);
}
const res = await fetch(`${HEYGEN_BASE}${path}`, opts);
if (!res.ok) {
const detail = await res.text().catch(() => "");
throw new Error(
`HeyGen ${method} ${path} → HTTP ${res.status}${detail ? `\n${detail.slice(0, 300)}` : ""}`,
);
}
return res.json();
}
// Download a (presigned) URL to destPath; returns byte length.
export async function downloadTo(url, destPath) {
const res = await fetch(url);
if (!res.ok) throw new Error(`download HTTP ${res.status}: ${String(url).slice(0, 80)}`);
const bytes = Buffer.from(await res.arrayBuffer());
mkdirSync(dirname(destPath), { recursive: true });
writeFileSync(destPath, bytes);
return bytes.length;
}
// Retrieval search over HeyGen's audio catalog (NOT generation). type =
// "music" | "sound_effects". Returns the ranked results array (best first); each
// item has a presigned `audio_url` (+ `duration`, `description`, `name`, `score`).
// `query` is required (≥1 char, empty → HTTP 400) and `limit` is capped at 50.
// `minScore`: omit to use the server default (0.7). That default is TOO HIGH for
// sound_effects — good SFX hits score ~0.50.67, so callers wanting SFX should
// pass a lower floor (~0.4); music scores high and is fine at the default.
export async function searchSounds(query, type, headers, { limit = 5, minScore } = {}) {
const params = new URLSearchParams({ query, type, limit: String(limit) });
if (minScore != null) params.set("min_score", String(minScore));
const payload = await heygenJSON(`/audio/sounds?${params.toString()}`, { headers });
// `data` comes back as a ranked array (best first). Older responses keyed it by
// numeric index ("0","1",…); normalize both shapes to an array (empty → []).
const data = payload?.data ?? payload;
if (Array.isArray(data)) return data;
if (data && typeof data === "object") return Object.values(data);
throw new Error(
`unexpected /audio/sounds shape — top keys: ${Object.keys(payload ?? {}).join(", ")}`,
);
}
@@ -0,0 +1,100 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { heygenAuthHeaders, heygenAuthMethod } from "./heygen.mjs";
function withCleanHeygenEnv(fn) {
const previousApiKey = process.env.HEYGEN_API_KEY;
const previousHyperframesApiKey = process.env.HYPERFRAMES_API_KEY;
const previousConfigDir = process.env.HEYGEN_CONFIG_DIR;
try {
delete process.env.HEYGEN_API_KEY;
delete process.env.HYPERFRAMES_API_KEY;
delete process.env.HEYGEN_CONFIG_DIR;
return fn();
} finally {
if (previousApiKey === undefined) delete process.env.HEYGEN_API_KEY;
else process.env.HEYGEN_API_KEY = previousApiKey;
if (previousHyperframesApiKey === undefined) delete process.env.HYPERFRAMES_API_KEY;
else process.env.HYPERFRAMES_API_KEY = previousHyperframesApiKey;
if (previousConfigDir === undefined) delete process.env.HEYGEN_CONFIG_DIR;
else process.env.HEYGEN_CONFIG_DIR = previousConfigDir;
}
}
test("heygenAuthHeaders does not tag API-key requests as CLI traffic", () => {
withCleanHeygenEnv(() => {
process.env.HEYGEN_API_KEY = "hg_test";
// API-key requests use normal billing; the backend ignores the cli-source
// header for them, so it's not sent.
assert.deepEqual(heygenAuthHeaders(), {
"X-Api-Key": "hg_test",
});
});
});
test("heygenAuthHeaders tags OAuth requests as CLI traffic", () => {
withCleanHeygenEnv(() => {
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
try {
process.env.HEYGEN_CONFIG_DIR = dir;
writeFileSync(
join(dir, "credentials"),
JSON.stringify({
oauth: {
access_token: "at_test",
expires_at: "2099-01-01T00:00:00Z",
},
}),
);
assert.deepEqual(heygenAuthHeaders(), {
Authorization: "Bearer at_test",
"X-HeyGen-Source": "cli",
});
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
test("heygenAuthMethod returns api_key for an env API key, without tagging headers", () => {
withCleanHeygenEnv(() => {
process.env.HEYGEN_API_KEY = "hg_test";
assert.equal(heygenAuthMethod(), "api_key");
});
});
test("heygenAuthMethod returns oauth for a live OAuth credential", () => {
withCleanHeygenEnv(() => {
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
try {
process.env.HEYGEN_CONFIG_DIR = dir;
writeFileSync(
join(dir, "credentials"),
JSON.stringify({
oauth: {
access_token: "at_test",
expires_at: "2099-01-01T00:00:00Z",
},
}),
);
assert.equal(heygenAuthMethod(), "oauth");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
test("heygenAuthMethod returns null with no credential at all", () => {
withCleanHeygenEnv(() => {
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
try {
process.env.HEYGEN_CONFIG_DIR = dir; // no credentials file written
assert.equal(heygenAuthMethod(), null);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
@@ -0,0 +1,63 @@
// python.mjs — resolve which Python 3 executable to spawn, per platform.
//
// The audio engine (tts.mjs, bgm.mjs) shells out to `python3` for ElevenLabs
// TTS and the local Lyria/MusicGen BGM paths. `python3` is the right name on
// macOS/Linux, but on Windows the python.org installer only creates
// `python.exe` plus the `py` launcher — there is no `python3.exe` (only the
// Microsoft Store build adds one). So a bare `spawn("python3", …)` ENOENTs on a
// standard Windows Python install, silently disabling every Python-backed audio
// feature until the user hand-creates a `python3.exe` shim (reported twice).
//
// Resolve once, per process: probe the platform's candidates in order and take
// the first that actually runs. `py` is the launcher, so it needs a `-3` arg to
// select Python 3 — hence candidates are argv PREFIXES, not bare names.
import { spawnSync } from "node:child_process";
function defaultProbe(cmd, args) {
try {
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
} catch {
return false;
}
}
/**
* Pick the argv prefix that launches Python 3 on this platform.
* Returns e.g. `["python3"]`, `["python"]`, or `["py", "-3"]`.
*
* Pure except for `probe` (which runs `<cmd> … --version`); both `platform`
* and `probe` are injectable so every branch is unit-testable without spawning.
* If nothing probes OK, falls back to the canonical name for the platform so
* the eventual spawn fails loudly exactly as it did before — never worse.
*/
export function resolvePythonCommand(platform = process.platform, probe = defaultProbe) {
const candidates =
platform === "win32" ? [["python3"], ["python"], ["py", "-3"]] : [["python3"], ["python"]];
for (const prefix of candidates) {
if (probe(prefix[0], [...prefix.slice(1), "--version"])) return prefix;
}
return candidates[0];
}
let cached = null;
/** Cached `resolvePythonCommand()` — probing spawns, so resolve at most once. */
export function pythonCommand() {
if (!cached) cached = resolvePythonCommand();
return cached;
}
/**
* Build a `{ cmd, args }` for running Python 3 with `extraArgs`, using the
* resolved (or supplied) prefix. Keeps the launcher's `-3` (and any future
* prefix args) ahead of the caller's own arguments.
*/
export function pythonInvocation(extraArgs, prefix = pythonCommand()) {
return { cmd: prefix[0], args: [...prefix.slice(1), ...extraArgs] };
}
/** Test-only: clear the cached resolution so a test can re-probe. */
export function _resetPythonCommandCacheForTests() {
cached = null;
}
@@ -0,0 +1,68 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { resolvePythonCommand, pythonInvocation } from "./python.mjs";
// Regression: on Windows a standard python.org install has no `python3.exe`
// (only `python.exe` + the `py` launcher), so `spawn("python3", …)` ENOENTs and
// every Python-backed audio feature silently no-ops. resolvePythonCommand takes
// injectable platform/probe params so all branches are testable without
// spawning a real interpreter.
// probeFor(names): a probe that reports success only for the given argv-0 names.
function probeFor(...names) {
const ok = new Set(names);
return (cmd) => ok.has(cmd);
}
test("non-win32 uses python3 when it runs", () => {
assert.deepEqual(resolvePythonCommand("linux", probeFor("python3")), ["python3"]);
assert.deepEqual(resolvePythonCommand("darwin", probeFor("python3")), ["python3"]);
});
test("win32 prefers python3 when the Microsoft Store build provides it", () => {
assert.deepEqual(resolvePythonCommand("win32", probeFor("python3", "python", "py")), ["python3"]);
});
test("win32 falls back to python.exe when python3 is absent (python.org install)", () => {
// The exact reported scenario: no python3, but `python` exists.
assert.deepEqual(resolvePythonCommand("win32", probeFor("python", "py")), ["python"]);
});
test("win32 falls back to the py launcher with -3 when only py exists", () => {
assert.deepEqual(resolvePythonCommand("win32", probeFor("py")), ["py", "-3"]);
});
test("py launcher is probed as `py -3 --version`, not bare `py`", () => {
const seen = [];
const probe = (cmd, args) => {
seen.push([cmd, ...args]);
return cmd === "py";
};
resolvePythonCommand("win32", probe);
assert.deepEqual(seen.at(-1), ["py", "-3", "--version"]);
});
test("falls back to the canonical name (loud failure, unchanged) when nothing runs", () => {
// No interpreter anywhere — must not throw, and must return python3 so the
// eventual spawn fails exactly as it did before this fix, never worse.
assert.deepEqual(
resolvePythonCommand("win32", () => false),
["python3"],
);
assert.deepEqual(
resolvePythonCommand("linux", () => false),
["python3"],
);
});
test("pythonInvocation prepends the resolved prefix ahead of caller args", () => {
assert.deepEqual(pythonInvocation(["-c", "import x"], ["python"]), {
cmd: "python",
args: ["-c", "import x"],
});
// The py launcher's -3 must stay ahead of the caller's own arguments.
assert.deepEqual(pythonInvocation(["-c", "import x"], ["py", "-3"]), {
cmd: "py",
args: ["-3", "-c", "import x"],
});
});
+143
View File
@@ -0,0 +1,143 @@
// sfx.mjs — sound effects for the media audio engine. Provider-gated (NOT a
// per-cue merge): the decision is made once, by whether HeyGen is configured —
// mirroring how TTS and BGM degrade.
//
// HeyGen credential present → retrieve EVERY cue from HeyGen's audio library
// (/v3/audio/sounds, type=sound_effects, min_score=0.4). The bundled
// library is NOT consulted.
// HeyGen credential absent → resolve cues against the bundled 21-file
// library (assets/sfx/manifest.json), copying matched files into the
// project. Offline, deterministic, free.
//
// A cue that matches nothing is skipped (recorded as an anomaly); SFX never
// blocks a render. Every cue sits at volume ~0.35, under voice + BGM.
import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { downloadTo, searchSounds } from "./heygen.mjs";
const SFX_VOLUME = 0.35;
const slug = (s) =>
s
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, 40) || "x";
const r3 = (x) => Number(x.toFixed(3));
// cues: [{ id, name }] (id = the line/frame/scene the cue fires in). Returns
// { sfx: [{ id, name, file, source, offset_s, duration_s, volume }], anomalies }.
export async function resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir }) {
const sfx = [];
const anomalies = [];
const destDir = join(hyperframesDir, "assets", "sfx");
// Dedupe identical (id,name) cues — the same effect named twice in one line
// downloads/copies once.
const seen = new Set();
const uniq = cues.filter((c) => {
const k = `${c.id}:${c.name}`;
if (seen.has(k)) return false;
seen.add(k);
return true;
});
if (heygenOK) {
for (const { id, name } of uniq) {
try {
// SFX hits score low (~0.50.67), below the API's default 0.7 which
// silently drops most named cues — floor to 0.4. (BGM/music score high
// and keep the default.)
const results = await searchSounds(name, "sound_effects", headers, {
limit: 3,
minScore: 0.4,
});
if (!results.length) {
anomalies.push(`sfx "${name}" (id ${id}): no HeyGen match — skipped`);
continue;
}
const top = results[0];
const file = `assets/sfx/${slug(name)}.mp3`;
await downloadTo(top.audio_url, join(hyperframesDir, file));
sfx.push({
id,
name,
file,
source: "heygen",
offset_s: 0,
duration_s: typeof top.duration === "number" ? r3(top.duration) : 1.0,
volume: SFX_VOLUME,
});
} catch (e) {
anomalies.push(`sfx "${name}" (id ${id}): retrieval failed — ${e.message}`);
}
}
return { sfx, anomalies };
}
// ── offline: bundled library ──
const manifestPath = join(sfxLibDir, "manifest.json");
if (!existsSync(manifestPath)) {
if (uniq.length)
anomalies.push(`no HeyGen credential and no SFX library at ${sfxLibDir} — all cues dropped`);
return { sfx, anomalies };
}
let manifest;
try {
manifest = JSON.parse(readFileSync(manifestPath, "utf8"));
} catch (e) {
anomalies.push(`SFX manifest parse failed (${e.message}) — all cues dropped`);
return { sfx, anomalies };
}
// Build lookups: by manifest key, by file basename, and by slug of either, so
// a cue can name "whoosh", "whoosh.mp3", or "ui click" (→ slug match).
const byKey = new Map();
for (const [key, entry] of Object.entries(manifest)) {
if (!entry?.file || !isFinite(entry.duration)) continue;
const rec = { key, file: entry.file, duration: entry.duration };
byKey.set(key, rec);
byKey.set(entry.file, rec);
byKey.set(slug(key), rec);
byKey.set(slug(entry.file.replace(/\.\w+$/, "")), rec);
}
mkdirSync(destDir, { recursive: true });
for (const { id, name } of uniq) {
const hit = byKey.get(name) ?? byKey.get(slug(name));
if (!hit) {
const known = [...new Set([...byKey.values()].map((v) => v.key))].slice(0, 8).join(", ");
anomalies.push(
`sfx "${name}" (id ${id}): not in bundled library — skipped (have: ${known}…)`,
);
continue;
}
const src = join(sfxLibDir, hit.file);
const destRel = `assets/sfx/${hit.file}`;
const dest = join(hyperframesDir, destRel);
// The bundled library may be incomplete: some installs of the skill ship
// manifest.json without the actual mp3s. Pushing an sfx entry that points at
// a file we never copied produces a dangling reference that silently drops
// downstream ("not on disk"). Surface it as a loud anomaly and skip the cue
// instead, so the audio_meta never references a missing file.
if (!existsSync(dest)) {
if (!existsSync(src)) {
anomalies.push(
`sfx "${name}" (id ${id}): bundled file ${hit.file} missing from the offline ` +
`library (${sfxLibDir}) — skipped. Reinstall the media-use skill to ` +
`restore assets/sfx/*.mp3, or configure a HeyGen credential for retrieval.`,
);
continue;
}
copyFileSync(src, dest);
}
sfx.push({
id,
name,
file: destRel,
source: "local",
offset_s: 0,
duration_s: r3(hit.duration),
volume: SFX_VOLUME,
});
}
return { sfx, anomalies };
}
@@ -0,0 +1,69 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, mkdirSync, writeFileSync, existsSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { resolveSfx } from "./sfx.mjs";
// Offline (no HeyGen) SFX resolution: the bundled library may ship manifest.json
// without the actual mp3s. The old code copied only when the source existed but
// pushed the sfx entry unconditionally — producing a dangling reference that
// silently dropped downstream ("not on disk"). These tests lock in the loud
// behavior: a present file is copied + referenced; a missing file yields an
// anomaly and NO dangling entry.
async function withDirs(fn) {
const root = mkdtempSync(join(tmpdir(), "hf-sfx-"));
const libDir = join(root, "lib");
const projDir = join(root, "proj");
mkdirSync(libDir, { recursive: true });
mkdirSync(projDir, { recursive: true });
try {
// `await` is load-bearing: without it the finally cleanup runs before the
// async test body resolves, deleting the temp dir mid-assertion.
return await fn({ libDir, projDir });
} finally {
rmSync(root, { recursive: true, force: true });
}
}
test("offline: copies and references a present bundled file", async () => {
await withDirs(async ({ libDir, projDir }) => {
writeFileSync(
join(libDir, "manifest.json"),
JSON.stringify({ whoosh: { file: "whoosh.mp3", duration: 0.8 } }),
);
writeFileSync(join(libDir, "whoosh.mp3"), "ID3-fake-bytes");
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "s1", name: "whoosh" }],
heygenOK: false,
hyperframesDir: projDir,
sfxLibDir: libDir,
});
assert.equal(sfx.length, 1);
assert.equal(sfx[0].file, "assets/sfx/whoosh.mp3");
assert.equal(sfx[0].source, "local");
assert.ok(existsSync(join(projDir, "assets/sfx/whoosh.mp3")), "mp3 copied into project");
assert.equal(anomalies.length, 0);
});
});
test("offline: a matched-but-missing bundled file yields an anomaly and NO dangling entry", async () => {
await withDirs(async ({ libDir, projDir }) => {
// Manifest names whoosh.mp3, but the mp3 was never shipped (the reported bug).
writeFileSync(
join(libDir, "manifest.json"),
JSON.stringify({ whoosh: { file: "whoosh.mp3", duration: 0.8 } }),
);
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "s1", name: "whoosh" }],
heygenOK: false,
hyperframesDir: projDir,
sfxLibDir: libDir,
});
assert.equal(sfx.length, 0, "no dangling entry for a file that was never copied");
assert.equal(anomalies.length, 1);
assert.match(anomalies[0], /missing from the offline library/);
assert.ok(!existsSync(join(projDir, "assets/sfx/whoosh.mp3")), "nothing copied");
});
});
+378
View File
@@ -0,0 +1,378 @@
// tts.mjs — multi-provider TTS for the media audio engine. The provider chain,
// auto-detected from env, is the one documented in ../SKILL.md:
//
// 1. HeyGen (Starfish) — $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY / ~/.heygen.
// Direct v3 REST (NOT `hyperframes tts`, which in the published build is
// Kokoro-only and silently ignores a HeyGen key). Returns word_timestamps
// in the same call, so no separate transcribe pass.
// 2. ElevenLabs — $ELEVENLABS_API_KEY + `pip install elevenlabs`. No
// word timings → caller chains transcribeWav().
// 3. Kokoro-82M (local) — always available, via the published `hyperframes tts`
// CLI. No word timings → caller chains transcribeWav().
//
// "HeyGen available" is decided by CREDENTIAL presence (heygenCredential), never
// by the CLI — see the note above.
import { spawn, spawnSync } from "node:child_process";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { heygenAuthHeaders, heygenCredential, heygenJSON } from "./heygen.mjs";
import { pythonInvocation } from "./python.mjs";
// ── provider detection ────────────────────────────────────────────────────────
export function heygenAvailable() {
return heygenCredential() !== null;
}
export function elevenlabsAvailable() {
if (!process.env.ELEVENLABS_API_KEY) return false;
const { cmd, args } = pythonInvocation(["-c", "import elevenlabs"]);
const r = spawnSync(cmd, args, {
stdio: "ignore",
});
return r.status === 0;
}
// First available provider wins; an explicit choice is honored (and validated).
export function pickProvider(userProvider) {
if (userProvider) {
if (!["heygen", "elevenlabs", "kokoro"].includes(userProvider))
throw new Error(`invalid provider "${userProvider}" (heygen | elevenlabs | kokoro)`);
if (userProvider === "heygen" && !heygenAvailable())
throw new Error(
"provider=heygen but no HeyGen credentials (set $HEYGEN_API_KEY or run `npx hyperframes auth login`)",
);
if (userProvider === "elevenlabs" && !process.env.ELEVENLABS_API_KEY)
throw new Error("provider=elevenlabs but $ELEVENLABS_API_KEY is not set");
return userProvider;
}
return heygenAvailable() ? "heygen" : elevenlabsAvailable() ? "elevenlabs" : "kokoro";
}
// ── voice resolution ──────────────────────────────────────────────────────────
// HeyGen /v3/voices/speech only accepts STARFISH voice_ids; auto-pick the first
// English public starfish voice when none is pinned. ElevenLabs/Kokoro have
// their own defaults.
export async function resolveVoiceId({ provider, userVoice, lang = "en" }) {
if (userVoice) return userVoice;
if (provider === "elevenlabs") return "21m00Tcm4TlvDq8ikWAM"; // Rachel
if (provider === "kokoro") {
if (lang === "en") return "am_michael";
throw new Error("Kokoro non-English needs an explicit --voice (see references/tts.md)");
}
// heygen — pin a fixed English default so the choice is deterministic. The old
// "first English voice the API returns" drifts whenever HeyGen re-sorts the
// public catalog. Marcia (mature, low female). Override with --voice / request.voice.
if (lang === "en") return "05f19352e8f74b0392a8f411eba40de1"; // Marcia · English · female
// Non-English: no fixed default — fall back to the first matching catalog voice.
const payload = await heygenJSON(`/voices?engine=starfish&type=public&limit=50`, {
headers: heygenAuthHeaders(),
});
const voices = payload.data ?? payload.voices ?? [];
const pick = voices.find((v) => v.language === "English") ?? voices[0];
if (!pick) throw new Error("no public starfish voice to default to — pass --voice");
return pick.voice_id;
}
// ── helpers ─────────────────────────────────────────────────────────────────
export function withWordIds(words) {
return (words ?? []).map((w, i) => ({
id: `w${i}`,
text: w.text,
start: w.start,
end: w.end,
}));
}
// `ffmpeg -i <file>` prints a `Duration: HH:MM:SS.ms` line to stderr even
// though it exits non-zero with no output requested. Parsing pulled out as
// a pure function so the ENOENT fallback below can be tested without
// depending on whether ffprobe/ffmpeg are actually installed on the
// machine running the tests.
export function parseFfmpegDurationBanner(stderrText) {
const match = /Duration:\s*(\d+):(\d+):(\d+(?:\.\d+)?)/.exec(stderrText ?? "");
if (!match) return NaN;
const [, hours, minutes, seconds] = match;
return Number(hours) * 3600 + Number(minutes) * 60 + Number(seconds);
}
// Some "essentials"-style ffmpeg distributions (common on Windows) ship
// ffmpeg.exe without ffprobe.exe. ffprobeDuration's caller (audio.mjs)
// otherwise reads a spurious NaN as "the WAV file is corrupt" and drops an
// already-successfully-synthesized TTS line, rather than "the tool for
// measuring it is missing".
function ffmpegDurationFallback(absPath) {
const r = spawnSync("ffmpeg", ["-i", absPath], { encoding: "utf8" });
return parseFfmpegDurationBanner(r.stderr);
}
export function ffprobeDuration(absPath) {
const r = spawnSync(
"ffprobe",
["-v", "error", "-show_entries", "format=duration", "-of", "default=nw=1:nk=1", absPath],
{ encoding: "utf8" },
);
if (r.error?.code === "ENOENT") return ffmpegDurationFallback(absPath);
if (r.status !== 0) return NaN;
return parseFloat(String(r.stdout).trim());
}
export function resolveNpxCliFromNpmExecPath(
npmExecPath = process.env.npm_execpath,
pathExists = existsSync,
) {
if (!npmExecPath) return null;
const fileName = npmExecPath.replace(/\\/g, "/").split("/").pop()?.toLowerCase();
const npxCliPath =
fileName === "npx-cli.js" ? npmExecPath : join(dirname(npmExecPath), "npx-cli.js");
return pathExists(npxCliPath) ? npxCliPath : null;
}
export function resolveNpxCliPath(
npmExecPath = process.env.npm_execpath,
nodeExecPath = process.env.npm_node_execpath || process.execPath,
pathExists = existsSync,
) {
const fromNpm = resolveNpxCliFromNpmExecPath(npmExecPath, pathExists);
if (fromNpm) return fromNpm;
const besideNode = join(dirname(nodeExecPath), "node_modules", "npm", "bin", "npx-cli.js");
return pathExists(besideNode) ? besideNode : null;
}
export function resolveSpawnCommand(
cmd,
args,
opts = {},
platform = process.platform,
env = process.env,
pathExists = existsSync,
) {
if (cmd !== "npx" || platform !== "win32") {
return { cmd, args, opts: { stdio: "ignore", ...opts } };
}
// On Windows, npx resolves to npx.cmd, which Node cannot execute directly.
// Avoid `shell:true` and the .cmd shim entirely by invoking npm's JS CLI with
// node, preserving request-provided values as argv data instead of shell text.
const nodeExecPath = env.npm_node_execpath || process.execPath;
const npxCliPath = resolveNpxCliPath(env.npm_execpath, nodeExecPath, pathExists);
if (!npxCliPath) return null;
return {
cmd: nodeExecPath,
args: [npxCliPath, ...args.map((arg) => String(arg))],
opts: { stdio: "ignore", windowsHide: true, ...opts },
};
}
// `platform`/`spawnFn` params (default process.platform / the real spawn)
// exist so tests can exercise the win32 branch without mocking node:child_process
// (its ESM exports are non-configurable, so mock.method can't patch it).
// One-shot so a whole batch of TTS lines doesn't repeat the same diagnostic.
let _warnedNpxResolution = false;
/** Test-only: reset the one-shot npx-resolution warning latch. */
export function _resetNpxResolutionWarnForTests() {
_warnedNpxResolution = false;
}
export function spawnP(
cmd,
args,
opts = {},
platform = process.platform,
spawnFn = spawn,
env = process.env,
pathExists = existsSync,
) {
const resolved = resolveSpawnCommand(cmd, args, opts, platform, env, pathExists);
if (!resolved) {
// resolveSpawnCommand only returns null for the npx-on-win32 case where
// neither npm's configured CLI nor the beside-node fallback exists. Without
// this, every call silently returns status:-1 and stdio:"ignore" hides why.
if (!_warnedNpxResolution) {
_warnedNpxResolution = true;
const reason = env.npm_execpath
? `npm_execpath (${env.npm_execpath}) and the beside-node npm fallback could not be found`
: "npm_execpath is unset and the beside-node npm fallback could not be found";
console.error(
`[media-use] Cannot run "${cmd}" on Windows: ${reason}. ` +
`Every "${cmd}" call is being skipped. Install npm with Node, or run via ` +
`\`npx\`/\`npm run\` with a valid npm_execpath.`,
);
}
return Promise.resolve({ status: -1 });
}
return new Promise((resolve) => {
const p = spawnFn(resolved.cmd, resolved.args, resolved.opts);
p.on("exit", (code) => resolve({ status: code ?? -1 }));
p.on("error", () => resolve({ status: -1 }));
});
}
// mp3/whatever bytes → wav 44.1k mono at destWav (ffmpeg detects true format).
function transcodeToWav(bytes, destWav) {
const td = mkdtempSync(join(tmpdir(), "hf-tts-"));
const tmp = join(td, "a.mp3");
writeFileSync(tmp, bytes);
mkdirSync(dirname(destWav), { recursive: true });
const ff = spawnSync(
"ffmpeg",
["-y", "-loglevel", "error", "-i", tmp, "-ar", "44100", "-ac", "1", destWav],
{ stdio: "ignore" },
);
rmSync(td, { recursive: true, force: true });
return ff.status === 0 && existsSync(destWav);
}
const ELEVENLABS_PY = `
import os, sys
from elevenlabs.client import ElevenLabs
from elevenlabs import save
client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])
text = open(sys.argv[1]).read()
audio = client.text_to_speech.convert(
text=text, voice_id=sys.argv[2],
model_id="eleven_multilingual_v2", output_format="mp3_44100_128",
)
save(audio, sys.argv[3])
`;
// ── synthesize one line ───────────────────────────────────────────────────────
// Writes wav at wavAbs. Returns { ok, words, error } — words is the raw
// [{text,start,end}] array for HeyGen (native), or null for ElevenLabs/Kokoro
// (caller must transcribeWav). Never throws; failures return { ok:false, error }
// where `error` states WHY (so the caller can surface it, not a bare "TTS failed").
export async function synthesizeOne({
provider,
text,
voiceId,
lang = "en",
speed = 1.0,
wavAbs,
hyperframesDir,
}) {
if (provider === "heygen") return synthesizeHeygen({ text, voiceId, lang, speed, wavAbs });
if (provider === "elevenlabs") {
// The Python helper writes straight to wavAbs; unlike heygen (transcodeToWav)
// and kokoro (the `hyperframes tts` CLI), it does NOT create the parent dir,
// so on a fresh project (no assets/voice/ yet) the save fails and the line is
// silently dropped as "TTS failed - omitted". Create it first, like the other
// providers do. Guarded so a mkdir failure (EACCES/EROFS) returns
// { ok:false } like the rest of this branch rather than throwing (the
// function's contract is "never throws; failures return { ok:false }").
try {
mkdirSync(dirname(wavAbs), { recursive: true });
} catch {
return { ok: false, words: null };
}
const { cmd, args } = pythonInvocation([
"-c",
ELEVENLABS_PY,
writeTmpText(text),
voiceId,
wavAbs,
]);
const r = await spawnP(cmd, args, {});
return synthResult(r, wavAbs, "elevenlabs (python)");
}
// kokoro — via the published CLI; --output is relative to the project dir.
const wavRel = relTo(hyperframesDir, wavAbs);
const args = ["hyperframes", "tts", writeTmpText(text), "--voice", voiceId, "--output", wavRel];
if (lang !== "en") args.push("--lang", lang);
const r = await spawnP("npx", args, { cwd: hyperframesDir });
return synthResult(r, wavAbs, "kokoro (npx hyperframes tts)");
}
// Shape a spawn result into { ok, words, error }, naming why on failure so the
// caller surfaces it instead of a bare "TTS failed".
export function synthResult(r, wavAbs, label) {
if (r.status === 0 && existsSync(wavAbs)) return { ok: true, words: null };
const why =
r.status !== 0 ? `${label} exited with status ${r.status}` : `${label} produced no wav file`;
return { ok: false, words: null, error: why };
}
// `deps` is injectable for tests; production uses the real network/ffmpeg impls.
// Every failure path returns an `error` string so the caller can surface WHY a
// line was dropped instead of the bare "TTS failed" that hid the real cause
// (e.g. an HTTP 402 plan_upgrade_required thrown by heygenJSON was swallowed).
export async function synthesizeHeygen({ text, voiceId, lang, speed, wavAbs }, deps = {}) {
const requestJSON = deps.heygenJSON ?? heygenJSON;
const authHeaders = deps.heygenAuthHeaders ?? heygenAuthHeaders;
const fetchImpl = deps.fetch ?? fetch;
const transcode = deps.transcodeToWav ?? transcodeToWav;
try {
const body = { text, voice_id: voiceId, speed };
if (lang !== "en") body.language = lang;
const payload = await requestJSON(`/voices/speech`, {
method: "POST",
headers: authHeaders(),
body,
});
const inner = payload.data ?? payload;
if (!inner.audio_url) {
return { ok: false, words: null, error: "HeyGen /voices/speech returned no audio_url" };
}
const res = await fetchImpl(inner.audio_url);
if (!res.ok) {
return { ok: false, words: null, error: `audio_url fetch failed: HTTP ${res.status}` };
}
const bytes = Buffer.from(await res.arrayBuffer());
// .wav output → transcode to 44.1k mono; .mp3 → raw bytes (no ffmpeg). The
// engine always asks for .wav; the standalone heygen-tts CLI may ask for .mp3.
if (wavAbs.endsWith(".wav")) {
if (!transcode(bytes, wavAbs)) {
return {
ok: false,
words: null,
error: "wav transcode failed (ffmpeg)",
};
}
} else {
mkdirSync(dirname(wavAbs), { recursive: true });
writeFileSync(wavAbs, bytes);
}
const words = Array.isArray(inner.word_timestamps)
? inner.word_timestamps
.filter((w) => w && typeof w.word === "string" && isFinite(w.start) && isFinite(w.end))
.filter((w) => !/^<.*>$/.test(w.word.trim())) // drop <start>/<end> sentinels
.map((w) => ({ text: w.word, start: w.start, end: w.end }))
: [];
return { ok: true, words };
} catch (e) {
return { ok: false, words: null, error: e?.message ? String(e.message) : String(e) };
}
}
// ElevenLabs/Kokoro have no word timings — run Whisper over the wav. Returns the
// flat [{id,text,start,end}] word array, or null. Each call uses a throwaway
// --dir so parallel scenes don't collide on transcript.json.
export async function transcribeWav({ wavRel, lang = "en", hyperframesDir }) {
const model = lang === "en" ? "small.en" : "small";
const td = mkdtempSync(join(tmpdir(), "hf-trans-"));
const args = ["hyperframes", "transcribe", wavRel, "--model", model, "--dir", td];
if (lang !== "en") args.push("--language", lang);
const r = await spawnP("npx", args, { cwd: hyperframesDir });
let words = null;
if (r.status === 0) {
const src = join(td, "transcript.json");
if (existsSync(src)) {
try {
const arr = JSON.parse(readFileSync(src, "utf8"));
if (Array.isArray(arr) && arr.length) words = arr;
} catch {}
}
}
rmSync(td, { recursive: true, force: true });
return words;
}
// ── tiny local utils ──────────────────────────────────────────────────────────
function writeTmpText(text) {
const td = mkdtempSync(join(tmpdir(), "hf-txt-"));
const p = join(td, "line.txt");
writeFileSync(p, text);
return p;
}
function relTo(base, abs) {
return abs.startsWith(base + "/") ? abs.slice(base.length + 1) : abs;
}
@@ -0,0 +1,158 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { EventEmitter } from "node:events";
import {
resolveNpxCliFromNpmExecPath,
resolveNpxCliPath,
resolveSpawnCommand,
spawnP,
_resetNpxResolutionWarnForTests,
} from "./tts.mjs";
// Regression: on Windows, npx resolves to npx.cmd, which spawn() cannot exec
// without shell:true — it fails ENOENT, silently swallowed as ok:false by the
// caller. spawnP takes injectable platform/spawnFn params so this doesn't
// need to touch the real process.platform or mock node:child_process (whose
// ESM exports are non-configurable).
function fakeSpawn(captured) {
return (cmd, args, opts) => {
captured.push({ cmd, args, opts });
const p = new EventEmitter();
setImmediate(() => p.emit("exit", 0));
return p;
};
}
const envWithNpxCli = {
npm_execpath: "/opt/node/lib/node_modules/npm/bin/npm-cli.js",
npm_node_execpath: "/opt/node/bin/node",
};
const npxCliPath = "/opt/node/lib/node_modules/npm/bin/npx-cli.js";
const pathExists = (path) => path === npxCliPath;
test("resolveNpxCliFromNpmExecPath finds npx-cli next to npm-cli", () => {
assert.equal(resolveNpxCliFromNpmExecPath(envWithNpxCli.npm_execpath, pathExists), npxCliPath);
});
test("resolveNpxCliPath finds npx-cli beside node when npm_execpath is unset", () => {
const node = "C:/Program Files/nodejs/node.exe";
const expected = "C:/Program Files/nodejs/node_modules/npm/bin/npx-cli.js";
assert.equal(
resolveNpxCliPath(undefined, node, (path) => path === expected),
expected,
);
});
test("resolveSpawnCommand routes npx through node+npx-cli on win32 without shell:true", () => {
const resolved = resolveSpawnCommand(
"npx",
["hyperframes", "tts", "C:\\Users\\Test User\\line.txt", "--voice", "am_michael"],
{},
"win32",
envWithNpxCli,
pathExists,
);
assert.ok(resolved);
assert.equal(resolved.cmd, envWithNpxCli.npm_node_execpath);
assert.deepEqual(resolved.args, [
npxCliPath,
"hyperframes",
"tts",
"C:\\Users\\Test User\\line.txt",
"--voice",
"am_michael",
]);
assert.equal(resolved.opts.shell, undefined);
});
test("resolveSpawnCommand preserves Windows npx shell metacharacters as argv data", () => {
const resolved = resolveSpawnCommand(
"npx",
["hyperframes", "tts", "hello & calc"],
{},
"win32",
envWithNpxCli,
pathExists,
);
assert.ok(resolved);
assert.deepEqual(resolved.args, [npxCliPath, "hyperframes", "tts", "hello & calc"]);
});
test("spawnP uses the resolved node+npx-cli command for npx on win32", async () => {
const captured = [];
await spawnP(
"npx",
["hyperframes", "tts"],
{},
"win32",
fakeSpawn(captured),
envWithNpxCli,
pathExists,
);
assert.equal(captured.length, 1);
assert.equal(captured[0].cmd, envWithNpxCli.npm_node_execpath);
assert.deepEqual(captured[0].args, [npxCliPath, "hyperframes", "tts"]);
assert.equal(captured[0].opts.shell, undefined);
});
test("spawnP does not enable shell for npx on darwin/linux", async () => {
const captured = [];
await spawnP("npx", ["hyperframes", "tts"], {}, "darwin", fakeSpawn(captured));
assert.equal(captured[0].cmd, "npx");
assert.deepEqual(captured[0].args, ["hyperframes", "tts"]);
assert.equal(captured[0].opts.shell, undefined);
});
test("spawnP does not enable shell for non-npx commands even on win32", async () => {
const captured = [];
await spawnP("python3", ["-c", "pass"], {}, "win32", fakeSpawn(captured));
assert.equal(captured[0].cmd, "python3");
assert.deepEqual(captured[0].args, ["-c", "pass"]);
assert.equal(captured[0].opts.shell, undefined);
});
test("spawnP resolves npx beside node when npm_execpath is unset on win32", async () => {
_resetNpxResolutionWarnForTests();
const captured = [];
const node = "C:/Program Files/nodejs/node.exe";
const npxCli = "C:/Program Files/nodejs/node_modules/npm/bin/npx-cli.js";
const result = await spawnP(
"npx",
["hyperframes", "tts"],
{},
"win32",
fakeSpawn(captured),
{ npm_node_execpath: node },
(path) => path === npxCli,
);
assert.equal(result.status, 0);
assert.equal(captured.length, 1);
assert.equal(captured[0].cmd, node);
assert.deepEqual(captured[0].args, [npxCli, "hyperframes", "tts"]);
});
test("spawnP warns once with an accurate diagnostic when neither npx path exists", async () => {
_resetNpxResolutionWarnForTests();
const errors = [];
const originalError = console.error;
console.error = (message) => errors.push(String(message));
try {
const env = { npm_execpath: "C:/missing/npm-cli.js", npm_node_execpath: "C:/node/node.exe" };
const missing = () => false;
assert.equal(
(await spawnP("npx", ["hyperframes", "tts"], {}, "win32", fakeSpawn([]), env, missing))
.status,
-1,
);
assert.equal(
(await spawnP("npx", ["hyperframes", "tts"], {}, "win32", fakeSpawn([]), env, missing))
.status,
-1,
);
} finally {
console.error = originalError;
}
assert.equal(errors.length, 1);
assert.match(errors[0], /npm_execpath \(C:\/missing\/npm-cli\.js\)/);
assert.doesNotMatch(errors[0], /npm_execpath is not set/);
});
@@ -0,0 +1,157 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync, chmodSync, rmSync, existsSync } from "node:fs";
import { join, dirname } from "node:path";
import { tmpdir } from "node:os";
import {
parseFfmpegDurationBanner,
ffprobeDuration,
synthesizeOne,
synthesizeHeygen,
synthResult,
} from "./tts.mjs";
test("parseFfmpegDurationBanner reads ffmpeg's stderr Duration line", () => {
const stderr = [
"ffmpeg version 6.0",
"Input #0, wav, from 'a.wav':",
" Duration: 00:00:03.42, bitrate: 705 kb/s",
"At least one output file must be specified",
].join("\n");
assert.equal(parseFfmpegDurationBanner(stderr), 3.42);
});
test("parseFfmpegDurationBanner handles an hours component", () => {
const stderr = " Duration: 01:02:03.50, start: 0.000000, bitrate: 128 kb/s";
assert.equal(parseFfmpegDurationBanner(stderr), 3723.5);
});
test("parseFfmpegDurationBanner returns NaN when there is no Duration line", () => {
assert.ok(Number.isNaN(parseFfmpegDurationBanner("ffmpeg: command not found")));
assert.ok(Number.isNaN(parseFfmpegDurationBanner("")));
assert.ok(Number.isNaN(parseFfmpegDurationBanner(undefined)));
});
// Regression for the actual bug: ffprobeDuration used to collapse "ffprobe
// binary is missing" (ENOENT — the "essentials"-style Windows ffmpeg build
// with no ffprobe.exe) and "file is genuinely unreadable" into the same NaN,
// giving audio.mjs no way to tell "measure differently" from "give up".
//
// Builds an isolated PATH containing only a fake `ffmpeg` stub (no `ffprobe`
// at all) so ffprobeDuration's spawnSync("ffprobe", ...) call ENOENTs for
// real, then verifies it recovers the duration via the ffmpeg fallback
// instead of returning NaN.
test("ffprobeDuration falls back to ffmpeg when the ffprobe binary itself is missing", () => {
const dir = mkdtempSync(join(tmpdir(), "tts-ffprobe-fallback-"));
const fakeFfmpeg = join(dir, "ffmpeg");
writeFileSync(
fakeFfmpeg,
"#!/bin/sh\necho 'Duration: 00:00:02.50, start: 0.000000, bitrate: 128 kb/s' 1>&2\nexit 1\n",
);
chmodSync(fakeFfmpeg, 0o755);
const originalPath = process.env.PATH;
try {
process.env.PATH = dir; // only the fake ffmpeg resolves; no real ffprobe on this PATH
assert.equal(ffprobeDuration("/does/not/matter.wav"), 2.5);
} finally {
process.env.PATH = originalPath;
rmSync(dir, { recursive: true, force: true });
}
});
test("ffprobeDuration returns NaN when neither ffprobe nor ffmpeg resolve", () => {
const dir = mkdtempSync(join(tmpdir(), "tts-no-binaries-"));
const originalPath = process.env.PATH;
try {
process.env.PATH = dir; // empty directory — nothing resolves
assert.ok(Number.isNaN(ffprobeDuration("/does/not/matter.wav")));
} finally {
process.env.PATH = originalPath;
rmSync(dir, { recursive: true, force: true });
}
});
test("synthesizeOne(elevenlabs) creates the output dir before writing", async () => {
const dir = mkdtempSync(join(tmpdir(), "tts-el-mkdir-"));
const wavAbs = join(dir, "assets", "voice", "line-0.wav"); // nested, not yet created
const savedKey = process.env.ELEVENLABS_API_KEY;
try {
// Unset the key so the Python side fails fast — the mkdir must run before
// the spawn regardless, which is what this guards.
delete process.env.ELEVENLABS_API_KEY;
await synthesizeOne({
provider: "elevenlabs",
text: "hi",
voiceId: "v",
wavAbs,
hyperframesDir: dir,
});
assert.ok(existsSync(dirname(wavAbs)), "output directory should be created");
} finally {
if (savedKey === undefined) delete process.env.ELEVENLABS_API_KEY;
else process.env.ELEVENLABS_API_KEY = savedKey;
rmSync(dir, { recursive: true, force: true });
}
});
test("synthesizeHeygen surfaces a thrown HTTP error (e.g. 402) instead of swallowing it", async () => {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
{
heygenAuthHeaders: () => ({}),
heygenJSON: async () => {
throw new Error("HeyGen POST /voices/speech → HTTP 402\nplan_upgrade_required");
},
},
);
assert.equal(res.ok, false);
assert.match(res.error, /402/);
assert.match(res.error, /plan_upgrade_required/);
});
test("synthesizeHeygen surfaces a failed audio_url fetch with its status", async () => {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
{
heygenAuthHeaders: () => ({}),
heygenJSON: async () => ({ data: { audio_url: "http://audio.example/x" } }),
fetch: async () => ({ ok: false, status: 403 }),
},
);
assert.equal(res.ok, false);
assert.match(res.error, /HTTP 403/);
});
test("synthesizeHeygen reports a missing audio_url", async () => {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
{ heygenAuthHeaders: () => ({}), heygenJSON: async () => ({}) },
);
assert.equal(res.ok, false);
assert.match(res.error, /no audio_url/);
});
test("synthesizeHeygen reports wav transcode failures", async () => {
const dir = mkdtempSync(join(tmpdir(), "hf-tts-test-"));
try {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: join(dir, "voice.wav") },
{
heygenAuthHeaders: () => ({}),
heygenJSON: async () => ({ data: { audio_url: "http://audio.example/x" } }),
fetch: async () => ({ ok: true, status: 200, arrayBuffer: async () => new ArrayBuffer(0) }),
transcodeToWav: () => false,
},
);
assert.equal(res.ok, false);
assert.equal(res.error, "wav transcode failed (ffmpeg)");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("synthResult names a non-zero subprocess exit", () => {
const res = synthResult({ status: 2 }, "/tmp/none.wav", "kokoro (npx hyperframes tts)");
assert.equal(res.ok, false);
assert.match(res.error, /kokoro .* exited with status 2/);
});
@@ -0,0 +1,128 @@
#!/usr/bin/env python3
"""Generate BGM using Google Lyria RealTime API.
Usage:
python lyria-recipe.py --output <path> --duration <seconds> [tuning flags]
Requires:
$GOOGLE_API_KEY or $GEMINI_API_KEY environment variable (treated as aliases).
pip install google-genai python-dotenv. audio.mjs Step 4b installs these on
demand when a key is set but google.genai is not importable; if that install
fails it falls back to local MusicGen rather than leaving the video with no BGM.
"""
from __future__ import annotations
import argparse
import asyncio
import os
import sys
import wave
from pathlib import Path
DEFAULT_PROMPT = "Uplifting corporate tech, bright and modern, gentle piano with synth pads"
SAMPLE_RATE = 48000
CHANNELS = 2
SAMPLE_WIDTH = 2 # 16-bit
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description="Generate BGM via Google Lyria RealTime.")
p.add_argument("--output", required=True, help="Output WAV path.")
p.add_argument("--duration", type=float, required=True, help="Target duration in seconds.")
p.add_argument("--prompt", default=DEFAULT_PROMPT, help="Mood / instrumentation prompt.")
p.add_argument("--negative-prompt", default=None, help="Styles to exclude (optional).")
p.add_argument("--bpm", type=int, default=110)
p.add_argument("--brightness", type=float, default=0.8, help="0-1, higher = brighter mood.")
p.add_argument("--density", type=float, default=0.5, help="0-1, higher = fuller mix.")
p.add_argument(
"--scale",
default="MAJOR",
help="MAJOR / MINOR / PENTATONIC / etc. — see google.genai.types.Scale. Pass empty string for none.",
)
return p.parse_args()
async def generate_bgm(args: argparse.Namespace) -> dict:
from google import genai
from google.genai import types
api_key = os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY") or ""
if not api_key:
raise RuntimeError("Neither GOOGLE_API_KEY nor GEMINI_API_KEY is set.")
client = genai.Client(
api_key=api_key,
http_options={"api_version": "v1alpha"},
)
out_path = Path(args.output)
out_path.parent.mkdir(parents=True, exist_ok=True)
target_bytes = int(args.duration * SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH)
cfg: dict = {"bpm": args.bpm, "temperature": 1.0}
if args.density is not None:
cfg["density"] = args.density
if args.brightness is not None:
cfg["brightness"] = args.brightness
if args.scale:
scale_enum = getattr(types.Scale, args.scale, None)
if scale_enum:
cfg["scale"] = scale_enum
prompts = [types.WeightedPrompt(text=args.prompt, weight=1.0)]
if args.negative_prompt:
prompts.append(types.WeightedPrompt(text=args.negative_prompt, weight=-1.0))
buf = bytearray()
timeout = args.duration + 8
async with client.aio.live.music.connect(
model="models/lyria-realtime-exp",
) as session:
await session.set_weighted_prompts(prompts=prompts)
await session.set_music_generation_config(
config=types.LiveMusicGenerationConfig(**cfg),
)
await session.play()
async def collect():
while len(buf) < target_bytes:
async for msg in session.receive():
sc = msg.server_content
if sc and sc.audio_chunks:
for chunk in sc.audio_chunks:
buf.extend(chunk.data)
if len(buf) >= target_bytes:
return
await asyncio.sleep(1e-6)
try:
await asyncio.wait_for(collect(), timeout=timeout)
except TimeoutError:
print(f"Timeout after {timeout:.0f}s, collected {len(buf)} bytes", file=sys.stderr)
audio = bytes(buf[:target_bytes])
with wave.open(str(out_path), "wb") as wf:
wf.setnchannels(CHANNELS)
wf.setsampwidth(SAMPLE_WIDTH)
wf.setframerate(SAMPLE_RATE)
wf.writeframes(audio)
actual_duration = len(audio) / (SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH)
print(f"BGM: {out_path} ({actual_duration:.2f}s)")
return {"file": str(out_path), "duration_sec": round(actual_duration, 2)}
def main() -> None:
args = parse_args()
try:
asyncio.run(generate_bgm(args))
except RuntimeError as exc:
print(f"BGM generation failed: {exc}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/env node
// Phase 4c pre-assemble helper — wait for detached BGM, then write status.
//
// audio.mjs may launch Lyria / MusicGen in a detached process so voice work can
// keep moving. Before assemble-index.mjs decides whether to emit the BGM audio
// track, this script gives the background renderer a bounded chance to finish
// and converts log/process state into a small bgm_status.json file.
//
// Always exits 0 for normal pipeline use: missing/failed BGM should not block a
// voice/captions/SFX render. Structural invocation errors still exit 1.
//
// Usage:
// node wait-bgm.mjs --audio-meta ./audio_meta.json --hyperframes . \
// [--timeout-ms 120000] [--interval-ms 2000] [--out ./bgm_status.json]
import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
const argv = process.argv.slice(2);
const flag = (name, def) => {
const i = argv.indexOf(`--${name}`);
return i >= 0 && i + 1 < argv.length ? argv[i + 1] : def;
};
function die(msg) {
console.error(`✗ wait-bgm.mjs: ${msg}`);
process.exit(1);
}
const audioMetaPath = resolve(flag("audio-meta", "./audio_meta.json"));
const hyperframesDir = resolve(flag("hyperframes", "."));
const outPath = resolve(flag("out", join(hyperframesDir, "bgm_status.json")));
const timeoutMs = Math.max(0, Number(flag("timeout-ms", "120000")) || 0);
const intervalMs = Math.max(250, Number(flag("interval-ms", "2000")) || 2000);
function sleep(ms) {
return new Promise((resolveSleep) => setTimeout(resolveSleep, ms));
}
function isProcessAlive(pid) {
if (!pid || !Number.isFinite(Number(pid))) return false;
try {
process.kill(Number(pid), 0);
return true;
} catch {
return false;
}
}
function readTail(path, maxChars = 6000) {
if (!path || !existsSync(path)) return "";
const s = statSync(path);
const txt = readFileSync(path, "utf8");
return txt.slice(Math.max(0, txt.length - Math.min(maxChars, s.size)));
}
function detectFailure(logTail) {
if (!logTail) return "";
const lines = logTail.split("\n");
// Bare "out of range" over-matched benign BGM-renderer logs (e.g. a "sample rate
// out of range, resampling" notice), mislabelling a healthy track as failed and
// silently dropping the music. Anchor to the actual crash strings instead:
// Python "(list) index out of range" and torch "index … out of bounds".
const idx = lines.findIndex((line) =>
/(Traceback|IndexError|RuntimeError|Exception|Killed|No space left|Cannot allocate|index out of range|out of bounds)/i.test(
line,
),
);
if (idx < 0) return "";
return lines.slice(idx).join("\n").trim();
}
function writeStatus(status) {
const payload = {
generated_at: new Date().toISOString(),
...status,
};
writeFileSync(outPath, JSON.stringify(payload, null, 2) + "\n");
return payload;
}
if (!existsSync(audioMetaPath)) die(`audio_meta.json missing at ${audioMetaPath}`);
const audioMeta = JSON.parse(readFileSync(audioMetaPath, "utf8"));
const bgmPath = audioMeta.bgm?.path || "";
const bgmAbsPath = bgmPath ? join(hyperframesDir, bgmPath) : "";
const logPath = audioMeta.bgm_log || "";
const pid = audioMeta.bgm_pid || null;
const base = {
enabled: Boolean(audioMeta.bgm_pending && bgmPath),
provider: audioMeta.bgm_provider || null,
mode: audioMeta.bgm_mode || null,
path: bgmPath || null,
log: logPath || null,
pid,
target_duration_s: audioMeta.bgm_target_duration_s || null,
seed_duration_s: audioMeta.bgm_seed_duration_s || null,
loop_count: audioMeta.bgm_loop_count || null,
timeout_ms: timeoutMs,
};
if (!base.enabled) {
const status = writeStatus({
...base,
status: "disabled",
ready: false,
waited_ms: 0,
message: "BGM not requested or disabled in audio_meta.json.",
});
console.log(`✓ bgm: ${status.status} (${status.message})`);
process.exit(0);
}
const started = Date.now();
let lastFailure = "";
let lastTail = "";
while (Date.now() - started <= timeoutMs) {
if (existsSync(bgmAbsPath)) {
const size = statSync(bgmAbsPath).size;
writeStatus({
...base,
status: "ready",
ready: true,
waited_ms: Date.now() - started,
size_bytes: size,
message: `BGM ready at ${bgmPath}.`,
});
console.log(`✓ bgm: ready (${bgmPath}, ${size}B)`);
process.exit(0);
}
lastTail = readTail(logPath);
lastFailure = detectFailure(lastTail);
const alive = isProcessAlive(pid);
if (lastFailure || (!alive && logPath && existsSync(logPath))) {
const message = lastFailure
? `BGM renderer failed; see ${logPath}.`
: `BGM renderer exited without writing ${bgmPath}; see ${logPath}.`;
const status = writeStatus({
...base,
status: "failed",
ready: false,
waited_ms: Date.now() - started,
process_alive: alive,
message,
error_tail: lastFailure || lastTail.slice(-2000),
});
console.log(`! bgm: failed (${status.message})`);
process.exit(0);
}
if (timeoutMs === 0) break;
await sleep(Math.min(intervalMs, Math.max(0, timeoutMs - (Date.now() - started))));
}
const status = writeStatus({
...base,
status: "timeout",
ready: false,
waited_ms: Date.now() - started,
process_alive: isProcessAlive(pid),
message: `Timed out waiting for ${bgmPath}; assemble-index will skip BGM if still absent.`,
log_tail: lastTail.slice(-2000),
});
console.log(`! bgm: timeout after ${status.waited_ms}ms (${bgmPath})`);
+28
View File
@@ -0,0 +1,28 @@
# LUT library (authoring)
`index.json` is the agent-consumed catalog of color-grade looks. Each entry resolves
on demand — no `.cube` bodies are committed to the repo.
Each look has:
- `id`, `description`, `tags`, `intensity` — matching + application metadata.
- `url` (optional) — a hosted `.cube` downloaded, validated, and frozen at resolve
time, exactly like bgm/image assets.
- `params` (optional) — a deterministic `buildCube` spec used offline (`--local-only`)
or as a fallback if the `url` download/validation fails.
An entry needs at least one of `url` or `params`; prefer both (CDN url with a params
fallback) so resolution is never blocked on the network.
## Hosting a new look (operators)
1. Generate the `.cube` (e.g. `resolve -t lut --params '{...}'` or a graded export).
2. Upload it to the public CDN origin bucket:
```
aws s3 cp <id>.cube s3://heygen-public/luts/<id>.cube
```
It is then served at `https://static.heygen.ai/luts/<id>.cube` (CloudFront).
3. Add an entry to `index.json` with that `url` (and ideally a `params` fallback).
+57
View File
@@ -0,0 +1,57 @@
{
"notes": "Look entries resolve on-demand from a CDN .cube `url` (downloaded + frozen, like bgm/image); `params` is a deterministic buildCube fallback used offline (--local-only) or if the download/validation fails. No .cube bodies are committed. See README.md to author or host a new look.",
"looks": [
{
"id": "teal-orange-blockbuster",
"description": "Teal shadows and warm orange highlights for blockbuster-style cinematic footage.",
"tags": ["teal", "orange", "blockbuster", "cinematic", "split tone", "movie"],
"intensity": 0.85,
"url": "https://static.heygen.ai/luts/teal-orange-blockbuster.cube",
"params": {
"contrast": 0.18,
"saturation": 0.08,
"vibrance": 0.12,
"splitTone": {
"intensity": 0.62,
"balance": 0.52,
"shadows": [-0.04, 0.05, 0.09],
"highlights": [0.1, 0.04, -0.03]
}
}
},
{
"id": "bleach-bypass",
"description": "High-contrast desaturated bleach bypass look with strong blacks.",
"tags": ["bleach", "bypass", "desaturated", "high contrast", "gritty", "film"],
"intensity": 0.8,
"url": "https://static.heygen.ai/luts/bleach-bypass.cube",
"params": {
"blacks": 0.04,
"shadows": -0.08,
"highlights": 0.08,
"whites": 0.18,
"contrast": 0.55,
"temperature": -0.02,
"saturation": -0.72,
"vibrance": -0.25
}
},
{
"id": "film-fade",
"description": "Soft faded film wash with lifted blacks and warm highlights.",
"tags": ["film", "fade", "faded", "wash", "warm", "vintage"],
"intensity": 0.75,
"url": "https://static.heygen.ai/luts/film-fade.cube",
"params": {
"blacks": 0.35,
"shadows": 0.18,
"highlights": 0.02,
"whites": -0.03,
"contrast": -0.28,
"temperature": 0.16,
"saturation": -0.12,
"vibrance": -0.08
}
}
]
}
+228
View File
@@ -0,0 +1,228 @@
# Media operations: agent guidance
media-use resolves and remembers assets. For **operating** on them: cutting,
reframing, stitching, transforming, it does not wrap every action as a bespoke
command. Instead it points you at the right local tool (decision OP1). Run the
tool, then register the output with `resolve --from <output> --type <type>` so the
result lands in the ledger and the global cache like any other asset.
All tools below are local and free. ffmpeg is assumed present (it backs the
engine already).
## Cut / trim: keep a slice
```bash
ffmpeg -i in.mp4 -ss 00:00:12 -to 00:00:20 -c copy out.mp4 # 0:120:20, no re-encode
```
In-composition trimming usually needs **no new file**: a clip plays a sub-window
via `data-media-start` + `data-duration` (see hyperframes-core). Only cut a
physical file when exporting/assembling outside the composition.
## Reframe / crop: change aspect ratio
```bash
# 16:9 -> 9:16, crop centered
ffmpeg -i in.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" out.mp4
```
For a non-destructive crop, set a `clip-path` on the element in the composition
itself (render-time, source file untouched) instead of re-encoding with ffmpeg.
## Montage / stitch: join clips
```bash
printf "file '%s'\n" a.mp4 b.mp4 c.mp4 > list.txt
ffmpeg -f concat -safe 0 -i list.txt -c copy out.mp4
```
## Silence-cut / highlight: trim dead air, grab the best moment
```bash
auto-editor in.mp4 --edit audio:threshold=4% -o tight.mp4 # pip install auto-editor
scenedetect -i in.mp4 detect-adaptive list-scenes # pip install scenedetect
```
## Transforms with a quality choice (process)
These have a local option AND a higher-quality HeyGen-CLI option. Run the local
one for free/offline; use the HeyGen CLI when quality matters. Showing the user
a **side-by-side** (local vs HeyGen) is the honest way to let them choose.
| Op | Local (free) | HeyGen CLI (quality) |
| ------------------ | -------------------------------------------------- | --------------------------- |
| Background removal | `hyperframes remove-background in.png` (u2net) | `heygen background-removal` |
| Upscale | `realesrgan-ncnn-vulkan -i in.png -o out.png -s 4` | n/a |
| Lipsync (dub) | n/a | `heygen lipsync` |
| Translate | n/a | `heygen video-translate` |
After any op: `resolve --from out.ext --type <type>` to register the derived
asset (it records provenance and auto-promotes to the global cache).
> ponytail: media-use doesn't re-wrap ffmpeg/heygen here, that's deliberate
> (OP1). The value it adds is the ledger + global reuse on the _output_, via
> `--from`. Add a thin `process` verb only if agents repeatedly fumble these
> recipes.
## Transcription (default: Parakeet, better than whisper.cpp)
`transcribe.mjs` is the default local transcription path. It runs **NVIDIA
Parakeet-TDT via parakeet-mlx**, which beats whisper.cpp on the Open ASR
Leaderboard (avg WER ~6.05% vs 7.44%; on NOISY audio 4.73% vs 5.96%, where
whisper-large-v3 hallucinated to 308% WER on meetings) and is 5-10x faster.
It emits `{ text, words:[{text,start,end}] }` with word timestamps (merged from
Parakeet's sub-word tokens), feeding transcript-cut, captions, and the audio
engine directly.
```bash
# install once: uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx
node <SKILL_DIR>/scripts/transcribe.mjs --input talk.mp4 --out talk.transcribe.json
# equivalently, the hyperframes CLI has Parakeet built in (auto-detects it, whisper fallback):
npx hyperframes transcribe talk.mp4 --engine parakeet # or --engine auto (default)
```
VERIFIED on 24GB: accurate, ~3s (cached) for 8s audio. Parakeet covers English +
25 European languages. For other languages, or when parakeet-mlx is not
installed, transcribe.mjs auto-falls-back to whisper.cpp (99 languages) via
`hyperframes transcribe`. `--engine parakeet|whisper` forces one. (Cohere
Transcribe tops the leaderboard on paper but its mlx-audio quants produced
garbage and ran 40-70x slower on a Mac in testing, so it is not wired in.)
## Text-based editing (transcript cut)
`transcript-cut.mjs` is a compiler, not a wrapper: it turns word timestamps and
agent cut decisions into exact kept segments. It is provided even though the rest
of this file is guidance-only.
```bash
node <SKILL_DIR>/scripts/transcript-cut.mjs \
--input talk.mp4 \
--transcript talk.transcribe.json \
--remove "12.41-15.02,88.3-91.7" \
--remove-fillers "um,uh,like" \
--cut-silence 0.8 \
--out talk.cut.mp4
resolve --from talk.cut.mp4 --type video
```
Use `--plan` first when you want to inspect the kept segment JSON before encoding.
## Ducking (declare in-composition / bake for export)
B1, declare ducking in the composition. `audio-duck.mjs` emits GSAP volume
keyframes. Paste them into the composition timeline, the source file stays
untouched.
```bash
node <SKILL_DIR>/scripts/audio-duck.mjs \
--meta audio_meta.json \
--target "#bgm" \
--composition index.html
```
```js
// auto-duck: #bgm under narration (generated; base volume 0.6)
tl.to("#bgm", { volume: 0.15, duration: 0.15 }, 3.42);
tl.to("#bgm", { volume: 0.6, duration: 0.4 }, 9.87);
```
B2, bake ducking only for exported or standalone files.
```bash
ffmpeg -i bgm.mp3 -i voice.wav \
-filter_complex "[0][1]sidechaincompress=threshold=0.03:ratio=8:attack=200:release=400[ducked]" \
-map "[ducked]" bgm.ducked.wav
```
Declare inside compositions. Bake only for assets leaving the hyperframes
pipeline.
## Publish loudness
Two-pass `loudnorm` measures first, then applies the measured values with the
target LUFS baked in.
Socials target, -14 LUFS:
```bash
ffmpeg -i mix.wav \
-af loudnorm=I=-14:TP=-1.5:LRA=11:print_format=json \
-f null -
ffmpeg -i mix.wav \
-af loudnorm=I=-14:TP=-1.5:LRA=11:measured_I=<input_i>:measured_TP=<input_tp>:measured_LRA=<input_lra>:measured_thresh=<input_thresh>:offset=<target_offset>:linear=true:print_format=summary \
mix.social.wav
```
Podcast target, -16 LUFS:
```bash
ffmpeg -i mix.wav \
-af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json \
-f null -
ffmpeg -i mix.wav \
-af loudnorm=I=-16:TP=-1.5:LRA=11:measured_I=<input_i>:measured_TP=<input_tp>:measured_LRA=<input_lra>:measured_thresh=<input_thresh>:offset=<target_offset>:linear=true:print_format=summary \
mix.podcast.wav
```
## Generate: images (local first, cloud upsell)
`resolve --type image` retrieves from the HeyGen catalog first; on a miss it
GENERATES. Two paths, best-for-the-machine picked automatically:
1. **Local (default, free, private): mflux** (FLUX-on-MLX). `resolve` spec-checks
AVAILABLE RAM and runs the best FLUX-class model that fits, via
`scripts/lib/local-models.mjs` (`imagegen` ladder) + `mflux-provider.mjs`.
The RAM ladder (agent sees it via `describeModelLadder("imagegen", specs)`):
| Tier | Model | Needs (available RAM) | Notes |
| ------ | -------------------- | --------------------- | ----------------------------------- |
| medium | FLUX.1 schnell int4 | ~8GB (`--low-ram`) | ~20s/512px on 24GB. VERIFIED. Fast. |
| large | FLUX.2 Klein 4B int4 | ~32GB | higher quality, full-resident |
| xlarge | Qwen-Image | ~64GB | top quality, 64GB+ Macs only |
Gotchas baked into the table: the official FLUX repos are HF-gated, so it
points at non-gated community 4-bit re-uploads; and `--low-ram` is MANDATORY
at the medium tier (without it a 768x512 run swap-thrashed to 90 minutes on
24GB; with it, 20 seconds).
2. **Cloud upsell (better quality): the `codex` CLI** `image_gen` tool, on the
user's ChatGPT subscription (codex owns auth, no key here, no per-call
charge). It is the automatic fallback when no local model fits AND the
explicit "make it better" choice on any machine. Users who just want codex
can ask for it directly. Verified: prompt -> raster -> frozen + ledgered.
`--local-only` keeps mflux (once cached) and skips codex (network).
## Generate: video (local first, HeyGen avatar upsell)
Operate-on-video ships now; GENERATING video is local-first with a HeyGen
avatar upsell (decision X3).
- **Local (default): LTX 2.3 on MLX** via `dgrauet/ltx-2-mlx`, the `videogen`
ladder in `local-models.mjs`. Generative clips (t2v / i2v), spec-gated to RAM.
Verified on 24GB: 512x320 x 33f with audio.
- **HeyGen avatar upsell (better, script-driven): the `heygen` CLI**, NOT the
raw API. For a talking-head / avatar video, `heygen video create` (avatar
engine IV by default) beats a generative clip when you want a real presenter.
Browser OAuth uses the web-plan/free avatar-video allowance where eligible;
API keys follow the normal API billing path:
```bash
# discover an avatar + a starfish voice, then create + wait
heygen avatar list --ownership public --limit 5
heygen voice list --engine starfish --limit 5
heygen video create --wait -d '{
"type": "avatar",
"avatar_id": "<avatar-id>",
"script": "Your narration here.",
"voice_id": "<voice-id>"
}'
```
Avatar videos are deterministic + script-driven (lip-sync from a script or a
pre-recorded `audio_url`), distinct from the generative LTX clips. After it
renders, `resolve --from <downloaded.mp4> --type video` to ledger it.
@@ -0,0 +1,50 @@
# media-use usage dashboard (PostHog)
Reproducible definition of the media-use usage dashboard. The dashboard answers
"how much is media-use used, for what, is reuse working, and what can't it
satisfy" from the telemetry `scripts/lib/telemetry.mjs` already emits. Build it
in PostHog project **Hyperframes (356858)**; this doc is the source of truth so
it can be recreated. Local complement: `resolve --stats` (same questions, from
`.media/` + `~/.media`, no PostHog access needed).
## Identity (see `scripts/lib/telemetry.mjs`)
Events attribute to the **same PostHog person as the hyperframes CLI and studio**
— the shared install id in `~/.hyperframes/config.json` (`anonymousId`), stitched
to the HeyGen account (`$identify`, `distinct_id` = email/username) on sign-in.
Not fully anonymous by design; pseudonymous before sign-in, account-linked after.
`$ip:null`. Opt-out: `HYPERFRAMES_NO_TELEMETRY=1` / `DO_NOT_TRACK=1` (also CI, dev).
## Event catalog (verified present in-project)
Every event carries `surface: "media-use"`. Event **properties are coarse**
never intent text, file names, or paths.
| Event | Fires on | Key properties |
| ---------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------- |
| `media_use_resolve` | a resolve that produced/returned an asset | `type`, `source`, `provider`, `via`, `local_only`, `provider_override` |
| `media_use_resolve_miss` | a resolve that found nothing | `type`, `local_only`, `provider_override` (no intent) |
| `media_use_candidates` | `--candidates` / `--dry-run` listing | `type`, counts |
| `media_use_doctor_run` | `--doctor` | `ok`, `checks_failed`, `failed[]` |
| `media_use_compare` | `grade-compare` / `compare` | `command`, `cells`, `truncated`, `total`, `render_ready_timed_out` |
| `media_use_transcribe` · `media_use_duck` · `media_use_transcript_cut` | audio-engine ops | op-specific |
## Dashboard tiles
1. **Invocation volume**`query-trends`, count of `media_use_resolve` over time (daily). "How much."
2. **By media type**`media_use_resolve` broken down by `type` (bgm/sfx/image/icon/logo/voice/grade/lut). "For what."
3. **Resolve hit-rate** — trends formula: `A / (A + B)` where A = `media_use_resolve`, B = `media_use_resolve_miss`. "Is the catalog covering needs."
4. **Provider mix**`media_use_resolve` broken down by `provider`; a second tile by `via` (`url` / `params-fallback` / `params`) to catch CDN→params LUT downgrades.
5. **Top misses**`media_use_resolve_miss` broken down by `type` (the tuning signal — pair with local `resolve --stats`, which also shows the missed _intents_ that telemetry deliberately omits).
6. **Doctor health**`media_use_doctor_run` broken down by `failed[]` (which dependency check fails most) + `checks_failed` distribution.
7. **Compare cost**`media_use_compare` by `command`, plus `truncated` / `render_ready_timed_out` rates (observe before lifting the 16-cell cap).
8. **Adoption (optional)** — if the `first_run` property ships (plan U5), segment `media_use_resolve` first-run vs repeat.
## Recreate via the PostHog MCP
For each tile: `read-data-schema` to confirm the event/property, then a
`query-*` tool (`query-trends` for 17), then `insight-create`, then
`dashboard-create` collecting the insights. Keep names prefixed `media-use:` so
the dashboard is greppable. Cross-surface note: because identity is shared with
CLI/studio, you can also break these down by the same person across `cli_command*`
and `studio:*` events.
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env node
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
import { parseArgs } from "node:util";
import { duckKeyframes, speechSpans } from "./lib/duck.mjs";
import { track } from "./lib/telemetry.mjs";
const { values: args } = parseArgs({
options: {
meta: { type: "string" },
target: { type: "string" },
duck: { type: "string", default: "0.25" },
attack: { type: "string", default: "0.15" },
release: { type: "string", default: "0.4" },
"merge-gap": { type: "string", default: "0.6" },
sequential: { type: "boolean", default: false },
gap: { type: "string", default: "0" },
offsets: { type: "string" },
composition: { type: "string" },
json: { type: "boolean", default: false },
help: { type: "boolean", short: "h", default: false },
},
strict: true,
});
if (args.help) {
console.log(`media-use audio-duck — generate GSAP volume ducking keyframes
Usage:
node audio-duck.mjs --meta audio_meta.json --target "#bgm"
Options:
--meta audio_meta.json or JSON word transcript
--target GSAP selector for the background audio element
--duck Duck multiplier (default: 0.25)
--attack Duck-in duration seconds (default: 0.15)
--release Restore duration seconds (default: 0.4)
--merge-gap Bridge speech gaps smaller than this many seconds (default: 0.6)
--sequential Place multi-line meta back to back at composition time
--gap Extra seconds between sequential lines (default: 0)
--offsets Explicit placement, "l1=0,l2=3.4" (voice id = start seconds)
--composition Read target data-volume from this HTML file
--json Output { spans, keyframes }
--help, -h Show this help`);
process.exit(0);
}
try {
run();
await track("media_use_duck", { sequential: !!args.sequential });
} catch (err) {
if (args.json) console.log(JSON.stringify({ ok: false, error: err.message }));
else console.error(`error: ${err.message}`);
process.exit(1);
}
function run() {
if (!args.meta || !args.target) throw new Error("--meta and --target are required");
const meta = JSON.parse(readFileSync(resolve(args.meta), "utf8"));
const target = args.target;
const baseVolume = readBaseVolume(args.composition, target);
const offsets = args.offsets
? Object.fromEntries(
args.offsets.split(",").map((pair) => {
const [id, t] = pair.split("=");
return [id.trim(), Number(t)];
}),
)
: undefined;
const spans = speechSpans(meta, {
mergeGap: Number(args["merge-gap"]),
sequential: args.sequential,
gap: Number(args.gap),
offsets,
});
const keyframes = duckKeyframes(spans, {
duck: Number(args.duck),
attack: Number(args.attack),
release: Number(args.release),
baseVolume,
});
if (args.json) {
console.log(JSON.stringify({ spans, keyframes }));
return;
}
console.log(
`// auto-duck: ${target} under narration (generated; base volume ${fmt(baseVolume)})`,
);
for (const keyframe of keyframes) {
console.log(
`tl.to(${JSON.stringify(target)}, { volume: ${fmt(keyframe.volume)}, duration: ${fmt(
keyframe.duration,
)} }, ${fmt(keyframe.time)});`,
);
}
}
function readBaseVolume(composition, target) {
if (!composition || !target.startsWith("#")) return 1;
const id = target.slice(1);
const html = readFileSync(resolve(composition), "utf8");
// ponytail: regex is enough here because this only reads one attribute from
// one user-authored composition element, not arbitrary HTML.
const tag = html.match(new RegExp(`<[^>]*\\bid=["']${escapeRegExp(id)}["'][^>]*>`, "i"))?.[0];
const raw = tag?.match(/\bdata-volume=["']([^"']+)["']/i)?.[1];
const volume = Number(raw);
return Number.isFinite(volume) ? volume : 1;
}
function escapeRegExp(value) {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
function fmt(n) {
return Number(n)
.toFixed(3)
.replace(/\.?0+$/, "");
}
+369
View File
@@ -0,0 +1,369 @@
#!/usr/bin/env node
/**
* media-use eval — compare baseline (no media-use) vs. with media-use
* on real registry blocks. Produces an HTML report.
*/
import {
mkdtempSync,
cpSync,
rmSync,
readFileSync,
readdirSync,
existsSync,
writeFileSync,
} from "node:fs";
import { join, basename, resolve, dirname } from "node:path";
import { execFileSync } from "node:child_process";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(SCRIPT_DIR, "..", "..", "..");
const RESOLVE_SCRIPT = join(SCRIPT_DIR, "resolve.mjs");
const TEST_BLOCKS = [
"registry/blocks/nyc-paris-flight",
"registry/blocks/macos-tahoe-liquid-glass",
"registry/blocks/blue-sweater-intro-video",
"registry/blocks/vpn-youtube-spot",
"registry/blocks/apple-money-count",
"registry/blocks/liquid-glass-notification",
"registry/blocks/instagram-follow",
];
// Run resolve.mjs with args as a literal argv array (no shell), so values
// interpolated from manifest metadata (--intent prompt, --type) can't inject
// shell. Mirrors the execFileSync fix in probe.mjs / heygen-search.mjs.
function run(args, opts = {}) {
try {
return {
ok: true,
output: execFileSync(process.execPath, [RESOLVE_SCRIPT, ...args], {
encoding: "utf8",
timeout: 15000,
stdio: "pipe",
...opts,
}).trim(),
};
} catch (err) {
return { ok: false, output: (err.stdout || "") + (err.stderr || ""), code: err.status };
}
}
function countAssetFiles(dir) {
const assetsDir = join(dir, "assets");
if (!existsSync(assetsDir)) return { count: 0, files: [] };
const files = [];
function walk(d, base = "") {
for (const e of readdirSync(d, { withFileTypes: true })) {
const rel = base ? `${base}/${e.name}` : e.name;
if (e.isDirectory()) walk(join(d, e.name), rel);
else files.push(rel);
}
}
walk(assetsDir);
return { count: files.length, files };
}
function evalBlock(blockPath) {
const fullPath = join(REPO_ROOT, blockPath);
if (!existsSync(fullPath)) return null;
const name = basename(blockPath);
const tmp = mkdtempSync(join(tmpdir(), `mu-eval-${name}-`));
try {
cpSync(fullPath, tmp, { recursive: true });
// baseline: what the agent sees WITHOUT media-use
const baseline = countAssetFiles(tmp);
const htmlFiles = readdirSync(tmp).filter((f) => f.endsWith(".html"));
// parse compositions for asset references
const assetRefs = [];
for (const hf of htmlFiles) {
const html = readFileSync(join(tmp, hf), "utf8");
const srcMatches = html.matchAll(/src=["']([^"']+?)["']/g);
for (const m of srcMatches) {
const ref = m[1];
if (ref.startsWith("data:") || ref.startsWith("http")) continue;
assetRefs.push({ composition: hf, ref });
}
const urlMatches = html.matchAll(/url\(["']?([^"')]+?)["']?\)/g);
for (const m of urlMatches) {
const ref = m[1];
if (ref.startsWith("data:") || ref.startsWith("http") || ref.startsWith("#")) continue;
assetRefs.push({ composition: hf, ref });
}
}
// with media-use: run --adopt
const adoptResult = run(["--adopt", "--project", tmp, "--json"]);
let adopted = { ok: false, adopted: 0, assets: [] };
if (adoptResult.ok) {
try {
adopted = JSON.parse(adoptResult.output);
} catch {
/* */
}
}
// read the generated index
const indexPath = join(tmp, ".media", "index.md");
const indexContent = existsSync(indexPath)
? readFileSync(indexPath, "utf8")
: "(no index generated)";
// read manifest for detail
const manifestPath = join(tmp, ".media", "manifest.jsonl");
const manifest = existsSync(manifestPath)
? readFileSync(manifestPath, "utf8")
.trim()
.split("\n")
.map((l) => {
try {
return JSON.parse(l);
} catch {
return null;
}
})
.filter(Boolean)
: [];
// test resolve cache hit: try resolving something that was adopted
let resolveTest = null;
if (manifest.length > 0) {
const first = manifest[0];
const prompt = first.provenance?.prompt || first.description;
const r = run(["--type", first.type, "--intent", prompt, "--project", tmp, "--json"]);
if (r.ok) {
try {
resolveTest = JSON.parse(r.output);
} catch {
/* */
}
}
}
// test resolve miss: try resolving something that doesn't exist
const missResult = run([
"--type",
"bgm",
"--intent",
"nonexistent query xyz",
"--project",
tmp,
"--json",
]);
let resolveMiss = null;
if (!missResult.ok) {
try {
resolveMiss = JSON.parse(missResult.output);
} catch {
/* */
}
}
// coverage: which composition refs are covered by the manifest
const manifestPaths = new Set(manifest.map((m) => m.path));
const coverage = assetRefs.map((r) => ({
...r,
covered: manifestPaths.has(r.ref),
}));
return {
name,
baseline: { fileCount: baseline.count, files: baseline.files, htmlCount: htmlFiles.length },
compositions: htmlFiles,
assetRefs: coverage,
adopted: { count: adopted.adopted, assets: adopted.assets || [] },
index: indexContent,
manifest,
resolveTest,
resolveMiss,
};
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}
function generateReport(results) {
const all = results.filter(Boolean);
const passed = all.filter((r) => r.adopted.count > 0);
const rows = results
.filter(Boolean)
.map((r) => {
const hasMetadata = r.manifest.some((m) => m.duration || m.width);
const cacheHit = r.resolveTest?._source === "cached";
const missHandled = r.resolveMiss?.ok === false;
return `<tr>
<td><strong>${r.name}</strong></td>
<td>${r.baseline.fileCount} files, ${r.baseline.htmlCount} comp${r.baseline.htmlCount === 1 ? "" : "s"}</td>
<td>${r.adopted.count} adopted</td>
<td>${hasMetadata ? "<span class='pass'>with metadata</span>" : "<span class='warn'>no metadata</span>"}</td>
<td>${cacheHit ? "<span class='pass'>cache hit</span>" : "<span class='warn'>no hit</span>"}</td>
<td>${missHandled ? "<span class='pass'>handled</span>" : "<span class='fail'>unexpected</span>"}</td>
</tr>`;
})
.join("\n");
const details = results
.filter(Boolean)
.filter((r) => r.adopted.count > 0)
.map((r) => {
const assetRows = r.manifest
.map((m) => {
const dur = m.duration != null ? `${m.duration}s` : "—";
const dims = m.width && m.height ? `${m.width}×${m.height}` : "—";
return `<tr><td>${m.id}</td><td>${m.type}</td><td>${dur}</td><td>${dims}</td><td class="path">${m.path}</td><td>${m.description || ""}</td></tr>`;
})
.join("\n");
const coveredCount = r.assetRefs.filter((c) => c.covered).length;
const totalRefs = r.assetRefs.length;
const coveragePct = totalRefs > 0 ? Math.round((coveredCount / totalRefs) * 100) : 100;
const refRows = r.assetRefs
.map(
(c) =>
`<tr><td class="path">${c.composition}</td><td class="path">${c.ref}</td><td>${c.covered ? "<span class='pass'>covered</span>" : "<span class='warn'>not in manifest</span>"}</td></tr>`,
)
.join("\n");
return `<div class="block-detail">
<h3>${r.name}</h3>
<p style="font-size:13px;color:var(--muted)">${r.compositions.length} composition${r.compositions.length === 1 ? "" : "s"}: ${r.compositions.join(", ")}</p>
<div class="comparison">
<div class="col">
<h4>Baseline (no media-use)</h4>
<p>Agent sees: ${r.baseline.fileCount} raw files in assets/<br>No metadata, no type info, no relationship to compositions.</p>
<pre class="file-list">${r.baseline.files.join("\n") || "(no assets)"}</pre>
</div>
<div class="col">
<h4>With media-use (after --adopt)</h4>
<p>Agent reads index.md — structured, typed, with metadata:</p>
<pre class="index">${escapeHtml(r.index)}</pre>
</div>
</div>
${
totalRefs > 0
? `<h4>Composition → asset coverage <span class="${coveragePct === 100 ? "pass" : "warn"}">${coveragePct}%</span> (${coveredCount}/${totalRefs} refs)</h4>
<table class="manifest">
<thead><tr><th>composition</th><th>asset reference</th><th>in manifest?</th></tr></thead>
<tbody>${refRows}</tbody>
</table>`
: ""
}
<h4>Manifest records</h4>
<table class="manifest">
<thead><tr><th>id</th><th>type</th><th>dur</th><th>dims</th><th>path</th><th>description</th></tr></thead>
<tbody>${assetRows}</tbody>
</table>
</div>`;
})
.join("\n");
return `<title>media-use eval report</title>
<style>
:root { --bg: #fafaf7; --text: #1b1b18; --muted: #7a756a; --accent: #0d7377; --good: #1a7a3a; --warn: #b45309; --fail: #dc2626; --border: #e8e5df; --surface: #fff; --mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; --sans: system-ui, -apple-system, sans-serif; --serif: Georgia, serif }
* { box-sizing: border-box; margin: 0 } body { background: var(--bg); color: var(--text); font-family: var(--serif); line-height: 1.6; font-size: 15px; padding: 40px 24px }
.wrap { max-width: 1100px; margin: 0 auto }
h1 { font-family: var(--sans); font-size: 28px; font-weight: 700; margin-bottom: 8px; letter-spacing: -.02em }
h2 { font-family: var(--sans); font-size: 20px; font-weight: 650; margin: 32px 0 12px; letter-spacing: -.01em }
h3 { font-family: var(--sans); font-size: 17px; font-weight: 650; margin: 24px 0 8px }
h4 { font-family: var(--sans); font-size: 14px; font-weight: 600; margin: 16px 0 6px; color: var(--muted) }
p { margin-bottom: 10px }
.meta { font-family: var(--mono); font-size: 12px; color: var(--muted); margin-bottom: 24px }
.summary { display: flex; gap: 16px; margin: 16px 0; flex-wrap: wrap }
.stat { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 14px 18px; flex: 1; min-width: 140px }
.stat .num { font-family: var(--sans); font-size: 28px; font-weight: 700; color: var(--accent) }
.stat .label { font-family: var(--mono); font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: .1em }
table { width: 100%; border-collapse: collapse; font-size: 13px; font-family: var(--sans); margin: 8px 0 }
th { text-align: left; font-family: var(--mono); font-size: 10px; letter-spacing: .08em; text-transform: uppercase; color: var(--muted); border-bottom: 2px solid var(--border); padding: 6px 8px; font-weight: 700 }
td { border-bottom: 1px solid var(--border); padding: 7px 8px; vertical-align: top }
td.path { font-family: var(--mono); font-size: 12px; color: var(--muted); max-width: 300px; overflow: hidden; text-overflow: ellipsis }
.pass { color: var(--good); font-weight: 600 } .warn { color: var(--warn); font-weight: 600 } .fail { color: var(--fail); font-weight: 600 }
.comparison { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin: 12px 0 }
@media(max-width:700px) { .comparison { grid-template-columns: 1fr } }
.col { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 14px 16px }
.col h4 { margin-top: 0 }
pre { font-family: var(--mono); font-size: 12px; background: #1b1b18; color: #d4d0c8; border-radius: 6px; padding: 12px 14px; overflow-x: auto; margin: 6px 0; line-height: 1.5 }
pre.file-list { background: var(--bg); color: var(--muted); border: 1px solid var(--border) }
pre.index { white-space: pre; }
.block-detail { border-top: 1px solid var(--border); padding-top: 20px; margin-top: 20px }
.verdict { margin-top: 24px; padding: 16px 20px; border-radius: 8px; font-family: var(--sans); font-size: 15px }
.verdict.ship { background: #edfbf0; border: 1px solid #1a7a3a; color: #1a7a3a }
.verdict.wait { background: #fff3ec; border: 1px solid #d94f04; color: #d94f04 }
</style>
<div class="wrap">
<h1>media-use eval report</h1>
<p class="meta">${new Date().toISOString().slice(0, 10)} · ${all.length} blocks evaluated · baseline vs. media-use --adopt</p>
<div class="summary">
<div class="stat"><div class="num">${all.length}</div><div class="label">blocks tested</div></div>
<div class="stat"><div class="num">${passed.length}</div><div class="label">with assets</div></div>
<div class="stat"><div class="num">${all.reduce((s, r) => s + r.adopted.count, 0)}</div><div class="label">assets adopted</div></div>
<div class="stat"><div class="num">${all.filter((r) => r.manifest.some((m) => m.duration || m.width)).length}</div><div class="label">with ffprobe metadata</div></div>
<div class="stat"><div class="num">${(() => {
const refs = all.flatMap((r) => r.assetRefs);
const covered = refs.filter((c) => c.covered).length;
return refs.length > 0 ? Math.round((covered / refs.length) * 100) + "%" : "—";
})()}</div><div class="label">composition coverage</div></div>
</div>
<h2>Results matrix</h2>
<table>
<thead><tr><th>Block</th><th>Baseline</th><th>Adopted</th><th>Metadata</th><th>Cache hit</th><th>Miss handling</th></tr></thead>
<tbody>${rows}</tbody>
</table>
<h2>Before / after comparisons</h2>
${details}
<div class="verdict ${passed.length >= 3 ? "ship" : "wait"}">
${
passed.length >= 3
? `<strong>Ship it.</strong> ${passed.length}/${all.length} blocks adopted successfully with metadata. Resolve cache hits work. Miss handling is clean.`
: `<strong>Needs work.</strong> Only ${passed.length} blocks adopted. Check the failures above.`
}
</div>
</div>`;
}
function escapeHtml(str) {
return str.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
}
console.log("media-use eval · running against registry blocks...\n");
const results = [];
for (const block of TEST_BLOCKS) {
const fullPath = join(REPO_ROOT, block);
if (!existsSync(fullPath)) {
console.log(` skip ${basename(block)} (not found)`);
results.push(null);
continue;
}
process.stdout.write(` ${basename(block)}...`);
const result = evalBlock(block);
if (result) {
console.log(
` ${result.adopted.count} adopted, ${result.manifest.filter((m) => m.duration || m.width).length} with metadata`,
);
} else {
console.log(" failed");
}
results.push(result);
}
const report = generateReport(results);
const outPath = join(SCRIPT_DIR, "..", "eval-report.html");
writeFileSync(outPath, report);
console.log(`\nReport: ${outPath}`);
+128
View File
@@ -0,0 +1,128 @@
import { readdirSync, statSync, existsSync } from "node:fs";
import { join, extname, basename } from "node:path";
import { readManifest, appendRecord, nextId } from "./manifest.mjs";
import { regenerateIndex } from "./index-gen.mjs";
import { probe } from "./probe.mjs";
import { matchTokens } from "./match.mjs";
const AUDIO_EXT = new Set([".mp3", ".wav", ".ogg", ".m4a", ".aac"]);
const IMAGE_EXT = new Set([".jpg", ".jpeg", ".png", ".gif", ".webp", ".svg", ".ico"]);
const VIDEO_EXT = new Set([".mp4", ".webm", ".mov"]);
function inferType(filePath) {
const ext = extname(filePath).toLowerCase();
if (AUDIO_EXT.has(ext)) {
const lower = filePath.toLowerCase();
if (lower.includes("/bgm/") || lower.includes("/music/") || lower.startsWith("bgm/"))
return "bgm";
if (lower.includes("/sfx/") || lower.includes("/sound") || lower.startsWith("sfx/"))
return "sfx";
if (lower.includes("/voice/") || lower.includes("/narrat") || lower.startsWith("voice/"))
return "voice";
return "bgm";
}
if (IMAGE_EXT.has(ext)) {
if (ext === ".svg" || ext === ".ico") return "icon";
return "image";
}
if (VIDEO_EXT.has(ext)) return "video";
return null;
}
function walkDir(dir, base = "") {
const files = [];
if (!existsSync(dir)) return files;
for (const entry of readdirSync(dir, { withFileTypes: true })) {
const rel = base ? `${base}/${entry.name}` : entry.name;
if (entry.isDirectory()) {
files.push(...walkDir(join(dir, entry.name), rel));
} else {
files.push(rel);
}
}
return files;
}
export function scanExistingAssets(projectDir) {
const assetsDir = join(projectDir, "assets");
if (!existsSync(assetsDir)) return [];
const files = walkDir(assetsDir);
const found = [];
for (const rel of files) {
const type = inferType(rel);
if (!type) continue;
const fullPath = join(assetsDir, rel);
const stat = statSync(fullPath);
if (stat.size === 0) {
// A 0-byte asset would register clean but fail at render — skip it loudly
// rather than adopt a broken file.
console.error(`media-use: skipping 0-byte asset assets/${rel}`);
continue;
}
const meta = probe(fullPath);
found.push({
relativePath: `assets/${rel}`,
type,
size: stat.size,
name: basename(rel, extname(rel)),
...meta,
});
}
return found;
}
export function adoptExistingAssets(projectDir) {
const existing = scanExistingAssets(projectDir);
if (existing.length === 0) return [];
const manifest = readManifest(projectDir);
const knownPaths = new Set(manifest.map((r) => r.path));
const adopted = [];
for (const asset of existing) {
if (knownPaths.has(asset.relativePath)) continue;
const id = nextId(projectDir, asset.type);
const record = {
id,
type: asset.type,
path: asset.relativePath,
source: "existing",
description: asset.name.replace(/[-_]/g, " "),
...(asset.duration != null && { duration: asset.duration }),
...(asset.width != null && { width: asset.width }),
...(asset.height != null && { height: asset.height }),
provenance: { provider: "local", adopted: true },
};
appendRecord(projectDir, record);
adopted.push(record);
}
if (adopted.length > 0) regenerateIndex(projectDir);
return adopted;
}
// Adopt a pre-existing assets/ file only when it shares a meaningful word with
// the intent. The old test — `name.includes(intent) || intent.includes(name)` —
// silently returned the WRONG file: "whoosh" grabbed a stray who.mp3, and a
// one-letter filename matched every intent. A false negative just falls through
// to a catalog search (safe); a false positive ships the wrong asset. So bias to
// precision: require a shared token, don't guess from substrings.
export function findExistingAsset(projectDir, intent, type) {
const assetsDir = join(projectDir, "assets");
if (!existsSync(assetsDir)) return null;
const intentTokens = matchTokens(intent);
if (intentTokens.size === 0) return null;
for (const rel of walkDir(assetsDir)) {
const t = inferType(rel);
if (!t || (type && t !== type)) continue;
const stem = basename(rel, extname(rel));
for (const tok of matchTokens(stem)) {
if (intentTokens.has(tok)) {
return { relativePath: `assets/${rel}`, type: t, name: stem };
}
}
}
return null;
}
@@ -0,0 +1,95 @@
import { strict as assert } from "node:assert";
import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { findExistingAsset, adoptExistingAssets } from "./adopt.mjs";
let tmp;
function setup() {
tmp = mkdtempSync(join(tmpdir(), "mu-adopt-test-"));
}
function cleanup() {
if (tmp) rmSync(tmp, { recursive: true, force: true });
}
function drop(rel) {
const full = join(tmp, "assets", rel);
mkdirSync(join(full, ".."), { recursive: true });
writeFileSync(full, "x");
}
function runTests() {
const tests = [];
const test = (name, fn) => tests.push({ name, fn });
test("does NOT false-match a short filename stem (who.mp3 vs 'whoosh')", () => {
setup();
drop("sfx/who.mp3");
assert.equal(findExistingAsset(tmp, "whoosh", "sfx"), null);
cleanup();
});
test("does NOT match a one-letter filename against any intent", () => {
setup();
drop("images/a.jpg");
assert.equal(findExistingAsset(tmp, "gradient tech background", "image"), null);
cleanup();
});
test("matches on a shared meaningful word", () => {
setup();
drop("images/hero-shot.jpg");
const hit = findExistingAsset(tmp, "hero image", "image");
assert.ok(hit);
assert.equal(hit.relativePath, "assets/images/hero-shot.jpg");
cleanup();
});
test("matches multi-word overlap", () => {
setup();
drop("images/gradient-tech-bg.jpg");
assert.ok(findExistingAsset(tmp, "gradient tech background", "image"));
cleanup();
});
test("a shared stopword alone does not match", () => {
setup();
drop("video/the-video.mp4");
assert.equal(findExistingAsset(tmp, "the rocket", null), null);
cleanup();
});
test("respects the type filter", () => {
setup();
drop("sfx/rocket.mp3");
assert.equal(findExistingAsset(tmp, "rocket", "image"), null, "wrong type is skipped");
assert.ok(findExistingAsset(tmp, "rocket", "sfx"), "right type matches");
cleanup();
});
test("adoptExistingAssets still imports every typed file (unaffected by match rule)", () => {
setup();
drop("sfx/who.mp3");
drop("images/a.jpg");
assert.equal(adoptExistingAssets(tmp).length, 2);
cleanup();
});
let passed = 0;
let failed = 0;
for (const { name, fn } of tests) {
try {
fn();
passed++;
console.log(` \x1b[32m✓\x1b[0m ${name}`);
} catch (err) {
failed++;
console.log(` \x1b[31m✗\x1b[0m ${name}`);
console.log(` ${err.message}`);
}
}
console.log(`\n${passed} passed, ${failed} failed`);
if (failed > 0) process.exit(1);
}
console.log("media-use · adopt / findExistingAsset tests\n");
runTests();
@@ -0,0 +1,20 @@
import { heygenSearch } from "./heygen-search.mjs";
export const bgmProvider = {
async search(intent) {
const results = heygenSearch("audio sounds list", intent, { type: "music" });
if (!results) return null;
const best = results[0];
return {
url: best.audio_url,
source: "search",
// ext derived from audio_url by resolve.mjs — catalog tracks are .mp3 or .wav
metadata: {
description: best.description || intent,
duration: best.duration || null,
provider: "heygen.audio.sounds",
provenance: { track_id: best.id, score: best.score, query: intent },
},
};
},
};
@@ -0,0 +1,59 @@
import { readFileSync, existsSync } from "node:fs";
import { join } from "node:path";
function findDesignSpec(projectDir) {
for (const name of ["frame.md", "design.md", "DESIGN.md"]) {
const p = join(projectDir, name);
if (existsSync(p)) return { path: p, name };
}
return null;
}
function parseFrontmatter(content) {
const match = content.match(/^---\n([\s\S]*?)\n---/);
if (!match) return null;
const yaml = match[1];
const tokens = {};
for (const line of yaml.split("\n")) {
const m = line.match(/^\s*(\w[\w-]*):\s*(.+)/);
if (m) tokens[m[1]] = m[2].trim().replace(/^["']|["']$/g, "");
}
return tokens;
}
function extractColors(tokens) {
const colors = [];
for (const [k, v] of Object.entries(tokens)) {
if (typeof v === "string" && /^#[0-9a-fA-F]{3,8}$/.test(v)) {
colors.push({ name: k, hex: v });
}
}
return colors;
}
export const brandProvider = {
async search(intent, { projectDir } = {}) {
if (!projectDir) return null;
const spec = findDesignSpec(projectDir);
if (!spec) return null;
const content = readFileSync(spec.path, "utf8");
const tokens = parseFrontmatter(content);
if (!tokens) return null;
const colors = extractColors(tokens);
return {
localPath: spec.path,
source: "local",
ext: ".md",
metadata: {
description: "Brand tokens from " + spec.name,
provider: "design_spec",
provenance: {
file: spec.name,
colors,
font: tokens.font || tokens.typography || null,
logo: tokens.logo || null,
},
},
};
},
};
@@ -0,0 +1,58 @@
import { existsSync, readFileSync } from "node:fs";
import { extname, join } from "node:path";
const LIB_DIR = join(import.meta.dirname, "..", "..", "audio", "assets", "sfx");
const normalize = (value) =>
String(value)
.toLowerCase()
.replace(/[^a-z0-9]+/g, " ")
.trim();
export function extensionForBundledSfxFile(filename) {
return extname(filename) || ".mp3";
}
function score(intent, key, entry) {
const query = normalize(intent);
const name = normalize(key);
if (query === name) return 100;
if (query.includes(name) || name.includes(query)) return 50;
const haystack = new Set(normalize(`${key} ${entry.description || ""}`).split(/\s+/));
return query.split(/\s+/).filter((token) => token && haystack.has(token)).length;
}
export const bundledSfxProvider = {
async search(intent) {
const manifestPath = join(LIB_DIR, "manifest.json");
if (!existsSync(manifestPath)) return null;
let manifest;
try {
manifest = JSON.parse(readFileSync(manifestPath, "utf8"));
} catch {
return null;
}
const ranked = Object.entries(manifest)
.map(([key, entry]) => ({ key, entry, score: score(intent, key, entry) }))
.filter(({ entry, score }) => entry?.file && score > 0)
.sort((a, b) => b.score - a.score || a.key.localeCompare(b.key));
const best = ranked[0];
if (!best) return null;
const localPath = join(LIB_DIR, best.entry.file);
if (!existsSync(localPath)) return null;
return {
localPath,
ext: extensionForBundledSfxFile(best.entry.file),
source: "bundled",
metadata: {
description: best.entry.description || best.key,
duration: best.entry.duration ?? null,
provider: "bundled.sfx",
provenance: { library_key: best.key },
},
};
},
};
@@ -0,0 +1,9 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { extensionForBundledSfxFile } from "./bundled-sfx-provider.mjs";
test("derives bundled SFX extension from the manifest filename", () => {
assert.equal(extensionForBundledSfxFile("impact.wav"), ".wav");
assert.equal(extensionForBundledSfxFile("whoosh.ogg"), ".ogg");
assert.equal(extensionForBundledSfxFile("extensionless"), ".mp3");
});
+150
View File
@@ -0,0 +1,150 @@
import { readFileSync, writeFileSync, mkdirSync, existsSync, copyFileSync } from "node:fs";
import { join, basename } from "node:path";
import { createHash } from "node:crypto";
import { homedir } from "node:os";
import { readManifest, appendRecord, normalizePrompt } from "./manifest.mjs";
const SCHEMA_PREFIX = "mu-v1-";
const KEY_HEX_CHARS = 16;
const COMPLETE_SENTINEL = ".hf-complete";
export function globalMediaDir() {
return join(homedir(), ".media");
}
export function contentHash(filePath) {
const bytes = readFileSync(filePath);
return createHash("sha256").update(bytes).digest("hex");
}
function cacheEntryDir(rootDir, sha) {
return join(rootDir, SCHEMA_PREFIX + sha.slice(0, KEY_HEX_CHARS));
}
function isComplete(entryDir) {
return existsSync(join(entryDir, COMPLETE_SENTINEL));
}
function markComplete(entryDir) {
writeFileSync(join(entryDir, COMPLETE_SENTINEL), "", "utf8");
}
// The manifest helpers append their own ".media" to the dir they get, so the
// global manifest must be addressed by HOME, not by globalMediaDir() — passing
// the latter nested it at ~/.media/.media/manifest.jsonl, invisible to the
// Studio /api/assets/global route (which reads the documented flat path).
export function readGlobalManifest() {
return readManifest(homedir());
}
// Resolve a content-sha (full or unambiguous prefix) to a reusable global-cache
// record, for `resolve --reuse <sha>`. Returns null on no match, or
// { ambiguous: true, count } when a prefix matches multiple distinct entries.
// Completeness (the .hf-complete sentinel) is left to importFromCache so the
// caller can surface an "incomplete cache entry" error distinctly from a miss.
export function findGlobalBySha(shaPrefix) {
const p = String(shaPrefix || "")
.toLowerCase()
.trim();
if (!p) return null;
const matches = readGlobalManifest().filter(
(r) => r.reusable && typeof r.sha === "string" && r.sha.startsWith(p),
);
if (matches.length === 0) return null;
if (matches.length > 1) {
const exact = matches.find((r) => r.sha === p);
if (exact) return exact;
return { ambiguous: true, count: matches.length };
}
return matches[0];
}
function validateCacheHit(match) {
if (!match?.sha) return null;
return isComplete(cacheEntryDir(globalMediaDir(), match.sha)) ? match : null;
}
export function cacheGet(prompt, type) {
const key = normalizePrompt(prompt);
if (!key) return null;
return validateCacheHit(
readGlobalManifest().find(
(r) =>
r.reusable &&
normalizePrompt(r.provenance?.prompt) === key &&
(type == null || r.type === type),
),
);
}
export function cacheGetByEntity(entity) {
const lower = entity.toLowerCase();
return validateCacheHit(
readGlobalManifest().find((r) => r.reusable && r.entity && r.entity.toLowerCase() === lower),
);
}
export function cachePut(filePath, record) {
const sha = contentHash(filePath);
// Idempotent: same content already promoted -> don't duplicate the global
// record. ponytail: skips usage_count bump; add it when the metric is needed.
const existing = readGlobalManifest().find((r) => r.sha === sha);
if (existing) return { sha, cached_path: existing.cached_path, deduped: true };
const dir = globalMediaDir();
const entryDir = cacheEntryDir(dir, sha);
mkdirSync(entryDir, { recursive: true });
const dest = join(entryDir, basename(filePath));
copyFileSync(filePath, dest);
markComplete(entryDir);
const globalRecord = {
...record,
sha,
reusable: true,
cached_path: dest,
};
appendRecord(homedir(), globalRecord);
return { sha, cached_path: dest };
}
export function importFromCache(cacheRecord, projectDir, localId, localPath) {
const sha = cacheRecord.sha;
const entryDir = cacheEntryDir(globalMediaDir(), sha);
if (!isComplete(entryDir)) return null;
const cachedFile = cacheRecord.cached_path;
if (!cachedFile || !existsSync(cachedFile)) return null;
mkdirSync(join(projectDir, ".media"), { recursive: true });
const fullDest = join(projectDir, localPath);
mkdirSync(join(fullDest, ".."), { recursive: true });
copyFileSync(cachedFile, fullDest);
const projectRecord = {
...cacheRecord,
id: localId,
path: localPath,
provenance: {
...cacheRecord.provenance,
imported_from: sha,
},
};
delete projectRecord.sha;
delete projectRecord.reusable;
delete projectRecord.cached_path;
return projectRecord;
}
export function promote(projectDir, id) {
const records = readManifest(projectDir);
const record = records.find((r) => r.id === id);
if (!record) throw new Error(`asset not found in project manifest: ${id}`);
const filePath = join(projectDir, record.path);
if (!existsSync(filePath)) throw new Error(`asset file not found: ${filePath}`);
return cachePut(filePath, record);
}
@@ -0,0 +1,91 @@
// Reuse candidates: a side-effect-free view of assets already available to this
// project (its own manifest) and across every project (the global ~/.media
// cache), so the calling agent can judge semantic fit itself. No download, no
// provider, no mutation. The ranker only *surfaces* — it orders by lexical
// overlap but never filters a candidate out on zero overlap (that would
// pre-empt the agent's judgment); the agent does the semantic call.
import { readManifest } from "./manifest.mjs";
import { readGlobalManifest } from "./cache.mjs";
import { tokenOverlap, typesMatch } from "./match.mjs";
export const CANDIDATE_CAP = 8;
function shape(record, scope, intent) {
const description = record.description || record.provenance?.prompt || "";
const prompt = record.provenance?.prompt || null;
return {
id: record.id,
type: record.type,
scope,
description,
prompt,
provider: record.provenance?.provider || null,
duration: record.duration ?? null,
width: record.width ?? null,
height: record.height ?? null,
// Only global records carry a content sha — it is the stable reuse handle
// for `resolve --reuse <sha>`. Project assets are reused by referencing
// their path directly, so they need no handle.
sha: scope === "global" ? record.sha || null : null,
path: scope === "project" ? record.path : null,
score: intent ? tokenOverlap(intent, `${description} ${prompt || ""}`) : 0,
};
}
// Rank one scope: type-matched (icon<->image aware), newest-first within equal
// overlap, ordered by overlap desc. Returns the full ranked list (uncapped).
function rankScope(records, scope, type, intent) {
return records
.filter((r) => typesMatch(r.type, type))
.reverse() // manifest is append-order (oldest first); newest-first at equal score
.map((r) => shape(r, scope, intent))
.sort((a, b) => b.score - a.score); // Array.sort is stable → recency preserved
}
// List reuse candidates for `type`, capped per scope. Returns:
// candidates: capped project candidates followed by capped global candidates
// truncated: true if either scope had more than `cap`
// total: { project, global } counts before the cap (machine-readable)
// similar: count of candidates with lexical overlap > 0 (drives the nudge)
export function listCandidates({ projectDir, type, intent = "", cap = CANDIDATE_CAP }) {
const project = rankScope(readManifest(projectDir), "project", type, intent);
const global = rankScope(readGlobalManifest(), "global", type, intent);
const candidates = [...project.slice(0, cap), ...global.slice(0, cap)];
return {
candidates,
truncated: project.length > cap || global.length > cap,
total: { project: project.length, global: global.length },
similar: [...project, ...global].filter((c) => c.score > 0).length,
};
}
function meta(c) {
const parts = [];
if (c.duration != null) parts.push(`${c.duration}s`);
if (c.width && c.height) parts.push(`${c.width}x${c.height}`);
if (c.provider) parts.push(c.provider);
return parts.join(", ");
}
// Human-readable listing. The agent can read this directly; --json is for
// programmatic use. Reuse handle differs by scope: path for project, sha for
// global.
export function formatCandidates(candidates, { truncated, total } = {}) {
if (candidates.length === 0) return "no reuse candidates found (project or global cache)";
const lines = [`${candidates.length} reuse candidate${candidates.length === 1 ? "" : "s"}:`, ""];
for (const c of candidates) {
const handle =
c.scope === "global" ? `--reuse ${String(c.sha).slice(0, 16)}` : c.path || `manifest:${c.id}`;
const m = meta(c);
lines.push(` [${c.scope}] ${c.description}${m ? ` (${m})` : ""}`);
lines.push(` ${handle}`);
}
if (truncated && total) {
lines.push("");
lines.push(
` (showing top ${CANDIDATE_CAP} per scope; ${total.project} project / ${total.global} global total — refine --intent to narrow)`,
);
}
return lines.join("\n");
}
@@ -0,0 +1,155 @@
import { test } from "node:test";
import { strict as assert } from "node:assert";
import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { listCandidates, formatCandidates, CANDIDATE_CAP } from "./candidates.mjs";
import { findGlobalBySha } from "./cache.mjs";
// candidates + findGlobalBySha are offline (no heygen), so we can override HOME
// to a temp dir and seed a fake global ~/.media manifest deterministically.
function sandbox() {
const root = mkdtempSync(join(tmpdir(), "mu-cand-"));
const project = join(root, "proj");
const home = join(root, "home");
process.env.HOME = home;
return { root, project, home };
}
function seedManifest(dir, records) {
const md = join(dir, ".media");
mkdirSync(md, { recursive: true });
writeFileSync(md + "/manifest.jsonl", records.map((r) => JSON.stringify(r)).join("\n") + "\n");
}
function proj(id, type, description, prompt) {
return { id, type, path: `.media/audio/bgm/${id}.wav`, description, provenance: { prompt } };
}
function glob(id, type, description, prompt, sha) {
return {
id,
type,
sha,
reusable: true,
cached_path: `/x/${sha}/${id}.wav`,
description,
provenance: { prompt, provider: "heygen.audio.sounds" },
};
}
test("ranks project + global by overlap, tags scope", () => {
const { root, project, home } = sandbox();
try {
seedManifest(project, [proj("bgm_001", "bgm", "calm ambient piano", "calm ambient piano")]);
seedManifest(home, [
glob("bgm_009", "bgm", "energetic tech launch", "energetic tech launch", "a".repeat(64)),
glob("bgm_010", "bgm", "sad corporate piano", "sad corporate piano", "b".repeat(64)),
]);
const { candidates } = listCandidates({
projectDir: project,
type: "bgm",
intent: "tech launch",
});
assert.equal(candidates[0].scope, "project"); // project listed first
const g = candidates.filter((c) => c.scope === "global");
assert.equal(g[0].description, "energetic tech launch"); // higher overlap ranks first
assert.ok(g[0].score >= g[1].score);
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("zero-overlap intent still lists candidates (no hard filter)", () => {
const { root, project, home } = sandbox();
try {
seedManifest(project, []);
seedManifest(home, [glob("bgm_009", "bgm", "driving synth", "driving synth", "c".repeat(64))]);
const { candidates, similar } = listCandidates({
projectDir: project,
type: "bgm",
intent: "totally unrelated words xyz",
});
assert.equal(candidates.length, 1, "listed despite zero overlap");
assert.equal(candidates[0].score, 0);
assert.equal(similar, 0, "similar counts only overlap>0");
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("caps per scope and reports truncation + totals", () => {
const { root, project, home } = sandbox();
try {
const many = Array.from({ length: CANDIDATE_CAP + 3 }, (_, i) =>
glob(`bgm_${i}`, "bgm", `track ${i}`, `track ${i}`, String(i).padStart(64, "0")),
);
seedManifest(project, []);
seedManifest(home, many);
const { candidates, truncated, total } = listCandidates({ projectDir: project, type: "bgm" });
assert.equal(candidates.length, CANDIDATE_CAP);
assert.equal(truncated, true);
assert.equal(total.global, CANDIDATE_CAP + 3);
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("honors icon<->image adjacency", () => {
const { root, project, home } = sandbox();
try {
seedManifest(project, []);
seedManifest(home, [glob("image_1", "image", "rocket logo", "rocket logo", "d".repeat(64))]);
const { candidates } = listCandidates({ projectDir: project, type: "icon", intent: "rocket" });
assert.equal(candidates.length, 1, "image asset surfaces for icon request");
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("sha only on global, path only on project", () => {
const { root, project, home } = sandbox();
try {
seedManifest(project, [proj("bgm_001", "bgm", "x", "x")]);
seedManifest(home, [glob("bgm_009", "bgm", "y", "y", "e".repeat(64))]);
const { candidates } = listCandidates({ projectDir: project, type: "bgm" });
const p = candidates.find((c) => c.scope === "project");
const g = candidates.find((c) => c.scope === "global");
assert.ok(p.path && !p.sha);
assert.ok(g.sha && !g.path);
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("findGlobalBySha resolves unique prefix, flags ambiguity, misses cleanly", () => {
const { root, project, home } = sandbox();
try {
seedManifest(project, []);
seedManifest(home, [
glob("bgm_1", "bgm", "a", "a", "abc" + "0".repeat(61)),
glob("bgm_2", "bgm", "b", "b", "abd" + "0".repeat(61)),
glob("bgm_3", "bgm", "c", "c", "fff" + "0".repeat(61)),
]);
assert.equal(findGlobalBySha("fff").id, "bgm_3", "unique prefix resolves");
assert.deepEqual(
{ ambiguous: findGlobalBySha("ab").ambiguous, count: findGlobalBySha("ab").count },
{ ambiguous: true, count: 2 },
"ambiguous prefix flagged",
);
assert.equal(findGlobalBySha("zzz"), null, "miss returns null");
assert.equal(findGlobalBySha(""), null, "empty returns null");
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("formatCandidates shows reuse handles by scope; empty message", () => {
const { candidates } = {
candidates: [
{ scope: "project", description: "p", path: ".media/audio/bgm/bgm_001.wav" },
{ scope: "global", description: "g", sha: "f".repeat(64) },
],
};
const out = formatCandidates(candidates, {});
assert.match(out, /\.media\/audio\/bgm\/bgm_001\.wav/);
assert.match(out, /--reuse ffffffffffffffff/);
assert.match(formatCandidates([], {}), /no reuse candidates/);
});
@@ -0,0 +1,143 @@
import { execFileSync } from "node:child_process";
import { copyFileSync, existsSync, readdirSync, statSync, unlinkSync } from "node:fs";
import { homedir, tmpdir } from "node:os";
import { join } from "node:path";
// Image generation via the OpenAI Codex CLI's built-in image tool (gpt-image-2)
// on the user's ChatGPT subscription: the codex CLI owns auth, media-use holds
// no key (CLI-only). The image UPSELL behind local mflux; skipped by --local-only.
//
// Retrieval mirrors illo-skill rather than trusting the model to save a file:
// `--enable imagegenext` makes the built-in tool drop the rendered artifact into
// $CODEX_HOME/generated_images/, and we fetch the freshest file that postdates
// this run. The save-to-path instruction is only a best-effort verify-first.
const TIMEOUT_MS = 600000; // codex exec round-trips the sub; first-run tool spin-up is slow
const MTIME_SKEW_MS = 2000; // tolerate mtime granularity / clock skew (illo uses 2s)
function codexGeneratedDir() {
// Codex relocates CODEX_HOME on some hosts, so resolve it at run time.
return join(process.env.CODEX_HOME || join(homedir(), ".codex"), "generated_images");
}
// Newest artifact that postdates `sinceMs` (minus skew), so a stale prior render
// or a concurrent session's file can't be mistaken for this run's output.
function freshestGeneratedImage(sinceMs) {
const dir = codexGeneratedDir();
if (!existsSync(dir)) return null;
const floor = sinceMs - MTIME_SKEW_MS;
let best = null;
for (const name of readdirSync(dir)) {
let st;
try {
st = statSync(join(dir, name));
} catch {
continue;
}
if (!st.isFile() || st.mtimeMs < floor) continue;
if (!best || st.mtimeMs > best.mtimeMs) best = { path: join(dir, name), mtimeMs: st.mtimeMs };
}
return best?.path ?? null;
}
// Short `codex` subcommand → combined stdout+stderr, or null if it can't run.
function codexRun(args) {
try {
return execFileSync("codex", args, {
encoding: "utf8",
stdio: ["ignore", "pipe", "pipe"],
timeout: 10000,
});
} catch (err) {
return `${err.stdout?.toString() ?? ""}${err.stderr?.toString() ?? ""}` || null;
}
}
// Fail-fast host check (mirrors illo): don't burn a minutes-long exec when Codex
// isn't usable. Returns null when ready, else a human reason. imagegenext ships
// default-disabled ("under development"), so we check the ROW is present (the
// capability signal) — the exec enables it per-render with --enable.
function codexUnavailableReason() {
try {
const which = process.platform === "win32" ? "where" : "which";
execFileSync(which, ["codex"], { stdio: ["ignore", "ignore", "ignore"], timeout: 5000 });
} catch {
// A shell alias (e.g. `codex → /Applications/Codex.app/...`) is NOT enough:
// aliases live only in the interactive shell, so a spawned subprocess's PATH
// lookup can't see them. Symlink the real binary onto PATH.
return 'codex CLI not reachable on PATH (a shell alias won\'t work — spawned processes can\'t see aliases; symlink the real binary onto PATH, e.g. ln -s "$(readlink -f "$(command -v codex)")" ~/.local/bin/codex)';
}
// Auth marker: presence of the credentials file, NOT `codex login status`.
// That command prints "Logged in using ChatGPT" only to a human stream
// (stderr / TTY) and exits 0, so its piped stdout — how media-use spawns it —
// is empty, and the gate falsely reported "not logged in", blocking codex
// image gen in every headless / CI / agent run even when fully authed.
// auth.json is the durable, TTY-independent signal; token validity is proven
// by the exec itself, which fails cleanly if the login is stale.
const authPath = join(process.env.CODEX_HOME || join(homedir(), ".codex"), "auth.json");
if (!existsSync(authPath)) return "codex not logged in (run: codex login)";
const feats = codexRun(["features", "list"]);
if (feats == null) return "could not read `codex features list`";
if (!/\bimage_generation\b/.test(feats)) return "codex image_generation feature unavailable";
if (!/\bimagegenext\b/.test(feats)) return "codex imagegenext unavailable (upgrade Codex CLI)";
return null;
}
export async function codexImageGenerate(intent) {
const unavailable = codexUnavailableReason();
if (unavailable) {
console.error(`media-use: codex image upsell unavailable: ${unavailable}`);
return null;
}
const outPath = join(tmpdir(), `media-use-codex-${process.pid}-${Date.now()}.png`);
const prompt =
`${intent}\n\n` +
`Use your built-in image generation tool to render this, then save the image ` +
`to ${outPath} (overwrite if it exists). Do not ask for confirmation. ` +
`If you have no built-in image tool, do nothing (no PIL/matplotlib/SVG substitute).`;
try {
unlinkSync(outPath); // clear any prior file so verify-first can't accept a stale render
} catch {
/* no prior file */
}
const started = Date.now();
try {
execFileSync(
"codex",
[
"exec",
"--cd",
tmpdir(),
"-s",
"workspace-write",
"--skip-git-repo-check",
"--enable",
"imagegenext",
"-",
],
{ input: prompt, encoding: "utf8", timeout: TIMEOUT_MS, stdio: ["pipe", "pipe", "pipe"] },
);
} catch (err) {
console.error(
`media-use: \`codex exec\` image generation failed: ${err.stderr?.toString().trim().slice(-200) || err.message}`,
);
return null;
}
// Verify-first (save-to-path may have worked), else fetch the imagegenext artifact.
const produced =
existsSync(outPath) && statSync(outPath).size > 0 ? outPath : freshestGeneratedImage(started);
if (!produced) return null;
if (produced !== outPath) {
try {
copyFileSync(produced, outPath);
} catch {
return null;
}
}
return {
localPath: outPath,
ext: ".png",
source: "generated",
metadata: { description: intent, provider: "codex.image_gen", provenance: { prompt: intent } },
};
}
@@ -0,0 +1,117 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { existsSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
import { listTypes, getProviders } from "./registry.mjs";
import { CAPABILITIES, listModels } from "./local-models.mjs";
// Capstone: media-use must actually OWN each hyperframes media weakness. This
// test enforces the weakness→owner matrix in SKILL.md so a claim can't rot — if
// a capability's entrypoint disappears, this fails.
const SKILL = join(dirname(fileURLToPath(import.meta.url)), "..", "..");
test("weakness: audio-only → media-use resolves image + icon", () => {
for (const t of ["image", "icon"]) {
assert.ok(getProviders(t).length > 0, `no provider for ${t}`);
}
});
test("weakness: no third-party brand logos → media-use resolves logo", () => {
assert.ok(listTypes().includes("logo"), "logo type missing");
assert.ok(getProviders("logo").length >= 4, "logo cascade incomplete");
});
test("weakness: no voice/audio gen → media-use exposes voice + the audio engine", () => {
assert.ok(listTypes().includes("voice"), "voice type missing");
assert.ok(getProviders("voice").length > 0, "no enabled voice provider (Bin approved)");
assert.ok(existsSync(join(SKILL, "audio", "scripts", "audio.mjs")), "audio engine missing");
});
test("weakness: scattered audio engine → consolidated under media-use (hyperframes-media gone)", () => {
assert.ok(existsSync(join(SKILL, "audio", "scripts", "lib", "tts.mjs")), "tts engine missing");
assert.ok(
existsSync(join(SKILL, "audio", "assets", "sfx", "manifest.json")),
"bundled SFX missing",
);
});
test("weakness: no media-ops → ops guidance reference exists", () => {
assert.ok(existsSync(join(SKILL, "references", "operations.md")), "operations.md missing");
});
test("weakness: no transcript-driven cutting → cut compiler entrypoints exist", async () => {
assert.ok(existsSync(join(SKILL, "scripts", "transcript-cut.mjs")), "transcript-cut missing");
assert.ok(existsSync(join(SKILL, "scripts", "lib", "cutlist.mjs")), "cutlist lib missing");
const cutlist = await import("./cutlist.mjs");
assert.equal(typeof cutlist.compileCutList, "function");
});
test("weakness: whisper.cpp is weak → better local ASR (Parakeet) entrypoint exists", async () => {
assert.ok(existsSync(join(SKILL, "scripts", "transcribe.mjs")), "transcribe.mjs missing");
const pw = await import("./parakeet-words.mjs");
assert.equal(typeof pw.mergeTokensToWords, "function", "token->word merge missing");
const lm = await import("./local-models.mjs");
const asr = lm.listModels("asr");
const parakeet = asr.find((m) => m.id === "parakeet-mlx");
assert.ok(parakeet && parakeet.rank === 0, "Parakeet must be the rank-0 preferred ASR");
});
test("weakness: no auto-duck/loudness → duck compiler and recipes exist", async () => {
assert.ok(existsSync(join(SKILL, "scripts", "audio-duck.mjs")), "audio-duck missing");
assert.ok(existsSync(join(SKILL, "scripts", "lib", "duck.mjs")), "duck lib missing");
assert.ok(existsSync(join(SKILL, "references", "operations.md")), "operations.md missing");
const duck = await import("./duck.mjs");
assert.equal(typeof duck.speechSpans, "function");
assert.equal(typeof duck.duckKeyframes, "function");
});
test("weakness: no cross-project memory → global cache + ingest entrypoints exist", async () => {
const cache = await import("./cache.mjs");
assert.equal(typeof cache.cachePut, "function");
assert.equal(typeof cache.promote, "function");
assert.equal(typeof cache.globalMediaDir, "function");
const freeze = await import("./freeze.mjs");
assert.equal(typeof freeze.isDirectMediaUrl, "function", "ingest URL guard missing");
});
// Wenbo (06-29): heygen free-usage is the default; local models are the opt-out
// fallback ("if user no, then local"). We still assert the fallback table is
// populated so the opt-out path stays real.
test("weakness: weak local defaults → local models exist as the opt-out fallback (tts/asr/upscale)", () => {
for (const cap of ["tts", "asr", "upscale"]) {
assert.ok(CAPABILITIES.includes(cap), `capability ${cap} missing`);
assert.ok(listModels(cap).length > 0, `no local models for ${cap}`);
}
});
test("weakness: no image generation → local mflux (RAM-graded) + codex upsell", async () => {
const ps = getProviders("image");
assert.ok(
ps.some((p) => p.name === "mflux.local" && typeof p.generate === "function"),
"local image gen missing",
);
assert.ok(
ps.some((p) => p.name === "codex.image_gen" && typeof p.generate === "function"),
"codex image upsell missing",
);
const lm = await import("./local-models.mjs");
assert.ok(lm.CAPABILITIES.includes("imagegen"), "imagegen capability missing");
assert.ok(lm.listModels("imagegen").length >= 3, "imagegen RAM ladder too small");
assert.equal(typeof lm.describeModelLadder, "function", "agent-facing ladder missing");
});
test("weakness: no video generation → local videogen ladder + heygen avatar upsell", async () => {
const lm = await import("./local-models.mjs");
assert.ok(lm.CAPABILITIES.includes("videogen"), "videogen capability missing");
assert.ok(lm.listModels("videogen").length >= 2, "videogen ladder too small");
const ops = existsSync(join(SKILL, "references", "operations.md"));
assert.ok(ops, "operations.md (avatar-upsell recipe) missing");
});
test("every resolve type has at least one enabled provider", () => {
for (const t of listTypes()) {
assert.ok(getProviders(t).length > 0, `type ${t} has no enabled provider`);
}
});
+181
View File
@@ -0,0 +1,181 @@
const DEFAULT_SIZE = 33;
const MAX_SIZE = 64;
function clamp(value, min, max) {
if (!Number.isFinite(value)) return 0;
return Math.min(max, Math.max(min, value));
}
function clampUnit(value) {
return clamp(value, 0, 1);
}
function readParam(params, key, min, max) {
return clamp(Number(params?.[key] ?? 0), min, max);
}
function luma([r, g, b]) {
// Rec.709 luma weightings (matches the color space the grading runtime uses).
return r * 0.2126 + g * 0.7152 + b * 0.0722;
}
function smoothstep(edge0, edge1, value) {
const t = clampUnit((value - edge0) / (edge1 - edge0));
return t * t * (3 - 2 * t);
}
function applyLiftGain(color, params) {
const y = luma(color);
const blacks = readParam(params, "blacks", -1, 1);
const shadows = readParam(params, "shadows", -1, 1);
const highlights = readParam(params, "highlights", -1, 1);
const whites = readParam(params, "whites", -1, 1);
const shadowMask = 1 - smoothstep(0.18, 0.62, y);
const highlightMask = smoothstep(0.38, 0.82, y);
const offset =
blacks * 0.08 + shadows * 0.12 * shadowMask + highlights * 0.12 * highlightMask + whites * 0.08;
return color.map((channel) => clampUnit(channel + offset));
}
function applyExposure(color, params) {
const exposure = readParam(params, "exposure", -2, 2);
const gain = 2 ** exposure;
const lift = Math.max(0, exposure) * 0.015;
return color.map((channel) => clampUnit(channel * gain + lift));
}
function applyContrast(color, params) {
const contrast = readParam(params, "contrast", -1, 1);
if (contrast === 0) return color;
const factor = 1 + contrast * 1.2;
return color.map((channel) => clampUnit(0.5 + (channel - 0.5) * factor));
}
function applyWhiteBalance(color, params) {
const temperature = readParam(params, "temperature", -1, 1);
const tint = readParam(params, "tint", -1, 1);
const redScale = 1 + temperature * 0.28 + tint * 0.08;
const greenScale = 1 - Math.abs(tint) * 0.1 - tint * 0.08;
const blueScale = 1 - temperature * 0.28 + tint * 0.08;
return [
clampUnit(color[0] * redScale),
clampUnit(color[1] * greenScale),
clampUnit(color[2] * blueScale),
];
}
function applySplitTone(color, params) {
const split = params?.splitTone;
if (!split) return color;
const intensity = clampUnit(Number(split.intensity ?? 0));
if (intensity === 0) return color;
const balance = clampUnit(Number(split.balance ?? 0.5));
const y = luma(color);
const shadowMask = 1 - smoothstep(balance - 0.25, balance + 0.2, y);
const highlightMask = smoothstep(balance - 0.2, balance + 0.25, y);
const shadows = Array.isArray(split.shadows) ? split.shadows : [0, 0, 0];
const highlights = Array.isArray(split.highlights) ? split.highlights : [0, 0, 0];
return color.map((channel, i) =>
clampUnit(
channel +
Number(shadows[i] ?? 0) * shadowMask * intensity +
Number(highlights[i] ?? 0) * highlightMask * intensity,
),
);
}
function applySaturation(color, params) {
const saturation = readParam(params, "saturation", -1, 1);
const vibrance = readParam(params, "vibrance", -1, 1);
if (saturation === 0 && vibrance === 0) return color;
const y = luma(color);
const currentSat = Math.max(
Math.abs(color[0] - y),
Math.abs(color[1] - y),
Math.abs(color[2] - y),
);
const vibranceWeight = 1 - clampUnit(currentSat * 2);
const factor = clamp(1 + saturation + vibrance * vibranceWeight, 0, 2.5);
return color.map((channel) => clampUnit(y + (channel - y) * factor));
}
function applyParams(color, params) {
let out = applyLiftGain(color, params);
out = applyExposure(out, params);
out = applyContrast(out, params);
out = applyWhiteBalance(out, params);
out = applySplitTone(out, params);
out = applySaturation(out, params);
return out;
}
function formatNumber(value) {
return clampUnit(value).toFixed(6);
}
export function buildCube(params = {}, size = DEFAULT_SIZE) {
if (!Number.isInteger(size) || size < 2 || size > MAX_SIZE) {
throw new Error(`LUT size must be an integer from 2 to ${MAX_SIZE}`);
}
const lines = [
`TITLE "media-use parametric grade"`,
"DOMAIN_MIN 0 0 0",
"DOMAIN_MAX 1 1 1",
`LUT_3D_SIZE ${size}`,
];
const denom = size - 1;
for (let b = 0; b < size; b++) {
for (let g = 0; g < size; g++) {
for (let r = 0; r < size; r++) {
const out = applyParams([r / denom, g / denom, b / denom], params);
lines.push(`${formatNumber(out[0])} ${formatNumber(out[1])} ${formatNumber(out[2])}`);
}
}
}
return `${lines.join("\n")}\n`;
}
export function paramsFromIntent(intent) {
const text = String(intent ?? "").toLowerCase();
const params = {};
let matched = false;
if (/\b(warm|golden|sunlit|sunny)\b/.test(text)) {
params.temperature = 0.18;
matched = true;
} else if (/\b(cool|blue|icy|crisp)\b/.test(text)) {
params.temperature = -0.16;
matched = true;
}
if (/\b(cinematic|film|movie)\b/.test(text)) {
params.contrast = 0.08;
params.saturation = 0.04;
matched = true;
}
if (/\b(punchy|contrast|dramatic|bold)\b/.test(text)) {
params.contrast = Math.max(params.contrast ?? 0, 0.22);
matched = true;
}
if (/\b(bright|airy|lift)\b/.test(text)) {
params.exposure = 0.16;
params.shadows = 0.08;
matched = true;
}
if (/\b(dark|moody|low-key)\b/.test(text)) {
params.exposure = -0.12;
params.contrast = Math.max(params.contrast ?? 0, 0.12);
matched = true;
}
if (/\b(vibrant|saturated|colorful)\b/.test(text)) {
params.saturation = Math.max(params.saturation ?? 0, 0.16);
params.vibrance = 0.12;
matched = true;
}
if (/\b(muted|desaturated|washed)\b/.test(text)) {
params.saturation = Math.min(params.saturation ?? 0, -0.16);
matched = true;
}
return matched ? params : null;
}
@@ -0,0 +1,80 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { buildCube, paramsFromIntent } from "./cube-build.mjs";
import { validateCube } from "./cube-validate.mjs";
function rows(cube) {
return cube
.split(/\r?\n/)
.map((line) => line.trim())
.filter((line) => /^[+-]?(?:\d|\.\d)/.test(line))
.map((line) => line.split(/\s+/).map(Number));
}
function rowAt(cubeRows, size, r, g, b) {
return cubeRows[(b * size + g) * size + r];
}
function luma(row) {
return row[0] * 0.2126 + row[1] * 0.7152 + row[2] * 0.0722;
}
test("all-zero params produce a near-identity LUT", () => {
const cube = buildCube({}, 3);
assert.equal(validateCube(cube).ok, true);
const parsed = rows(cube);
for (let b = 0; b < 3; b++) {
for (let g = 0; g < 3; g++) {
for (let r = 0; r < 3; r++) {
const row = rowAt(parsed, 3, r, g, b);
assert.ok(Math.abs(row[0] - r / 2) < 0.000001);
assert.ok(Math.abs(row[1] - g / 2) < 0.000001);
assert.ok(Math.abs(row[2] - b / 2) < 0.000001);
}
}
}
});
test("positive exposure increases unclipped output luma", () => {
const identity = rows(buildCube({}, 5));
const exposed = rows(buildCube({ exposure: 0.3 }, 5));
for (let i = 0; i < identity.length; i++) {
const before = luma(identity[i]);
if (before > 0.02 && before < 0.95) {
assert.ok(luma(exposed[i]) > before, `row ${i} should brighten`);
}
}
});
test("positive temperature warms mid-gray", () => {
const parsed = rows(buildCube({ temperature: 0.2 }, 3));
const mid = rowAt(parsed, 3, 1, 1, 1);
assert.ok(mid[0] > 0.5, "red channel should rise");
assert.ok(mid[2] < 0.5, "blue channel should fall");
});
test("positive contrast darkens shadows and brightens highlights", () => {
const parsed = rows(buildCube({ contrast: 0.3 }, 5));
const shadow = rowAt(parsed, 5, 1, 1, 1);
const highlight = rowAt(parsed, 5, 3, 3, 3);
assert.ok(luma(shadow) < 0.25, "below-mid gray should darken");
assert.ok(luma(highlight) > 0.75, "above-mid gray should brighten");
});
test("outputs validate at the default size and are deterministic", () => {
const params = { exposure: 0.15, contrast: 0.2, temperature: -0.1, saturation: 0.12 };
const a = buildCube(params);
const b = buildCube(params);
assert.equal(a, b);
assert.equal(validateCube(a).ok, true);
assert.equal(validateCube(a).size, 33);
});
test("paramsFromIntent declines zero-overlap prompts and maps technical words", () => {
assert.equal(paramsFromIntent("zqxv imaginary neutron look"), null);
assert.deepEqual(paramsFromIntent("warm cinematic"), {
temperature: 0.18,
contrast: 0.08,
saturation: 0.04,
});
});
@@ -0,0 +1,198 @@
#!/usr/bin/env node
// Standalone mirror of packages/core/src/colorLuts.ts. media-use cannot import
// the TypeScript source at runtime, so cube-validate.test.mjs mirrors core
// parser cases to catch drift in accepted .cube files before freezing them.
import { readFileSync } from "node:fs";
import { resolve as resolvePath } from "node:path";
import { fileURLToPath } from "node:url";
export const DEFAULT_MAX_CUBE_LUT_SIZE = 64;
const DEFAULT_DOMAIN_MIN = [0, 0, 0];
const DEFAULT_DOMAIN_MAX = [1, 1, 1];
class CubeValidateError extends Error {
constructor(message, lineNumber = null) {
super(lineNumber == null ? message : `${message} at line ${lineNumber}`);
this.name = "CubeValidateError";
this.lineNumber = lineNumber;
}
}
function stripComment(line) {
let inQuote = false;
for (let i = 0; i < line.length; i++) {
const char = line[i];
if (char === '"') inQuote = !inQuote;
if (char === "#" && !inQuote) return line.slice(0, i);
}
return line;
}
function parseFiniteNumber(value, lineNumber) {
const parsed = Number(value);
if (!Number.isFinite(parsed)) {
throw new CubeValidateError(`Invalid number "${value}"`, lineNumber);
}
return parsed;
}
function parseVec3(parts, keyword, lineNumber) {
if (parts.length !== 3) {
throw new CubeValidateError(`${keyword} expects three numbers`, lineNumber);
}
return [
parseFiniteNumber(parts[0], lineNumber),
parseFiniteNumber(parts[1], lineNumber),
parseFiniteNumber(parts[2], lineNumber),
];
}
function parseSize(value, keyword, lineNumber) {
if (!value) throw new CubeValidateError(`${keyword} expects a size`, lineNumber);
const parsed = Number(value);
if (!Number.isInteger(parsed) || parsed < 2) {
throw new CubeValidateError(`${keyword} must be an integer greater than 1`, lineNumber);
}
return parsed;
}
function validateDomain(domainMin, domainMax) {
if (
domainMax[0] <= domainMin[0] ||
domainMax[1] <= domainMin[1] ||
domainMax[2] <= domainMin[2]
) {
throw new CubeValidateError("DOMAIN_MAX values must be greater than DOMAIN_MIN values");
}
}
function isNumericDataLine(token) {
return /^[+-]?(?:\d|\.\d)/.test(token);
}
function parseCube(input, options = {}) {
const maxSize = options.maxSize ?? DEFAULT_MAX_CUBE_LUT_SIZE;
let domainMin = DEFAULT_DOMAIN_MIN;
let domainMax = DEFAULT_DOMAIN_MAX;
let lut1dSize = null;
let lut3dSize = null;
let rows = 0;
const lines = String(input)
.replace(/^\uFEFF/, "")
.split(/\r?\n/);
for (let i = 0; i < lines.length; i++) {
const lineNumber = i + 1;
const line = stripComment(lines[i] ?? "").trim();
if (!line) continue;
const parts = line.split(/\s+/);
const keyword = (parts[0] ?? "").toUpperCase();
const rest = parts.slice(1);
if (keyword === "TITLE") continue;
if (keyword === "DOMAIN_MIN") {
domainMin = parseVec3(rest, keyword, lineNumber);
continue;
}
if (keyword === "DOMAIN_MAX") {
domainMax = parseVec3(rest, keyword, lineNumber);
continue;
}
if (keyword === "LUT_3D_INPUT_RANGE") {
if (rest.length !== 2) {
throw new CubeValidateError(`${keyword} expects two numbers`, lineNumber);
}
const min = parseFiniteNumber(rest[0], lineNumber);
const max = parseFiniteNumber(rest[1], lineNumber);
if (max <= min) {
throw new CubeValidateError("LUT_3D_INPUT_RANGE max must exceed min", lineNumber);
}
domainMin = [min, min, min];
domainMax = [max, max, max];
continue;
}
if (keyword === "LUT_1D_SIZE") {
lut1dSize = parseSize(rest[0], keyword, lineNumber);
continue;
}
if (keyword === "LUT_3D_SIZE") {
lut3dSize = parseSize(rest[0], keyword, lineNumber);
if (lut3dSize > maxSize) {
throw new CubeValidateError(`LUT_3D_SIZE ${lut3dSize} exceeds max ${maxSize}`, lineNumber);
}
continue;
}
if (!isNumericDataLine(keyword)) {
if (keyword.startsWith("LUT_")) {
throw new CubeValidateError(`Unsupported cube keyword ${keyword}`, lineNumber);
}
continue;
}
if (!lut3dSize) {
if (lut1dSize) {
throw new CubeValidateError("1D cube LUTs are not supported yet", lineNumber);
}
throw new CubeValidateError("LUT data appears before LUT_3D_SIZE", lineNumber);
}
if (parts.length !== 3) {
throw new CubeValidateError("LUT data rows must contain three numbers", lineNumber);
}
parseFiniteNumber(parts[0], lineNumber);
parseFiniteNumber(parts[1], lineNumber);
parseFiniteNumber(parts[2], lineNumber);
rows++;
}
if (lut1dSize && lut3dSize) {
throw new CubeValidateError("Mixed 1D and 3D cube LUTs are not supported yet");
}
if (!lut3dSize) {
if (lut1dSize) throw new CubeValidateError("1D cube LUTs are not supported yet");
throw new CubeValidateError("Missing LUT_3D_SIZE");
}
validateDomain(domainMin, domainMax);
const expectedRows = lut3dSize * lut3dSize * lut3dSize;
if (rows !== expectedRows) {
throw new CubeValidateError(
`Expected ${expectedRows} LUT rows for size ${lut3dSize}, found ${rows}`,
);
}
return { size: lut3dSize };
}
export function validateCube(input, options = {}) {
try {
const parsed = parseCube(input, options);
return { ok: true, size: parsed.size };
} catch (err) {
return { ok: false, error: err.message };
}
}
export function validateCubeFile(filePath, options = {}) {
return validateCube(readFileSync(filePath, "utf8"), options);
}
function main(argv) {
const file = argv[2];
if (!file) {
console.error("usage: cube-validate.mjs <file.cube>");
process.exit(2);
}
const result = validateCubeFile(file);
if (!result.ok) {
console.error(`error: ${result.error}`);
process.exit(1);
}
console.log(`ok: LUT_3D_SIZE ${result.size}`);
}
if (process.argv[1] && resolvePath(process.argv[1]) === fileURLToPath(import.meta.url)) {
main(process.argv);
}
@@ -0,0 +1,125 @@
import { strict as assert } from "node:assert";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { execFileSync } from "node:child_process";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { test } from "node:test";
import { validateCube } from "./cube-validate.mjs";
const IDENTITY_2 = `
# comment
TITLE "Identity 2"
DOMAIN_MIN 0 0 0
DOMAIN_MAX 1 1 1
LUT_3D_SIZE 2
0 0 0
1 0 0
0 1 0
1 1 0
0 0 1
1 0 1
0 1 1
1 1 1
`;
test("accepts a valid minimal 3D cube LUT", () => {
const result = validateCube(IDENTITY_2);
assert.deepEqual(result, { ok: true, size: 2 });
});
test("rejects oversize LUTs with the core parser message", () => {
const result = validateCube("LUT_3D_SIZE 65", { maxSize: 64 });
assert.equal(result.ok, false);
assert.match(result.error, /LUT_3D_SIZE 65 exceeds max 64/);
});
test("rejects data rows before LUT_3D_SIZE", () => {
const result = validateCube("0 0 0\nLUT_3D_SIZE 2");
assert.equal(result.ok, false);
assert.match(result.error, /LUT data appears before LUT_3D_SIZE/);
});
test("rejects missing LUT_3D_SIZE", () => {
const result = validateCube('TITLE "No Size"');
assert.equal(result.ok, false);
assert.match(result.error, /Missing LUT_3D_SIZE/);
});
test("rejects row count mismatches with the core parser message", () => {
const result = validateCube("LUT_3D_SIZE 2\n0 0 0");
assert.equal(result.ok, false);
assert.match(result.error, /Expected 8 LUT rows/);
});
test("rejects inverted domains", () => {
const result = validateCube(`
DOMAIN_MIN 0 0 0
DOMAIN_MAX 1 0 1
LUT_3D_SIZE 2
0 0 0
1 0 0
0 1 0
1 1 0
0 0 1
1 0 1
0 1 1
1 1 1
`);
assert.equal(result.ok, false);
assert.match(result.error, /DOMAIN_MAX values must be greater than DOMAIN_MIN values/);
});
test("rejects unsupported 1D and mixed cube LUTs", () => {
const oneD = validateCube("LUT_1D_SIZE 2\n0 0 0\n1 1 1");
assert.equal(oneD.ok, false);
assert.match(oneD.error, /1D cube LUTs are not supported yet/);
const mixed = validateCube(`
LUT_1D_SIZE 2
LUT_3D_SIZE 2
0 0 0
1 0 0
0 1 0
1 1 0
0 0 1
1 0 1
0 1 1
1 1 1
`);
assert.equal(mixed.ok, false);
assert.match(mixed.error, /Mixed 1D and 3D cube LUTs are not supported yet/);
});
test("CLI exits zero for valid files and non-zero for invalid files", () => {
const dir = mkdtempSync(join(tmpdir(), "mu-cube-validate-"));
try {
const valid = join(dir, "valid.cube");
const invalid = join(dir, "invalid.cube");
writeFileSync(valid, IDENTITY_2);
writeFileSync(invalid, "LUT_3D_SIZE 65");
const out = execFileSync(
process.execPath,
[new URL("./cube-validate.mjs", import.meta.url).pathname, valid],
{
encoding: "utf8",
},
);
assert.match(out, /ok: LUT_3D_SIZE 2/);
assert.throws(
() =>
execFileSync(
process.execPath,
[new URL("./cube-validate.mjs", import.meta.url).pathname, invalid],
{
encoding: "utf8",
stdio: "pipe",
},
),
(err) => err.status === 1 && String(err.stderr).includes("exceeds max 64"),
);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
+184
View File
@@ -0,0 +1,184 @@
import { normalizeWords } from "./words.mjs";
const MIN_SEGMENT_SECONDS = 0.2;
const SILENCE_PAD_SECONDS = 0.15;
export function compileCutList(transcript, opts = {}) {
const words = normalizeWords(transcript);
if (opts.keep != null && hasRemovalSource(opts)) {
throw new Error("--keep is mutually exclusive with removal options");
}
if (opts.keep != null) {
const duration = durationFrom(words, opts);
const ranges = parseTimeRanges(opts.keep);
return finalizeKept(duration != null ? clampRanges(ranges, duration) : ranges);
}
const duration = durationFrom(words, opts);
if (!duration) return [];
const removals = [
...parseTimeRanges(opts.remove),
...wordIndexRanges(words, opts.removeWords),
...fillerRanges(words, opts.removeFillers),
...silenceRanges(words, opts.cutSilence),
];
const mergedRemovals = mergeRanges(clampRanges(removals, duration));
return finalizeKept(invertRanges(mergedRemovals, duration));
}
function hasRemovalSource(opts) {
return (
opts.remove != null ||
opts.removeWords != null ||
opts.removeFillers != null ||
opts.cutSilence != null
);
}
function durationFrom(words, opts) {
const explicit = Number(opts.duration ?? opts.totalDuration);
if (Number.isFinite(explicit) && explicit > 0) return explicit;
const last = words.at(-1);
return last && Number.isFinite(last.end) && last.end > 0 ? last.end : null;
}
function parseTimeRanges(value) {
if (value == null || value === false || value === "") return [];
if (typeof value === "string") {
return value
.split(",")
.map((part) => part.trim())
.filter(Boolean)
.map(parseRangeString);
}
if (!Array.isArray(value)) throw new Error("range list must be a string or array");
return value.map((range) => {
if (Array.isArray(range)) return cleanRange(Number(range[0]), Number(range[1]));
return cleanRange(Number(range?.start), Number(range?.end));
});
}
function parseRangeString(value) {
const match = value.match(/^([0-9]*\.?[0-9]+)\s*-\s*([0-9]*\.?[0-9]+)$/);
if (!match) throw new Error(`invalid range: ${value}`);
return cleanRange(Number(match[1]), Number(match[2]));
}
function cleanRange(start, end) {
if (!Number.isFinite(start) || !Number.isFinite(end)) {
throw new Error("range start/end must be finite numbers");
}
if (end < start) throw new Error(`range end ${end} is before start ${start}`);
return { start, end };
}
function wordIndexRanges(words, value) {
if (value == null || value === false || value === "") return [];
const ranges = typeof value === "string" ? value.split(",") : value;
if (!Array.isArray(ranges)) throw new Error("--remove-words must be a string or array");
return ranges
.map((range) => (typeof range === "string" ? range.trim() : range))
.filter(Boolean)
.map((range) => {
const [first, last = first] =
typeof range === "string" ? range.split("-").map((n) => n.trim()) : range;
const startIndex = Number(first);
const endIndex = Number(last);
if (!Number.isInteger(startIndex) || !Number.isInteger(endIndex)) {
throw new Error(`invalid word range: ${range}`);
}
if (startIndex < 0 || endIndex < startIndex || endIndex >= words.length) {
throw new Error(`word range out of bounds: ${range}`);
}
return { start: words[startIndex].start, end: words[endIndex].end };
});
}
function fillerRanges(words, value) {
if (value == null || value === false || value === "") return [];
const fillers = Array.isArray(value)
? value
: String(value)
.split(",")
.map((s) => s.trim());
const set = new Set(fillers.filter(Boolean).map(bareToken));
if (set.size === 0) return [];
// Whisper emits words with attached punctuation and arbitrary case
// ("UM," / "Um."), so compare bare tokens.
return words
.filter((word) => set.has(bareToken(word.text)))
.map((word) => ({ start: word.start, end: word.end }));
}
function bareToken(text) {
return String(text)
.toLowerCase()
.replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, "");
}
function silenceRanges(words, value) {
if (value == null || value === false || value === "") return [];
const threshold = Number(value);
if (!Number.isFinite(threshold) || threshold <= 0) {
throw new Error("--cut-silence must be a positive number");
}
const ranges = [];
for (let i = 0; i < words.length - 1; i++) {
const current = words[i];
const next = words[i + 1];
const gap = next.start - current.end;
if (gap <= threshold) continue;
const start = current.end + SILENCE_PAD_SECONDS;
const end = next.start - SILENCE_PAD_SECONDS;
if (end > start) ranges.push({ start, end });
}
return ranges;
}
function clampRanges(ranges, duration) {
return ranges
.map((range) => ({
start: Math.max(0, Math.min(duration, range.start)),
end: Math.max(0, Math.min(duration, range.end)),
}))
.filter((range) => range.end > range.start);
}
function mergeRanges(ranges) {
const sorted = ranges
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
.sort((a, b) => a.start - b.start || a.end - b.end);
const merged = [];
for (const range of sorted) {
const prev = merged.at(-1);
if (prev && range.start <= prev.end) {
prev.end = Math.max(prev.end, range.end);
} else {
merged.push({ ...range });
}
}
return merged;
}
function invertRanges(removals, duration) {
const kept = [];
let cursor = 0;
for (const range of removals) {
if (range.start > cursor) kept.push({ start: cursor, end: range.start });
cursor = Math.max(cursor, range.end);
}
if (cursor < duration) kept.push({ start: cursor, end: duration });
return kept;
}
function finalizeKept(ranges) {
return mergeRanges(ranges)
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
.filter((range) => round3(range.end - range.start) >= MIN_SEGMENT_SECONDS);
}
function round3(n) {
return Math.round(Number(n) * 1000) / 1000;
}
@@ -0,0 +1,148 @@
import { strict as assert } from "node:assert";
import { execFileSync } from "node:child_process";
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { join, dirname } from "node:path";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
import { test } from "node:test";
import { compileCutList } from "./cutlist.mjs";
const HERE = dirname(fileURLToPath(import.meta.url));
const SCRIPT = join(HERE, "..", "transcript-cut.mjs");
test("explicit --remove ranges invert to kept segments", () => {
const transcript = [
word("w0", "alpha", 0, 1),
word("w1", "beta", 1.2, 2),
word("w2", "gamma", 2.2, 5),
];
assert.deepEqual(compileCutList(transcript, { remove: "1-2.5" }), [
{ start: 0, end: 1 },
{ start: 2.5, end: 5 },
]);
});
test("--remove-words resolves inclusive word-index ranges to time ranges", () => {
const transcript = [
word("w0", "zero", 0, 0.5),
word("w1", "one", 0.6, 1),
word("w2", "two", 1.1, 1.5),
word("w3", "three", 2, 3),
];
assert.deepEqual(compileCutList(transcript, { removeWords: "1-2" }), [
{ start: 0, end: 0.6 },
{ start: 1.5, end: 3 },
]);
});
test("--remove-fillers drops case-insensitive matching words", () => {
const transcript = [
word("w0", "Hello", 0, 0.5),
word("w1", "Um", 0.5, 0.7),
word("w2", "world", 0.8, 1.2),
word("w3", "LIKE", 1.3, 1.5),
word("w4", "done", 1.6, 2),
];
assert.deepEqual(compileCutList(transcript, { removeFillers: "um,like" }), [
{ start: 0, end: 0.5 },
{ start: 0.7, end: 1.3 },
{ start: 1.5, end: 2 },
]);
});
test("--cut-silence removes only the center of long inter-word gaps", () => {
const transcript = [word("w0", "a", 0, 0.5), word("w1", "b", 2, 2.5), word("w2", "c", 2.7, 3)];
assert.deepEqual(compileCutList(transcript, { cutSilence: 0.8 }), [
{ start: 0, end: 0.65 },
{ start: 1.85, end: 3 },
]);
});
test("overlapping removal sources merge before inversion", () => {
const transcript = [
word("w0", "start", 0, 0.5),
word("w1", "um", 0.9, 1.1),
word("w2", "middle", 2.5, 2.8),
word("w3", "more", 3.1, 3.4),
word("w4", "end", 5.5, 6),
];
assert.deepEqual(
compileCutList(transcript, {
remove: "1-2.7",
removeWords: "2-3",
removeFillers: "um",
}),
[
{ start: 0, end: 0.9 },
{ start: 3.4, end: 6 },
],
);
});
test("kept slivers shorter than 0.2s are dropped", () => {
const transcript = [word("w0", "start", 0, 0.5), word("w1", "end", 2.5, 3)];
assert.deepEqual(compileCutList(transcript, { remove: "0.1-2.95" }), []);
});
test("--keep is inverse mode and coalesces direct kept ranges", () => {
const transcript = [word("w0", "start", 0, 0.5), word("w1", "end", 4.5, 5)];
assert.deepEqual(compileCutList(transcript, { keep: "3-4,1-2,1.5-2.5,4.1-4.2" }), [
{ start: 1, end: 2.5 },
{ start: 3, end: 4 },
]);
});
test("--plan on a fixture transcript prints the exact segment JSON", () => {
const dir = mkdtempSync(join(tmpdir(), "media-use-cutlist-"));
try {
const transcriptPath = join(dir, "fixture.json");
writeFileSync(
transcriptPath,
JSON.stringify([
word("w0", "hello", 0, 0.4),
word("w1", "um", 0.5, 0.65),
word("w2", "there", 0.7, 1),
word("w3", "pause", 2.2, 2.5),
word("w4", "end", 2.7, 3.2),
]),
);
const out = execFileSync(
process.execPath,
[
SCRIPT,
"--input",
"ignored.mp4",
"--transcript",
transcriptPath,
"--remove",
"0.9-1.2",
"--remove-fillers",
"um",
"--cut-silence",
"0.8",
"--plan",
],
{ encoding: "utf8" },
);
assert.deepEqual(JSON.parse(out), [
{ start: 0, end: 0.5 },
{ start: 0.65, end: 0.9 },
{ start: 2.05, end: 3.2 },
]);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
function word(id, text, start, end) {
return { id, text, start, end };
}
+89
View File
@@ -0,0 +1,89 @@
import { wordListsFromMediaMeta } from "./words.mjs";
/**
* Speech spans from word timestamps.
*
* audio_meta.json word times are relative to EACH LINE'S OWN FILE, not to the
* composition. Without placement info, multiple lines would overlap at t=0 and
* merge into one bogus span. Placement options:
* offsets: { [voiceId]: startSeconds } explicit composition placement
* sequential: stack lines back to back (plus `gap` seconds between lines)
* A single word list (bare transcript) needs neither.
*/
export function speechSpans(meta, { mergeGap = 0.6, offsets, sequential = false, gap = 0 } = {}) {
const merge = Number(mergeGap);
const lists = wordListsFromMediaMeta(meta);
const voices = Array.isArray(meta?.voices) ? meta.voices : [];
if (lists.length > 1 && !offsets && !sequential) {
throw new Error(
"audio_meta has multiple voice lines with file-relative times; pass --sequential or --offsets so spans land at composition time",
);
}
const intervals = [];
let cursor = 0;
for (let i = 0; i < lists.length; i++) {
const voice = voices[i];
let offset = 0;
if (offsets) {
const id = voice?.id ?? String(i);
if (!(id in offsets)) throw new Error(`--offsets is missing voice "${id}"`);
offset = Number(offsets[id]) || 0;
} else if (sequential) {
offset = cursor;
const lineDuration = Number(voice?.duration_s) || Math.max(...lists[i].map((w) => w.end), 0);
cursor += lineDuration + (Number(gap) || 0);
}
for (const word of lists[i]) {
if (word.end > word.start)
intervals.push({ start: word.start + offset, end: word.end + offset });
}
}
return mergeIntervals(intervals, Number.isFinite(merge) && merge >= 0 ? merge : 0.6);
}
export function duckKeyframes(
spans,
{ duck = 0.25, attack = 0.15, release = 0.4, baseVolume = 1 } = {},
) {
const base = finiteOr(baseVolume, 1);
const ducked = round3(base * finiteOr(duck, 0.25));
const keyframes = [];
for (const span of spans) {
keyframes.push({
time: round3(Math.max(0, finiteOr(span.start, 0))),
volume: ducked,
duration: round3(finiteOr(attack, 0.15)),
});
keyframes.push({
time: round3(Math.max(0, finiteOr(span.end, 0))),
volume: round3(base),
duration: round3(finiteOr(release, 0.4)),
});
}
return keyframes.sort((a, b) => a.time - b.time);
}
function mergeIntervals(intervals, mergeGap) {
const sorted = intervals
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
.sort((a, b) => a.start - b.start || a.end - b.end);
const merged = [];
for (const range of sorted) {
const prev = merged.at(-1);
if (prev && (range.start <= prev.end || range.start - prev.end < mergeGap)) {
prev.end = Math.max(prev.end, range.end);
} else {
merged.push({ ...range });
}
}
return merged;
}
function finiteOr(value, fallback) {
const n = Number(value);
return Number.isFinite(n) ? n : fallback;
}
function round3(n) {
return Math.round(Number(n) * 1000) / 1000;
}
+118
View File
@@ -0,0 +1,118 @@
import { strict as assert } from "node:assert";
import { execFileSync } from "node:child_process";
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { join, dirname } from "node:path";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
import { test } from "node:test";
import { duckKeyframes, speechSpans } from "./duck.mjs";
const HERE = dirname(fileURLToPath(import.meta.url));
const SCRIPT = join(HERE, "..", "audio-duck.mjs");
test("speechSpans bridges gaps smaller than mergeGap", () => {
const meta = {
words: [word("w0", "one", 0, 0.5), word("w1", "two", 0.8, 1), word("w2", "three", 2, 2.2)],
};
assert.deepEqual(speechSpans(meta, { mergeGap: 0.4 }), [
{ start: 0, end: 1 },
{ start: 2, end: 2.2 },
]);
});
test("speechSpans refuses multi-line meta without placement (file-relative times)", () => {
const meta = {
voices: [
{ id: "a", words: [word("w0", "one", 0, 1)] },
{ id: "b", words: [word("w1", "two", 0, 1)] },
],
};
assert.throws(() => speechSpans(meta, { mergeGap: 0.2 }), /--sequential or --offsets/);
});
test("speechSpans sequential stacks lines by duration plus gap", () => {
const meta = {
voices: [
{ id: "a", duration_s: 2, words: [word("w0", "one", 0.1, 1.9)] },
{ id: "b", duration_s: 1, words: [word("w1", "two", 0.1, 0.9)] },
],
};
assert.deepEqual(speechSpans(meta, { mergeGap: 0.2, sequential: true, gap: 0.5 }), [
{ start: 0.1, end: 1.9 },
{ start: 2.6, end: 3.4 },
]);
});
test("speechSpans explicit offsets place each line at composition time", () => {
const meta = {
voices: [
{ id: "a", words: [word("w0", "one", 0, 1)] },
{ id: "b", words: [word("w1", "two", 0, 1)] },
],
};
assert.deepEqual(speechSpans(meta, { mergeGap: 0.2, offsets: { a: 0, b: 4 } }), [
{ start: 0, end: 1 },
{ start: 4, end: 5 },
]);
assert.throws(() => speechSpans(meta, { offsets: { a: 0 } }), /missing voice "b"/);
});
test("speechSpans returns empty spans for empty input", () => {
assert.deepEqual(speechSpans({ voices: [] }, { mergeGap: 0.6 }), []);
});
test("duckKeyframes shapes attack and release from base volume", () => {
assert.deepEqual(
duckKeyframes([{ start: 3, end: 5 }], {
duck: 0.25,
attack: 0.15,
release: 0.4,
baseVolume: 0.6,
}),
[
{ time: 3, volume: 0.15, duration: 0.15 },
{ time: 5, volume: 0.6, duration: 0.4 },
],
);
});
test("--json spans match --merge-gap semantics exactly", () => {
const dir = mkdtempSync(join(tmpdir(), "media-use-duck-"));
try {
const metaPath = join(dir, "audio_meta.json");
writeFileSync(
metaPath,
JSON.stringify({
voices: [
{
id: "narration",
words: [
word("w0", "one", 0, 0.4),
word("w1", "two", 0.9, 1.2),
word("w2", "three", 1.8, 2.1),
],
},
],
}),
);
const out = execFileSync(
process.execPath,
[SCRIPT, "--meta", metaPath, "--target", "#bgm", "--merge-gap", "0.6", "--json"],
{ encoding: "utf8" },
);
const parsed = JSON.parse(out);
assert.deepEqual(parsed.spans, [
{ start: 0, end: 1.2 },
{ start: 1.8, end: 2.1 },
]);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
function word(id, text, start, end) {
return { id, text, start, end };
}
+70
View File
@@ -0,0 +1,70 @@
import { writeFileSync, copyFileSync, mkdirSync } from "node:fs";
import { dirname } from "node:path";
// ponytail: bound the download so a hostile/runaway URL can't fill the disk.
// 256MB covers any real media asset; raise if 4K video sources ever exceed it.
const MAX_FREEZE_BYTES = 256 * 1024 * 1024;
export async function freezeUrl(url, destPath) {
const where = String(url).slice(0, 80);
const res = await fetch(url);
if (!res.ok) throw new Error(`freeze failed: HTTP ${res.status} for ${where}`);
// Fail fast on an advertised oversize body before reading a single byte.
const declared = Number(res.headers.get("content-length"));
if (declared > MAX_FREEZE_BYTES)
throw new Error(
`freeze failed: ${declared} bytes exceeds ${MAX_FREEZE_BYTES} cap for ${where}`,
);
// Stream and abort once the cap is crossed, so a lying/chunked hostile URL
// can't buffer the whole payload into memory before the check (M1).
const chunks = [];
let total = 0;
for await (const chunk of res.body) {
total += chunk.length;
if (total > MAX_FREEZE_BYTES)
throw new Error(`freeze failed: stream exceeds ${MAX_FREEZE_BYTES} cap for ${where}`);
chunks.push(chunk);
}
if (total === 0) throw new Error(`freeze failed: empty response for ${where}`);
mkdirSync(dirname(destPath), { recursive: true });
writeFileSync(destPath, Buffer.concat(chunks, total));
return total;
}
export function freezeLocalFile(srcPath, destPath) {
mkdirSync(dirname(destPath), { recursive: true });
copyFileSync(srcPath, destPath);
}
// Ingest accepts a DIRECT public media URL only — not a platform page. yt-dlp is
// deliberately out (cloud IPs get blocked, and it's brittle); the supported case
// is "user points at their own file or a direct asset link". A direct URL is a
// non-platform host whose path ends in a known media extension.
const PLATFORM_HOSTS =
/(^|\.)(youtube\.com|youtu\.be|vimeo\.com|tiktok\.com|instagram\.com|twitter\.com|x\.com|facebook\.com|dailymotion\.com)$/i;
const MEDIA_EXT = /\.(mp3|wav|m4a|aac|ogg|flac|mp4|mov|webm|mkv|png|jpe?g|webp|gif|svg|avif)$/i;
// SSRF guard (m11): a user-supplied --from URL must not point at the local host
// or a private network. Blocks loopback/localhost, RFC1918, link-local, and the
// IPv6 equivalents on the literal hostname.
// ponytail: literal-host check only; a DNS name that *resolves* to a private IP
// (rebinding) still passes — add resolve-then-check if --from ever fetches from
// untrusted hostnames at scale.
const PRIVATE_HOST =
/^(localhost|.*\.local|.*\.internal|127\.|10\.|0\.|169\.254\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.|\[?(::1|::ffff:127\.|f[cd][0-9a-f]{2}:|fe80:))/i;
export function isDirectMediaUrl(u) {
let url;
try {
url = new URL(u);
} catch {
return false;
}
if (url.protocol !== "http:" && url.protocol !== "https:") return false;
if (PLATFORM_HOSTS.test(url.hostname)) return false;
if (PRIVATE_HOST.test(url.hostname)) return false;
return MEDIA_EXT.test(url.pathname);
}
@@ -0,0 +1,46 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { isDirectMediaUrl } from "./freeze.mjs";
test("accepts direct public media URLs", () => {
assert.equal(isDirectMediaUrl("https://cdn.example.com/clip.mp4"), true);
assert.equal(isDirectMediaUrl("https://example.com/a/b/track.mp3"), true);
assert.equal(isDirectMediaUrl("http://example.com/logo.svg"), true);
});
test("rejects platform pages (no yt-dlp)", () => {
assert.equal(isDirectMediaUrl("https://www.youtube.com/watch?v=abc"), false);
assert.equal(isDirectMediaUrl("https://youtu.be/abc"), false);
assert.equal(isDirectMediaUrl("https://vimeo.com/12345"), false);
assert.equal(isDirectMediaUrl("https://x.com/u/status/1"), false);
});
test("rejects non-direct / non-media URLs", () => {
assert.equal(isDirectMediaUrl("https://example.com/page"), false, "no media extension");
assert.equal(isDirectMediaUrl("ftp://example.com/a.mp4"), false, "non-http(s)");
assert.equal(isDirectMediaUrl("not a url"), false);
});
test("rejects local / private hosts (SSRF guard, m11)", () => {
for (const u of [
"http://localhost/a.mp4",
"http://127.0.0.1/a.mp4",
"http://127.1.2.3/a.mp4",
"http://0.0.0.0/a.mp4",
"http://10.0.0.5/a.mp4",
"http://192.168.1.1/a.mp4",
"http://172.16.0.1/a.mp4",
"http://172.31.255.255/a.mp4",
"http://169.254.169.254/a.mp4", // cloud metadata endpoint
"http://printer.local/a.mp4",
"http://svc.internal/a.mp4",
"http://[::1]/a.mp4",
"http://[fe80::1]/a.mp4",
"http://[fd00::1]/a.mp4",
]) {
assert.equal(isDirectMediaUrl(u), false, `should block ${u}`);
}
// A public host that merely starts with similar digits is still allowed.
assert.equal(isDirectMediaUrl("https://172.40.0.1/a.mp4"), true, "172.40 is public");
assert.equal(isDirectMediaUrl("https://11.example.com/a.mp4"), true);
});
@@ -0,0 +1,165 @@
import { execFileSync } from "node:child_process";
import { basename, extname } from "node:path";
const IMAGE_EXT = new Set([".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp", ".tif", ".tiff"]);
const SAMPLE_FRAMES = 5;
// A long HD clip on slow storage can exceed the default 15s signalstats window;
// override without a code change via HYPERFRAMES_ANALYZE_TIMEOUT_MS.
const SIGNALSTATS_TIMEOUT_MS = Number(process.env.HYPERFRAMES_ANALYZE_TIMEOUT_MS) || 15000;
const ADJUST_LIMITS = {
exposure: { min: -2, max: 2 },
contrast: { min: -1, max: 1 },
highlights: { min: -1, max: 1 },
shadows: { min: -1, max: 1 },
whites: { min: -1, max: 1 },
blacks: { min: -1, max: 1 },
temperature: { min: -1, max: 1 },
tint: { min: -1, max: 1 },
vibrance: { min: -1, max: 1 },
saturation: { min: -1, max: 1 },
};
function clamp(value, key) {
const limit = ADJUST_LIMITS[key];
if (!Number.isFinite(value)) return 0;
return Math.min(limit.max, Math.max(limit.min, value));
}
function round(value) {
return Math.round(value * 1000) / 1000;
}
function avg(values) {
if (values.length === 0) return 0;
return values.reduce((sum, value) => sum + value, 0) / values.length;
}
function probeDuration(mediaPath) {
try {
const raw = execFileSync(
"ffprobe",
["-v", "quiet", "-print_format", "json", "-show_format", mediaPath],
{ encoding: "utf8", timeout: 5000 },
);
const parsed = JSON.parse(raw);
const duration = Number(parsed.format?.duration);
return Number.isFinite(duration) && duration > 0 ? duration : null;
} catch {
return null;
}
}
function filterFor(mediaPath) {
const ext = extname(mediaPath).toLowerCase();
if (IMAGE_EXT.has(ext)) return "signalstats,metadata=print:file=-";
const duration = probeDuration(mediaPath);
if (!duration || duration <= 1) return "signalstats,metadata=print:file=-";
const fps = Math.max(0.1, Math.min(2, SAMPLE_FRAMES / duration));
return `fps=${fps.toFixed(4)},signalstats,metadata=print:file=-`;
}
function parseSignalStats(raw) {
const frames = [];
let current = null;
for (const line of String(raw).split(/\r?\n/)) {
const frameMatch = line.match(/^frame:/);
if (frameMatch) {
if (current) frames.push(current);
current = {};
continue;
}
const match = line.match(/lavfi\.signalstats\.([A-Z]+)=([+-]?(?:\d+(?:\.\d+)?|\.\d+))/);
if (!match) continue;
if (!current) current = {};
current[match[1]] = Number(match[2]);
}
if (current) frames.push(current);
const complete = frames.filter(
(frame) =>
Number.isFinite(frame.YMIN) &&
Number.isFinite(frame.YMAX) &&
Number.isFinite(frame.YAVG) &&
Number.isFinite(frame.UAVG) &&
Number.isFinite(frame.VAVG),
);
if (complete.length === 0) {
throw new Error("no signalstats frames found");
}
return {
frames: complete.length,
yMin: Math.min(...complete.map((frame) => frame.YMIN)),
yMax: Math.max(...complete.map((frame) => frame.YMAX)),
yAvg: avg(complete.map((frame) => frame.YAVG)),
uAvg: avg(complete.map((frame) => frame.UAVG)),
vAvg: avg(complete.map((frame) => frame.VAVG)),
};
}
export function statsToAdjust(stats) {
const yMin = Number(stats.yMin);
const yMax = Number(stats.yMax);
const yAvg = Number(stats.yAvg);
const uAvg = Number(stats.uAvg);
const vAvg = Number(stats.vAvg);
const spread = (yMax - yMin) / 255;
const normalizedAvg = yAvg / 255;
const exposure = clamp((0.45 - normalizedAvg) * 1.8, "exposure");
const contrast = clamp((0.42 - spread) * 0.9, "contrast");
const whites =
yMax > 230 ? clamp(-((yMax - 230) / 40 + Math.max(0, normalizedAvg - 0.74)), "whites") : 0;
const blacks = yMin < 12 ? clamp((12 - yMin) / 80, "blacks") : 0;
const chromaWarmth = (vAvg - 128 + (128 - uAvg)) / 128;
const temperature = clamp(-chromaWarmth * 0.7, "temperature");
const tint = clamp(-(uAvg - 128 + (vAvg - 128)) / 256, "tint");
return {
adjust: {
exposure: round(exposure),
contrast: round(contrast),
blacks: round(blacks),
whites: round(whites),
temperature: round(temperature),
tint: round(tint),
},
measured: {
frames: Number(stats.frames ?? 1),
yMin: round(yMin),
yMax: round(yMax),
yAvg: round(yAvg),
uAvg: round(uAvg),
vAvg: round(vAvg),
},
};
}
export function analyzeMediaGrade(mediaPath) {
try {
const raw = execFileSync(
"ffmpeg",
[
"-hide_banner",
"-nostdin",
"-v",
"error",
"-i",
mediaPath,
"-vf",
filterFor(mediaPath),
"-frames:v",
String(SAMPLE_FRAMES),
"-f",
"null",
"-",
],
{ encoding: "utf8", timeout: SIGNALSTATS_TIMEOUT_MS, stdio: ["ignore", "pipe", "pipe"] },
);
return statsToAdjust(parseSignalStats(raw));
} catch (err) {
throw new Error(`grade analysis failed for ${mediaPath}: ${err.message}`);
}
}
export function formatMeasuredNote(mediaPath, measured) {
return `media-use: measured ${basename(mediaPath)}: frames=${measured.frames}, YMIN=${measured.yMin}, YMAX=${measured.yMax}, YAVG=${measured.yAvg}, UAVG=${measured.uAvg}, VAVG=${measured.vAvg}; adjust is a starting suggestion`;
}
@@ -0,0 +1,155 @@
import { strict as assert } from "node:assert";
import { execFileSync, spawnSync } from "node:child_process";
import { existsSync, mkdtempSync, rmSync, unlinkSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { test } from "node:test";
import { analyzeMediaGrade, formatMeasuredNote, statsToAdjust } from "./grade-analyzer.mjs";
// The "Test: skills" CI job runs bare `node --test` with no ffmpeg on PATH (by
// design — skills tests are meant to be node-builtin-only). Tests that shell to
// ffmpeg skip there and run wherever ffmpeg is present (locally, dev).
const FFMPEG_SKIP =
spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0
? false
: "ffmpeg not on PATH";
const ADJUST_LIMITS = {
exposure: { min: -2, max: 2 },
contrast: { min: -1, max: 1 },
highlights: { min: -1, max: 1 },
shadows: { min: -1, max: 1 },
whites: { min: -1, max: 1 },
blacks: { min: -1, max: 1 },
temperature: { min: -1, max: 1 },
tint: { min: -1, max: 1 },
vibrance: { min: -1, max: 1 },
saturation: { min: -1, max: 1 },
};
function makeFrame(dir, name, color) {
const out = join(dir, name);
execFileSync(
"ffmpeg",
[
"-hide_banner",
"-loglevel",
"error",
"-f",
"lavfi",
"-i",
`color=c=${color}:s=64x64`,
"-frames:v",
"1",
"-y",
out,
],
{ stdio: "pipe" },
);
return out;
}
function assertWithinLimits(adjust) {
for (const [key, value] of Object.entries(adjust)) {
const limit = ADJUST_LIMITS[key];
assert.ok(limit, `unexpected adjust key ${key}`);
assert.ok(value >= limit.min && value <= limit.max, `${key} out of range: ${value}`);
}
}
test("under-exposed synthetic frame suggests positive exposure", { skip: FFMPEG_SKIP }, () => {
const dir = mkdtempSync(join(tmpdir(), "mu-grade-under-"));
try {
const file = makeFrame(dir, "under.png", "0x202020");
const { adjust, measured } = analyzeMediaGrade(file);
assert.ok(measured.frames >= 1);
assert.ok(adjust.exposure > 0, `expected positive exposure, got ${adjust.exposure}`);
assertWithinLimits(adjust);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("over-exposed synthetic frame pulls exposure and whites down", { skip: FFMPEG_SKIP }, () => {
const dir = mkdtempSync(join(tmpdir(), "mu-grade-over-"));
try {
const file = makeFrame(dir, "over.png", "white");
const { adjust } = analyzeMediaGrade(file);
assert.ok(adjust.exposure < 0, `expected negative exposure, got ${adjust.exposure}`);
assert.ok(adjust.whites < 0, `expected negative whites, got ${adjust.whites}`);
assertWithinLimits(adjust);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test(
"warm-cast synthetic frame suggests negative temperature correction",
{ skip: FFMPEG_SKIP },
() => {
const dir = mkdtempSync(join(tmpdir(), "mu-grade-warm-"));
try {
const file = makeFrame(dir, "warm.png", "orange");
const { adjust } = analyzeMediaGrade(file);
assert.ok(adjust.temperature < 0, `expected cooling correction, got ${adjust.temperature}`);
assertWithinLimits(adjust);
} finally {
rmSync(dir, { recursive: true, force: true });
}
},
);
test("low-spread stats suggest positive contrast", () => {
const { adjust } = statsToAdjust({
frames: 1,
yMin: 104,
yMax: 116,
yAvg: 110,
uAvg: 128,
vAvg: 128,
});
assert.ok(adjust.contrast > 0, `expected positive contrast, got ${adjust.contrast}`);
assertWithinLimits(adjust);
});
test("malformed media fails cleanly", () => {
assert.throws(
() => analyzeMediaGrade(join(tmpdir(), "does-not-exist.png")),
/grade analysis failed/,
);
});
test(
"media path with shell metacharacters is passed as argv, not a shell string",
{
skip: FFMPEG_SKIP,
},
() => {
const dir = mkdtempSync(join(tmpdir(), "mu-grade-shell-"));
const sentinel = join(process.cwd(), "mu-grade-shell-sentinel.png");
try {
if (existsSync(sentinel)) unlinkSync(sentinel);
const file = makeFrame(dir, "frame; touch mu-grade-shell-sentinel.png", "orange");
const result = analyzeMediaGrade(file);
assert.ok(result.measured.frames >= 1);
assert.equal(existsSync(sentinel), false);
} finally {
if (existsSync(sentinel)) unlinkSync(sentinel);
rmSync(dir, { recursive: true, force: true });
}
},
);
test("measured note is a stderr-safe single-line summary", () => {
const note = formatMeasuredNote("/tmp/frame.png", {
frames: 1,
yMin: 10,
yMax: 240,
yAvg: 80,
uAvg: 120,
vAvg: 140,
});
assert.match(note, /^media-use: measured /);
assert.match(note, /YAVG=80/);
assert.equal(note.includes("\n"), false);
});
+162
View File
@@ -0,0 +1,162 @@
import { track } from "./telemetry.mjs";
// v0.3.0 is the first CLI that can use an OAuth session; v0.1.x/0.2.x reject it
// ("heygen-cli can't use OAuth yet"), and OAuth is what the free-usage path
// needs — so anything below this can't authenticate for free usage at all.
export const HEYGEN_MIN_VERSION = "0.3.0";
// Free-usage path is OAuth (`--oauth` → subscription/free credits); `--api-key`
// bills API credits, so the onboarding steers to OAuth.
export const HEYGEN_INSTALL_COMMAND =
"curl -fsSL https://static.heygen.ai/cli/install.sh | bash && heygen auth login --oauth";
export const HEYGEN_AUTH_COMMAND = "heygen auth login --oauth";
export const HEYGEN_UPDATE_COMMAND = "heygen update";
export const HEYGEN_NOT_FOUND_MESSAGE = `media-use: heygen CLI not found — it's the free path for bgm/image/voice/avatar-video. Install: ${HEYGEN_INSTALL_COMMAND}`;
export const HEYGEN_NOT_AUTHENTICATED_MESSAGE = `media-use: heygen CLI not authenticated (free usage) — run: ${HEYGEN_AUTH_COMMAND}`;
export const HEYGEN_OUTDATED_MESSAGE = `media-use: heygen CLI is outdated — run: ${HEYGEN_UPDATE_COMMAND} (need >= v${HEYGEN_MIN_VERSION})`;
const ACTIONABLE_MESSAGES = new Set([
HEYGEN_NOT_FOUND_MESSAGE,
HEYGEN_NOT_AUTHENTICATED_MESSAGE,
HEYGEN_OUTDATED_MESSAGE,
]);
export function classifyHeygenError(err) {
return classifyHeygenErrorResult(err).message;
}
export function classifyHeygenErrorCode(err) {
return classifyHeygenErrorResult(err).code;
}
function classifyHeygenErrorResult(err) {
const detail = heygenErrorDetail(err);
const text = [err?.stderr, err?.stdout, err?.message, detail]
.map((value) => textOf(value))
.filter(Boolean)
.join("\n");
const lower = text.toLowerCase();
// Only ENOENT (spawn of a missing binary) or a shell's "command not found"
// mean the CLI itself is absent. A bare "not found" would misfire on the CLI's
// own resource errors (e.g. a stale voiceId → "voice not found"), whose message
// embeds the `heygen ...` command line — sending users to reinstall a CLI they
// just ran successfully. Keep this narrow.
if (err?.code === "ENOENT" || lower.includes("command not found")) {
return { code: "not_found", message: HEYGEN_NOT_FOUND_MESSAGE };
}
if (
lower.includes("unauthorized") ||
lower.includes("unauthenticated") ||
// \b401\b, not a bare "401" substring — otherwise request IDs (req-401abc),
// URLs, and retry-after headers would misclassify as an auth failure.
/\b401\b/.test(lower) ||
lower.includes("not logged in") ||
lower.includes("no api key") ||
lower.includes("missing api key") ||
lower.includes("invalid api key") ||
lower.includes("login required") ||
lower.includes("auth required") ||
lower.includes("authentication required")
) {
return { code: "not_authenticated", message: HEYGEN_NOT_AUTHENTICATED_MESSAGE };
}
const version = firstSemver(text);
if (version && versionLessThan(version, HEYGEN_MIN_VERSION)) {
return { code: "outdated", message: HEYGEN_OUTDATED_MESSAGE };
}
if (
lower.includes("rate limit") ||
lower.includes("quota") ||
lower.includes("insufficient credit") ||
lower.includes("too many requests") ||
lower.includes("throttled") ||
/\b429\b/.test(lower)
) {
return { code: "rate_limited", message: detail };
}
return { code: "other", message: detail };
}
// reportHeygenFailure's callers (voice-provider.mjs, heygen-search.mjs) are
// synchronous and several layers below the CLI's process.exit() calls, so
// they can't await this tracking call themselves. Stash each attempt's
// promise here so a caller closer to exit (resolve.mjs) can join it first —
// same "awaited so a short-lived run flushes it" discipline telemetry.mjs's
// track() already documents, just reachable from a sync call site.
const pendingFailureTracking = new Set();
// resolve.mjs is a single-shot CLI (one resolve per process), so one shared
// consume-once slot is sufficient. If resolve becomes an in-process/concurrent
// API, move this state into a per-resolve context before reusing that path.
let pendingRemediation = null;
export function consumeHeygenRemediation() {
const remediation = pendingRemediation;
pendingRemediation = null;
return remediation;
}
export function reportHeygenFailure(err, context, trackEvent = track) {
const { code, message } = classifyHeygenErrorResult(err);
if (code === "not_found" || code === "outdated") {
pendingRemediation = { code, message };
}
if (ACTIONABLE_MESSAGES.has(message)) {
console.error(message);
} else {
console.error(`media-use: \`${context}\` failed: ${message}`);
}
try {
const tracked = Promise.resolve(
trackEvent("media_use_provider_error", { provider: "heygen", reason: code }),
).catch(() => {});
pendingFailureTracking.add(tracked);
void tracked.finally(() => pendingFailureTracking.delete(tracked));
return tracked;
} catch {
// Telemetry must never affect the provider failure path.
return Promise.resolve();
}
}
// Awaits every provider-error track fired since the last flush, so a caller
// about to process.exit() doesn't orphan one mid-request (both are separate,
// non-keepalive HTTP connections with no ordering guarantee otherwise).
// Never rejects: each tracked promise already swallows its own failure.
export async function flushHeygenFailureTracking() {
if (pendingFailureTracking.size === 0) return;
await Promise.all(pendingFailureTracking);
}
export function firstSemver(text) {
const match = String(text || "").match(/\bv?(\d+)\.(\d+)\.(\d+)\b/);
return match ? `${match[1]}.${match[2]}.${match[3]}` : null;
}
export function versionLessThan(version, minimum) {
const left = versionParts(version);
const right = versionParts(minimum);
if (!left || !right) return false;
for (let i = 0; i < 3; i++) {
if (left[i] < right[i]) return true;
if (left[i] > right[i]) return false;
}
return false;
}
function heygenErrorDetail(err) {
return textOf(err?.stderr) || textOf(err?.stdout) || err?.message || String(err);
}
function textOf(value) {
return value == null ? "" : String(value).trim();
}
function versionParts(version) {
const match = String(version || "").match(/^v?(\d+)\.(\d+)\.(\d+)$/);
return match ? match.slice(1).map((part) => Number.parseInt(part, 10)) : null;
}
@@ -0,0 +1,301 @@
import { strict as assert } from "node:assert";
import { spawnSync } from "node:child_process";
import { test } from "node:test";
import {
classifyHeygenError,
classifyHeygenErrorCode,
consumeHeygenRemediation,
flushHeygenFailureTracking,
HEYGEN_NOT_AUTHENTICATED_MESSAGE,
HEYGEN_NOT_FOUND_MESSAGE,
HEYGEN_OUTDATED_MESSAGE,
reportHeygenFailure,
} from "./heygen-cli.mjs";
function captureFailureReport(err, context, trackEvent) {
const originalError = console.error;
const stderrCalls = [];
console.error = (...args) => stderrCalls.push(args);
try {
if (trackEvent) {
reportHeygenFailure(err, context, trackEvent);
} else {
reportHeygenFailure(err, context);
}
} finally {
console.error = originalError;
}
return stderrCalls;
}
test("classifies ENOENT-style missing heygen errors with install instructions", () => {
const message = classifyHeygenError({ code: "ENOENT", message: "spawn heygen ENOENT" });
assert.equal(message, HEYGEN_NOT_FOUND_MESSAGE);
});
test("classifies auth failures with login instructions", () => {
const message = classifyHeygenError({ stderr: Buffer.from("Error: not logged in") });
assert.equal(message, HEYGEN_NOT_AUTHENTICATED_MESSAGE);
});
test("classifies a real 401 as auth, but not a bare 401 substring in prose", () => {
assert.equal(
classifyHeygenError({ stderr: Buffer.from("HTTP 401 Unauthorized") }),
HEYGEN_NOT_AUTHENTICATED_MESSAGE,
);
// A request id that merely contains "401" must NOT read as an auth failure.
const noise = classifyHeygenError({ stderr: Buffer.from("upload failed (request req-401abc)") });
assert.notEqual(noise, HEYGEN_NOT_AUTHENTICATED_MESSAGE);
});
test("classifies old heygen versions with update instructions", () => {
const message = classifyHeygenError({
stderr: Buffer.from("heygen v0.1.5 does not support --headers"),
});
assert.equal(message, HEYGEN_OUTDATED_MESSAGE);
});
test("does not misclassify a resource 'not found' error as a missing CLI", () => {
// A stale voiceId makes `heygen voice speech create` fail with "voice not
// found"; the error message embeds the `heygen ...` command line. This must
// pass through as detail, not send the user to reinstall a working CLI.
const message = classifyHeygenError({
stderr: Buffer.from("Error: voice not found (id: stale-123)"),
message: "Command failed: heygen voice speech create --voice stale-123",
});
assert.notEqual(message, HEYGEN_NOT_FOUND_MESSAGE);
assert.equal(message, "Error: voice not found (id: stale-123)");
});
test("classifies a shell 'command not found' as a missing CLI", () => {
const message = classifyHeygenError({ stderr: Buffer.from("bash: heygen: command not found") });
assert.equal(message, HEYGEN_NOT_FOUND_MESSAGE);
});
test("passes through unrelated errors", () => {
const message = classifyHeygenError({
stderr: Buffer.from("rate limit exceeded"),
message: "Command failed",
});
assert.equal(message, "rate limit exceeded");
});
test("classifies existing HeyGen failures with stable reason codes", () => {
assert.equal(classifyHeygenErrorCode({ code: "ENOENT" }), "not_found");
assert.equal(
classifyHeygenErrorCode({ stderr: Buffer.from("HTTP 401 Unauthorized") }),
"not_authenticated",
);
assert.equal(
classifyHeygenErrorCode({ stderr: Buffer.from("heygen v0.1.5 is unsupported") }),
"outdated",
);
assert.equal(classifyHeygenErrorCode({ stderr: Buffer.from("provider unavailable") }), "other");
});
test("classifies rate-limit text case-insensitively", () => {
assert.equal(
classifyHeygenErrorCode({ stderr: Buffer.from("RATE LIMIT exceeded") }),
"rate_limited",
);
});
test("classifies quota and insufficient-credit errors as rate limited", () => {
for (const detail of ["Quota exhausted", "INSUFFICIENT CREDIT remaining"]) {
assert.equal(classifyHeygenErrorCode({ stderr: Buffer.from(detail) }), "rate_limited");
}
});
test("classifies the literal 429 reason phrase and throttling language as rate limited", () => {
for (const detail of ["Too Many Requests", "Error: throttled by upstream, retry later"]) {
assert.equal(classifyHeygenErrorCode({ stderr: Buffer.from(detail) }), "rate_limited");
}
});
test("does not misclassify unrelated errors that share a word with the new phrasing", () => {
// Shares "too many" with "too many requests" but is a distinct failure (fd
// exhaustion, not a rate limit) — the match must require the full phrase.
assert.equal(
classifyHeygenErrorCode({ stderr: Buffer.from("Too many open file descriptors") }),
"other",
);
});
test("classifies a bare 429 as rate limited without matching request IDs", () => {
assert.equal(
classifyHeygenErrorCode({ stderr: Buffer.from("HTTP 429 Too Many Requests") }),
"rate_limited",
);
assert.equal(
classifyHeygenErrorCode({ stderr: Buffer.from("request req-429abc failed") }),
"other",
);
});
test("tracks not-found failures without changing actionable output", () => {
const trackingCalls = [];
const stderrCalls = captureFailureReport({ code: "ENOENT" }, "heygen asset search", (...args) =>
trackingCalls.push(args),
);
assert.deepEqual(stderrCalls, [[HEYGEN_NOT_FOUND_MESSAGE]]);
assert.deepEqual(trackingCalls, [
["media_use_provider_error", { provider: "heygen", reason: "not_found" }],
]);
});
test("records missing and outdated CLI remediation once", () => {
consumeHeygenRemediation();
captureFailureReport({ code: "ENOENT" }, "heygen audio sounds list", () => {});
assert.deepEqual(consumeHeygenRemediation(), {
code: "not_found",
message: HEYGEN_NOT_FOUND_MESSAGE,
});
assert.equal(consumeHeygenRemediation(), null);
captureFailureReport(
{ stderr: Buffer.from("heygen v0.1.5 does not support --headers") },
"heygen audio sounds list",
() => {},
);
assert.deepEqual(consumeHeygenRemediation(), {
code: "outdated",
message: HEYGEN_OUTDATED_MESSAGE,
});
assert.equal(consumeHeygenRemediation(), null);
});
test("does not record non-install remediation", () => {
consumeHeygenRemediation();
for (const error of [
{ stderr: Buffer.from("HTTP 401 Unauthorized") },
{ stderr: Buffer.from("quota exhausted") },
{ stderr: Buffer.from("provider unavailable") },
]) {
captureFailureReport(error, "heygen audio sounds list", () => {});
assert.equal(consumeHeygenRemediation(), null);
}
});
test("tracks generic failures without including raw detail", () => {
const trackingCalls = [];
const stderrCalls = captureFailureReport(
{ stderr: Buffer.from("private provider detail") },
"heygen asset search",
(...args) => trackingCalls.push(args),
);
assert.deepEqual(stderrCalls, [
["media-use: `heygen asset search` failed: private provider detail"],
]);
assert.deepEqual(trackingCalls, [
["media_use_provider_error", { provider: "heygen", reason: "other" }],
]);
});
test("keeps failure output observable when telemetry is opted out", () => {
const previousOptOut = process.env.HYPERFRAMES_NO_TELEMETRY;
process.env.HYPERFRAMES_NO_TELEMETRY = "1";
try {
const stderrCalls = captureFailureReport({ code: "ENOENT" }, "heygen asset search");
assert.deepEqual(stderrCalls, [[HEYGEN_NOT_FOUND_MESSAGE]]);
} finally {
if (previousOptOut === undefined) {
delete process.env.HYPERFRAMES_NO_TELEMETRY;
} else {
process.env.HYPERFRAMES_NO_TELEMETRY = previousOptOut;
}
}
});
test("keeps failure output observable when tracking throws synchronously", () => {
const originalError = console.error;
const stderrCalls = [];
let thrown;
console.error = (...args) => stderrCalls.push(args);
try {
try {
reportHeygenFailure({ code: "ENOENT" }, "heygen asset search", () => {
throw new Error("tracking failed");
});
} catch (err) {
thrown = err;
}
} finally {
console.error = originalError;
}
assert.deepEqual(stderrCalls, [[HEYGEN_NOT_FOUND_MESSAGE]]);
assert.equal(thrown, undefined);
});
test("does not leave rejected tracking promises unhandled", () => {
const moduleUrl = new URL("./heygen-cli.mjs", import.meta.url).href;
const script = `
import { reportHeygenFailure } from ${JSON.stringify(moduleUrl)};
reportHeygenFailure(
{ stderr: "provider unavailable" },
"heygen asset search",
() => Promise.reject(new Error("tracking failed")),
);
await new Promise((resolve) => setImmediate(resolve));
`;
const child = spawnSync(
process.execPath,
["--unhandled-rejections=strict", "--input-type=module", "--eval", script],
{ encoding: "utf8", timeout: 5000 },
);
assert.equal(child.error, undefined);
assert.equal(child.signal, null);
assert.equal(child.status, 0, child.stderr);
assert.equal(child.stderr, "media-use: `heygen asset search` failed: provider unavailable\n");
});
test("flushHeygenFailureTracking waits for a pending report before resolving", async () => {
const events = [];
let releaseTrack;
const gate = new Promise((resolve) => {
releaseTrack = resolve;
});
// Mirrors the real call sites (voice-provider.mjs, heygen-search.mjs):
// fire-and-forget, the return value is never awaited by the caller.
reportHeygenFailure({ code: "ENOENT" }, "heygen voice speech", () =>
gate.then(() => {
events.push("track-settled");
}),
);
const flushed = flushHeygenFailureTracking().then(() => {
events.push("flush-resolved");
});
// Let several pending microtasks drain before releasing the gate, so this
// proves flush is genuinely still waiting on the tracked promise -- not
// merely that it hasn't had a tick yet.
await Promise.resolve();
await Promise.resolve();
await Promise.resolve();
assert.deepEqual(events, [], "flush must not resolve while the tracked promise is still pending");
releaseTrack();
await flushed;
assert.deepEqual(
events,
["track-settled", "flush-resolved"],
"flush must resolve only after the pending track settles, in that order",
);
});
test("flushHeygenFailureTracking resolves immediately when nothing is pending", async () => {
await flushHeygenFailureTracking();
});
@@ -0,0 +1,51 @@
import { execFileSync } from "node:child_process";
import { reportHeygenFailure } from "./heygen-cli.mjs";
export function heygenSearch(subcommand, query, { type, limit = 5, minScore } = {}) {
// execFileSync with an argv array (no shell), so query/type/etc. are passed as
// literal arguments — no quoting tricks, no command injection. subcommand is a
// hardcoded multi-word string (e.g. "audio sounds list"), split into tokens.
// Tag the caller via the CLI's allowlisted attribution header (heygen >= v0.3.0).
const args = [
"--headers",
"X-HeyGen-Client-Source: media-use",
...subcommand.split(" "),
"--query",
query,
];
if (type) args.push("--type", type);
args.push("--limit", String(limit));
// Server-side score floor. Honored by `audio sounds list`; the `asset search`
// backend rejects it, so only audio providers pass minScore (see image-provider).
if (minScore != null) args.push("--min-score", String(minScore));
let out;
try {
out = execFileSync("heygen", args, {
encoding: "utf8",
timeout: 15000,
stdio: ["pipe", "pipe", "pipe"],
});
} catch (err) {
// Don't swallow a broken command / auth failure as "no results" — that turns
// a typo or expired key into a silent dead end. Surface it, then give up.
reportHeygenFailure(err, `heygen ${subcommand}`);
return null;
}
let parsed;
try {
parsed = JSON.parse(out);
} catch {
console.error(`media-use: \`heygen ${subcommand}\` returned non-JSON output`);
return null;
}
if (parsed?.error) {
const e = parsed.error;
console.error(`media-use: \`heygen ${subcommand}\` error: ${e.message ?? JSON.stringify(e)}`);
return null;
}
const data = parsed?.data;
return Array.isArray(data) && data.length > 0 ? data : null;
}
@@ -0,0 +1,44 @@
import { heygenSearch } from "./heygen-search.mjs";
export const imageProvider = {
async search(intent) {
const results = heygenSearch("asset search", intent, { type: "image" });
if (!results) return null;
const best = results[0];
return {
url: best.url,
source: "search",
// ext derived from the asset URL by resolve.mjs (.jpg/.png/.webp)
metadata: {
description: intent,
width: best.width || null,
height: best.height || null,
transparent: best.is_transparent || false,
provider: "heygen.asset.search",
provenance: { asset_id: best.id, score: best.score },
},
};
},
};
export const iconProvider = {
async search(intent) {
// No minScore: the `asset search` backend rejects --min-score and returns no score field.
const results = heygenSearch("asset search", intent, { type: "icon" });
if (!results) return null;
const best = results[0];
return {
url: best.url,
source: "search",
// ext derived from the asset URL by resolve.mjs — catalog icons are .png, not .svg
metadata: {
description: intent,
width: best.width || null,
height: best.height || null,
transparent: best.is_transparent ?? true,
provider: "heygen.asset.search",
provenance: { asset_id: best.id, score: best.score, type: "icon" },
},
};
},
};
@@ -0,0 +1,63 @@
import { writeFileSync, mkdirSync } from "node:fs";
import { dirname } from "node:path";
import { readManifest, indexPath } from "./manifest.mjs";
function pad(str, len) {
return String(str ?? "").padEnd(len);
}
function formatDur(record) {
if (record.duration == null) return "—";
return `${record.duration}s`;
}
function formatDims(record) {
if (record.width && record.height) return `${record.width}×${record.height}`;
if (record.type === "icon" && record.transparent) return "svg";
return "—";
}
export function generateIndexContent(records) {
const count = records.length;
const header = `# .media · ${count} asset${count === 1 ? "" : "s"}\n`;
if (count === 0) return header;
const cols = { id: 4, type: 5, dur: 4, dims: 5, path: 5, desc: 11 };
for (const r of records) {
cols.id = Math.max(cols.id, (r.id ?? "").length);
cols.type = Math.max(cols.type, (r.type ?? "").length);
cols.dur = Math.max(cols.dur, formatDur(r).length);
cols.dims = Math.max(cols.dims, formatDims(r).length);
cols.path = Math.max(cols.path, (r.path ?? "").length);
}
const heading =
pad("id", cols.id + 2) +
pad("type", cols.type + 2) +
pad("dur", cols.dur + 2) +
pad("dims", cols.dims + 2) +
pad("path", cols.path + 2) +
"description";
const lines = [header, heading];
for (const r of records) {
lines.push(
pad(r.id, cols.id + 2) +
pad(r.type, cols.type + 2) +
pad(formatDur(r), cols.dur + 2) +
pad(formatDims(r), cols.dims + 2) +
pad(r.path, cols.path + 2) +
(r.description ?? ""),
);
}
return lines.join("\n") + "\n";
}
export function regenerateIndex(projectDir) {
const records = readManifest(projectDir);
const content = generateIndexContent(records);
const p = indexPath(projectDir);
mkdirSync(dirname(p), { recursive: true });
writeFileSync(p, content);
return content;
}
@@ -0,0 +1,279 @@
// Declarative table of USER-INSTALLED local models, for the spec-gated fallback.
//
// These models run on the user's own machine for their own use; media-use
// recommends, spec-checks, and assists install; it does not bundle, redistribute,
// or sell them. Because nothing is redistributed, selection is purely by
// quality / size / spec-fit / word-timestamp support (there is deliberately NO
// license field gating availability).
//
// Tiers (`small`|`medium`|`large`|`xlarge`) are human labels; `needs.ramMB` is
// what selection actually gates on. selectModel() returns the best model that
// fits the machine's AVAILABLE RAM, best-first: by explicit `rank` when set
// (quality that is NOT size, e.g. ASR), else by RAM footprint (the quality
// proxy for generation). No fit -> recommend the CLI/cloud path.
//
// Picks reflect the 2026 research pass, verified live where noted.
export const CAPABILITIES = ["tts", "asr", "upscale", "videogen", "imagegen"];
const MODELS = {
tts: [
{
id: "kokoro",
tier: "medium",
sizeMB: 330,
needs: { ramMB: 2048, gpu: false },
wordTimestamps: "native",
install: "pip install kokoro",
invoke: "python -m kokoro --text {text} --voice {voice} --out {out}",
notes: "CPU, faster-than-realtime, native per-word timestamps. Default floor.",
},
{
id: "fish-speech",
tier: "large",
sizeMB: 1100,
needs: { ramMB: 16000, gpu: true, vramMB: 12000 },
wordTimestamps: "whisperx", // needs forced alignment (run ASR over output)
install: "pip install fish-speech",
invoke: "fish-speech synth --text {text} --ref {ref} --out {out}",
notes: "Expressive zero-shot voice cloning; meeting pick. WhisperX for word timing.",
},
],
asr: [
// Parakeet is BETTER than Whisper yet SMALLER (0.6B vs 1.5B), so quality is
// not size here: `rank` pins it ahead of whisper regardless of footprint.
// Open ASR Leaderboard avg WER: Parakeet ~6.05% vs whisper-large-v3 7.44%
// (~19% better); on NOISY test-other 4.73% vs 5.96%, and whisper-v3
// hallucinated to 308% WER on meetings where Parakeet held. 5-10x faster.
//
// Cohere Transcribe 2B tops the leaderboard (5.42%) and is nominally the most
// accurate, but its mlx-audio community MLX quants (4bit AND 8bit, with and
// without --language en) produced multilingual token-soup garbage AND ran
// 40-70x slower than Parakeet on a 24GB Mac (live-tested 2026-07). Excluded
// until the mlx-audio Cohere decoder stabilizes; Parakeet is the default.
{
id: "parakeet-mlx",
tier: "small",
rank: 0,
sizeMB: 2400,
needs: { ramMB: 4000, gpu: true },
wordTimestamps: "tokens", // sub-word tokens; merged to words by parakeet-words.mjs
repo: "mlx-community/parakeet-tdt-0.6b-v3",
install:
"uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx",
invoke:
"parakeet-mlx {audio} --model mlx-community/parakeet-tdt-0.6b-v3 --output-format json --output-dir {outdir}",
notes:
"NVIDIA Parakeet-TDT 0.6B via parakeet-mlx. VERIFIED on 24GB: accurate transcript, ~3s (cached model) for 8s audio, word timestamps drive transcript-cut. English + 25 European languages. Beats whisper.cpp on accuracy (6.05% vs 7.44% WER) AND speed (5-10x).",
},
{
id: "whisperx",
tier: "medium",
rank: 1,
sizeMB: 1500,
needs: { ramMB: 4096, gpu: false },
wordTimestamps: "native", // faster-whisper + wav2vec2 forced alignment
install: "pip install whisperx",
invoke: "whisperx {audio} --output_format json --out {out}",
notes:
"CPU-only fallback (no GPU): faster-whisper + wav2vec2 forced alignment, native word timestamps. The packaged `hyperframes transcribe` (whisper.cpp) is the zero-setup baseline below this.",
},
],
upscale: [
{
id: "real-esrgan",
tier: "medium",
sizeMB: 70,
needs: { ramMB: 2048, gpu: false },
wordTimestamps: false,
install: "brew install real-esrgan-ncnn-vulkan # or download the ncnn binary",
invoke: "realesrgan-ncnn-vulkan -i {in} -o {out} -s 4",
notes: "ncnn-vulkan binary, CPU-capable. GFPGAN for faces.",
},
{
id: "seedvr2",
tier: "large",
sizeMB: 6000,
needs: { ramMB: 24000, gpu: true, vramMB: 16000 },
wordTimestamps: false,
install: "pip install seedvr2",
invoke: "seedvr2 upscale --in {in} --out {out}",
notes: "Diffusion upscaler, GPU-only. Video2X for video.",
},
],
videogen: [
// 2026-07 X research pass + live verification on a 24GB M-series Mac.
// The Mac-local video story is LTX 2.3 on MLX via dgrauet/ltx-2-mlx (the
// pipeline these weights were converted for; also powers Phosphene).
// Wan 2.x MLX exists only as A14B conversions (too large for consumer
// unified memory); revisit when a 5B Wan MLX conversion lands.
// IMPORTANT: download the weights with a targeted include list first;
// pointing tools at the repo blind snapshot-downloads all 60 GB:
// hf download dgrauet/ltx-2.3-mlx-q4 --include \
// transformer-distilled-1.1.safetensors connector.safetensors \
// "vae_*.safetensors" audio_vae.safetensors vocoder.safetensors "*.json"
{
id: "ltx-2.3-mlx-q4",
tier: "medium",
sizeMB: 20000, // distilled subset; gemma-3-12b-4bit text encoder adds ~7GB
needs: { ramMB: 16384, gpu: true },
wordTimestamps: false,
install:
"git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras",
invoke:
"ltx-2-mlx generate --prompt {prompt} --distilled --low-ram --model dgrauet/ltx-2.3-mlx-q4 --width {w} --height {h} --frames {frames} --frame-rate 24 --output {out}",
notes:
"LTX 2.3 int4 on MLX. Verified on 24GB unified: 512x320 x 33 frames in ~19 min cold (incl. text-encoder download), t2v with audio. Dims must be multiples of 64. i2v, retake/extend, keyframe interpolation supported.",
},
{
id: "ltx-2.3-mlx-bf16",
tier: "large",
sizeMB: 45000,
needs: { ramMB: 32768, gpu: true },
wordTimestamps: false,
install:
"git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras",
invoke:
"ltx-2-mlx generate --prompt {prompt} --two-stage --model dgrauet/ltx-2.3-mlx-bf16 --width {w} --height {h} --frames {frames} --frame-rate 24 --output {out}",
notes:
"Full-precision two-stage pipeline (upstream production default). 32GB with --low-ram block streaming; 64-128GB Macs for long/HD runs (the 25s multi-scene spots seen in the wild).",
},
],
imagegen: [
// 2026-07 X research + live verification on a 24GB M-series Mac. mflux
// (FLUX-on-MLX) is the Mac-native runner; FLUX is the quality leader. Two
// hard-won findings baked into `needs.ramMB`:
// 1. The OFFICIAL FLUX repos are HF-gated (license wall). Point --path at a
// non-gated community 4-bit re-upload (self-contained, incl. VAE).
// 2. Without --low-ram, FLUX's T5-XXL text encoder + transformer blow past
// 24GB into swap: a 768x512 run took 90 MINUTES. With --low-ram (streams
// components from disk) the SAME machine did 512x512 in ~20s at 7.6GB
// free. So the medium tier's needs.ramMB is the streamed floor, not the
// resident footprint; the large tiers are the no-streaming thresholds.
// The runner resolves `repo` to a local snapshot (hf download) before --path;
// a bare repo id in --path breaks mlx unflatten.
{
id: "flux-schnell-mflux-q4",
tier: "medium",
sizeMB: 8700,
needs: { ramMB: 8000, gpu: true },
repo: "dhairyashil/FLUX.1-schnell-mflux-4bit",
wordTimestamps: false,
install: "uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6",
invoke:
"mflux-generate --model schnell --path {model_path} --low-ram --steps 4 --prompt {prompt} --width {w} --height {h} --seed {seed} --output {out}",
notes:
"FLUX.1 schnell int4. VERIFIED on 24GB (7.6GB free): --low-ram 512x512 in ~20s, photoreal. --low-ram is MANDATORY at this tier (streams to avoid swap). Few-step, fast.",
},
{
id: "flux2-klein-mflux-q4",
tier: "large",
sizeMB: 12000,
needs: { ramMB: 32000, gpu: true },
repo: "Runpod/FLUX.2-klein-4B-mflux-4bit",
wordTimestamps: false,
install: "uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux",
invoke:
"mflux-generate --base-model flux2-klein-4b --path {model_path} --steps 8 --prompt {prompt} --width {w} --height {h} --seed {seed} --output {out}",
notes:
"FLUX.2 Klein 4B int4 (most-downloaded mflux community repo). Newer, higher quality than schnell; full-resident (no streaming) so needs 32GB+ to stay fast. Needs mflux >= 0.18 for the flux2-klein base model.",
},
{
id: "qwen-image-mflux",
tier: "xlarge",
sizeMB: 40000,
needs: { ramMB: 64000, gpu: true },
repo: "Qwen/Qwen-Image",
wordTimestamps: false,
install: "uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux",
invoke:
"mflux-generate --base-model qwen --steps 20 --prompt {prompt} --width {w} --height {h} --seed {seed} --output {out}",
notes:
"Qwen-Image, top-tier quality. Heavy: 'several minutes' even on 128GB M4 Max, 'almost fried' a 32GB M4 Pro. 64GB+ only. Below that, the cloud upsell (codex) is faster and better.",
},
],
};
function tableFor(capability) {
const t = MODELS[capability];
if (!t) throw new Error(`unknown local-model capability: ${capability}`);
return t;
}
/** All local models for a capability. */
export function listModels(capability) {
return tableFor(capability).slice();
}
/** Does this machine meet a model's needs? Apple Silicon unified memory counts as VRAM. */
export function meetsSpecs(model, specs) {
const n = model.needs || {};
// Gate on AVAILABLE RAM when the probe reported it (the real budget with the
// OS + open apps resident); fall back to total RAM otherwise. Older specs
// objects (and unit fixtures) that only set ramMB keep working unchanged.
const budget = specs.availableRamMB ?? specs.ramMB;
if (n.ramMB && budget < n.ramMB) return false;
if (n.gpu && !specs.gpu?.present) return false;
if (n.vramMB) {
const vram = specs.gpu?.vramMB ?? 0;
if (vram < n.vramMB) return false;
}
return true;
}
// "Best model the machine can run" == best-first among those that fit. Ordering:
// 1. explicit `rank` (lower = better) when a model declares it. Needed where
// quality is NOT size: Parakeet-0.6B beats Whisper-large-1.5B at ASR, so
// footprint would pick the wrong one.
// 2. otherwise RAM footprint descending, the quality proxy for generation
// (a 40GB image model out-renders a 12GB one).
function rankedByPreference(table) {
return [...table].sort((a, b) => {
const ra = a.rank ?? Infinity;
const rb = b.rank ?? Infinity;
if (ra !== rb) return ra - rb;
return (b.needs?.ramMB ?? 0) - (a.needs?.ramMB ?? 0);
});
}
/**
* Pick the best local model the machine can run for a capability: the
* highest-footprint model that fits the available-RAM budget (and GPU/VRAM).
* `preferTier` pins the search to one tier (e.g. force a smaller/faster model).
* Returns `{ model, tier }`, or `{ recommend: "cli", reason }` when nothing fits.
*/
export function selectModel(capability, specs, { preferTier } = {}) {
const table = tableFor(capability);
const pool = preferTier ? table.filter((m) => m.tier === preferTier) : table;
for (const model of rankedByPreference(pool)) {
if (meetsSpecs(model, specs)) return { model, tier: model.tier };
}
const smallest = table.reduce((a, b) => (a.sizeMB <= b.sizeMB ? a : b));
return {
recommend: "cli",
reason: `machine does not meet specs for any local ${capability} model (smallest needs ~${smallest.needs.ramMB}MB RAM${smallest.needs.gpu ? " + GPU" : ""}); use the CLI path instead`,
};
}
/**
* Agent-facing ladder: every model for a capability, best-first, each flagged
* with whether it fits this machine and why. Lets the agent see the RAM-graded
* options and choose (e.g. trade the auto-picked best for a smaller/faster one,
* or step up to a cloud upsell) rather than only getting one auto-selection.
*/
export function describeModelLadder(capability, specs) {
const budget = specs.availableRamMB ?? specs.ramMB;
return rankedByPreference(tableFor(capability)).map((model) => {
const fits = meetsSpecs(model, specs);
return {
id: model.id,
tier: model.tier,
needsRamMB: model.needs?.ramMB ?? 0,
fits,
reason: fits
? `fits (needs ~${model.needs?.ramMB}MB, ${budget}MB available)`
: `too big (needs ~${model.needs?.ramMB}MB${model.needs?.gpu ? " + GPU" : ""}, ${budget}MB available)`,
notes: model.notes,
};
});
}
@@ -0,0 +1,154 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import {
listModels,
meetsSpecs,
selectModel,
describeModelLadder,
CAPABILITIES,
} from "./local-models.mjs";
const TIERS = ["small", "medium", "large", "xlarge"];
const strongGpu = {
ramMB: 64000,
gpu: { present: true, kind: "nvidia", vramMB: 24000 },
appleSilicon: false,
};
const cpuOnly = { ramMB: 16000, gpu: { present: false, vramMB: 0 }, appleSilicon: false };
const tiny = { ramMB: 1024, gpu: { present: false, vramMB: 0 }, appleSilicon: false };
test("every capability table is non-empty and well-formed", () => {
for (const cap of CAPABILITIES) {
const models = listModels(cap);
assert.ok(models.length > 0, `no models for ${cap}`);
for (const m of models) {
assert.ok(m.id && m.tier && m.needs, `${cap}/${m.id} missing fields`);
assert.ok(TIERS.includes(m.tier), `${cap}/${m.id} bad tier: ${m.tier}`);
assert.equal(typeof m.install, "string", `${cap}/${m.id} needs an install command`);
assert.equal(typeof m.invoke, "string", `${cap}/${m.id} needs an invoke command`);
// user-installed, local-use-only: there is NO license gate on selection
assert.equal("license" in m, false, `${cap}/${m.id} must not carry a license gate`);
}
}
});
test("meetsSpecs enforces RAM, GPU presence, and VRAM", () => {
const gpuModel = { needs: { ramMB: 8000, gpu: true, vramMB: 12000 } };
assert.equal(meetsSpecs(gpuModel, strongGpu), true);
assert.equal(meetsSpecs(gpuModel, cpuOnly), false, "no GPU -> fails a GPU model");
const cpuModel = { needs: { ramMB: 2000, gpu: false } };
assert.equal(meetsSpecs(cpuModel, cpuOnly), true);
assert.equal(meetsSpecs(cpuModel, tiny), false, "too little RAM");
});
test("Apple Silicon unified memory counts as VRAM", () => {
const apple = {
ramMB: 24000,
appleSilicon: true,
gpu: { present: true, kind: "apple", vramMB: 24000 },
};
const gpuModel = { needs: { ramMB: 8000, gpu: true, vramMB: 16000 } };
assert.equal(meetsSpecs(gpuModel, apple), true);
});
test("selectModel picks the large tier on a strong machine", () => {
const r = selectModel("tts", strongGpu);
assert.equal(r.tier, "large");
assert.ok(r.model.id);
});
test("selectModel falls back to medium on a CPU-only machine", () => {
const r = selectModel("tts", cpuOnly);
assert.equal(r.tier, "medium");
assert.equal(r.model.id, "kokoro", "Kokoro is the CPU/medium default (native word timestamps)");
});
test("selectModel recommends the CLI path when no tier fits", () => {
const r = selectModel("tts", tiny);
assert.equal(r.recommend, "cli");
assert.ok(r.reason && /spec/i.test(r.reason));
assert.equal(r.model, undefined);
});
test("preferTier:'medium' avoids the large model even on a strong machine", () => {
const r = selectModel("tts", strongGpu, { preferTier: "medium" });
assert.equal(r.tier, "medium");
});
test("selectModel gates on AVAILABLE RAM, not total, when both are present", () => {
// 64GB total but only 6GB free right now -> the large tier must not be chosen.
const busy = {
ramMB: 64000,
availableRamMB: 6000,
appleSilicon: true,
gpu: { present: true, kind: "apple", vramMB: 64000 },
};
const r = selectModel("tts", busy);
assert.equal(r.tier, "medium", "available RAM (6GB) rules out the 16GB large tier");
});
test("imagegen is a RAM-graduated ladder; agent picks the best that fits", () => {
const ladder = describeModelLadder("imagegen", {
ramMB: 24000,
availableRamMB: 12000,
appleSilicon: true,
gpu: { present: true, kind: "apple", vramMB: 24000 },
});
// best-first order, each flagged with fit
assert.ok(ladder.length >= 3, "imagegen offers multiple RAM tiers");
assert.ok(
ladder[0].needsRamMB >= ladder[ladder.length - 1].needsRamMB,
"ladder is ordered best (biggest) first",
);
// on 24GB / 12GB-free the schnell --low-ram tier fits, the 32GB+ tiers do not
const fitting = ladder.filter((m) => m.fits);
assert.ok(fitting.length >= 1, "at least the low-ram tier fits a 24GB Mac");
assert.ok(
fitting.every((m) => m.needsRamMB <= 12000),
"only sub-budget models flagged as fitting",
);
const pick = selectModel("imagegen", {
ramMB: 24000,
availableRamMB: 12000,
gpu: { present: true, vramMB: 24000 },
});
assert.equal(
pick.model.id,
"flux-schnell-mflux-q4",
"best fit on 24GB is the low-ram schnell tier",
);
});
test("imagegen on a 64GB Mac steps up to the higher-quality tier", () => {
const pick = selectModel("imagegen", {
ramMB: 96000,
availableRamMB: 80000,
gpu: { present: true, vramMB: 96000 },
});
assert.equal(pick.tier, "xlarge", "80GB free unlocks the top-quality model");
});
test("ASR prefers Parakeet by rank even though it is smaller than whisper", () => {
// quality != size for ASR: Parakeet 0.6B beats whisper-1.5B, so `rank` wins
// over footprint. On a capable machine both fit; Parakeet must be chosen.
const capable = {
ramMB: 24000,
availableRamMB: 12000,
appleSilicon: true,
gpu: { present: true, kind: "apple", vramMB: 24000 },
};
const pick = selectModel("asr", capable);
assert.equal(pick.model.id, "parakeet-mlx", "Parakeet is the rank-0 preferred ASR");
// whisperx (rank 1, CPU-only) is the fallback when no GPU
const cpu = { ramMB: 16000, availableRamMB: 12000, gpu: { present: false, vramMB: 0 } };
assert.equal(selectModel("asr", cpu).model.id, "whisperx", "CPU-only falls back to whisperx");
});
test("ASR offers word-timestamp-capable models (better than plain whisper)", () => {
const asr = listModels("asr");
assert.ok(
asr.every((m) => m.wordTimestamps),
"every ASR model must support word timestamps",
);
});
@@ -0,0 +1,64 @@
import { execFileSync } from "node:child_process";
import { selectModel } from "./local-models.mjs";
import { probeSpecs } from "./specs.mjs";
// Run a USER-INSTALLED local model for a capability (tts/asr/upscale).
// Picks the best tier the machine supports (selectModel), checks the tool is on
// PATH, fills the model's invoke template, and runs it. Returns:
// { model, tier, out } on success
// { recommend:"install", model, command, reason } when the tool isn't installed
// { recommend:"cli", reason } when no tier fits the machine
// `exec` / `which` are injectable for tests.
//
// ponytail: "installed" = the invoke's first token is on PATH (e.g. `whisperx`,
// `realesrgan-ncnn-vulkan`). For `python -m kokoro` this only proves python
// exists; good enough to gate — the recommend.command names the real package.
// Upgrade to a per-tool probe if a "python present but package missing" run ever
// produces a confusing error instead of a clean recommend.
function defaultWhich(bin) {
execFileSync("command", ["-v", bin], { stdio: "ignore", shell: true });
}
function defaultExec(cmd) {
execFileSync(cmd, { stdio: ["ignore", "pipe", "pipe"], shell: true, timeout: 600000 });
}
const fill = (tpl, vars) =>
tpl.replace(/\{(\w+)\}/g, (_, k) => (vars[k] != null ? String(vars[k]) : ""));
export function runLocalModel(capability, opts = {}) {
const {
specs = probeSpecs(),
exec = defaultExec,
which = defaultWhich,
vars = {},
preferTier,
} = opts;
const sel = selectModel(capability, specs, { preferTier });
if (sel.recommend) return sel; // no tier fits -> recommend the CLI path
const { model } = sel;
const bin = model.invoke.split(/\s+/)[0];
try {
which(bin);
} catch {
return {
recommend: "install",
model: model.id,
command: model.install,
reason: `${model.id} not installed`,
};
}
try {
exec(fill(model.invoke, vars));
} catch (e) {
return {
recommend: "install",
model: model.id,
command: model.install,
reason: e.message || String(e),
};
}
return { model: model.id, tier: sel.tier, out: vars.out };
}
@@ -0,0 +1,54 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { runLocalModel } from "./local-run.mjs";
const strongCpu = { ramMB: 16000, gpu: { present: false, vramMB: 0 }, appleSilicon: false };
const tiny = { ramMB: 512, gpu: { present: false, vramMB: 0 }, appleSilicon: false };
const ok = () => {}; // which/exec that succeed
test("recommends the CLI path when no local tier fits the machine", () => {
const r = runLocalModel("tts", { specs: tiny, which: ok, exec: ok });
assert.equal(r.recommend, "cli");
});
test("recommends install when the tool is not on PATH", () => {
const r = runLocalModel("tts", {
specs: strongCpu,
which: () => {
throw new Error("not found");
},
exec: ok,
vars: { text: "hi", out: "/tmp/v.wav" },
});
assert.equal(r.recommend, "install");
assert.equal(r.model, "kokoro");
assert.match(r.command, /pip install kokoro/);
});
test("runs the model and returns the output path when installed", () => {
let ran = "";
const r = runLocalModel("tts", {
specs: strongCpu,
which: ok,
exec: (cmd) => {
ran = cmd;
},
vars: { text: "hello world", voice: "af_heart", out: "/tmp/v.wav" },
});
assert.equal(r.model, "kokoro");
assert.equal(r.out, "/tmp/v.wav");
assert.match(ran, /hello world/, "invoke template filled with vars");
assert.match(ran, /\/tmp\/v\.wav/);
});
test("a failing run degrades to an install recommendation, never throws", () => {
const r = runLocalModel("upscale", {
specs: strongCpu,
which: ok,
exec: () => {
throw new Error("boom");
},
vars: { in: "a.png", out: "b.png" },
});
assert.equal(r.recommend, "install");
});
@@ -0,0 +1,222 @@
// Official brand marks — the `logo` type's provider tiers, tried in registry
// order. Every tier was verified against a 54-brand stress test (2026-07,
// 100% cascade hit). Hit counts below are a snapshot of that run — they
// drift as the alias/org maps grow; re-run the stress test to refresh them.
//
// 1. svgl — official full-color vector SVGs (+ wordmark variants);
// 40/54 first-hits. Search is substring-based, so
// entities go through alias normalization first
// ("nextjs" never matches "Next.js" raw).
// 2. simple-icons — monochrome official glyphs; caught the long tail the
// others miss (nike, visa, toyota, wechat, bytedance).
// Pinned CDN build for determinism.
// 3. github avatar — the org's official logo for brands with a GitHub
// presence. Known orgs only: guessing a login risks a
// same-named personal account.
// 4. domain favicon — small-raster last resort (DuckDuckGo ip3). Responses
// under ~500B are DDG's globe placeholder, not a hit.
//
// HeyGen asset search is deliberately absent: for brand queries it returns
// generic look-alike icons (0/3 in testing) — worse than a miss. A total miss
// falls through to resolve's normal failure path (`no provider could resolve
// logo`, exit 1).
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
const SVGL_API = "https://api.svgl.app";
const SIMPLE_ICONS_CDN = "https://cdn.jsdelivr.net/npm/simple-icons@16.25.0/icons";
const FAVICON_MIN_BYTES = 500;
// svgl search queries per entity, tried in order after the raw entity.
const SVGL_ALIASES = {
nextjs: ["next.js", "next"],
aws: ["amazon web services"],
huggingface: ["hugging face"],
cocacola: ["coca-cola"],
mcdonalds: ["mcdonald's"],
};
// simple-icons slugs that differ from the normalized entity.
const SIMPLE_ICON_SLUGS = {
nextjs: "nextdotjs",
aws: "amazonwebservices",
};
// Known GitHub orgs. Only mapped entities resolve at this tier — a brand name
// is NOT a GitHub login, and guessing hits same-named personal accounts.
const GITHUB_ORGS = {
slack: "slackhq",
meta: "facebook",
google: "google",
microsoft: "microsoft",
aws: "aws",
vercel: "vercel",
nextjs: "vercel",
alibaba: "alibaba",
heygen: "heygen-com",
};
// Favicon domains that aren't `<entity>.com`.
const FAVICON_DOMAINS = {
cocacola: "coca-cola.com",
aws: "aws.amazon.com",
nextjs: "nextjs.org",
};
const norm = (s) =>
String(s)
.toLowerCase()
.replace(/[^a-z0-9]/g, "");
/** The brand entity for a query: --entity wins; else the intent minus filler. */
export function entityFrom(intent, entity) {
if (entity) return entity.toLowerCase().trim();
return String(intent)
.toLowerCase()
.replace(/\b(logo|logos|icon|brand|official|mark)\b/g, "")
.trim()
.replace(/\s+/g, " ");
}
/** Exact match after stripping case/spacing/punctuation — "Next.js" ≡ "nextjs". */
export function titleMatches(title, entity) {
return norm(title) === norm(entity);
}
export function svglQueriesFor(entity) {
return [entity, ...(SVGL_ALIASES[norm(entity)] || [])];
}
export function simpleIconSlugsFor(entity) {
const slugs = [norm(entity)];
const alias = SIMPLE_ICON_SLUGS[norm(entity)];
if (alias) slugs.push(alias);
return slugs;
}
export function githubOrgFor(entity) {
return GITHUB_ORGS[norm(entity)] || null;
}
export function faviconDomainFor(entity) {
return FAVICON_DOMAINS[norm(entity)] || `${norm(entity)}.com`;
}
async function fetchJson(url) {
const res = await fetch(url, { signal: AbortSignal.timeout(10_000) });
if (!res.ok) return null;
return res.json();
}
async function urlExists(url) {
const res = await fetch(url, { method: "HEAD", signal: AbortSignal.timeout(10_000) });
return res.ok;
}
export async function svglSearch(intent, ctx = {}) {
const entity = entityFrom(intent, ctx.entity);
for (const q of svglQueriesFor(entity)) {
let items;
try {
items = await fetchJson(`${SVGL_API}?search=${encodeURIComponent(q)}`);
} catch {
return null; // network down — let the next tier try its own host
}
if (!Array.isArray(items)) continue;
const hit = items.find((it) => titleMatches(it.title, q) || titleMatches(it.title, entity));
if (!hit) continue;
const route = typeof hit.route === "string" ? hit.route : hit.route?.light;
if (!route) continue;
return {
url: route,
ext: ".svg",
source: "search",
metadata: {
description: `${hit.title} logo (official mark)`,
provider: "svgl",
provenance: { entity, query: q, route, wordmark: Boolean(hit.wordmark) },
},
};
}
return null;
}
export async function simpleIconsSearch(intent, ctx = {}) {
const entity = entityFrom(intent, ctx.entity);
for (const slug of simpleIconSlugsFor(entity)) {
const url = `${SIMPLE_ICONS_CDN}/${slug}.svg`;
let ok;
try {
ok = await urlExists(url);
} catch {
return null;
}
if (!ok) continue;
return {
url,
ext: ".svg",
source: "search",
metadata: {
description: `${entity} logo (official monochrome glyph)`,
provider: "simple-icons",
provenance: { entity, slug, pinned: "simple-icons@16.25.0" },
},
};
}
return null;
}
export async function githubAvatarSearch(intent, ctx = {}) {
const entity = entityFrom(intent, ctx.entity);
const org = githubOrgFor(entity);
if (!org) return null;
const url = `https://github.com/${org}.png?size=460`;
try {
if (!(await urlExists(url))) return null;
} catch {
return null;
}
return {
url,
ext: ".png",
source: "search",
metadata: {
description: `${entity} logo (GitHub org avatar)`,
provider: "github.avatar",
provenance: { entity, org },
},
};
}
export async function faviconSearch(intent, ctx = {}) {
const entity = entityFrom(intent, ctx.entity);
const domain = faviconDomainFor(entity);
const url = `https://icons.duckduckgo.com/ip3/${domain}.ico`;
let body;
try {
const res = await fetch(url, { signal: AbortSignal.timeout(10_000) });
if (!res.ok) return null;
body = Buffer.from(await res.arrayBuffer());
} catch {
return null;
}
if (body.byteLength < FAVICON_MIN_BYTES) return null; // DDG placeholder, not a logo
// Hand the verified bytes over as a local file: the freeze step copies it
// instead of re-downloading, so the size check is authoritative over what
// gets frozen and the favicon tier costs one network round-trip, not two.
const bytes = body.byteLength;
const tmp = join(mkdtempSync(join(tmpdir(), "media-use-logo-")), `${domain}.ico`);
writeFileSync(tmp, body);
return {
localPath: tmp,
ext: ".ico",
source: "search",
metadata: {
description: `${entity} favicon (small raster — chip-size use only)`,
provider: "favicon.ddg",
provenance: { entity, domain, bytes, low_res: true },
},
};
}
@@ -0,0 +1,137 @@
import test from "node:test";
import assert from "node:assert";
import { readFileSync } from "node:fs";
import {
entityFrom,
titleMatches,
svglQueriesFor,
simpleIconSlugsFor,
githubOrgFor,
faviconDomainFor,
svglSearch,
simpleIconsSearch,
githubAvatarSearch,
faviconSearch,
} from "./logo-provider.mjs";
import { getProviders, runProviders } from "./registry.mjs";
test("entityFrom strips filler words from the intent; --entity wins", () => {
assert.equal(entityFrom("LinkedIn logo"), "linkedin");
assert.equal(entityFrom("official Slack brand mark"), "slack");
assert.equal(entityFrom("anything", "Notion"), "notion");
});
test("titleMatches ignores case, spacing, punctuation — and rejects lookalikes", () => {
assert.ok(titleMatches("Next.js", "nextjs"));
assert.ok(titleMatches("Coca-Cola", "coca cola"));
assert.ok(!titleMatches("Slackware", "slack"));
});
test("svgl queries include the alias forms the raw entity can't match", () => {
assert.ok(svglQueriesFor("nextjs").includes("next.js"));
assert.ok(svglQueriesFor("aws").includes("amazon web services"));
assert.deepEqual(svglQueriesFor("figma"), ["figma"]);
});
test("simple-icons slugs cover the renamed entries", () => {
assert.ok(simpleIconSlugsFor("nextjs").includes("nextdotjs"));
assert.ok(simpleIconSlugsFor("aws").includes("amazonwebservices"));
assert.deepEqual(simpleIconSlugsFor("nike"), ["nike"]);
});
test("github avatar tier never guesses an org", () => {
assert.equal(githubOrgFor("slack"), "slackhq");
assert.equal(githubOrgFor("heygen"), "heygen-com");
assert.equal(githubOrgFor("some-random-startup"), null);
});
test("favicon domain defaults to <entity>.com with explicit overrides", () => {
assert.equal(faviconDomainFor("cocacola"), "coca-cola.com");
assert.equal(faviconDomainFor("stripe"), "stripe.com");
});
// --- async tiers, network mocked -------------------------------------------
// The 54-brand stress test is a manual snapshot; these pin the same behavior
// as CI gates: descriptor shape, alias retry, error→null fallthrough, the
// placeholder filter, and the real cascade order under a mocked network.
const json = (data) => new Response(JSON.stringify(data), { status: 200 });
const status = (code) => new Response(null, { status: code });
const bin = (n) => new Response(new Uint8Array(n), { status: 200 });
test("svglSearch returns the descriptor shape on an exact title hit", async (t) => {
t.mock.method(globalThis, "fetch", async () =>
json([{ title: "Figma", route: "https://svgl.app/library/figma.svg" }]),
);
const res = await svglSearch("Figma logo", {});
assert.equal(res.url, "https://svgl.app/library/figma.svg");
assert.equal(res.ext, ".svg");
assert.equal(res.metadata.provider, "svgl");
});
test("svglSearch skips a non-array payload and retries with the alias query", async (t) => {
const seen = [];
t.mock.method(globalThis, "fetch", async (url) => {
seen.push(decodeURIComponent(String(url)));
return seen.length === 1
? json({ error: "unexpected shape" })
: json([{ title: "Next.js", route: "https://svgl.app/library/nextjs.svg" }]);
});
const res = await svglSearch("nextjs logo", {});
assert.equal(res.metadata.provenance.query, "next.js", "hit came from the alias query");
assert.ok(seen.length >= 2, "raw query then alias");
});
test("svglSearch returns null when the network is down — the cascade falls through", async (t) => {
t.mock.method(globalThis, "fetch", async () => {
throw new Error("network down");
});
assert.equal(await svglSearch("figma logo", {}), null);
});
test("simpleIconsSearch falls to the next slug on a 404", async (t) => {
const seen = [];
t.mock.method(globalThis, "fetch", async (url) => {
seen.push(String(url));
return String(url).includes("amazonwebservices") ? status(200) : status(404);
});
const res = await simpleIconsSearch("aws logo", {});
assert.ok(res.url.endsWith("amazonwebservices.svg"));
assert.equal(seen.length, 2, "plain slug 404s first, alias slug hits");
});
test("faviconSearch rejects DDG's sub-500B placeholder with null", async (t) => {
t.mock.method(globalThis, "fetch", async () => bin(120));
assert.equal(await faviconSearch("someco logo", {}), null);
});
test("faviconSearch hands verified bytes over as a local file — one fetch, no re-download", async (t) => {
const fetchMock = t.mock.method(globalThis, "fetch", async () => bin(600));
const res = await faviconSearch("someco logo", {});
assert.ok(res.localPath, "returns a localPath, not a url");
assert.equal(readFileSync(res.localPath).byteLength, 600, "frozen bytes are the verified bytes");
assert.equal(fetchMock.mock.callCount(), 1, "single network round-trip");
assert.equal(res.metadata.provenance.low_res, true);
});
test("githubAvatarSearch never touches the network for an unmapped entity", async (t) => {
const fetchMock = t.mock.method(globalThis, "fetch", async () => status(200));
assert.equal(await githubAvatarSearch("some-random-startup logo", {}), null);
assert.equal(fetchMock.mock.callCount(), 0);
});
test("the real logo cascade falls through tier by tier to the first hit", async (t) => {
t.mock.method(globalThis, "fetch", async (url) => {
const u = String(url);
if (u.includes("api.svgl.app")) return json([]); // tier 1: no hit
if (u.includes("jsdelivr")) return status(404); // tier 2: no such slug
// tier 3 (github) is never called: entity is unmapped
if (u.includes("duckduckgo")) return bin(600); // tier 4: real favicon
throw new Error(`unexpected fetch: ${u}`);
});
const res = await runProviders(getProviders("logo"), "search", "zzzbrand logo", {
entity: "zzzbrand",
});
assert.ok(res, "cascade must land on the favicon tier");
assert.equal(res.metadata.provider, "favicon.ddg");
});
@@ -0,0 +1,232 @@
import { existsSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { allocateId } from "./manifest.mjs";
import { freezeUrl } from "./freeze.mjs";
import { tokenOverlap } from "./match.mjs";
import { buildCube } from "./cube-build.mjs";
import { validateCube, validateCubeFile } from "./cube-validate.mjs";
const SKILL_DIR = join(import.meta.dirname, "..", "..");
const LUT_DIR = join(SKILL_DIR, "luts");
const LUT_INDEX = join(LUT_DIR, "index.json");
export const LIBRARY_LUT_OFFLINE_CODE = "MEDIA_USE_LIBRARY_LUT_OFFLINE";
// Mirrored from packages/core/src/colorGrading.ts HfColorGradingPresetId.
// Keep this list in lockstep with the core runtime contract.
export const CORE_PRESET_IDS = [
"neutral",
"natural-lift",
"fresh-pop",
"warm-daylight",
"clean-studio",
"skin-soft",
"food-pop",
"night-lift",
"muted-editorial",
"vintage-wash",
"mono-clean",
"mono-fade",
"warm-clean",
"cool-clean",
"soft-boost",
"bright-pop",
"deep-contrast",
];
const PRESET_SYNONYMS = {
neutral: ["neutral", "identity", "none", "ungraded", "natural base"],
"natural-lift": ["natural lift", "natural light", "gentle lift", "soft natural"],
"fresh-pop": ["fresh pop", "fresh", "bright fresh", "clean colorful"],
"warm-daylight": [
"warm daylight",
"warm natural light",
"golden daylight",
"sunlit",
"warm sunny",
],
"clean-studio": ["clean studio", "studio clean", "cool studio", "product studio"],
"skin-soft": ["skin soft", "soft skin", "portrait soft", "beauty skin"],
"food-pop": ["food pop", "food vibrant", "appetizing", "restaurant color"],
"night-lift": ["night lift", "night", "low light lift", "city night"],
"muted-editorial": ["muted editorial", "editorial muted", "magazine muted"],
"vintage-wash": ["vintage wash", "vintage", "retro wash", "aged film"],
"mono-clean": ["mono clean", "black white clean", "monochrome clean"],
"mono-fade": ["mono fade", "black white fade", "faded monochrome"],
"warm-clean": ["warm clean", "clean warm", "warm product"],
"cool-clean": ["cool clean", "clean cool", "cool crisp"],
"soft-boost": ["soft boost", "soft bright", "gentle boost"],
"bright-pop": ["bright pop", "bright punchy", "vivid bright"],
"deep-contrast": ["deep contrast", "high contrast punchy", "punchy contrast", "bold contrast"],
};
function presetCandidates() {
return CORE_PRESET_IDS.map((id) => ({
kind: "preset",
preset: id,
synonyms: PRESET_SYNONYMS[id] ?? [],
text: [id, ...(PRESET_SYNONYMS[id] ?? [])].join(" "),
}));
}
export function readBundledLutIndex() {
if (!existsSync(LUT_INDEX)) return [];
const parsed = JSON.parse(readFileSync(LUT_INDEX, "utf8"));
const entries = Array.isArray(parsed) ? parsed : parsed.looks;
if (!Array.isArray(entries)) return [];
return entries.map((entry) => {
const params =
entry.params && typeof entry.params === "object" && !Array.isArray(entry.params)
? entry.params
: null;
const url = typeof entry.url === "string" && entry.url.trim() ? entry.url.trim() : null;
return {
id: String(entry.id),
description: String(entry.description ?? entry.id),
tags: Array.isArray(entry.tags) ? entry.tags.map(String) : [],
intensity: Number.isFinite(Number(entry.intensity)) ? Number(entry.intensity) : 1,
...(params && { params }),
...(url && { url }),
};
});
}
function libraryCandidates() {
return readBundledLutIndex().map((entry) => ({
kind: "library",
...entry,
text: [entry.id, entry.description, ...entry.tags].join(" "),
}));
}
export function matchColorLook(intent) {
const normalized = String(intent ?? "")
.trim()
.toLowerCase()
.replace(/\s+/g, " ");
for (const candidate of presetCandidates()) {
if (candidate.preset === normalized) {
return { kind: "preset", preset: candidate.preset, score: 99 };
}
}
const candidates = [...presetCandidates(), ...libraryCandidates()]
.map((candidate, index) => ({
...candidate,
index,
score: tokenOverlap(intent, candidate.text),
}))
.filter((candidate) => candidate.score >= 2)
.sort((a, b) => b.score - a.score || a.index - b.index);
if (candidates.length === 0) return null;
const best = candidates[0];
if (best.kind === "preset") {
return { kind: "preset", preset: best.preset, score: best.score };
}
return {
kind: "library",
id: best.id,
description: best.description,
tags: best.tags,
intensity: best.intensity,
...(best.params && { params: best.params }),
...(best.url && { url: best.url }),
score: best.score,
};
}
export function isLibraryLutOfflineMiss(err) {
return err?.code === LIBRARY_LUT_OFFLINE_CODE;
}
function libraryRecord(match, { id, localPath, fullPath, via }) {
return {
id,
localPath,
fullPath,
lut: { src: localPath, intensity: match.intensity },
source: "library",
description: match.description,
metadata: {
provider: "cube_lut.library",
provenance: {
look_id: match.id,
tags: match.tags,
via,
},
},
};
}
function assertValidCubeText(cube, label) {
const check = validateCube(cube);
if (!check.ok) throw new Error(`${label}: ${check.error}`);
}
function assertValidCubeFile(path, label) {
const check = validateCubeFile(path);
if (!check.ok) throw new Error(`${label}: ${check.error}`);
}
function offlineLibraryMiss(match) {
const err = new Error(`library LUT "${match.id}" is CDN-only and --local-only is set`);
err.code = LIBRARY_LUT_OFFLINE_CODE;
return err;
}
export async function freezeLibraryLut(match, { projectDir, type, localOnly = false }) {
if (!match || match.kind !== "library") {
throw new Error("freezeLibraryLut requires a library match");
}
// Prefer the CDN url so looks download on-demand (like bgm/image). Fall back
// to deterministic buildCube params when offline (--local-only) or if the
// download/validation fails, so resolution is never blocked on the network.
if (match.url && !localOnly) {
const { id, localPath } = allocateId(projectDir, type, ".cube");
const fullPath = join(projectDir, localPath);
const tmpPath = `${fullPath}.tmp`;
try {
// Download + validate at a .tmp path, then atomically rename. A crash
// (SIGKILL/OOM) between write and validate can't orphan an invalid .cube
// at the final path — only a validated cube is ever renamed into place.
await freezeUrl(match.url, tmpPath);
assertValidCubeFile(tmpPath, `downloaded library LUT ${match.id} failed validation`);
renameSync(tmpPath, fullPath);
return libraryRecord(match, { id, localPath, fullPath, via: "url" });
} catch (err) {
rmSync(tmpPath, { force: true });
if (!match.params) {
throw new Error(`failed to freeze library LUT ${match.id}: ${err.message}`);
}
// else: fall through to the params fallback below
}
}
if (match.params) {
const { id, localPath } = allocateId(projectDir, type, ".cube");
const fullPath = join(projectDir, localPath);
const tmpPath = `${fullPath}.tmp`;
try {
const cube = buildCube(match.params);
assertValidCubeText(cube, `invalid library LUT ${match.id}`);
// Write + validate at .tmp, then atomic rename — same no-orphan guarantee
// as the url path above.
writeFileSync(tmpPath, cube);
assertValidCubeFile(tmpPath, `invalid frozen LUT ${localPath}`);
renameSync(tmpPath, fullPath);
} catch (err) {
rmSync(tmpPath, { force: true });
throw err;
}
return libraryRecord(match, {
id,
localPath,
fullPath,
via: match.url ? "params-fallback" : "params",
});
}
if (match.url) throw offlineLibraryMiss(match); // url-only entry, offline
throw new Error(`misconfigured library LUT "${match.id}": expected params or url`);
}
@@ -0,0 +1,130 @@
import { strict as assert } from "node:assert";
import { mkdtempSync, rmSync, existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { test } from "node:test";
import {
CORE_PRESET_IDS,
LIBRARY_LUT_OFFLINE_CODE,
freezeLibraryLut,
matchColorLook,
readBundledLutIndex,
} from "./lut-preset-provider.mjs";
import { buildCube } from "./cube-build.mjs";
import { validateCube, validateCubeFile } from "./cube-validate.mjs";
const REPO_ROOT = join(import.meta.dirname, "..", "..", "..", "..");
function corePresetIdsFromSource() {
const src = readFileSync(join(REPO_ROOT, "packages/core/src/colorGrading.ts"), "utf8");
const match = src.match(/export type HfColorGradingPresetId =([\s\S]*?);/);
assert.ok(match, "core preset union should be readable");
return [...match[1].matchAll(/"([^"]+)"/g)].map((m) => m[1]);
}
test("warm daylight and warm natural light resolve to the core warm-daylight preset", () => {
assert.deepEqual(matchColorLook("warm daylight"), {
kind: "preset",
preset: "warm-daylight",
score: 2,
});
assert.equal(matchColorLook("warm natural light").preset, "warm-daylight");
});
test("high contrast punchy resolves to deep-contrast", () => {
assert.equal(matchColorLook("high contrast punchy").preset, "deep-contrast");
});
test("library look freezes a validated cube from params offline (--local-only)", async () => {
const projectDir = mkdtempSync(join(tmpdir(), "mu-lut-provider-"));
try {
const match = matchColorLook("teal orange blockbuster");
assert.equal(match.kind, "library");
// localOnly forces the deterministic params path (no network); online, the
// same look downloads its .cube from the CDN url (via "url").
const frozen = await freezeLibraryLut(match, { projectDir, type: "grade", localOnly: true });
assert.match(frozen.localPath, /^\.media\/luts\/grade_001\.cube$/);
assert.ok(existsSync(join(projectDir, frozen.localPath)));
assert.equal(validateCubeFile(join(projectDir, frozen.localPath)).ok, true);
assert.equal(frozen.lut.src, frozen.localPath);
assert.equal(frozen.metadata.provenance.via, "params-fallback");
} finally {
rmSync(projectDir, { recursive: true, force: true });
}
});
test("preset IDs stay in sync with packages/core/src/colorGrading.ts", () => {
assert.deepEqual(CORE_PRESET_IDS, corePresetIdsFromSource());
for (const id of CORE_PRESET_IDS) {
const match = matchColorLook(id);
assert.equal(match.kind, "preset");
assert.equal(match.preset, id);
}
});
test("zero-overlap intent returns no preset or library match", () => {
assert.equal(matchColorLook("zqxv imaginary neutron look"), null);
});
test("bundled LUT index entries resolve from params or url", () => {
for (const entry of readBundledLutIndex()) {
assert.ok(entry.id);
assert.ok(entry.description);
assert.ok(entry.params || entry.url, `${entry.id} should define params or url`);
if (entry.params) {
assert.equal(typeof entry.params, "object");
assert.equal(validateCube(buildCube(entry.params)).ok, true, `${entry.id} params validate`);
}
if (entry.url) assert.equal(typeof entry.url, "string");
}
});
test("url library entries respect localOnly and freeze through fetch", async () => {
const projectDir = mkdtempSync(join(tmpdir(), "mu-lut-url-provider-"));
const match = {
kind: "library",
id: "cdn-look",
description: "CDN-hosted look",
tags: ["cdn"],
intensity: 0.7,
url: "https://example.invalid/look.cube",
};
const originalFetch = globalThis.fetch;
let fetchCalls = 0;
try {
globalThis.fetch = async () => {
fetchCalls++;
throw new Error("network should be skipped under localOnly");
};
await assert.rejects(
freezeLibraryLut(match, { projectDir, type: "lut", localOnly: true }),
(err) => {
assert.equal(err.code, LIBRARY_LUT_OFFLINE_CODE);
assert.match(err.message, /--local-only/);
return true;
},
);
assert.equal(fetchCalls, 0);
const cube = buildCube({ contrast: 0.1 });
const body = Buffer.from(cube);
globalThis.fetch = async (url) => {
fetchCalls++;
assert.equal(url, match.url);
return {
ok: true,
headers: { get: () => String(body.length) },
body: [body],
};
};
const frozen = await freezeLibraryLut(match, { projectDir, type: "lut" });
assert.equal(fetchCalls, 1);
assert.match(frozen.localPath, /^\.media\/luts\/lut_001\.cube$/);
assert.equal(validateCubeFile(join(projectDir, frozen.localPath)).ok, true);
assert.equal(frozen.metadata.provenance.via, "url");
} finally {
globalThis.fetch = originalFetch;
rmSync(projectDir, { recursive: true, force: true });
}
});
+191
View File
@@ -0,0 +1,191 @@
import {
readFileSync,
appendFileSync,
mkdirSync,
existsSync,
readdirSync,
openSync,
closeSync,
writeFileSync,
rmSync,
statSync,
} from "node:fs";
import { join } from "node:path";
const MANIFEST_FILE = "manifest.jsonl";
const INDEX_FILE = "index.md";
const TYPE_DIRS = {
bgm: "audio/bgm",
sfx: "audio/sfx",
voice: "audio/voice",
image: "images",
icon: "images",
logo: "images",
brand: "images",
video: "video",
grade: "luts",
lut: "luts",
};
export function mediaDir(projectDir) {
return join(projectDir, ".media");
}
export function manifestPath(projectDir) {
return join(mediaDir(projectDir), MANIFEST_FILE);
}
export function indexPath(projectDir) {
return join(mediaDir(projectDir), INDEX_FILE);
}
export function typeSubdir(type) {
const sub = TYPE_DIRS[type];
if (!sub) throw new Error(`unknown media type: ${type}`);
return sub;
}
export function typeDirPath(projectDir, type) {
return join(mediaDir(projectDir), typeSubdir(type));
}
export function readManifest(projectDir) {
const p = manifestPath(projectDir);
if (!existsSync(p)) return [];
const raw = readFileSync(p, "utf8");
const records = [];
for (const line of raw.split(/\r?\n/)) {
const trimmed = line.trim();
if (!trimmed) continue;
try {
records.push(JSON.parse(trimmed));
} catch {
// ponytail: skip malformed lines, don't crash
}
}
return records;
}
export function appendRecord(projectDir, record) {
const dir = mediaDir(projectDir);
mkdirSync(dir, { recursive: true });
const typeDir = typeDirPath(projectDir, record.type);
mkdirSync(typeDir, { recursive: true });
const p = manifestPath(projectDir);
const line = JSON.stringify(record) + "\n";
appendFileSync(p, line);
}
// Match prompts forgivingly. Agents rarely re-emit a byte-identical intent, so
// keying cache lookups on exact equality meant "Calm piano" and "calm piano"
// re-searched and re-downloaded. Normalize (trim, lowercase, collapse internal
// whitespace) on both sides; the raw prompt is still stored for audit.
export function normalizePrompt(prompt) {
return String(prompt ?? "")
.trim()
.toLowerCase()
.replace(/\s+/g, " ");
}
export function findByPrompt(projectDir, prompt, type) {
const key = normalizePrompt(prompt);
if (!key) return null;
const records = readManifest(projectDir);
return (
records.find(
(r) => normalizePrompt(r.provenance?.prompt) === key && (type == null || r.type === type),
) || null
);
}
export function findByEntity(projectDir, entity) {
const lower = entity.toLowerCase();
const records = readManifest(projectDir);
return records.find((r) => r.entity && r.entity.toLowerCase() === lower) || null;
}
export function nextId(projectDir, type) {
const records = readManifest(projectDir);
const prefix = type;
let max = 0;
for (const r of records) {
if (r.type !== type) continue;
const m = r.id?.match(new RegExp(`^${prefix}_(\\d+)$`));
if (m) max = Math.max(max, parseInt(m[1], 10));
}
return `${prefix}_${String(max + 1).padStart(3, "0")}`;
}
// Sync sleep (no busy-spin) for the allocation lock retry.
function sleepMs(ms) {
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
}
// Coarse per-project lock so concurrent resolves don't race on id allocation.
// ponytail: one lock file with a 15s stale-steal (a crashed holder can't wedge
// the project); fine for agent-scale concurrency — revisit if throughput needs
// finer locking. Date.now() is available here (a normal Node CLI, not a
// workflow DSL), so mtime-based staleness is safe.
const LOCK_STALE_MS = 15000;
const LOCK_TIMEOUT_MS = 20000;
function withLock(dir, fn) {
const lock = join(dir, ".lock");
const start = Date.now();
for (;;) {
try {
closeSync(openSync(lock, "wx")); // O_EXCL: atomic acquire
break;
} catch (err) {
if (err.code !== "EEXIST") throw err;
try {
if (Date.now() - statSync(lock).mtimeMs > LOCK_STALE_MS) {
rmSync(lock, { force: true }); // steal a stale lock from a dead holder
continue;
}
} catch {
continue; // lock vanished between check and stat — retry the acquire
}
if (Date.now() - start > LOCK_TIMEOUT_MS) {
throw new Error("media-use: timed out acquiring .media/.lock");
}
sleepMs(25);
}
}
try {
return fn();
} finally {
rmSync(lock, { force: true });
}
}
// Atomically allocate the next free id for `type` AND reserve its file, so a
// slow download/copy between allocation and appendRecord can't let a concurrent
// caller grab the same id (the MU-23 clobber). Under the lock we take the max id
// across BOTH the manifest and any already-reserved files in the type dir, then
// O_EXCL-create an empty placeholder at the target path; freeze/copy overwrites
// it. Returns { id, localPath }.
export function allocateId(projectDir, type, ext) {
mkdirSync(mediaDir(projectDir), { recursive: true });
const typeDir = typeDirPath(projectDir, type);
mkdirSync(typeDir, { recursive: true });
return withLock(mediaDir(projectDir), () => {
const re = new RegExp(`^${type}_(\\d+)`);
let max = 0;
for (const r of readManifest(projectDir)) {
if (r.type !== type) continue;
const m = r.id?.match(re);
if (m) max = Math.max(max, parseInt(m[1], 10));
}
for (const f of readdirSync(typeDir)) {
const m = f.match(re);
if (m) max = Math.max(max, parseInt(m[1], 10)); // skip ids reserved but not yet appended
}
const id = `${type}_${String(max + 1).padStart(3, "0")}`;
const localPath = `.media/${typeSubdir(type)}/${id}${ext}`;
writeFileSync(join(projectDir, localPath), "", { flag: "wx" }); // durable reservation
return { id, localPath };
});
}
@@ -0,0 +1,347 @@
import { strict as assert } from "node:assert";
import { mkdtempSync, rmSync, readFileSync, writeFileSync, mkdirSync, existsSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import {
readManifest,
appendRecord,
findByPrompt,
findByEntity,
nextId,
allocateId,
normalizePrompt,
manifestPath,
mediaDir,
typeDirPath,
typeSubdir,
} from "./manifest.mjs";
import { regenerateIndex, generateIndexContent } from "./index-gen.mjs";
import {
contentHash,
cachePut,
cacheGet,
cacheGetByEntity,
importFromCache,
promote,
} from "./cache.mjs";
let tmp;
function setup() {
tmp = mkdtempSync(join(tmpdir(), "mu-test-"));
}
function cleanup() {
if (tmp) rmSync(tmp, { recursive: true, force: true });
}
function makeRecord(overrides = {}) {
return {
id: "bgm_001",
type: "bgm",
path: ".media/audio/bgm/bgm_001.wav",
source: "search",
description: "soft minimal ambient",
duration: 11,
provenance: { provider: "heygen.audio.sounds", prompt: "subtle tech" },
...overrides,
};
}
function runTests() {
const tests = [];
function test(name, fn) {
tests.push({ name, fn });
}
// --- manifest.mjs ---
test("readManifest returns empty array when no manifest exists", () => {
setup();
const result = readManifest(tmp);
assert.deepStrictEqual(result, []);
cleanup();
});
test("appendRecord writes valid JSONL and readManifest parses it back", () => {
setup();
const record = makeRecord();
appendRecord(tmp, record);
const records = readManifest(tmp);
assert.equal(records.length, 1);
assert.deepStrictEqual(records[0], record);
cleanup();
});
test("appendRecord creates .media/ and type subdirs on first write", () => {
setup();
appendRecord(tmp, makeRecord());
assert.ok(existsSync(mediaDir(tmp)));
assert.ok(existsSync(typeDirPath(tmp, "bgm")));
cleanup();
});
test("lut and grade artifacts use the shared .media/luts subdir", () => {
assert.equal(typeSubdir("lut"), "luts");
assert.equal(typeSubdir("grade"), "luts");
setup();
const allocated = allocateId(tmp, "lut", ".cube");
assert.equal(allocated.localPath, ".media/luts/lut_001.cube");
assert.ok(existsSync(join(tmp, allocated.localPath)));
cleanup();
});
test("appendRecord appends multiple records", () => {
setup();
appendRecord(tmp, makeRecord({ id: "bgm_001" }));
appendRecord(tmp, makeRecord({ id: "bgm_002", provenance: { prompt: "energetic" } }));
const records = readManifest(tmp);
assert.equal(records.length, 2);
assert.equal(records[0].id, "bgm_001");
assert.equal(records[1].id, "bgm_002");
cleanup();
});
test("findByPrompt returns exact-match record", () => {
setup();
appendRecord(tmp, makeRecord());
const found = findByPrompt(tmp, "subtle tech", "bgm");
assert.ok(found);
assert.equal(found.id, "bgm_001");
cleanup();
});
test("findByPrompt returns null on miss", () => {
setup();
appendRecord(tmp, makeRecord());
assert.equal(findByPrompt(tmp, "nonexistent", "bgm"), null);
cleanup();
});
test("findByPrompt filters by type", () => {
setup();
appendRecord(tmp, makeRecord({ type: "sfx" }));
assert.equal(findByPrompt(tmp, "subtle tech", "bgm"), null);
assert.ok(findByPrompt(tmp, "subtle tech", "sfx"));
cleanup();
});
test("findByPrompt matches across case and whitespace variants", () => {
setup();
appendRecord(tmp, makeRecord({ provenance: { provider: "x", prompt: "calm ambient piano" } }));
assert.ok(findByPrompt(tmp, "Calm Ambient Piano", "bgm"), "case-insensitive");
assert.ok(findByPrompt(tmp, " calm ambient piano ", "bgm"), "whitespace-insensitive");
assert.equal(findByPrompt(tmp, "calm ambient guitar", "bgm"), null, "still a real miss");
cleanup();
});
test("normalizePrompt trims, lowercases, collapses whitespace", () => {
assert.equal(normalizePrompt(" Upbeat Tech Launch "), "upbeat tech launch");
assert.equal(normalizePrompt(null), "");
});
test("allocateId reserves the id on disk so a pre-append caller can't reuse it (MU-23)", () => {
setup();
const a = allocateId(tmp, "bgm", ".wav");
assert.equal(a.id, "bgm_001");
assert.ok(existsSync(join(tmp, a.localPath)), "placeholder reserved on disk");
// Second allocation BEFORE any manifest append (the download window) must not
// hand back bgm_001 again, even with a different extension.
const b = allocateId(tmp, "bgm", ".mp3");
assert.equal(b.id, "bgm_002");
assert.notEqual(a.localPath, b.localPath);
// Lock file is released (not left behind).
assert.ok(!existsSync(join(tmp, ".media", ".lock")), "lock released");
cleanup();
});
test("allocateId continues past the highest manifest id", () => {
setup();
appendRecord(tmp, makeRecord({ id: "bgm_005" }));
assert.equal(allocateId(tmp, "bgm", ".wav").id, "bgm_006");
cleanup();
});
test("findByEntity matches case-insensitively", () => {
setup();
appendRecord(tmp, makeRecord({ entity: "GitHub", type: "icon" }));
assert.ok(findByEntity(tmp, "github"));
assert.ok(findByEntity(tmp, "GITHUB"));
assert.equal(findByEntity(tmp, "gitlab"), null);
cleanup();
});
test("nextId generates sequential ids", () => {
setup();
assert.equal(nextId(tmp, "bgm"), "bgm_001");
appendRecord(tmp, makeRecord({ id: "bgm_001" }));
assert.equal(nextId(tmp, "bgm"), "bgm_002");
appendRecord(tmp, makeRecord({ id: "bgm_002" }));
assert.equal(nextId(tmp, "bgm"), "bgm_003");
cleanup();
});
// --- index-gen.mjs ---
test("regenerateIndex produces plain-column table", () => {
setup();
appendRecord(tmp, makeRecord());
regenerateIndex(tmp);
const content = readFileSync(join(tmp, ".media", "index.md"), "utf8");
assert.ok(content.includes("# .media · 1 asset"));
assert.ok(content.includes("bgm_001"));
assert.ok(content.includes("soft minimal ambient"));
assert.ok(content.includes("11s"));
cleanup();
});
test("regenerateIndex handles empty manifest", () => {
setup();
mkdirSync(join(tmp, ".media"), { recursive: true });
writeFileSync(manifestPath(tmp), "");
regenerateIndex(tmp);
const content = readFileSync(join(tmp, ".media", "index.md"), "utf8");
assert.ok(content.includes("# .media · 0 assets"));
cleanup();
});
test("generateIndexContent includes dims for images", () => {
const records = [
makeRecord({ id: "img_001", type: "image", width: 1920, height: 1080, duration: null }),
];
const content = generateIndexContent(records);
assert.ok(content.includes("1920×1080"));
assert.ok(content.includes("img_001"));
});
test("regenerateIndex matches manifest content after multiple writes", () => {
setup();
appendRecord(tmp, makeRecord({ id: "bgm_001" }));
appendRecord(
tmp,
makeRecord({ id: "sfx_001", type: "sfx", description: "whoosh", duration: 3 }),
);
regenerateIndex(tmp);
const content = readFileSync(join(tmp, ".media", "index.md"), "utf8");
assert.ok(content.includes("# .media · 2 assets"));
assert.ok(content.includes("bgm_001"));
assert.ok(content.includes("sfx_001"));
assert.ok(content.includes("whoosh"));
cleanup();
});
// --- cache.mjs ---
test("cacheGet returns null when cache is empty", () => {
const result = cacheGet("nonexistent prompt", "bgm");
assert.equal(result, null);
});
test("cachePut + cacheGet round-trip", () => {
setup();
const filePath = join(tmp, "test.wav");
writeFileSync(filePath, "fake audio bytes for testing");
const record = makeRecord({ provenance: { prompt: "cache test" } });
const { sha } = cachePut(filePath, record);
assert.ok(sha);
assert.equal(sha.length, 64);
const found = cacheGet("cache test", "bgm");
assert.ok(found);
assert.equal(found.reusable, true);
assert.equal(found.sha, sha);
// cross-project reuse must survive trivial prompt variation, not just
// byte-identical intents (the whole point of normalizePrompt).
assert.ok(cacheGet(" Cache Test ", "bgm"), "cacheGet is case/whitespace-insensitive");
cleanup();
});
test("cacheGetByEntity finds cached asset", () => {
setup();
const filePath = join(tmp, "logo.png");
writeFileSync(filePath, "fake png bytes");
const record = makeRecord({
type: "icon",
entity: "TestCorp",
provenance: { prompt: "TestCorp logo" },
});
cachePut(filePath, record);
const found = cacheGetByEntity("testcorp");
assert.ok(found);
assert.equal(found.entity, "TestCorp");
cleanup();
});
test("contentHash is deterministic", () => {
setup();
const filePath = join(tmp, "det.bin");
writeFileSync(filePath, "deterministic content");
const h1 = contentHash(filePath);
const h2 = contentHash(filePath);
assert.equal(h1, h2);
cleanup();
});
test("promote copies project asset to global cache", () => {
setup();
const record = makeRecord();
appendRecord(tmp, record);
const filePath = join(tmp, record.path);
mkdirSync(join(filePath, ".."), { recursive: true });
writeFileSync(filePath, "promotable audio data");
const { sha } = promote(tmp, "bgm_001");
assert.ok(sha);
const cached = cacheGet("subtle tech", "bgm");
assert.ok(cached);
assert.equal(cached.sha, sha);
cleanup();
});
test("importFromCache copies cached file into project", () => {
setup();
const filePath = join(tmp, "source.wav");
writeFileSync(filePath, "importable audio");
const record = makeRecord({ provenance: { prompt: "import test" } });
const { sha } = cachePut(filePath, record);
const cached = cacheGet("import test", "bgm");
const projectDir = mkdtempSync(join(tmpdir(), "mu-import-"));
const imported = importFromCache(cached, projectDir, "bgm_001", ".media/audio/bgm/bgm_001.wav");
assert.ok(imported);
assert.equal(imported.id, "bgm_001");
assert.equal(imported.provenance.imported_from, sha);
assert.ok(existsSync(join(projectDir, ".media/audio/bgm/bgm_001.wav")));
rmSync(projectDir, { recursive: true, force: true });
cleanup();
});
// --- run ---
let passed = 0;
let failed = 0;
for (const { name, fn } of tests) {
try {
fn();
passed++;
console.log(` \x1b[32m✓\x1b[0m ${name}`);
} catch (err) {
failed++;
console.log(` \x1b[31m✗\x1b[0m ${name}`);
console.log(` ${err.message}`);
}
}
console.log(`\n${passed} passed, ${failed} failed`);
if (failed > 0) process.exit(1);
}
console.log("media-use · manifest/index/cache tests\n");
runTests();
+47
View File
@@ -0,0 +1,47 @@
// Shared lexical-matching helpers used by both the assets/ scan (adopt.mjs) and
// the reuse-candidate ranker (candidates.mjs), and the type-equivalence check
// used by resolve.mjs and candidates.mjs. Kept in one place so the icon<->image
// equivalence and the token rules can't drift between the "do" path (resolve)
// and the "look" path (candidates).
// Common filler words that should never, on their own, make two strings match.
const MATCH_STOPWORDS = new Set([
"the",
"and",
"for",
"with",
"from",
"this",
"that",
"your",
"our",
]);
// Split into lowercased word tokens of length >= 3, minus stopwords.
export function matchTokens(text) {
return new Set(
String(text)
.toLowerCase()
.split(/[^a-z0-9]+/)
.filter((t) => t.length >= 3 && !MATCH_STOPWORDS.has(t)),
);
}
// Count of shared meaningful word tokens between two strings. 0 = no lexical
// overlap (the candidate ranker still surfaces these, ordered after overlaps).
export function tokenOverlap(a, b) {
const ta = matchTokens(a);
const tb = matchTokens(b);
let n = 0;
for (const t of ta) if (tb.has(t)) n++;
return n;
}
// icon, image, and logo are interchangeable: all live in images/, and
// figma-imported brand marks are recorded as type image while agents ask for
// logos as icon or logo.
export function typesMatch(a, b) {
if (a === b) return true;
const visual = new Set(["icon", "image", "logo"]);
return visual.has(a) && visual.has(b);
}
@@ -0,0 +1,95 @@
import { execFileSync } from "node:child_process";
import { existsSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { probeSpecs } from "./specs.mjs";
import { selectModel } from "./local-models.mjs";
// Local image generation via mflux (FLUX-on-MLX), the Mac-native runner.
// Spec-gated: selectModel("imagegen", specs) returns the best FLUX-class model
// the machine's AVAILABLE RAM can actually run (medium FLUX-schnell --low-ram on
// ~24GB, up to Qwen-Image on 64GB+). When nothing local fits, this returns null
// so the registry falls through to the codex image upsell.
//
// The official FLUX repos are HF-gated, so the model entries point --path at
// non-gated community 4-bit re-uploads; the repo is resolved to a local snapshot
// (hf download, idempotent) because a bare repo id breaks mlx unflatten.
// Tokenize the invoke template on whitespace FIRST, then substitute each token,
// so a {prompt}/{model_path} value with spaces stays a single argv entry.
function buildArgv(template, vars) {
return template
.trim()
.split(/\s+/)
.map((tok) => tok.replace(/\{(\w+)\}/g, (_, k) => (k in vars ? String(vars[k]) : `{${k}}`)));
}
// Resolve an HF repo to its local snapshot dir. `hf download` is idempotent and
// prints the snapshot path as its last line.
function resolveSnapshot(repo) {
const out = execFileSync("hf", ["download", repo], {
encoding: "utf8",
timeout: 1_800_000,
stdio: ["ignore", "pipe", "pipe"],
});
const path = out.trim().split(/\r?\n/).pop()?.trim();
return path && existsSync(path) ? path : null;
}
export async function mfluxImageGenerate(intent, ctx) {
const specs = ctx?.specs || probeSpecs();
const sel = selectModel("imagegen", specs, { preferTier: ctx?.preferTier });
if (sel.recommend) return null; // no local model fits -> codex upsell/fallback
const { model } = sel;
const bin = model.invoke.trim().split(/\s+/)[0];
// Not installed? Surface the exact enable-command FIRST (before the model
// download) so the agent learns the free local path is available instead of
// silently taking the codex upsell.
try {
execFileSync("which", [bin], { stdio: ["ignore", "ignore", "ignore"] });
} catch {
console.error(
`media-use: local image gen not enabled (\`${bin}\` not on PATH). Install for free on-device FLUX: ${model.install}`,
);
return null;
}
const outPath = join(tmpdir(), `media-use-mflux-${process.pid}-${Date.now()}.png`);
const width = ctx?.width || 512;
const height = ctx?.height || 512;
const seed = ctx?.seed ?? 42;
const vars = { prompt: intent, w: width, h: height, seed, out: outPath };
if (model.repo && model.invoke.includes("{model_path}")) {
const snap = model.repo ? resolveSnapshot(model.repo) : null;
if (!snap) return null;
vars.model_path = snap;
}
const argv = buildArgv(model.invoke, vars);
argv.shift(); // drop the bin (already validated)
try {
execFileSync(bin, argv, {
encoding: "utf8",
timeout: 1_800_000,
stdio: ["ignore", "pipe", "pipe"],
});
} catch (err) {
console.error(
`media-use: local image gen (${model.id}) failed: ${err.stderr?.toString().trim().slice(-200) || err.message}`,
);
return null;
}
if (!existsSync(outPath)) return null;
return {
localPath: outPath,
ext: ".png",
source: "generated",
metadata: {
description: intent,
provider: `mflux.${model.id}`,
provenance: { model: model.id, tier: model.tier, prompt: intent },
},
};
}
+49
View File
@@ -0,0 +1,49 @@
import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
const MISSES_FILE = "misses.jsonl";
function missesPath() {
return join(homedir(), ".media", MISSES_FILE);
}
export function recordMiss({ type, intent, provider_override, local_only }) {
try {
const dir = join(homedir(), ".media");
mkdirSync(dir, { recursive: true });
appendFileSync(
join(dir, MISSES_FILE),
JSON.stringify({
ts: new Date().toISOString(),
type,
intent,
provider_override: !!provider_override,
local_only: !!local_only,
}) + "\n",
);
} catch {
// local miss logging is best-effort; never surface into resolve
}
}
export function readMisses() {
const p = missesPath();
try {
if (!existsSync(p)) return [];
const raw = readFileSync(p, "utf8");
const records = [];
for (const line of raw.split(/\r?\n/)) {
const trimmed = line.trim();
if (!trimmed) continue;
try {
records.push(JSON.parse(trimmed));
} catch {
// skip malformed local lines, don't crash stats
}
}
return records;
} catch {
return [];
}
}
@@ -0,0 +1,78 @@
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { readMisses, recordMiss } from "./misses.mjs";
function sandbox() {
const root = mkdtempSync(join(tmpdir(), "mu-misses-"));
const home = join(root, "home");
mkdirSync(home, { recursive: true });
process.env.HOME = home;
return { root, home };
}
function restoreEnv(saved) {
for (const k of Object.keys(process.env)) if (!(k in saved)) delete process.env[k];
Object.assign(process.env, saved);
}
test("recordMiss appends a well-formed local miss", () => {
const savedEnv = { ...process.env };
const { root, home } = sandbox();
try {
recordMiss({ type: "bgm", intent: "moody synth pulse", provider_override: true });
const misses = readMisses();
assert.equal(misses.length, 1);
assert.equal(misses[0].type, "bgm");
assert.equal(misses[0].intent, "moody synth pulse");
assert.equal(misses[0].provider_override, true);
assert.equal(misses[0].local_only, false);
assert.ok(!Number.isNaN(Date.parse(misses[0].ts)));
const raw = readFileSync(join(home, ".media/misses.jsonl"), "utf8");
assert.match(raw, /moody synth pulse/);
} finally {
restoreEnv(savedEnv);
rmSync(root, { recursive: true, force: true });
}
});
test("recordMiss swallows filesystem failures", () => {
const savedEnv = { ...process.env };
const { root, home } = sandbox();
try {
writeFileSync(join(home, ".media"), "not a directory");
assert.doesNotThrow(() =>
recordMiss({ type: "image", intent: "unwritable", local_only: true }),
);
assert.equal(existsSync(join(home, ".media/misses.jsonl")), false);
} finally {
restoreEnv(savedEnv);
rmSync(root, { recursive: true, force: true });
}
});
test("readMisses skips corrupt lines", () => {
const savedEnv = { ...process.env };
const { root, home } = sandbox();
try {
mkdirSync(join(home, ".media"), { recursive: true });
writeFileSync(
join(home, ".media/misses.jsonl"),
[
JSON.stringify({ ts: "2026-07-09T00:00:00.000Z", type: "bgm", intent: "one" }),
"{not json",
JSON.stringify({ ts: "2026-07-09T00:00:01.000Z", type: "sfx", intent: "two" }),
].join("\n"),
);
assert.deepEqual(
readMisses().map((miss) => miss.intent),
["one", "two"],
);
} finally {
restoreEnv(savedEnv);
rmSync(root, { recursive: true, force: true });
}
});
@@ -0,0 +1,27 @@
// Merge Parakeet-MLX token timestamps into word timestamps.
//
// parakeet-mlx JSON emits SUB-WORD tokens (" H", "ello", ...) with per-token
// start/end. Captions + transcript-cut need WORD timestamps, so join tokens
// into words on the space boundary: a token whose text starts with a space
// (or the very first token) begins a new word; the rest append. Output matches
// the { words: [{ text, start, end }] } shape the rest of media-use consumes
// (see words.mjs / cutlist.mjs).
export function mergeTokensToWords(parakeet) {
const sentences = Array.isArray(parakeet?.sentences) ? parakeet.sentences : [];
const words = [];
for (const s of sentences) {
for (const t of s.tokens ?? []) {
const raw = typeof t.text === "string" ? t.text : "";
const startsWord = raw.startsWith(" ") || words.length === 0;
if (startsWord) {
words.push({ text: raw.trim(), start: t.start, end: t.end });
} else {
const w = words[words.length - 1];
w.text += raw;
w.end = t.end;
}
}
}
return { text: (parakeet?.text ?? "").trim(), words: words.filter((w) => w.text.length > 0) };
}

Some files were not shown because too many files have changed in this diff Show More