elizaos--eliza/packages/benchmarks/MEETING_VOICE_BENCHMARK_MATRIX.md

# Meeting, Transcription, and Voice Benchmark Matrix

Research matrix for issue #13352. This file records which external meeting,
speech, and voice-assistant benchmarks are already covered by elizaOS, which
ones need legal or infrastructure review, and where an adapter should land.

Status is based on repo inspection plus public source pages on 2026-07-04. The
source review checked the QMSum, MeetingBank, ELITR-Bench, VoiceCodeBench,
VoiceBench, and Sierra 3-Bench primary pages directly; the remaining P1/P2 rows
are conservative until their adapter PRs recheck license/access terms. Do not
commit raw external data unless the allowed-use column explicitly permits a repo
fixture and the source license is rechecked in the adapter PR.

## Existing elizaOS Coverage

| Surface | Existing coverage | Gap this matrix should drive |
| --- | --- | --- |
| `packages/scenario-runner` | Real `AgentRuntime` scenarios, deterministic/live lanes, JSON reports, native JSONL export. | Meeting transcript and voice task adapters that produce scenario definitions or scenario metadata from external datasets. |
| `packages/benchmarks/lifeops-bench` | Multi-turn tool-use benchmark with deterministic world-state scoring across Eliza/Hermes/OpenClaw/Cerebras adapters. | Knowledge-grounded and voice-interruption domains inspired by Sierra tau-Knowledge/tau-Voice without copying restricted data. |
| `packages/benchmarks/meeting-transcription-proof` | Planned adapter-contract scaffold from #13378 / #13359 for transcript, diarization, speaker identity, consent, retention, and meeting note metrics; not pre-existing `develop` coverage at the time of the matrix. | Dataset adapters for QMSum, MeetingBank, ELITR-Bench, TCR, and controlled public meeting slices. |
| `packages/benchmarks/voice-speaker-validation` | Speaker profile lifecycle, diarization, single-stream, and async identity checks. | Public speaker-count/diarization stress slices and trait-aware regression fixtures. |
| `packages/benchmarks/registry` | Integrated `voicebench`, `voicebench_quality`, `mmau`, and scoring gates that reject mock results for publishable runs. | New registry entries for VoiceCodeBench, QMSum/MeetingBank smoke slices, and tau-style knowledge/voice methodology. |
| `packages/training` | Eliza-1 benchmark matrix and voice gates for ASR/TTS/runtime metrics. | Exact structured-token recovery, long-form ASR, dropped-frame/interruption robustness, and voice task pass-at-1 gates. |
| `plugins/plugin-google` | Google Meet artifact import path with the current `GoogleMeetReport` export; canonical `elizaos.meeting_artifact.v1` is planned adapter/schema work rather than an existing plugin-google export. | Reference-based artifact grading: summary, quote grounding, action items, topic relevance, and privacy/retention checks. |

## Benchmark Matrix

Allowed-use values:

- `repo-fixture`: small derived fixture can be committed after license check.
- `downloaded-eval`: adapter downloads public data at run time; no raw data in repo.
- `private-eval`: use only with credentials/access approval; do not redistribute.
- `training-ok`: candidate training/fine-tuning data after contamination review.
- `eval-only`: evaluate only; keep out of training.
- `needs-legal`: license/access is unclear or restricted.
- `do-not-use`: avoid until legal/product explicitly changes the decision.

| Source | Task/modality | Status in elizaOS | License/access posture | Allowed use | Harness target | Priority | Minimal starting slice |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MeetBench / MeetAll / MeetMaster: https://github.com/huyuelin/MeetBench-MeetAll.github.io | Meeting-agent QA, meeting summaries, factuality and structure judging; multilingual/multimodal meetings. | New; only indirectly overlaps with meeting-proof and scenario-runner. | Project notes describe MeetAll data as CC BY-NC 4.0 / non-commercial research only. | `needs-legal`, likely `eval-only`/`do-not-use` for commercial repo fixtures. | Scenario-runner live-only meeting-agent scenarios; no raw data in repo. | P2 | Metadata-only adapter proof plus one synthetic eliza-owned fixture that mirrors task shape. |
| QMSum: https://github.com/Yale-LILY/QMSum | Query-focused meeting summarization over academic/product/committee meetings. | New; maps cleanly to meeting artifact QA. | GitHub repo is MIT; underlying source meeting corpora still need citation/license review per corpus. | `downloaded-eval`; small query/summary fixture only after corpus-level review. | `meeting-transcription-proof` and scenario-runner transcript QA scenarios. | P0 | 10 query-summary pairs with transcript excerpts, exact reference spans, and judge fallback. |
| MeetingBank / MeetingBank-utils: https://meetingbank.github.io/ and https://github.com/YebowenHu/MeetingBank-utils | City council meeting summarization with transcripts, videos, agendas, and minutes. | New; target adapter scaffold is #13378 / #13359, not existing dataset coverage on `develop`. | Public meeting data; site and Zenodo publish dataset, but city/source terms and video reuse need review. | `downloaded-eval`; repo fixture should be minimal metadata/excerpt only. | Meeting artifact generation and long-context summarization benchmark. | P0 | 5 public meeting sections with transcript excerpt, agenda, reference minutes, and action-item extraction. |
| ELITR-Bench: https://github.com/utter-project/ELITR-Bench | Long-context meeting transcript QA with ground-truth answers and metadata. | New. | GitHub lists CC BY 4.0; verify upstream ELITR meeting-data terms before raw fixture use. | `downloaded-eval`; `repo-fixture` only for tiny CC-compatible examples. | Scenario-runner long-context QA and retrieval/quote-grounding checks. | P1 | 20 QA rows with answer/reference metadata and no audio. |
| Topic-Conversation Relevance (TCR): https://github.com/microsoft/topic_conversation | Topic relevance over 1,500 meetings and 15k+ topics. | New. | Paper/repo indicate CC BY 4.0 data, but source-meeting provenance needs review. | `downloaded-eval`; possible small `repo-fixture` after provenance check. | Scenario-runner topic tracking and meeting artifact topic relevance. | P1 | 50 topic/transcript-window pairs with exact-match relevance metrics. |
| AMI Meeting Corpus: https://groups.inf.ed.ac.uk/ami/corpus/ | Meeting audio/video/transcripts, diarization, summaries. | Partially covered by generic meeting proof metrics, not dataset-specific. | Research corpus with separate license/access terms. | `needs-legal`, likely `private-eval` or `downloaded-eval`. | Meeting transcription, diarization, speaker attribution, summarization. | P1 | License-gated manifest adapter; no committed media. |
| ICSI Meeting Corpus: https://groups.inf.ed.ac.uk/ami/icsi/ | Meeting speech, transcripts, speaker labels. | Partially covered conceptually only. | Research corpus with access restrictions and citation requirements. | `needs-legal`, likely `private-eval`. | Speaker validation and meeting transcription proof. | P1 | Manifest-only adapter plus documented acquisition step. |
| CHiME-6: https://chimechallenge.github.io/chime6/ | Dinner-party multi-speaker ASR/diarization under real noise. | Partially covered by voice-speaker-validation abstractions, not dataset. | Challenge data requires agreement and download process. | `private-eval`; no repo data. | ASR/diarization stress lane and Eliza-1 ASR gates. | P1 | 1-hour manifest slice with WER/DER/JER metrics. |
| SAMSum: https://huggingface.co/datasets/Samsung/samsum | Dialogue summarization, not meeting-specific. | New; lower-fidelity baseline. | HF endpoint returned 401 during the 2026-07-04 link check; treat access and license as gated until rechecked. | `needs-legal`, likely `private-eval`; no repo fixture unless access terms change. | Baseline summarization scorer only, clearly marked non-meeting. | P2 | Metadata-only until access and license are verified. |
| DialogSum: https://huggingface.co/datasets/knkarthick/dialogsum | Dialogue summarization, not meeting-specific. | New; lower-fidelity baseline. | HF dataset; license must be checked before fixture. | `downloaded-eval`. | Baseline summarization scorer. | P2 | 100 short dialogue summaries. |
| MediaSum: https://github.com/zcgzcgzcg1/MediaSum | Interview/media transcript summarization. | New. | Dataset terms need review. | `needs-legal`; likely `downloaded-eval`. | Long transcript summarization baseline. | P2 | Metadata-only until license verified. |
| LongSpeech: https://arxiv.org/html/2601.13539v1 | Long-duration speech benchmark for ASR, ST, summarization, language detection, speaker count, QA, temporal localization, and emotion. | New. | Paper describes dataset; public code/data availability and license need review. | `needs-legal`, likely `downloaded-eval`. | Eliza-1 long-form ASR and audio QA gates. | P2 | 10-minute public subset if released with redistributable terms. |
| Sierra mu-Bench: https://huggingface.co/datasets/sierra-research/mu-bench | Multilingual transcription from real customer-service calls. | New. | Restricted access, non-commercial/restricted terms and deletion/redistribution limits. | `private-eval`, `eval-only`; no training or repo fixtures. | Private ASR regression lane. | P2 | Access-gated manifest and local-only result schema. |
| VoiceCodeBench: https://huggingface.co/datasets/besimple-ai/voice-code-bench | Exact structured-token recovery for workplace speech: paths, URLs, numbers, code-like entities. | New. | HF page indicates MIT and small human-recorded WAV set. | `downloaded-eval`; candidate `repo-fixture` for a tiny sample if license confirmed. | Eliza-1 ASR gate and meeting-transcription-proof exact-token metric. | P0 | 30 clips covering URLs, file paths, IDs, and punctuation; report CTEM/TSR. |
| Picovoice STT benchmark: https://github.com/Picovoice/speech-to-text-benchmark | WER/punctuation over LibriSpeech, TED-LIUM, Common Voice, MLS, VoxPopuli, FLEURS. | New harness; overlaps with ASR metrics. | Framework repo and component datasets have separate licenses. | `downloaded-eval`; dataset-specific gates. | Eliza-1 ASR comparison harness. | P1 | Common Voice or FLEURS 50-row smoke slice with checksums. |
| LibriSpeech: https://www.openslr.org/12 | Read-speech ASR baseline. | Not directly integrated. | Widely used OpenSLR corpus; verify license before committing slices. | `downloaded-eval`; possible small fixture after review. | Eliza-1 ASR baseline. | P2 | 100 dev-clean utterances. |
| TED-LIUM: https://www.openslr.org/51 | Lecture/talk ASR baseline. | Not directly integrated. | OpenSLR dataset with its own terms. | `downloaded-eval`. | Long-form ASR and punctuation baseline. | P2 | 30 short segments plus transcript refs. |
| Common Voice: https://commonvoice.mozilla.org/datasets | Multilingual ASR. | Not directly integrated. | Mozilla Common Voice has versioned licenses; review selected release. | `downloaded-eval`; possible training after contamination review. | Eliza-1 multilingual ASR gates. | P1 | 50 clips per target language. |
| MLS: https://www.openslr.org/94 | Multilingual read-speech ASR. | Not directly integrated. | OpenSLR dataset terms. | `downloaded-eval`. | Multilingual ASR. | P2 | 50 clips across high-priority locales. |
| VoxPopuli: https://github.com/facebookresearch/voxpopuli | Multilingual parliamentary speech ASR. | Not directly integrated. | Research dataset; terms vary by source. | `downloaded-eval`, `needs-legal` for fixtures. | Long-form multilingual ASR. | P2 | 50 short segments. |
| FLEURS: https://huggingface.co/datasets/google/fleurs | Multilingual speech recognition/translation. | Not directly integrated. | HF dataset; verify license/version. | `downloaded-eval`. | Multilingual ASR and language-ID gate. | P1 | 50 clips per launch locale. |
| VoiceBench: https://github.com/matthewcym/voicebench and https://huggingface.co/datasets/hlt-lab/voicebench | LLM-based voice assistant benchmark across spoken instruction subsets. | Partially covered: `voicebench_quality` is registered; `voicebench` latency harness exists. | HF/GitHub pages indicate Apache-2.0 for public dataset/code; verify exact subset terms. | `downloaded-eval`; selected `repo-fixture` possible for smoke. | Existing `voicebench_quality`; add missing subsets or adapter parity tests. | P0 | Close gap between registered suite list and current HF subsets; document skipped subsets. |
| VoiceAssistant-Eval: https://github.com/mathllm/VoiceAssistant-Eval and https://huggingface.co/datasets/MathLLMs/VoiceAssistant-Eval | Listening, speaking, and viewing; content quality, speech quality, consistency. | New. | Public HF/GitHub; license needs check before fixtures. | `downloaded-eval`, `needs-legal` for raw media fixtures. | Eliza-1 voice gates and scenario-runner multimodal live lane. | P2 | Metadata-only adapter plus 20 listening/speaking rows after license check. |
| VocalBench: https://github.com/SJTU-OmniAgent/VocalBench | Semantic, acoustic, chat, latency/RTF dimensions for speech interaction models. | New; overlaps with voicebench latency. | Public repo; data/license needs review. | `downloaded-eval`, `needs-legal`. | Eliza-1 speech-to-speech release gate. | P2 | Latency/RTF metric schema first, data second. |
| VCB-Bench: https://github.com/Tencent/VCB-Bench | Chinese voice chat benchmark with robustness and multi-turn tasks. | New. | Public repo; license/data terms need review. | `downloaded-eval`, `needs-legal`. | Multilingual voice assistant lane. | P2 | 20 Chinese prompt/audio rows with pass-at-1 task scoring. |
| MTalk-Bench: https://github.com/FreedomIntelligence/MTalk-Bench | Multi-turn speech-to-speech dialogue with semantic, paralinguistic, ambient, multiparty dimensions. | New. | Research license noted by project; verify terms before use. | `needs-legal`, likely `eval-only`. | Scenario-runner live voice and Eliza-1 full-duplex gates. | P2 | Methodology-only first; no raw data committed. |
| aiewf-eval: https://github.com/kwindla/aiewf-eval | Multi-turn workflow evals with text, realtime audio, speech-to-speech, tool-use, instruction, and grounding dimensions. | New; methodology overlaps LifeOpsBench and scenario-runner. | Public repo; license needs review. | `downloaded-eval` for code ideas, not data until verified. | LifeOpsBench adapter design and realtime audio scenario design. | P1 | Borrow report/schema concepts; implement an elizaOS-native workflow smoke fixture. |
| Vox-Profile: https://arxiv.org/html/2505.14648v1 | Speaker/speech trait robustness benchmark. | New; overlaps speaker validation. | Paper/data release status needs review. | `needs-legal`. | Voice-speaker-validation robustness report. | P2 | Trait taxonomy mapped to existing speaker validation metrics. |
| Picovoice TTS benchmark: https://github.com/Picovoice/text-to-speech-benchmark | Streaming TTS responsiveness under assistant-like generated text. | New harness; overlaps TTS latency gates. | Framework and datasets need license review. | `downloaded-eval`. | Eliza-1 TTS latency/resource gate. | P1 | 20 generated-response prompts with first-audio/p95/resource metrics. |
| Sierra tau-Knowledge / tau-Voice / 3-Bench: https://sierra.ai/blog/bench-advancing-agent-benchmarking-to-knowledge-and-voice | Knowledge-grounded backend task completion and realistic voice interactions with interruptions/noise/dropped frames. | New as dataset; methodology overlaps LifeOpsBench and Eliza-1 voice gates. | Blog describes benchmark release; public code/data/license must be verified separately. | `needs-legal`; methodology can be used without copying data. | LifeOpsBench knowledge domain and scenario-runner live voice task lane. | P0 methodology, P2 data | Add eliza-owned knowledge-base task and voice-interruption scenario using the same scoring philosophy. |
| MultiVox: https://arxiv.org/html/2507.10859v1 | Omni voice assistant benchmark with speech plus visual cues and paralinguistic features. | New. | Paper/data release and license need review. | `needs-legal`. | Multimodal scenario-runner and Eliza-1 audio-vision gate. | P2 | Methodology-only until data terms are clear. |
| SOVA-Bench: https://arxiv.org/abs/2506.02457 | Speech conversational voice assistant benchmark, including understanding and generated speech quality. | New. | Paper/data license needs review. | `needs-legal`. | Eliza-1 speech generation quality and acoustic metrics. | P2 | Metric taxonomy and scoring schema first. |
| AudioBench / OpenAudioBench: https://github.com/audiollms/audiobench | AudioLLM benchmark across speech, audio-scene, and voice understanding tasks. | Partially covered by registered MMAU and voicebench, but AudioBench itself is new. | Public repo; dataset licenses vary by subset. | `downloaded-eval`, subset-specific legal review. | Benchmark registry audio understanding lane. | P1 | Map only speech/voice subsets first; avoid scene/music until model support is explicit. |
| Big Bench Audio: https://huggingface.co/blog/big-bench-audio-release | Speech reasoning benchmark adapted from Big Bench Hard, evaluated across speech-to-speech/text paths. | New; overlaps MMAU/VoiceBench reasoning. | Public blog/dataset; license and access need verification. | `needs-legal` until source package checked. | Eliza-1 speech reasoning gate and hosted provider comparison. | P1 | 100-question reasoning slice, pass-at-1 exact answer. |
| ADU-Bench: https://arxiv.org/abs/2412.05167 | Open-ended audio dialogue understanding, multilingual ambiguity handling. | New. | Paper/data release and license need review. | `needs-legal`. | Scenario-runner voice ambiguity scenarios. | P2 | Methodology and ambiguity taxonomy first. |
| AIR-Bench: https://aclanthology.org/2024.acl-long.109/ | Generative audio-language benchmark with speech, natural sounds, and music. | New; broader than current voice-only lanes. | Public benchmark; subset licenses need review. | `downloaded-eval`, subset-specific. | Audio understanding registry entry after speech subset triage. | P2 | Speech-only foundation rows, no music/sound scene rows initially. |

## Recommended P0 Implementation Issues

1. VoiceCodeBench exact-token ASR gate: #13358
   - Target: `packages/training` plus `packages/benchmarks/registry`.
   - Metrics: CTEM/TSR, URL/path/ID exact recovery, punctuation-critical WER.
   - Evidence: real Eliza-1 or configured ASR provider run over a downloaded slice, score JSON, and manually reviewed failures.

2. QMSum and MeetingBank meeting artifact adapter: #13359
   - Target: `packages/benchmarks/meeting-transcription-proof` with scenario-runner export.
   - Metrics: query answer correctness, quote grounding, action-item extraction, agenda/topic coverage, summary faithfulness.
   - Evidence: real provider run over a tiny downloaded slice and `elizaos.meeting_artifact.v1` outputs inspected by hand.

3. VoiceBench coverage closeout: #13360
   - Target: existing `voicebench_quality` and `voicebench` registry entries.
   - Metrics: current suite coverage vs public VoiceBench subsets, per-suite score, STT provider, judge model, mock-result rejection.
   - Evidence: real `voicebench_quality` run with non-mock STT and Cerebras judge, plus report JSON.

4. Sierra-style tau-Knowledge/tau-Voice eliza-owned fixtures: #13361
   - Target: `lifeops-bench` and scenario-runner live voice.
   - Metrics: deterministic backend state success, knowledge-source grounding, interruption recovery, dropped-frame robustness, task pass-at-1.
   - Evidence: eliza-owned synthetic domain only; no Sierra data unless license/access is confirmed.

## Source And Repo Review Notes

- QMSum's primary repository describes 1,808 query-summary pairs over 232
  meetings across Academic, Product, and Committee domains and carries an MIT
  repository license, but adapter PRs still need corpus-level provenance review.
- MeetingBank's project page describes 1,366 city-council meetings, more than
  3,579 hours of video, transcripts, minutes, agendas, and 6,892 segment-level
  summarization instances; adapters should download by manifest rather than
  commit raw data.
- ELITR-Bench includes password-protected JSON archives with manually crafted
  questions, ground-truth answers, metadata, and generated responses. GitHub
  reports mixed code/data licenses, including CC BY 4.0 for data, so keep it P1
  and recheck upstream ELITR terms before fixtures.
- VoiceCodeBench's Hugging Face card reports MIT licensing, 300 test rows, audio
  modality, and structured entities; it is the cleanest P0 exact-token ASR slice.
- VoiceBench's GitHub repository and Hugging Face dataset report Apache-2.0
  licensing, public data, and 20,554 rows. elizaOS already registers
  `voicebench` and `voicebench_quality`, so the P0 work is coverage closeout, not
  a greenfield benchmark.
- Sierra's 3-Bench blog describes tau-Voice as realistic full-duplex voice with
  interruptions, noisy/compressed audio, dropped frames, and task pass@1
  failures. The P0 value is methodology and eliza-owned fixtures until public
  code/data/license status is reviewed.
- Local repo inspection found existing benchmark registry coverage for
  `voicebench`, `voicebench_quality`, `mmau`, and `tau_bench`, plus
  voice-speaker-validation, LifeOpsBench, plugin-google Meet report imports, and
  Eliza-1 training/eval surfaces. #13378 is the proposed
  meeting-transcription-proof adapter-contract scaffold; no direct QMSum,
  MeetingBank, VoiceCodeBench, ELITR-Bench, or TCR adapter exists on `develop`
  today.

## Follow-Up Triage

- Legal: MeetBench/MeetAll, mu-Bench, MTalk-Bench, AMI, ICSI, CHiME-6, and any dataset with non-commercial, restricted, or source-corpus terms.
- Infra: long audio downloads, GPU/ASR providers, and nightly-only release lanes.
- Product/model: which Eliza-1 gates become blocking versus dashboard-only.
- Contamination: keep training data and evaluation data separated; do not fine-tune on benchmark rows without explicit approval.

## Verification Notes

This PR is documentation and planning only. It does not add adapters, benchmark
data, model behavior, or runtime code. Applicable evidence is source review and
repo coverage inspection; real run evidence belongs to the adapter PRs listed
above.