wehub-resource-sync
c3749daf48
Tests / test-linux (3.13) (push) Failing after 0s
Tests / test-linux (3.11) (push) Failing after 1s
Tests / lint (push) Failing after 0s
Tests / test-linux (3.9) (push) Failing after 1s
Docker / build (push) Failing after 1s
Docker / build-gpu (push) Failing after 2s
Tests / test-windows (push) Has been cancelled
Tests / test-macos (push) Has been cancelled
169 lines
8.6 KiB
Markdown
169 lines
8.6 KiB
Markdown
# MemPalace Small-Model Benchmark Datasets
|
|
|
|
Synthetic, hand-curated evaluation datasets for testing small (≤4B parameter) Ollama models on the four classification and extraction tasks that matter for MemPalace's local-first memory pipeline.
|
|
|
|
- **Synthetic only.** No real-person names, no real organizations, no real PII.
|
|
- **Public-safe.** Intended to be committed to the public MemPalace open-source repository.
|
|
- **Generated 2026-05-10.**
|
|
- **211 samples** total across four tasks (100 synthetic + 1 real-format-flavored sample in room_classification).
|
|
|
|
## Tasks
|
|
|
|
| Task | Samples | Purpose |
|
|
|---|---|---|
|
|
| `room_classification/` | 101 | Given a session summary and a room list, pick the right room (or `"other"`). Tests the closed-set room-routing path used at filing time. 100 synthetic + 1 real-format-flavored (rc_101). |
|
|
| `entity_extraction/` | 50 | Given a conversation excerpt, extract typed entities (person / project / place / organization). Tests the entity detector that feeds the knowledge graph. |
|
|
| `memory_extraction/` | 40 | Given a snippet, extract memory-worthy items (decision / preference / fact / opinion / commitment). Tests the layer-promotion heuristic. |
|
|
| `calibration/` | 20 | Sentence-type classification (question / command / statement / exclamation / greeting). A trivially-easy baseline used to detect a broken model load before more meaningful evals run. |
|
|
|
|
## The five fictional personae
|
|
|
|
The 100 synthetic room-classification samples are split 20-per-agent across these synthetic agents (rc_101 is a separate real-format-flavored sample assigned to Solas). Each agent has a fixed room taxonomy; per-sample room lists are drawn as 5-10 room subsets that always include `general`.
|
|
|
|
### 1. Aria — research assistant (she/her)
|
|
|
|
Synthesizes academic papers.
|
|
|
|
Rooms: `about`, `projects/meta-cognition`, `projects/embedding-spaces`, `projects/topic-clustering`, `skills/latex`, `skills/python`, `skills/statistical-tests`, `daily-logs`, `general`
|
|
|
|
### 2. Solas — coding agent (they/them)
|
|
|
|
Writes and reviews code; focuses on systems work.
|
|
|
|
Rooms: `about`, `projects/distributed-tracing`, `projects/parser-combinators`, `projects/type-systems`, `skills/rust`, `skills/ocaml`, `skills/llvm`, `daily-logs`, `general`
|
|
|
|
### 3. Fenra — creative writing agent (she/her)
|
|
|
|
Drafts fiction and worldbuilds.
|
|
|
|
Rooms: `about`, `projects/world-building`, `projects/character-arcs`, `projects/dialogue-engine`, `skills/narrative-design`, `skills/etymology`, `skills/mythology`, `daily-logs`, `general`
|
|
|
|
### 4. Bramble — gardening agent (he/him)
|
|
|
|
Companion-plant and ecological-gardening advice.
|
|
|
|
Rooms: `about`, `projects/pollinator-paths`, `projects/soil-microbes`, `projects/native-species`, `skills/phenology`, `skills/plant-pathology`, `skills/propagation`, `daily-logs`, `general`
|
|
|
|
### 5. Thresh — finance/accounting agent (they/them)
|
|
|
|
Small-business bookkeeping.
|
|
|
|
Rooms: `about`, `projects/invoice-parsing`, `projects/tax-prep`, `projects/cashflow-models`, `skills/double-entry`, `skills/depreciation-rules`, `skills/ifrs`, `daily-logs`, `general`
|
|
|
|
## Distribution stats
|
|
|
|
### `room_classification/` (101 samples)
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| Samples per agent | Aria 20, Solas 20, Fenra 20, Bramble 20, Thresh 20 |
|
|
| Messy samples (`include_messy_features: true`) | 14 (target: ~15%) |
|
|
| Closed-set `"other"` | 18 (target: ~20%) |
|
|
| Closed-set `"general"` (escape hatch used as best fit) | 6 |
|
|
| Closed-set match to a non-`general` non-`other` room | 76 |
|
|
|
|
`"other"` distribution by agent: Aria 2, Solas 3, Fenra 3, Bramble 7, Thresh 3. Bramble skews higher because gardening conversations frequently span topics that don't cleanly fit a single skill room (e.g. seed-saving + plant pathology, soil chemistry, vegetable-pest control).
|
|
|
|
### `entity_extraction/` (50 samples)
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| Total entities | 247 |
|
|
| `person` | 114 |
|
|
| `organization` | 74 |
|
|
| `project` | 32 |
|
|
| `place` | 27 |
|
|
| Entities per sample (min / max / avg) | 3 / 9 / 4.9 |
|
|
|
|
Person-skew is intentional: real MemPalace sessions are heavily person-centric, and the entity detector's hardest job is disambiguating people (the dataset deliberately reuses some last names like "Halloran" across distinct individuals to test this).
|
|
|
|
### `memory_extraction/` (40 samples)
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| Total memories | 55 |
|
|
| `decision` | 12 |
|
|
| `preference` | 9 |
|
|
| `commitment` | 12 |
|
|
| `fact` | 15 |
|
|
| `opinion` | 7 |
|
|
| Memories per sample | 1 (25 samples) or 2 (15 samples) |
|
|
|
|
### `calibration/` (20 samples)
|
|
|
|
Exactly 4 samples per class: `question`, `command`, `statement`, `exclamation`, `greeting`. Designed to be unambiguous — any working model should hit ≥95% accuracy here. If a model fails calibration, it is broken or misconfigured and the more interesting tasks should not be run.
|
|
|
|
## Schema reference
|
|
|
|
### `room_classification/dataset.jsonl`
|
|
|
|
```json
|
|
{"id": "rc_001", "agent": "Aria", "session_summary": "...", "include_messy_features": false}
|
|
```
|
|
|
|
### `room_classification/labels.jsonl`
|
|
|
|
```json
|
|
{"id": "rc_001", "closed_set_label": "projects/meta-cognition", "preferred_open_label": "meta-cognition-research"}
|
|
```
|
|
|
|
`closed_set_label` is either a room from that sample's `rooms` list, or exactly `"other"`.
|
|
`preferred_open_label` is the slug a human annotator would invent if labeling freely (lowercase, hyphenated).
|
|
|
|
### `room_classification/room_lists.jsonl`
|
|
|
|
```json
|
|
{"id": "rc_001", "rooms": ["about", "projects/meta-cognition", "skills/python", "daily-logs", "general"]}
|
|
```
|
|
|
|
Always includes `"general"`. Always 5-10 rooms. Composition varies per sample so the model can't just memorize an agent's full taxonomy.
|
|
|
|
### `entity_extraction/dataset.jsonl` + `labels.jsonl`
|
|
|
|
```json
|
|
{"id": "ent_001", "text": "..."}
|
|
{"id": "ent_001", "entities": [{"name": "Aria", "type": "person"}, {"name": "Embedding Spaces", "type": "project"}]}
|
|
```
|
|
|
|
Entity types: `person`, `project`, `place`, `organization`.
|
|
|
|
### `memory_extraction/dataset.jsonl` + `labels.jsonl`
|
|
|
|
```json
|
|
{"id": "mem_001", "text": "..."}
|
|
{"id": "mem_001", "memories": [{"type": "decision", "content": "..."}]}
|
|
```
|
|
|
|
Memory types: `decision`, `preference`, `fact`, `opinion`, `commitment`.
|
|
|
|
### `calibration/dataset.jsonl` + `labels.jsonl`
|
|
|
|
```json
|
|
{"id": "cal_001", "text": "Could you fix the indentation on this Python file?", "classes": ["question", "command", "statement", "exclamation", "greeting"]}
|
|
{"id": "cal_001", "label": "command"}
|
|
```
|
|
|
|
The `classes` field is identical across all 20 samples, included in each line for self-contained downstream loading.
|
|
|
|
## Annotation conventions
|
|
|
|
- **Decision vs. commitment.** A *decision* changes how something will be done from now on (a policy or design choice); a *commitment* binds the speaker to a specific deliverable, often with a deadline. "Switching to Jaccard similarity" is a decision; "I'll deliver the draft Friday" is a commitment. When both apply to one sentence, both are emitted.
|
|
- **Fact vs. opinion.** A *fact* is a claim about the world that is checkable in principle. An *opinion* is signaled by hedge words ("I think", "honestly", "my read") or by being a value judgement. Borderline cases (e.g. an arguable empirical claim about a paper) lean to opinion when the speaker frames it as their own assessment.
|
|
- **Preference.** Preferences attach to a person and describe how that person likes things done. They are durable across conversations, unlike one-off decisions.
|
|
- **Closed-set "other".** Used when no listed room is a substantively better home for the content than `general`. If `general` is a clean fit (catch-all small talk, identity questions), the label is `general` rather than `other`.
|
|
- **Ambiguous closed-set samples** (~10% of room_classification) have two rooms that fit; the label picks the one that better matches the dominant content of the session. `preferred_open_label` may name the secondary topic when the closed-set choice is forced.
|
|
|
|
## Extending
|
|
|
|
To regenerate or extend:
|
|
|
|
1. **Add new personae.** Define a name, pronouns, role, and 7-9 rooms (always include `about`, `daily-logs`, `general`). Add 20 samples covering each room at least once, plus a few drift / messy / `"other"` cases.
|
|
2. **Add more samples.** Keep the per-agent balance even. Maintain ~70/20/10 distribution for closed-set / `"other"` / ambiguous.
|
|
3. **Add new tasks.** Pick a discriminative task that small models would plausibly fail in interesting ways (relation extraction, coreference, stance, etc.). Match the pattern of separate `dataset.jsonl` and `labels.jsonl` files keyed by `id`.
|
|
|
|
## Provenance
|
|
|
|
- Synthetic, generated 2026-05-10 for the MemPalace project.
|
|
- No real-person names. Fictional personae and place/organization names invented for this benchmark.
|
|
- Safe to commit to a public repository.
|