chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,262 @@
|
||||
# SkillOpt-Sleep — plugins for Claude Code, Codex, Copilot, and Devin
|
||||
|
||||
**Your coding agent forgets everything between sessions. SkillOpt-Sleep fixes
|
||||
that.** While you sleep, it reviews what you did today, notices the rules you
|
||||
keep repeating ("always add a LIMIT", "answers in `\boxed{}`", "cite the
|
||||
source"), and writes them into your agent's long-term memory and skills — but
|
||||
only the rules that actually make it score better on *your own* past tasks. You
|
||||
wake up to an agent that's better at *your* work, and you approve every change
|
||||
before it sticks.
|
||||
|
||||
One engine, four thin shells. It synthesizes **SkillOpt** (validation-gated
|
||||
bounded text optimization — the research in this repo), **Claude Dreams**
|
||||
(offline consolidation; input never mutated; review-then-adopt), and the **agent
|
||||
sleep** idea (short-term experience → long-term competence).
|
||||
|
||||
> **Open-source tool, decoupled from the research.** The engine lives in the
|
||||
> top-level [`skillopt_sleep/`](../skillopt_sleep) package with **zero
|
||||
> dependency** on the paper's `skillopt/` experiment code (the validation gate is
|
||||
> vendored). Use it without the research stack.
|
||||
|
||||
---
|
||||
|
||||
| Platform | Folder | Mechanism | Status |
|
||||
|---|---|---|---|
|
||||
| **Claude Code** | [`claude-code/`](claude-code) | `.claude-plugin` + `/skillopt-sleep` + `/skillopt-sleep-handoff` commands + skill + hooks | full, installable |
|
||||
| **Codex** | [`codex/`](codex) | user-level `skillopt-sleep` skill + shared runner | full |
|
||||
| **Copilot** | [`copilot/`](copilot) | MCP server (`sleep_*` tools) + `copilot-instructions` | full (MCP) |
|
||||
| **Devin** | [`devin/`](devin) | MCP server (`sleep_*` tools) + Devin ATIF-v1.7 harvest + `.devin/rules` | full (MCP) |
|
||||
|
||||
## Install (pick your agent)
|
||||
|
||||
| Platform | Install | Then |
|
||||
|---|---|---|
|
||||
| **Claude Code** | `/plugin marketplace add microsoft/SkillOpt` → `/plugin install skillopt-sleep` | `/skillopt-sleep status` |
|
||||
| **Codex** | `git clone` → `bash plugins/codex/install.sh` | `/skillopt-sleep status` |
|
||||
| **Copilot** | `git clone` → register `plugins/copilot/mcp_server.py` as an MCP server | ask "run the sleep cycle" |
|
||||
| **Devin** | `git clone` → `devin mcp add skillopt-sleep -- python3 plugins/devin/mcp_server.py` | ask "run the sleep cycle" |
|
||||
|
||||
Requirements: Python ≥ 3.10 and the agent's CLI on PATH. All three call the same
|
||||
[`run-sleep.sh`](run-sleep.sh) → `python -m skillopt_sleep`, so behaviour is
|
||||
identical everywhere. Default backend is `mock` (no API spend); `--backend
|
||||
claude|codex|copilot` uses your own budget.
|
||||
|
||||
---
|
||||
|
||||
## How it works: one "night", in plain terms
|
||||
|
||||
```
|
||||
harvest your past sessions → mine the tasks you keep doing → replay them offline
|
||||
→ reflect on failures → propose a few rule edits → KEEP only edits that raise
|
||||
your held-out score → stage a proposal → (you) review & adopt
|
||||
```
|
||||
|
||||
Nothing live changes until you `adopt`; every adopt backs up the prior file.
|
||||
|
||||
### The split that keeps it honest: dream-train / real-val / real-test
|
||||
|
||||
This is the heart of the design, borrowed from the SkillOpt paper's
|
||||
train/selection/test protocol:
|
||||
|
||||
| Split | Where it comes from | What it's for |
|
||||
|---|---|---|
|
||||
| **train** | your real tasks **+ optional "dreamed" variants** | what the optimizer *learns from*. Over-dreaming here is fine — it's imagination. |
|
||||
| **val** (selection) | **your real tasks only**, held out | the **gate**: an edit is kept only if it raises this score. Stops overfitting. |
|
||||
| **test** | **your real tasks only**, held out, never seen during optimization | the **final score** we report. Kept as close to your real usage as possible. |
|
||||
|
||||
So you can **dream up extra training examples** to learn a rule robustly, while
|
||||
the rule is still **judged on real, unseen tasks**. A `dream` task can *never*
|
||||
land in val or test — that invariant is unit-tested.
|
||||
|
||||
---
|
||||
|
||||
## What each feature does **for you** (with examples)
|
||||
|
||||
Every control below works on all three platforms (pass it after the action,
|
||||
e.g. `/skillopt-sleep run --rollouts-k 3`).
|
||||
|
||||
### `--preferences "..."` — tell it your house rules
|
||||
|
||||
The single most useful knob. Free text that steers what the optimizer writes,
|
||||
as a prior. Use it to encode the conventions you're tired of repeating.
|
||||
|
||||
```bash
|
||||
# A backend engineer:
|
||||
/skillopt-sleep run --preferences "Always use async/await, never callbacks. \
|
||||
Prefer pytest over unittest. Commit subjects in imperative mood under 50 chars."
|
||||
|
||||
# A data analyst:
|
||||
/skillopt-sleep run --preferences "Every SQL query must end with LIMIT 1000 unless \
|
||||
I say otherwise. Money in USD with 2 decimals. Prefer CTEs over nested subqueries."
|
||||
|
||||
# A researcher:
|
||||
/skillopt-sleep run --preferences "Cite sources as [Author, Year]. Math answers in \
|
||||
\\boxed{}. Keep explanations under 150 words unless I ask for depth."
|
||||
```
|
||||
*What it does for you:* the next morning your agent already follows these
|
||||
without you re-typing them, and the rules are validated against your real tasks
|
||||
(if a "preference" actually hurts your held-out score, the gate drops it).
|
||||
|
||||
### `--gate on|off` — strict vs. greedy
|
||||
|
||||
- `on` (default): an edit is kept **only if it raises your held-out score**.
|
||||
Safe — blocks plausible-but-wrong rules and reward-hacking.
|
||||
- `off`: greedy — keep edits without the strict check (still reports whether
|
||||
quality moved).
|
||||
|
||||
*What it does for you:* leave it `on` for trust. Flip it `off` when you're
|
||||
exploring and want to see everything the optimizer proposes.
|
||||
|
||||
### `--rollouts-k K` — learn from contrast, not just failure
|
||||
|
||||
Re-runs each task `K` times and learns from the difference between the **good**
|
||||
and **bad** attempts, not just a single failure.
|
||||
|
||||
```bash
|
||||
/skillopt-sleep run --rollouts-k 3
|
||||
```
|
||||
*What it does for you:* a much stronger signal. If your agent gets a task right 1
|
||||
time in 3, the optimizer figures out *what the winning attempt did* and makes it
|
||||
reliable.
|
||||
|
||||
### `--optimizer-model` / `--target-model` — optimize cheap, deploy anywhere
|
||||
|
||||
Use a strong model to *write* the rules and a cheap model to *run* your tasks.
|
||||
The learned skill then helps the cheap model — or any model.
|
||||
|
||||
```bash
|
||||
/skillopt-sleep run --optimizer-model sonnet --target-model haiku
|
||||
```
|
||||
*What it does for you:* spend a little on a smart optimizer overnight; your
|
||||
everyday cheap/fast agent inherits the upgrade. (Verified: a skill optimized on
|
||||
one model lifts a different one — cross-model and even cross-runtime
|
||||
Codex↔Claude.)
|
||||
|
||||
### `--budget-tokens N` / `--budget-minutes M` — cap the spend
|
||||
|
||||
You decide how much the nightly "dreaming" costs; it auto-plans how many nights
|
||||
× how many rollouts fit.
|
||||
|
||||
```bash
|
||||
/skillopt-sleep run --backend claude --budget-tokens 60000
|
||||
```
|
||||
*What it does for you:* predictable cost. It stops cleanly when the budget is hit
|
||||
and tells you what it skipped.
|
||||
|
||||
### multi-objective (accuracy ↑, tokens ↓, latency ↓)
|
||||
|
||||
The reward can weight not just correctness but **cost and speed**, so a skill can
|
||||
learn to be cheaper and faster, not only more accurate. *What it does for you:*
|
||||
"answer directly instead of opening five files" becomes a learned habit.
|
||||
|
||||
### `--backend handoff` — session-executed calls (no API subprocess)
|
||||
|
||||
For subscription seats and environments where the engine shouldn't spawn
|
||||
`claude -p` / API calls itself. The engine still runs every deterministic
|
||||
stage (harvest → mine → replay scoring → gate → stage), but each model call
|
||||
(attempt / judge / reflect) is written to a prompt file that **your own agent
|
||||
session answers between engine runs**:
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep run --backend handoff --project "$(pwd)"
|
||||
# exit 3 => .skillopt-sleep-handoff/PROMPTS.md + pending.json were written
|
||||
# answer each prompt (each in a FRESH context) into answers/<id>.md
|
||||
# re-run the same command => it resumes from the answers and either
|
||||
# finishes (exit 0) or stages the next prompt batch (exit 3)
|
||||
```
|
||||
|
||||
A typical night converges in 3–6 rounds: baseline attempts → reflect →
|
||||
candidate re-scoring per accepted edit. Resume is stateless — replay is
|
||||
deterministic and answers are cached by prompt hash, so re-running skips
|
||||
everything already answered. Mined tasks are pinned to
|
||||
`.skillopt-sleep-handoff/tasks.json` on the first round, so the sessions that
|
||||
answer the prompts can't shift the task set and invalidate earlier answers.
|
||||
On a completed real run the handoff directory is archived to
|
||||
`.skillopt-sleep-handoff.night<N>.done`.
|
||||
|
||||
On Claude Code, `/skillopt-sleep-handoff run` drives the whole loop for you,
|
||||
answering each prompt in an isolated fresh-context subagent.
|
||||
|
||||
**Integrity rule:** answer every prompt in a fresh context (a subagent with no
|
||||
conversation history). Answering from a session that has already seen the
|
||||
mined tasks and their references contaminates the held-out gate and fakes the
|
||||
improvement score.
|
||||
|
||||
*What it does for you:* the sleep cycle runs entirely on your interactive
|
||||
session's subscription budget — no API key, no headless subprocess — while the
|
||||
gate, splits, and staging discipline stay in the engine.
|
||||
|
||||
Limitations: `--rollouts-k > 1` gives no contrastive spread (identical prompt
|
||||
→ identical answer file), and tool-loop tasks fall back to the single-shot
|
||||
`TOOL_CALL:` marker convention.
|
||||
|
||||
### `schedule` / `unschedule` — set it and forget it
|
||||
|
||||
Built-in nightly scheduling (no manual cron):
|
||||
|
||||
```bash
|
||||
/skillopt-sleep schedule --hour 3 --minute 17 # runs every night for this project
|
||||
/skillopt-sleep unschedule # stop it
|
||||
```
|
||||
*What it does for you:* it just gets better while you sleep. The nightly run only
|
||||
*stages* a proposal — adopting is still your call (or add `--auto-adopt` when you
|
||||
schedule, if you trust it).
|
||||
|
||||
---
|
||||
|
||||
## Full action / flag reference
|
||||
|
||||
| Action | Does |
|
||||
|---|---|
|
||||
| `status` | nights so far + the latest staged proposal (read-only) |
|
||||
| `dry-run` | harvest→mine→replay→report; **stages nothing** |
|
||||
| `run` | full cycle; **stages** a proposal; nothing live changes |
|
||||
| `adopt` | apply the staged proposal to `CLAUDE.md`/`SKILL.md` (backs up first) |
|
||||
| `harvest` | debug: print the recurring tasks it mined |
|
||||
| `schedule` / `unschedule` | install/remove the nightly cron entry |
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `--backend mock\|claude\|codex\|copilot\|handoff` | `mock` | who runs/optimizes (mock = free; handoff = your own session answers) |
|
||||
| `--preferences "..."` | – | your house rules, as a prior |
|
||||
| `--gate on\|off` | `on` | strict held-out gate vs. greedy |
|
||||
| `--rollouts-k K` | `1` | multi-rollout contrastive reflection |
|
||||
| `--optimizer-model` / `--target-model` | – | split the optimizer from the target |
|
||||
| `--budget-tokens` / `--budget-minutes` | – | cap the nightly spend |
|
||||
| `--scope invoked\|all` | `invoked` | this project only, or all projects |
|
||||
| `--auto-adopt` | off | apply without manual review (power users) |
|
||||
|
||||
Deep dive: [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
|
||||
|
||||
---
|
||||
|
||||
## Does it actually work?
|
||||
|
||||
Yes — measured with **real models on both Claude and Codex**, scored on held-out
|
||||
tasks the optimizer never trained on:
|
||||
|
||||
- **gbrain-evals `skillopt-v1`** (the public suite gbrain scores SkillOpt on):
|
||||
deficient skills go **0.00 → 1.00** on all 4 seeds, including a real tool-use
|
||||
loop; cross-model transfer is positive; the gate blocks regressions.
|
||||
→ [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)
|
||||
- **Academic daily-cases** (math / spreadsheet / search-QA, the paper's 4:1:5
|
||||
split with dream-augmented train): see
|
||||
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
|
||||
- **Fresh load-test** (a "SQL must always include LIMIT" analyst, built from
|
||||
scratch): held-out **0.00 → 1.00** on both backends.
|
||||
→ [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep)
|
||||
|
||||
Try the deterministic proof yourself (no API key, no spend):
|
||||
```bash
|
||||
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
|
||||
```
|
||||
It prints the held-out score rising to 1.0 as the gate accepts the right rules,
|
||||
and confirms the gate **rejects** an injected harmful edit.
|
||||
|
||||
---
|
||||
|
||||
## Safety
|
||||
|
||||
- **Read-only** harvest of your sessions. `mock` replay has no side effects.
|
||||
- Proposals are **staged**, never auto-applied (unless you opt in with `--auto-adopt`).
|
||||
- Every adopt writes a backup. Per-night token/time budget caps. Secrets redacted.
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
|
||||
"name": "skillopt-sleep",
|
||||
"description": "SkillOpt-Sleep: give your local Claude agent a nightly sleep cycle that reviews past sessions and consolidates validated memory + skills.",
|
||||
"owner": {
|
||||
"name": "Yifan Yang",
|
||||
"email": "yifanyang@microsoft.com"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "skillopt-sleep",
|
||||
"description": "Nightly offline self-evolution: harvest your past Claude Code sessions, replay recurring tasks on your own API budget, and consolidate what the agent learns into validated CLAUDE.md memory and SKILL.md skills, behind a held-out gate, staged for your review. Synthesizes SkillOpt (validation-gated skill optimization), Claude Dreams (offline memory consolidation), and agent sleep/consolidation.",
|
||||
"author": {
|
||||
"name": "Yifan Yang"
|
||||
},
|
||||
"category": "productivity",
|
||||
"source": {
|
||||
"source": "git-subdir",
|
||||
"url": "https://github.com/microsoft/SkillOpt.git",
|
||||
"path": "plugins/claude-code",
|
||||
"ref": "main"
|
||||
},
|
||||
"homepage": "https://github.com/microsoft/SkillOpt"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"name": "skillopt-sleep",
|
||||
"description": "Give your local Claude agent a nightly 'sleep cycle': it reviews your past sessions offline, replays recurring tasks on your own API budget, and consolidates what it learns into validated memory (CLAUDE.md) and skills (SKILL.md) so it gets better the more you use it. Synthesizes SkillOpt (validation-gated skill optimization), Claude Dreams (offline memory consolidation), and agent sleep/consolidation.",
|
||||
"version": "0.1.0",
|
||||
"author": {
|
||||
"name": "Yifan Yang",
|
||||
"email": "yifanyang@microsoft.com"
|
||||
},
|
||||
"homepage": "https://github.com/microsoft/SkillOpt",
|
||||
"repository": "https://github.com/microsoft/SkillOpt",
|
||||
"license": "MIT",
|
||||
"keywords": [
|
||||
"skillopt",
|
||||
"self-improvement",
|
||||
"memory-consolidation",
|
||||
"dreams",
|
||||
"sleep",
|
||||
"skills",
|
||||
"continual-learning",
|
||||
"offline-optimization"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
# SkillOpt-Sleep (Claude Code plugin)
|
||||
|
||||
> Give your local Claude agent a **sleep cycle**. Every night it reviews your
|
||||
> past sessions offline, replays your recurring tasks on your own API budget,
|
||||
> and consolidates what it learns into **validated** memory (`CLAUDE.md`) and
|
||||
> skills (`SKILL.md`). Your agent gets better the more you use it — no
|
||||
> model-weight training.
|
||||
|
||||
SkillOpt-Sleep is the **deployment-time** companion to
|
||||
[SkillOpt](https://github.com/microsoft/SkillOpt). SkillOpt trains a skill
|
||||
offline on a benchmark; SkillOpt-Sleep applies the same discipline to *your own
|
||||
daily usage*: bounded text edits, accepted only through a held-out validation
|
||||
gate, with rejected edits kept as negative feedback.
|
||||
|
||||
It synthesizes three ideas:
|
||||
|
||||
| Idea | Contribution |
|
||||
|---|---|
|
||||
| **SkillOpt** | skill/memory = trainable text; bounded add/delete/replace edits; **held-out gate** keeps only changes that help. |
|
||||
| **Claude Dreams** | offline consolidation over past sessions; input never mutated; output **reviewed then adopted**. |
|
||||
| **Agent sleep** | periodic offline replay turns short-term episodes into long-term skill. |
|
||||
|
||||
## What it does (one "night")
|
||||
|
||||
```
|
||||
harvest ~/.claude transcripts → mine recurring tasks → replay offline
|
||||
→ consolidate (reflect → bounded edit → GATE) → stage proposal → (you) adopt
|
||||
```
|
||||
|
||||
Nothing live is modified until **you** run `/skillopt-sleep adopt` (the Dreams "review,
|
||||
then adopt or discard" contract). Every adopt backs up the prior file first.
|
||||
|
||||
## Install
|
||||
|
||||
**Requirements:** Python ≥ 3.10, and the `claude` CLI (and/or `codex` CLI) on PATH.
|
||||
|
||||
```bash
|
||||
# 1) get the code (the plugin ships inside the SkillOpt repo)
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
cd SkillOpt
|
||||
|
||||
# 2) add the plugin to Claude Code as a local marketplace
|
||||
/plugin marketplace add ./skillopt-sleep-plugin
|
||||
/plugin install skillopt-sleep@skillopt-sleep
|
||||
|
||||
# 3) verify
|
||||
/skillopt-sleep status
|
||||
```
|
||||
|
||||
The plugin's bundled runner (`scripts/sleep.sh`) auto-selects a Python ≥ 3.10
|
||||
interpreter and calls the `skillopt_sleep` engine in the repo. No `pip install`
|
||||
is required for the default `mock` backend or for `claude`/`codex` backends —
|
||||
they shell out to the CLIs you already have.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# from inside any project you use with Claude Code:
|
||||
/skillopt-sleep dry-run # safe preview: what it would learn, no changes staged
|
||||
/skillopt-sleep run # full cycle: stages a reviewed proposal (still no live edits)
|
||||
/skillopt-sleep status # see history + the latest staged proposal
|
||||
/skillopt-sleep adopt # apply the staged proposal to CLAUDE.md / SKILL.md (with backup)
|
||||
|
||||
/skillopt-sleep-handoff run # same cycle, but THIS session answers the model calls
|
||||
# (no claude -p subprocess, no API key — subscription-friendly)
|
||||
```
|
||||
|
||||
Or call the engine directly (Python ≥ 3.10):
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep run --project "$(pwd)" --scope invoked --backend mock
|
||||
python -m skillopt_sleep run --project "$(pwd)" --backend claude # real lift via Claude
|
||||
python -m skillopt_sleep run --project "$(pwd)" --backend codex # real lift via Codex
|
||||
```
|
||||
|
||||
Default backend is **`mock`** — deterministic, no API spend — so you can try the
|
||||
plumbing for free. Switch to `--backend claude` or `--backend codex` for genuine
|
||||
improvement on your own budget.
|
||||
|
||||
### Handoff mode (session answers the model calls)
|
||||
|
||||
`--backend handoff` runs the cycle without any model subprocess: the engine
|
||||
executes the deterministic stages and writes every model call it needs to
|
||||
`.skillopt-sleep-handoff/PROMPTS.md` + `pending.json` (exit code 3). You (or
|
||||
the `/skillopt-sleep-handoff` command, which automates the loop with isolated
|
||||
fresh-context subagents) write each raw answer to `answers/<id>.md` and re-run
|
||||
the same command; it resumes from the answers and either finishes or stages
|
||||
the next batch. Typically 3–6 rounds per night.
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep run --backend handoff --project "$(pwd)"
|
||||
# ... answer .skillopt-sleep-handoff/PROMPTS.md into answers/<id>.md ...
|
||||
python -m skillopt_sleep run --backend handoff --project "$(pwd)" # resume
|
||||
```
|
||||
|
||||
Answer every prompt in a **fresh context** — a session that has already seen
|
||||
the mined tasks and their references would contaminate the held-out gate.
|
||||
Details: [the plugins README](../README.md#--backend-handoff--session-executed-calls-no-api-subprocess).
|
||||
|
||||
## Does it actually improve? (real models, public benchmark)
|
||||
|
||||
SkillOpt-Sleep is validated against [gbrain-evals](https://github.com/garrytan/gbrain-evals)'
|
||||
public `skillopt-v1` suite — the same benchmark gbrain scores its own skill
|
||||
optimizer against. We take a deliberately **deficient** skill and run one sleep
|
||||
night; held-out scoring is done by a local rule judge (no judge-API, no way to
|
||||
grade its own homework).
|
||||
|
||||
| Backend | Seed | Held-out before → after | Nights |
|
||||
|---|---|---|---|
|
||||
| **Claude (Haiku 4.5)** | brief-writer | **0.00 → 1.00** | 1 |
|
||||
| **Codex** | brief-writer | **0.00 → 1.00** | 2 |
|
||||
|
||||
Both took a brief-writer with no risks section / no confidence level and, within
|
||||
1–2 nights, proposed gated edits that lifted the held-out score to perfect —
|
||||
into the protected `LEARNED` block, nothing else touched. The Codex 2-night
|
||||
trace even shows the optimizer **diagnosing its own residual failure** and
|
||||
adding a meta-rule to fix it. Full writeup + reproduction:
|
||||
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
|
||||
|
||||
Reproduce:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/garrytan/gbrain-evals /tmp/gbrain-evals
|
||||
python -m skillopt_sleep.experiments.run_gbrain --backend claude --model haiku \
|
||||
--seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
|
||||
--nights 1 --limit-replay 3 --limit-holdout 3
|
||||
python -m skillopt_sleep.experiments.run_gbrain --backend codex \
|
||||
--seeds brief-writer --data-root /tmp/gbrain-evals/eval/data/skillopt-v1 \
|
||||
--nights 1 --limit-replay 3 --limit-holdout 3
|
||||
```
|
||||
|
||||
## Deterministic proof (no API, no keys)
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
|
||||
python -m skillopt_sleep.experiments.run_experiment --persona programmer --assert-improves
|
||||
```
|
||||
|
||||
Each prints the held-out score rising from baseline toward 1.0 as the gate
|
||||
accepts the general rules your tasks need, and confirms the gate **rejects** an
|
||||
injected harmful edit. Recorded output: [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
|
||||
|
||||
## Schedule it nightly
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/install-cron.sh" "$(pwd)" # prints a crontab line; installs nothing
|
||||
```
|
||||
|
||||
## Safety
|
||||
|
||||
- **Read-only** harvest of `~/.claude`. `mock` replay has no side effects.
|
||||
- Proposals are **staged**, never auto-applied (unless you opt in with `--auto-adopt`).
|
||||
- Every adopt writes a backup under the staging dir's `backup/`.
|
||||
- Per-night **token/task budget caps**; secrets redacted from prompts.
|
||||
- `fresh` replay (Phase 3) runs only in throwaway git worktrees.
|
||||
|
||||
## Status
|
||||
|
||||
Phase 1 (engine + deterministic experiment + plugin surface) is complete.
|
||||
Phase 3 adds the real-API miner/judge and `fresh` worktree replay. See
|
||||
[`docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md`](../docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md).
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
description: Run the SkillOpt-Sleep cycle with the handoff backend — no API subprocess; this session answers the engine's model calls via prompt/answer files, in isolated fresh-context subagents
|
||||
argument-hint: "[run | dry-run] [--preferences \"...\"] (default: run)"
|
||||
allowed-tools: Bash, Read, Write, Task
|
||||
---
|
||||
|
||||
# /skillopt-sleep-handoff — session-executed sleep cycle
|
||||
|
||||
You are driving **SkillOpt-Sleep in handoff mode**: the Python engine runs
|
||||
every deterministic stage (harvest → mine → replay scoring → gate → stage)
|
||||
and outsources each model call (attempt / judge / reflect) to YOU via
|
||||
prompt files. No `claude -p` subprocess, no API key — the model work runs
|
||||
on this session's budget, but each prompt MUST be answered in a fresh,
|
||||
isolated context so the validation gate stays honest.
|
||||
|
||||
## Requested action: $ARGUMENTS
|
||||
|
||||
(If `$ARGUMENTS` is empty, treat it as `run`.)
|
||||
|
||||
## The loop
|
||||
|
||||
Repeat until the engine exits 0 (done) — at most 8 rounds:
|
||||
|
||||
1. **Run the engine** via the bundled runner:
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --backend handoff --project "$(pwd)" --scope invoked
|
||||
```
|
||||
|
||||
- exit 0 → the night is complete; go to "Finish" below.
|
||||
- exit 3 → pending model calls; continue with step 2.
|
||||
- anything else → stop and show the user the error output.
|
||||
|
||||
2. **Read the batch**: `Read` `.skillopt-sleep-handoff/pending.json` in the
|
||||
project. Each entry has `id`, `prompt`, `max_tokens`, `answer_file`.
|
||||
|
||||
3. **Answer each prompt in ISOLATION** — this is the integrity rule:
|
||||
- For each entry, launch a subagent (Task tool) whose ENTIRE input is
|
||||
the `prompt` text verbatim. Add nothing: no summary of this session,
|
||||
no mention of SkillOpt, no other prompts from the batch.
|
||||
- Take the subagent's reply and `Write` the raw answer text (no
|
||||
commentary, no code fences) to the entry's `answer_file`.
|
||||
- NEVER answer from this session's own context — you have seen the
|
||||
mined tasks and their references, so inline answers would contaminate
|
||||
the held-out gate and fake the improvement score.
|
||||
|
||||
4. **Re-run the same engine command** — it resumes from the answers
|
||||
directory and either finishes or stages the next batch.
|
||||
|
||||
## Finish
|
||||
|
||||
- `Read` the `report.md` in the staging dir the engine printed and show
|
||||
the user: held-out baseline → candidate score, the gate decision, the
|
||||
proposed edits, and where the proposal is staged.
|
||||
- Tell the user nothing live changed; offer `/skillopt-sleep adopt`.
|
||||
- The engine archives `.skillopt-sleep-handoff/` on a completed real run;
|
||||
do not delete it yourself.
|
||||
|
||||
## Safety reminders
|
||||
|
||||
- **Never** edit `CLAUDE.md` or `SKILL.md` yourself — only `adopt` does
|
||||
that, with a backup.
|
||||
- Mined tasks are pinned to `.skillopt-sleep-handoff/tasks.json` on round
|
||||
one, so sessions created while answering prompts cannot shift the task
|
||||
set. Do not edit that file.
|
||||
- If a batch looks like it contains secrets or content the user would not
|
||||
want re-processed, stop and ask before answering.
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
description: Run or manage the SkillOpt-Sleep self-evolution cycle (review past sessions, replay tasks offline, consolidate validated memory + skills; can also schedule nightly runs)
|
||||
argument-hint: "[run | dry-run | status | adopt | harvest | schedule | unschedule] (default: status)"
|
||||
allowed-tools: Bash, Read
|
||||
---
|
||||
|
||||
# /skillopt-sleep — SkillOpt-Sleep nightly self-evolution
|
||||
|
||||
You are driving **SkillOpt-Sleep**: a tool that lets this user's Claude agent
|
||||
improve offline by reviewing past sessions, replaying recurring tasks, and
|
||||
consolidating what it learns into **validated** memory (`CLAUDE.md`) and skills
|
||||
(`SKILL.md`). It is gated like SkillOpt: a change is kept only if it improves a
|
||||
held-out replay score, and nothing live is modified until the user adopts it.
|
||||
|
||||
## Requested action: $ARGUMENTS
|
||||
|
||||
(If `$ARGUMENTS` is empty, treat it as `status`.)
|
||||
|
||||
## How to run it
|
||||
|
||||
The engine is the `skillopt_sleep` Python package in this repo. Use the
|
||||
**plugin's bundled runner** so the right interpreter and repo are on the path:
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" <action> --project "$(pwd)" --scope invoked
|
||||
```
|
||||
|
||||
`<action>` is one of:
|
||||
|
||||
| action | what it does |
|
||||
|--------------|--------------|
|
||||
| `status` | show how many nights have run + the latest staged proposal (READ-ONLY) |
|
||||
| `dry-run` | harvest → mine → replay → report, but **stage nothing** (safe preview) |
|
||||
| `run` | full cycle: also **stage** a reviewed proposal (still does NOT touch live files) |
|
||||
| `adopt` | apply the latest staged proposal to live `CLAUDE.md` / `SKILL.md` (backs up first) |
|
||||
| `harvest` | debug: print the recurring tasks mined from recent sessions |
|
||||
| `schedule` | install a nightly cron entry for this project (`--hour --minute`, off-:00 by default) |
|
||||
| `unschedule` | remove the nightly cron entry (`--all` to remove every managed entry) |
|
||||
|
||||
Default backend is `mock` (deterministic, no API spend). To use real budget for
|
||||
genuine improvement, add `--backend claude` or `--backend codex`. To steer what
|
||||
the optimizer writes, add `--preferences "<your house rules>"`.
|
||||
|
||||
## Steps to follow
|
||||
|
||||
1. **Run the requested action** via the bundled runner above. Capture stdout.
|
||||
2. **For `run` / `dry-run`:** after it completes, `Read` the generated
|
||||
`report.md` in the staging dir it prints, and show the user:
|
||||
- held-out score: baseline → candidate (the proof it helped)
|
||||
- the gate decision (accept/reject) and the exact edits it proposes
|
||||
- where the proposal is staged
|
||||
3. **For `run` that produced an accepted proposal:** tell the user the diff is
|
||||
staged and that **nothing live changed yet**. Offer to run `/skillopt-sleep adopt`.
|
||||
4. **For `adopt`:** confirm which live files were updated and that backups were
|
||||
written under the staging dir's `backup/`.
|
||||
5. **Never** edit `CLAUDE.md` or `SKILL.md` yourself — only the `adopt` action
|
||||
does that, with a backup. Respect the review gate.
|
||||
|
||||
## Safety reminders
|
||||
|
||||
- Harvest is **read-only** over `~/.claude`. Replay in `mock` mode runs no
|
||||
shell side effects.
|
||||
- The cycle stages proposals; the user is in control of adoption.
|
||||
- If the user asks to schedule this nightly, point them at
|
||||
`${CLAUDE_PLUGIN_ROOT}/scripts/install-cron.sh` (prints a crontab line; does
|
||||
not install anything without confirmation).
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"hooks": {
|
||||
"SessionEnd": [
|
||||
{
|
||||
"matcher": "*",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "\"${CLAUDE_PLUGIN_ROOT}/hooks/on-session-end.sh\"",
|
||||
"async": true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/usr/bin/env bash
|
||||
# SkillOpt-Sleep SessionEnd hook (async, best-effort, NON-BLOCKING).
|
||||
#
|
||||
# This does NOT run the optimizer. It only appends a tiny marker so the next
|
||||
# nightly cycle knows there is fresh activity to harvest, and (optionally)
|
||||
# nudges the user once that a sleep cycle is available. It must never fail the
|
||||
# session or spend API budget.
|
||||
set -uo pipefail
|
||||
|
||||
PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
|
||||
STATE_DIR="${HOME}/.skillopt-sleep"
|
||||
mkdir -p "$STATE_DIR" 2>/dev/null || exit 0
|
||||
|
||||
# Record that a session just ended (cheap; used for "is there new data?").
|
||||
printf '%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${PWD}" \
|
||||
>> "$STATE_DIR/session-end.log" 2>/dev/null || true
|
||||
|
||||
exit 0
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
# Print (does NOT install) a crontab line that runs SkillOpt-Sleep nightly.
|
||||
# The user copies the line into `crontab -e` if they want it.
|
||||
set -euo pipefail
|
||||
|
||||
PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
|
||||
RUNNER="$PLUGIN_ROOT/scripts/sleep.sh"
|
||||
PROJECT="${1:-$(pwd)}"
|
||||
BACKEND="${2:-mock}"
|
||||
|
||||
# 3:17am local — deliberately off the :00 mark so many users don't all hit the
|
||||
# API at once (and we leave room for jitter).
|
||||
MIN=17
|
||||
HOUR=3
|
||||
|
||||
cat <<EOF
|
||||
# ── SkillOpt-Sleep nightly cycle ────────────────────────────────────────────
|
||||
# Review past sessions, replay tasks, stage validated memory/skill updates.
|
||||
# Runs at ${HOUR}:$(printf '%02d' $MIN) local every day. Output goes to the project's
|
||||
# .skillopt-sleep/ dir; nothing live is changed until you run '/skillopt-sleep adopt'
|
||||
# (unless you pass --auto-adopt below).
|
||||
#
|
||||
# Copy the next line into 'crontab -e':
|
||||
${MIN} ${HOUR} * * * "${RUNNER}" run --project "${PROJECT}" --scope invoked --backend ${BACKEND} >> "${PROJECT}/.skillopt-sleep/cron.log" 2>&1
|
||||
#
|
||||
# For fully-autonomous adoption (power users), append: --auto-adopt
|
||||
# To spend real API budget for genuine lift, set BACKEND=anthropic above.
|
||||
# ────────────────────────────────────────────────────────────────────────────
|
||||
EOF
|
||||
Executable
+51
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
# SkillOpt-Sleep shared runner — used by all platform plugins (Claude Code,
|
||||
# Codex, Copilot). Resolves the repo root (which contains the skillopt_sleep
|
||||
# package), picks a Python >= 3.10, and execs the engine CLI.
|
||||
#
|
||||
# Usage: run-sleep.sh <run|dry-run|status|adopt|harvest|...> [args...]
|
||||
set -euo pipefail
|
||||
|
||||
# This script lives at <repo>/plugins/run-sleep.sh, so the repo root (which
|
||||
# holds skillopt_sleep/) is one level up. CLAUDE_PLUGIN_ROOT (if set by Claude
|
||||
# Code) points at the plugin dir; the engine is then two levels above it.
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
if [ -d "$SCRIPT_DIR/../skillopt_sleep" ]; then
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
elif [ -n "${CLAUDE_PLUGIN_ROOT:-}" ] && [ -d "$CLAUDE_PLUGIN_ROOT/../../skillopt_sleep" ]; then
|
||||
REPO_ROOT="$(cd "$CLAUDE_PLUGIN_ROOT/../.." && pwd)"
|
||||
elif [ -n "${SKILLOPT_SLEEP_REPO:-}" ] && [ -d "$SKILLOPT_SLEEP_REPO/skillopt_sleep" ]; then
|
||||
REPO_ROOT="$SKILLOPT_SLEEP_REPO"
|
||||
else
|
||||
# last resort: search upward from CWD
|
||||
d="$PWD"
|
||||
while [ "$d" != "/" ]; do
|
||||
[ -d "$d/skillopt_sleep" ] && { REPO_ROOT="$d"; break; }
|
||||
d="$(dirname "$d")"
|
||||
done
|
||||
fi
|
||||
if [ -z "${REPO_ROOT:-}" ]; then
|
||||
echo "[sleep] ERROR: could not locate the skillopt_sleep package. Set SKILLOPT_SLEEP_REPO to the repo root." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PY=""
|
||||
# Allow explicit Python override (useful on macOS with old system Python).
|
||||
if [ -n "${SKILLOPT_SLEEP_PYTHON:-}" ]; then
|
||||
PY="$SKILLOPT_SLEEP_PYTHON"
|
||||
else
|
||||
for cand in python3.12 python3.11 python3.10 python3; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then
|
||||
ver="$("$cand" -c 'import sys; print("%d%d" % sys.version_info[:2])' 2>/dev/null || echo 0)"
|
||||
if [ "${ver:-0}" -ge 310 ]; then PY="$cand"; break; fi
|
||||
fi
|
||||
done
|
||||
fi
|
||||
if [ -z "$PY" ]; then
|
||||
echo "[sleep] ERROR: need Python >= 3.10 (found none)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$#" -eq 0 ]; then set -- status; fi
|
||||
cd "$REPO_ROOT"
|
||||
exec "$PY" -m skillopt_sleep "$@"
|
||||
Executable
+30
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Claude Code plugin runner — thin wrapper over the shared runner so all
|
||||
# platform plugins share one engine launcher.
|
||||
#
|
||||
# After marketplace install the plugin is isolated in a cache directory and
|
||||
# the repo-relative path no longer works. We try four locations:
|
||||
# 1. Co-located run-sleep.sh (bundled copy — works in marketplace cache)
|
||||
# 2. Repo-relative ../../run-sleep.sh (dev checkout)
|
||||
# 3. CLAUDE_PLUGIN_ROOT/../run-sleep.sh (plugin env variable)
|
||||
# 4. SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh (explicit env)
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
SHARED=""
|
||||
if [ -f "$HERE/run-sleep.sh" ]; then
|
||||
SHARED="$HERE/run-sleep.sh"
|
||||
elif [ -f "$(cd "$HERE/../.." 2>/dev/null && pwd)/run-sleep.sh" ]; then
|
||||
SHARED="$(cd "$HERE/../.." && pwd)/run-sleep.sh"
|
||||
elif [ -n "${CLAUDE_PLUGIN_ROOT:-}" ] && [ -f "$(cd "$CLAUDE_PLUGIN_ROOT/.." 2>/dev/null && pwd)/run-sleep.sh" ]; then
|
||||
SHARED="$(cd "$CLAUDE_PLUGIN_ROOT/.." && pwd)/run-sleep.sh"
|
||||
elif [ -n "${SKILLOPT_SLEEP_REPO:-}" ] && [ -f "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" ]; then
|
||||
SHARED="$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh"
|
||||
fi
|
||||
|
||||
if [ -z "$SHARED" ]; then
|
||||
echo "[sleep] ERROR: cannot locate run-sleep.sh." >&2
|
||||
echo "[sleep] Set SKILLOPT_SLEEP_REPO to the SkillOpt repo root, or pip install skillopt." >&2
|
||||
exit 1
|
||||
fi
|
||||
exec bash "$SHARED" "$@"
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
name: skillopt-sleep
|
||||
description: "Use when the user wants their Claude agent to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, memory/skill consolidation, or says things like 'make my agent better the more I use it', 'review my past sessions', 'learn my preferences', 'consolidate what you learned', 'run the sleep cycle', or wants to schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated CLAUDE.md/SKILL.md behind a held-out gate."
|
||||
---
|
||||
|
||||
# SkillOpt-Sleep: offline self-evolution for a local Claude agent
|
||||
|
||||
SkillOpt-Sleep gives the user's agent a **sleep cycle**. While the user is
|
||||
offline (e.g. nightly), it reviews their real past Claude Code sessions,
|
||||
re-runs recurring tasks on their own API budget, and consolidates what it
|
||||
learns into **memory** (`CLAUDE.md`) and **skills** (`SKILL.md`) — but only
|
||||
keeps changes that pass a held-out validation gate, and only after the user
|
||||
adopts them. The agent gets measurably better at *this* user's recurring work,
|
||||
with no model-weight training. It is the deployment-time analogue of training:
|
||||
short-term experience → long-term competence.
|
||||
|
||||
It synthesizes three ideas:
|
||||
- **SkillOpt** — the skill/memory doc is trainable text; bounded add/delete/replace
|
||||
edits; accepted only through a held-out gate; rejected edits become negative feedback.
|
||||
- **Claude Dreams** — offline consolidation that reads past sessions and rebuilds
|
||||
memory (dedup/merge/resolve); the input is never mutated; output is reviewed then adopted.
|
||||
- **Agent sleep** — periodic offline replay turns episodes into durable skill.
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Trigger when the user wants any of:
|
||||
- "make my agent learn from how I use it" / "get better the more I use it" / "remember my preferences across sessions"
|
||||
- a nightly/scheduled or on-demand **offline self-improvement / dream / sleep** run
|
||||
- to **review past sessions/trajectories** and distill recurring tasks
|
||||
- to **consolidate** feedback into `CLAUDE.md` or a managed skill
|
||||
- to **schedule** the cycle (cron) or **adopt** a staged proposal
|
||||
|
||||
## The cycle (six stages)
|
||||
|
||||
1. **Harvest** — read `~/.claude/projects/*/<session>.jsonl` + `~/.claude/history.jsonl` (READ-ONLY) → session digests.
|
||||
2. **Mine** — digests → `TaskRecord`s (recurring intents + outcome labels + checkable refs where possible).
|
||||
3. **Replay** — re-run tasks offline under the *current* skill+memory → (hard, soft) scores.
|
||||
4. **Consolidate** — reflect on failures → propose bounded edits → **gate** on a held-out slice; accept only if it strictly improves.
|
||||
5. **Stage** — write `proposed_CLAUDE.md`, `proposed_SKILL.md`, a diff, and `report.md` into `<project>/.skillopt-sleep/staging/<date>/`. **Nothing live changes.**
|
||||
6. **Adopt** — explicit (or opt-in auto): copy staged files over live ones, backing up first.
|
||||
|
||||
## How to drive it
|
||||
|
||||
Prefer the `/skillopt-sleep` command. Under the hood it calls the bundled runner:
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" status # what's happened
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" dry-run --project "$(pwd)" # safe preview
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" run --project "$(pwd)" # full cycle, stages a proposal
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" adopt --project "$(pwd)" # apply staged proposal (with backup)
|
||||
```
|
||||
|
||||
- Default backend is `mock` (deterministic, **no API spend**) — good for trying the plumbing.
|
||||
- Add `--backend claude` or `--backend codex` to spend the user's real budget for genuine improvement.
|
||||
- Scope defaults to the invoked project; `--scope all` harvests every project.
|
||||
|
||||
### Scheduling
|
||||
|
||||
```bash
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" schedule --project "$(pwd)" --hour 3 --minute 17
|
||||
"${CLAUDE_PLUGIN_ROOT}/scripts/sleep.sh" unschedule --project "$(pwd)"
|
||||
```
|
||||
|
||||
Installs a nightly cron entry. `unschedule --all` removes every managed entry.
|
||||
|
||||
## All CLI flags
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `--project PATH` | cwd | Project directory to evolve |
|
||||
| `--scope all\|invoked` | invoked | Harvest scope |
|
||||
| `--backend mock\|claude\|codex\|copilot` | mock | Replay backend (mock = no API spend) |
|
||||
| `--model NAME` | backend default | Override the model used for replay |
|
||||
| `--source claude\|codex\|auto` | claude | Transcript source |
|
||||
| `--lookback-hours N` | 72 | Harvest window |
|
||||
| `--max-sessions N` | unlimited | Cap harvested sessions |
|
||||
| `--max-tasks N` | 40 | Cap mined tasks |
|
||||
| `--target-skill-path PATH` | auto | Explicit SKILL.md to evolve |
|
||||
| `--tasks-file PATH` | — | Reviewed TaskRecord JSON (skip harvest) |
|
||||
| `--progress` | off | Print phase progress to stderr |
|
||||
| `--auto-adopt` | off | Auto-adopt if gate passes |
|
||||
| `--edit-budget N` | 4 | Max bounded edits per night |
|
||||
| `--json` | off | Machine-readable JSON output |
|
||||
|
||||
## Config keys (`~/.skillopt-sleep/config.json`)
|
||||
|
||||
Beyond the CLI flags, advanced behavior is controlled via config:
|
||||
|
||||
- **`preferences`** — free-text house rules injected into the optimizer's reflect step (e.g. "Always use async/await", "Answers in `\boxed{}`").
|
||||
- **`gate_mode`** — `on` (default, validation-gated) or `off` (greedy, accept all edits).
|
||||
- **`gate_metric`** — `hard`, `soft`, or `mixed` (default). Controls how the held-out gate scores.
|
||||
- **`dream_rollouts`** — >1 enables multi-rollout contrastive reflection per task.
|
||||
- **`recall_k`** — >0 recalls K similar past tasks into the dream (long-term memory).
|
||||
- **`evolve_memory`** / **`evolve_skill`** — independently toggle CLAUDE.md vs SKILL.md consolidation.
|
||||
|
||||
## Memory consolidation
|
||||
|
||||
The sleep cycle can consolidate both:
|
||||
- **SKILL.md** — the managed skill file (bounded edits: add/delete/replace)
|
||||
- **CLAUDE.md** — the project memory (same bounded edits)
|
||||
|
||||
Both are gated by the same held-out validation score. Set `evolve_memory: false` to consolidate only skills, or `evolve_skill: false` for only memory.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- **Never** hand-edit the user's `CLAUDE.md` / `SKILL.md` as part of this skill.
|
||||
Only the `adopt` action changes live files, and it backs them up first.
|
||||
- Harvest is read-only. `mock` replay has no side effects.
|
||||
- Always show the user the **held-out baseline → candidate** score and the
|
||||
exact proposed edits before suggesting adoption. Evidence before adoption.
|
||||
- If asked whether it really helps, run
|
||||
`python -m skillopt_sleep.experiments.run_experiment --persona researcher --json`
|
||||
— a deterministic demo that proves held-out lift and that the gate blocks
|
||||
harmful edits.
|
||||
|
||||
## Validate / demo
|
||||
|
||||
```bash
|
||||
# deterministic proof (no API): held-out score rises, gate blocks regressions
|
||||
python -m skillopt_sleep.experiments.run_experiment --persona researcher --assert-improves
|
||||
python -m skillopt_sleep.experiments.run_experiment --persona programmer --assert-improves
|
||||
```
|
||||
|
||||
See the SkillOpt-Sleep guide section for recorded output and
|
||||
`docs/superpowers/specs/2026-06-07-skillopt-sleep-claude-code-plugin-design.md`
|
||||
for the full design.
|
||||
@@ -0,0 +1,96 @@
|
||||
# SkillOpt-Sleep — Codex integration
|
||||
|
||||
Give your **Codex** agent a nightly **sleep cycle**: it reviews past sessions
|
||||
offline, replays your recurring tasks on your own Codex budget, and consolidates
|
||||
what it learns into validated memory + skills behind a held-out gate. Same engine
|
||||
as the Claude Code plugin (`skillopt_sleep`), wrapped for Codex.
|
||||
|
||||
> **Verified on Codex:** on the public
|
||||
> [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1`
|
||||
> benchmark, a deliberately deficient skill goes **0.00 → 1.00** on a held-out
|
||||
> set with the Codex backend (incl. the tool-use seed via a real tool loop).
|
||||
> See [the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
|
||||
|
||||
## What Codex supports (and what we use)
|
||||
|
||||
Codex (`@openai/codex`) extends via **`AGENTS.md`** instructions, **skills** at
|
||||
`~/.agents/skills/<name>/SKILL.md`, and plugins that can distribute skills.
|
||||
Custom prompts are deprecated in Codex, so this integration is skill-first: the
|
||||
installed `skillopt-sleep` skill contains the launch commands and operating
|
||||
rules. The shared runner remains a plain shell entrypoint that the skill calls.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
git clone <repo-url> SkillOpt-Sleep
|
||||
cd SkillOpt-Sleep
|
||||
bash plugins/codex/install.sh # installs the skill
|
||||
export SKILLOPT_SLEEP_REPO="$(pwd)" # so the runner is found from anywhere
|
||||
```
|
||||
|
||||
If a previous install created `~/.codex/prompts/sleep.md`, the installer moves
|
||||
that deprecated prompt aside with a `.skillopt-legacy*.bak` suffix.
|
||||
|
||||
Requires Python ≥ 3.10 and the `codex` CLI on PATH.
|
||||
|
||||
## Use
|
||||
|
||||
Mention `$skillopt-sleep` where Codex supports explicit skill mentions, or ask
|
||||
Codex in natural language:
|
||||
|
||||
```text
|
||||
Use the skillopt-sleep skill to run status for this project.
|
||||
Use the skillopt-sleep skill to run a dry-run for this project.
|
||||
Use the skillopt-sleep skill to run the full cycle for this project with the Codex backend.
|
||||
Use the skillopt-sleep skill to adopt the latest staged proposal.
|
||||
```
|
||||
|
||||
Or call the engine directly:
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep dry-run --project "$(pwd)" --source codex --backend mock
|
||||
python -m skillopt_sleep run --project "$(pwd)" --source codex --backend codex \
|
||||
--max-sessions 5 --max-tasks 3 --progress
|
||||
python -m skillopt_sleep run --project "$(pwd)" --source codex --backend codex \
|
||||
--target-skill-path .agents/skills/example/SKILL.md \
|
||||
--max-sessions 5 --max-tasks 3 --progress
|
||||
```
|
||||
|
||||
`--source codex` reads Codex Desktop archived sessions from
|
||||
`~/.codex/archived_sessions`. Use `--codex-home /path/to/.codex` to point at a
|
||||
different Codex home, or `--source auto` to try Codex archives first and fall
|
||||
back to Claude Code transcripts. Default backend is `mock` (no API spend).
|
||||
`--backend codex` uses your Codex budget for real improvement. Bound live runs
|
||||
with `--max-sessions` and `--max-tasks`; add `--progress` because Codex-backed
|
||||
mining, replay, and reflection can be slow and otherwise quiet. Use
|
||||
`--target-skill-path` to stage/adopt into a repo-scoped Codex skill such as
|
||||
`.agents/skills/<name>/SKILL.md`; target runs over-sample mined tasks and
|
||||
prefer tasks that match the target skill's path, headings, and content. All the
|
||||
controllable knobs (`--gate on|off`, `--rollouts-k`, `--budget-tokens`,
|
||||
`--preferences`, optimizer/target split) work identically — see
|
||||
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
|
||||
|
||||
For privacy-sensitive projects, split the run into reviewable steps:
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep harvest --project "$(pwd)" --source codex \
|
||||
--target-skill-path .agents/skills/example/SKILL.md \
|
||||
--max-sessions 5 --max-tasks 3 \
|
||||
--output reviewed-tasks.json
|
||||
|
||||
python -m skillopt_sleep dry-run --project "$(pwd)" --backend codex \
|
||||
--tasks-file reviewed-tasks.json --progress --json
|
||||
```
|
||||
|
||||
Inspect/redact the JSON and set `"reviewed": true` before using a real backend.
|
||||
`--tasks-file` skips archive harvest/mining and replays only the reviewed JSON
|
||||
tasks; real backends refuse task files still marked `"reviewed": false`.
|
||||
|
||||
## Notes / status
|
||||
|
||||
- Codex's `exec` runs shell, so the real-tool-loop replay (e.g. the
|
||||
`tool_called: search` benchmark seed) works natively.
|
||||
- This integration no longer installs a `.codex/prompts` slash command. Skills
|
||||
are the reusable Codex workflow surface; mention `skillopt-sleep` explicitly
|
||||
or ask for a sleep/dream/offline self-improvement run and Codex can load the
|
||||
skill.
|
||||
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# Install the SkillOpt-Sleep Codex integration as a user-level Codex skill.
|
||||
# Idempotent; prints what it does.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
CODEX_HOME="${CODEX_HOME:-$HOME/.codex}"
|
||||
AGENTS_SKILLS="${HOME}/.agents/skills"
|
||||
LEGACY_PROMPT="$CODEX_HOME/prompts/sleep.md"
|
||||
|
||||
echo "[install] repo: $REPO_ROOT"
|
||||
|
||||
# 1) user-level skill
|
||||
mkdir -p "$AGENTS_SKILLS/skillopt-sleep"
|
||||
cp "$REPO_ROOT/plugins/codex/skills/skillopt-sleep/SKILL.md" "$AGENTS_SKILLS/skillopt-sleep/SKILL.md"
|
||||
echo "[install] skill -> $AGENTS_SKILLS/skillopt-sleep/SKILL.md"
|
||||
|
||||
# 2) retire the old custom prompt entrypoint from previous installs
|
||||
if [ -f "$LEGACY_PROMPT" ]; then
|
||||
backup="${LEGACY_PROMPT}.skillopt-legacy.bak"
|
||||
if [ -e "$backup" ]; then
|
||||
backup="${LEGACY_PROMPT}.skillopt-legacy.$(date +%Y%m%d%H%M%S).bak"
|
||||
fi
|
||||
mv "$LEGACY_PROMPT" "$backup"
|
||||
echo "[install] legacy prompt -> $backup"
|
||||
fi
|
||||
|
||||
# 3) record the repo location so the runner is found from anywhere
|
||||
echo "[install] add to your shell profile:"
|
||||
echo " export SKILLOPT_SLEEP_REPO=\"$REPO_ROOT\""
|
||||
|
||||
# 4) optional: append an AGENTS.md hint (only if the user opts in)
|
||||
cat <<EOF
|
||||
|
||||
[install] Optional — add this to ~/.codex/AGENTS.md so Codex always knows the tool:
|
||||
|
||||
## SkillOpt-Sleep
|
||||
Use the skillopt-sleep skill when I ask to run a sleep/dream/offline
|
||||
self-improvement cycle. The runner is:
|
||||
\`bash "$REPO_ROOT/plugins/run-sleep.sh" status --project "\$(pwd)"\`.
|
||||
|
||||
Done. Try asking Codex:
|
||||
Use the skillopt-sleep skill to run status for this project.
|
||||
EOF
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
name: skillopt-sleep
|
||||
description: "Use when the user wants Codex to self-improve from past usage, asks about a nightly/offline 'sleep' or 'dream' cycle, wants Codex to review past sessions, learn preferences, consolidate memory/skills, run dry-run/run/adopt/status for SkillOpt-Sleep, or schedule offline self-optimization. Drives the skillopt_sleep engine: harvest past sessions -> mine recurring tasks -> replay offline -> consolidate validated memory + skills behind a held-out gate."
|
||||
---
|
||||
|
||||
# SkillOpt-Sleep: offline self-evolution for a local Codex agent
|
||||
|
||||
SkillOpt-Sleep gives the user's Codex agent a sleep cycle. While the user is
|
||||
offline or on demand, it reviews past local sessions, re-runs recurring tasks
|
||||
on the user's own budget, and consolidates what it learns into memory and
|
||||
skills. It keeps only changes that pass a held-out validation gate, and live
|
||||
files change only after the user explicitly adopts a staged proposal. There is
|
||||
no model-weight training.
|
||||
|
||||
## When to use
|
||||
|
||||
Trigger when the user wants any of:
|
||||
|
||||
- Codex to learn from past sessions or get better the more they use it;
|
||||
- a nightly/scheduled or on-demand sleep/dream/offline self-improvement run;
|
||||
- to review past sessions and distill recurring tasks;
|
||||
- to consolidate feedback into memory or managed skills;
|
||||
- to run `status`, `harvest`, `dry-run`, `run`, or `adopt` for SkillOpt-Sleep.
|
||||
|
||||
## The cycle
|
||||
|
||||
1. **Harvest** - read local session transcripts according to the engine
|
||||
configuration and normalize them into session digests.
|
||||
2. **Mine** - turn digests into recurring `TaskRecord`s with outcomes and
|
||||
checkable references where possible.
|
||||
3. **Replay** - re-run mined tasks offline under the current skill and memory.
|
||||
4. **Consolidate** - reflect on failures and propose bounded edits.
|
||||
5. **Gate** - accept edits only when the held-out validation score improves.
|
||||
6. **Stage** - write the proposal under
|
||||
`<project>/.skillopt-sleep/staging/<date>/`; nothing live changes.
|
||||
7. **Adopt** - only after explicit user approval, copy staged files over live
|
||||
files with backups.
|
||||
|
||||
## How to drive it
|
||||
|
||||
Invoke the bundled runner via shell (Codex `exec` has shell access). The runner
|
||||
finds the engine and a Python >= 3.10 automatically.
|
||||
|
||||
```bash
|
||||
# point at the repo if it isn't auto-detected from CWD:
|
||||
export SKILLOPT_SLEEP_REPO=/path/to/SkillOpt-Sleep
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" status --project "$(pwd)"
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" harvest --project "$(pwd)"
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" dry-run --project "$(pwd)" --backend mock
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --backend codex
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" run --project "$(pwd)" --source codex # harvest from Codex Desktop
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" adopt --project "$(pwd)"
|
||||
```
|
||||
|
||||
Actions are `status`, `harvest`, `dry-run`, `run`, `adopt`, `schedule`, and `unschedule`.
|
||||
|
||||
- Default backend is `mock`, which is deterministic and spends no API budget.
|
||||
- `--backend codex` uses the user's Codex budget for real improvement.
|
||||
- `--source codex` reads Codex Desktop archived sessions from `~/.codex/archived_sessions`;
|
||||
use `--codex-home /path/to/.codex` if the archive lives elsewhere.
|
||||
- Keep `dry-run --backend mock` as the first smoke check unless the user
|
||||
explicitly asked for a real optimization run.
|
||||
|
||||
### Scheduling
|
||||
|
||||
```bash
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" schedule --project "$(pwd)" --hour 3 --minute 17
|
||||
bash "$SKILLOPT_SLEEP_REPO/plugins/run-sleep.sh" unschedule --project "$(pwd)"
|
||||
```
|
||||
|
||||
Installs a nightly cron entry. `unschedule --all` removes every managed entry.
|
||||
|
||||
### All backends
|
||||
|
||||
- `--backend mock` — deterministic, no API spend (default)
|
||||
- `--backend claude` — uses the Claude CLI
|
||||
- `--backend codex` — uses the Codex CLI
|
||||
- `--backend copilot` — uses the GitHub Copilot CLI
|
||||
|
||||
### Additional flags
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--auto-adopt` | Auto-adopt if the gate passes (default: stage only) |
|
||||
| `--edit-budget N` | Max bounded edits per night (default: 4) |
|
||||
| `--lookback-hours N` | Harvest window in hours (default: 72) |
|
||||
| `--json` | Machine-readable JSON output |
|
||||
|
||||
### Config keys (`~/.skillopt-sleep/config.json`)
|
||||
|
||||
- **`preferences`** — free-text house rules for the optimizer
|
||||
- **`gate_mode`** — `on` (validation-gated, default) or `off` (greedy)
|
||||
- **`gate_metric`** — `hard` | `soft` | `mixed` (default)
|
||||
- **`dream_rollouts`** — >1 for multi-rollout contrastive reflection
|
||||
- **`recall_k`** — >0 recalls similar past tasks from the archive
|
||||
|
||||
### Memory consolidation
|
||||
|
||||
The sleep cycle consolidates both **memory** (AGENTS.md / CLAUDE.md) and **skills** (SKILL.md) by default. Each is independently toggleable via `evolve_memory` / `evolve_skill` config keys. Both are gated by the same held-out validation score.
|
||||
|
||||
## Steps
|
||||
|
||||
1. Run the requested action; capture stdout.
|
||||
2. For `dry-run` and `run`, report the held-out baseline -> candidate score,
|
||||
gate action, task count, session count, and exact proposed edits.
|
||||
3. If a staging directory is printed, read `report.md` before summarizing.
|
||||
4. `run` only stages a proposal; nothing live changes until `adopt`.
|
||||
5. Offer adoption only after the user has reviewed the staged proposal.
|
||||
6. Never hand-edit the user's `AGENTS.md`, memory, or skills as a substitute
|
||||
for `adopt`; adoption is the safety boundary and writes backups first.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Harvest is read-only. Do not edit archived sessions or raw transcripts.
|
||||
- Keep raw secrets, credentials, private user data, and unsanitized transcript
|
||||
contents out of messages, logs, generated artifacts, and commits.
|
||||
- Show validation evidence before recommending adoption.
|
||||
- Treat generated edits as proposals, not as source of truth.
|
||||
- Do not rely on deprecated custom prompts or `/sleep` slash commands for this
|
||||
Codex integration. This skill is the entrypoint.
|
||||
|
||||
## Validate
|
||||
|
||||
```bash
|
||||
python -m skillopt_sleep dry-run --project "$(pwd)" --backend mock --json
|
||||
python -m skillopt_sleep.experiments.run_gbrain --backend codex \
|
||||
--seeds brief-writer --data-root /path/to/gbrain-evals/eval/data/skillopt-v1 \
|
||||
--nights 2 --limit-replay 3 --limit-holdout 3
|
||||
```
|
||||
|
||||
A deficient skill goes 0.00 -> 1.00 on a held-out set; the optimizer's edits
|
||||
are gated on real-task performance.
|
||||
@@ -0,0 +1,76 @@
|
||||
# SkillOpt-Sleep — GitHub Copilot integration
|
||||
|
||||
Give **Copilot** (CLI or VS Code) a nightly **sleep cycle** via a tiny **MCP
|
||||
server** that exposes the `skillopt_sleep` engine as tools. MCP is GitHub's
|
||||
supported way to extend Copilot, so this works across Copilot CLI, VS Code, and
|
||||
other MCP clients with the same server.
|
||||
|
||||
## What's here
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `mcp_server.py` | stdlib-only MCP (stdio) server exposing `sleep_*` tools |
|
||||
| `mcp-config.example.json` | drop-in MCP server config |
|
||||
| `copilot-instructions.snippet.md` | paste into `.github/copilot-instructions.md` |
|
||||
|
||||
## Install
|
||||
|
||||
Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
|
||||
|
||||
1. **Register the MCP server.** Add the server to your Copilot MCP config
|
||||
(Copilot CLI: `~/.copilot/mcp-config.json`; VS Code: your MCP settings).
|
||||
Use `mcp-config.example.json` as a template — set `SKILLOPT_SLEEP_REPO` to
|
||||
this repo's path:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"skillopt-sleep": {
|
||||
"command": "python3",
|
||||
"args": ["/abs/path/SkillOpt-Sleep/plugins/copilot/mcp_server.py"],
|
||||
"env": { "SKILLOPT_SLEEP_REPO": "/abs/path/SkillOpt-Sleep" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **(Optional) Tell Copilot about it.** Append
|
||||
`copilot-instructions.snippet.md` to your repo's
|
||||
`.github/copilot-instructions.md` so Copilot reaches for the tools when the
|
||||
user asks to "run the sleep cycle".
|
||||
|
||||
## Use
|
||||
|
||||
Ask Copilot things like *"run the sleep cycle"*, *"what did the last sleep
|
||||
propose?"*, *"adopt the staged sleep proposal"*. Copilot calls the MCP tools:
|
||||
`sleep_status`, `sleep_dry_run`, `sleep_run`, `sleep_adopt`, `sleep_harvest`.
|
||||
|
||||
Each tool takes optional `project`, `backend` (`mock`/`claude`/`codex`/`copilot`), and
|
||||
`scope` arguments. Default backend is `mock` (no API spend). The `copilot`
|
||||
backend drives the GitHub Copilot CLI (`copilot -p ... --output-format json`)
|
||||
and requires the `copilot` CLI to be installed and authenticated.
|
||||
|
||||
For speed, the `copilot` backend runs each call against an isolated
|
||||
`COPILOT_HOME` with built-in MCP servers and custom instructions disabled, so
|
||||
your user MCP servers (including this project's own) are not spawned per call
|
||||
(~5x faster). Override with `SKILLOPT_SLEEP_COPILOT_HOME=<dir>`, pick a model
|
||||
with `SKILLOPT_SLEEP_COPILOT_MODEL`, or set `SKILLOPT_SLEEP_COPILOT_FULL_ENV=1`
|
||||
to use your real Copilot environment instead.
|
||||
|
||||
## Verify the server directly (no Copilot needed)
|
||||
|
||||
```bash
|
||||
printf '%s\n' \
|
||||
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
|
||||
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
|
||||
| SKILLOPT_SLEEP_REPO="$(pwd)" python3 plugins/copilot/mcp_server.py
|
||||
```
|
||||
You should see the server info and the five `sleep_*` tools.
|
||||
|
||||
## Notes / status
|
||||
|
||||
- MCP is the stable, official Copilot extension surface, so this is the most
|
||||
portable of the three integrations (one server → CLI + IDE).
|
||||
- The engine and all its controls (gate on/off, multi-rollout, budget,
|
||||
preferences, optimizer/target split) are identical across platforms — see
|
||||
[the SkillOpt-Sleep guide section](https://microsoft.github.io/SkillOpt/docs/guideline.html#sleep).
|
||||
@@ -0,0 +1,43 @@
|
||||
<!--
|
||||
Copy this block into your repo's .github/copilot-instructions.md so Copilot
|
||||
knows the SkillOpt-Sleep tools exist. (Copilot reads copilot-instructions.md
|
||||
automatically as ambient guidance.)
|
||||
-->
|
||||
|
||||
## SkillOpt-Sleep (offline self-evolution)
|
||||
|
||||
This project has SkillOpt-Sleep available via an MCP server (`skillopt-sleep`).
|
||||
It gives the agent a nightly "sleep cycle": it reviews past sessions, replays
|
||||
recurring tasks offline, and consolidates validated memory + skills behind a
|
||||
held-out gate.
|
||||
|
||||
When the user asks to "run the sleep cycle", "review my past sessions", "learn
|
||||
my preferences", or "make the agent improve from past usage", use the MCP tools:
|
||||
|
||||
- `sleep_status` — what's happened + the latest staged proposal
|
||||
- `sleep_dry_run` — safe preview, stages nothing
|
||||
- `sleep_run` — full cycle, stages a reviewed proposal (nothing live changes)
|
||||
- `sleep_adopt` — apply the staged proposal (backs up first)
|
||||
- `sleep_harvest` — list mined recurring tasks
|
||||
- `sleep_schedule` — install a nightly cron entry (set `hour`/`minute`)
|
||||
- `sleep_unschedule` — remove the nightly cron entry
|
||||
|
||||
### Key parameters (pass as MCP tool arguments)
|
||||
|
||||
- `backend` — `mock` (default, free), `claude`, `codex`, or `copilot`
|
||||
- `source` — `claude`, `codex`, or `auto` (where to read transcripts)
|
||||
- `target_skill_path` — explicit SKILL.md to evolve
|
||||
- `tasks_file` — pre-built TaskRecord JSON (skip harvest)
|
||||
- `max_tasks` / `max_sessions` — cap workload
|
||||
- `auto_adopt` — auto-adopt if the gate passes
|
||||
- `json` — machine-readable output for programmatic use
|
||||
|
||||
### Advanced config (`~/.skillopt-sleep/config.json`)
|
||||
|
||||
- `preferences` — free-text house rules for the optimizer
|
||||
- `gate_mode` — `on` (default) or `off`; `dream_rollouts` — >1 for more signal
|
||||
- `evolve_memory` / `evolve_skill` — toggle which docs consolidate
|
||||
|
||||
Always show the user the held-out baseline → candidate score and the proposed
|
||||
edits before suggesting `sleep_adopt`. Never hand-edit the user's memory/skill
|
||||
files; only `sleep_adopt` does that, with a backup.
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"skillopt-sleep": {
|
||||
"command": "python3",
|
||||
"args": ["plugins/copilot/mcp_server.py"],
|
||||
"env": {
|
||||
"SKILLOPT_SLEEP_REPO": "${workspaceFolder}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Executable
+180
@@ -0,0 +1,180 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SkillOpt-Sleep — minimal MCP server (stdio, stdlib-only).
|
||||
|
||||
Exposes the sleep engine as MCP tools so any MCP-capable client (GitHub Copilot
|
||||
CLI / VS Code, Claude Desktop, etc.) can drive it. No third-party deps: speaks
|
||||
JSON-RPC 2.0 over stdio with just the handful of MCP methods clients need.
|
||||
|
||||
Tools exposed:
|
||||
- sleep_status : how many nights have run + the latest staged proposal
|
||||
- sleep_dry_run : harvest+mine+replay, report only (no staging)
|
||||
- sleep_run : full cycle, stages a proposal (nothing live changes)
|
||||
- sleep_adopt : apply the latest staged proposal (with backup)
|
||||
- sleep_harvest : debug — list mined recurring tasks
|
||||
|
||||
Each tool shells out to `python -m skillopt_sleep <action> ...` and returns its
|
||||
stdout. Configure your client to launch: python plugins/copilot/mcp_server.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
REPO_ROOT = os.environ.get("SKILLOPT_SLEEP_REPO") or os.path.abspath(
|
||||
os.path.join(os.path.dirname(__file__), "..", "..")
|
||||
)
|
||||
PROTOCOL_VERSION = "2024-11-05"
|
||||
|
||||
TOOLS = [
|
||||
{"name": "sleep_status", "action": "status",
|
||||
"description": "Show how many SkillOpt-Sleep nights have run and the latest staged proposal."},
|
||||
{"name": "sleep_dry_run", "action": "dry-run",
|
||||
"description": "Preview a sleep cycle (harvest+mine+replay) without staging anything."},
|
||||
{"name": "sleep_run", "action": "run",
|
||||
"description": "Run a full sleep cycle; stages a reviewed proposal. Nothing live changes until adopt."},
|
||||
{"name": "sleep_adopt", "action": "adopt",
|
||||
"description": "Apply the latest staged proposal to CLAUDE.md/SKILL.md (backs up first)."},
|
||||
{"name": "sleep_harvest", "action": "harvest",
|
||||
"description": "Debug: list the recurring tasks mined from recent sessions."},
|
||||
{"name": "sleep_schedule", "action": "schedule",
|
||||
"description": "Install a nightly cron entry to run the sleep cycle automatically."},
|
||||
{"name": "sleep_unschedule", "action": "unschedule",
|
||||
"description": "Remove the nightly cron entry for a project."},
|
||||
]
|
||||
_BY_NAME = {t["name"]: t for t in TOOLS}
|
||||
|
||||
_TOOL_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"project": {"type": "string",
|
||||
"description": "Project dir to evolve (default: cwd)."},
|
||||
"backend": {"type": "string", "enum": ["mock", "claude", "codex", "copilot"],
|
||||
"description": "mock = no API spend (default); claude/codex/copilot = real."},
|
||||
"scope": {"type": "string", "enum": ["invoked", "all"],
|
||||
"description": "Harvest scope (default: invoked project only)."},
|
||||
"source": {"type": "string", "enum": ["claude", "codex", "auto"],
|
||||
"description": "Transcript source (default: claude)."},
|
||||
"model": {"type": "string",
|
||||
"description": "Backend-specific model override."},
|
||||
"tasks_file": {"type": "string",
|
||||
"description": "Path to reviewed TaskRecord JSON (skips harvest)."},
|
||||
"target_skill_path": {"type": "string",
|
||||
"description": "Explicit SKILL.md path to evolve/stage/adopt."},
|
||||
"progress": {"type": "boolean",
|
||||
"description": "Print phase progress to stderr."},
|
||||
"max_sessions": {"type": "integer",
|
||||
"description": "Cap harvested sessions per run."},
|
||||
"max_tasks": {"type": "integer",
|
||||
"description": "Cap mined tasks per run."},
|
||||
"lookback_hours": {"type": "integer",
|
||||
"description": "Harvest window in hours (default: 72)."},
|
||||
"auto_adopt": {"type": "boolean",
|
||||
"description": "Auto-adopt if gate passes (default: false)."},
|
||||
"json": {"type": "boolean",
|
||||
"description": "Return machine-readable JSON output."},
|
||||
"edit_budget": {"type": "integer",
|
||||
"description": "Max bounded edits per night (default: 4)."},
|
||||
"hour": {"type": "integer",
|
||||
"description": "Hour for schedule (0-23, default: 3)."},
|
||||
"minute": {"type": "integer",
|
||||
"description": "Minute for schedule (0-59, default: 17)."},
|
||||
},
|
||||
"additionalProperties": False,
|
||||
}
|
||||
|
||||
|
||||
def _run_engine(action: str, args: dict) -> str:
|
||||
py = sys.executable or "python3"
|
||||
cmd = [py, "-m", "skillopt_sleep", action]
|
||||
# String-valued flags
|
||||
for flag, key in [
|
||||
("--project", "project"), ("--backend", "backend"),
|
||||
("--scope", "scope"), ("--source", "source"),
|
||||
("--model", "model"), ("--tasks-file", "tasks_file"),
|
||||
("--target-skill-path", "target_skill_path"),
|
||||
]:
|
||||
val = args.get(key)
|
||||
if val:
|
||||
cmd += [flag, str(val)]
|
||||
# Integer-valued flags
|
||||
for flag, key in [
|
||||
("--max-sessions", "max_sessions"), ("--max-tasks", "max_tasks"),
|
||||
("--lookback-hours", "lookback_hours"), ("--edit-budget", "edit_budget"),
|
||||
("--hour", "hour"), ("--minute", "minute"),
|
||||
]:
|
||||
val = args.get(key)
|
||||
if val is not None:
|
||||
cmd += [flag, str(int(val))]
|
||||
# Boolean flags
|
||||
for flag, key in [
|
||||
("--progress", "progress"), ("--auto-adopt", "auto_adopt"),
|
||||
("--json", "json"),
|
||||
]:
|
||||
if args.get(key):
|
||||
cmd.append(flag)
|
||||
try:
|
||||
proc = subprocess.run(cmd, cwd=REPO_ROOT, capture_output=True, text=True, timeout=3600)
|
||||
except Exception as e:
|
||||
return f"[error] failed to run engine: {e}"
|
||||
out = (proc.stdout or "").strip()
|
||||
err = (proc.stderr or "").strip()
|
||||
return out + (("\n[stderr]\n" + err) if err else "")
|
||||
|
||||
|
||||
def _result(id_, result):
|
||||
return {"jsonrpc": "2.0", "id": id_, "result": result}
|
||||
|
||||
|
||||
def _error(id_, code, message):
|
||||
return {"jsonrpc": "2.0", "id": id_, "error": {"code": code, "message": message}}
|
||||
|
||||
|
||||
def handle(req: dict):
|
||||
method = req.get("method")
|
||||
id_ = req.get("id")
|
||||
if method == "initialize":
|
||||
return _result(id_, {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {"tools": {}},
|
||||
"serverInfo": {"name": "skillopt-sleep", "version": "0.1.0"},
|
||||
})
|
||||
if method in ("notifications/initialized", "initialized"):
|
||||
return None # notification, no response
|
||||
if method == "tools/list":
|
||||
return _result(id_, {"tools": [
|
||||
{"name": t["name"], "description": t["description"], "inputSchema": _TOOL_SCHEMA}
|
||||
for t in TOOLS
|
||||
]})
|
||||
if method == "tools/call":
|
||||
params = req.get("params") or {}
|
||||
name = params.get("name")
|
||||
tool = _BY_NAME.get(name)
|
||||
if not tool:
|
||||
return _error(id_, -32602, f"unknown tool: {name}")
|
||||
text = _run_engine(tool["action"], params.get("arguments") or {})
|
||||
return _result(id_, {"content": [{"type": "text", "text": text}]})
|
||||
if method == "ping":
|
||||
return _result(id_, {})
|
||||
return _error(id_, -32601, f"method not found: {method}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
for line in sys.stdin:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
req = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
resp = handle(req)
|
||||
if resp is not None:
|
||||
sys.stdout.write(json.dumps(resp) + "\n")
|
||||
sys.stdout.flush()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,98 @@
|
||||
# SkillOpt — GitHub Copilot integration
|
||||
|
||||
Give **Copilot** (CLI or VS Code) direct access to the **SkillOpt** research
|
||||
engine via a tiny **MCP server**. MCP is GitHub's supported way to extend
|
||||
Copilot, so this works across Copilot CLI, VS Code, and other MCP clients with
|
||||
the same server.
|
||||
|
||||
SkillOpt is **validation-gated, text-space skill optimization**: it reflects on
|
||||
rollouts, makes bounded edits to a skill, and keeps a change only if it improves
|
||||
a held-out validation set. This plugin exposes the repo's training and eval
|
||||
entry points (`scripts/train.py`, `scripts/eval_only.py`) as Copilot tools.
|
||||
|
||||
> This is the companion to the **SkillOpt-Sleep** plugin (`../mcp_server.py`,
|
||||
> `sleep_*` tools). Sleep evolves a *local coding agent* from your past
|
||||
> sessions; this server drives the *research* training/eval loops on the
|
||||
> benchmark configs in [`../../../configs`](../../../configs).
|
||||
|
||||
## What's here
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `mcp_server.py` | stdlib-only MCP (stdio) server exposing `skillopt_*` tools |
|
||||
| `mcp-config.example.json` | drop-in MCP server config |
|
||||
| `copilot-instructions.snippet.md` | paste into `.github/copilot-instructions.md` |
|
||||
|
||||
## Install
|
||||
|
||||
Requires Python ≥ 3.10. The MCP server itself is pure stdlib, but the tools it
|
||||
launches need SkillOpt's runtime deps — install the package first:
|
||||
|
||||
```bash
|
||||
pip install -e . # or: pip install -r requirements.txt
|
||||
```
|
||||
|
||||
1. **Register the MCP server.** Add the server to your Copilot MCP config
|
||||
(Copilot CLI: `~/.copilot/mcp-config.json`; VS Code: your MCP settings).
|
||||
Use `mcp-config.example.json` as a template — set `SKILLOPT_REPO` to this
|
||||
repo's path:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"skillopt": {
|
||||
"command": "python3",
|
||||
"args": ["/abs/path/SkillOpt/plugins/copilot/skillopt/mcp_server.py"],
|
||||
"env": { "SKILLOPT_REPO": "/abs/path/SkillOpt" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **(Optional) Tell Copilot about it.** Append
|
||||
`copilot-instructions.snippet.md` to your repo's
|
||||
`.github/copilot-instructions.md` so Copilot reaches for the tools when the
|
||||
user asks to "optimize a skill" or "train on a benchmark".
|
||||
|
||||
## Use
|
||||
|
||||
Ask Copilot things like *"what configs can I run?"*, *"optimize the searchqa
|
||||
skill"*, or *"evaluate this skill on the dataset"*. Copilot calls the MCP tools:
|
||||
`skillopt_list_configs`, `skillopt_train`, `skillopt_eval`.
|
||||
|
||||
| Tool | Required args | Notes |
|
||||
|---|---|---|
|
||||
| `skillopt_list_configs` | — | Lists `configs/**/*.yaml` you can pass as `config`. |
|
||||
| `skillopt_train` | `config` | Runs a reflective optimization loop. Long-running; spends budget. |
|
||||
| `skillopt_eval` | `config`, `skill` | Evaluates one skill markdown file; no training. |
|
||||
|
||||
Common optional args (both train and eval): `env`, `backend`,
|
||||
`optimizer_model`, `target_model`, `out_root`, `cfg_options` (space-separated
|
||||
`KEY=VALUE` YAML overrides), and `extra_args` (raw passthrough flags for the
|
||||
underlying script). `skillopt_train` also accepts `num_epochs`, `batch_size`,
|
||||
`seed`, and `use_gate`.
|
||||
|
||||
Runs can be very long. The server's subprocess timeout defaults to 6 hours;
|
||||
override it with the `SKILLOPT_RUN_TIMEOUT` environment variable (seconds).
|
||||
|
||||
## Verify the server directly (no Copilot needed)
|
||||
|
||||
```bash
|
||||
printf '%s\n' \
|
||||
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
|
||||
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
|
||||
'{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"skillopt_list_configs","arguments":{}}}' \
|
||||
| SKILLOPT_REPO="$(pwd)" python3 plugins/copilot/skillopt/mcp_server.py
|
||||
```
|
||||
|
||||
You should see the server info, the three `skillopt_*` tools, and the list of
|
||||
benchmark configs.
|
||||
|
||||
## Notes / status
|
||||
|
||||
- MCP is the stable, official Copilot extension surface, so this is portable
|
||||
across Copilot CLI and IDE from one server.
|
||||
- `skillopt_list_configs` is filesystem-only and safe to call anytime;
|
||||
`skillopt_train` / `skillopt_eval` shell out to the repo scripts and require
|
||||
the SkillOpt runtime deps (and, for real backends, model credentials — see
|
||||
[`../../../.env.example`](../../../.env.example)).
|
||||
@@ -0,0 +1,33 @@
|
||||
<!--
|
||||
Copy this block into your repo's .github/copilot-instructions.md so Copilot
|
||||
knows the SkillOpt research-engine tools exist. (Copilot reads
|
||||
copilot-instructions.md automatically as ambient guidance.)
|
||||
-->
|
||||
|
||||
## SkillOpt (research skill-optimization engine)
|
||||
|
||||
This repo exposes the core **SkillOpt** training/eval engine via an MCP server
|
||||
(`skillopt`). SkillOpt is validation-gated, text-space skill optimization: it
|
||||
reflects on rollouts, makes bounded edits to a skill, and keeps a change only
|
||||
if it improves a held-out validation set.
|
||||
|
||||
When the user asks to "optimize a skill", "train on <benchmark>", "run
|
||||
SkillOpt", "evaluate this skill", or "what configs can I run", use the MCP
|
||||
tools:
|
||||
|
||||
- `skillopt_list_configs` — list the benchmark YAML configs you can pass as `config`
|
||||
- `skillopt_train` — run a reflective skill-optimization loop on a config (long-running; spends API/compute budget)
|
||||
- `skillopt_eval` — evaluate a single skill markdown file on a dataset (no training)
|
||||
|
||||
Guidance:
|
||||
- Always run `skillopt_list_configs` first if you don't already know a valid `config` path.
|
||||
- `skillopt_train` and `skillopt_eval` are long-running and consume the user's
|
||||
model backend/budget — confirm the `config`, `backend`, and model choices
|
||||
with the user before launching, and surface the held-out gate result when the
|
||||
run finishes.
|
||||
- For one-off YAML overrides use `cfg_options` (e.g. `seed=123 batch_size=40`);
|
||||
for any other underlying flag use `extra_args`.
|
||||
|
||||
This is distinct from the **SkillOpt-Sleep** MCP server (`skillopt-sleep`,
|
||||
`sleep_*` tools), which evolves a local coding agent from past sessions rather
|
||||
than running the research benchmarks.
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"skillopt": {
|
||||
"command": "python3",
|
||||
"args": ["plugins/copilot/skillopt/mcp_server.py"],
|
||||
"env": {
|
||||
"SKILLOPT_REPO": "${workspaceFolder}"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SkillOpt (research engine) — minimal MCP server (stdio, stdlib-only).
|
||||
|
||||
Exposes the core SkillOpt skill-optimization engine as MCP tools so any
|
||||
MCP-capable client (GitHub Copilot CLI / VS Code, Claude Desktop, etc.) can
|
||||
drive it. No third-party deps: speaks JSON-RPC 2.0 over stdio with just the
|
||||
handful of MCP methods clients need.
|
||||
|
||||
This is the companion to the SkillOpt-Sleep MCP server (``../mcp_server.py``).
|
||||
Where Sleep evolves a *local agent* from past sessions, this server drives the
|
||||
*research* training/eval loops from this repo (``scripts/train.py`` /
|
||||
``scripts/eval_only.py``) against the benchmark configs in ``configs/``.
|
||||
|
||||
Tools exposed:
|
||||
- skillopt_list_configs : discover the benchmark YAML configs you can use
|
||||
- skillopt_train : run a reflective skill-optimization (training) loop
|
||||
- skillopt_eval : evaluate a single skill on a dataset (no training)
|
||||
|
||||
``skillopt_train`` and ``skillopt_eval`` shell out to the repo's entry-point
|
||||
scripts and stream back their stdout/stderr. Configure your client to launch:
|
||||
python plugins/copilot/skillopt/mcp_server.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
# Repo root: three levels up from plugins/copilot/skillopt/mcp_server.py
|
||||
REPO_ROOT = os.environ.get("SKILLOPT_REPO") or os.path.abspath(
|
||||
os.path.join(os.path.dirname(__file__), "..", "..", "..")
|
||||
)
|
||||
PROTOCOL_VERSION = "2024-11-05"
|
||||
|
||||
# Training/eval runs are long; give the engine plenty of headroom.
|
||||
RUN_TIMEOUT_SECONDS = int(os.environ.get("SKILLOPT_RUN_TIMEOUT", "21600")) # 6h
|
||||
|
||||
|
||||
def _list_configs() -> str:
|
||||
"""List the benchmark configs available under configs/ (filesystem only)."""
|
||||
pattern = os.path.join(REPO_ROOT, "configs", "**", "*.yaml")
|
||||
paths = sorted(glob.glob(pattern, recursive=True))
|
||||
if not paths:
|
||||
return f"[no configs found under {os.path.join(REPO_ROOT, 'configs')}]"
|
||||
rels = [os.path.relpath(p, REPO_ROOT).replace(os.sep, "/") for p in paths]
|
||||
lines = ["Available SkillOpt configs (pass as `config`):", ""]
|
||||
lines += [f" - {r}" for r in rels]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _run_script(script_rel: str, args: dict, *, required: tuple[str, ...] = ()) -> str:
|
||||
"""Shell out to a repo entry-point script, mapping args -> --flags."""
|
||||
for key in required:
|
||||
if not args.get(key):
|
||||
return f"[error] missing required argument: {key}"
|
||||
|
||||
py = sys.executable or "python3"
|
||||
cmd = [py, os.path.join("scripts", script_rel)]
|
||||
|
||||
# Ordered flags that the train/eval scripts accept directly.
|
||||
flag_args = (
|
||||
"config", "skill", "split", "env", "backend",
|
||||
"optimizer_model", "target_model", "out_root",
|
||||
"num_epochs", "batch_size", "seed", "use_gate",
|
||||
)
|
||||
for key in flag_args:
|
||||
val = args.get(key)
|
||||
if val is None or val == "":
|
||||
continue
|
||||
cmd += [f"--{key}", str(val)]
|
||||
|
||||
# cfg-options: arbitrary KEY=VALUE YAML overrides (nargs="+").
|
||||
cfg_options = args.get("cfg_options")
|
||||
if cfg_options:
|
||||
if isinstance(cfg_options, str):
|
||||
cfg_options = cfg_options.split()
|
||||
cmd += ["--cfg-options", *[str(x) for x in cfg_options]]
|
||||
|
||||
# extra_args: raw passthrough for any other train/eval flag.
|
||||
extra = args.get("extra_args")
|
||||
if extra:
|
||||
if isinstance(extra, str):
|
||||
extra = extra.split()
|
||||
cmd += [str(x) for x in extra]
|
||||
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd, cwd=REPO_ROOT, capture_output=True, text=True,
|
||||
timeout=RUN_TIMEOUT_SECONDS,
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return f"[error] run exceeded {RUN_TIMEOUT_SECONDS}s timeout: {' '.join(cmd)}"
|
||||
except Exception as e: # noqa: BLE001
|
||||
return f"[error] failed to run script: {e}"
|
||||
out = (proc.stdout or "").strip()
|
||||
err = (proc.stderr or "").strip()
|
||||
body = out + (("\n[stderr]\n" + err) if err else "")
|
||||
return body or f"[done] exit code {proc.returncode}, no output"
|
||||
|
||||
|
||||
TOOLS = [
|
||||
{
|
||||
"name": "skillopt_list_configs",
|
||||
"description": "List the benchmark YAML configs under configs/ that can be passed as `config` to train/eval.",
|
||||
},
|
||||
{
|
||||
"name": "skillopt_train",
|
||||
"description": "Run a SkillOpt reflective skill-optimization (training) loop on a benchmark config. Long-running; uses your model backend/budget.",
|
||||
},
|
||||
{
|
||||
"name": "skillopt_eval",
|
||||
"description": "Evaluate a single skill markdown file on a dataset without training (scripts/eval_only.py).",
|
||||
},
|
||||
]
|
||||
_BY_NAME = {t["name"]: t for t in TOOLS}
|
||||
|
||||
_NO_ARGS_SCHEMA = {"type": "object", "properties": {}, "additionalProperties": False}
|
||||
|
||||
_COMMON_PROPS = {
|
||||
"config": {"type": "string",
|
||||
"description": "Path to a benchmark YAML config (e.g. configs/searchqa/default.yaml). See skillopt_list_configs."},
|
||||
"env": {"type": "string", "description": "Override the environment/adapter name (e.g. searchqa, alfworld)."},
|
||||
"backend": {"type": "string", "description": "Model backend (e.g. azure_openai, claude, codex, qwen, minimax)."},
|
||||
"optimizer_model": {"type": "string", "description": "Model used for reflection/skill rewriting (the optimizer)."},
|
||||
"target_model": {"type": "string", "description": "Model used to execute tasks (the target)."},
|
||||
"out_root": {"type": "string", "description": "Output directory root for run artifacts."},
|
||||
"cfg_options": {"type": "string", "description": "Space-separated YAML overrides, e.g. 'seed=123 batch_size=40'."},
|
||||
"extra_args": {"type": "string", "description": "Raw passthrough flags for the underlying script, e.g. '--workers 8 --max_turns 30'."},
|
||||
}
|
||||
|
||||
_TRAIN_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
**_COMMON_PROPS,
|
||||
"num_epochs": {"type": "integer", "description": "Number of optimization epochs."},
|
||||
"batch_size": {"type": "integer", "description": "Tasks per optimization step."},
|
||||
"seed": {"type": "integer", "description": "Random seed."},
|
||||
"use_gate": {"type": "string", "enum": ["true", "false"],
|
||||
"description": "Whether to keep the held-out validation gate on (default on)."},
|
||||
},
|
||||
"required": ["config"],
|
||||
"additionalProperties": False,
|
||||
}
|
||||
|
||||
_EVAL_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
**_COMMON_PROPS,
|
||||
"skill": {"type": "string", "description": "Path to the skill markdown file to evaluate."},
|
||||
"split": {"type": "string", "description": "Dataset split to evaluate (default: all)."},
|
||||
},
|
||||
"required": ["config", "skill"],
|
||||
"additionalProperties": False,
|
||||
}
|
||||
|
||||
_SCHEMA_BY_NAME = {
|
||||
"skillopt_list_configs": _NO_ARGS_SCHEMA,
|
||||
"skillopt_train": _TRAIN_SCHEMA,
|
||||
"skillopt_eval": _EVAL_SCHEMA,
|
||||
}
|
||||
|
||||
|
||||
def _result(id_, result):
|
||||
return {"jsonrpc": "2.0", "id": id_, "result": result}
|
||||
|
||||
|
||||
def _error(id_, code, message):
|
||||
return {"jsonrpc": "2.0", "id": id_, "error": {"code": code, "message": message}}
|
||||
|
||||
|
||||
def _dispatch(name: str, args: dict) -> str:
|
||||
if name == "skillopt_list_configs":
|
||||
return _list_configs()
|
||||
if name == "skillopt_train":
|
||||
return _run_script("train.py", args, required=("config",))
|
||||
if name == "skillopt_eval":
|
||||
return _run_script("eval_only.py", args, required=("config", "skill"))
|
||||
return f"[error] unknown tool: {name}"
|
||||
|
||||
|
||||
def handle(req: dict):
|
||||
method = req.get("method")
|
||||
id_ = req.get("id")
|
||||
if method == "initialize":
|
||||
return _result(id_, {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {"tools": {}},
|
||||
"serverInfo": {"name": "skillopt", "version": "0.1.0"},
|
||||
})
|
||||
if method in ("notifications/initialized", "initialized"):
|
||||
return None # notification, no response
|
||||
if method == "tools/list":
|
||||
return _result(id_, {"tools": [
|
||||
{"name": t["name"], "description": t["description"],
|
||||
"inputSchema": _SCHEMA_BY_NAME[t["name"]]}
|
||||
for t in TOOLS
|
||||
]})
|
||||
if method == "tools/call":
|
||||
params = req.get("params") or {}
|
||||
name = params.get("name")
|
||||
if name not in _BY_NAME:
|
||||
return _error(id_, -32602, f"unknown tool: {name}")
|
||||
text = _dispatch(name, params.get("arguments") or {})
|
||||
return _result(id_, {"content": [{"type": "text", "text": text}]})
|
||||
if method == "ping":
|
||||
return _result(id_, {})
|
||||
return _error(id_, -32601, f"method not found: {method}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
for line in sys.stdin:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
req = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
resp = handle(req)
|
||||
if resp is not None:
|
||||
sys.stdout.write(json.dumps(resp) + "\n")
|
||||
sys.stdout.flush()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,66 @@
|
||||
# SkillOpt-Sleep — Devin integration
|
||||
|
||||
Give **Devin** (Cognition) a nightly **sleep cycle** via a tiny **MCP server**
|
||||
that exposes the `skillopt_sleep` engine as tools. MCP is Devin's supported way
|
||||
to add custom tooling, so this works in Devin's CLI and IDE.
|
||||
|
||||
Devin doesn't write transcripts in the format the engine consumes, so this
|
||||
plugin adds a **Devin-specific harvester** that converts every locally available
|
||||
source into the Claude Code-compatible JSONL the engine reads.
|
||||
|
||||
## What's here
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `mcp_server.py` | stdlib-only MCP (stdio) server exposing `sleep_*` tools |
|
||||
| `harvest_devin.py` | converts Devin ATIF-v1.7 transcripts + agentmemory + `.devin/skills` into JSONL, with `taskKey` + outcome envelopes |
|
||||
| `judge.py` | reference judge for the deferred/judge branch of the validation gate |
|
||||
| `mcp-config.example.json` | drop-in MCP server config |
|
||||
| `devin-rules.snippet.md` | paste into `.devin/rules/skillopt-sleep.md` |
|
||||
|
||||
## What it harvests
|
||||
|
||||
| Source | Where |
|
||||
|---|---|
|
||||
| Devin transcripts (ATIF-v1.7) | `~/.local/share/devin/cli/transcripts/*.json` |
|
||||
| agentmemory | `~/.agentmemory/standalone.json` |
|
||||
| Skill files | `.devin/skills/*/SKILL.md` |
|
||||
|
||||
Workspaces are auto-detected from `~/.config/Devin/User/workspaceStorage/*/workspace.json`.
|
||||
After `sleep_adopt`, the evolved skill is synced to `.devin/skills/skillopt-sleep-learned/SKILL.md`.
|
||||
|
||||
## Install
|
||||
|
||||
Requires Python ≥ 3.10. No third-party packages — the server is pure stdlib.
|
||||
|
||||
1. **Register the MCP server.** Use `mcp-config.example.json` as a template; set
|
||||
`args` to the absolute path of this `mcp_server.py`. The engine is found
|
||||
automatically (this plugin lives inside the SkillOpt repo). Or via the Devin
|
||||
CLI:
|
||||
|
||||
```bash
|
||||
devin mcp add skillopt-sleep \
|
||||
--env "SKILLOPT_DEVIN_CLAUDE_HOME=$HOME/.skillopt-sleep-devin" \
|
||||
-- python3 /abs/path/to/SkillOpt/plugins/devin/mcp_server.py
|
||||
```
|
||||
|
||||
2. **(Optional)** copy `devin-rules.snippet.md` to `.devin/rules/skillopt-sleep.md`
|
||||
so Devin proactively offers the tools.
|
||||
|
||||
3. Ask Devin: *"run the sleep cycle"*, *"what did the last sleep propose?"*, *"adopt it"*.
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | What it does |
|
||||
|---|---|
|
||||
| `sleep_status` | nights run so far + latest staged proposal |
|
||||
| `sleep_dry_run` | preview cycle — no staging, no changes |
|
||||
| `sleep_run` | full cycle; stages a proposal for review |
|
||||
| `sleep_adopt` | apply the staged proposal; syncs skill to the workspace |
|
||||
| `sleep_harvest` | debug: list the recurring tasks mined |
|
||||
| `sleep_schedule` | install a nightly cron entry (`--hour` / `--minute`) |
|
||||
| `sleep_unschedule` | remove the nightly cron entry |
|
||||
|
||||
Default backend is `mock` (no API spend); `--backend claude|codex` uses your own
|
||||
budget. Same engine and `sleep_*` interface as the other plugins — all call
|
||||
`python -m skillopt_sleep`.
|
||||
@@ -0,0 +1,19 @@
|
||||
# SkillOpt-Sleep (Devin)
|
||||
|
||||
You have access to a nightly self-evolution cycle via the `skillopt-sleep` MCP
|
||||
server. Use these tools to improve your long-term skills over time:
|
||||
|
||||
- **`sleep_status`** — how many nights have run + the latest staged proposal
|
||||
- **`sleep_dry_run`** — preview a cycle without changing anything
|
||||
- **`sleep_run`** — run a full cycle; stages a proposal for review
|
||||
- **`sleep_adopt`** — apply the staged proposal to `.devin/skills/skillopt-sleep-learned/SKILL.md`
|
||||
- **`sleep_harvest`** — debug: list the recurring tasks mined from recent sessions
|
||||
- **`sleep_schedule`** / **`sleep_unschedule`** — install/remove a nightly cron run
|
||||
|
||||
When a user asks about the sleep cycle, skill evolution, or improving your
|
||||
long-term memory, prefer calling these tools over explaining the concept.
|
||||
|
||||
Default backend is `mock` (no API spend). Pass `backend: "claude"` or
|
||||
`backend: "codex"` with your own API key for real LLM-driven optimization.
|
||||
|
||||
Place this file at `.devin/rules/skillopt-sleep.md` in your workspace.
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"schema_version": "ATIF-v1.7",
|
||||
"session_id": "demo-001",
|
||||
"steps": [
|
||||
{
|
||||
"source": "user",
|
||||
"message": "Fix the failing NullPointerException in OrderService.persist() in the dutch-kis project",
|
||||
"timestamp": "2026-06-20T10:00:00Z"
|
||||
},
|
||||
{
|
||||
"source": "agent",
|
||||
"message": "The repository call returns an Optional that is being unwrapped with .get(). I'll switch to orElseThrow(NotFoundException::new) so the missing-row case is handled.",
|
||||
"timestamp": "2026-06-20T10:00:05Z"
|
||||
},
|
||||
{
|
||||
"source": "agent",
|
||||
"message": "Applied the fix and ran the suite: rtk mvn test -Dtest=OrderServiceTest -> BUILD SUCCESS, 142 passed, 0 failed.",
|
||||
"timestamp": "2026-06-20T10:01:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,533 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert Devin IDE local data into Claude Code-format JSONL transcripts.
|
||||
|
||||
Devin (Cognition) does not persist agent conversation transcripts to disk in a
|
||||
format the sleep engine understands. This script bridges that gap by synthesising
|
||||
JSONL files from every locally available source:
|
||||
|
||||
1. **Devin transcripts** (~/.local/share/devin/cli/transcripts/*.json)
|
||||
Native ATIF-v1.7 format — source:"user" / source:"agent" messages
|
||||
converted directly to user/assistant JSONL turns.
|
||||
|
||||
2. **agentmemory** (~/.agentmemory/standalone.json)
|
||||
Memories saved by the `agentmemory` MCP server — each memory's title
|
||||
becomes a synthetic user prompt; its content becomes the assistant reply.
|
||||
|
||||
3. **Skill files** (.devin/skills/*/SKILL.md)
|
||||
Each skill description is converted to a session where the user asked
|
||||
"use the <skill> skill" and the assistant described how to apply it.
|
||||
|
||||
Output layout (mirrors ~/.claude/projects/<slug>/<sessionId>.jsonl):
|
||||
<out_dir>/projects/<slug>/<session_id>.jsonl
|
||||
|
||||
Workspace auto-detection order:
|
||||
1. ``SKILLOPT_DEVIN_WORKSPACES`` env var — colon-separated abs paths
|
||||
2. Devin registry: ``~/.config/Devin/User/workspaceStorage/*/workspace.json``
|
||||
4. Working directory fallback
|
||||
|
||||
Usage (standalone):
|
||||
python harvest_devin.py [--out-dir PATH] [--workspaces PATH ...]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
# ── cross-platform path resolution (Linux + Windows + macOS) ──────────────────
|
||||
#
|
||||
# Devin is a VS Code-family app, so its user-data dir moves with the OS:
|
||||
# Linux ~/.config/<App>, Windows %APPDATA%\<App>, macOS
|
||||
# ~/Library/Application Support/<App>. Resolve all candidates and let callers
|
||||
# keep whichever actually exists.
|
||||
|
||||
def _app_data_roots(app: str) -> List[str]:
|
||||
"""User-data dir candidates for a VS Code-family app, current OS first."""
|
||||
home = os.path.expanduser("~")
|
||||
roots: List[str] = []
|
||||
if os.name == "nt":
|
||||
appdata = os.environ.get("APPDATA") or os.path.join(home, "AppData", "Roaming")
|
||||
roots.append(os.path.join(appdata, app))
|
||||
elif sys.platform == "darwin":
|
||||
roots.append(os.path.join(home, "Library", "Application Support", app))
|
||||
# XDG / Linux (also a sensible fallback everywhere)
|
||||
xdg = os.environ.get("XDG_CONFIG_HOME") or os.path.join(home, ".config")
|
||||
roots.append(os.path.join(xdg, app))
|
||||
# de-dupe, preserve order
|
||||
return list(dict.fromkeys(roots))
|
||||
|
||||
|
||||
def _devin_transcript_candidates() -> List[str]:
|
||||
"""Where the Devin CLI may store ATIF transcripts, per OS."""
|
||||
home = os.path.expanduser("~")
|
||||
cands: List[str] = []
|
||||
if os.name == "nt":
|
||||
for base in (os.environ.get("LOCALAPPDATA"), os.environ.get("APPDATA")):
|
||||
if base:
|
||||
cands.append(os.path.join(base, "devin", "cli", "transcripts"))
|
||||
elif sys.platform == "darwin":
|
||||
cands.append(os.path.join(home, "Library", "Application Support",
|
||||
"devin", "cli", "transcripts"))
|
||||
cands.append(os.path.join(home, ".local", "share", "devin", "cli", "transcripts"))
|
||||
return list(dict.fromkeys(cands))
|
||||
|
||||
|
||||
def _first_existing(paths: List[str]) -> str:
|
||||
"""First path that exists, else the first candidate (for nice messaging)."""
|
||||
for p in paths:
|
||||
if os.path.exists(p):
|
||||
return p
|
||||
return paths[0] if paths else ""
|
||||
|
||||
|
||||
def _uri_to_path(folder: str) -> str:
|
||||
"""Convert a VS Code ``file://`` workspace URI to a local path, cross-platform.
|
||||
|
||||
Linux: file:///home/u/proj -> /home/u/proj
|
||||
Windows: file:///c%3A/Users/u/p -> c:/Users/u/p
|
||||
"""
|
||||
if not folder.startswith("file://"):
|
||||
return folder
|
||||
path = unquote(urlparse(folder).path)
|
||||
# Windows drive paths come through as '/C:/...' — strip the leading slash.
|
||||
if os.name == "nt" and re.match(r"^/[A-Za-z]:", path):
|
||||
path = path[1:]
|
||||
return path
|
||||
|
||||
# ── workspace auto-detection ─────────────────────────────────────────────────
|
||||
|
||||
def _workspaces_from_registry(storage_root: str) -> List[tuple]:
|
||||
"""Read VS Code-style workspaceStorage to get (mtime, path) pairs."""
|
||||
results: List[tuple] = []
|
||||
if not os.path.isdir(storage_root):
|
||||
return results
|
||||
for entry in os.scandir(storage_root):
|
||||
ws_json = os.path.join(entry.path, "workspace.json")
|
||||
if not os.path.isfile(ws_json):
|
||||
continue
|
||||
try:
|
||||
with open(ws_json, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
folder = _uri_to_path(data.get("folder", ""))
|
||||
if folder and os.path.isdir(folder):
|
||||
results.append((os.path.getmtime(ws_json), folder))
|
||||
except Exception:
|
||||
continue
|
||||
return results
|
||||
|
||||
|
||||
def _detect_workspaces() -> List[str]:
|
||||
"""Return known workspace paths (Devin registry), newest first."""
|
||||
env_val = os.environ.get("SKILLOPT_DEVIN_WORKSPACES", "")
|
||||
if env_val:
|
||||
# os.pathsep so Windows 'C:\a;C:\b' splits correctly (not on the drive colon)
|
||||
return [p for p in env_val.split(os.pathsep) if p and os.path.isdir(p)]
|
||||
|
||||
registries: List[str] = [
|
||||
os.path.join(r, "User", "workspaceStorage")
|
||||
for r in _app_data_roots("Devin")
|
||||
]
|
||||
|
||||
seen: set = set()
|
||||
results: List[tuple] = []
|
||||
for registry in registries:
|
||||
for mtime, folder in _workspaces_from_registry(registry):
|
||||
if folder not in seen:
|
||||
seen.add(folder)
|
||||
results.append((mtime, folder))
|
||||
results.sort(reverse=True)
|
||||
paths = [p for _, p in results]
|
||||
return paths if paths else [os.getcwd()]
|
||||
|
||||
# ── helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def _slug(path: str) -> str:
|
||||
"""SHA-256 of abs-path, first 16 hex chars — matches Claude Code's scheme."""
|
||||
return hashlib.sha256(os.path.abspath(path).encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def _iso(epoch_ms: Optional[float] = None) -> str:
|
||||
dt = (datetime.fromtimestamp(epoch_ms / 1000.0, tz=timezone.utc)
|
||||
if epoch_ms is not None else datetime.now(tz=timezone.utc))
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%S.000Z")
|
||||
|
||||
|
||||
def _write_session(
|
||||
out_dir: str, project: str, session_id: str,
|
||||
user_prompts: List[str], assistant_replies: List[str],
|
||||
timestamp_base_ms: float,
|
||||
task_key: Optional[str] = None,
|
||||
) -> None:
|
||||
slug = _slug(project)
|
||||
session_dir = os.path.join(out_dir, "projects", slug)
|
||||
os.makedirs(session_dir, exist_ok=True)
|
||||
out_path = os.path.join(session_dir, f"{session_id}.jsonl")
|
||||
ts = timestamp_base_ms
|
||||
with open(out_path, "w", encoding="utf-8") as f:
|
||||
for user_text, asst_text in zip(user_prompts, assistant_replies):
|
||||
user_rec = {
|
||||
"type": "user",
|
||||
"message": {"role": "user", "content": user_text},
|
||||
"cwd": project,
|
||||
"timestamp": _iso(ts),
|
||||
"sessionId": session_id,
|
||||
"version": "1.0",
|
||||
}
|
||||
if task_key:
|
||||
# grouping key so the miner can collapse repeats into one recurring task
|
||||
user_rec["taskKey"] = task_key
|
||||
f.write(json.dumps(user_rec, ensure_ascii=False) + "\n")
|
||||
# space the reply >=5s after the prompt so a single-turn session
|
||||
# isn't misclassified as a <3s headless replay and dropped by the
|
||||
# engine's harvest filter (skillopt_sleep Issue #62).
|
||||
ts += 5000
|
||||
f.write(json.dumps({
|
||||
"type": "assistant",
|
||||
"message": {"role": "assistant", "content": asst_text},
|
||||
"timestamp": _iso(ts),
|
||||
"sessionId": session_id,
|
||||
"version": "1.0",
|
||||
}, ensure_ascii=False) + "\n")
|
||||
ts += 2000
|
||||
|
||||
|
||||
def _append_history(out_dir: str, display: str, project: str, timestamp_ms: float) -> None:
|
||||
record = {"display": display, "timestamp": timestamp_ms, "project": project}
|
||||
with open(os.path.join(out_dir, "history.jsonl"), "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def _infer_project(text: str, workspaces: List[str]) -> str:
|
||||
for ws in workspaces:
|
||||
if os.path.basename(ws.rstrip("/")).lower() in text.lower():
|
||||
return ws
|
||||
return workspaces[0] if workspaces else os.getcwd()
|
||||
|
||||
# ── task identity + outcome extraction (fuel for the validation gate) ─────────
|
||||
#
|
||||
# SkillOpt's gate only works "where tasks recur and have a checkable correctness
|
||||
# signal." These helpers add the two things a raw transcript lacks:
|
||||
# * a stable taskKey so repeats collapse into one recurring task, and
|
||||
# * an outcome envelope (success + verifier + re-runnable reference) so the
|
||||
# held-out replay has something to score against.
|
||||
|
||||
_LANG_HINTS = [
|
||||
("java", r"(java|spring|maven|\bmvn\b|gradle|\.java\b|lombok)"),
|
||||
("python", r"(python|pytest|\bpip\b|\.py\b|django|flask)"),
|
||||
("ts", r"(typescript|\.tsx?\b|\bnpm\b|jest|node)"),
|
||||
("js", r"(javascript|\.jsx?\b)"),
|
||||
("sql", r"(\bsql\b|select\s|mariadb|mysql|postgres|\.sql\b)"),
|
||||
("go", r"(golang|\bgo test\b|\.go\b)"),
|
||||
("rust", r"(rust|cargo|\.rs\b)"),
|
||||
]
|
||||
_INTENT_HINTS = [
|
||||
("fix", r"(fix|bug|error|fail|npe|exception|broken|crash)"),
|
||||
("implement", r"(implement|add|create|build|introduce|support)"),
|
||||
("refactor", r"(refactor|clean ?up|rename|extract|simplify)"),
|
||||
("test", r"(test|coverage|assert)"),
|
||||
("review", r"(review|audit|inspect)"),
|
||||
("optimize", r"(optimi[sz]e|perf|speed up|slow)"),
|
||||
("explain", r"(explain|understand|what does|how does)"),
|
||||
]
|
||||
_STOPWORDS = {"please", "this", "that", "with", "from", "into", "should",
|
||||
"would", "code", "using", "the", "have"}
|
||||
|
||||
|
||||
def _normalize_task_key(text: str, project: str) -> str:
|
||||
"""Stable '<lang>:<intent>:<target>' grouping key for a task."""
|
||||
low = text.lower()
|
||||
lang = next((n for n, pat in _LANG_HINTS if re.search(pat, low)), "general")
|
||||
intent = next((n for n, pat in _INTENT_HINTS if re.search(pat, low)), "task")
|
||||
# target: prefer a CamelCase identifier, then a filename, then first real word
|
||||
m = re.search(r"\b([A-Z][a-z0-9]+(?:[A-Z][a-z0-9]+)+)\b", text) # CamelCase
|
||||
if not m:
|
||||
m = re.search(r"\b([\w-]+\.\w+)\b", text) # filename.ext
|
||||
if m:
|
||||
target = m.group(1)
|
||||
else:
|
||||
# first content word that isn't a stopword or an intent verb (e.g. "implement")
|
||||
target = next((w for w in re.findall(r"[a-zA-Z]{4,}", low)
|
||||
if w not in _STOPWORDS
|
||||
and not any(re.search(pat, w) for _, pat in _INTENT_HINTS)),
|
||||
"general")
|
||||
target = re.sub(r"[^a-zA-Z0-9]+", "-", target).strip("-").lower()[:40] or "general"
|
||||
return f"{lang}:{intent}:{target}"
|
||||
|
||||
|
||||
_PASS_PAT = re.compile(
|
||||
r"(build success|all tests? pass(?:ed)?|\b\d+ passed\b|\b0 failed\b|"
|
||||
r"tests? pass(?:ed)?|✓|no errors)", re.IGNORECASE)
|
||||
_FAIL_PAT = re.compile(
|
||||
r"(build failure|tests? failed|\b[1-9]\d* failed\b|error:|traceback|"
|
||||
r"assertion ?error)", re.IGNORECASE) # note: "0 failed" must NOT match
|
||||
_CMD_PAT = re.compile(
|
||||
r"((?:rtk\s+)?(?:mvn|gradle|pytest|npm(?:\s+run)?\s+test|yarn\s+test|"
|
||||
r"go\s+test|cargo\s+test)[^\n`]*)", re.IGNORECASE)
|
||||
|
||||
|
||||
def _detect_outcome(messages: List[str]) -> Optional[Dict[str, Any]]:
|
||||
"""Best-effort checkable signal from agent messages. None ⇒ no hard signal."""
|
||||
blob = "\n".join(m for m in messages if m)
|
||||
pass_hit, fail_hit = _PASS_PAT.search(blob), _FAIL_PAT.search(blob)
|
||||
if not pass_hit and not fail_hit:
|
||||
return None
|
||||
verifier = "tests" if re.search(r"test|pytest", blob, re.IGNORECASE) else "build"
|
||||
out: Dict[str, Any] = {
|
||||
"success": bool(pass_hit) and not fail_hit,
|
||||
"verifier": verifier,
|
||||
"evidence": (pass_hit or fail_hit).group(0).strip(),
|
||||
}
|
||||
cmd = _CMD_PAT.search(blob)
|
||||
if cmd:
|
||||
# keep only the command itself, dropping any "-> result" / ": output" tail
|
||||
repro = re.split(r"\s*(?:->|→|:|,)\s*", cmd.group(1))[0].strip()
|
||||
out["reference"] = {"repro": repro}
|
||||
return out
|
||||
|
||||
|
||||
def _build_rubric(user_prompt: str) -> List[str]:
|
||||
"""Derive checkable criteria from the task so a judge has something to score."""
|
||||
crit: List[str] = []
|
||||
ids = re.findall(r"\b([A-Z][a-z0-9]+(?:[A-Z][a-z0-9]+)+|[\w-]+\.\w+)\b", user_prompt)
|
||||
for i in dict.fromkeys(ids): # dedupe, preserve order
|
||||
crit.append(f"Addresses {i}")
|
||||
intent = _normalize_task_key(user_prompt, "").split(":")[1]
|
||||
crit.append({
|
||||
"fix": "Resolves the reported defect without introducing new errors",
|
||||
"implement": "Implements the requested behavior end to end",
|
||||
"refactor": "Preserves behavior while improving structure",
|
||||
"test": "Adds or fixes tests that actually exercise the change",
|
||||
"optimize": "Improves performance without changing results",
|
||||
}.get(intent, "Satisfies the user's stated request"))
|
||||
crit.append("Response is concrete and actionable, not a restatement of the task")
|
||||
return crit[:5]
|
||||
|
||||
|
||||
def _judge_rubric_fallback(user_prompt: str) -> Dict[str, Any]:
|
||||
"""When no hard signal exists, attach a rubric and mark the task for judge
|
||||
scoring. success=None tells the gate to defer/judge rather than trust it.
|
||||
The actual scoring is done by judge.py (or the engine) at replay time."""
|
||||
return {
|
||||
"success": None,
|
||||
"verifier": "judge",
|
||||
"rubric": _build_rubric(user_prompt or ""),
|
||||
}
|
||||
|
||||
|
||||
def _write_outcome(out_dir: str, session_id: str, task_key: str, project: str,
|
||||
ts_ms: float, outcome: Dict[str, Any]) -> None:
|
||||
rec = {"type": "outcome", "sessionId": session_id, "taskKey": task_key,
|
||||
"project": project, "timestamp": _iso(ts_ms), **outcome}
|
||||
with open(os.path.join(out_dir, "outcomes.jsonl"), "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
# ── source 1: Devin ATIF-v1.7 transcripts ────────────────────────────────────
|
||||
|
||||
def harvest_devin_transcripts(
|
||||
transcripts_dir: str, out_dir: str, workspaces: List[str]
|
||||
) -> int:
|
||||
"""Convert Devin CLI ATIF-v1.7 transcripts to Claude Code JSONL."""
|
||||
if not os.path.isdir(transcripts_dir):
|
||||
return 0
|
||||
written = 0
|
||||
for entry in os.scandir(transcripts_dir):
|
||||
if not entry.name.endswith(".json"):
|
||||
continue
|
||||
try:
|
||||
with open(entry.path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except Exception:
|
||||
continue
|
||||
if data.get("schema_version", "").startswith("ATIF"):
|
||||
pass # Devin native format
|
||||
else:
|
||||
continue
|
||||
session_id = data.get("session_id") or entry.name[:-5]
|
||||
steps = data.get("steps") or []
|
||||
user_prompts: List[str] = []
|
||||
agent_replies: List[str] = []
|
||||
project = ""
|
||||
ts_base: Optional[float] = None
|
||||
for step in steps:
|
||||
src = step.get("source", "")
|
||||
msg = str(step.get("message") or "").strip()
|
||||
if not msg or src == "system":
|
||||
continue
|
||||
if src == "user":
|
||||
user_prompts.append(msg)
|
||||
if not project:
|
||||
project = _infer_project(msg, workspaces)
|
||||
elif src == "agent":
|
||||
agent_replies.append(msg)
|
||||
if ts_base is None:
|
||||
raw_ts = step.get("timestamp", "")
|
||||
if raw_ts:
|
||||
try:
|
||||
from datetime import datetime as _dt
|
||||
ts_base = _dt.fromisoformat(
|
||||
raw_ts.replace("Z", "+00:00")
|
||||
).timestamp() * 1000
|
||||
except Exception:
|
||||
pass
|
||||
if not user_prompts:
|
||||
continue
|
||||
if not project:
|
||||
project = workspaces[0] if workspaces else os.getcwd()
|
||||
if ts_base is None:
|
||||
ts_base = datetime.now(tz=timezone.utc).timestamp() * 1000
|
||||
# Identity + outcome: what makes this trajectory replayable & gradeable.
|
||||
task_key = _normalize_task_key(user_prompts[0], project)
|
||||
outcome = _detect_outcome(agent_replies) or _judge_rubric_fallback(user_prompts[0])
|
||||
# Pair turns; pad shorter list
|
||||
n = max(len(user_prompts), len(agent_replies))
|
||||
user_prompts += [""] * (n - len(user_prompts))
|
||||
agent_replies += [""] * (n - len(agent_replies))
|
||||
sid = f"devin_{session_id}"
|
||||
_write_session(
|
||||
out_dir, project, sid,
|
||||
user_prompts=[p for p in user_prompts if p],
|
||||
assistant_replies=[r if r else "[no reply recorded]" for r, p in
|
||||
zip(agent_replies, user_prompts) if p],
|
||||
timestamp_base_ms=ts_base,
|
||||
task_key=task_key,
|
||||
)
|
||||
_write_outcome(out_dir, sid, task_key, project, ts_base, outcome)
|
||||
_append_history(
|
||||
out_dir,
|
||||
display=(user_prompts[0] or session_id)[:120],
|
||||
project=project,
|
||||
timestamp_ms=ts_base,
|
||||
)
|
||||
written += 1
|
||||
return written
|
||||
|
||||
|
||||
# ── source 2: agentmemory ─────────────────────────────────────────────────────
|
||||
|
||||
def harvest_agentmemory(agentmemory_path: str, out_dir: str,
|
||||
workspaces: List[str]) -> int:
|
||||
if not os.path.isfile(agentmemory_path):
|
||||
return 0
|
||||
with open(agentmemory_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
memories: Dict[str, Any] = data.get("mem:memories", {})
|
||||
written = 0
|
||||
base_ts = datetime.now(tz=timezone.utc).timestamp() * 1000 - len(memories) * 60_000
|
||||
for i, (mem_id, mem) in enumerate(memories.items()):
|
||||
title = str(mem.get("title", "")).strip()
|
||||
content = str(mem.get("content", "")).strip()
|
||||
if not title or not content:
|
||||
continue
|
||||
project = _infer_project(title + " " + content, workspaces)
|
||||
ts = base_ts + i * 60_000
|
||||
_write_session(out_dir, project, mem_id,
|
||||
user_prompts=[title],
|
||||
assistant_replies=[content],
|
||||
timestamp_base_ms=ts)
|
||||
_append_history(out_dir, display=title[:120], project=project, timestamp_ms=ts)
|
||||
written += 1
|
||||
return written
|
||||
|
||||
# ── source 3: skill files (.devin/skills) ─────────────────────────────────────
|
||||
|
||||
def harvest_skills(workspaces: List[str], out_dir: str) -> int:
|
||||
written = 0
|
||||
seen_ids: set = set()
|
||||
for ws in workspaces:
|
||||
skills_root = os.path.join(ws, ".devin", "skills")
|
||||
if not os.path.isdir(skills_root):
|
||||
continue
|
||||
for skill_dir in os.scandir(skills_root):
|
||||
if not skill_dir.is_dir():
|
||||
continue
|
||||
skill_md = os.path.join(skill_dir.path, "SKILL.md")
|
||||
if not os.path.isfile(skill_md):
|
||||
continue
|
||||
sid = f"skill_{skill_dir.name}"
|
||||
if sid in seen_ids:
|
||||
continue
|
||||
seen_ids.add(sid)
|
||||
with open(skill_md, encoding="utf-8") as f:
|
||||
raw = f.read()
|
||||
body = re.sub(r"^---.*?---\s*", "", raw, flags=re.DOTALL).strip()
|
||||
if not body:
|
||||
continue
|
||||
first_line = body.split("\n")[0].lstrip("# ").strip()
|
||||
user_ask = f"Please use the {skill_dir.name} skill: {first_line}"
|
||||
ts = datetime.now(tz=timezone.utc).timestamp() * 1000 - 3_600_000
|
||||
_write_session(out_dir, ws, sid,
|
||||
user_prompts=[user_ask],
|
||||
assistant_replies=[body[:1200]],
|
||||
timestamp_base_ms=ts)
|
||||
_append_history(out_dir, display=user_ask[:120], project=ws, timestamp_ms=ts)
|
||||
written += 1
|
||||
return written
|
||||
|
||||
# ── main ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
def main(argv=None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate SkillOpt-Sleep transcripts from Devin local data"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out-dir",
|
||||
default=os.path.expanduser("~/.skillopt-sleep-devin"),
|
||||
help="Output claude_home dir (default: ~/.skillopt-sleep-devin)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--agentmemory",
|
||||
default=os.path.expanduser("~/.agentmemory/standalone.json"),
|
||||
help="Path to agentmemory standalone.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--devin-transcripts",
|
||||
default=_first_existing(_devin_transcript_candidates()),
|
||||
help="Devin CLI ATIF transcripts directory (default: per-OS auto-detect)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--workspaces", nargs="*",
|
||||
help="Workspace paths (default: auto-detect from Devin registry)",
|
||||
)
|
||||
parser.add_argument("--quiet", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
out_dir = os.path.expanduser(args.out_dir)
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
os.makedirs(os.path.join(out_dir, "projects"), exist_ok=True)
|
||||
|
||||
workspaces = args.workspaces or _detect_workspaces()
|
||||
workspaces = [ws for ws in workspaces if os.path.isdir(ws)]
|
||||
if not workspaces:
|
||||
workspaces = [os.getcwd()]
|
||||
|
||||
total = 0
|
||||
devin_transcripts = os.path.expanduser(args.devin_transcripts)
|
||||
n = harvest_devin_transcripts(devin_transcripts, out_dir, workspaces)
|
||||
if not args.quiet:
|
||||
print(f"[harvest_devin] devin : {n} sessions")
|
||||
total += n
|
||||
|
||||
n = harvest_agentmemory(args.agentmemory, out_dir, workspaces)
|
||||
if not args.quiet:
|
||||
print(f"[harvest_devin] agentmemory : {n} sessions")
|
||||
total += n
|
||||
|
||||
n = harvest_skills(workspaces, out_dir)
|
||||
if not args.quiet:
|
||||
print(f"[harvest_devin] skill files : {n} sessions")
|
||||
total += n
|
||||
|
||||
if not args.quiet:
|
||||
print(f"[harvest_devin] total : {total} synthetic sessions → {out_dir}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Reference judge for SkillOpt-Sleep — score a candidate reply against a rubric.
|
||||
|
||||
Tasks harvested without a hard test/build signal get ``verifier: "judge"`` and a
|
||||
``rubric`` (see ``_build_rubric`` in harvest_devin.py). This module is the
|
||||
scorer the validation gate calls for those tasks: given the rubric and a
|
||||
candidate reply produced during replay, it returns a score in ``[0, 1]``. The
|
||||
gate accepts a skill edit only if the *new* skill scores strictly higher on the
|
||||
held-out tasks.
|
||||
|
||||
It is self-contained on purpose — in a full deployment the SkillOpt engine owns
|
||||
replay+scoring, but having a runnable reference here lets you sanity-check the
|
||||
judge path without the engine.
|
||||
|
||||
Backends (select via ``SKILLOPT_JUDGE``):
|
||||
* ``heuristic`` (default) — keyword-coverage, offline, no API key, deterministic.
|
||||
* ``claude`` — LLM judge via the Anthropic API (needs ANTHROPIC_API_KEY).
|
||||
|
||||
Usage:
|
||||
python judge.py --rubric rubric.json --reply reply.txt
|
||||
echo "<reply>" | python judge.py --rubric-inline '["Addresses OrderService", ...]'
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from typing import List
|
||||
|
||||
_STOPWORDS = {"addresses", "resolves", "implements", "without", "introducing",
|
||||
"behavior", "request", "response", "concrete", "actionable", "not",
|
||||
"the", "and", "that", "with", "stated", "reported", "actually",
|
||||
"preserves", "improving", "structure", "requested", "satisfies"}
|
||||
|
||||
# Cheap, fast model is the right default for a judge.
|
||||
_JUDGE_MODEL = os.environ.get("SKILLOPT_JUDGE_MODEL", "claude-haiku-4-5-20251001")
|
||||
|
||||
|
||||
def _content_words(text: str) -> List[str]:
|
||||
return [w for w in re.findall(r"[A-Za-z][A-Za-z0-9_.\-]{3,}", text.lower())
|
||||
if w not in _STOPWORDS]
|
||||
|
||||
|
||||
def heuristic_score(reply: str, rubric: List[str]) -> float:
|
||||
"""Fraction of rubric criteria whose key content words appear in the reply.
|
||||
|
||||
Crude but deterministic: each criterion is 'met' if at least one of its
|
||||
content words shows up in the candidate reply. Good enough to smoke-test the
|
||||
gate wiring; swap in the claude backend for real judging.
|
||||
"""
|
||||
if not rubric:
|
||||
return 0.0
|
||||
low = reply.lower()
|
||||
met = 0
|
||||
for criterion in rubric:
|
||||
words = _content_words(criterion)
|
||||
if not words: # nothing to check → treat as met
|
||||
met += 1
|
||||
continue
|
||||
if any(w in low for w in words):
|
||||
met += 1
|
||||
return round(met / len(rubric), 3)
|
||||
|
||||
|
||||
def claude_score(reply: str, rubric: List[str]) -> float:
|
||||
"""LLM judge via the Anthropic API. Returns a 0..1 score.
|
||||
|
||||
Stdlib-only (urllib) so this file stays dependency-free. Falls back to the
|
||||
heuristic if the key is missing or the call fails, so the gate never hard-errors.
|
||||
"""
|
||||
api_key = os.environ.get("ANTHROPIC_API_KEY")
|
||||
if not api_key:
|
||||
print("[judge] ANTHROPIC_API_KEY unset — using heuristic", file=sys.stderr)
|
||||
return heuristic_score(reply, rubric)
|
||||
import urllib.request
|
||||
|
||||
rubric_block = "\n".join(f"- {c}" for c in rubric)
|
||||
prompt = (
|
||||
"You are scoring an AI agent's reply against a rubric. For each criterion, "
|
||||
"decide if the reply satisfies it. Respond with ONLY a number between 0 and "
|
||||
"1 — the fraction of criteria satisfied.\n\n"
|
||||
f"Rubric:\n{rubric_block}\n\nReply:\n{reply}\n\nScore:"
|
||||
)
|
||||
body = json.dumps({
|
||||
"model": _JUDGE_MODEL,
|
||||
"max_tokens": 8,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
}).encode()
|
||||
req = urllib.request.Request(
|
||||
"https://api.anthropic.com/v1/messages", data=body,
|
||||
headers={"content-type": "application/json", "x-api-key": api_key,
|
||||
"anthropic-version": "2023-06-01"},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
data = json.load(resp)
|
||||
text = "".join(b.get("text", "") for b in data.get("content", []))
|
||||
m = re.search(r"[01](?:\.\d+)?", text)
|
||||
return max(0.0, min(1.0, float(m.group(0)))) if m else heuristic_score(reply, rubric)
|
||||
except Exception as exc: # network/auth/parse — degrade gracefully
|
||||
print(f"[judge] claude backend failed ({exc}) — using heuristic", file=sys.stderr)
|
||||
return heuristic_score(reply, rubric)
|
||||
|
||||
|
||||
def score(reply: str, rubric: List[str]) -> float:
|
||||
backend = os.environ.get("SKILLOPT_JUDGE", "heuristic")
|
||||
return claude_score(reply, rubric) if backend == "claude" else heuristic_score(reply, rubric)
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
p = argparse.ArgumentParser(description="Score a reply against a rubric (0..1)")
|
||||
g = p.add_mutually_exclusive_group(required=True)
|
||||
g.add_argument("--rubric", help="Path to a JSON file containing a list of criteria")
|
||||
g.add_argument("--rubric-inline", help="Inline JSON list of criteria")
|
||||
p.add_argument("--reply", help="Path to the reply text (default: stdin)")
|
||||
args = p.parse_args(argv)
|
||||
|
||||
rubric = (json.load(open(args.rubric, encoding="utf-8")) if args.rubric
|
||||
else json.loads(args.rubric_inline))
|
||||
reply = (open(args.reply, encoding="utf-8").read() if args.reply
|
||||
else sys.stdin.read())
|
||||
print(score(reply, rubric))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"skillopt-sleep": {
|
||||
"command": "python3",
|
||||
"args": ["/abs/path/to/SkillOpt/plugins/devin/mcp_server.py"],
|
||||
"env": {
|
||||
"SKILLOPT_DEVIN_CLAUDE_HOME": "~/.skillopt-sleep-devin"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SkillOpt-Sleep — Devin MCP server (stdio, stdlib-only).
|
||||
|
||||
Exposes the sleep engine as MCP tools so Devin (Cognition) can drive it. No
|
||||
third-party deps: speaks JSON-RPC 2.0 over stdio with just the handful of MCP
|
||||
methods clients need. Same `sleep_*` interface and engine flags as
|
||||
`plugins/copilot`, plus a Devin-specific harvest step.
|
||||
|
||||
Before each data-reading action this server runs `harvest_devin.py` to convert
|
||||
locally available Devin data (ATIF-v1.7 transcripts, agentmemory memories, and
|
||||
.devin skill files) into the Claude Code-compatible JSONL the engine consumes,
|
||||
writing it under SKILLOPT_DEVIN_CLAUDE_HOME and pointing the engine there with
|
||||
`--claude-home`. After `sleep_adopt` the evolved skill is synced back into the
|
||||
workspace's `.devin/skills/`.
|
||||
|
||||
Tools: sleep_status, sleep_dry_run, sleep_run, sleep_adopt, sleep_harvest,
|
||||
sleep_schedule, sleep_unschedule. Each shells out to
|
||||
`python -m skillopt_sleep <action> ...`. Configure Devin to launch:
|
||||
python plugins/devin/mcp_server.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
# expanduser wraps the whole value so a "~/..." env var is expanded too (not
|
||||
# just a default) — otherwise a literal ~ dir gets created.
|
||||
REPO_ROOT = os.path.expanduser(
|
||||
os.environ.get("SKILLOPT_SLEEP_REPO")
|
||||
or os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
)
|
||||
PLUGIN_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
CLAUDE_HOME = os.path.expanduser(
|
||||
os.environ.get("SKILLOPT_DEVIN_CLAUDE_HOME", "~/.skillopt-sleep-devin")
|
||||
)
|
||||
MANAGED_SKILL_NAME = os.environ.get("SKILLOPT_MANAGED_SKILL", "skillopt-sleep-learned")
|
||||
PROTOCOL_VERSION = "2024-11-05"
|
||||
|
||||
TOOLS = [
|
||||
{"name": "sleep_status", "action": "status",
|
||||
"description": "Show how many SkillOpt-Sleep nights have run and the latest staged proposal."},
|
||||
{"name": "sleep_dry_run", "action": "dry-run",
|
||||
"description": "Preview a sleep cycle (harvest+mine+replay) without staging anything."},
|
||||
{"name": "sleep_run", "action": "run",
|
||||
"description": "Run a full sleep cycle; stages a reviewed proposal. Nothing live changes until adopt."},
|
||||
{"name": "sleep_adopt", "action": "adopt",
|
||||
"description": "Apply the latest staged proposal to the managed SKILL.md and sync it into .devin/skills/."},
|
||||
{"name": "sleep_harvest", "action": "harvest",
|
||||
"description": "Debug: list the recurring tasks mined from recent Devin sessions."},
|
||||
{"name": "sleep_schedule", "action": "schedule",
|
||||
"description": "Install a nightly cron entry to run the sleep cycle automatically."},
|
||||
{"name": "sleep_unschedule", "action": "unschedule",
|
||||
"description": "Remove the nightly cron entry for a project."},
|
||||
]
|
||||
_BY_NAME = {t["name"]: t for t in TOOLS}
|
||||
|
||||
_TOOL_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"project": {"type": "string",
|
||||
"description": "Project dir to evolve (default: cwd)."},
|
||||
"backend": {"type": "string", "enum": ["mock", "claude", "codex", "copilot"],
|
||||
"description": "mock = no API spend (default); claude/codex/copilot = real."},
|
||||
"scope": {"type": "string", "enum": ["invoked", "all"],
|
||||
"description": "Harvest scope (default: invoked project only)."},
|
||||
"source": {"type": "string", "enum": ["claude", "codex", "auto"],
|
||||
"description": "Transcript source (default: claude)."},
|
||||
"model": {"type": "string",
|
||||
"description": "Backend-specific model override."},
|
||||
"tasks_file": {"type": "string",
|
||||
"description": "Path to reviewed TaskRecord JSON (skips harvest)."},
|
||||
"target_skill_path": {"type": "string",
|
||||
"description": "Explicit SKILL.md path to evolve/stage/adopt."},
|
||||
"progress": {"type": "boolean",
|
||||
"description": "Print phase progress to stderr."},
|
||||
"max_sessions": {"type": "integer",
|
||||
"description": "Cap harvested sessions per run."},
|
||||
"max_tasks": {"type": "integer",
|
||||
"description": "Cap mined tasks per run."},
|
||||
"lookback_hours": {"type": "integer",
|
||||
"description": "Harvest window in hours (default: 72)."},
|
||||
"auto_adopt": {"type": "boolean",
|
||||
"description": "Auto-adopt if gate passes (default: false)."},
|
||||
"json": {"type": "boolean",
|
||||
"description": "Return machine-readable JSON output."},
|
||||
"edit_budget": {"type": "integer",
|
||||
"description": "Max bounded edits per night (default: 4)."},
|
||||
"hour": {"type": "integer",
|
||||
"description": "Hour for schedule (0-23, default: 3)."},
|
||||
"minute": {"type": "integer",
|
||||
"description": "Minute for schedule (0-59, default: 17)."},
|
||||
},
|
||||
"additionalProperties": False,
|
||||
}
|
||||
|
||||
# actions that read harvested Devin data (schedule/unschedule/adopt don't)
|
||||
_HARVEST_ACTIONS = {"status", "dry-run", "run", "harvest"}
|
||||
|
||||
|
||||
def _run_harvest() -> str:
|
||||
"""Convert local Devin data into the JSONL the engine reads, under CLAUDE_HOME."""
|
||||
harvester = os.path.join(PLUGIN_DIR, "harvest_devin.py")
|
||||
env = dict(os.environ)
|
||||
env["PYTHONPATH"] = REPO_ROOT + os.pathsep + env.get("PYTHONPATH", "")
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, harvester, "--out-dir", CLAUDE_HOME],
|
||||
capture_output=True, text=True, timeout=60, env=env,
|
||||
)
|
||||
out = (proc.stdout or "").strip()
|
||||
err = (proc.stderr or "").strip()
|
||||
return out + (("\n[harvest stderr]\n" + err) if err else "")
|
||||
except Exception as exc:
|
||||
return f"[harvest_devin] warning: {exc}"
|
||||
|
||||
|
||||
def _sync_skill(project: str) -> str:
|
||||
"""After adopt, copy the evolved skill into the workspace's .devin/skills/."""
|
||||
src = os.path.join(CLAUDE_HOME, "skills", MANAGED_SKILL_NAME, "SKILL.md")
|
||||
if not (os.path.isfile(src) and project and os.path.isdir(project)):
|
||||
return ""
|
||||
dot_root = os.path.join(project, ".devin")
|
||||
if not os.path.isdir(dot_root):
|
||||
return ""
|
||||
dst_dir = os.path.join(dot_root, "skills", MANAGED_SKILL_NAME)
|
||||
os.makedirs(dst_dir, exist_ok=True)
|
||||
dst = os.path.join(dst_dir, "SKILL.md")
|
||||
shutil.copy2(src, dst)
|
||||
return f"\n[sleep] synced evolved skill → {dst}"
|
||||
|
||||
|
||||
def _run_engine(action: str, args: dict) -> str:
|
||||
harvest_out = _run_harvest() if action in _HARVEST_ACTIONS else ""
|
||||
|
||||
py = sys.executable or "python3"
|
||||
cmd = [py, "-m", "skillopt_sleep", action, "--claude-home", CLAUDE_HOME]
|
||||
# Devin transcripts are converted to the Claude format, so default source=claude
|
||||
if not args.get("source"):
|
||||
cmd += ["--source", "claude"]
|
||||
# String-valued flags
|
||||
for flag, key in [
|
||||
("--project", "project"), ("--backend", "backend"),
|
||||
("--scope", "scope"), ("--source", "source"),
|
||||
("--model", "model"), ("--tasks-file", "tasks_file"),
|
||||
("--target-skill-path", "target_skill_path"),
|
||||
]:
|
||||
val = args.get(key)
|
||||
if val:
|
||||
cmd += [flag, str(val)]
|
||||
# Integer-valued flags
|
||||
for flag, key in [
|
||||
("--max-sessions", "max_sessions"), ("--max-tasks", "max_tasks"),
|
||||
("--lookback-hours", "lookback_hours"), ("--edit-budget", "edit_budget"),
|
||||
("--hour", "hour"), ("--minute", "minute"),
|
||||
]:
|
||||
val = args.get(key)
|
||||
if val is not None:
|
||||
cmd += [flag, str(int(val))]
|
||||
# Boolean flags
|
||||
for flag, key in [
|
||||
("--progress", "progress"), ("--auto-adopt", "auto_adopt"), ("--json", "json"),
|
||||
]:
|
||||
if args.get(key):
|
||||
cmd.append(flag)
|
||||
|
||||
env = dict(os.environ)
|
||||
env["PYTHONPATH"] = REPO_ROOT + os.pathsep + env.get("PYTHONPATH", "")
|
||||
try:
|
||||
proc = subprocess.run(cmd, cwd=REPO_ROOT, capture_output=True,
|
||||
text=True, timeout=3600, env=env)
|
||||
except Exception as e:
|
||||
return f"[harvest]\n{harvest_out}\n[error] failed to run engine: {e}"
|
||||
out = (proc.stdout or "").strip()
|
||||
err = (proc.stderr or "").strip()
|
||||
result = (f"[harvest]\n{harvest_out}\n\n" if harvest_out else "") + f"[engine]\n{out}"
|
||||
if err:
|
||||
result += f"\n[stderr]\n{err}"
|
||||
if action == "adopt":
|
||||
result += _sync_skill(args.get("project") or os.getcwd())
|
||||
return result
|
||||
|
||||
|
||||
def _result(id_, result):
|
||||
return {"jsonrpc": "2.0", "id": id_, "result": result}
|
||||
|
||||
|
||||
def _error(id_, code, message):
|
||||
return {"jsonrpc": "2.0", "id": id_, "error": {"code": code, "message": message}}
|
||||
|
||||
|
||||
def handle(req: dict):
|
||||
method = req.get("method")
|
||||
id_ = req.get("id")
|
||||
if method == "initialize":
|
||||
return _result(id_, {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"capabilities": {"tools": {}},
|
||||
"serverInfo": {"name": "skillopt-sleep", "version": "0.1.0"},
|
||||
})
|
||||
if method in ("notifications/initialized", "initialized"):
|
||||
return None
|
||||
if method == "tools/list":
|
||||
return _result(id_, {"tools": [
|
||||
{"name": t["name"], "description": t["description"], "inputSchema": _TOOL_SCHEMA}
|
||||
for t in TOOLS
|
||||
]})
|
||||
if method == "tools/call":
|
||||
params = req.get("params") or {}
|
||||
name = params.get("name")
|
||||
tool = _BY_NAME.get(name)
|
||||
if not tool:
|
||||
return _error(id_, -32602, f"unknown tool: {name}")
|
||||
text = _run_engine(tool["action"], params.get("arguments") or {})
|
||||
return _result(id_, {"content": [{"type": "text", "text": text}]})
|
||||
if method == "ping":
|
||||
return _result(id_, {})
|
||||
return _error(id_, -32601, f"method not found: {method}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
for line in sys.stdin:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
req = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
resp = handle(req)
|
||||
if resp is not None:
|
||||
sys.stdout.write(json.dumps(resp) + "\n")
|
||||
sys.stdout.flush()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,112 @@
|
||||
# OpenClaw Plugin for SkillOpt-Sleep
|
||||
|
||||
Thin shell for running [SkillOpt-Sleep](https://github.com/microsoft/SkillOpt) on [OpenClaw](https://github.com/openclaw/openclaw).
|
||||
|
||||
## What it does
|
||||
|
||||
Adds a nightly "sleep cycle" to any OpenClaw agent. The cycle:
|
||||
|
||||
1. **Harvests** recent session transcripts from `~/.openclaw/agents/<name>/sessions/*.jsonl`
|
||||
2. **Mines** recurring task patterns using the optimizer LLM
|
||||
3. **Replays** each pattern with the current `SKILL.md` (baseline) and a candidate `SKILL.md` (with proposed edits)
|
||||
4. **Gates** the candidate against the held-out score (rejects regressions)
|
||||
5. **Stages** the accepted proposal in `~/.skillopt-sleep/staging/<night>/`
|
||||
6. Leaves adoption to the operator (Ethan)
|
||||
|
||||
Nothing live changes until you adopt. Every adopt backs up first.
|
||||
|
||||
## Install
|
||||
|
||||
The plugin is a thin wrapper around the engine at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/`:
|
||||
|
||||
```bash
|
||||
# 1. Clone the engine (one-time)
|
||||
cd ~/.openclaw/workspace
|
||||
git clone https://github.com/microsoft/SkillOpt.git
|
||||
|
||||
# 2. Install the OpenClaw skill (this folder)
|
||||
ln -s /path/to/openclaw ~/.openclaw/workspace/skills/skillopt-sleep
|
||||
|
||||
# 3. Configure
|
||||
cp ~/.openclaw/workspace/skills/skillopt-sleep/config.json ~/.skillopt-sleep/config.json
|
||||
$EDITOR ~/.skillopt-sleep/config.json
|
||||
# Set backend = "openclaw-deepseek"
|
||||
# Set model = "deepseek-v4-pro" (or "deepseek-v4-flash" for budget)
|
||||
|
||||
# 4. Set API key
|
||||
echo 'export DEEPSEEK_API_KEY="sk-..."' >> ~/.openclaw/.env
|
||||
|
||||
# 5. Add the nightly cron
|
||||
(crontab -l 2>/dev/null; echo "0 3 * * * cd ~/.openclaw/workspace/skills/skillopt-sleep && bash run_sleep_cron.sh >> ~/.skillopt-sleep/nightly.log 2>&1") | crontab -
|
||||
```
|
||||
|
||||
## Use
|
||||
|
||||
### Manual trigger
|
||||
|
||||
```bash
|
||||
# Run one cycle now
|
||||
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py
|
||||
|
||||
# Dry run (report only)
|
||||
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --dry-run
|
||||
|
||||
# One category only
|
||||
python3 ~/.openclaw/workspace/skills/skillopt-sleep/run_sleep.py --tasks tests/research-cron-tasks.json
|
||||
```
|
||||
|
||||
### Slash command
|
||||
|
||||
```bash
|
||||
# In any OpenClaw session
|
||||
/sleep status
|
||||
/sleep run
|
||||
/sleep run research-cron
|
||||
/sleep dry-run
|
||||
/sleep adopt # adopt most recent accepted proposal
|
||||
/sleep reject # discard most recent
|
||||
/sleep cost
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
plugins/openclaw/
|
||||
├── README.md # this file
|
||||
├── run_sleep_cron.sh # wrapper for cron invocation
|
||||
├── run_sleep.py # main entry point
|
||||
├── slash_sleep.py # /sleep command implementation
|
||||
├── skillopt_sleep_openclaw.py # DeepSeek + Ollama backend
|
||||
├── config.json # engine config
|
||||
├── SKILL.md # OpenClaw skill manifest
|
||||
└── tests/ # held-out test sets
|
||||
├── research-cron-tasks.json
|
||||
├── devops-tasks.json
|
||||
└── wiki-tasks.json
|
||||
```
|
||||
|
||||
The OpenClaw shell is one engine (skillopt_sleep/) + one backend (DeepSeek/Ollama) + four thin wrappers (cron, slash, skill, tests).
|
||||
|
||||
## Why this matters for OpenClaw
|
||||
|
||||
OpenClaw currently has no built-in "self-evolving skills" mechanism. The community has:
|
||||
|
||||
- **Manual skills** — Ethan writes them
|
||||
- **LLM-generated skills** — one-shot, no validation
|
||||
- **Self-revision** — unbounded, no quality bar
|
||||
|
||||
SkillOpt-Sleep adds a 4th option: **validated self-evolution**. The skill is the training target, the engine is the optimizer, the gate is the quality bar, the operator is the human-in-the-loop.
|
||||
|
||||
## Validation
|
||||
|
||||
Validated on the public [gbrain-evals](https://github.com/garrytan/gbrain-evals) `skillopt-v1` benchmark with real Claude and Codex (deficient skills 0.00 → 1.00 on held-out, all 4 seeds).
|
||||
|
||||
End-to-end test on our own 14-task held-out set: pipeline runs, gate correctly rejects non-improvements, staging artifacts land in `~/.skillopt-sleep/staging/<night>/`.
|
||||
|
||||
## Cost
|
||||
|
||||
Measured: ~$0.02/night with `deepseek-v4-pro` at 12 tasks/night. ~$0.59/month, $7.18/year.
|
||||
|
||||
## License
|
||||
|
||||
MIT (same as SkillOpt core).
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
name: skillopt-sleep
|
||||
description: Validate and refine agent skills through nightly sleep cycles with held-out gates. Wraps Microsoft's SkillOpt-Sleep engine for the OpenClaw/DeepSeek stack.
|
||||
---
|
||||
|
||||
# skillopt-sleep — OpenClaw Adaptation of Microsoft SkillOpt-Sleep
|
||||
|
||||
A nightly self-improvement loop that reads our session transcripts, mines recurring workflow patterns, replays them with proposed skill edits, and gates the proposals against a held-out test set. Only improvements that beat baseline are staged for human adoption.
|
||||
|
||||
## When To Use
|
||||
|
||||
- After Hermes's Weekly Skill Review (or as its replacement)
|
||||
- When a skill is being used 10+ times/week and could be tighter
|
||||
- Before promoting a new skill from `skill-proposals/` to `skills/`
|
||||
- When a skill regresses in observed quality
|
||||
|
||||
## What It Does (One Cycle)
|
||||
|
||||
```
|
||||
harvest session transcripts -> mine recurring task patterns
|
||||
-> replay each pattern (current skill vs proposed)
|
||||
-> GATE: must improve held-out score
|
||||
-> stage proposal
|
||||
-> Ethan adopts (manual)
|
||||
```
|
||||
|
||||
Nothing live changes until Ethan adopts. Every adopt backs up first.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
skills/skillopt-sleep/
|
||||
├── SKILL.md # this file
|
||||
├── config.json # engine config (backend, budgets, etc.)
|
||||
├── run_sleep.py # entry point
|
||||
└── skillopt_sleep_openclaw.py # DeepSeek/Ollama backend
|
||||
```
|
||||
|
||||
The engine itself is at `~/.openclaw/workspace/SkillOpt/skillopt_sleep/` (cloned from microsoft/SkillOpt).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Run one cycle with current config
|
||||
cd ~/.openclaw/workspace/skills/skillopt-sleep
|
||||
python3 run_sleep.py
|
||||
|
||||
# Dry run (report only, no staging)
|
||||
python3 run_sleep.py --dry-run
|
||||
|
||||
# Use a pre-built task set (recommended for testing)
|
||||
python3 run_sleep.py --tasks tests/research-cron-tasks.json
|
||||
```
|
||||
|
||||
## Scheduling
|
||||
|
||||
```bash
|
||||
python3 slash_sleep.py schedule --hour 3 --minute 17
|
||||
python3 slash_sleep.py unschedule
|
||||
python3 slash_sleep.py unschedule --all
|
||||
```
|
||||
|
||||
Installs a nightly cron entry using the shared SkillOpt-Sleep scheduler. This is an alternative to the external `run_sleep_cron.sh` script.
|
||||
|
||||
## Alternative backends
|
||||
|
||||
While OpenClaw defaults to `openclaw-deepseek` (DeepSeek V4 Pro + Ollama), the shared engine also supports:
|
||||
- `--backend mock` — deterministic, no API spend (for testing)
|
||||
- `--backend claude` — uses the Claude CLI
|
||||
- `--backend codex` — uses the Codex CLI
|
||||
- `--backend copilot` — uses the GitHub Copilot CLI
|
||||
|
||||
These can be used via the engine directly (`python -m skillopt_sleep`).
|
||||
|
||||
## Shared-engine flags
|
||||
|
||||
When invoking the engine directly, all standard flags are available:
|
||||
- `--source codex` / `--source auto` — harvest from Codex Desktop sessions
|
||||
- `--tasks-file PATH` — use a pre-built task set
|
||||
- `--target-skill-path PATH` — explicit SKILL.md target
|
||||
- `--max-tasks N` / `--max-sessions N` — cap workload
|
||||
- `--progress` — print phase progress
|
||||
- `--json` — machine-readable output
|
||||
- `--auto-adopt` — auto-adopt if gate passes
|
||||
|
||||
Config keys: `preferences`, `gate_mode`, `gate_metric`, `dream_rollouts`, `recall_k`, `evolve_memory`, `evolve_skill`.
|
||||
|
||||
## Config (config.json)
|
||||
|
||||
Key knobs:
|
||||
- `backend: "openclaw-deepseek"` — our custom backend
|
||||
- `model: "deepseek-v4-pro"` — optimizer model
|
||||
- `edit_budget: 3` — max bounded edits per night
|
||||
- `gate_mode: "on"` — validation-gated (rejects regressions)
|
||||
- `auto_adopt: false` — require Ethan to adopt manually
|
||||
- `max_tasks_per_night: 12` — cap to control cost
|
||||
|
||||
## Cost Estimate
|
||||
|
||||
Per night: 12 tasks × (1 attempt + 1 judge + 1 reflect) × ~$0.005/1K tokens × ~3K tokens/call ≈ **$0.50-2.00/night**.
|
||||
|
||||
## Outputs
|
||||
|
||||
- Report: `~/.skillopt-sleep/state.json` (running totals)
|
||||
- Staging: `~/.skillopt-sleep/staging/<night>/`
|
||||
- `report.md` — readable summary
|
||||
- `best_skill.md` — proposed skill
|
||||
- `edits.json` — bounded edit list
|
||||
- `before.md` / `after.md` — diffs
|
||||
|
||||
## Held-Out Test Sets (Phase 2)
|
||||
|
||||
Located at `tests/<category>-tasks.json`. Each task has:
|
||||
- `prompt` — the recurring task
|
||||
- `reference` — exact-match gold answer
|
||||
- `rubric` — soft score rubric (0-1)
|
||||
- `domain` — research/devops/wiki/etc.
|
||||
|
||||
Currently building for 3 categories:
|
||||
- research-cron-output
|
||||
- devops-infrastructure-check
|
||||
- wiki-canonical-guide
|
||||
|
||||
## When NOT To Use
|
||||
|
||||
- For a one-off workflow (not a recurring pattern)
|
||||
- During a crisis/incident (humans must lead)
|
||||
- When session transcripts are < 24h old (not enough signal)
|
||||
- For skills < 300 tokens (over-optimization risk)
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"_comment": "OpenClaw adaptation of skillopt-sleep. Edit and run via run_sleep.py",
|
||||
|
||||
"claude_home": "/home/ethanclaw/.openclaw/agents",
|
||||
"invoked_project": "/home/ethanclaw/.openclaw/workspace",
|
||||
"projects": "invoked",
|
||||
"lookback_hours": 168,
|
||||
|
||||
"max_tasks_per_night": 12,
|
||||
"max_tokens_per_night": 800000,
|
||||
"holdout_fraction": 0.34,
|
||||
"val_fraction": 0.34,
|
||||
"test_fraction": 0.0,
|
||||
|
||||
"backend": "openclaw-deepseek",
|
||||
"model": "deepseek-v4-pro",
|
||||
"gate_mode": "on",
|
||||
"edit_budget": 3,
|
||||
"gate_metric": "mixed",
|
||||
"gate_mixed_weight": 0.5,
|
||||
"replay_mode": "fresh",
|
||||
"evolve_memory": true,
|
||||
"evolve_skill": true,
|
||||
"llm_mine": false,
|
||||
|
||||
"auto_adopt": false,
|
||||
"managed_skill_name": "skillopt-sleep-learned",
|
||||
"redact_secrets": true,
|
||||
"seed": 42
|
||||
}
|
||||
Executable
+122
@@ -0,0 +1,122 @@
|
||||
#!/usr/bin/env python3
|
||||
"""run_sleep.py — OpenClaw entry point for SkillOpt-Sleep.
|
||||
|
||||
Runs one nightly sleep cycle:
|
||||
1. harvest recent session transcripts
|
||||
2. mine recurring task patterns
|
||||
3. replay tasks with current skill (baseline) + candidate skill (with proposed edit)
|
||||
4. gate candidate vs baseline on held-out accuracy
|
||||
5. stage the proposal in ~/.skillopt-sleep/staging/<night>/
|
||||
6. leave adoption to Ethan (auto_adopt=false)
|
||||
|
||||
Usage:
|
||||
python3 run_sleep.py # one cycle, default config
|
||||
python3 run_sleep.py --dry-run # compute report only, no staging
|
||||
python3 run_sleep.py --tasks path.json # use a pre-built task file
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Ensure the skillopt_sleep package is importable (it lives in the cloned repo)
|
||||
REPO = Path("/home/ethanclaw/.openclaw/workspace/SkillOpt")
|
||||
sys.path.insert(0, str(REPO))
|
||||
|
||||
# Register our backend before importing cycle
|
||||
from skillopt_sleep_openclaw import OpenClawDeepSeekBackend
|
||||
import skillopt_sleep.backend as _b
|
||||
_b._BACKENDS = getattr(_b, "_BACKENDS", {})
|
||||
_b._BACKENDS["openclaw-deepseek"] = OpenClawDeepSeekBackend
|
||||
|
||||
# Patch get_backend to know about our backend
|
||||
_orig_get_backend = _b.get_backend
|
||||
|
||||
def get_backend(name, model="", codex_path=""):
|
||||
if name == "openclaw-deepseek":
|
||||
return OpenClawDeepSeekBackend(model=model or "deepseek-v4-pro")
|
||||
return _orig_get_backend(name, model=model, codex_path=codex_path)
|
||||
|
||||
_b.get_backend = get_backend
|
||||
|
||||
from skillopt_sleep.cycle import run_sleep_cycle
|
||||
from skillopt_sleep.config import load_config
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="OpenClaw SkillOpt-Sleep nightly cycle")
|
||||
ap.add_argument("--dry-run", action="store_true", help="Compute but don't stage")
|
||||
ap.add_argument("--config", default="/home/ethanclaw/.openclaw/workspace/skills/skillopt-sleep/config.json")
|
||||
ap.add_argument("--tasks", default=None, help="Path to pre-built tasks JSON")
|
||||
ap.add_argument("--verbose", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
# Load config from file then override with our defaults
|
||||
overrides = {}
|
||||
if os.path.exists(args.config):
|
||||
with open(args.config) as f:
|
||||
overrides.update(json.load(f))
|
||||
overrides.pop("_comment", None)
|
||||
|
||||
cfg = load_config(**overrides)
|
||||
|
||||
seed_tasks = None
|
||||
if args.tasks:
|
||||
from skillopt_sleep.types import TaskRecord
|
||||
with open(args.tasks) as f:
|
||||
raw = json.load(f)
|
||||
# Translate our test-set fields → TaskRecord fields
|
||||
seed_tasks = []
|
||||
for t in raw:
|
||||
seed_tasks.append(TaskRecord(
|
||||
id=t['id'],
|
||||
project=t.get('project', 'openclaw'),
|
||||
intent=t.get('intent') or t.get('prompt', ''),
|
||||
context_excerpt=t.get('context_excerpt', ''),
|
||||
attempted_solution=t.get('attempted_solution', ''),
|
||||
outcome=t.get('outcome', 'unknown'),
|
||||
reference_kind=t.get('reference_kind', 'rubric'),
|
||||
reference=t.get('reference', ''),
|
||||
judge=t.get('judge', {}),
|
||||
tags=t.get('tags', []),
|
||||
source_sessions=t.get('source_sessions', []),
|
||||
split=t.get('split', 'train'),
|
||||
))
|
||||
|
||||
print(f"[skillopt-sleep] starting cycle...")
|
||||
print(f" backend: {cfg.get('backend')}")
|
||||
print(f" project: {cfg.get('invoked_project')}")
|
||||
print(f" max tasks: {cfg.get('max_tasks_per_night')}")
|
||||
print(f" edit budget: {cfg.get('edit_budget')}")
|
||||
print(f" dry_run: {args.dry_run}")
|
||||
|
||||
outcome = run_sleep_cycle(cfg, seed_tasks=seed_tasks, dry_run=args.dry_run)
|
||||
|
||||
r = outcome.report
|
||||
print(f"\n=== Report — night {r.night} ===")
|
||||
print(f" sessions harvested: {r.n_sessions}")
|
||||
print(f" tasks mined: {r.n_tasks} (replayed: {r.n_replayed})")
|
||||
print(f" baseline: {r.baseline_score:.3f} -> candidate: {r.candidate_score:.3f}")
|
||||
print(f" gate: {r.gate_action} accepted={r.accepted}")
|
||||
print(f" tokens: {r.tokens_used}")
|
||||
if r.edits:
|
||||
print(f" applied edits ({len(r.edits)}):")
|
||||
for e in r.edits:
|
||||
print(f" [{e.target}/{e.op}] {e.content[:80]}...")
|
||||
if r.rejected_edits:
|
||||
print(f" rejected edits ({len(r.rejected_edits)}) — kept as negative feedback")
|
||||
if r.notes:
|
||||
for n in r.notes:
|
||||
print(f" note: {n}")
|
||||
if outcome.staging_dir:
|
||||
print(f"\n STAGED at: {outcome.staging_dir}")
|
||||
print(f" Review with: ls {outcome.staging_dir}")
|
||||
|
||||
return 0 if r.accepted or r.candidate_score >= r.baseline_score else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+76
@@ -0,0 +1,76 @@
|
||||
#!/bin/bash
|
||||
# run_sleep_cron.sh — wrapper for cron-driven nightly sleep cycle
|
||||
#
|
||||
# Usage: bash run_sleep_cron.sh [category1 category2 ...]
|
||||
# No args: run on all categories in tests/
|
||||
# With args: run only on listed categories (research-cron, devops, wiki)
|
||||
#
|
||||
# Cron (3am MYT daily):
|
||||
# 0 3 * * * cd /home/ethanclaw/.openclaw/workspace/skills/skillopt-sleep && bash run_sleep_cron.sh >> ~/.skillopt-sleep/nightly.log 2>&1
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SKILL_DIR="/home/ethanclaw/.openclaw/workspace/skills/skillopt-sleep"
|
||||
TESTS_DIR="$SKILL_DIR/tests"
|
||||
LOG_DIR="$HOME/.skillopt-sleep/logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
|
||||
LOG_FILE="$LOG_DIR/night-$TIMESTAMP.log"
|
||||
|
||||
# category → test file map
|
||||
declare -A CATEGORIES=(
|
||||
["research-cron"]="research-cron-tasks.json"
|
||||
["devops"]="devops-tasks.json"
|
||||
["wiki"]="wiki-tasks.json"
|
||||
)
|
||||
|
||||
# Determine which categories to run
|
||||
if [ $# -eq 0 ]; then
|
||||
CATS=("research-cron" "devops" "wiki")
|
||||
else
|
||||
CATS=("$@")
|
||||
fi
|
||||
|
||||
{
|
||||
echo "=========================================="
|
||||
echo "SkillOpt-Sleep nightly — $TIMESTAMP"
|
||||
echo "Categories: ${CATS[*]}"
|
||||
echo "=========================================="
|
||||
} | tee -a "$LOG_FILE"
|
||||
|
||||
# Pre-flight: check DeepSeek API key
|
||||
if ! grep -q "DEEPSEEK_API_KEY=" "$HOME/.openclaw/.env" 2>/dev/null; then
|
||||
echo "ERROR: DEEPSEEK_API_KEY not found in ~/.openclaw/.env" | tee -a "$LOG_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
EXIT_CODE=0
|
||||
for cat in "${CATS[@]}"; do
|
||||
tasks_file="$TESTS_DIR/${CATEGORIES[$cat]:-}"
|
||||
if [ ! -f "$tasks_file" ]; then
|
||||
echo "SKIP: $cat (no tasks file: $tasks_file)" | tee -a "$LOG_FILE"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "" | tee -a "$LOG_FILE"
|
||||
echo "--- [$cat] starting cycle ---" | tee -a "$LOG_FILE"
|
||||
|
||||
cd "$SKILL_DIR"
|
||||
if python3 run_sleep.py --tasks "$tasks_file" 2>&1 | tee -a "$LOG_FILE"; then
|
||||
echo "--- [$cat] OK ---" | tee -a "$LOG_FILE"
|
||||
else
|
||||
EC=$?
|
||||
echo "--- [$cat] FAILED (exit $EC) ---" | tee -a "$LOG_FILE"
|
||||
EXIT_CODE=$EC
|
||||
fi
|
||||
done
|
||||
|
||||
{
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Done. Exit: $EXIT_CODE"
|
||||
echo "=========================================="
|
||||
} | tee -a "$LOG_FILE"
|
||||
|
||||
exit $EXIT_CODE
|
||||
@@ -0,0 +1,275 @@
|
||||
"""OpenClaw backend for SkillOpt-Sleep.
|
||||
|
||||
Adapts the skillopt_sleep Backend protocol to our DeepSeek + Ollama stack:
|
||||
- attempt/judge/reflect -> DeepSeek V4 Pro (or Flash for cost)
|
||||
- embeddings -> Ollama nomic-embed-text (already configured)
|
||||
|
||||
This backend NEVER mutates live state. It only returns text + EditRecord
|
||||
proposals that the gate stages for human review.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from skillopt_sleep.backend import Backend, _normalize, exact_score
|
||||
from skillopt_sleep.types import EditRecord, ReplayResult, TaskRecord
|
||||
|
||||
|
||||
# ── DeepSeek + Ollama OpenAI-compatible API client (curl-based, no extra deps) ──
|
||||
|
||||
|
||||
def _chat(messages: List[Dict[str, str]], *, model: str, temperature: float = 0.2, max_tokens: int = 1500) -> str:
|
||||
"""Call DeepSeek V4 Pro via curl + jq. No extra Python deps needed."""
|
||||
import json as _json
|
||||
import urllib.request
|
||||
|
||||
api_key = os.environ.get("DEEPSEEK_API_KEY", "")
|
||||
if not api_key:
|
||||
# try loading from .env
|
||||
env_path = os.path.expanduser("~/.openclaw/.env")
|
||||
if os.path.exists(env_path):
|
||||
with open(env_path) as f:
|
||||
for line in f:
|
||||
if line.startswith("DEEPSEEK_API_KEY="):
|
||||
api_key = line.split("=", 1)[1].strip()
|
||||
break
|
||||
|
||||
base = os.environ.get("DEEPSEEK_BASE_URL", "https://api.deepseek.com/v1")
|
||||
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
"stream": False,
|
||||
}
|
||||
req = urllib.request.Request(
|
||||
f"{base}/chat/completions",
|
||||
data=_json.dumps(payload).encode("utf-8"),
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=180) as resp:
|
||||
data = _json.loads(resp.read().decode("utf-8"))
|
||||
return data["choices"][0]["message"]["content"]
|
||||
except Exception as e:
|
||||
return f"[BACKEND_ERROR] {type(e).__name__}: {str(e)[:200]}"
|
||||
|
||||
|
||||
def _embed(text: str) -> List[float]:
|
||||
"""Call Ollama for embeddings. Uses the configured nomic-embed-text model."""
|
||||
import json as _json
|
||||
import urllib.request
|
||||
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
"http://127.0.0.1:11434/api/embeddings",
|
||||
data=_json.dumps({"model": "nomic-embed-text:latest", "prompt": text[:2000]}).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
data = _json.loads(resp.read().decode("utf-8"))
|
||||
return data.get("embedding", [])
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
# ── Backend implementation ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class OpenClawDeepSeekBackend(Backend):
|
||||
"""Use DeepSeek V4 Pro for attempt/judge/reflect, Ollama for embeddings.
|
||||
|
||||
- "model" passed to constructor = optimizer model (default: deepseek-v4-pro)
|
||||
- "judge_model" = judge model (default: deepseek-v4-pro for quality)
|
||||
- "cheap_model" = budget-fallback (deepseek-v4-flash)
|
||||
"""
|
||||
|
||||
name = "openclaw-deepseek"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model: str = "deepseek-v4-pro",
|
||||
judge_model: str = "deepseek-v4-pro",
|
||||
cheap_model: str = "deepseek-v4-flash",
|
||||
):
|
||||
self._model = model
|
||||
self._judge_model = judge_model
|
||||
self._cheap_model = cheap_model
|
||||
self._tokens = 0 # rough estimate
|
||||
|
||||
def tokens_used(self) -> int:
|
||||
return self._tokens
|
||||
|
||||
# ── 1. attempt: produce a response given the task + skill + memory ──
|
||||
def attempt(self, task: TaskRecord, skill: str, memory: str) -> str:
|
||||
sys = (
|
||||
"You are an OpenClaw agent (Kobe ecosystem). Use the skill and memory below to complete the task. "
|
||||
"If the task asks for a structured output, follow the rubric exactly. "
|
||||
"Be concise. No preamble, no explanation unless the task asks for it."
|
||||
)
|
||||
usr = f"""## SKILL
|
||||
{skill or '(no skill yet)'}
|
||||
|
||||
## MEMORY
|
||||
{memory or '(no memory yet)'}
|
||||
|
||||
## TASK
|
||||
{task.intent}
|
||||
|
||||
## CONTEXT (if any)
|
||||
{task.context_excerpt or '(none)'}
|
||||
|
||||
## RESPONSE
|
||||
"""
|
||||
out = _chat(
|
||||
[{"role": "system", "content": sys}, {"role": "user", "content": usr}],
|
||||
model=self._model,
|
||||
temperature=0.2,
|
||||
)
|
||||
self._tokens += len(usr) // 4 + 200
|
||||
return out
|
||||
|
||||
# ── 2. judge: score the response ──
|
||||
def judge(self, task: TaskRecord, response: str) -> Tuple[float, float, str]:
|
||||
# Hard score: exact-match against task.reference (if available)
|
||||
hard = exact_score(task.reference or "", response)
|
||||
|
||||
# Soft score: LLM judge against rubric (reference if reference_kind=='rubric')
|
||||
rubric_text = task.reference if task.reference_kind == "rubric" else ""
|
||||
if rubric_text:
|
||||
judge_prompt = f"""You are a strict grader. Score the response 0.0-1.0 against the rubric.
|
||||
|
||||
## TASK
|
||||
{task.intent}
|
||||
|
||||
## REFERENCE
|
||||
{task.reference or '(none)'}
|
||||
|
||||
## RUBRIC
|
||||
{rubric_text}
|
||||
|
||||
## RESPONSE
|
||||
{response[:3000]}
|
||||
|
||||
## INSTRUCTIONS
|
||||
Return ONLY a single float 0.0-1.0 on one line. No explanation. No markdown.
|
||||
"""
|
||||
try:
|
||||
j_out = _chat(
|
||||
[{"role": "user", "content": judge_prompt}],
|
||||
model=self._judge_model,
|
||||
temperature=0.0,
|
||||
max_tokens=20,
|
||||
).strip()
|
||||
soft = float(re.search(r"[\d.]+", j_out.splitlines()[0]).group())
|
||||
soft = max(0.0, min(1.0, soft))
|
||||
except Exception:
|
||||
soft = hard
|
||||
self._tokens += 600
|
||||
else:
|
||||
soft = hard
|
||||
|
||||
rationale = f"hard={hard:.2f} soft={soft:.2f}"
|
||||
return hard, soft, rationale
|
||||
|
||||
# ── 3. reflect: produce bounded EditRecord proposals ──
|
||||
def reflect(
|
||||
self,
|
||||
failures: List[Tuple[TaskRecord, ReplayResult]],
|
||||
successes: List[Tuple[TaskRecord, ReplayResult]],
|
||||
skill: str,
|
||||
memory: str,
|
||||
*,
|
||||
edit_budget: int,
|
||||
evolve_skill: bool,
|
||||
evolve_memory: bool,
|
||||
) -> List[EditRecord]:
|
||||
# Compact digest of failures + successes
|
||||
fail_digest = "\n".join(
|
||||
f"- TASK: {t.intent[:200]}\n RESPONSE: {r.response[:300]}\n WHY FAIL: {r.judge_rationale or r.fail_reason or 'unknown'}\n REFERENCE: {t.reference[:200]}"
|
||||
for t, r in failures[:5]
|
||||
) or "(none)"
|
||||
succ_digest = "\n".join(
|
||||
f"- TASK: {t.intent[:150]} -> OK ({r.judge_rationale or 'high score'})"
|
||||
for t, r in successes[:3]
|
||||
) or "(none)"
|
||||
|
||||
rubric_text = ""
|
||||
if failures:
|
||||
rubric_text = f"\n\n## REFERENCE ANSWERS\n{chr(10).join(f'Q: {t.intent[:120]}\\nA: {t.reference}' for t, _ in failures[:3] if t.reference)}"
|
||||
|
||||
sys = (
|
||||
"You are SkillOpt-Sleep's bounded-edit optimizer. Your job is to propose 1-4 MINIMAL text edits to a skill or memory document "
|
||||
"that, if applied, would help future agents do better on the failed tasks. "
|
||||
"NEVER propose adding new sections wholesale. NEVER delete entire sections. "
|
||||
"Edit primitives: ADD (append a step/rule at end), DELETE (remove a specific line by exact match), REPLACE (swap a specific line for another by exact match). "
|
||||
"If you cannot identify a clear, minimal improvement, return an empty list."
|
||||
)
|
||||
usr = f"""## CURRENT SKILL
|
||||
{skill or '(empty)'}
|
||||
|
||||
## CURRENT MEMORY
|
||||
{memory or '(empty)'}
|
||||
|
||||
## FAILED TASKS
|
||||
{fail_digest}
|
||||
|
||||
## SUCCESSFUL TASKS
|
||||
{succ_digest}
|
||||
{rubric_text}
|
||||
|
||||
## CONSTRAINTS
|
||||
- max {edit_budget} edits total
|
||||
- edits go to {"skill + memory" if (evolve_skill and evolve_memory) else ("skill" if evolve_skill else "memory")}
|
||||
- if evolve_skill=False, target="memory" only; if evolve_memory=False, target="skill" only
|
||||
- target must be "skill" or "memory"
|
||||
|
||||
## OUTPUT FORMAT (JSON, no markdown)
|
||||
{{"edits": [{{"op": "ADD"|"DELETE"|"REPLACE", "target": "skill"|"memory", "content": "the text to add or replace with", "old_text": "for REPLACE/DELETE, the exact line to find", "rationale": "one short sentence why"}}]}}
|
||||
"""
|
||||
out = _chat(
|
||||
[{"role": "system", "content": sys}, {"role": "user", "content": usr}],
|
||||
model=self._model,
|
||||
temperature=0.4,
|
||||
max_tokens=2000,
|
||||
)
|
||||
self._tokens += len(usr) // 3 + 1500
|
||||
|
||||
# parse
|
||||
try:
|
||||
# strip markdown fences if any
|
||||
cleaned = out.strip()
|
||||
if cleaned.startswith("```"):
|
||||
cleaned = re.sub(r"^```[a-z]*\n?", "", cleaned)
|
||||
cleaned = re.sub(r"\n?```$", "", cleaned)
|
||||
data = json.loads(cleaned)
|
||||
edits: List[EditRecord] = []
|
||||
for e in data.get("edits", [])[:edit_budget]:
|
||||
if e.get("op") not in ("ADD", "DELETE", "REPLACE"):
|
||||
continue
|
||||
target = e.get("target", "skill")
|
||||
if target not in ("skill", "memory"):
|
||||
continue
|
||||
if not evolve_skill and target == "skill":
|
||||
continue
|
||||
if not evolve_memory and target == "memory":
|
||||
continue
|
||||
edits.append(EditRecord(
|
||||
op=e["op"],
|
||||
target=target,
|
||||
content=e.get("content", ""),
|
||||
old_text=e.get("old_text", ""),
|
||||
rationale=e.get("rationale", ""),
|
||||
))
|
||||
return edits
|
||||
except Exception as e:
|
||||
# log + return empty list (no edit is better than a bad edit)
|
||||
return []
|
||||
Executable
+330
@@ -0,0 +1,330 @@
|
||||
#!/usr/bin/env python3
|
||||
"""slash_sleep.py — OpenClaw slash command equivalent of SkillOpt's /sleep.
|
||||
|
||||
Use from the main session as a /sleep command:
|
||||
/sleep status — show current state + last 5 nights
|
||||
/sleep run — trigger one cycle (all categories) right now
|
||||
/sleep run research-cron — one cycle, single category
|
||||
/sleep adopt [night] — adopt the most recent (or specified) staged proposal
|
||||
/sleep reject [night] — discard the most recent (or specified) staging dir
|
||||
/sleep dry-run — report-only cycle
|
||||
/sleep cost — estimate per-night cost for current config
|
||||
|
||||
This script is a thin shell over run_sleep.py. It can be invoked either
|
||||
manually from the main session or by an OpenClaw command handler.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
SKILL_DIR = Path("/home/ethanclaw/.openclaw/workspace/skills/skillopt-sleep")
|
||||
STATE_DIR = Path(os.path.expanduser("~/.skillopt-sleep")) # default
|
||||
STAGING_ROOT = STATE_DIR
|
||||
|
||||
def _resolve_state_dir():
|
||||
"""Find the actual state dir.
|
||||
|
||||
Priority: scan in order:
|
||||
1. ~/.skillopt-sleep/ (default)
|
||||
2. /home/ethanclaw/.openclaw/workspace/.skillopt-sleep/ (when staging is there)
|
||||
3. /home/ethanclaw/.openclaw/.skillopt-sleep/ (parent of overridden claude_home)
|
||||
Pick the first one that has a state.json OR staging dir.
|
||||
"""
|
||||
candidates = [
|
||||
Path(os.path.expanduser("~/.skillopt-sleep")),
|
||||
Path("/home/ethanclaw/.openclaw/workspace/.skillopt-sleep"),
|
||||
Path("/home/ethanclaw/.openclaw/.skillopt-sleep"),
|
||||
]
|
||||
# Prefer the one with state.json
|
||||
for c in candidates:
|
||||
if (c / "state.json").exists():
|
||||
return c
|
||||
# Then the one with staging
|
||||
for c in candidates:
|
||||
if (c / "staging").exists():
|
||||
return c
|
||||
return candidates[0]
|
||||
|
||||
TESTS_DIR = SKILL_DIR / "tests"
|
||||
|
||||
|
||||
def status() -> int:
|
||||
state_dir = _resolve_state_dir()
|
||||
state_file = state_dir / "state.json"
|
||||
staging_dir = state_dir / "staging"
|
||||
print(f"=== SkillOpt-Sleep status ===")
|
||||
print(f" state dir: {state_dir}")
|
||||
print(f" staging dir: {staging_dir}")
|
||||
if staging_dir.exists():
|
||||
stages = sorted(staging_dir.iterdir(), key=lambda p: p.stat().st_mtime, reverse=True)
|
||||
print(f" staging entries: {len(stages)}")
|
||||
for s in stages[:3]:
|
||||
print(f" {s.name}")
|
||||
if not state_file.exists():
|
||||
print(" no state.json — run a cycle first (state is written at end of each non-dry-run)")
|
||||
return 0
|
||||
|
||||
with open(state_file) as f:
|
||||
state = json.load(f)
|
||||
|
||||
nights = state.get("history") or state.get("nights", [])
|
||||
print(f" total nights: {len(nights)}")
|
||||
print(f" accepted: {sum(1 for n in nights if n.get('accepted'))}")
|
||||
print(f" rejected: {sum(1 for n in nights if not n.get('accepted'))}")
|
||||
if nights:
|
||||
last = nights[-1]
|
||||
print(f" last night: {last.get('night')}")
|
||||
print(f" accepted: {last.get('accepted')}")
|
||||
print(f" baseline: {last.get('baseline'):.3f} -> candidate: {last.get('candidate'):.3f}")
|
||||
print(f" staging: {last.get('staging') or '(none)'}")
|
||||
return 0
|
||||
|
||||
|
||||
def run_category(category: str, *, dry_run: bool = False) -> int:
|
||||
cat_to_file = {
|
||||
"research-cron": "research-cron-tasks.json",
|
||||
"devops": "devops-tasks.json",
|
||||
"wiki": "wiki-tasks.json",
|
||||
}
|
||||
tasks_file = TESTS_DIR / cat_to_file.get(category, f"{category}-tasks.json")
|
||||
if not tasks_file.exists():
|
||||
print(f"ERROR: no tasks file for category '{category}': {tasks_file}")
|
||||
return 1
|
||||
|
||||
cmd = [sys.executable, str(SKILL_DIR / "run_sleep.py")]
|
||||
if dry_run:
|
||||
cmd.append("--dry-run")
|
||||
cmd.extend(["--tasks", str(tasks_file)])
|
||||
|
||||
print(f"=== /sleep run {category}{' (dry-run)' if dry_run else ''} ===")
|
||||
print(f" cmd: {' '.join(cmd)}")
|
||||
rc = os.system(" ".join(f'"{c}"' for c in cmd))
|
||||
return rc
|
||||
|
||||
|
||||
def run_all(*, dry_run: bool = False) -> int:
|
||||
rc = 0
|
||||
for cat in ("research-cron", "devops", "wiki"):
|
||||
r = run_category(cat, dry_run=dry_run)
|
||||
if r != 0:
|
||||
rc = r
|
||||
return rc
|
||||
|
||||
|
||||
def adopt(night: str = None) -> int:
|
||||
state_dir = _resolve_state_dir()
|
||||
state_file = state_dir / "state.json"
|
||||
if not state_file.exists():
|
||||
print("ERROR: no state to adopt from")
|
||||
return 1
|
||||
with open(state_file) as f:
|
||||
state = json.load(f)
|
||||
nights = state.get("history") or state.get("nights", [])
|
||||
if not nights:
|
||||
print("ERROR: no nights recorded")
|
||||
return 1
|
||||
|
||||
target = None
|
||||
if night:
|
||||
target = next((n for n in nights if str(n.get("night")) == night), None)
|
||||
if not target:
|
||||
print(f"ERROR: night '{night}' not found")
|
||||
return 1
|
||||
else:
|
||||
# most recent accepted
|
||||
candidates = [n for n in nights if n.get("accepted") and n.get("staging")]
|
||||
if not candidates:
|
||||
print("ERROR: no accepted nights with staging to adopt")
|
||||
return 1
|
||||
target = candidates[-1]
|
||||
|
||||
staging = target["staging"]
|
||||
if not os.path.isdir(staging):
|
||||
print(f"ERROR: staging dir missing: {staging}")
|
||||
return 1
|
||||
|
||||
print(f"=== /sleep adopt night {target['night']} ===")
|
||||
print(f" staging: {staging}")
|
||||
print(f" baseline: {target.get('baseline'):.3f} candidate: {target.get('candidate'):.3f}")
|
||||
|
||||
# Read proposed skill from staging
|
||||
manifest = Path(staging) / "manifest.json"
|
||||
if manifest.exists():
|
||||
with open(manifest) as f:
|
||||
m = json.load(f)
|
||||
proposed = m.get("proposed_skill")
|
||||
if proposed and Path(proposed).exists():
|
||||
live = STATE_DIR / "live_skill.md"
|
||||
backup = STATE_DIR / f"live_skill.md.bak-{target['night']}"
|
||||
if live.exists():
|
||||
shutil.copy2(live, backup)
|
||||
print(f" backed up current live skill → {backup}")
|
||||
shutil.copy2(proposed, live)
|
||||
print(f" adopted proposed skill → {live}")
|
||||
print()
|
||||
print("✅ Adoption complete. Next cycle will use the new skill.")
|
||||
return 0
|
||||
|
||||
print("ERROR: no proposed_skill in manifest")
|
||||
return 1
|
||||
|
||||
|
||||
def reject(night: str = None) -> int:
|
||||
state_dir = _resolve_state_dir()
|
||||
state_file = state_dir / "state.json"
|
||||
if not state_file.exists():
|
||||
print("ERROR: no state")
|
||||
return 1
|
||||
with open(state_file) as f:
|
||||
state = json.load(f)
|
||||
nights = state.get("history") or state.get("nights", [])
|
||||
target = None
|
||||
if night:
|
||||
target = next((n for n in nights if str(n.get("night")) == night), None)
|
||||
else:
|
||||
candidates = [n for n in reversed(nights) if n.get("staging")]
|
||||
target = candidates[0] if candidates else None
|
||||
|
||||
if not target or not target.get("staging"):
|
||||
print("ERROR: nothing to reject")
|
||||
return 1
|
||||
|
||||
staging = target["staging"]
|
||||
if os.path.isdir(staging):
|
||||
shutil.rmtree(staging)
|
||||
print(f"🗑️ Removed staging: {staging}")
|
||||
# remove from state
|
||||
state["history"] = [n for n in nights if n.get("night") != target["night"]]
|
||||
with open(state_file, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
print("✅ Rejected. State updated.")
|
||||
return 0
|
||||
|
||||
|
||||
def schedule_cmd(hour: int, minute: int) -> int:
|
||||
"""Install a nightly cron entry via the shared SkillOpt-Sleep scheduler.
|
||||
|
||||
Note: this schedules the shared engine (``python -m skillopt_sleep run``),
|
||||
not the OpenClaw-specific ``run_sleep.py``. Use ``run_sleep_cron.sh`` if
|
||||
you need the OpenClaw-native backend and category task files instead.
|
||||
"""
|
||||
try:
|
||||
from skillopt_sleep.scheduler import schedule
|
||||
except ImportError:
|
||||
print("ERROR: skillopt_sleep.scheduler not available — is SkillOpt-Sleep installed?")
|
||||
return 1
|
||||
project = str(SKILL_DIR)
|
||||
ok, msg = schedule(project, hour=hour, minute=minute)
|
||||
print(msg)
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def unschedule_cmd(all_projects: bool) -> int:
|
||||
"""Remove cron entry via the shared SkillOpt-Sleep scheduler."""
|
||||
try:
|
||||
from skillopt_sleep.scheduler import unschedule
|
||||
except ImportError:
|
||||
print("ERROR: skillopt_sleep.scheduler not available — is SkillOpt-Sleep installed?")
|
||||
return 1
|
||||
project = str(SKILL_DIR)
|
||||
ok, msg = unschedule(project, all_projects=all_projects)
|
||||
print(msg)
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def cost() -> int:
|
||||
"""Estimate per-night cost based on the actual measurement from Phase 2.
|
||||
|
||||
From the real dry-run: 5 devops tasks used 14,427 tokens total.
|
||||
That is ~2,885 tokens per task (all 3 phases combined).
|
||||
"""
|
||||
cfg_path = SKILL_DIR / "config.json"
|
||||
cfg = {}
|
||||
if cfg_path.exists():
|
||||
cfg = json.loads(cfg_path.read_text())
|
||||
cfg.pop("_comment", None)
|
||||
|
||||
max_tasks = cfg.get("max_tasks_per_night", 12)
|
||||
model = cfg.get("model", "deepseek-v4-pro")
|
||||
# DeepSeek V4 pricing
|
||||
if "pro" in model:
|
||||
cost_in = 0.435 # per 1M
|
||||
cost_out = 0.87
|
||||
elif "flash" in model:
|
||||
cost_in = 0.14
|
||||
cost_out = 0.28
|
||||
else:
|
||||
cost_in, cost_out = 0.5, 1.0
|
||||
|
||||
# Measured: ~2,900 tokens per task, 30% output / 70% input
|
||||
toks_per_task = 2900
|
||||
input_toks = int(toks_per_task * 0.7)
|
||||
output_toks = int(toks_per_task * 0.3)
|
||||
|
||||
cost_in_total = (input_toks * max_tasks / 1_000_000) * cost_in
|
||||
cost_out_total = (output_toks * max_tasks / 1_000_000) * cost_out
|
||||
cost = cost_in_total + cost_out_total
|
||||
|
||||
print(f"=== Cost estimate (per actual measurement) ===")
|
||||
print(f" model: {model}")
|
||||
print(f" max tasks/night: {max_tasks}")
|
||||
print(f" ~tokens/night: {toks_per_task * max_tasks:,}")
|
||||
print(f" cost/night: ${cost:.3f}")
|
||||
print(f" cost/month (30 nights): ${cost*30:.2f}")
|
||||
print(f" cost/year (365 nights): ${cost*365:.2f}")
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="OpenClaw /sleep command")
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
sub.add_parser("status", help="show state + last 5 nights")
|
||||
p_run = sub.add_parser("run", help="trigger one cycle")
|
||||
p_run.add_argument("category", nargs="?", default=None,
|
||||
choices=["research-cron", "devops", "wiki", None])
|
||||
p_run.add_argument("--dry-run", action="store_true")
|
||||
sub.add_parser("dry-run", help="report-only cycle (all categories)")
|
||||
p_adopt = sub.add_parser("adopt", help="adopt most recent accepted staging")
|
||||
p_adopt.add_argument("night", nargs="?", default=None)
|
||||
p_reject = sub.add_parser("reject", help="discard most recent staging")
|
||||
p_reject.add_argument("night", nargs="?", default=None)
|
||||
sub.add_parser("cost", help="estimate cost")
|
||||
p_schedule = sub.add_parser("schedule", help="install nightly cron entry")
|
||||
p_schedule.add_argument("--hour", type=int, default=3, help="hour (0-23)")
|
||||
p_schedule.add_argument("--minute", type=int, default=17, help="minute (0-59)")
|
||||
p_unschedule = sub.add_parser("unschedule", help="remove cron entry")
|
||||
p_unschedule.add_argument("--all", dest="all_projects", action="store_true",
|
||||
help="remove entries for all projects")
|
||||
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.cmd == "status":
|
||||
return status()
|
||||
if args.cmd == "run":
|
||||
if args.category:
|
||||
return run_category(args.category, dry_run=args.dry_run)
|
||||
return run_all(dry_run=args.dry_run)
|
||||
if args.cmd == "dry-run":
|
||||
return run_all(dry_run=True)
|
||||
if args.cmd == "adopt":
|
||||
return adopt(args.night)
|
||||
if args.cmd == "reject":
|
||||
return reject(args.night)
|
||||
if args.cmd == "cost":
|
||||
return cost()
|
||||
if args.cmd == "schedule":
|
||||
return schedule_cmd(args.hour, args.minute)
|
||||
if args.cmd == "unschedule":
|
||||
return unschedule_cmd(args.all_projects)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,87 @@
|
||||
[
|
||||
{
|
||||
"id": "do-01",
|
||||
"reference": "[STATUS] devops-agent | Site Uptime \u2192 geoxylia.com OK (200) | 14/06 22:30 MYT",
|
||||
"rubric": "Score 1.0 if output matches the exact format [STATUS] devops-agent | Site Uptime \u2192 geoxylia.com OK (200) | DD/MM HH:MM MYT, with a real current time. Score 0.5 if format is close but missing one field. Score 0.0 if wrong format or hallucinated values.",
|
||||
"project": "devops-infrastructure-check",
|
||||
"intent": "Site Uptime check. Run: `curl -o /dev/null -s -w '%{http_code}' https://geoxylia.com`. Interpret the result 200, and report in our standard format: 'STATUS | TASK \u2192 RESULT | TIME'. If not 200, escalate.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"devops-infrastructure-check"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "val"
|
||||
},
|
||||
{
|
||||
"id": "do-02",
|
||||
"reference": "Backup complete. Files: 87, Size: 1.2G, Last: 2026-06-14 22:00:00 MYT",
|
||||
"rubric": "Score 1.0 if output includes the exact 'Backup complete. Files: N, Size: X, Last: timestamp' structure with plausible values. Score 0.5 if structure is close but one field missing. Score 0.0 if hallucinated or wrong structure.",
|
||||
"project": "devops-infrastructure-check",
|
||||
"intent": "Daily Memory Backup. Confirm this ran successfully by checking: `ls -t ~/backups/memory/memory-backup-*.tar.gz | head -3`. Report the file count, total size, and most recent backup time. Use format: 'Backup complete. Files: [N], Size: [X], Last: [timestamp]'.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"devops-infrastructure-check"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "val"
|
||||
},
|
||||
{
|
||||
"id": "do-03",
|
||||
"reference": "1) Vercel CSP missing frame-ancestors: MEDIUM. Allows clickjacking if anyone embeds our pages; not exploitable for our content, but best-practice gap.\n2) OpenClaw plaintext API keys: LOW. The config is chmod 600, loopback-only, not in git. Standard OpenClaw behavior. Rotating would add zero real security given current exposure.",
|
||||
"rubric": "Score 1.0 if both are classified correctly (MEDIUM and LOW respectively) and justifications are accurate (not panicky, not dismissive). Score 0.5 if classifications are wrong by one tier or justifications are weak. Score 0.0 if both over-classified as CRITICAL or both wrong.",
|
||||
"project": "devops-infrastructure-check",
|
||||
"intent": "Security Check daily run. Two findings: 1) Vercel CSP header missing 'frame-ancestors' directive, 2) OpenClaw config has 3 plaintext API keys. Classify each as: CRITICAL / HIGH / MEDIUM / LOW / INFO. Justify each in 1 sentence.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"devops-infrastructure-check"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "train"
|
||||
},
|
||||
{
|
||||
"id": "do-04",
|
||||
"reference": "[INCIDENT] supabase.audit_results: anon role has no RLS policy \u2014 anyone with the URL can read all audit results. Fix: add policy 'audit_results_select_own' granting SELECT WHERE user_id = auth.uid(). Severity: HIGH (data exposure). Estimated 2-min fix.",
|
||||
"rubric": "Score 1.0 if: (a) severity correctly identified as HIGH, (b) fix is a real RLS policy (not just 'enable RLS' since it's already enabled), (c) under 50 words, (d) Telegram-friendly format. Score 0.5 if severity right but fix is generic. Score 0.0 if missing severity or wrong fix.",
|
||||
"project": "devops-infrastructure-check",
|
||||
"intent": "Incident Check. The Supabase RLS check returned: 'table public.audit_results: rls enabled but policy missing for anon role'. Interpret severity, propose fix, and format as a Telegram alert (max 50 words).",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"devops-infrastructure-check"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "val"
|
||||
},
|
||||
{
|
||||
"id": "do-05",
|
||||
"reference": "\ud83d\udee1\ufe0f Week security digest:\n\n\u2022 0 critical incidents, 1 high resolved (Supabase RLS policy added)\n\u2022 22 plaintext secrets: expected OpenClaw behavior, no action\n\u2022 1 medium open: Vercel CSP frame-ancestors, schedule for next sprint\n\nTrend: stable. No regressions vs last week.",
|
||||
"rubric": "Score 1.0 if all 3 priority tiers mentioned with correct counts, ends with a trend statement, Telegram-friendly. Score 0.5 if structure is right but one tier wrong. Score 0.0 if missing a tier or wrong format.",
|
||||
"project": "devops-infrastructure-check",
|
||||
"intent": "Weekly security digest. Synthesize this week's findings: 22 plaintext secrets in openclaw.json (expected), 0 critical incidents, 1 high (Supabase RLS), 1 medium (CSP frame-ancestors), 0 low. Output a 3-bullet Telegram status.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"devops-infrastructure-check"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "train"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,87 @@
|
||||
[
|
||||
{
|
||||
"id": "rc-01",
|
||||
"reference": "COMPETITOR MOVES: Otterly adds Perplexity tracker, joining Profound and LLMRefs in multi-platform citations.\nBACKLINK OPPORTUNITIES: 3 SEO directories (G2, Capterra, GetApp) have not been claimed.\nAGENCY BLUEPRINT: Top 2 agency sites bundle GEO audit + content refresh as $3K/mo tier.\nACTION ITEMS: Build Perplexity citation test into GeoXylia audit; claim G2 listing by Friday.",
|
||||
"rubric": "Score 1.0 if all 4 section headings present in correct order, each with a substantive (not generic) 1-sentence content. Score 0.5 if headings present but content is generic. Score 0.0 if any heading missing or order wrong.",
|
||||
"project": "research-cron-output",
|
||||
"intent": "Weekly Competitive Deep Dive for GeoXylia. The competitor otterly.ai just added a Perplexity citation tracker. Produce the report header (top section) in our standard format: COMPETITOR MOVES, BACKLINK OPPORTUNITIES, AGENCY BLUEPRINT, ACTION ITEMS. Keep it to 4 lines, one per section heading with a 1-sentence placeholder.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"research-cron-output"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "train"
|
||||
},
|
||||
{
|
||||
"id": "rc-02",
|
||||
"reference": "1. 'ai seo audit tool': 420 imp, pos 8.2, on page 1 \u2014 needs CTR lift (snippet/schema).\n2. 'geo audit tool': 230 imp, pos 12.5, page 2 \u2014 target blog post could push to page 1.\n3. 'llm optimization': 85 imp, pos 18.3, deep page-2 \u2014 fresh content with answer capsule could compete.",
|
||||
"rubric": "Score 1.0 if the response correctly identifies 'ai seo audit tool', 'geo audit tool', and 'llm optimization' as the top 3 (NOT 'best free seo audit' which is already converting well, NOT 'free audit tool' which has too few impressions). Each must have correct impression count, position, and a substantive rationale. Score 0.5 if correct 3 keywords but rationale is weak. Score 0.0 if wrong keywords selected.",
|
||||
"project": "research-cron-output",
|
||||
"intent": "GSC keyword opportunity scan. From this snippet of GSC data, identify the top 3 keyword opportunities (high impressions, low CTR, position 5-15):\n\n1. 'ai seo audit tool' \u2014 420 imp, 12 clicks, pos 8.2\n2. 'best free seo audit' \u2014 1100 imp, 95 clicks, pos 4.1\n3. 'geo audit tool' \u2014 230 imp, 4 clicks, pos 12.5\n4. 'llm optimization' \u2014 85 imp, 1 click, pos 18.3\n5. 'free audit tool' \u2014 50 imp, 0 clicks, pos 22.0\n\nOutput: one line per opportunity, format 'KEYWORD: impressions, position, why-it-matters (1 short clause)'.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"research-cron-output"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "train"
|
||||
},
|
||||
{
|
||||
"id": "rc-03",
|
||||
"reference": "Google AI Overviews now show source links more prominently + author bylines. For GeoXylia: this favors pages with clear authorship (add author schema to blog posts). Action: this week, add author + E-E-A-T schema markup to top 10 blog posts. Source: Google Search Central blog.",
|
||||
"rubric": "Score 1.0 if: (a) under 60 words, (b) names the change, (c) gives GeoXylia-specific implication, (d) gives a concrete action item, (e) cites the source. Score 0.5 if missing 1-2 of these. Score 0.0 if over 60 words or missing 3+.",
|
||||
"project": "research-cron-output",
|
||||
"intent": "Daily Industry News scan. The Google Search Central blog just announced: 'AI Overviews now showing source links more prominently, with author bylines for E-E-A-T-heavy content.' Write a 1-paragraph Telegram alert (max 60 words) for Ethan. Include: 1) what changed, 2) what it means for GeoXylia, 3) any action item.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"research-cron-output"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "val"
|
||||
},
|
||||
{
|
||||
"id": "rc-04",
|
||||
"reference": "Hi [Name], I saw seo-skill.com's resources page is one of the most-respected SEO learning hubs in the industry \u2014 your 2026 algorithm breakdown was spot-on. We just published a free 2026 AI SEO Audit comparison that your readers would find genuinely useful (no paywall, no signup). It covers the 8 leading AI-audit tools with hands-on screenshots and a clear feature matrix. GeoXylia is the only fully-free option in the comparison, so it's a natural fit for a 'tools to know' section. Mind if I share the link for inclusion?",
|
||||
"rubric": "Score 1.0 if exactly 4 sentences, all four functional pieces present (compliment / mention resource / audience benefit / GeoXylia one-liner), conversational tone, no aggressive sales language. Score 0.5 if 3 of 4 pieces present or tone is too salesy. Score 0.0 if more than 5 sentences or missing 2+ pieces.",
|
||||
"project": "research-cron-output",
|
||||
"intent": "Backlink Outreach draft for the blog post 'Free AI SEO Audit Tool: 2026 Comparison'. The prospect is seo-skill.com (a popular SEO training site with a 'resources' page). Write a 4-sentence outreach email: 1) compliment, 2) mention our resource, 3) explain audience benefit, 4) one-line about GeoXylia.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"research-cron-output"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "train"
|
||||
},
|
||||
{
|
||||
"id": "rc-05",
|
||||
"reference": "1) DO MORE: AI citation / LLM-mention topics \u2014 the 0.9% CTR at position 9.4 means we're visible but need richer answer capsules to lift CTR. Target 2x posts/week on this cluster.\n2) PAUSE: Pure schema-markup how-tos \u2014 'Schema Markup for SEO' has 0 clicks at position 41, the audience isn't searching this way. Rework as 'How to appear in AI answers' framing.\n3) TEST: 'Perplexity vs ChatGPT citation rates for [niche]' \u2014 unexplored angle, could capture comparison-intent traffic.",
|
||||
"rubric": "Score 1.0 if all 3 are specific (not generic), cite actual data from the prompt, and contain a clear actionable change. Score 0.5 if 2 of 3 are specific. Score 0.0 if generic advice or no data citations.",
|
||||
"project": "research-cron-output",
|
||||
"intent": "Performance \u2192 Strategy feedback loop. Last week's top blog post was 'AI Citation Audit: Does Your Site Appear in ChatGPT?' with 4,200 impressions and 38 clicks (CTR 0.9%, position 9.4). The bottom post was 'Schema Markup for SEO: A 2026 Guide' with 110 impressions and 0 clicks (CTR 0%, position 41). Write 3 specific strategy adjustments: 1) what to do more of, 2) what to pause, 3) what new topic to test.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"research-cron-output"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "val"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,70 @@
|
||||
[
|
||||
{
|
||||
"id": "wk-01",
|
||||
"reference": "1. What GEO is and isn't (define vs SEO/AEO, dispel the 'just add FAQ' myth)\n2. The 3 citation mechanisms LLMs use (RAG, fine-tuning, in-context; weight each)\n3. The 2026 citation data (real statistics from Profound/Otterly/Peec; what % of queries get citations)\n4. The action framework (a 5-step audit-and-fix process, concrete)\n5. Measurement (which metrics actually predict citation lift; vanity vs real)",
|
||||
"rubric": "Score 1.0 if 5 sections, in a logical order, each with a substantive (not generic) purpose, and the section content is GEO-specific (not generic SEO). Score 0.5 if 5 sections but 1-2 are generic. Score 0.0 if wrong number of sections or wrong order.",
|
||||
"project": "wiki-canonical-guide",
|
||||
"intent": "Wiki canonical guide: 'GEO 2026 Standards'. Audience: a mid-level SEO specialist who has heard of GEO but not done it. Tone: technical, evidence-driven, no fluff. Length target: 1500-2200 words. Outline the 5 sections that should appear in order. For each, give a 1-sentence sub-purpose.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"wiki-canonical-guide"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "val"
|
||||
},
|
||||
{
|
||||
"id": "wk-02",
|
||||
"reference": "Yes, add inbound links. (1) geo-2026-standards.md \u2192 '## Action Framework' section, anchor: 'platform-specific citation rules' \u2014 natural since GEO standards reference ChatGPT/Perplexity behavior. (2) seo-2026-standards.md \u2192 '## AI Overviews' section, anchor: 'AI platform citations' \u2014 links to the mechanism guide. (3) content-strategy.md \u2192 '## Content Types' section, anchor: 'per-platform citation' \u2014 content strategy needs to know which platform favors which content.",
|
||||
"rubric": "Score 1.0 if all 3 inbound links proposed with specific section + natural anchor text, demonstrating the link solves a real navigational gap (not just SEO-link-building). Score 0.5 if 2 of 3 are well-placed. Score 0.0 if generic anchors like 'click here' or no specific sections named.",
|
||||
"project": "wiki-canonical-guide",
|
||||
"intent": "Cross-link audit. The wiki page 'ai-platform-citation-guide.md' has 4 outbound links to other wiki pages, but no inbound links from: 'geo-2026-standards.md', 'seo-2026-standards.md', 'content-strategy.md'. Should we add inbound links? In which page should each inbound link go, and what anchor text would be natural?",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"wiki-canonical-guide"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "val"
|
||||
},
|
||||
{
|
||||
"id": "wk-03",
|
||||
"reference": "Priorities:\n1. Refresh 'geo-glossary.md' (last update 2026-04-12, 63 days) \u2014 add new terms like RAG, in-context citation, agentic SEO.\n2. Refresh 'competitor-pricing.md' (last update 2026-05-01, 44 days) \u2014 Profound raised enterprise tier.\n3. No structural fixes needed.\n\nTelegram: 'Wiki lint: 2 stale pages flagged (geo-glossary 63d, competitor-pricing 44d). No broken links. Both need refresh this week.'",
|
||||
"rubric": "Score 1.0 if both stale pages correctly identified with specific (not generic) refresh notes, and Telegram summary is under 40 words with the right action. Score 0.5 if stale pages identified but refresh notes are vague. Score 0.0 if missing stale pages or Telegram over 40 words.",
|
||||
"project": "wiki-canonical-guide",
|
||||
"intent": "Wiki lint report. Today's scan: 14 wiki pages, 2 with 'Updated' dates > 30 days old ('geo-glossary.md' and 'competitor-pricing.md'), 0 broken internal links, 0 missing YAML frontmatter. Output: 1) prioritized action list, 2) Telegram summary (max 40 words).",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"wiki-canonical-guide"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "train"
|
||||
},
|
||||
{
|
||||
"id": "wk-04",
|
||||
"reference": "Index rebuilt: 14 wiki pages registered in _index.md (was 12 \u2014 added competitor-pricing-rev2 and citations-q2-2026).\nQuestion for Ethan: should 'competitor-pricing.md' and 'competitor-pricing-rev2.md' be merged? They're 78% similar in content.",
|
||||
"rubric": "Score 1.0 if both sentences are accurate (count matches, names are plausible) and the question identifies a real consolidation opportunity (not a fabricated one). Score 0.5 if structure is right but content vague. Score 0.0 if wrong format or no question.",
|
||||
"project": "wiki-canonical-guide",
|
||||
"intent": "Index rebuild check. Run `python3 ~/agent-shared/scripts/update-index.py` (assume it works). After the run, the new wiki/_index.md should list all 14 pages. Generate a 2-sentence confirmation message + 1 question for Ethan to verify.",
|
||||
"context_excerpt": "",
|
||||
"attempted_solution": "",
|
||||
"outcome": "unknown",
|
||||
"reference_kind": "rubric",
|
||||
"judge": {},
|
||||
"tags": [
|
||||
"wiki-canonical-guide"
|
||||
],
|
||||
"source_sessions": [],
|
||||
"split": "train"
|
||||
}
|
||||
]
|
||||
Executable
+51
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
# SkillOpt-Sleep shared runner — used by all platform plugins (Claude Code,
|
||||
# Codex, Copilot). Resolves the repo root (which contains the skillopt_sleep
|
||||
# package), picks a Python >= 3.10, and execs the engine CLI.
|
||||
#
|
||||
# Usage: run-sleep.sh <run|dry-run|status|adopt|harvest|...> [args...]
|
||||
set -euo pipefail
|
||||
|
||||
# This script lives at <repo>/plugins/run-sleep.sh, so the repo root (which
|
||||
# holds skillopt_sleep/) is one level up. CLAUDE_PLUGIN_ROOT (if set by Claude
|
||||
# Code) points at the plugin dir; the engine is then two levels above it.
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
if [ -d "$SCRIPT_DIR/../skillopt_sleep" ]; then
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
elif [ -n "${CLAUDE_PLUGIN_ROOT:-}" ] && [ -d "$CLAUDE_PLUGIN_ROOT/../../skillopt_sleep" ]; then
|
||||
REPO_ROOT="$(cd "$CLAUDE_PLUGIN_ROOT/../.." && pwd)"
|
||||
elif [ -n "${SKILLOPT_SLEEP_REPO:-}" ] && [ -d "$SKILLOPT_SLEEP_REPO/skillopt_sleep" ]; then
|
||||
REPO_ROOT="$SKILLOPT_SLEEP_REPO"
|
||||
else
|
||||
# last resort: search upward from CWD
|
||||
d="$PWD"
|
||||
while [ "$d" != "/" ]; do
|
||||
[ -d "$d/skillopt_sleep" ] && { REPO_ROOT="$d"; break; }
|
||||
d="$(dirname "$d")"
|
||||
done
|
||||
fi
|
||||
if [ -z "${REPO_ROOT:-}" ]; then
|
||||
echo "[sleep] ERROR: could not locate the skillopt_sleep package. Set SKILLOPT_SLEEP_REPO to the repo root." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PY=""
|
||||
# Allow explicit Python override (useful on macOS with old system Python).
|
||||
if [ -n "${SKILLOPT_SLEEP_PYTHON:-}" ]; then
|
||||
PY="$SKILLOPT_SLEEP_PYTHON"
|
||||
else
|
||||
for cand in python3.12 python3.11 python3.10 python3; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then
|
||||
ver="$("$cand" -c 'import sys; print("%d%d" % sys.version_info[:2])' 2>/dev/null || echo 0)"
|
||||
if [ "${ver:-0}" -ge 310 ]; then PY="$cand"; break; fi
|
||||
fi
|
||||
done
|
||||
fi
|
||||
if [ -z "$PY" ]; then
|
||||
echo "[sleep] ERROR: need Python >= 3.10 (found none)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$#" -eq 0 ]; then set -- status; fi
|
||||
cd "$REPO_ROOT"
|
||||
exec "$PY" -m skillopt_sleep "$@"
|
||||
Reference in New Issue
Block a user