wehub/microsoft--skillopt
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:24:16 +08:00
commit 4b92209caa
317 changed files with 67848 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guide/configuration.md
+139
View File
@@ -0,0 +1,139 @@
# Configuration Guide
SkillOpt uses YAML configuration files with a hierarchical override system.
## Config Structure
```
configs/
├── _base_/
│ └── default.yaml # Global defaults
├── searchqa/
│ └── default.yaml # SearchQA overrides
├── docvqa/
│ └── default.yaml # DocVQA overrides
└── alfworld/
└── default.yaml # ALFWorld overrides
```
Benchmark configs inherit from `_base_/default.yaml` and override specific values.
## Key Parameters
### Model
```yaml
model:
backend: azure_openai # azure_openai | openai_chat | claude_code_exec | qwen
optimizer: gpt-5.5 # Optimizer model (for reflection)
target: gpt-5.5 # Target model (for rollout)
```
### Training
```yaml
train:
num_epochs: 4 # Number of training epochs
batch_size: 40 # Tasks per step (batch size)
accumulation: 1 # Gradient accumulation
seed: 42
```
### Gradient (Reflection)
```yaml
gradient:
minibatch_size: 8 # Reflect minibatch size
analyst_workers: 16 # Parallel reflection workers
max_analyst_rounds: 3 # Max rounds of analyst reflection
failure_only: false # Only reflect on failures
```
### Optimizer
```yaml
optimizer:
learning_rate: 4 # Max edits per step (edit budget)
min_learning_rate: 2 # Min edits for decay schedulers
lr_scheduler: cosine # constant | linear | cosine | autonomous
use_slow_update: true # Momentum-like blending at epoch boundary
slow_update_samples: 20 # Samples for slow update evaluation
use_meta_skill: true # Cross-epoch strategy memory
```
### Skill-Aware Reflection (optional, off by default)
EmbodiSkill-style failure routing: the failure analyst classifies each
failure pattern as **SKILL_DEFECT** (the rule is wrong or missing → normal
gated body edit) or **EXECUTION_LAPSE** (a valid rule exists but was not
followed → a short reminder appended to a protected appendix region inside
the skill that step-level edits can never modify).
```yaml
optimizer:
use_skill_aware_reflection: false # Master switch (default off = baseline-identical)
skill_aware_appendix_source: both # both | failure_only (paper-faithful S_app)
skill_aware_consolidate_threshold: 0 # >0: LLM-compact the appendix past N notes (experimental)
```
Notes:
- The switch is resolved process-wide from the config
(`configure_skill_aware_reflection`), so it applies to every benchmark
with no per-adapter wiring.
- `failure_only` restricts appendix notes to the failure analyst, matching
the original S_app formulation; `both` additionally lets the success
analyst re-emphasize existing rules.
- Appendix notes bypass the validation gate by design and accumulate with
order-preserving dedup; lapse-only steps (no body edits) still flush
their notes.
- Not supported together with `skill_update_mode=rewrite_from_suggestions`
or the full-rewrite modes: whole-document rewrites can drop the appendix
region.
### Evaluation
```yaml
evaluation:
use_gate: true # Validation gating (accept/reject updates)
eval_test: true # Run test evaluation after training
```
### Environment (Data)
```yaml
env:
name: searchqa # Benchmark name
split_mode: ratio # ratio | split_dir
split_ratio: "2:1:7" # train:val:test ratio
data_path: "" # Path to dataset
exec_timeout: 120 # Per-task timeout (seconds)
```
## CLI Overrides
Override any config value from the command line:
```bash
python scripts/train.py \
--config configs/searchqa/default.yaml \
optimizer.learning_rate=16 \
optimizer.lr_scheduler=linear \
gradient.analyst_workers=8
```
## Environment Variables
Model credentials are loaded from environment variables:
| Variable | Backend | Description |
|---|---|---|
| `AZURE_OPENAI_ENDPOINT` | azure_openai | Azure resource endpoint |
| `AZURE_OPENAI_API_KEY` | azure_openai | Azure API key |
| `OPENAI_API_KEY` | openai | OpenAI API key |
| `ANTHROPIC_API_KEY` | claude | Anthropic API key |
| `QWEN_API_BASE` | qwen | Local Qwen vLLM endpoint |
## Full Reference
See [Configuration Reference](../reference/config.md) for the complete parameter list.
docs/guide/dl-analogy.md
+51
View File
@@ -0,0 +1,51 @@
# Deep Learning ↔ SkillOpt Analogy
SkillOpt is designed around a core insight: **optimizing natural-language prompts follows the same structure as training neural networks**. This page maps every DL concept to its SkillOpt counterpart.
## Complete Mapping
| Deep Learning | SkillOpt | Description |
|---|---|---|
| **Model weights** | Skill document (Markdown) | The thing being optimized |
| **Forward pass** | Rollout | Target executes tasks using current skill |
| **Loss function** | Task evaluator | Scores task execution quality |
| **Backpropagation** | Reflect | Optimizer analyzes failures → edit patches |
| **Gradients** | Edit patches | Proposed changes to the skill |
| **Gradient aggregation** | Patch aggregation | Merge similar edits |
| **Gradient clipping** | Edit selection | Cap max edits per step |
| **Learning rate** | `learning_rate` | Max number of edits applied per step |
| **LR scheduler** | `lr_scheduler` | Decay schedule: cosine, linear, constant |
| **SGD step** | Skill update | Apply selected patches to document |
| **Validation set** | Selection split | Gate checks improvement before accepting |
| **Early stopping** | Gate patience | Reject updates that don't improve |
| **Training step** | Step | One rollout → reflect → update cycle |
| **Epoch** | Epoch | Full pass with slow update + meta memory |
| **Momentum** | Slow update | Longitudinal comparison at epoch boundary |
| **Meta-learning** | Meta skill | Cross-epoch optimizer strategy memory |
| **Batch size** | `batch_size` | Tasks sampled per rollout |
| **Data parallelism** | `analyst_workers` | Parallel reflection workers |
| **Training set** | Train split | Items used for rollout |
| **Test set** | Test split | Held-out final evaluation |
| **Warm-up** | (implicit) | High LR early steps explore broadly |
| **Checkpointing** | Skill snapshots | Saved after each accepted step |
| **Transfer learning** | Seed skill / cross-benchmark init | Start from pre-trained skill |
## Why This Analogy Matters
1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt
2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL
3. **Proven mechanisms**: Gating ≈ validation-based selection, patience ≈ early stopping, slow update ≈ momentum — all with strong theoretical motivation
## Hyperparameter Transfer Rules
From our experiments, these DL intuitions transfer well:
!!! success "What transfers"
- **Cosine schedule > constant** — same as in DL, cosine annealing helps convergence
- **Moderate LR (4-16) > very high/low** — too few edits = slow learning, too many = noisy
- **Slow update helps** — longitudinal comparison prevents catastrophic forgetting across epochs
- **Meta skill memory improves reflection** — optimizer benefits from cross-epoch strategy notes
!!! warning "What doesn't transfer"
- **Batch size ≠ better** — larger rollout batches have diminishing returns due to API costs
- **More epochs ≠ better** — skills converge faster than neural networks (2-4 epochs usually enough)
docs/guide/first-experiment.md
+110
View File
@@ -0,0 +1,110 @@
# Your First Experiment
This guide walks through running a complete SkillOpt training on SearchQA.
## 1. Choose a Benchmark
SkillOpt includes ready-to-use configs for several benchmarks:
| Benchmark | Difficulty | Typical Runtime |
|---|---|---|
| SearchQA | ⭐ Easy | ~30 min |
| DocVQA | ⭐⭐ Medium | ~2 hours |
| ALFWorld | ⭐⭐⭐ Hard | ~3 hours |
We'll use **SearchQA** as it's the fastest to complete.
## 2. Configure
Review the config file:
```bash
cat configs/searchqa/default.yaml
```
Key parameters (deep learning analogy in parentheses):
```yaml
train:
num_epochs: 4 # (epochs)
batch_size: 40 # (batch size)
optimizer:
learning_rate: 4 # (max edits per step)
lr_scheduler: cosine # (learning rate schedule)
use_slow_update: true # (momentum at epoch boundary)
use_meta_skill: true # (cross-epoch optimizer memory)
gradient:
analyst_workers: 16 # (parallel reflection workers)
evaluation:
use_gate: true # (validation gating)
```
## 3. Train
```bash
python scripts/train.py --config configs/searchqa/default.yaml
```
You'll see output like:
```
[Step 1/8] Rollout: 20 items, 4 workers...
[Step 1/8] Score: 0.65 → Reflect...
[Step 1/8] 6 edit patches generated
[Step 1/8] Selected 4 edits (lr=8, cosine → 7.7)
[Step 1/8] Gate: val score 0.68 > 0.65 ✓ ACCEPT
[Step 2/8] ...
```
## 4. Monitor
Training outputs are saved to `outputs/<benchmark>/<run_id>/`:
```
outputs/searchqa/2024-01-15_10-30-00/
├── steps/
│ ├── step_0001/
│ │ ├── candidate_skill.md
│ │ ├── step_record.json
│ │ └── trajectory_digest.json
│ └── step_0002/
├── slow_update/
│ └── epoch_02/
├── meta_skill/
│ └── epoch_02/
├── skills/
│ └── step_0001.md
├── best_skill.md
├── history.json
└── config.yaml
```
## 5. Evaluate
Evaluate the best skill on the test split:
```bash
python scripts/eval_only.py \
--config configs/searchqa/default.yaml \
--skill outputs/searchqa/<run_id>/skills/best_skill.md
```
## WebUI
Prefer a graphical interface? Launch the WebUI:
```bash
pip install -e ".[webui]"
python -m skillopt_webui.app
```
Then open `http://localhost:7860` in your browser to configure parameters and launch training.
## Next Steps
- [Understand the training loop](training-loop.md)
- [Configuration reference](../reference/config.md)
- [Add a new benchmark](new-benchmark.md)
docs/guide/installation.md
+89
View File
@@ -0,0 +1,89 @@
# Installation
## Requirements
- Python ≥ 3.10
- At least one model API key (Azure OpenAI, OpenAI, Anthropic, or local Qwen)
## Quick Install
```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
pip install -e .
```
## Optional Dependencies
Install extras for specific benchmarks or backends:
=== "ALFWorld"
```bash
pip install -e ".[alfworld]"
```
=== "Claude Backend"
```bash
pip install -e ".[claude]"
```
=== "Qwen (Local)"
```bash
pip install -e ".[qwen]"
```
=== "WebUI"
```bash
pip install -e ".[webui]"
```
=== "Development"
```bash
pip install -e ".[dev]"
```
=== "All"
```bash
pip install -e ".[alfworld,claude,qwen,webui,dev]"
```
## Environment Variables
Copy the example `.env` file and fill in your credentials:
```bash
cp .env.example .env
```
Edit `.env` with your API keys:
```ini
# Azure OpenAI (default backend)
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_KEY=your-key
# Or use OpenAI directly
OPENAI_API_KEY=sk-...
# Or Anthropic Claude
ANTHROPIC_API_KEY=sk-ant-...
```
!!! tip
You only need credentials for the backend you plan to use. Azure OpenAI is the default.
## Verify Installation
```bash
python -c "import skillopt; print('SkillOpt ready!')"
```
## Next Steps
→ [Run your first experiment](first-experiment.md)
docs/guide/local-env-smoke.md
+143
View File
@@ -0,0 +1,143 @@
# Local Environment Smoke Tests
This guide describes a lightweight pattern for testing a custom SkillOpt environment before connecting it to expensive model calls or a full benchmark dataset.
The goal is to validate the training loop plumbing first:
- config loading
- adapter construction
- dataloader splits
- rollout output shape
- reflection patch shape
- merge/rank/update control flow
- artifact creation under `out_root`
Once those are stable, you can switch the same environment to real model calls and larger evaluation splits.
## 1. Add a tiny fixture split
Start with a handful of deterministic examples that cover the expected pass/fail cases for your environment. Keep them small enough that a single training step can run locally.
A minimal fixture item usually needs:
```json
{
"id": "example-1",
"split": "train",
"question": "...",
"expected": "..."
}
```
Use the split names your adapter maps to SkillOpt phases:
- `train` for optimization rollouts
- `val` or `valid_seen` for selection/gating
- `test` or `valid_unseen` for final evaluation
## 2. Support an offline mock mode
Add a configuration flag such as `mock: true` to your adapter. In mock mode, `rollout()` should return deterministic responses without calling external model APIs.
This lets you verify the SkillOpt loop with a fast command such as:
```bash
python scripts/train.py \
--config configs/myenv/tiny_mock.yaml
```
Mock mode should still write the same artifacts as a real run, for example:
- `responses.json`
- `rollout_results.json`
- `ranked_edits.json`
- `candidate_skill.md`
- `summary.json`
## 3. Keep the smoke config tiny
A CI-friendly smoke config should run a single small step:
```yaml
train:
num_epochs: 1
train_size: 3
batch_size: 3
gradient:
minibatch_size: 1
merge_batch_size: 2
analyst_workers: 1
max_analyst_rounds: 1
optimizer:
learning_rate: 1
min_learning_rate: 1
lr_scheduler: constant
skill_update_mode: patch
use_slow_update: false
evaluation:
use_gate: true
sel_env_num: 2
test_env_num: 2
eval_test: false
env:
name: myenv
out_root: outputs/myenv_tiny_mock
mock: true
```
Prefer a mock config that runs without credentials. That makes it useful for contributors and CI.
## 4. Validate optimizer JSON before returning it
If your environment or extension asks an LLM to merge or rank skill edits, validate the returned JSON before passing it back into SkillOpt. This avoids silent fallbacks from empty, malformed, or out-of-range responses.
Useful checks for edit payloads:
- response is a JSON object
- `edits` is a non-empty list
- every edit is an object
- every edit has an allowed operation
- required fields such as `content` or `target` are present for that operation
Useful checks for ranking payloads:
- `selected_indices` exists
- indices are integers
- indices are unique
- indices are within the candidate edit range
- selected count does not exceed the edit budget
On failure, retry with a compact prompt that includes the schema error. If retries fail, raise an explicit error instead of silently accepting malformed output.
## 5. Run progressively stronger checks
A good development sequence is:
```bash
python -m py_compile scripts/train.py skillopt/envs/myenv/adapter.py
python scripts/train.py --config configs/myenv/tiny_mock.yaml
python scripts/train.py --config configs/myenv/tiny.yaml
```
For the real tiny run, verify that:
- the run completes
- `summary.json` is written
- `ranked_edits.json` contains the expected ranking metadata
- any optimizer bridge log marks the response schema as valid
- no generated files are written outside `out_root`
## 6. Keep custom environments isolated
When adding a custom environment to the registry, avoid side effects for existing benchmarks:
- lazy-import optional dependencies
- install environment-specific hooks only when `cfg["env"]` matches your environment
- keep mock behavior behind an explicit config flag
- write generated artifacts only under `out_root`
This makes it easier to review and test a custom integration without affecting the built-in benchmarks.
docs/guide/new-backend.md
+130
View File
@@ -0,0 +1,130 @@
# Add a New Model Backend
SkillOpt supports multiple LLM backends. This guide shows how to add your own.
## Backend Architecture
```
skillopt/model/
├── base.py # Abstract base class
├── azure_openai.py # Azure OpenAI backend
├── openai_model.py # Direct OpenAI backend
├── claude.py # Anthropic Claude backend
├── qwen.py # Local Qwen (vLLM) backend
└── your_backend.py # Your new backend
```
## Step 1: Create the Backend
Create `skillopt/model/your_backend.py`:
```python
from skillopt.model.base import ModelBackend, ModelResponse
class YourBackend(ModelBackend):
"""Your custom model backend."""
def __init__(self, cfg: dict):
super().__init__(cfg)
self.model_name = cfg.get('model_name', 'your-default-model')
self.api_key = os.environ.get('YOUR_API_KEY', '')
self.client = self._init_client()
def _init_client(self):
"""Initialize API client."""
# TODO: Set up your API client
pass
async def generate(
self,
messages: list[dict],
temperature: float = 0.7,
max_tokens: int = 4096,
**kwargs
) -> ModelResponse:
"""
Generate a completion.
Args:
messages: Chat messages [{"role": "...", "content": "..."}]
temperature: Sampling temperature
max_tokens: Maximum tokens in response
Returns:
ModelResponse with content, usage, and metadata
"""
response = await self.client.chat(
model=self.model_name,
messages=messages,
temperature=temperature,
max_tokens=max_tokens,
)
return ModelResponse(
content=response.text,
usage={
'prompt_tokens': response.usage.input,
'completion_tokens': response.usage.output,
},
model=self.model_name,
)
async def generate_with_tools(
self,
messages: list[dict],
tools: list[dict],
**kwargs
) -> ModelResponse:
"""Generate with tool/function calling support."""
# Optional: implement if your model supports tool use
raise NotImplementedError("Tool use not supported")
```
## Step 2: Register the Backend
Add to `skillopt/model/__init__.py`:
```python
from .your_backend import YourBackend
BACKEND_REGISTRY = {
# ... existing backends ...
'your_backend': YourBackend,
}
```
## Step 3: Configure
Use your backend in any config:
```yaml
model:
backend: your_backend
model_name: your-model-id
temperature: 0.7
max_tokens: 4096
```
Set credentials via environment variable:
```bash
export YOUR_API_KEY="your-key"
```
## Required Interface
Your backend must implement these methods:
| Method | Required | Description |
|---|---|---|
| `generate()` | ✅ | Basic text generation |
| `generate_with_tools()` | Optional | Tool/function calling |
| `count_tokens()` | Optional | Token counting for context management |
## Tips
!!! tip
- Test your backend with `python -c "from skillopt.model.your_backend import YourBackend"` first
- Use `async` methods for all API calls — SkillOpt uses asyncio throughout
- Implement retry logic with exponential backoff for production use
- Add your API key to `.env.example` when submitting a PR
docs/guide/new-benchmark.md
+371
View File
@@ -0,0 +1,371 @@
# Add a New Benchmark
Extend SkillOpt with your own benchmark in ~200 lines of code. We will use
a tiny worked example, `docfaithful`, that scores a target model on
how faithfully it answers questions grounded in a small reference doc.
> **Working reference.** The easiest way to copy-cargo-cult a new env is
> to read [`skillopt/envs/officeqa/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt/envs/officeqa).
> Everything below is the same shape, simplified.
## What you need to build
To add a benchmark you implement four things:
1. **A `SplitDataLoader` subclass** — knows how to load train / val / test
item dicts from disk.
2. **A rollout helper** — runs the target model on a batch of items
under the current skill and scores each prediction.
3. **An `EnvAdapter` subclass** — wires the loader + rollout helper into
SkillOpt's lifecycle (`build_*_env`, `rollout`, `reflect`,
`get_task_types`).
4. **A YAML config** — references your env name plus the standard
train / optimizer / gradient knobs.
Then one line in `scripts/train.py`'s `_register_builtins()` makes it
discoverable.
---
## Step 1 — Create the package
```bash
mkdir -p skillopt/envs/docfaithful
touch skillopt/envs/docfaithful/__init__.py
```
## Step 2 — Implement the data loader
`skillopt/envs/docfaithful/dataloader.py`:
```python
from __future__ import annotations
import json
from pathlib import Path
from skillopt.datasets.base import SplitDataLoader
def _normalize(raw: dict) -> dict:
"""Make sure every item has an ``id``. Other keys are env-specific."""
return {
"id": str(raw["uid"]),
"question": raw["question"],
"ground_truth": raw["answer"],
"reference_text": raw.get("reference", ""),
"task_type": raw.get("category", "docfaithful"),
}
class DocFaithfulDataLoader(SplitDataLoader):
"""Load DocFaithful items from JSON files inside each split dir."""
def load_split_items(self, split_path: str) -> list[dict]:
# split_path is e.g. data/docfaithful_split/train/
json_files = sorted(Path(split_path).glob("*.json"))
if not json_files:
raise FileNotFoundError(f"No .json file found in {split_path}")
with json_files[0].open(encoding="utf-8") as f:
raw = json.load(f)
return [_normalize(item) for item in raw]
```
Only `load_split_items()` is mandatory. If you also want to support
`split_mode="ratio"` (auto-split a single raw file into train/val/test),
override `load_raw_items(data_path)` as well — see
`skillopt/datasets/base.py` docstrings.
## Step 3 — Write the rollout helper
`skillopt/envs/docfaithful/rollout.py`:
```python
from __future__ import annotations
import json
import os
from pathlib import Path
from skillopt.model import chat_target
def _score(prediction: str, ground_truth: str) -> tuple[int, float]:
"""Trivial exact-match scorer. Replace with F1 / ROUGE / LLM-judge."""
p = (prediction or "").strip().lower()
g = (ground_truth or "").strip().lower()
hard = int(p == g and bool(g))
soft = 1.0 if hard else 0.0
return hard, soft
def _rollout_one(item: dict, skill_content: str,
*, max_completion_tokens: int) -> dict:
system = skill_content
user = (
f"Question: {item['question']}\n\n"
f"Reference:\n{item.get('reference_text', '')}\n\n"
"Answer:"
)
prediction, _usage = chat_target(
system=system,
user=user,
max_completion_tokens=max_completion_tokens,
)
hard, soft = _score(prediction, item.get("ground_truth", ""))
return {
"id": str(item["id"]),
"hard": hard,
"soft": soft,
"predicted_answer": prediction,
"question": item.get("question", ""),
"reference_text": item.get("reference_text", ""),
"task_type": item.get("task_type", "docfaithful"),
}
def run_batch(*, items: list[dict], skill_content: str, out_root: str,
workers: int = 4, max_completion_tokens: int = 4096) -> list[dict]:
"""Run a batch of episodes sequentially or with a thread pool."""
os.makedirs(out_root, exist_ok=True)
# For brevity we go sequentially — swap in concurrent.futures.ThreadPoolExecutor
# when network / model latency dominates.
results = [
_rollout_one(item, skill_content,
max_completion_tokens=max_completion_tokens)
for item in items
]
Path(out_root, "rollouts.json").write_text(
json.dumps(results, ensure_ascii=False, indent=2)
)
return results
```
Two design points worth flagging:
- **Scoring lives here, not in `EnvAdapter`.** There is no `evaluate()`
method on the ABC. Whatever signal you put in `hard` (0/1, or a float
in [0, 1] for smoothed reward) and `soft` (float in [0, 1]) is what
the optimizer reads.
- **Use `skillopt.model.chat_target`**, not raw OpenAI/Claude calls.
That routes through whichever **chat** target backend the user
configured (`openai_chat` / `claude_chat` / `qwen_chat` /
`minimax_chat`) without your adapter caring. Exec-style backends
(`codex_exec`, `claude_code_exec`) need env-specific rollout code —
see `skillopt/envs/swebench/` for an example.
## Step 4 — Implement the environment adapter
`skillopt/envs/docfaithful/adapter.py`:
```python
from __future__ import annotations
from skillopt.datasets.base import BatchSpec
from skillopt.envs.base import EnvAdapter
from skillopt.envs.docfaithful.dataloader import DocFaithfulDataLoader
from skillopt.envs.docfaithful.rollout import run_batch
class DocFaithfulAdapter(EnvAdapter):
"""SkillOpt adapter for the DocFaithful benchmark."""
def __init__(
self,
split_dir: str = "",
data_path: str = "",
split_mode: str = "split_dir",
split_ratio: str = "2:1:7",
split_seed: int = 42,
split_output_dir: str = "",
workers: int = 4,
analyst_workers: int = 4,
failure_only: bool = False,
minibatch_size: int = 8,
edit_budget: int = 4,
seed: int = 42,
limit: int = 0,
max_completion_tokens: int = 4096,
) -> None:
self.workers = workers
self.analyst_workers = analyst_workers
self.failure_only = failure_only
self.minibatch_size = minibatch_size
self.edit_budget = edit_budget
self.max_completion_tokens = int(max_completion_tokens)
self.dataloader = DocFaithfulDataLoader(
split_dir=split_dir,
data_path=data_path,
split_mode=split_mode,
split_ratio=split_ratio,
split_seed=split_seed,
split_output_dir=split_output_dir,
seed=seed,
limit=limit,
)
# ── Lifecycle ───────────────────────────────────────────────────────
def setup(self, cfg: dict) -> None:
super().setup(cfg)
self.dataloader.setup(cfg)
def get_dataloader(self):
return self.dataloader
# ── Env construction ────────────────────────────────────────────────
def build_env_from_batch(self, batch: BatchSpec, **kwargs):
# For dataset-backed envs the "manager" is just the items list.
return list(batch.payload or [])
def build_train_env(self, batch_size: int, seed: int, **kwargs):
batch = self.dataloader.build_train_batch(
batch_size=batch_size, seed=seed, **kwargs
)
return self.build_env_from_batch(batch, **kwargs)
def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
batch = self.dataloader.build_eval_batch(
env_num=env_num, split=split, seed=seed, **kwargs
)
return self.build_env_from_batch(batch, **kwargs)
# ── The rollout method (reflect is inherited) ───────────────────────
def rollout(self, env_manager, skill_content: str,
out_dir: str, **kwargs) -> list[dict]:
items: list[dict] = env_manager
return run_batch(
items=items,
skill_content=skill_content,
out_root=out_dir,
workers=self.workers,
max_completion_tokens=self.max_completion_tokens,
)
# reflect() is inherited from EnvAdapter — it delegates to
# run_minibatch_reflect with your analyst_error_* / analyst_success_*
# prompts. Override it only if you need custom reflection logic.
def get_task_types(self) -> list[str]:
seen: list[str] = []
for item in (
self.dataloader.train_items
+ self.dataloader.val_items
+ self.dataloader.test_items
):
tt = str(item.get("task_type") or "docfaithful")
if tt not in seen:
seen.append(tt)
return seen or ["docfaithful"]
```
### What the rollout actually does
Look back at `run_batch` from Step 3 — it sends each `item["question"]`
to the target model with `skill_content` as the system prompt, scores
the answer against `item["ground_truth"]`, and returns a list of dicts:
```python
[
{"id": "ex_001", "hard": 1, "soft": 0.92,
"predicted_answer": "...", "question": "...",
"reference_text": item["reference_text"]},
{"id": "ex_002", "hard": 0, "soft": 0.13, "fail_reason": "...", ...},
...
]
```
The trainer only requires `id`, `hard`, `soft`. The rest is preserved on
`RolloutResult.extras` (see `skillopt/types.py`) and is what your
`reflect()` consumes via `run_minibatch_reflect`.
## Step 5 — Register the adapter
Edit [`scripts/train.py`](https://github.com/microsoft/SkillOpt/blob/main/scripts/train.py)
and add to `_register_builtins()`:
```python
try:
from skillopt.envs.docfaithful.adapter import DocFaithfulAdapter
_ENV_REGISTRY["docfaithful"] = DocFaithfulAdapter
except ImportError:
pass # docfaithful deps not installed — skip
```
There is **no `BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`** —
the registry lives in `scripts/train.py` and is populated lazily so that
optional deps don't break `--help`.
## Step 6 — Create the YAML config
`configs/docfaithful/default.yaml`:
```yaml
_base_: ../_base_/default.yaml # NOTE: string, not list
model:
reasoning_effort: medium
train:
batch_size: 16
accumulation: 1
num_epochs: 4
gradient:
minibatch_size: 8
merge_batch_size: 8
optimizer:
learning_rate: 4
env:
name: docfaithful
# Optional: a seed skill document. Create this file (or any markdown
# file) yourself before the first run, or omit the key to let SkillOpt
# start from an empty skill.
skill_init: skillopt/envs/docfaithful/skills/initial.md
split_mode: split_dir
split_dir: data/docfaithful_split
workers: 4
max_completion_tokens: 4096
limit: 0
```
> ⚠️ `_base_` is currently parsed as a **string path**, not a list. Write
> `_base_: ../_base_/default.yaml`, not `_base_: ['../_base_/default.yaml']`.
> See [`skillopt/config.py`](https://github.com/microsoft/SkillOpt/blob/main/skillopt/config.py)
> if you want to add list-form inheritance.
## Step 7 — Run
```bash
# If you set skill_init above, create the seed skill first:
# mkdir -p skillopt/envs/docfaithful/skills
# echo "# DocFaithful initial skill" > skillopt/envs/docfaithful/skills/initial.md
python scripts/train.py --config configs/docfaithful/default.yaml
```
If you get `ValueError: Unknown environment 'docfaithful'. Available: [...]`,
you forgot Step 5.
If you get `TypeError: Can't instantiate abstract class DocFaithfulAdapter`,
you forgot to implement one of the four abstract methods on `EnvAdapter`:
`build_train_env`, `build_eval_env`, `rollout`, `get_task_types`.
## Tips
- Start with `train.batch_size: 4` and `limit: 10` while debugging.
- The `evaluate` half lives **inside your `rollout`**, not as a separate
method — there is no `evaluate()` in the `EnvAdapter` ABC. Score the
prediction in `run_batch` and put the score on each result dict's
`hard` / `soft`.
- Noisy scoring kills the optimizer. Spend time on `run_batch`'s scoring
before you spend time on prompts.
- If your benchmark needs heavy optional deps (selenium, vllm, ...),
wrap the registration block with `try / except ImportError` (Step 5)
so people without those deps can still `--help`.
- Copy `skillopt/envs/_template/` as a starting skeleton — it now
implements the real abstract methods.
docs/guide/skill-document.md
+78
View File
@@ -0,0 +1,78 @@
# Skill Document
A **skill document** is a Markdown file that serves as the "prompt weights" of your agent. SkillOpt trains this document through iterative optimization.
## What is a Skill Document?
A skill document is a structured set of instructions that tells a language model **how** to approach a specific type of task. It's analogous to learned weights in a neural network — encoding task-specific knowledge in natural language rather than floating-point parameters.
## Structure
A typical skill document contains:
```markdown
# Task Strategy
## General Approach
- Break complex problems into sub-steps
- Always verify intermediate results
## Common Patterns
- When you see X, try approach Y
- Avoid Z because it leads to errors
## Edge Cases
- If the input contains A, handle it specially by...
- Watch out for B — it requires C
## Output Format
- Always include reasoning before the answer
- Format numbers with proper units
```
## How It Evolves
During training, the skill document is modified by **edit patches**:
1. **Additions**: New rules or strategies discovered from failed trajectories
2. **Modifications**: Refining existing rules that are partially correct
3. **Deletions**: Removing rules that consistently lead to errors
Each edit is validated through the **gate** mechanism before being permanently accepted.
## Initial Skill
You can start training with:
- **Empty skill**: The system learns everything from scratch
- **Seed skill**: Provide initial instructions to bootstrap training
- **Pre-trained skill**: Transfer a skill from a related benchmark
Configure the initial skill in your YAML:
```yaml
train:
init_skill: "path/to/initial_skill.md" # or omit for empty
```
## Skill Quality Metrics
Track your skill's evolution through:
- **Validation score**: Primary metric on the selection split
- **Test score**: Final metric on held-out test data
- **Skill length**: Total tokens in the document
- **Edit acceptance rate**: Fraction of proposed edits that pass gating
## Best Practices
!!! tip "Tips for better skills"
1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster
2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement
3. **Enable slow update** (`use_slow_update: true`) to prevent forgetting across epochs
4. **Enable meta skill** (`use_meta_skill: true`) so the optimizer accumulates strategy memory
## Next Steps
- [Deep Learning Analogy](dl-analogy.md)
- [Configuration Reference](../reference/config.md)
docs/guide/training-loop.md
+92
View File
@@ -0,0 +1,92 @@
# The Training Loop
SkillOpt's core insight: **optimizing natural-language skill documents follows the same structure as training neural networks**.
## Overview
```
┌─────────────────────────────────────────────────────────┐
│ Training Loop │
│ │
│ for epoch in epochs: │
│ for step in steps: │
│ 1. Rollout — Target executes tasks │
│ 2. Reflect — Optimizer analyzes trajectories │
│ 3. Aggregate — Hierarchical merge of patches │
│ 4. Select — Rank & clip edits (learning rate) │
│ 5. Update — Apply patches to skill doc │
│ 6. Gate — Validate & accept/reject │
│ │
│ Epoch Boundary: │
│ • Slow Update (longitudinal comparison & guidance) │
│ • Meta Skill (cross-epoch strategy memory) │
└─────────────────────────────────────────────────────────┘
```
## Stage Details
### 1. Rollout (Forward Pass)
The **target** model executes tasks using the current skill document as its prompt. Each task produces a trajectory and a score.
```python
# Analogy: forward pass through the network
predictions = model(input, skill_document)
scores = evaluate(predictions, ground_truth)
```
### 2. Reflect (Backward Pass)
The **optimizer** model analyzes failed trajectories and produces **edit patches** — structured suggestions for improving the skill document.
Two modes:
- **Shallow**: Analyze each trajectory independently
- **Deep**: Cross-reference multiple failures to find systemic issues
```python
# Analogy: computing gradients
gradients = loss.backward() # → edit patches
```
### 3. Aggregate
Semantically similar edit patches are merged to avoid redundant edits.
### 4. Select (Gradient Clipping)
Edits are ranked by relevance score. The `learning_rate` parameter caps how many edits are applied per step — just like gradient clipping prevents overshooting.
```python
# Analogy: gradient clipping + optimizer step size
selected = top_k(edits, k=learning_rate)
```
The `lr_scheduler` adjusts this over training:
- **cosine**: Start aggressive, taper smoothly
- **linear**: Linear decay
- **constant**: Fixed rate
### 5. Update (Parameter Update)
Selected edits are applied to the skill document, producing a new version.
### 6. Gate (Validation)
The updated skill is evaluated on a **selection split** (analogous to a validation set). The update is only accepted if performance improves.
## Epoch Boundary Mechanisms
### Slow Update
At the end of each epoch (starting from epoch 2), the system performs a **longitudinal comparison**: it rolls out both the previous epoch's skill and the current skill on the same samples, categorizes items as improved/regressed/persistent_fail/stable_success, then generates high-level **guidance** that is injected into the skill document. This prevents catastrophic forgetting of earlier improvements.
### Meta Skill
A **meta-skill memory** accumulates high-level strategy notes across the entire training run. At the end of each epoch, the optimizer reflects on what changed between epochs and produces a compact memory that is provided as additional context during future reflection steps.
## Next Steps
- [Understand Skill Documents](skill-document.md)
- [DL ↔ SkillOpt analogy table](dl-analogy.md)