wehub/1jehuang--jcode
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
FreeBSD Smoke / FreeBSD Smoke (x86_64) (push) Has been cancelled
Details
CI / Quality Guardrails (push) Has been cancelled
Details
CI / Build & Test (macos-latest) (push) Has been cancelled
Details
CI / Build & Test (ubuntu-latest) (push) Has been cancelled
Details
CI / Build & Test (windows-latest) (push) Has been cancelled
Details
CI / Format (push) Has been cancelled
Details
CI / PowerShell Syntax (push) Has been cancelled
Details
CI / Windows Cross-Target Check (Linux) (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:10:34 +08:00
commit a789495a98
1551 changed files with 718128 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/AGENT_NATIVE_VCS_CORE_BEHAVIOR.md
+343
View File
@@ -0,0 +1,343 @@
# Agent-Native VCS: Core Behavior
## Status
Draft project note synthesizing the core ideas from this session.
## One-line thesis
This system is **not primarily a better merge algorithm**. It is a Git/jj-layer VCS that preserves the **meaning, ownership, and maintenance context** of local changes so intelligent agents can continuously coordinate with each other and re-maintain local deltas against a moving upstream.
## Core framing
- Git is largely **branch-first**.
- jj is largely **change-first**.
- This system should be **lane-first** and **maintenance-packet-first**.
The underlying storage and transport can remain Git-compatible. The innovation is in the metadata, grouping, and workflows the VCS makes first-class.
## What problem this VCS is solving
There are two related but distinct problems:
```mermaid
flowchart TD
G[Git/jj storage and transport] --> L[Lane-first coordination layer]
L --> P[Owned draft patches]
L --> M[Maintenance packets]
P --> H[Clean published history]
M --> U[Smarter upstream maintenance]
```
1. **Parallel multi-agent editing inside one codebase**
- Agents work in short bursts of coherent edits.
- Their work may interleave and may conflict.
- Anonymous dirty state causes hesitation and contamination.
2. **Long-lived local customization over moving upstream**
- Users will increasingly maintain personalized downstream variants of open source projects.
- The hard problem is not patch application; it is preserving the **intent** of the local delta as upstream changes.
## Design goals
1. Make agent work naturally representable.
2. Eliminate anonymous dirty state.
3. Preserve enough context for future agents to maintain local changes against upstream.
4. Keep machine history rich while allowing human-facing history to stay clean.
5. Remain compatible with Git ecosystems and remote hosting.
## Non-goals
- Do not try to replace Git object storage first.
- Do not promise that all major upstream divergence can be automatically resolved.
- Do not depend on raw reasoning traces as the main artifact of understanding.
## Core entities
### 1. Lane
A **lane** is the primary unit of ongoing work.
A lane is usually keyed by:
- `goal`
- `agent`
For downstream maintenance, a lane may instead be a long-lived **customization lane** representing a persistent local delta.
A lane is not just a label. It has:
- local sequence/order
- owned draft state
- provenance
- anchors into code/upstream
- contracts/invariants
- maintenance policy
### 2. Draft patch (or micro-commit)
Every meaningful edit produced by an agent should be captured as an owned, replayable unit.
Properties:
- associated with one lane
- attributable to one agent/model/session
- based on a specific revision
- revertible and replayable
- safe to compact later
This avoids the model of a shared dirty working tree with unclear ownership.
### 3. Burst
Agents often emit several rapid, coherent edits while pursuing one subtask.
A **burst** groups these temporally adjacent draft patches into one operational work episode.
### 4. Published commit
A human-facing commit may be compacted from one or more draft patches or bursts.
This gives a two-level model:
- **operational history** for concurrency and maintenance
- **published history** for human review and sharing
### 5. Maintenance packet
Every local delta should carry a context packet rich enough for future maintenance.
A maintenance packet contains at least:
- intent/goal
- behavioral contract
- semantic anchors
- assumptions
- validation hooks
- provenance
- lifecycle/upstream policy
- concise rationale
### 6. Anchor
An anchor records what upstream concept a lane or patch is attached to.
Examples:
- symbol/function/type
- endpoint
- config key
- UI element
- file region
- protocol/schema field
Anchors are stronger than line-level diffs for long-term maintenance.
## Core invariants
### Invariant 1: No anonymous dirtiness
All uncommitted changes must belong to something:
- a lane
- a draft patch
- a scratch area with explicit ownership
The system should discourage or auto-resolve unattributed dirty state.
### Invariant 2: Capture is separate from publish
Agents should not have to decide immediately whether to create a final human-facing commit.
Instead:
- edits are captured automatically into owned draft units
- publishing/compaction happens later
### Invariant 3: Local meaning matters more than exact old patch shape
For upstream maintenance, the system preserves the **meaning of the delta**, not just its old textual diff.
### Invariant 4: Interleaving is normal
Commits from different lanes may interleave in global history while each lane retains its own local coherence.
```mermaid
flowchart LR
subgraph LA[Lane A]
a1[a1] --> a2[a2] --> a3[a3]
end
subgraph LB[Lane B]
b1[b1] --> b2[b2]
end
subgraph GI[Global integration order]
g1[a1] --> g2[b1] --> g3[a2] --> g4[b2] --> g5[a3]
end
```
## Core behaviors
### 1. Group work by lane, not only by branch
The primary grouping is not a branch but a **lane**.
A lane answers:
- what goal is this serving?
- which agent produced it?
- what code/upstream concepts is it attached to?
- how should it be maintained later?
### 2. Treat edits as owned draft units as soon as possible
When an agent edits code, the system should quickly capture those edits into a draft patch for the active lane.
This avoids:
- commit contamination
- uncertainty about ownership
- fear of losing unrelated work
### 3. Keep dirty state explicit and attributable
If a workspace is not clean, the state should read as something like:
- draft changes for lane A
- draft changes for lane B
- scratch changes owned by session X
not merely "repo dirty".
```mermaid
flowchart TD
WT[Workspace state] --> C{Attributed?}
C -- Yes --> LA[Draft patch in Lane A]
C -- Yes --> LB[Draft patch in Lane B]
C -- Yes --> S[Explicit scratch area]
C -- No --> BAD[Anonymous dirty state\nDisallowed / auto-captured]
```
### 4. Support interleaved global integration
Lanes may be globally interleaved. That is acceptable.
Example:
- Lane A: `a1 -> a2 -> a3`
- Lane B: `b1 -> b2`
- Integrated order: `a1 -> b1 -> a2 -> b2 -> a3`
The system should preserve both:
- local lane order
- global integration order
### 5. Preserve a maintenance packet for each local delta
For each lane/customization, store:
- why it exists
- what behavior must remain true
- what it is attached to upstream
- what assumptions it depends on
- how to test it
- when to drop, adapt, or regenerate it
This is the core enabler for intelligent downstream maintenance.
### 6. Maintenance is replay/adapt/regenerate/drop — not only merge
When upstream changes, the system should help agents decide among:
1. replay the delta
2. structurally adapt the delta
3. regenerate from goal + contract
4. drop because upstream subsumed it
5. redesign because upstream changed the world too much
```mermaid
flowchart TD
U[Upstream changed] --> Q{Does old delta still fit?}
Q -- Yes --> R[Replay]
Q -- Partially --> A[Adapt structurally]
Q -- No but goal still valid --> G[Regenerate from intent + contract]
Q -- Upstream already covers it --> D[Drop local delta]
Q -- Goal/world changed too much --> X[Redesign or escalate]
```
### 7. Continuous classification matters more than blind merge
For each lane relative to upstream, the system should help classify:
- clean
- drifting
- partially subsumed
- conflicted
- broken
- needs regeneration
- should be dropped
- needs human/product decision
### 8. Published history should stay clean
Machine history can be noisy and fine-grained.
Human history should remain compact, reviewable, and comprehensible.
## What information the VCS should encourage
For each local lane/customization, strongly encourage or require:
### Intent
- why this change exists
- user/stakeholder need
- must-have vs preference
- non-goals
### Behavioral contract
- invariants
- acceptance criteria
- relevant tests
- performance/security/UX constraints
### Semantic anchors
- symbols/types/APIs/config keys/endpoints/UI elements touched or depended on
### Assumptions
- ordering assumptions
- environment assumptions
- dependency assumptions
- extension-point assumptions
### Provenance
- agent id
- model id
- prompts/specs/task references
- authored-against revision
- confidence/review status
### Rationale
- concise explanation of the chosen path
- important alternatives rejected
- known uncertainty
### Upstream policy
- override upstream / defer to upstream / drop if subsumed / candidate for upstreaming
### Lifecycle
- permanent / temporary / experiment / workaround / expiry conditions
### Validation hooks
- tests, commands, fixtures, benchmarks, snapshots, smoke checks
## What this VCS can realistically solve
It can dramatically improve:
- agent coordination in one repo
- attribution of uncommitted changes
- structured integration of interleaved work
- preservation of local-delta meaning across upstream updates
- the ability of intelligent agents to re-maintain forks
## What it cannot promise
It cannot guarantee automatic maintenance when upstream changes are radical.
If upstream replaces the subsystem, changes architecture, or invalidates the original local goal, the right action may be to regenerate or redesign rather than merge.
This VCS should therefore promise:
> Preserve enough meaning and structure that an intelligent agent can make the right maintenance decision.
## Core promise
Given a local codebase with parallel agents and a moving upstream, this VCS should make the following true:
1. Every local change has ownership.
2. Every important local delta retains its meaning.
3. Dirty state is explicit and attributable.
4. Interleaving is representable without losing lane coherence.
5. Future agents can understand not just **what changed**, but **why it changed** and **how to keep it alive**.
## Minimal conceptual model
At minimum, the system needs first-class support for:
- lanes
- draft patches
- bursts
- published commits
- anchors
- maintenance packets
- upstream maintenance policies
```mermaid
flowchart LR
Lane --> DraftPatch[Draft patch]
DraftPatch --> Burst
Burst --> Publish[Published commit]
Lane --> Packet[Maintenance packet]
Packet --> Anchor
Packet --> Policy[Upstream policy]
```
## Suggested next step
Translate this into a concrete schema for:
- `lane`
- `draft_patch`
- `maintenance_packet`
- `anchor`
- `publish`
- `upstream_status`
docs/AMBIENT_MODE.md
+966
View File
@@ -0,0 +1,966 @@
# Ambient Mode
> **Status:** Design
> **Updated:** 2026-02-08
A proactive, always-on agent mode that works autonomously without user prompting. Like a brain consolidating memories during sleep, ambient mode tends to the memory graph, identifies useful work, and acts on the user's behalf — all while staying within resource limits.
## Overview
Ambient mode operates as a background loop that:
1. **Gardens** — consolidates, prunes, and strengthens the memory graph
2. **Scouts** — analyzes recent sessions, git history, and memories to understand what the user cares about
3. **Works** — proactively completes tasks the user would appreciate being surprised by
These aren't separate phases. The agent does all three in a single pass — while looking at memories it naturally discovers maintenance work and identifies proactive opportunities simultaneously.
**Key Design Decisions:**
1. **Single agent at a time** — only one ambient instance ever runs, no parallelism
2. **Subscription-first** — defaults to OAuth (OpenAI/Anthropic), never uses API keys unless explicitly configured
3. **User priority** — interactive sessions always take precedence over ambient work
4. **Strong models** — uses the strongest available model from the selected provider so the agent can reason well about what's actually useful
5. **Self-scheduling** — the agent decides when to wake next, constrained by adaptive resource limits
---
## Architecture
```mermaid
graph TB
subgraph "Scheduling Layer"
EV[Event Triggers<br/>session close, crash, git push]
TM[Timer<br/>agent-scheduled wake]
RC[Resource Calculator<br/>adaptive interval]
SQ[(Scheduled Queue<br/>persistent)]
end
subgraph "Ambient Agent"
QC[Check Queue]
SC[Scout<br/>memories + sessions + git]
GD[Garden<br/>consolidate + prune + verify]
WK[Work<br/>proactive tasks]
SA[schedule_ambient tool<br/>set next wake + context]
end
subgraph "Resource Awareness"
UH[Usage History<br/>rolling window]
RL[Rate Limits<br/>per provider]
AU[Ambient Usage<br/>current window]
AC[Active Sessions<br/>user activity]
end
subgraph "Outputs"
MG[(Memory Graph<br/>consolidated)]
CM[Commits & Changes]
IW[Info Widget<br/>TUI display]
end
EV -->|wake early| RC
TM -->|scheduled wake| RC
RC -->|"gate: safe to run?"| QC
SQ -->|pending items| QC
QC --> SC
SC --> GD
SC --> WK
GD --> MG
WK --> CM
SA -->|next wake + context| SQ
SA -->|proposed interval| RC
UH --> RC
RL --> RC
AU --> RC
AC --> RC
QC --> IW
SC --> IW
GD --> IW
WK --> IW
style EV fill:#fff3e0
style TM fill:#fff3e0
style RC fill:#ffcdd2
style SQ fill:#e3f2fd
style QC fill:#e8f5e9
style SC fill:#e8f5e9
style GD fill:#e8f5e9
style WK fill:#e8f5e9
```
---
## Ambient Cycle
Each ambient cycle follows a single flow. The agent doesn't switch between "modes" — it naturally handles gardening, scouting, and work in one pass.
```mermaid
sequenceDiagram
participant SYS as System Scheduler
participant RES as Resource Calculator
participant AMB as Ambient Agent
participant MEM as Memory Graph
participant CB as Codebase
participant Q as Scheduled Queue
SYS->>RES: Timer/event fired
RES->>RES: Check usage headroom
alt Over budget
RES->>SYS: Delay (recalculate interval)
else Safe to run
RES->>AMB: Spawn ambient agent
end
AMB->>Q: Check scheduled queue
alt Has queued items
Q-->>AMB: Return items + context
AMB->>MEM: Scout relevant memories for queued work
MEM-->>AMB: Context memories
AMB->>CB: Execute queued work
end
AMB->>MEM: Load memory graph
MEM-->>AMB: Full graph state
Note over AMB: Garden pass
AMB->>AMB: Find duplicates → merge & reinforce
AMB->>AMB: Find contradictions → resolve
AMB->>AMB: Find decayed memories → prune or re-verify
AMB->>CB: Verify stale facts against codebase
CB-->>AMB: Verification results
AMB->>MEM: Apply consolidation changes
Note over AMB: Scout pass (simultaneous)
AMB->>AMB: Analyze recent sessions for missed extractions
AMB->>AMB: Check git history for active work
AMB->>AMB: Identify proactive work opportunities
Note over AMB: Work pass
AMB->>CB: Execute proactive tasks
AMB->>MEM: Store new memories from findings
AMB->>AMB: end_ambient_cycle(summary, schedule)
AMB->>SYS: Done (summary → widget + email)
```
---
## Ambient Agent Tools
The ambient agent has access to a subset of jcode tools plus ambient-specific tools.
### `end_ambient_cycle` (required)
Every ambient cycle **must** end with this tool call. The system uses the summary for the notification email and the info widget.
```rust
// Tool: end_ambient_cycle
{
"summary": "Merged 3 duplicate memories, pruned 2 stale facts,
extracted memories from crashed session jcode-red-fox-1234",
"memories_modified": 8,
"compactions": 2,
"proactive_work": null,
"next_schedule": {
"wake_in_minutes": 25,
"context": "Verify 4 remaining stale facts"
}
}
```
| Field | Required | Description |
|-------|----------|-------------|
| `summary` | yes | Human-readable summary of what was done (goes into email/widget) |
| `memories_modified` | yes | Count of memories created/merged/pruned/updated |
| `compactions` | yes | Number of context compactions during this cycle |
| `proactive_work` | no | Description of proactive code changes, if any |
| `next_schedule` | no | When to wake next + context (falls back to system default if omitted) |
### `schedule_ambient`
Can also be called mid-cycle to queue future work:
```rust
// Tool: schedule_ambient
{
"wake_in_minutes": 15,
"context": "Check if CI passed for auth refactor PR",
"priority": "normal"
}
```
### `todos`
The agent should use a todos tool to plan its cycle. This provides:
- Visibility into what the agent planned vs what it actually did
- If the cycle is interrupted, we know what's left
- Structure for the agent's reasoning
### `request_permission`
From the [Safety System](./SAFETY_SYSTEM.md). Used for any Tier 2 action.
---
## Handling Unexpected Stops
The model may stop unexpectedly (output length limit, API error, random stop). The system handles this:
```mermaid
stateDiagram-v2
[*] --> Running: Cycle started
Running --> Stopped: Model output ends
Stopped --> CheckTool{Called end_ambient_cycle?}
CheckTool --> Complete: Yes → normal completion
CheckTool --> Continuation: No → send continuation message
Continuation --> Running: Model continues work
Continuation --> Stopped: Model stops again
Stopped --> ForcedEnd: Second stop without end_ambient_cycle
ForcedEnd --> Incomplete: Generate partial transcript,\nschedule default wake
Complete --> [*]
Incomplete --> [*]
```
**Continuation message** (injected as user message):
```
You stopped unexpectedly without calling end_ambient_cycle.
If you are done with your work, call end_ambient_cycle with a
summary of what you accomplished and schedule your next wake.
If you are not done, continue what you were doing.
```
**If no `end_ambient_cycle` is called after two attempts:**
- System generates a partial transcript marked as `incomplete`
- Compaction count is pulled from system metrics
- Default wake interval is scheduled
- Warning logged for debugging
**If no `schedule_ambient` or `next_schedule` in `end_ambient_cycle`:**
- System schedules a default wake at `max_interval_minutes` from config
- Warning logged — the agent should always schedule its next wake
---
## System Prompt
The ambient agent's system prompt is built dynamically each cycle with real data. The prompt gives the agent information to reason with, not rigid instructions for how to think.
```
You are the ambient agent for jcode. You operate autonomously without
user prompting. Your job is to maintain and improve the user's
development environment.
## Current State
- Last ambient cycle: {timestamp} ({time_ago})
- Machine was off/idle since: {if applicable}
- Active user sessions: {count, or "none"}
- Cycle budget: ~{estimated_max_tokens} tokens
## Scheduled Queue
{queued items with context, or "empty — do general ambient work"}
## Recent Sessions (since last cycle)
{for each session:
- id, status (closed/crashed/active), duration, topic summary
- extraction status (extracted/missed/partial)
}
## Memory Graph Health
- Total memories: {count} ({active} active, {inactive} inactive)
- Memories with confidence < 0.1: {count}
- Unresolved contradictions: {count}
- Memories without embeddings: {count}
- Duplicate candidates (similarity > 0.95): {count}
- Last consolidation: {timestamp}
## User Feedback History
{recent memories about ambient approval/rejection patterns}
## Resource Budget
- Provider: {name}
- Tokens remaining in window: {count}
- Window resets: {timestamp}
- User usage rate: {tokens/min average}
- Budget for this cycle: stay under {limit} tokens
## Instructions
Start by using the todos tool to plan what you'll do this cycle.
Priority order:
1. Execute any scheduled queue items first.
2. Garden the memory graph — consolidate duplicates, resolve
contradictions, prune dead memories, verify stale facts,
extract from missed sessions.
3. Scout for proactive work (only if enabled and past cold start) —
look at recent sessions and git history to identify useful work
the user would appreciate.
For gardening: focus on highest-value maintenance first. Duplicates
and contradictions before pruning. Verify stale facts only if you
have budget left.
For proactive work: be conservative. A bad surprise is worse than
no surprise. Check the user feedback memories — if they've rejected
similar work before, don't do it. Code changes must go on a worktree
branch with a PR via request_permission.
When done, you MUST call end_ambient_cycle with a summary of
everything you did, including compaction count. Always schedule
your next wake time with context for what you plan to do next.
```
---
## Usage Calculation
### Tracking
Every API call (user or ambient) is logged:
```rust
struct UsageRecord {
timestamp: DateTime<Utc>,
source: UsageSource, // User | Ambient
tokens_input: u32,
tokens_output: u32,
provider: String,
}
```
### Rate Limit Discovery
Rate limits are learned from provider response headers:
```
x-ratelimit-limit-requests: 50
x-ratelimit-remaining-requests: 42
x-ratelimit-limit-tokens: 100000
x-ratelimit-remaining-tokens: 85000
x-ratelimit-reset-requests: 2026-02-08T15:00:00Z
```
When headers aren't available, fall back to conservative defaults and adjust based on whether rate limit errors occur.
### Adaptive Interval Algorithm
```
# Known from headers or defaults
window_remaining = reset_time - now
tokens_remaining = ratelimit_remaining_tokens
requests_remaining = ratelimit_remaining_requests
# Estimate user consumption from rolling history
user_rate = rolling_average(
usage_log.filter(source=User, last_hour),
per_minute
)
# Project user usage for rest of window
user_projected = user_rate * window_remaining
# Reserve 20% buffer so user never feels throttled
ambient_budget = (tokens_remaining - user_projected) * 0.8
# Estimate cost per ambient cycle from recent cycles
tokens_per_cycle = rolling_average(
recent_ambient_cycles.last(5).tokens_used
)
# How many cycles fit in remaining budget?
cycles_available = ambient_budget / tokens_per_cycle
# Spread evenly across remaining window
if cycles_available > 0:
interval = window_remaining / cycles_available
else:
interval = window_remaining # wait for reset
# Clamp to configured bounds
interval = clamp(interval, min_interval, max_interval)
```
### Behavioral Rules
| Condition | Behavior |
|-----------|----------|
| User is active in a session | Pause ambient (or multiply interval by 3-5x) |
| User has been idle for hours | Run cycles more frequently |
| Hit a rate limit | Exponential backoff (double interval each time) |
| No rate limit errors for N cycles | Gradually decrease interval |
| No headers available | Start with 30min interval, adjust from errors |
| Approaching end of window with budget left | Squeeze in extra cycles |
| Over 80% of budget consumed | Fall back to max_interval |
---
## Memory Consolidation
### Two-Layer Architecture
Memory consolidation happens at two levels, mirroring how the brain encodes during the day and consolidates during sleep.
```mermaid
graph LR
subgraph "Layer 1: Sidecar (every turn, fast)"
S1[Memory retrieved<br/>for relevance check]
S2{New memory<br/>similar to existing?}
S3[Reinforce existing<br/>+ breadcrumb]
S4[Create new memory]
S5[Supersede if<br/>contradicts]
end
subgraph "Layer 2: Ambient Garden (background, deep)"
A1[Full graph scan]
A2[Cross-session<br/>dedup]
A3[Fact verification<br/>against codebase]
A4[Retroactive<br/>session extraction]
A5[Prune dead<br/>memories]
A6[Relationship<br/>discovery]
end
S1 --> S2
S2 -->|yes| S3
S2 -->|no| S4
S2 -->|contradicts| S5
A1 --> A2
A1 --> A3
A1 --> A4
A1 --> A5
A1 --> A6
style S1 fill:#e8f5e9
style S2 fill:#e8f5e9
style S3 fill:#e8f5e9
style S4 fill:#e8f5e9
style S5 fill:#e8f5e9
style A1 fill:#e3f2fd
style A2 fill:#e3f2fd
style A3 fill:#e3f2fd
style A4 fill:#e3f2fd
style A5 fill:#e3f2fd
style A6 fill:#e3f2fd
```
### Layer 1: Sidecar Consolidation
Runs after every turn, only on memories already retrieved for relevance checking. Zero added latency — runs after results are returned to the main agent.
**Operations:**
- **Duplicate detection** — if the sidecar is about to create a memory that's semantically identical to one it just retrieved, reinforce the existing one instead
- **Contradiction detection** — if a new memory contradicts an existing one in the retrieved set, supersede the old one
- **Reinforcement** — bump strength on memories that keep appearing relevant
**Cost:** Near zero. Only operates on memories already in hand.
### Layer 2: Ambient Garden
Deep consolidation that runs during ambient cycles. Has access to the full memory graph and codebase.
**Operations:**
| Operation | Description | Trigger |
|-----------|-------------|---------|
| **Graph-wide dedup** | Find semantically similar memories across entire graph | Embedding similarity > 0.95 |
| **Contradiction resolution** | Resolve `Contradicts` edges by checking current state | Contradicts edges exist |
| **Fact verification** | Check factual memories against codebase | Facts older than confidence half-life |
| **Retroactive extraction** | Analyze recent sessions that lack memory extraction | Sessions with status Crashed, Closed without extraction |
| **Pruning** | Remove memories with near-zero confidence and low strength | confidence < 0.05 AND strength <= 1 |
| **Relationship discovery** | Find new connections between memories | Co-occurrence in sessions, semantic similarity |
| **Embedding backfill** | Generate embeddings for memories that lack them | embedding is None |
| **Cluster refinement** | Re-run clustering on updated embeddings | Every N ambient cycles |
### Reinforcement Provenance
When a memory is reinforced (by sidecar or ambient), the system records a breadcrumb for traceability:
```rust
pub struct Reinforcement {
pub session_id: String,
pub message_index: usize,
pub timestamp: DateTime<Utc>,
}
pub struct MemoryEntry {
// ... existing fields ...
pub reinforcements: Vec<Reinforcement>,
}
impl MemoryEntry {
pub fn reinforce(&mut self, session_id: &str, message_index: usize) {
self.strength += 1;
self.updated_at = Utc::now();
self.reinforcements.push(Reinforcement {
session_id: session_id.to_string(),
message_index,
timestamp: Utc::now(),
});
}
}
```
The consolidation agent can later trace back through reinforcements to understand *why* a memory has the strength it does, and whether those reinforcements still hold.
---
## Scheduling
### Two-Layer Scheduling
```mermaid
graph TB
subgraph "Agent Layer (proposes)"
AT[schedule_ambient tool]
AT -->|"wake in 15m,<br/>context: check CI"| PROP[Proposed Schedule]
end
subgraph "System Layer (constrains)"
PROP --> ADAPT[Adaptive Calculator]
MAX[Max Interval Ceiling] --> ADAPT
MIN[Min Interval Floor] --> ADAPT
ADAPT --> FINAL[Final Schedule]
end
subgraph "Adaptive Calculator Inputs"
UH[User usage history<br/>rolling window]
AU[Ambient usage<br/>current window]
RL[Provider rate limits<br/>from headers]
TW[Time remaining<br/>in limit window]
AS[Active sessions<br/>user currently working?]
end
UH --> ADAPT
AU --> ADAPT
RL --> ADAPT
TW --> ADAPT
AS --> ADAPT
FINAL -->|"actual: 28m<br/>(headroom limited)"| TIMER[System Timer]
style AT fill:#e8f5e9
style ADAPT fill:#ffcdd2
style FINAL fill:#e3f2fd
```
### Agent-Initiated Scheduling
The ambient agent has a `schedule_ambient` tool to request its next wake-up:
```rust
// Tool: schedule_ambient
{
"wake_in_minutes": 15, // or "wake_at": "2026-02-08T15:30:00Z"
"context": "Check if CI passed for auth refactor PR",
"priority": "normal" // "low" | "normal" | "high"
}
```
The context is stored in the scheduled queue so when the agent wakes up, it knows what it planned to do.
### Adaptive Resource Calculation
The system calculates the safe interval based on usage patterns:
```
headroom = rate_limit - (user_usage_rate + ambient_usage_rate)
safe_interval = max(min_interval, target_budget_fraction / headroom)
```
**Inputs:**
- **User usage rate** — rolling average of tokens/requests per hour from interactive sessions
- **Ambient usage rate** — tokens/requests consumed by ambient in current window
- **Rate limits** — known per-provider limits (from response headers or config)
- **Time in window** — how much of the rate limit window remains
- **Active sessions** — if user is currently in a session, ambient pauses or throttles heavily
**Behavior:**
- Agent says "wake in 10m" but system calculates "not safe until 30m" → pushed to 30m
- Agent says "wake in 6h" but system sees unused budget → pulled forward to max interval
- User starts interactive session → ambient pauses, resumes when user goes idle
- Approaching rate limit → ambient backs off exponentially
### Event Triggers
Certain events can wake ambient early (still subject to resource gate):
| Event | Priority | Rationale |
|-------|----------|-----------|
| Session crashed | High | Likely missed memory extraction |
| Session closed | Normal | May have unextracted memories |
| Git push | Low | Codebase changed, facts may be stale |
| User idle > threshold | Low | Good time for ambient work |
| Explicit `/ambient` command | Immediate | User requested |
### Scheduled Queue
Persistent queue of scheduled ambient tasks:
```rust
pub struct ScheduledItem {
pub id: String,
pub scheduled_for: DateTime<Utc>,
pub context: String,
pub priority: Priority,
pub created_by_session: String, // which ambient cycle created this
pub created_at: DateTime<Utc>,
}
pub enum Priority {
Low,
Normal,
High,
}
```
**Queue rules:**
- Checked first when ambient wakes up
- Items sorted by priority then scheduled time
- Expired items (past their scheduled_for) are still executed
- System can delay items if over budget, but won't drop them
- Only one ambient agent at a time — if one is running, new triggers queue up
---
## Provider & Model Selection
### Default Priority
```mermaid
graph TD
START[Ambient Mode Start] --> CHECK1{OpenAI OAuth<br/>available?}
CHECK1 -->|yes| OAI[Use OpenAI<br/>strongest available]
CHECK1 -->|no| CHECK2{Anthropic OAuth<br/>available?}
CHECK2 -->|yes| ANT[Use Anthropic<br/>strongest available]
CHECK2 -->|no| CHECK3{API key or OpenRouter +<br/>config opt-in?}
CHECK3 -->|yes| API[Use API/OpenRouter<br/>with budget cap]
CHECK3 -->|no| DISABLED[Ambient mode disabled<br/>no provider available]
style OAI fill:#e8f5e9
style ANT fill:#fff3e0
style API fill:#ffcdd2
style DISABLED fill:#f5f5f5
```
**Rationale:**
- **OpenAI first** — separate rate limit pool from Anthropic, so ambient doesn't compete with interactive sessions
- **Anthropic second** — also subscription-based (OAuth), no per-token cost
- **OpenRouter/API keys last** — these are pay-per-token; opt-in only via config to avoid silently burning credits
- **Strong models** — ambient needs good judgment about what work is valuable. A weak model would do the wrong proactive work and annoy the user.
### Model Selection
| Provider | Default Model | Rationale |
|----------|--------------|-----------|
| OpenAI OAuth | Strongest available (e.g. `5.2-codex-xhigh`) | Best reasoning for judgment calls |
| Anthropic OAuth | Strongest available (e.g. `claude-opus-4-6`) | Best available on Anthropic |
| OpenRouter (opt-in) | Strongest available | Pay-per-token, requires config opt-in |
| API key (opt-in) | Configurable | User chooses cost/capability tradeoff |
### Resource Rules
1. **Subscription (OAuth — OpenAI/Anthropic):** Ambient is allowed, subject to adaptive rate limiting
2. **Pay-per-token (API keys, OpenRouter):** Off by default. Enable in config with optional daily budget cap
3. **User active:** Ambient pauses or throttles to minimum when user has an active session
4. **Rate limited:** If ambient hits a rate limit, back off aggressively (exponential backoff)
5. **Separate pools:** Prefer OpenAI for ambient when Anthropic is used interactively (and vice versa)
---
## Proactive Work
### What Ambient Does
The agent uses memories, recent sessions, and git history to identify useful work:
```mermaid
graph LR
subgraph "Context Gathering"
M[Memories<br/>user preferences,<br/>priorities]
S[Recent Sessions<br/>what user was<br/>working on]
G[Git History<br/>active branches,<br/>recent changes]
end
subgraph "Inference"
I[What does the user<br/>care about most?]
U[What upcoming work<br/>is there?]
O[What would surprise<br/>the user positively?]
end
subgraph "Actions"
T[Write/fix tests]
R[Small refactors]
D[Update stale docs]
F[Fix obvious issues]
C[Clean up TODOs]
end
M --> I
S --> I
G --> I
I --> O
U --> O
O --> T
O --> R
O --> D
O --> F
O --> C
```
### Safety
Ambient mode operates under the [Safety System](./SAFETY_SYSTEM.md) — a human-in-the-loop layer that classifies actions, requests permission for anything risky, and notifies the user via email/SMS/desktop.
Key constraints for ambient:
- **All actions classified** — auto-allowed (read, local branches, memory ops), requires permission (PRs, pushes, communication), or always denied (force-push, delete remote branches)
- **Commits to a separate branch** — never pushes to main/master directly
- **Code changes require worktree + PR** — modifications always go through review
- **Small, focused changes** — no large refactors without user request
- **Session transcript** — full log of every action, sent as summary after each cycle
- **Respects .gitignore and sensitive files** — same security rules as interactive mode
- **Can be reviewed** — user sees ambient work in the TUI and pending permission requests
---
## Info Widget
The TUI displays ambient mode status alongside existing widgets (memory, tokens, etc.).
### Widget Content
```
╭─ Ambient ─────────────────────────╮
│ ● Running (garden + scout) │
│ Queue: 2 items (next: check CI) │
│ Last: 12m ago — pruned 3, merged 1│
│ Next: ~18m (adaptive) │
│ Budget: ██████░░░░ 58% remaining │
╰───────────────────────────────────╯
```
**Fields:**
| Field | Description |
|-------|-------------|
| **Status** | `idle` / `running (detail)` / `scheduled` / `paused (rate limited)` |
| **Queue** | Count of scheduled items + preview of next one's context |
| **Last cycle** | Time since last run + summary of what it did |
| **Next wake** | Estimated time until next cycle (from adaptive calculator) |
| **Budget** | Visual bar showing usage: user + ambient + remaining headroom |
### Budget Breakdown
The budget bar shows three segments:
```
User usage Ambient usage Remaining
████████████ ████ ░░░░░░░░░░
45% 12% 43%
```
This gives the user immediate visibility into whether ambient is being too aggressive.
---
## Configuration
```toml
[ambient]
# Enable ambient mode (default: false until stable)
enabled = false
# Provider override (default: auto-select per priority chain)
# provider = "openai"
# Model override (default: provider's strongest)
# model = "5.2-codex-xhigh"
# Allow API key usage (default: false, only OAuth)
allow_api_keys = false
# Daily token budget when using API keys (ignored for OAuth)
# api_daily_budget = 100000
# Minimum interval between cycles in minutes (default: 5)
min_interval_minutes = 5
# Maximum interval between cycles in minutes (default: 120)
max_interval_minutes = 120
# Pause ambient when user has active session (default: true)
pause_on_active_session = true
# Enable proactive work (vs garden-only mode) (default: true)
proactive_work = true
# Proactive work branch prefix (default: "ambient/")
work_branch_prefix = "ambient/"
```
---
## Storage
```
~/.jcode/ambient/
├── state.json # Current ambient state (status, last run, etc.)
├── queue.json # Scheduled queue (persistent across restarts)
├── usage.json # Usage history for adaptive calculation
└── logs/
└── ambient-YYYY-MM-DD.log # Daily ambient activity logs
```
---
## Context Window Management
Ambient mode uses the same compaction strategy as interactive sessions: **compact at 80% context window usage.** No special handling needed — if an ambient cycle is analyzing a large memory graph or many sessions, it compacts and continues.
---
## User Feedback via Memory
Ambient learns from the user's approval/rejection decisions through the memory system itself. No separate feedback mechanism is needed.
- **User rejects a proactive change** → ambient stores a memory: *"User rejected ambient PR to refactor auth tests — prefers not to have tests auto-modified"*
- **User approves** → memory: *"User approved ambient fixing typos in docs"*
- **Pattern emerges** → these memories get reinforced over time, naturally influencing what ambient prioritizes
This works because ambient already scouts memories before deciding what to do. Its own approval/rejection history becomes part of the context it reasons about, and these memories consolidate, decay, and reinforce like everything else in the graph.
---
## Crash Safety & Recovery
Ambient must assume the process can die at any point (battery death, crash, OOM, etc.) and design so nothing is lost or corrupted.
### Principles
- **Atomic writes** — memory graph and state files are written to a temp file first, then atomically renamed. A crash mid-write doesn't corrupt existing data.
- **Incremental checkpointing** — if ambient is halfway through gardening 50 memories and crashes, it shouldn't redo the ones already finished. A "last processed" marker tracks progress within a cycle.
- **Persistent queue survives crashes** — scheduled queue and permission requests are on disk, not in memory. They survive restarts.
- **Interrupted transcripts** — if a cycle doesn't complete, the transcript is marked as `interrupted` rather than `completed`, so the user knows it didn't finish.
### Recovery on Restart
When ambient starts after an unexpected shutdown:
1. **Don't replay missed cycles** — don't try to run every cycle that was scheduled while the machine was off. Just run one cycle that examines current state.
2. **Check time since last run** — if the gap is large (hours/days), there may be a backlog of crashed sessions to extract, stale memories to verify, etc. The agent handles this naturally since it always checks current state rather than diffing from last run.
3. **Expired scheduled items** — still execute them. The context the agent stored is still valid, the work is just late.
4. **Resume, don't restart** — if a cycle was interrupted mid-way, check the checkpoint and continue from where it left off rather than starting over.
### State Diagram
```mermaid
stateDiagram-v2
[*] --> Starting: jcode starts
Starting --> CheckLastRun: ambient enabled?
CheckLastRun --> NormalCycle: last run recent
CheckLastRun --> CatchUpCycle: last run stale (hours/days)
CheckLastRun --> ResumeCycle: interrupted cycle found
NormalCycle --> Sleeping: cycle complete
CatchUpCycle --> Sleeping: cycle complete
ResumeCycle --> Sleeping: cycle complete
Sleeping --> NormalCycle: timer/event fires
Sleeping --> [*]: machine off / crash
note right of CatchUpCycle: Single cycle examining\ncurrent state, not\nreplaying missed cycles
note right of ResumeCycle: Continue from\ncheckpoint marker
```
---
## Cold Start
First time ambient runs, there's no usage history, no patterns, no feedback memories. Bootstrapping strategy:
- **Start conservative** — garden-only (memory maintenance), no proactive work until ambient has enough context
- **Build usage baseline** — first few cycles just observe and track usage patterns for the adaptive scheduler
- **Proactive work unlocks gradually** — after N successful garden cycles with user-approved results, ambient can start scouting for proactive work
- **Or user opts in immediately** — config option to skip the warm-up if the user trusts it
---
## Per-Project Configuration
Some projects may need different ambient behavior (e.g. sensitive work projects, personal repos with different preferences):
```toml
# In project-level .jcode/config.toml
[ambient]
# Disable ambient entirely for this project
enabled = false
# Or restrict to garden-only (no proactive code changes)
proactive_work = false
```
---
## Multi-Machine (Deferred)
When ambient runs on multiple machines (e.g. laptop + desktop), shared state could conflict: double-processing sessions, conflicting memory edits, overlapping proactive work.
This is a distributed systems problem that will be addressed once ambient is stable on a single machine. Potential approaches:
- Machine ID on memory writes for conflict resolution
- Lock file or leader election for exclusive operations
- Git worktrees are already isolated, so proactive work is naturally conflict-free
---
## Implementation Phases
### Phase 1: Foundation
- [ ] Ambient agent loop (spawn, run, sleep)
- [ ] Single-instance guard
- [ ] Basic scheduling (fixed interval with max ceiling)
- [ ] Provider selection chain (OpenAI OAuth → Anthropic OAuth → pay-per-token opt-in → disabled)
- [ ] Configuration (`[ambient]` section in config)
- [ ] Storage layout
### Phase 2: Memory Consolidation — Garden
- [ ] Full graph-wide dedup scan
- [ ] Fact verification against codebase
- [ ] Retroactive session extraction (crashed/missed sessions)
- [ ] Pruning dead memories (low confidence + low strength)
- [ ] Relationship discovery across sessions
- [ ] Embedding backfill
- [ ] Contradiction resolution
### Phase 3: Scheduling
- [ ] `schedule_ambient` tool for agent self-scheduling
- [ ] Scheduled queue (persistent, with context)
- [ ] Adaptive resource calculator
- [ ] Usage history tracking
- [ ] Rate limit awareness (from provider response headers)
- [ ] Event triggers (session close, crash, git push)
- [ ] Active session detection → pause/throttle
### Phase 4: Proactive Work
- [ ] Scout: analyze recent sessions + git history
- [ ] Infer user priorities from memories
- [ ] Identify actionable work
- [ ] Execute on separate branch
- [ ] Report results
### Phase 5: Info Widget
- [ ] Ambient status display in TUI
- [ ] Queue preview
- [ ] Last cycle summary
- [ ] Next wake estimate
- [ ] Budget bar (user vs ambient vs remaining)
---
*Last updated: 2026-02-08*
docs/AUTH_CREDENTIAL_SOURCES.md
+127
View File
@@ -0,0 +1,127 @@
# Auth Credential Sources (single source of truth)
This document exists because the same confusion keeps recurring: an agent (or a
human) greps for `ANTHROPIC_API_KEY` / `sk-ant-api`, finds nothing, reads a stale
`auth-validation.json` entry that says "expired", and wrongly concludes "there is
no working credential". In reality the credential is present and working; it just
lives somewhere the naive search did not look.
If you are debugging "does provider X have a credential?", read this first.
## The two "dual-auth" providers: Anthropic and OpenAI
Anthropic/Claude and OpenAI each support **two completely independent credential
paths**, surfaced as **two separate login providers**:
| Concept | Login provider id | Auth kind | Where the credential lives |
|--------------------|-------------------|-----------|-------------------------------------------------------------|
| Claude, OAuth/sub | `claude` | OAuth | `~/.jcode/auth.json` → `anthropic_accounts[].access` (`sk-ant-oat...`) |
| Claude, API key | `anthropic-api` | API key | `ANTHROPIC_API_KEY` env **or** `~/.config/jcode/anthropic.env` |
| OpenAI, OAuth | `openai` | OAuth | `~/.jcode/openai-auth.json` (Codex/ChatGPT login) |
| OpenAI, API key | `openai-api` | API key | `OPENAI_API_KEY` env **or** `~/.config/jcode/openai.env` |
Key facts that trip people up:
- The **OAuth token is not an API key.** Anthropic OAuth tokens are `sk-ant-oat01-...`
(and refresh tokens `sk-ant-ort01-...`). A direct API key is `sk-ant-api03-...`.
Grepping for `sk-ant-api` will miss an OAuth-only setup, and vice versa.
- The **API key is usually in the app config dir, not an env var.** The canonical
store is `~/.config/jcode/anthropic.env` (XDG `$XDG_CONFIG_HOME/jcode/anthropic.env`),
written by `jcode login --provider anthropic-api`. `printenv ANTHROPIC_API_KEY`
returning nothing does **not** mean there is no key.
- `~/.jcode/auth.json` holds **only OAuth accounts**, never the API key.
- `claude` and `anthropic-api` are **different providers** with different
availability. Having a Claude subscription login (OAuth) does **not** make
`anthropic-api` usable, and vice versa.
### How to actually check (don't guess)
```sh
# The honest, normalized answer for every provider:
jcode auth status --json
```
Each provider entry reports `status`, `auth_kind` ("OAuth" vs "API key"),
`credential_source` (env var / app config file / jcode-managed file), and the
exact `method`. This is the canonical surface; prefer it over grepping files.
Programmatically, the single source of truth is
`AuthStatus::assessment_for_provider(descriptor)` in
`crates/jcode-base/src/auth/mod.rs`, which returns a `ProviderAuthAssessment`.
## Selecting a default via config
`~/.jcode/config.toml`:
```toml
[provider]
default_provider = "claude" # Claude subscription (OAuth)
# default_provider = "anthropic-api" # Claude via direct Anthropic API key
default_model = "claude-opus-4-8"
anthropic_reasoning_effort = "xhigh"
```
- `default_provider = "claude"` uses the OAuth/subscription credential.
- `default_provider = "anthropic-api"` uses the direct API key. In this mode the
runtime **does not** fall back to OAuth: if no API key is configured the request
fails. Make sure `~/.config/jcode/anthropic.env` (or `ANTHROPIC_API_KEY`) exists.
The full alias/vocabulary mapping (runtime env, route stable-id, CLI `--provider`,
model prefix) is centralized in
`crates/jcode-provider-core/src/auth_mode.rs` (`AuthRoute`). Do not re-parse these
strings by hand; go through `AuthRoute`.
## Why "expired" was misleading: validation cache is not live state
`~/.jcode/auth-validation.json` caches the result of the **last** runtime
auth-test per provider. It is a historical record, not the current credential
state. An OAuth token that has since auto-refreshed can still show a days-old
"validation failed / expired" entry here.
To avoid presenting stale records as current fact, `format_record_label`
(`crates/jcode-base/src/auth/validation.rs`) flags any record older than
`doctor::VALIDATION_STALE_AFTER_MS` (7 days) as `stale, ... re-validate`. Treat a
stale record as "unknown, re-check", never as ground truth. Re-validate with:
```sh
jcode auth-test --provider <id>
```
## Quick decision tree for "is provider X authenticated?"
1. Run `jcode auth status --json` and read the entry for the **specific** login
provider id (`claude` vs `anthropic-api` are different!).
2. If you must inspect files: OAuth → `~/.jcode/auth.json` (and external imports);
API key → `ANTHROPIC_API_KEY` env or `~/.config/jcode/<provider>.env`.
3. Ignore `auth-validation.json` verdicts older than 7 days (shown as `stale`);
re-run `jcode auth-test` instead.
## Importing credentials from other agent tools
On a fresh install jcode can **reuse logins left behind by other coding
agents**, both OAuth tokens and API keys. Detection is consent-gated: jcode
lists the sources it found and only reads them after you approve each one
(`crates/jcode-base/src/auth/external.rs`, `unconsented_sources` /
`trust_external_auth_source`). Nothing is copied into jcode's own stores; the
external file is read in place.
Shared `auth.json`-style sources (`ExternalAuthSource`):
| Tool | Auth file path | On-disk shape |
|-----------|------------------------------------|------------------------------------------------------------------------------|
| OpenCode | `~/.local/share/opencode/auth.json`| flat `{ provider: { type: "oauth", access, refresh, expires } \| { type: "api", key } }` |
| pi | `~/.pi/agent/auth.json` | flat `{ provider: { type: "oauth", ... } \| { type: "api_key", key } }` (key may be `$ENV` ref) |
| OpenClaw | `~/.openclaw/agent/auth.json`, `~/.openclaw/agents/<id>/agent/auth-profiles.json`, `~/.openclaw/agents/<id>/agent/auth.json`, `~/.openclaw/credentials/oauth.json` | legacy flat pi shape, or the current `{ "profiles": { "<provider>:<name>": ... } }` store (first existing path wins; `main` agent and `:default` profiles preferred) |
| Hermes | `~/.hermes/auth.json` | nested `{ credential_pool: { provider: [ { auth_type, access_token, refresh_token, expires_at_ms } ] }, providers: {...} }` |
Notes:
- pi/OpenClaw API-key values that are `$ENV_VAR` references are resolved against
the environment; values that begin with `!` (shell commands) are **never
executed** and are skipped.
- Hermes stores literal API keys in the `access_token` field of `api_key`
credential-pool entries; many of its providers store only env-var *names*, so
those import nothing unless the env var is set.
- Other tool-specific importers exist for Claude Code, Codex, Gemini CLI,
GitHub Copilot, and Cursor (see `auth/claude.rs`, `auth/codex.rs`,
`auth/gemini.rs`, `auth/copilot.rs`, `auth/cursor.rs`).
docs/AWS_BEDROCK_PROVIDER.md
+165
View File
@@ -0,0 +1,165 @@
# AWS Bedrock provider
Jcode supports a native AWS Bedrock provider that talks directly to Bedrock Runtime with the AWS Rust SDK and `ConverseStream`.
## Configure credentials
Jcode supports two Bedrock auth styles:
- **Bedrock API key / bearer token**: easiest for local onboarding. Jcode stores the token in its config env file and sends it through the AWS SDK as `AWS_BEARER_TOKEN_BEDROCK`.
- **AWS IAM credentials**: best for normal AWS customer environments. This can be an AWS CLI/SSO profile, environment access keys, web identity, EC2/ECS metadata credentials, or another standard AWS SDK credential source.
For the guided API-key flow, run:
```bash
jcode login --provider bedrock
```
This saves `AWS_BEARER_TOKEN_BEDROCK` and `JCODE_BEDROCK_REGION` to `~/.config/jcode/bedrock.env`.
You can also configure manually:
```bash
export AWS_BEARER_TOKEN_BEDROCK=your-bedrock-api-key
export AWS_REGION=us-east-1
```
For AWS CLI/IAM/SSO credentials:
```bash
export AWS_PROFILE=my-profile
export AWS_REGION=us-east-1
# Optional Jcode-specific overrides:
export JCODE_BEDROCK_PROFILE=my-profile
export JCODE_BEDROCK_REGION=us-east-1
```
If you rely on instance/container metadata credentials and have no local profile env vars, opt in explicitly:
```bash
export JCODE_BEDROCK_ENABLE=1
export AWS_REGION=us-east-1
```
For AWS SSO profiles, run:
```bash
aws sso login --profile my-profile
```
For AWS CLI console-login profiles, Jcode can also use credentials exported by:
```bash
aws configure export-credentials --profile my-profile --format env-no-export
```
Jcode does not store these exported session credentials; it asks the AWS CLI profile provider when the Bedrock provider initializes.
## IAM permissions
The runtime path needs, at minimum:
```json
{
"Effect": "Allow",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"Resource": "*"
}
```
Model discovery additionally uses:
```json
{
"Effect": "Allow",
"Action": [
"bedrock:ListFoundationModels",
"bedrock:ListInferenceProfiles"
],
"Resource": "*"
}
```
If you enable STS validation with `JCODE_BEDROCK_VALIDATE_STS=1`, allow `sts:GetCallerIdentity`.
## Run Jcode with Bedrock
```bash
jcode --provider bedrock --model anthropic.claude-3-5-sonnet-20241022-v2:0
```
or:
```bash
jcode --model bedrock:anthropic.claude-3-5-sonnet-20241022-v2:0
```
Inference profile IDs/ARNs are accepted as model IDs, for example:
```bash
jcode --model bedrock:us.anthropic.claude-3-5-sonnet-20241022-v2:0
```
Recommended active profile-style choices, when your account has access, include:
```text
us.amazon.nova-2-lite-v1:0
us.amazon.nova-lite-v1:0
us.amazon.nova-micro-v1:0
us.anthropic.claude-sonnet-4-6
us.anthropic.claude-haiku-4-5-20251001-v1:0
us.deepseek.r1-v1:0
```
Prefer the region/profile ID such as `us.amazon.nova-2-lite-v1:0` when both a foundation model ID and a profile ID appear. Some Bedrock models do not support on-demand invocation and must be invoked through an inference profile.
## Model picker UX
`/model` keeps Bedrock rows compact:
- `×` means the route is not selectable. Select the row to see the full reason, such as legacy model access or missing credentials.
- `⚠` means the route is selectable but limited, most commonly no tool-use support.
- A selected inference-profile route shows which foundation model it targets.
If the model list looks stale after enabling model access, changing region, or refreshing credentials, run:
```text
/refresh-model-list
```
This forces `ListFoundationModels` and `ListInferenceProfiles`, updates cached legacy/profile metadata, and removes stale duplicate foundation rows when a usable inference profile route is available.
## Optional request parameters
```bash
export JCODE_BEDROCK_MAX_TOKENS=4096
export JCODE_BEDROCK_TEMPERATURE=0.2
export JCODE_BEDROCK_TOP_P=0.9
export JCODE_BEDROCK_STOP_SEQUENCES='</done>,STOP'
```
## Model discovery
Jcode will use a static Bedrock model list immediately. When model prefetch/catalog refresh runs, it calls `ListFoundationModels` and `ListInferenceProfiles`, then caches results in Jcode's config directory. Cached Bedrock catalogs are region-scoped; if you switch `JCODE_BEDROCK_REGION`/`AWS_REGION`, Jcode ignores the old-region cache and refreshes for the new region.
## Live smoke test
The live test is ignored by default. Run it only with valid AWS credentials and enabled model access:
```bash
JCODE_BEDROCK_LIVE_TEST=1 \
AWS_PROFILE=my-profile \
AWS_REGION=us-east-1 \
cargo test -p jcode --lib provider::bedrock::tests::bedrock_live_smoke_test -- --ignored
```
## Troubleshooting
- `AccessDenied`: grant Bedrock invoke/list permissions and enable model access in the AWS Console.
- `model not found` or validation errors: verify model ID/inference profile and region support.
- SSO token errors: run `aws sso login --profile <profile>`.
- API key auth: set `AWS_BEARER_TOKEN_BEDROCK` and `AWS_REGION`.
- Missing region: set `AWS_REGION` or `JCODE_BEDROCK_REGION`.
docs/BROWSER_PROVIDER_PROTOCOL.md
+817
View File
@@ -0,0 +1,817 @@
# Browser Provider Protocol
Status: draft
Owner: jcode
Audience: jcode core, browser bridge authors, adapter authors
## Why this exists
jcode should expose a single first-class `browser` tool while remaining compatible with multiple browser automation backends:
- Firefox Agent Bridge
- Chrome Agent Bridge
- Chrome remote debugging / CDP adapters
- WebDriver / BiDi adapters
- Safari automation adapters
- other third-party browser control systems
The protocol in this document defines the **normalized contract** between jcode and a browser provider.
This is intentionally **not** a demand that every bridge speak exactly the same native command language. Instead:
- jcode defines a **core semantic layer** it can rely on
- providers declare the capabilities and commands they support
- providers may expose **provider-specific commands** beyond the core
- adapters can translate a provider's native model into this protocol
That gives us both consistency and room for bridge-specific power features.
---
## Design goals
1. **One first-class tool in jcode**
- The model should use a single `browser` tool.
2. **Multiple provider implementations**
- Firefox, Chrome, Safari, Edge, WebDriver, and other systems should fit.
3. **Capability negotiation**
- jcode should know what each provider can and cannot do.
4. **Extensibility without fragmentation**
- We need a standard core, but providers must have room for browser-specific features.
5. **Stable session and element references**
- The model should be able to snapshot a page, then act on returned references.
6. **Transport-neutral semantics**
- The semantic protocol should be the same whether the provider is in-process, over stdio, over a socket, or wrapped through another adapter.
---
## Non-goals
1. Standardizing every low-level browser primitive.
2. Forcing all providers to support deep DOM, network, or JS introspection.
3. Requiring all providers to attach to the user's existing browser profile.
4. Making provider-specific commands part of the required core.
---
## Terminology
- **browser tool**: the user/model-facing jcode tool.
- **provider**: a backend implementation that satisfies this protocol.
- **bridge**: an external browser integration such as Firefox Agent Bridge.
- **adapter**: glue code that translates a bridge's native API into this protocol.
- **browser session**: the provider's isolated session or attachment scope for a jcode session.
- **page**: a tab, target, or browsing surface under a session.
- **element ref**: an opaque provider-issued handle for an actionable element.
---
## Conformance model
Providers do not need to implement everything.
### Core required for certification
A provider should support these normalized operations to be considered `certified`:
- `provider.describe`
- `provider.status`
- `session.ensure`
- `session.close`
- `page.open`
- `page.snapshot`
- `page.click`
- `page.type`
- `page.wait`
- `page.screenshot`
### Optional but recommended
- `page.go_back`
- `page.go_forward`
- `page.reload`
- `tab.list`
- `tab.activate`
- `tab.close`
- `page.eval`
- `page.press`
- `page.scroll`
- `page.select`
- `download.list`
### Provider-specific extensions
Providers may expose additional commands such as:
- `firefox.install_extension`
- `chrome.attach_debug_target`
- `cdp.send`
- `webdriver.perform_actions`
These are allowed, but they are not part of the required core.
---
## Transport model
This protocol defines **message semantics**, not one required wire format.
Supported implementation styles may include:
- direct Rust trait calls inside jcode
- stdio JSON request/response
- local socket RPC
- wrapped remote API
For external-process integrations, the recommended envelope is a JSON-RPC-like shape.
---
## Message envelope
For external providers, requests and responses should use a stable envelope.
### Request
```json
{
"protocol_version": "0.1",
"id": "req_123",
"method": "page.open",
"params": {
"session_id": "sess_abc",
"url": "https://example.com"
}
}
```
### Success response
```json
{
"protocol_version": "0.1",
"id": "req_123",
"ok": true,
"result": {
"page_id": "page_1",
"url": "https://example.com",
"title": "Example Domain"
},
"warnings": []
}
```
### Error response
```json
{
"protocol_version": "0.1",
"id": "req_123",
"ok": false,
"error": {
"code": "unsupported_method",
"message": "This provider does not implement page.eval",
"retryable": false,
"details": {}
}
}
```
### Event envelope
If a provider emits async events, use:
```json
{
"protocol_version": "0.1",
"event": "page.navigated",
"payload": {
"session_id": "sess_abc",
"page_id": "page_1",
"url": "https://example.com/next"
}
}
```
Events are optional in v1.
---
## Discovery and handshake
### `provider.describe`
Returns static and semi-static metadata about the provider.
Example:
```json
{
"provider_id": "firefox_agent_bridge",
"provider_label": "Firefox Agent Bridge",
"provider_version": "1.2.3",
"protocol_version": "0.1",
"browser_families": ["firefox"],
"transport": "stdio-json",
"certification_tier": "candidate",
"capabilities": {
"core_methods": [
"session.ensure",
"session.close",
"page.open",
"page.snapshot",
"page.click",
"page.type",
"page.wait",
"page.screenshot"
],
"optional_methods": [
"tab.list",
"tab.activate",
"page.eval"
],
"features": [
"element_refs",
"a11y_snapshot",
"attach_existing_browser",
"persistent_profile"
],
"custom_methods": [
{
"name": "firefox.install_extension",
"stability": "experimental",
"description": "Install or verify the Firefox extension"
}
]
}
}
```
### `provider.status`
Returns current availability and setup state.
Example fields:
```json
{
"availability": "ready",
"browser_detected": true,
"browser_running": true,
"setup_state": "complete",
"requires_manual_setup": false,
"recommended_browser": "firefox",
"manual_steps": [],
"diagnostics": [
{
"level": "info",
"code": "native_host_detected",
"message": "Native host manifest found"
}
]
}
```
Suggested enums:
- `availability`: `ready | degraded | unavailable`
- `setup_state`: `complete | partial | required | broken`
---
## Session model
jcode should not care whether a provider uses tabs, contexts, profiles, or remote targets internally.
It only needs a stable handle it can reuse.
### `session.ensure`
Creates or reuses a browser session for a jcode session.
Request:
```json
{
"client_session_id": "jcode_session_123",
"browser_preference": "auto",
"isolation": "per_jcode_session",
"attach": "prefer",
"persist": true,
"metadata": {
"owner": "agent",
"purpose": "browser_tool"
}
}
```
Response:
```json
{
"session_id": "browser_sess_1",
"browser_family": "firefox",
"browser_label": "Firefox",
"attached_to_existing_browser": true,
"isolation": "per_jcode_session",
"default_page_id": "page_1"
}
```
### `session.close`
Closes or detaches the provider session.
Providers may choose whether this closes tabs, detaches from a target, or merely releases provider-side state. The behavior should be documented in `provider.describe` or `provider.status` diagnostics.
---
## Resource identifiers
All resource identifiers are opaque strings.
Examples:
- `session_id`
- `page_id`
- `tab_id`
- `element_ref`
- `download_id`
jcode must not assume identifier shape or encode browser semantics into them.
---
## Normalized core methods
These are the semantics jcode can rely on.
### `page.open`
Open a URL in the current page or a new page.
Request fields:
- `session_id` required
- `url` required
- `page_id` optional
- `new_page` optional
- `foreground` optional
- `wait_until` optional: `load | domcontentloaded | networkidle | provider_default`
Response fields:
- `page_id`
- `url`
- `title` optional
- `navigation_state` optional
### `page.snapshot`
Return a normalized view of the current page for agent reasoning.
This is the most important method for model use.
Request fields:
- `session_id` required
- `page_id` optional
- `include_screenshot` optional
- `include_html` optional
- `include_dom` optional
- `include_a11y` optional
- `include_text` optional
- `max_nodes` optional
Response fields:
- `page_id`
- `url`
- `title`
- `snapshot`
- `elements`
- `text`
- `screenshot_ref` optional
- `provider_data` optional
#### Snapshot shape
Providers may use different internal representations, but `page.snapshot` should normalize into a common minimum format:
```json
{
"snapshot": {
"format": "jcode.page_snapshot.v1",
"root": {
"node_id": "n1",
"role": "document",
"name": "Example Domain",
"children": ["n2", "n3"]
},
"nodes": [
{
"node_id": "n2",
"role": "heading",
"name": "Example Domain",
"text": "Example Domain",
"element_ref": "el_1",
"actionable": false
},
{
"node_id": "n3",
"role": "link",
"name": "More information...",
"text": "More information...",
"element_ref": "el_2",
"actionable": true
}
]
}
}
```
#### Element list
For agent convenience, providers should also return a flattened actionable list when possible:
```json
{
"elements": [
{
"element_ref": "el_2",
"role": "link",
"name": "More information...",
"text": "More information...",
"actionable": true,
"enabled": true,
"selector_hint": "a"
}
]
}
```
A provider that cannot produce rich DOM/a11y data may still return a weaker snapshot, but it should say so in capabilities.
### `page.click`
Click an element.
Request should support multiple targeting modes:
- `element_ref`
- `selector`
- `text_query`
- `position`
At least one must be provided.
Response fields:
- `page_id`
- `clicked` boolean
- `navigation_occurred` optional
- `url` optional
Providers should prefer `element_ref` when available.
### `page.type`
Type or set text into an input-like target.
Request fields:
- `element_ref` optional
- `selector` optional
- `text` required
- `replace` optional
- `submit` optional
Response fields:
- `page_id`
- `typed` boolean
### `page.wait`
Wait for a condition.
Request fields may include:
- `text_present`
- `text_absent`
- `selector_present`
- `selector_absent`
- `element_ref_present`
- `url_matches`
- `navigation_complete`
- `timeout_ms`
Response fields:
- `satisfied` boolean
- `matched_condition` optional
- `url` optional
### `page.screenshot`
Capture a screenshot.
Request fields:
- `session_id`
- `page_id` optional
- `full_page` optional
- `clip` optional
- `element_ref` optional
Response fields:
- `page_id`
- `image` or `image_ref`
- `media_type`
- `width`
- `height`
Providers may return inline base64 data or a provider-managed image reference depending on transport constraints.
---
## Optional normalized methods
These methods are standardized when present, but not required for certification in the first pass.
### Navigation
- `page.go_back`
- `page.go_forward`
- `page.reload`
### Keyboard and form interaction
- `page.press`
- `page.select`
- `page.hover`
- `page.scroll`
### Tabs and pages
- `tab.list`
- `tab.activate`
- `tab.close`
- `tab.new`
### Introspection and debugging
- `page.eval`
- `network.list`
- `console.list`
- `storage.get`
- `cookie.list`
### Files and downloads
- `download.list`
- `download.wait`
- `upload.set_files`
---
## Extensibility model
This is the key part that allows leeway for provider-specific commands.
### Rule 1: providers may expose custom methods
Custom methods should use a namespaced method name, for example:
- `firefox.install_extension`
- `chrome.attach_debug_target`
- `cdp.send`
- `webdriver.actions`
### Rule 2: providers must advertise custom methods
Every custom method should appear in `provider.describe.capabilities.custom_methods` with:
- `name`
- `description`
- `stability`: `stable | experimental | deprecated`
- optional `input_schema`
- optional `output_schema`
### Rule 3: jcode core should only rely on normalized methods by default
The main `browser` tool should prefer the standard core and optional normalized methods.
Provider-specific methods should only be used when:
- the user explicitly asks for them
- a jcode-side adapter knows how to use them safely
- or a future advanced/debug mode is enabled
### Rule 4: provider-native passthrough is allowed, but should be explicit
If we want an escape hatch, the browser tool can support something like:
```json
{
"action": "provider_command",
"provider_method": "cdp.send",
"params": {
"method": "Network.enable"
}
}
```
This should be considered advanced/debug behavior, not the primary UX.
---
## Capability schema
Providers should report both methods and higher-level features.
### Methods
Concrete callable operations:
- `page.open`
- `page.snapshot`
- `tab.list`
### Features
Semantics or qualities that influence jcode behavior:
- `element_refs`
- `a11y_snapshot`
- `dom_snapshot`
- `html_snapshot`
- `full_page_screenshot`
- `attach_existing_browser`
- `persistent_profile`
- `isolated_contexts`
- `js_eval`
- `network_observe`
- `console_observe`
- `file_upload`
- `download_observe`
- `manual_setup_required`
- `extension_required`
- `remote_debugging_required`
### Stability
Each feature or method may optionally include a stability tag:
- `stable`
- `experimental`
- `deprecated`
---
## Setup and diagnostics
A browser provider often requires manual setup. The protocol should make that machine-readable.
### Diagnostic item
```json
{
"level": "warning",
"code": "extension_missing",
"message": "Firefox extension is not installed",
"manual_steps": [
"Open Firefox",
"Install the extension from /path/to/bridge.xpi",
"Restart Firefox if prompted"
]
}
```
### Recommended setup-oriented methods
- `provider.status`
- `provider.setup_guide` optional
- `provider.verify` optional
`provider.setup_guide` may return browser-specific instructions, URLs, file paths, permissions, or troubleshooting steps.
---
## Error model
Standard error codes should include:
- `unsupported_method`
- `unsupported_target`
- `invalid_request`
- `invalid_selector`
- `element_not_found`
- `element_not_actionable`
- `navigation_timeout`
- `not_ready`
- `setup_required`
- `permission_denied`
- `browser_not_running`
- `session_not_found`
- `page_not_found`
- `internal_error`
Providers may add provider-specific detail codes in `error.details`.
---
## Versioning
The protocol should be versioned independently from provider versions.
### Rules
- `protocol_version` identifies the semantic protocol version.
- Providers should declare the protocol version they implement.
- Minor additive changes should not break existing certified providers.
- Breaking changes require a new protocol version.
For now use:
- `protocol_version = "0.1"`
---
## Certification guidance
A provider can be classified as:
### Certified
- passes conformance tests for required core methods
- returns stable identifiers and normalized results
- reports setup/diagnostics correctly
- behaves predictably across repeated runs
### Compatible
- supports some or most normalized methods
- may have missing features or partial behavior
- useful, but not yet fully certified
### Experimental
- adapter exists, but semantics are incomplete or unstable
---
## Minimal conformance scenarios
A future conformance suite should verify at least:
1. `provider.describe` succeeds
2. `provider.status` reports a coherent state
3. `session.ensure` creates or reuses a session
4. `page.open` navigates to a test page
5. `page.snapshot` returns usable text and at least one actionable reference when applicable
6. `page.click` can activate a known element
7. `page.type` can fill a known input
8. `page.wait` observes a deterministic page change
9. `page.screenshot` returns an image
10. `session.close` cleans up or detaches cleanly
---
## Recommended jcode integration policy
The jcode `browser` tool should:
1. prefer normalized core methods
2. choose a provider based on user preference, availability, and capability quality
3. expose provider-specific methods only behind an explicit advanced path
4. return setup guidance when no ready provider is available
5. avoid baking Firefox-specific or Chrome-specific assumptions into the core tool API
---
## Open questions
These are intentionally left open for the next iteration.
1. Should screenshots always be inline, or can providers return file/image handles?
2. Should event streaming be required for advanced integrations?
3. How much of raw HTML/DOM should be normalized versus returned as provider data?
4. Should `page.snapshot` support multiple named formats beyond `jcode.page_snapshot.v1`?
5. Should provider-specific methods be invokable through the same `browser` tool or only via debug mode?
6. Should setup/install flows themselves be standardized beyond status and diagnostics?
---
## Proposed next steps
1. Review this document and tighten the core method set.
2. Decide the exact normalized `page.snapshot` format.
3. Define a Rust trait matching this protocol.
4. Implement the first provider adapter for Firefox Agent Bridge.
5. Build a conformance test harness.
6. Add README browser setup and compatibility documentation after the protocol stabilizes.
docs/CLIENT_CORE_PRESENTATION_SPLIT_PLAN.md
+877
View File
@@ -0,0 +1,877 @@
# Client-Core vs Presentation Split Plan
Status: Proposed
This document audits the current TUI/client stack and proposes a safe, incremental split between a reusable `client-core` layer and the ratatui/crossterm presentation layer.
The goal is to make the current single-surface client easier to maintain, while also unblocking the multi-surface direction described in [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md).
See also:
- [`REFACTORING.md`](./REFACTORING.md)
- [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md)
- [`SERVER_ARCHITECTURE.md`](./SERVER_ARCHITECTURE.md)
## Executive Summary
Today the client stack is functionally split, but not structurally split:
- `src/tui/app.rs` owns a very large `App` state object with session state, transport state, input state, transient UI state, and runtime handles mixed together.
- `src/tui/app/*.rs` acts like a distributed reducer, but mutation is expressed as direct `impl App` methods instead of typed actions and reducer entrypoints.
- `src/tui/ui.rs` and `src/tui/ui_*.rs` are already mostly presentation-only, but they still depend on a very wide `TuiState` trait and a few process-global render caches.
- `src/tui/workspace_client.rs` is process-global mutable state, which is the clearest current blocker for a true client-core split and for multi-surface clients.
The safest plan is:
1. Define a real `client-core` state model inside the existing crate first.
2. Move pure state and reducers behind that boundary without changing behavior.
3. Keep ratatui rendering, overlays, markdown, mermaid, and render caches in presentation.
4. Only after the boundary is clean, consider moving `client-core` into its own crate.
## Current Stack Audit
## Entry points and loops
Current runtime entrypoints:
- `src/cli/tui_launch.rs`
- boots terminal runtime
- constructs `tui::App`
- restores session/startup hints
- calls `app.run(...)`
- `src/tui/app/run_shell.rs`
- local loop: `App::run`
- remote loop: `App::run_remote`
- replay loop helpers
- `src/tui/app/local.rs`
- local tick handling
- terminal event handling
- bus event handling
- finish-turn bookkeeping
- `src/tui/app/remote.rs`
- remote tick and terminal event handling
- reconnect and disconnected handling
- `src/tui/app/remote/reconnect.rs`
- connect/reconnect orchestration
- `src/tui/app/remote/input_dispatch.rs`
- remote send/split submission path
- `src/tui/app/remote/server_events.rs`
- main remote event reducer today
Rendering entrypoints:
- `src/tui/mod.rs`
- `render_frame(frame, state)`
- `src/tui/ui.rs`
- `draw(frame, app: &dyn TuiState)`
- `draw_inner(...)`
- `src/tui/ui_prepare.rs`, `ui_viewport.rs`, `ui_messages.rs`, `ui_input.rs`, `ui_pinned.rs`, `ui_overlays.rs`, `ui_header.rs`, `ui_diagram_pane.rs`
- frame preparation and rendering
## Current state root
Primary root:
- `src/tui/app.rs`
- `pub struct App`
- `DisplayMessage`
- `ProcessingStatus`
- several transport and pending-operation helper structs
`App` currently mixes all of these concerns:
- runtime handles
- `provider`, `registry`, `skills`, `mcp_manager`, debug channel
- conversation/session data
- `messages`, `session`, `display_messages`, tool-output tracking
- composer/input state
- `input`, `cursor_pos`, pasted content, pending images, queueing
- turn execution state
- `is_processing`, `status`, `processing_started`, pending turn flags
- streaming state
- `streaming_text`, stream buffer, thinking state, token usage, TPS tracking
- remote client/session state
- remote provider hints, session ids, server metadata, reconnect/startup state, split launch state
- workspace state
- currently not in `App`, but in global `workspace_client.rs`
- surface-local UI state
- scroll offsets, copy selection, diagram pane focus/scroll, diff pane state, inline picker state, overlays, status notices
- config and feature toggles
- memory, swarm, diff mode, centered mode, diagram mode, auto-review, auto-judge
## Current mutation surface
Mutation is spread across many `impl App` files:
State helpers and pseudo-reducers:
- `src/tui/app/state_ui.rs`
- `src/tui/app/state_ui_runtime.rs`
- `src/tui/app/state_ui_messages.rs`
- `src/tui/app/state_ui_storage.rs`
- `src/tui/app/state_ui_input_helpers.rs`
- `src/tui/app/state_ui_maintenance.rs`
- `src/tui/app/conversation_state.rs`
Event and command handling:
- `src/tui/app/input.rs`
- `src/tui/app/turn.rs`
- `src/tui/app/local.rs`
- `src/tui/app/remote.rs`
- `src/tui/app/remote/input_dispatch.rs`
- `src/tui/app/remote/server_events.rs`
- `src/tui/app/remote/workspace.rs`
- `src/tui/app/navigation.rs`
- `src/tui/app/inline_interactive.rs`
- `src/tui/app/copy_selection.rs`
- `src/tui/app/model_context.rs`
- `src/tui/app/auth*.rs`
This is why the code already feels reducer-like, but is still tightly coupled. State transitions, runtime side effects, and redraw decisions are interleaved.
## Current presentation boundary
The renderer already has a partial boundary via `src/tui/mod.rs::TuiState`.
That boundary is promising, but still too wide because it currently includes:
- raw domain/session access
- surface state access
- auth/config lookups
- render-specific helpers such as `render_streaming_markdown`
- some expensive derived computations and caching behavior
- mutable behavior like `update_cost`
The result is that the trait is acting as a dump point rather than a narrow presentation model.
## Concrete pain points found in code
### 1. `App` is too large and semantically mixed
The state root in `src/tui/app.rs` is carrying:
- domain state
- surface/controller state
- transport state
- runtime handles
- presentation-adjacent state
This prevents reuse outside the current TUI runtime.
### 2. No typed action/reducer boundary
The main reducers are implicit:
- `local.rs::handle_tick`
- `local.rs::handle_terminal_event`
- `remote/server_events.rs::handle_server_event`
- `remote/input_dispatch.rs::*`
- `state_ui_messages.rs::*`
- `conversation_state.rs::*`
These should become named reducers over named state slices.
### 3. Workspace state is process-global
- `src/tui/workspace_client.rs`
- uses `static WORKSPACE_STATE: Mutex<Option<WorkspaceClientState>>`
This is incompatible with:
- multiple client instances in one process
- test isolation without global resets
- future multi-surface clients
- a clean client-core extraction
This state must become instance-owned.
### 4. Render layer still relies on globals
Examples in `src/tui/ui.rs`:
- `LAST_MAX_SCROLL`
- `PINNED_PANE_TOTAL_LINES`
- prompt viewport animation state
- visible copy targets
These are presentation concerns, but they should become renderer-instance state, not process-global state.
### 5. Runtime loops and rendering are tightly interwoven
`terminal.draw(|frame| crate::tui::ui::draw(frame, &self))` appears in many control-flow paths:
- `run_shell.rs`
- `turn.rs`
- `remote/reconnect.rs`
- `input.rs`
- `model_context.rs`
That makes controller extraction harder because redraw timing is coupled to mutation paths.
## Proposed Split
## Layer 1: `client-core`
Owns client behavior and state, but not ratatui rendering or terminal I/O.
Allowed in core:
- client/session/surface state
- reduction of user intents, server events, bus events, and ticks
- command parsing and routing decisions
- queueing and pending-operation state
- workspace model state
- feature toggles and mode state
- effects emitted for runtime adapters
Not allowed in core:
- `ratatui`
- `crossterm` event types
- direct terminal drawing
- process-global UI caches
- widget rendering
- mermaid/image/markdown rendering details
## Layer 2: presentation
Owns all ratatui and render-time concerns.
Includes:
- `src/tui/ui.rs` and `src/tui/ui_*.rs`
- `src/tui/info_widget*.rs`
- `src/tui/markdown*.rs`
- `src/tui/mermaid*.rs`
- `src/tui/session_picker*.rs`
- `src/tui/login_picker.rs`
- `src/tui/account_picker*.rs`
- `src/tui/usage_overlay.rs`
- `src/tui/visual_debug.rs`
Presentation should consume a narrow immutable snapshot or read-only trait from core.
## Proposed State Types
These types should exist before any crate split. Initially they can live in a new `src/client_core/` module inside the main crate.
### `ClientCoreState`
Top-level state for one client surface.
Suggested file:
- `src/client_core/state/mod.rs`
Suggested fields:
- `conversation: ConversationState`
- `composer: ComposerState`
- `turn: TurnState`
- `stream: StreamState`
- `remote: RemoteState`
- `workspace: WorkspaceState`
- `surface: SurfaceState`
- `features: FeatureState`
- `notices: NoticeState`
### `ConversationState`
Suggested files:
- `src/client_core/state/conversation.rs`
Move in from:
- `src/tui/app.rs`
- `src/tui/app/conversation_state.rs`
- `src/tui/app/state_ui_messages.rs`
Owns:
- `messages: Vec<Message>`
- `display_messages: Vec<DisplayMessage>`
- `display_messages_version: u64`
- tool output tracking
- `tool_call_ids`
- `tool_result_ids`
- `tool_output_scan_index`
- provider/session conversation hydration helpers
Reducer name:
- `conversation_reducer`
Primary responsibilities:
- append/replace/remove display messages
- replace provider transcript
- compact storage-friendly display messages
- maintain tool output tracking
### `ComposerState`
Suggested file:
- `src/client_core/state/composer.rs`
Move in from:
- `src/tui/app.rs`
- `src/tui/app/state_ui_input_helpers.rs`
- pure parts of `src/tui/app/input.rs`
Owns:
- `input`
- `cursor_pos`
- `pasted_contents`
- `pending_images`
- `queued_messages`
- `hidden_queued_system_messages`
- `interleave_message`
- `pending_soft_interrupts`
- `pending_soft_interrupt_requests`
- `stashed_input`
- `queue_mode`
- `submit_input_on_startup`
- route-next-prompt flags
Reducer names:
- `composer_reducer`
- `queue_reducer`
Primary responsibilities:
- text editing
- queueing/interleave behavior
- restore/save reload input decisions
- turning prepared input into a high-level send intent
### `TurnState`
Suggested file:
- `src/client_core/state/turn.rs`
Move in from:
- `src/tui/app.rs`
- `src/tui/app/local.rs`
- `src/tui/app/tui_lifecycle.rs`
- `src/tui/app/state_ui_maintenance.rs`
Owns:
- `is_processing`
- `status: ProcessingStatus`
- `processing_started`
- `pending_turn`
- `pending_queued_dispatch`
- `cancel_requested`
- `quit_pending`
- `pending_provider_failover`
- `session_save_pending`
- background maintenance state
- current-turn reminder state
Reducer names:
- `turn_reducer`
- `lifecycle_reducer`
- `maintenance_reducer`
Primary responsibilities:
- start/finish turn
- idle/sending/streaming/tool transitions
- failover countdown state
- maintenance banners/notices
### `StreamState`
Suggested file:
- `src/client_core/state/stream.rs`
Move in from:
- `src/tui/app.rs`
- `src/tui/app/remote/server_events.rs`
- `src/tui/app/misc_ui.rs`
Owns:
- `streaming_text`
- `stream_buffer`
- `streaming_tool_calls`
- token usage fields
- cache usage fields
- TPS tracking fields
- thinking/thought-line state
- `last_stream_activity`
- `subagent_status`
- `batch_progress`
Reducer names:
- `stream_reducer`
- `server_event_reducer`
Primary responsibilities:
- text delta/replace handling
- tool start/exec/done state
- token accounting
- thought-line handling
- stale activity tracking
### `RemoteState`
Suggested file:
- `src/client_core/state/remote.rs`
Move in from:
- `src/tui/app.rs`
- `src/tui/app/remote/input_dispatch.rs`
- `src/tui/app/remote/server_events.rs`
- `src/tui/app/remote/reconnect.rs`
- `src/tui/app/remote/queue_recovery.rs`
Owns:
- remote session identity and resume state
- provider/model/server metadata
- startup/reconnect phase
- split-launch state
- pending remote message state
- rate-limit retry state
- remote resume activity snapshot
- `current_message_id`
- server sessions / client count / swarm snapshots
Reducer names:
- `remote_reducer`
- `server_event_reducer`
- `remote_lifecycle_reducer`
Primary responsibilities:
- reduce `ServerEvent` into remote/session state
- own remote reconnect-visible state
- own split/new-session routing state
- own queue recovery bookkeeping
### `WorkspaceState`
Suggested file:
- `src/client_core/state/workspace.rs`
Move in from:
- `src/tui/workspace_client.rs`
Owns:
- `enabled`
- `map: WorkspaceMapModel`
- `imported_server_sessions`
- `pending_split_target`
- `pending_resume_session`
Reducer names:
- `workspace_reducer`
Primary responsibilities:
- enable/disable workspace mode
- import existing sessions
- update map after split/resume/history sync
- move focus left/right/up/down
Important rule:
- this state must become instance-owned, not global static state
### `SurfaceState`
Suggested file:
- `src/client_core/state/surface.rs`
Move in from:
- `src/tui/app.rs`
- `src/tui/app/navigation.rs`
- `src/tui/app/copy_selection.rs`
- `src/tui/app/inline_interactive.rs`
- selected non-render code from `src/tui/app/input.rs`
Owns:
- `scroll_offset`
- `auto_scroll_paused`
- copy selection state
- diff pane focus/scroll
- diagram focus/index/scroll/ratio state
- side-panel focus state
- inline interactive/view state
- help/changelog overlay visibility and scroll
- status notices
- mouse scroll animation queue
Reducer names:
- `surface_reducer`
- `navigation_reducer`
- `overlay_reducer`
Note:
This is still core, not presentation. It is surface-local controller state, not render cache state.
### `FeatureState`
Suggested file:
- `src/client_core/state/features.rs`
Move in from:
- `src/tui/app.rs`
- `src/tui/app/observe.rs`
- `src/tui/app/split_view.rs`
Owns:
- memory, swarm, autoreview, autojudge, improve mode
- diff mode
- centered mode
- diagram mode/pinning defaults
- observe mode
- split view mode
- image pinning and native scrollbar toggles
Reducer names:
- `feature_reducer`
### `NoticeState`
Suggested file:
- `src/client_core/state/notices.rs`
Owns:
- transient status notices
- rate-limit/reset countdown notices
- background task wake/status notices
- startup hints / restored reload notices
Reducer names:
- `notice_reducer`
## Proposed Effects Boundary
Reducers should not directly call terminal, remote socket, or persistence APIs.
Introduce:
- `src/client_core/effects.rs`
Suggested effect enum:
- `ClientEffect::SendRemoteMessage { ... }`
- `ClientEffect::ResumeRemoteSession { session_id }`
- `ClientEffect::LaunchRemoteSplit`
- `ClientEffect::PersistSession`
- `ClientEffect::PersistReloadInput`
- `ClientEffect::ExtractMemories`
- `ClientEffect::StartCompaction`
- `ClientEffect::RunInputShell { ... }`
- `ClientEffect::RequestQuit`
- `ClientEffect::RequestRedraw`
Runtime adapters in `src/tui/app/local.rs`, `remote.rs`, `remote/reconnect.rs`, and `run_shell.rs` should execute these effects.
## Presentation: What Stays Put
The following should remain presentation-owned for the first split:
### Core renderer
- `src/tui/ui.rs`
- `src/tui/ui_prepare.rs`
- `src/tui/ui_viewport.rs`
- `src/tui/ui_messages.rs`
- `src/tui/ui_input.rs`
- `src/tui/ui_pinned.rs`
- `src/tui/ui_overlays.rs`
- `src/tui/ui_header.rs`
- `src/tui/ui_diagram_pane.rs`
- `src/tui/ui_layout.rs`
- `src/tui/ui_status.rs`
- `src/tui/ui_theme.rs`
### Rendering helpers and caches
- `src/tui/markdown*.rs`
- `src/tui/mermaid*.rs`
- `src/tui/image.rs`
- `src/tui/visual_debug.rs`
- render cache structs in `ui.rs`, `ui_messages_cache.rs`, `ui_file_diff.rs`, `ui_pinned.rs`
### Widgets and overlays
- `src/tui/info_widget*.rs`
- `src/tui/session_picker*.rs`
- `src/tui/login_picker.rs`
- `src/tui/account_picker*.rs`
- `src/tui/usage_overlay.rs`
## Concrete File Mapping
### Files that should become core-first
| Current file | Target module | Notes |
| --- | --- | --- |
| `src/tui/app.rs` | `src/client_core/state/*` + thin `App` shell | Split the giant `App` root by concern |
| `src/tui/app/conversation_state.rs` | `src/client_core/state/conversation.rs` | Mostly state logic already |
| `src/tui/app/state_ui_messages.rs` | `src/client_core/reducer/conversation.rs` | Clean first reducer extraction |
| `src/tui/app/state_ui_input_helpers.rs` | `src/client_core/reducer/composer.rs` | Pure text-edit logic |
| `src/tui/app/state_ui.rs` | `src/client_core/reducer/lifecycle.rs` | Save/restore and client focus helpers |
| `src/tui/app/state_ui_maintenance.rs` | `src/client_core/reducer/maintenance.rs` | Notice/message state |
| `src/tui/app/remote/server_events.rs` | `src/client_core/reducer/server_event.rs` | Highest-value reducer split |
| `src/tui/app/remote/queue_recovery.rs` | `src/client_core/reducer/queue_recovery.rs` | Already isolated |
| `src/tui/app/remote/workspace.rs` | `src/client_core/reducer/workspace.rs` + runtime adapter | Split commands from transport calls |
| `src/tui/workspace_client.rs` | `src/client_core/state/workspace.rs` | Must stop being global |
| `src/tui/app/navigation.rs` | `src/client_core/reducer/navigation.rs` | Move non-ratatui navigation state |
| `src/tui/app/copy_selection.rs` | `src/client_core/reducer/copy_selection.rs` | Surface interaction state |
| `src/tui/app/inline_interactive.rs` | `src/client_core/reducer/inline_ui.rs` | State transitions, not drawing |
### Files that should remain presentation-first
| Current file | Keep in presentation because... |
| --- | --- |
| `src/tui/ui.rs` | main ratatui frame renderer |
| `src/tui/ui_prepare.rs` | render-time wrapping/caching/layout prep |
| `src/tui/ui_viewport.rs` | draw-time viewport calculations |
| `src/tui/ui_messages.rs` | ratatui message rendering |
| `src/tui/ui_input.rs` | input box drawing |
| `src/tui/ui_pinned.rs` | side-pane drawing and caches |
| `src/tui/info_widget*.rs` | widget composition and rendering |
| `src/tui/markdown*.rs` | rendering pipeline, not client behavior |
| `src/tui/mermaid*.rs` | rendering pipeline and image management |
| `src/tui/session_picker*.rs`, `login_picker.rs`, `account_picker*.rs`, `usage_overlay.rs` | widget state can remain presentation initially |
## Recommended Reducer API
Do not start with a single mega-reducer.
Start with slice reducers and one coordinator:
- `reduce_tick(state, now) -> Effects`
- `reduce_terminal_intent(state, intent) -> Effects`
- `reduce_server_event(state, event) -> Effects`
- `reduce_bus_event(state, event) -> Effects`
- `reduce_workspace_action(state, action) -> Effects`
Suggested types:
- `ClientIntent`
- normalized user intent, not raw crossterm keys
- `ExternalEvent`
- server event, bus event, tick, lifecycle event
- `ClientEffect`
- runtime work for adapters
This keeps crossterm and ratatui out of core.
## Proposed Extraction Order
## Phase 0: docs and naming
- Land this document.
- Freeze naming for the future core slices.
- Do not move code yet.
## Phase 1: introduce state slices inside the current crate
Create empty or lightly-populated modules:
- `src/client_core/mod.rs`
- `src/client_core/state/mod.rs`
- `src/client_core/state/conversation.rs`
- `src/client_core/state/composer.rs`
- `src/client_core/state/turn.rs`
- `src/client_core/state/stream.rs`
- `src/client_core/state/remote.rs`
- `src/client_core/state/workspace.rs`
- `src/client_core/state/surface.rs`
- `src/client_core/state/features.rs`
- `src/client_core/state/notices.rs`
Safe rule:
- move types first
- keep method bodies where they are until state compiles cleanly
## Phase 2: extract the easiest pure reducers
First extractions should be the least coupled files:
1. `state_ui_messages.rs`
2. `conversation_state.rs`
3. `state_ui_input_helpers.rs`
4. `remote/queue_recovery.rs`
5. `state_ui_maintenance.rs`
Why first:
- mostly state mutation
- low terminal/runtime coupling
- easy to cover with unit tests
## Phase 3: move workspace state into the app instance
This is the highest-leverage architectural fix.
Do this before large event-loop refactors:
1. replace `workspace_client.rs` global static state with `WorkspaceState` inside app/core
2. keep the same commands and behavior
3. adjust `remote/workspace.rs` to operate on instance-owned state
Why now:
- removes the clearest multi-surface blocker
- lowers future complexity for everything else
## Phase 4: extract remote event reduction
Split `src/tui/app/remote/server_events.rs` into:
- core reduction
- state transitions
- display-message mutations
- token and tool-call accounting
- status transitions
- runtime adapter
- `RemoteEventState` parsing glue
- redraw policy
- transport-specific buffering
This is the single most important reducer extraction after workspace state.
## Phase 5: extract normalized terminal intents
Do not put raw `crossterm::Event` into core.
Instead:
1. keep key decoding in `local.rs`, `remote.rs`, and `input.rs`
2. introduce normalized intents such as:
- `SubmitPrompt`
- `MoveCursorLeft`
- `ScrollChatUp`
- `OpenSessionPicker`
- `ToggleCopySelection`
- `NavigateWorkspace(Direction)`
3. reduce those intents in core
## Phase 6: narrow the renderer boundary
Replace the current wide `TuiState` dependency with either:
- a much narrower trait, or
- a `PresentationSnapshot` built from core state
Recommended direction:
- build a `PresentationSnapshot` from core + presentation-owned caches
This keeps expensive derived computations out of ad hoc trait methods.
## Phase 7: move runtime adapters behind effects
Once reducers return `ClientEffect`, update:
- `src/tui/app/local.rs`
- `src/tui/app/remote.rs`
- `src/tui/app/remote/reconnect.rs`
- `src/tui/app/run_shell.rs`
to become thin shells that:
- collect external events
- reduce them
- run returned effects
- schedule redraws
## Phase 8: optional crate split
Only after ratatui/crossterm have been removed from core APIs:
- create `crates/jcode-client-core`
- move `src/client_core/*` into the crate
- keep presentation in the main crate or a future `jcode-tui-presentation` crate
Do not start with the crate split. Start with the boundary.
## Testing Strategy For The Split
Each extraction phase should preserve the existing user-visible behavior.
Recommended checks:
- existing TUI tests under `src/tui/ui_tests` and `src/tui/app/tests.rs`
- focused reducer tests for new `client_core` slices
- workspace state tests after de-globalizing `workspace_client.rs`
- remote `ServerEvent` reduction tests using captured event sequences
## Recommended First PR Sequence
If this work starts immediately, the first sequence should be:
1. docs only
- this plan
2. type-only move
- introduce `client_core::state::workspace::WorkspaceState`
- no behavior change yet
3. safe behavioral move
- make workspace state instance-owned
4. reducer move
- extract `state_ui_messages.rs`
5. reducer move
- extract `remote/server_events.rs`
That order minimizes risk while unlocking the most important future architecture work.
## Non-Goals For The First Split
Do not try to do these in the first wave:
- rewriting the renderer
- deleting the `TuiState` abstraction immediately
- moving mermaid/markdown rendering into core
- redesigning all overlays/widgets at once
- introducing a giant Redux-style universal action enum from day one
- making independent and workspace modes separate apps
## Bottom Line
The split should be:
- `client-core` = instance-owned client state + reducers + effects
- presentation = ratatui widgets, layout, drawing, render caches, visual debug
The safest first extraction is not a rendering change. It is making workspace state instance-owned and then extracting the existing pseudo-reducers, starting with display-message and remote-event reduction.
docs/CODE_QUALITY_10_10_PLAN.md
+326
View File
@@ -0,0 +1,326 @@
# Code Quality 10/10 Plan
This document defines the quality target for jcode, the standards required to reach it, and the phased execution plan to get there without destabilizing the product.
## Goal
Raise jcode from its current state of roughly **7/10 overall code quality** to a sustained **9+/10 engineering standard**, with a practical target that feels like "10/10" in day-to-day development:
- clean builds
- clear module ownership
- small and maintainable files
- low-risk refactors
- strong tests
- predictable behavior under stress
- strict CI guardrails that prevent regressions
Because jcode is a fast-moving product, "10/10" does **not** mean "perfect". It means:
1. defects are easier to prevent than to introduce
2. contributors can quickly understand where code belongs
3. the repo resists architectural drift
4. risky areas are well-tested and observable
5. quality does not depend on memory or heroics
## Current Problems
The main issues observed in the codebase today are:
### 1. Oversized modules
Several files are dramatically larger than they should be for long-term maintainability. Major hotspots currently include:
- `src/provider/openai.rs`
- `src/provider/mod.rs`
- `src/agent.rs`
- `src/server.rs`
- `src/tui/ui.rs`
- `src/tui/info_widget.rs`
- `tests/e2e/main.rs`
These files are doing too much at once and create review, testing, and onboarding friction.
### 2. Warning and dead-code debt
The repository currently tolerates a significant warning budget instead of targeting warning-free builds. There are also multiple broad `allow(dead_code)` suppressions that hide drift.
### 3. Inconsistent strictness around failure paths
The codebase contains many `unwrap`, `expect`, `panic!`, `todo!`, and `unimplemented!` usages. Some are valid in tests, but production code should be more defensive and explicit.
### 4. Test concentration
There are many tests, which is good, but some test coverage is concentrated inside very large files and does not yet provide ideal fault isolation.
### 5. Guardrails are present but not yet strict enough
There is already useful quality infrastructure in the repository, but it should be tightened so quality improves automatically over time.
## Definition of Done for "10/10"
We will consider this program successful when the codebase reaches the following state:
### Build and lint quality
- `cargo check --all-targets --all-features` passes cleanly
- `cargo clippy --all-targets --all-features -- -D warnings` passes cleanly or is very close with narrow, justified exceptions
- `cargo fmt --all -- --check` passes
- warning count is near zero and actively ratcheted downward
### Structural quality
- no production file exceeds **1200 LOC** without a documented reason
- most production files are below **800 LOC**
- most functions stay below **100 LOC** unless complexity is clearly justified
- major domains have clear boundaries and ownership
### Reliability quality
- e2e tests are split by feature instead of concentrated in mega-files
- critical state transitions have targeted tests
- reload, streaming, tool execution, and swarm coordination have explicit failure-mode coverage
- long-running reliability checks exist for memory, socket lifecycle, and reconnect/reload behavior
### Safety quality
- production `unwrap` / `expect` usage is significantly reduced and justified where it remains
- broad `allow(dead_code)` suppressions are eliminated or reduced to narrow local allowances
- tool, shell, path, and credential boundaries are explicit and tested
### Contributor quality
- contributors can tell where code belongs
- refactor rules are documented
- CI makes regressions hard to merge
- architecture docs match reality
## Non-Negotiable Principles
1. **No big-bang rewrite.** Refactor incrementally.
2. **Behavior-preserving changes first.** Extract, move, split, and test before changing logic.
3. **Quality must be enforceable.** Prefer CI guardrails over informal expectations.
4. **Delete dead code aggressively.** Simpler code is higher-quality code.
5. **Keep the product shippable throughout the program.**
## Metrics to Track
These metrics should be checked repeatedly during the program:
- warning count
- clippy violations
- count of broad `allow(dead_code)` suppressions
- count of production `unwrap` / `expect`
- top 20 largest Rust files
- test runtime and flake rate
- startup time, memory, and reload reliability
## Phased Plan
## Phase 0: Prevent Further Decay
**Objective:** stop quality from getting worse.
Tasks:
- add stricter CI checks for clippy and all-target/all-feature builds
- ratchet warning policy downward
- document code quality standards and file-size goals
- establish a tracked todo list for the quality program
Success criteria:
- no new warnings merge unnoticed
- no new giant files are added casually
- contributors can see the roadmap and standards in-repo
## Phase 1: Warning and Dead-Code Burn-Down
**Objective:** restore signal quality in builds.
Tasks:
- remove unused variables, methods, and stale helpers
- replace broad `#![allow(dead_code)]` with narrow scoped allows where truly needed
- delete abandoned code paths
- reduce dead code in TUI, memory, and provider modules
Success criteria:
- warning count materially reduced
- dead-code suppression becomes the exception, not the default
## Phase 2: Decompose the Biggest Files
**Objective:** eliminate the primary maintainability hazard.
Priority order:
1. `tests/e2e/main.rs`
2. `src/server.rs`
3. `src/agent.rs`
4. `src/provider/mod.rs`
5. `src/provider/openai.rs`
6. `src/tui/ui.rs`
7. `src/tui/info_widget.rs`
Approach:
- extract pure helpers first
- extract types and state machines second
- extract domain-specific submodules third
- keep public interfaces stable during moves
Success criteria:
- each hotspot file becomes materially smaller
- functionality remains stable
- tests remain green during each split
## Phase 3: Strengthen Error Handling
**Objective:** make failure modes explicit and recoverable.
Tasks:
- reduce production `unwrap` / `expect`
- improve error context with `anyhow` / `thiserror`
- classify retryable vs user-facing vs internal invariant failures
- add tests for malformed streams, reconnects, and tool interruption paths
Success criteria:
- fewer panic-prone production paths
- clearer logs and more diagnosable failures
## Phase 4: Rebalance the Test Pyramid
**Objective:** make failures faster, narrower, and more actionable.
Tasks:
- split e2e suites by feature
- add more unit tests for parsing, protocol, and state transitions
- add snapshot or golden tests for stable render outputs
- add property tests for serialization, tool parsing, and patch/edit invariants
- improve test support utilities and isolation
Success criteria:
- lower test maintenance cost
- failures localize to one subsystem quickly
## Phase 5: Reliability and Performance Guardrails
**Objective:** keep architectural quality aligned with runtime quality.
Tasks:
- add or strengthen memory and stress checks
- add repeated reload / attach / detach reliability tests
- track startup and idle resource regressions
- improve structured diagnostics around reload, sockets, and provider streaming
Success criteria:
- regressions are caught before release
- long-running behavior is measurably stable
## Phase 6: Finish the Ratchet
**Objective:** make quality self-sustaining.
Tasks:
- move from warning budget to effectively warning-free builds
- enforce stricter clippy rules where practical
- document module ownership expectations
- review and refresh architecture docs after refactors land
Success criteria:
- repo quality remains high without special cleanup pushes
- the codebase resists drift by default
## Immediate Execution Order
The first concrete actions should be:
1. land this quality plan and a tracked todo list
2. tighten CI guardrails
3. begin warning/dead-code cleanup
4. split `tests/e2e/main.rs`
5. continue into `src/server.rs`
## Initial Target Refactors
### `tests/e2e/main.rs`
Split into:
- `tests/e2e/session_flow.rs`
- `tests/e2e/tool_execution.rs`
- `tests/e2e/reload.rs`
- `tests/e2e/swarm.rs`
- `tests/e2e/provider_behavior.rs`
- `tests/e2e/test_support/mod.rs`
### `src/server.rs`
Split further into:
- `src/server/state.rs`
- `src/server/bootstrap.rs`
- `src/server/socket.rs`
- `src/server/session_registry.rs`
- `src/server/event_subscriptions.rs`
### `src/agent.rs`
Split into:
- `src/agent/loop.rs`
- `src/agent/stream.rs`
- `src/agent/tool_exec.rs`
- `src/agent/interrupts.rs`
- `src/agent/messages.rs`
- `src/agent/retry.rs`
### `src/provider/mod.rs`
Split into:
- `src/provider/traits.rs`
- `src/provider/model_route.rs`
- `src/provider/pricing.rs`
- `src/provider/http.rs`
- `src/provider/capabilities.rs`
## Working Rules for the Refactor Program
- every step must compile or fail for a very obvious temporary reason
- prefer moving code without changing behavior
- avoid mixing cleanup and feature work in the same commit when possible
- when a file is touched, leave it cleaner than it was
- if a new broad allow-suppression is added, it must be documented in the PR
## Validation Matrix
Minimum validation during this program:
- `cargo check -q`
- `cargo test -q`
- targeted tests for touched areas
- `scripts/check_warning_budget.sh`
- `cargo fmt --all -- --check`
Stricter validation when touching core orchestration or provider code:
- `cargo check --all-targets --all-features`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test --all-targets --all-features`
- `cargo test --test e2e`
## Ownership
This is an active engineering program, not a one-time cleanup document. The expectation is:
- the plan is updated as milestones are completed
- todo items are kept current
- progress is visible in the repo
- each completed phase leaves behind stronger guardrails than before
docs/CODE_QUALITY_AUDIT_2026-04-18.md
+561
View File
@@ -0,0 +1,561 @@
# Code Quality Audit - 2026-04-18
This report inventories the repo-wide code-quality issues detectable with static scanning and targeted structural heuristics. It is intended as a comprehensive backlog seed, not just a shortlist.
## Scope and method
- scanned all Rust files outside `target`, `.git`, and `node_modules`
- measured file size by LOC
- approximated function size by brace-balanced `fn` blocks
- counted panic-prone macros and methods with path-based test classification
- inventoried `allow(...)` suppressions and TODO/FIXME/HACK/XXX markers
- note: path-based production vs test classification is approximate and may overcount test-only code embedded inside production files
## Current positives
- `cargo clippy --all-targets --all-features -- -D warnings` passes cleanly
- no `#[allow(dead_code)]` suppressions remain in Rust sources
- formatting is currently clean
## Repo metrics
- Rust files scanned: **455**
- `src/` Rust files: **429** totaling **277,014 LOC**
- `tests/` Rust files: **11** totaling **4,802 LOC**
- `crates/` Rust files: **14** totaling **5,335 LOC**
- Production files over 1200 LOC: **50**
- Production files between 801 and 1200 LOC: **62**
- Approximate production functions over 100 LOC: **304** across **165** files
## `unwrap` / `expect` split by production vs test-only files
Using improved path-based classification for Rust files:
- production files exclude `tests/`, `*_test.rs`, `*_tests.rs`, and directories ending in `_test` / `_tests`
- test-only files include `tests/` and Rust files or directories explicitly marked as tests
- note: this is still path-based, so test-only code embedded inside production files is counted as production
### Counts
| Scope | `unwrap` / `expect` occurrences |
|---|---:|
| Production files | **1258** |
| Test-only files | **1334** |
### Highest-count production files
| Count | File |
|---:|---|
| 136 | `src/tool/communicate.rs` |
| 62 | `src/build.rs` |
| 52 | `src/auth/cursor.rs` |
| 46 | `src/auth/codex.rs` |
| 42 | `src/provider/openai.rs` |
| 37 | `src/auth/claude.rs` |
| 30 | `src/cli/dispatch.rs` |
| 28 | `src/tool/bash.rs` |
| 26 | `src/storage.rs` |
| 25 | `src/auth/gemini.rs` |
| 25 | `src/tool/read.rs` |
| 25 | `src/tui/session_picker/loading.rs` |
| 24 | `src/side_panel.rs` |
| 24 | `src/cli/args.rs` |
| 24 | `src/server/comm_control.rs` |
### Highest-count test-only files
| Count | File |
|---:|---|
| 788 | `src/tui/app/tests.rs` |
| 98 | `src/tool/selfdev/tests.rs` |
| 59 | `src/memory_tests.rs` |
| 44 | `src/import_tests.rs` |
| 26 | `src/provider/tests.rs` |
| 26 | `src/tool/agentgrep_tests.rs` |
| 24 | `src/tui/mermaid_tests.rs` |
| 24 | `src/server/socket_tests.rs` |
| 21 | `src/tui/markdown_tests/cases.rs` |
| 20 | `src/provider/openrouter_tests.rs` |
| 18 | `src/tui/ui_pinned_tests.rs` |
| 17 | `src/cli/provider_init_tests.rs` |
| 15 | `src/agent_tests.rs` |
| 12 | `tests/e2e/provider_behavior.rs` |
| 12 | `src/server/startup_tests.rs` |
## Structural debt
### Production files over 1200 LOC
| LOC | File |
|---:|---|
| 3228 | `src/server/comm_control.rs` |
| 3165 | `src/tool/communicate.rs` |
| 2729 | `src/session.rs` |
| 2704 | `src/server/client_lifecycle.rs` |
| 2683 | `src/provider/openai.rs` |
| 2437 | `src/tui/ui.rs` |
| 2397 | `src/memory.rs` |
| 2365 | `src/provider/mod.rs` |
| 2217 | `src/telemetry.rs` |
| 2131 | `src/tui/ui_messages.rs` |
| 2115 | `src/tui/session_picker.rs` |
| 2041 | `src/tui/app/inline_interactive.rs` |
| 2023 | `src/tui/app/input.rs` |
| 2005 | `src/config.rs` |
| 1969 | `src/provider/anthropic.rs` |
| 1919 | `src/tui/app/remote/key_handling.rs` |
| 1912 | `src/tui/app/auth.rs` |
| 1900 | `src/usage.rs` |
| 1888 | `src/tui/session_picker/loading.rs` |
| 1881 | `src/cli/login.rs` |
| 1794 | `src/replay.rs` |
| 1769 | `src/cli/provider_init.rs` |
| 1738 | `src/bin/tui_bench.rs` |
| 1718 | `src/compaction.rs` |
| 1708 | `src/tui/ui_prepare.rs` |
| 1696 | `src/memory_agent.rs` |
| 1688 | `src/tui/info_widget.rs` |
| 1678 | `src/tui/ui_pinned.rs` |
| 1670 | `src/cli/tui_launch.rs` |
| 1630 | `src/tui/app/commands.rs` |
| 1607 | `src/auth/mod.rs` |
| 1572 | `src/tui/ui_input.rs` |
| 1559 | `src/server.rs` |
| 1551 | `src/tui/app/helpers.rs` |
| 1516 | `src/tool/agentgrep.rs` |
| 1504 | `src/import.rs` |
| 1496 | `src/ambient.rs` |
| 1491 | `src/server/swarm.rs` |
| 1446 | `src/tui/ui_tools.rs` |
| 1375 | `src/tui/markdown.rs` |
| 1362 | `src/protocol.rs` |
| 1341 | `src/tool/ambient.rs` |
| 1308 | `src/auth/oauth.rs` |
| 1300 | `src/tui/app/remote.rs` |
| 1292 | `src/tui/app/turn.rs` |
| 1263 | `src/provider/models.rs` |
| 1257 | `src/server/client_actions.rs` |
| 1211 | `src/tui/app/model_context.rs` |
| 1210 | `src/tui/app/tui_state.rs` |
| 1202 | `src/provider/gemini.rs` |
### Production files between 801 and 1200 LOC
| LOC | File |
|---:|---|
| 1195 | `src/video_export.rs` |
| 1192 | `src/tui/app/auth_account_picker.rs` |
| 1167 | `src/tui/mod.rs` |
| 1155 | `src/provider/copilot.rs` |
| 1150 | `src/tui/app/state_ui.rs` |
| 1144 | `src/tool/browser.rs` |
| 1142 | `src/provider/claude.rs` |
| 1132 | `src/provider/openrouter.rs` |
| 1125 | `src/tui/app/remote/server_events.rs` |
| 1124 | `src/tui/app/debug_bench.rs` |
| 1116 | `src/tui/mermaid.rs` |
| 1109 | `src/update.rs` |
| 1094 | `src/server/client_session.rs` |
| 1093 | `src/provider/openai_stream_runtime.rs` |
| 1087 | `src/tool/mod.rs` |
| 1075 | `src/tui/app/state_ui_input_helpers.rs` |
| 1071 | `src/server/comm_session.rs` |
| 1057 | `src/ambient/runner.rs` |
| 1043 | `src/provider/cursor.rs` |
| 1039 | `src/cli/commands.rs` |
| 1038 | `src/server/debug.rs` |
| 1038 | `src/message.rs` |
| 1037 | `src/tui/app/commands_review.rs` |
| 1014 | `src/tui/app/navigation.rs` |
| 1012 | `src/tui/account_picker.rs` |
| 995 | `src/goal.rs` |
| 980 | `src/memory_graph.rs` |
| 979 | `src/tui/markdown_render_full.rs` |
| 976 | `src/auth/claude.rs` |
| 970 | `src/auth/cursor.rs` |
| 958 | `src/browser.rs` |
| 956 | `src/runtime_memory_log.rs` |
| 945 | `src/agent/turn_streaming_mpsc.rs` |
| 929 | `src/cli/dispatch.rs` |
| 925 | `src/tui/ui_animations.rs` |
| 923 | `src/tui/app/auth_account_commands.rs` |
| 918 | `src/tui/test_harness.rs` |
| 911 | `src/auth/codex.rs` |
| 902 | `src/tui/keybind.rs` |
| 900 | `src/tui/ui_inline_interactive.rs` |
| 897 | `src/tui/ui_header.rs` |
| 895 | `src/server/state.rs` |
| 892 | `src/build.rs` |
| 881 | `src/tui/backend.rs` |
| 878 | `src/tui/login_picker.rs` |
| 872 | `src/sidecar.rs` |
| 868 | `src/tui/app/tui_lifecycle.rs` |
| 865 | `src/tui/permissions.rs` |
| 865 | `src/tui/markdown_render_lazy.rs` |
| 863 | `src/gateway.rs` |
| 862 | `src/tool/read.rs` |
| 860 | `src/provider/antigravity.rs` |
| 859 | `src/tool/apply_patch.rs` |
| 858 | `src/tool/bash.rs` |
| 849 | `src/auth/gemini.rs` |
| 847 | `src/tui/visual_debug.rs` |
| 827 | `src/setup_hints.rs` |
| 826 | `src/server/reload.rs` |
| 815 | `src/auth/copilot.rs` |
| 812 | `src/tui/app.rs` |
| 804 | `src/tui/app/remote/reconnect.rs` |
| 803 | `src/server/debug_swarm_read.rs` |
### Test files over 1200 LOC
| LOC | File |
|---:|---|
| 13615 | `src/tui/app/tests.rs` |
| 1263 | `src/server/client_session_tests/resume.rs` |
| 1252 | `src/provider/tests.rs` |
| 1226 | `src/cli/auth_test.rs` |
### Files with the most >100 LOC production functions
| Count | File |
|---:|---|
| 8 | `src/server/comm_control.rs` |
| 7 | `src/tool/communicate.rs` |
| 6 | `src/provider/mod.rs` |
| 5 | `src/auth/mod.rs` |
| 5 | `src/tui/app/auth.rs` |
| 5 | `src/tui/app/debug_bench.rs` |
| 4 | `src/provider/anthropic.rs` |
| 4 | `src/tui/ui_pinned.rs` |
| 4 | `src/tui/ui_prepare.rs` |
| 4 | `src/tui/app/inline_interactive.rs` |
| 4 | `src/tui/app/auth_account_picker.rs` |
| 4 | `src/cli/tui_launch.rs` |
| 4 | `src/server/client_comm.rs` |
| 3 | `src/import.rs` |
| 3 | `src/memory_agent.rs` |
| 3 | `src/replay.rs` |
| 3 | `src/video_export.rs` |
| 3 | `src/server.rs` |
| 3 | `src/usage.rs` |
| 3 | `src/config.rs` |
| 3 | `src/bin/tui_bench.rs` |
| 3 | `src/provider/claude.rs` |
| 3 | `src/provider/copilot.rs` |
| 3 | `src/provider/openai_stream_runtime.rs` |
| 3 | `src/tui/ui_animations.rs` |
| 3 | `src/tui/ui_input.rs` |
| 3 | `src/tui/ui_header.rs` |
| 3 | `src/tui/info_widget.rs` |
| 3 | `src/tui/app/model_context.rs` |
| 3 | `src/tui/app/tui_state.rs` |
| 3 | `src/tui/app/auth_account_commands.rs` |
| 3 | `src/tui/app/commands.rs` |
| 3 | `src/tui/app/remote.rs` |
| 3 | `src/tui/app/debug_profile.rs` |
| 3 | `src/tui/session_picker/loading.rs` |
| 3 | `src/server/comm_plan.rs` |
| 3 | `src/server/client_actions.rs` |
| 3 | `src/server/client_session.rs` |
| 3 | `src/server/swarm.rs` |
| 3 | `src/server/client_lifecycle.rs` |
| 2 | `src/compaction.rs` |
| 2 | `src/telemetry.rs` |
| 2 | `src/background.rs` |
| 2 | `src/auth/oauth.rs` |
| 2 | `src/provider/dispatch.rs` |
| 2 | `src/tool/apply_patch.rs` |
| 2 | `src/tool/agentgrep.rs` |
| 2 | `src/tool/bash.rs` |
| 2 | `src/tool/browser.rs` |
| 2 | `src/tool/selfdev/build_queue.rs` |
### Longest production functions detected
| LOC | Function | Location |
|---:|---|---|
| 1827 | `handle_remote_key_internal` | `src/tui/app/remote/key_handling.rs:93-1919` |
| 1658 | `handle_client` | `src/server/client_lifecycle.rs:669-2326` |
| 1121 | `handle_server_event` | `src/tui/app/remote/server_events.rs:5-1125` |
| 1016 | `run_turn_interactive` | `src/tui/app/turn.rs:23-1038` |
| 976 | `render_markdown_with_width` | `src/tui/markdown_render_full.rs:4-979` |
| 941 | `run_turn_streaming_mpsc` | `src/agent/turn_streaming_mpsc.rs:4-944` |
| 863 | `render_markdown_lazy` | `src/tui/markdown_render_lazy.rs:3-865` |
| 783 | `maybe_handle_swarm_read_command` | `src/server/debug_swarm_read.rs:21-803` |
| 780 | `execute` | `src/tool/communicate.rs:727-1506` |
| 771 | `run_turn_streaming` | `src/agent/turn_streaming_broadcast.rs:4-774` |
| 760 | `run_turn` | `src/agent/turn_loops.rs:9-768` |
| 602 | `complete` | `src/provider/openrouter_provider_impl.rs:6-607` |
| 591 | `draw_inner` | `src/tui/ui.rs:1758-2348` |
| 556 | `handle_debug_command` | `src/tui/app/debug_cmds.rs:4-559` |
| 548 | `handle_lightweight_control_request` | `src/server/client_lifecycle.rs:105-652` |
| 525 | `draw_messages` | `src/tui/ui_viewport.rs:147-671` |
| 509 | `get_suggestions_for` | `src/tui/app/state_ui_input_helpers.rs:374-882` |
| 501 | `handle_login_input` | `src/tui/app/auth.rs:1166-1666` |
| 490 | `get_tool_summary_with_budget` | `src/tui/ui_tools.rs:887-1376` |
| 487 | `execute_debug_command` | `src/server/debug_command_exec.rs:88-574` |
| 482 | `spawn_background_tasks` | `src/server.rs:651-1132` |
| 470 | `main` | `src/bin/tui_bench.rs:1269-1738` |
| 443 | `test_parse_openai_response_function_call_arguments_streaming` | `src/provider/openai.rs:2241-2683` |
| 433 | `apply_env_overrides` | `src/config.rs:773-1205` |
| 429 | `draw_inline_interactive` | `src/tui/ui_inline_interactive.rs:259-687` |
| 422 | `maybe_handle_swarm_write_command` | `src/server/debug_swarm_write.rs:11-432` |
| 408 | `build_server_memory_payload` | `src/server/debug_server_state.rs:254-661` |
| 405 | `handle_resume_session` | `src/server/client_session.rs:686-1090` |
| 404 | `prepare_body_incremental` | `src/tui/ui_prepare.rs:608-1011` |
| 401 | `handle_info_command` | `src/tui/app/state_ui.rs:750-1150` |
| 401 | `handle_comm_task_control` | `src/server/comm_control.rs:1546-1946` |
| 393 | `render_preview` | `src/tui/session_picker.rs:795-1187` |
| 382 | `draw_pinned_content_cached` | `src/tui/ui_pinned.rs:842-1223` |
| 380 | `build_responses_input` | `src/provider/openai_request.rs:286-665` |
| 376 | `stream_response_websocket_persistent` | `src/provider/openai_stream_runtime.rs:551-926` |
| 371 | `handle_inline_interactive_key` | `src/tui/app/inline_interactive.rs:1551-1921` |
| 369 | `set_model` | `src/provider/mod.rs:821-1189` |
| 367 | `render_tool_message` | `src/tui/ui_messages.rs:864-1230` |
| 362 | `prepare_body` | `src/tui/ui_prepare.rs:1066-1427` |
| 358 | `handle_comm_assign_task` | `src/server/comm_control.rs:1008-1365` |
| 346 | `draw_help_overlay` | `src/tui/ui_overlays.rs:85-430` |
| 340 | `debug_app_owned_memory_profile` | `src/tui/app/debug_profile.rs:170-509` |
| 339 | `handle_session_command` | `src/tui/app/commands.rs:578-916` |
| 324 | `try_persistent_ws_continuation` | `src/provider/openai_stream_runtime.rs:224-547` |
| 320 | `init_provider_with_options` | `src/cli/provider_init.rs:1428-1747` |
| 316 | `new` | `src/tui/app/tui_lifecycle.rs:422-737` |
| 316 | `execute` | `src/tool/gmail.rs:93-408` |
| 315 | `draw_status` | `src/tui/ui_input.rs:397-711` |
| 313 | `model_routes` | `src/provider/mod.rs:1342-1654` |
| 312 | `handle_debug_client` | `src/server/debug.rs:184-495` |
| 307 | `get_relevant_parallel` | `src/memory.rs:1752-2058` |
| 304 | `list_sessions` | `src/cli/tui_launch.rs:1146-1449` |
| 303 | `execute` | `src/tool/memory.rs:116-418` |
| 296 | `complete` | `src/provider/openai_provider_impl.rs:8-303` |
| 294 | `run_loop` | `src/ambient/runner.rs:443-736` |
| 291 | `run_scroll_test` | `src/tui/app/debug_bench.rs:710-1000` |
| 290 | `new_minimal_with_session` | `src/tui/app/tui_lifecycle.rs:131-420` |
| 289 | `open_account_center` | `src/tui/app/auth_account_picker.rs:4-292` |
| 277 | `handle_model_command` | `src/tui/app/model_context.rs:862-1138` |
| 277 | `monitor_bus` | `src/server.rs:1162-1438` |
| 275 | `display_string` | `src/config.rs:1688-1962` |
| 267 | `emit_lifecycle_event` | `src/telemetry.rs:1929-2195` |
| 261 | `prepare_messages_inner` | `src/tui/ui_prepare.rs:300-560` |
| 261 | `render_mermaid_sized_internal` | `src/tui/mermaid_cache_render.rs:427-687` |
| 261 | `handle_mouse_event` | `src/tui/app/navigation.rs:683-943` |
| 260 | `open_model_picker` | `src/tui/app/inline_interactive.rs:728-987` |
| 258 | `build_all_inline_account_picker` | `src/tui/app/auth_account_picker.rs:445-702` |
| 256 | `buffer_to_svg` | `src/video_export.rs:610-865` |
| 253 | `info_widget_data` | `src/tui/app/tui_state.rs:769-1021` |
| 249 | `build_header_lines` | `src/tui/ui_header.rs:421-669` |
| 249 | `extract_from_context` | `src/memory_agent.rs:725-973` |
| 245 | `box_drawing_to_svg` | `src/video_export.rs:931-1175` |
| 243 | `do_build` | `src/tool/selfdev/build_queue.rs:340-582` |
| 241 | `spawn_assigned_task_run` | `src/server/comm_control.rs:441-681` |
| 240 | `complete` | `src/provider/gemini.rs:393-632` |
| 240 | `process_context` | `src/memory_agent.rs:393-632` |
| 240 | `create_default_config_file` | `src/config.rs:1446-1685` |
| 238 | `handle_comm_propose_plan` | `src/server/comm_plan.rs:25-262` |
| 238 | `login_google_flow` | `src/cli/login.rs:1538-1775` |
| 235 | `new_with_auth_status` | `src/provider/startup.rs:50-284` |
| 234 | `run_side_panel_latency_bench` | `src/tui/app/debug_bench.rs:79-312` |
| 233 | `try_auto_compact_and_retry` | `src/tui/app/model_context.rs:441-673` |
| 233 | `handle_config_command` | `src/tui/app/commands.rs:1279-1511` |
| 232 | `run_mermaid_ui_bench` | `src/tui/app/debug_bench.rs:314-545` |
| 232 | `build_ambient_system_prompt` | `src/ambient.rs:788-1019` |
| 229 | `maybe_handle_server_state_command` | `src/server/debug_server_state.rs:20-248` |
| 228 | `run_memory_command` | `src/cli/commands.rs:161-388` |
| 225 | `draw_side_panel_markdown` | `src/tui/ui_pinned.rs:1225-1449` |
| 224 | `compact_tool_input_for_display` | `src/tui/app/state_ui_storage.rs:3-226` |
| 223 | `send_history` | `src/server/client_state.rs:232-454` |
| 221 | `handle_debug_command` | `src/tui/app/debug.rs:538-758` |
| 221 | `run_main` | `src/cli/dispatch.rs:21-241` |
| 220 | `rebuild_items` | `src/tui/session_picker/filter.rs:125-344` |
| 220 | `export_timeline` | `src/replay.rs:138-357` |
| 217 | `handle_comm_message` | `src/server/client_comm_message.rs:149-365` |
| 215 | `bridge_request` | `src/tool/browser.rs:472-686` |
| 213 | `connect_with_retry` | `src/tui/app/remote/reconnect.rs:339-551` |
| 212 | `restore_input_for_reload` | `src/tui/app/state_ui.rs:240-451` |
| 210 | `run_replay_command` | `src/cli/tui_launch.rs:437-646` |
| 208 | `shape_char_3x3` | `src/tui/ui_animations.rs:574-781` |
| 208 | `selfdev_status_output` | `src/tool/selfdev/status.rs:3-210` |
| 208 | `execute` | `src/tool/goal.rs:141-348` |
| 207 | `parse_account_command` | `src/tui/app/auth_account_commands.rs:69-275` |
| 206 | `restore_session` | `src/tui/app/tui_lifecycle_runtime.rs:212-417` |
| 205 | `render_image_widget` | `src/tui/mermaid_widget.rs:91-295` |
| 205 | `render_image_widget_viewport` | `src/tui/mermaid_viewport.rs:552-756` |
| 205 | `spawn_swarm_agent` | `src/server/comm_session.rs:196-400` |
| 205 | `handle_subscribe` | `src/server/client_session.rs:339-543` |
| 204 | `parse_next_event` | `src/provider/openrouter_sse_stream.rs:264-467` |
| 203 | `cleanup_client_connection` | `src/server/client_disconnect_cleanup.rs:55-257` |
| 198 | `calculate_placements` | `src/tui/info_widget_layout.rs:39-236` |
| 198 | `calculate_widget_height` | `src/tui/info_widget.rs:751-948` |
| 195 | `parse_text_wrapped_tool_call` | `src/agent/response_recovery.rs:4-198` |
| 194 | `do_reload` | `src/tool/selfdev/reload.rs:88-281` |
| 192 | `render_image_widget_fit_inner` | `src/tui/mermaid_widget.rs:318-509` |
| 188 | `write_frame` | `src/tui/visual_debug.rs:573-760` |
| 187 | `emit_ndjson_event` | `src/cli/commands.rs:758-944` |
| 186 | `stream_request` | `src/provider/copilot.rs:608-793` |
| 183 | `handle_ws_connection` | `src/gateway.rs:282-464` |
| 182 | `process_sse_stream` | `src/provider/copilot.rs:795-976` |
## Error-handling and panic-surface debt
Path-classified counts below are approximate. Inline `#[cfg(test)]` modules inside production files may inflate production totals.
### Macro/method counts
| Scope | unwrap | expect | panic! | todo! | unimplemented! | total |
|---|---:|---:|---:|---:|---:|---:|
| prod | 361 | 978 | 92 | 0 | 11 | 1442 |
| testlike | 501 | 832 | 52 | 0 | 10 | 1395 |
### Highest-count production files
| Total | File | unwrap | expect | panic! | todo! | unimplemented! |
|---:|---|---:|---:|---:|---:|---:|
| 136 | `src/tool/communicate.rs` | 0 | 136 | 0 | 0 | 0 |
| 64 | `src/build.rs` | 9 | 53 | 2 | 0 | 0 |
| 54 | `src/provider/openai.rs` | 7 | 38 | 9 | 0 | 0 |
| 52 | `src/auth/cursor.rs` | 48 | 4 | 0 | 0 | 0 |
| 46 | `src/auth/codex.rs` | 45 | 1 | 0 | 0 | 0 |
| 41 | `src/server/comm_control.rs` | 0 | 30 | 11 | 0 | 0 |
| 40 | `src/cli/args.rs` | 24 | 0 | 16 | 0 | 0 |
| 37 | `src/auth/claude.rs` | 28 | 9 | 0 | 0 | 0 |
| 30 | `src/cli/dispatch.rs` | 0 | 28 | 2 | 0 | 0 |
| 28 | `src/tool/bash.rs` | 7 | 21 | 0 | 0 | 0 |
| 26 | `src/storage.rs` | 0 | 26 | 0 | 0 | 0 |
| 25 | `src/tui/session_picker/loading.rs` | 0 | 25 | 0 | 0 | 0 |
| 25 | `src/tool/read.rs` | 0 | 25 | 0 | 0 | 0 |
| 25 | `src/auth/gemini.rs` | 4 | 21 | 0 | 0 | 0 |
| 24 | `src/tool/apply_patch.rs` | 15 | 1 | 8 | 0 | 0 |
| 24 | `src/side_panel.rs` | 0 | 24 | 0 | 0 | 0 |
| 24 | `src/server/client_comm.rs` | 0 | 12 | 11 | 0 | 1 |
| 23 | `src/server/reload.rs` | 0 | 23 | 0 | 0 | 0 |
| 21 | `src/tui/session_picker.rs` | 7 | 13 | 1 | 0 | 0 |
| 21 | `src/server/debug.rs` | 0 | 18 | 2 | 0 | 1 |
| 20 | `src/tool/goal.rs` | 0 | 19 | 1 | 0 | 0 |
| 20 | `src/server/comm_session.rs` | 0 | 20 | 0 | 0 | 0 |
| 19 | `src/cli/tui_launch.rs` | 0 | 18 | 1 | 0 | 0 |
| 19 | `src/auth/external.rs` | 19 | 0 | 0 | 0 | 0 |
| 18 | `src/provider/gemini.rs` | 7 | 10 | 0 | 0 | 1 |
| 17 | `src/restart_snapshot.rs` | 0 | 17 | 0 | 0 | 0 |
| 16 | `src/server/client_state.rs` | 0 | 14 | 1 | 0 | 1 |
| 16 | `src/replay.rs` | 11 | 2 | 3 | 0 | 0 |
| 16 | `src/goal.rs` | 0 | 16 | 0 | 0 | 0 |
| 15 | `src/server/client_actions.rs` | 3 | 9 | 2 | 0 | 1 |
| 14 | `src/tui/app/remote.rs` | 0 | 13 | 0 | 0 | 1 |
| 14 | `src/memory_graph.rs` | 12 | 2 | 0 | 0 | 0 |
| 14 | `src/mcp/protocol.rs` | 11 | 2 | 1 | 0 | 0 |
| 14 | `src/cli/selfdev.rs` | 1 | 12 | 0 | 0 | 1 |
| 13 | `src/setup_hints/macos_launcher.rs` | 0 | 13 | 0 | 0 | 0 |
| 13 | `src/server/client_lifecycle.rs` | 0 | 10 | 3 | 0 | 0 |
| 13 | `src/registry.rs` | 0 | 13 | 0 | 0 | 0 |
| 12 | `src/tool/batch.rs` | 12 | 0 | 0 | 0 | 0 |
| 12 | `src/server/swarm_mutation_state.rs` | 0 | 8 | 4 | 0 | 0 |
| 12 | `src/provider_catalog.rs` | 0 | 12 | 0 | 0 | 0 |
| 12 | `src/prompt.rs` | 11 | 1 | 0 | 0 | 0 |
| 11 | `src/tool/agentgrep.rs` | 0 | 11 | 0 | 0 | 0 |
| 10 | `src/tool/ambient.rs` | 10 | 0 | 0 | 0 | 0 |
| 9 | `src/soft_interrupt_store.rs` | 0 | 9 | 0 | 0 | 0 |
| 9 | `src/server/provider_control.rs` | 3 | 6 | 0 | 0 | 0 |
| 9 | `src/platform.rs` | 0 | 9 | 0 | 0 | 0 |
| 9 | `src/cli/login.rs` | 0 | 8 | 1 | 0 | 0 |
| 9 | `src/cli/commands/restart.rs` | 0 | 9 | 0 | 0 | 0 |
| 8 | `src/tool/side_panel.rs` | 0 | 8 | 0 | 0 | 0 |
| 8 | `src/tool/browser.rs` | 6 | 2 | 0 | 0 | 0 |
| 8 | `src/stdin_detect.rs` | 0 | 8 | 0 | 0 | 0 |
| 8 | `src/sidecar.rs` | 0 | 8 | 0 | 0 | 0 |
| 8 | `src/runtime_memory_log.rs` | 0 | 8 | 0 | 0 | 0 |
| 8 | `src/message.rs` | 4 | 1 | 3 | 0 | 0 |
| 8 | `src/gateway.rs` | 1 | 7 | 0 | 0 | 0 |
| 8 | `src/ambient.rs` | 8 | 0 | 0 | 0 | 0 |
| 7 | `src/server/swarm.rs` | 0 | 6 | 1 | 0 | 0 |
| 7 | `src/server/debug_testers.rs` | 0 | 7 | 0 | 0 | 0 |
| 7 | `src/provider/cursor.rs` | 4 | 3 | 0 | 0 | 0 |
| 7 | `src/dictation.rs` | 0 | 7 | 0 | 0 | 0 |
### Production files still containing `todo!` or `unimplemented!`
| Count | File |
|---:|---|
| 7 | `src/tui/app/tests.rs` |
| 1 | `src/tui/ui_header.rs` |
| 1 | `src/tui/app/remote.rs` |
| 1 | `src/tool/mod.rs` |
| 1 | `src/server/startup_tests.rs` |
| 1 | `src/server/queue_tests.rs` |
| 1 | `src/server/debug_command_exec.rs` |
| 1 | `src/server/debug.rs` |
| 1 | `src/server/client_state.rs` |
| 1 | `src/server/client_session_tests.rs` |
| 1 | `src/server/client_comm.rs` |
| 1 | `src/server/client_actions.rs` |
| 1 | `src/provider/gemini.rs` |
| 1 | `src/cli/selfdev.rs` |
| 1 | `src/ambient/runner.rs` |
## Suppression inventory
- Rust files containing `allow(...)`: **17**
- Total `allow(...)` attributes found: **28**
### Most common suppressions
| Count | Suppression |
|---:|---|
| 13 | `clippy::too_many_arguments` |
| 7 | `unused_mut` |
| 2 | `non_upper_case_globals` |
| 2 | `deprecated` |
| 2 | `unused_imports` |
| 1 | `non_snake_case` |
| 1 | `unused_variables` |
### Files containing suppressions
| Count | File | Suppressions |
|---:|---|---|
| 5 | `src/server/client_session.rs` | `clippy::too_many_arguments`, `clippy::too_many_arguments`, `clippy::too_many_arguments`, `clippy::too_many_arguments`, `clippy::too_many_arguments` |
| 3 | `src/cli/dispatch.rs` | `deprecated`, `unused_mut`, `unused_mut` |
| 2 | `src/tui/app/remote.rs` | `unused_imports`, `unused_imports` |
| 2 | `src/server/comm_session.rs` | `clippy::too_many_arguments`, `clippy::too_many_arguments` |
| 2 | `src/server/client_lifecycle.rs` | `clippy::too_many_arguments`, `clippy::too_many_arguments` |
| 2 | `src/server.rs` | `unused_mut`, `unused_mut` |
| 2 | `src/main.rs` | `non_upper_case_globals`, `non_upper_case_globals` |
| 1 | `src/tui/info_widget.rs` | `deprecated` |
| 1 | `src/tui/app/state_ui.rs` | `unused_mut` |
| 1 | `src/server/startup_tests.rs` | `unused_mut` |
| 1 | `src/server/debug_swarm_write.rs` | `clippy::too_many_arguments` |
| 1 | `src/server/comm_sync.rs` | `clippy::too_many_arguments` |
| 1 | `src/server/comm_await.rs` | `clippy::too_many_arguments` |
| 1 | `src/server/client_actions.rs` | `clippy::too_many_arguments` |
| 1 | `src/perf.rs` | `non_snake_case` |
| 1 | `src/auth/mod.rs` | `unused_mut` |
| 1 | `src/agent/turn_loops.rs` | `unused_variables` |
## TODO/FIXME/HACK debt
| Count | File |
|---:|---|
| 9 | `docs/CODE_QUALITY_AUDIT_2026-04-18.md` |
| 5 | `src/tui/ui_tests/prepare.rs` |
| 4 | `src/tui/ui_tests/tools.rs` |
| 1 | `src/stdin_detect.rs` |
| 1 | `docs/MEMORY_ARCHITECTURE.md` |
| 1 | `docs/IOS_CLIENT.md` |
## Highest-value improvement themes
1. **Split mega-files before adding more logic.** The repo has a very large number of production files far above the documented 1200 LOC ceiling, with especially acute concentration in TUI, server, provider, session, and tooling modules.
2. **Break down monster functions.** The biggest maintainability risk is not only file size but massive single functions like `handle_remote_key_internal`, `handle_client`, `handle_server_event`, `run_turn_interactive`, and multiple markdown/rendering paths.
3. **Reduce argument fan-out in server control/session code.** Repeated `#[allow(clippy::too_many_arguments)]` in server modules indicates missing request-context structs or narrower helper boundaries.
4. **Harden failure paths in real production code.** Even with clippy clean, there is still broad `unwrap`/`expect`/`panic!` presence, especially in tool execution, auth, server control, build, and provider code. Some of this is test-only code inside production files and should be moved out or isolated.
5. **Move or isolate inline tests embedded in giant production files.** Several production files carry substantial test bodies, inflating file size and panic-prone call counts.
6. **Reduce test concentration.** `src/tui/app/tests.rs` is itself a giant hotspot and should be split by domain like auth, remote, commands, rendering, and state restoration.
7. **Trim suppression surface.** Most suppressions are test-only clippy escapes, but the `too_many_arguments` suppressions in server code are architectural smell, not just lint noise.
8. **Burn down deferred work markers.** There are not many TODO/FIXME markers, which is good, but the remaining ones should still be converted into issues or resolved.
## Suggested execution order
1. split `src/server/comm_control.rs`, `src/server/client_lifecycle.rs`, `src/provider/mod.rs`, `src/provider/openai.rs`, and TUI remote/input modules
2. extract context/request structs to eliminate `too_many_arguments` suppressions in server paths
3. move inline tests out of production mega-files where practical
4. replace easy production `unwrap`/`expect` hotspots with explicit error handling, starting with tool/auth/build modules
5. continue splitting TUI render and event-handling functions into domain-focused helpers
docs/CODE_QUALITY_TODO.md
+481
View File
@@ -0,0 +1,481 @@
# Code Quality Program Todo List
This file tracks the execution backlog for the code-quality uplift program described in `docs/CODE_QUALITY_10_10_PLAN.md`.
Status values:
- `pending`
- `in_progress`
- `blocked`
- `done`
## Phase 0: Prevent Further Decay
- [x] Add CI job for `cargo check --all-targets --all-features`
- [x] Add CI job for `cargo clippy --all-targets --all-features -- -D warnings`
- [x] Keep warning policy on a downward ratchet
- [x] Add documented file-size and function-size targets to contributor guidance
## Phase 1: Warning and Dead-Code Burn-Down
- [x] Inventory all `#![allow(dead_code)]` locations and justify or remove them
- [x] Reduce baseline warning count significantly from the current level
- [ ] Remove stale unused functions in `setup_hints.rs`
- [ ] Remove stale unused code in TUI support modules
- [ ] Audit broad suppressions and replace with narrow local allowances
## Phase 2: Decompose the Biggest Files
### Highest priority
- [x] Split `tests/e2e/main.rs` by feature area
- Started 2026-03-24: extracted feature modules `session_flow`, `transport`, `provider_behavior`, `binary_integration`, `safety`, and `ambient`
- Completed 2026-03-24: extracted shared helpers into `tests/e2e/test_support/mod.rs`
- Verified 2026-05-18: `tests/e2e/main.rs` is now only the feature-module entry point and the full non-ignored e2e target passed 44/44 tests.
- [ ] Continue splitting `src/server.rs` into focused submodules ([#53](https://github.com/1jehuang/jcode/issues/53))
- Progress 2026-03-24: extracted shared server/swarm state into `src/server/state.rs`
- Progress 2026-03-24: extracted socket/bootstrap helpers into `src/server/socket.rs`
- Progress 2026-03-24: extracted reload marker/signal state into `src/server/reload_state.rs`
- Progress 2026-03-24: extracted path/update/swarm identity utilities into `src/server/util.rs`
- [ ] Split `src/agent.rs` into orchestration, stream, interrupt, and tool-exec modules
### Next wave
- [ ] Split `src/provider/mod.rs` into traits, pricing, routes, and shared HTTP helpers ([#52](https://github.com/1jehuang/jcode/issues/52))
- [ ] Split `src/provider/openai.rs` into request, stream, tool, and response modules ([#52](https://github.com/1jehuang/jcode/issues/52))
- [ ] Split `src/tui/ui.rs` by render responsibility ([#51](https://github.com/1jehuang/jcode/issues/51))
- [ ] Split `src/tui/info_widget.rs` by widget/domain sections ([#51](https://github.com/1jehuang/jcode/issues/51))
## Phase 3: Error Handling Hardening
- [ ] Count production `unwrap` / `expect` separately from test-only usages
- [ ] Replace easy production `unwrap` / `expect` hotspots with explicit errors
- [ ] Add better error context for provider stream parsing failures
- [ ] Add better error context for reload and socket lifecycle failures ([#53](https://github.com/1jehuang/jcode/issues/53))
## Phase 4: Test Strategy Improvements
- [ ] Extract shared e2e test support helpers
- [ ] Add focused tests for reload state transitions
- [ ] Add focused tests for malformed provider stream chunks
- [ ] Add snapshot or golden tests for stable TUI render outputs
- [ ] Add property tests for protocol serialization and tool parsing
## Phase 5: Reliability and Performance Guardrails
- [ ] Add repeated reload reliability test coverage
- [ ] Add repeated attach/detach and reconnect coverage
- [ ] Track memory regression expectations in a documented budget
- [ ] Improve observability around reload, swarm, and tool execution paths
- [ ] Execute the compile-performance roadmap in `docs/COMPILE_PERFORMANCE_PLAN.md`
- [ ] Add repeatable compile timing checkpoints for warm/cold self-dev loops
## Immediate Active Work
- [x] Land the quality plan document
- [x] Land this todo list
- [x] Tighten CI guardrails
- [ ] Begin the first high-ROI cleanup or split
- Follow-up tracking issues: #51, #52, #53, #54
## Comprehensive Audit Backlog (2026-04-18)
Generated from `docs/CODE_QUALITY_AUDIT_2026-04-18.md`. This section enumerates the full file-level backlog detected by the audit so the todo list captures all current hotspots.
### Audit snapshot
- [x] Publish comprehensive audit report (`50` production files >1200 LOC, `62` production files 801-1200 LOC, `304` production functions >100 LOC across `165` files)
- [ ] Refresh this audit backlog after each major cleanup wave
### Structural backlog: production files over 1200 LOC
- [ ] Split `src/server/comm_control.rs` (3228 LOC)
- [ ] Split `src/tool/communicate.rs` (3165 LOC)
- [ ] Split `src/session.rs` (2729 LOC)
- [ ] Split `src/server/client_lifecycle.rs` (2704 LOC)
- [ ] Split `src/provider/openai.rs` (2683 LOC)
- [ ] Split `src/tui/ui.rs` (2437 LOC)
- [ ] Split `src/memory.rs` (2397 LOC)
- [ ] Split `src/provider/mod.rs` (2365 LOC)
- [ ] Split `src/telemetry.rs` (2217 LOC)
- [ ] Split `src/tui/ui_messages.rs` (2131 LOC)
- [ ] Split `src/tui/session_picker.rs` (2115 LOC)
- [ ] Split `src/tui/app/inline_interactive.rs` (2041 LOC)
- [ ] Split `src/tui/app/input.rs` (2023 LOC)
- [ ] Split `src/config.rs` (2005 LOC)
- [ ] Split `src/provider/anthropic.rs` (1969 LOC)
- [ ] Split `src/tui/app/remote/key_handling.rs` (1919 LOC)
- [ ] Split `src/tui/app/auth.rs` (1912 LOC)
- [ ] Split `src/usage.rs` (1900 LOC)
- [ ] Split `src/tui/session_picker/loading.rs` (1888 LOC)
- [ ] Split `src/cli/login.rs` (1881 LOC)
- [ ] Split `src/replay.rs` (1794 LOC)
- [ ] Split `src/cli/provider_init.rs` (1769 LOC)
- [ ] Split `src/bin/tui_bench.rs` (1738 LOC)
- [ ] Split `src/compaction.rs` (1718 LOC)
- [ ] Split `src/tui/ui_prepare.rs` (1708 LOC)
- [ ] Split `src/memory_agent.rs` (1696 LOC)
- [ ] Split `src/tui/info_widget.rs` (1688 LOC)
- [ ] Split `src/tui/ui_pinned.rs` (1678 LOC)
- [ ] Split `src/cli/tui_launch.rs` (1670 LOC)
- [ ] Split `src/tui/app/commands.rs` (1630 LOC)
- [ ] Split `src/auth/mod.rs` (1607 LOC)
- [ ] Split `src/tui/ui_input.rs` (1572 LOC)
- [ ] Split `src/server.rs` (1559 LOC)
- [ ] Split `src/tui/app/helpers.rs` (1551 LOC)
- [ ] Split `src/tool/agentgrep.rs` (1516 LOC)
- [ ] Split `src/import.rs` (1504 LOC)
- [ ] Split `src/ambient.rs` (1496 LOC)
- [ ] Split `src/server/swarm.rs` (1491 LOC)
- [ ] Split `src/tui/ui_tools.rs` (1446 LOC)
- [ ] Split `src/tui/markdown.rs` (1375 LOC)
- [ ] Split `src/protocol.rs` (1362 LOC)
- [ ] Split `src/tool/ambient.rs` (1341 LOC)
- [ ] Split `src/auth/oauth.rs` (1308 LOC)
- [ ] Split `src/tui/app/remote.rs` (1300 LOC)
- [ ] Split `src/tui/app/turn.rs` (1292 LOC)
- [ ] Split `src/provider/models.rs` (1263 LOC)
- [ ] Split `src/server/client_actions.rs` (1257 LOC)
- [ ] Split `src/tui/app/model_context.rs` (1211 LOC)
- [ ] Split `src/tui/app/tui_state.rs` (1210 LOC)
- [ ] Split `src/provider/gemini.rs` (1202 LOC)
### Structural backlog: production files between 801 and 1200 LOC
- [ ] Reduce `src/video_export.rs` below 800 LOC (1195 LOC today)
- [ ] Reduce `src/tui/app/auth_account_picker.rs` below 800 LOC (1192 LOC today)
- [ ] Reduce `src/tui/mod.rs` below 800 LOC (1167 LOC today)
- [ ] Reduce `src/provider/copilot.rs` below 800 LOC (1155 LOC today)
- [ ] Reduce `src/tui/app/state_ui.rs` below 800 LOC (1150 LOC today)
- [ ] Reduce `src/tool/browser.rs` below 800 LOC (1144 LOC today)
- [ ] Reduce `src/provider/claude.rs` below 800 LOC (1142 LOC today)
- [ ] Reduce `src/provider/openrouter.rs` below 800 LOC (1132 LOC today)
- [ ] Reduce `src/tui/app/remote/server_events.rs` below 800 LOC (1125 LOC today)
- [ ] Reduce `src/tui/app/debug_bench.rs` below 800 LOC (1124 LOC today)
- [ ] Reduce `src/tui/mermaid.rs` below 800 LOC (1116 LOC today)
- [ ] Reduce `src/update.rs` below 800 LOC (1109 LOC today)
- [ ] Reduce `src/server/client_session.rs` below 800 LOC (1094 LOC today)
- [ ] Reduce `src/provider/openai_stream_runtime.rs` below 800 LOC (1093 LOC today)
- [ ] Reduce `src/tool/mod.rs` below 800 LOC (1087 LOC today)
- [ ] Reduce `src/tui/app/state_ui_input_helpers.rs` below 800 LOC (1075 LOC today)
- [ ] Reduce `src/server/comm_session.rs` below 800 LOC (1071 LOC today)
- [ ] Reduce `src/ambient/runner.rs` below 800 LOC (1057 LOC today)
- [ ] Reduce `src/provider/cursor.rs` below 800 LOC (1043 LOC today)
- [ ] Reduce `src/cli/commands.rs` below 800 LOC (1039 LOC today)
- [ ] Reduce `src/server/debug.rs` below 800 LOC (1038 LOC today)
- [ ] Reduce `src/message.rs` below 800 LOC (1038 LOC today)
- [ ] Reduce `src/tui/app/commands_review.rs` below 800 LOC (1037 LOC today)
- [ ] Reduce `src/tui/app/navigation.rs` below 800 LOC (1014 LOC today)
- [ ] Reduce `src/tui/account_picker.rs` below 800 LOC (1012 LOC today)
- [ ] Reduce `src/goal.rs` below 800 LOC (995 LOC today)
- [ ] Reduce `src/memory_graph.rs` below 800 LOC (980 LOC today)
- [ ] Reduce `src/tui/markdown_render_full.rs` below 800 LOC (979 LOC today)
- [ ] Reduce `src/auth/claude.rs` below 800 LOC (976 LOC today)
- [ ] Reduce `src/auth/cursor.rs` below 800 LOC (970 LOC today)
- [ ] Reduce `src/browser.rs` below 800 LOC (958 LOC today)
- [ ] Reduce `src/runtime_memory_log.rs` below 800 LOC (956 LOC today)
- [ ] Reduce `src/agent/turn_streaming_mpsc.rs` below 800 LOC (945 LOC today)
- [ ] Reduce `src/cli/dispatch.rs` below 800 LOC (929 LOC today)
- [ ] Reduce `src/tui/ui_animations.rs` below 800 LOC (925 LOC today)
- [ ] Reduce `src/tui/app/auth_account_commands.rs` below 800 LOC (923 LOC today)
- [ ] Reduce `src/tui/test_harness.rs` below 800 LOC (918 LOC today)
- [ ] Reduce `src/auth/codex.rs` below 800 LOC (911 LOC today)
- [ ] Reduce `src/tui/keybind.rs` below 800 LOC (902 LOC today)
- [ ] Reduce `src/tui/ui_inline_interactive.rs` below 800 LOC (900 LOC today)
- [ ] Reduce `src/tui/ui_header.rs` below 800 LOC (897 LOC today)
- [ ] Reduce `src/server/state.rs` below 800 LOC (895 LOC today)
- [ ] Reduce `src/build.rs` below 800 LOC (892 LOC today)
- [ ] Reduce `src/tui/backend.rs` below 800 LOC (881 LOC today)
- [ ] Reduce `src/tui/login_picker.rs` below 800 LOC (878 LOC today)
- [ ] Reduce `src/sidecar.rs` below 800 LOC (872 LOC today)
- [ ] Reduce `src/tui/app/tui_lifecycle.rs` below 800 LOC (868 LOC today)
- [ ] Reduce `src/tui/permissions.rs` below 800 LOC (865 LOC today)
- [ ] Reduce `src/tui/markdown_render_lazy.rs` below 800 LOC (865 LOC today)
- [ ] Reduce `src/gateway.rs` below 800 LOC (863 LOC today)
- [ ] Reduce `src/tool/read.rs` below 800 LOC (862 LOC today)
- [ ] Reduce `src/provider/antigravity.rs` below 800 LOC (860 LOC today)
- [ ] Reduce `src/tool/apply_patch.rs` below 800 LOC (859 LOC today)
- [ ] Reduce `src/tool/bash.rs` below 800 LOC (858 LOC today)
- [ ] Reduce `src/auth/gemini.rs` below 800 LOC (849 LOC today)
- [ ] Reduce `src/tui/visual_debug.rs` below 800 LOC (847 LOC today)
- [ ] Reduce `src/setup_hints.rs` below 800 LOC (827 LOC today)
- [ ] Reduce `src/server/reload.rs` below 800 LOC (826 LOC today)
- [ ] Reduce `src/auth/copilot.rs` below 800 LOC (815 LOC today)
- [ ] Reduce `src/tui/app.rs` below 800 LOC (812 LOC today)
- [ ] Reduce `src/tui/app/remote/reconnect.rs` below 800 LOC (804 LOC today)
- [ ] Reduce `src/server/debug_swarm_read.rs` below 800 LOC (803 LOC today)
### Test concentration backlog: test files over 1200 LOC
- [x] Split test hotspot `src/tui/app/tests.rs` (was 13615 LOC; split into focused `src/tui/app/tests/*.rs` includes)
- [x] Split test hotspot `src/server/client_session_tests/resume.rs` (was 1263 LOC; split into focused `src/server/client_session_tests/resume/*.rs` includes)
- [x] Split test hotspot `src/provider/tests.rs` (was 1252 LOC; split into focused `src/provider/tests/*.rs` includes)
- [x] Split test hotspot `src/cli/auth_test.rs` (was 1226 LOC; split into focused `src/cli/auth_test/*.rs` includes)
### Long-function backlog outside already-oversized files
- [ ] Break down >100 LOC functions in `src/server/client_comm.rs` (4 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/debug_profile.rs` (3 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/comm_plan.rs` (3 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/ui_file_diff.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/session_picker/render.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/mermaid_widget.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/mermaid_cache_render.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/info_widget_todos.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/info_widget_model.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/tui_lifecycle_runtime.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/selfdev/build_queue.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/debug_server_state.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/client_state.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/provider/dispatch.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/background.rs` (2 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/ui_viewport.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/ui_overlays.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/ui_memory.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/ui_diagram_pane.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/session_picker/filter.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/mermaid_viewport.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/mermaid_debug.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/memory_profile.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/markdown_wrap.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/markdown_render_support.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/info_widget_layout.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/state_ui_storage.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/state_ui_maintenance.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/runtime_memory.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/run_shell.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/dictation.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/debug_script.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/debug_cmds.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tui/app/debug.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/task.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/session_search.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/selfdev/status.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/selfdev/reload.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/memory.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/grep.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/goal.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/gmail.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/conversation_search.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/bg.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/tool/batch.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/setup_hints/windows_setup.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/swarm_persistence.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/reload_state.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/headless.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/debug_swarm_write.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/debug_session_admin.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/debug_jobs.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/debug_help.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/debug_command_exec.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/debug_ambient.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/comm_await.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/client_disconnect_cleanup.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/client_comm_message.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/server/client_comm_context.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/provider/startup.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/provider/openrouter_sse_stream.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/provider/openrouter_provider_impl.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/provider/openai_request.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/provider/openai_provider_impl.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/provider/cli_common.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/memory_log.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/mcp/client.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/cli/selfdev.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/cli/hot_exec.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/catchup.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/bin/harness.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/agent/turn_streaming_broadcast.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/agent/turn_loops.rs` (1 oversized functions)
- [ ] Break down >100 LOC functions in `src/agent/response_recovery.rs` (1 oversized functions)
### Failure-path hardening backlog: production files with panic-prone calls
- [ ] Harden `src/tool/communicate.rs` (`unwrap`: 0, `expect`: 136, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 136)
- [ ] Harden `src/build.rs` (`unwrap`: 9, `expect`: 53, `panic!`: 2, `todo!`: 0, `unimplemented!`: 0, total: 64)
- [ ] Harden `src/provider/openai.rs` (`unwrap`: 7, `expect`: 38, `panic!`: 9, `todo!`: 0, `unimplemented!`: 0, total: 54)
- [ ] Harden `src/auth/cursor.rs` (`unwrap`: 48, `expect`: 4, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 52)
- [ ] Harden `src/auth/codex.rs` (`unwrap`: 45, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 46)
- [ ] Harden `src/server/comm_control.rs` (`unwrap`: 0, `expect`: 30, `panic!`: 11, `todo!`: 0, `unimplemented!`: 0, total: 41)
- [ ] Harden `src/cli/args.rs` (`unwrap`: 24, `expect`: 0, `panic!`: 16, `todo!`: 0, `unimplemented!`: 0, total: 40)
- [ ] Harden `src/auth/claude.rs` (`unwrap`: 28, `expect`: 9, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 37)
- [ ] Harden `src/cli/dispatch.rs` (`unwrap`: 0, `expect`: 28, `panic!`: 2, `todo!`: 0, `unimplemented!`: 0, total: 30)
- [ ] Harden `src/tool/bash.rs` (`unwrap`: 7, `expect`: 21, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 28)
- [ ] Harden `src/storage.rs` (`unwrap`: 0, `expect`: 26, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 26)
- [ ] Harden `src/tui/session_picker/loading.rs` (`unwrap`: 0, `expect`: 25, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 25)
- [ ] Harden `src/tool/read.rs` (`unwrap`: 0, `expect`: 25, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 25)
- [ ] Harden `src/auth/gemini.rs` (`unwrap`: 4, `expect`: 21, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 25)
- [ ] Harden `src/tool/apply_patch.rs` (`unwrap`: 15, `expect`: 1, `panic!`: 8, `todo!`: 0, `unimplemented!`: 0, total: 24)
- [ ] Harden `src/side_panel.rs` (`unwrap`: 0, `expect`: 24, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 24)
- [ ] Harden `src/server/client_comm.rs` (`unwrap`: 0, `expect`: 12, `panic!`: 11, `todo!`: 0, `unimplemented!`: 1, total: 24)
- [ ] Harden `src/server/reload.rs` (`unwrap`: 0, `expect`: 23, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 23)
- [ ] Harden `src/tui/session_picker.rs` (`unwrap`: 7, `expect`: 13, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 21)
- [ ] Harden `src/server/debug.rs` (`unwrap`: 0, `expect`: 18, `panic!`: 2, `todo!`: 0, `unimplemented!`: 1, total: 21)
- [ ] Harden `src/tool/goal.rs` (`unwrap`: 0, `expect`: 19, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 20)
- [ ] Harden `src/server/comm_session.rs` (`unwrap`: 0, `expect`: 20, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 20)
- [ ] Harden `src/cli/tui_launch.rs` (`unwrap`: 0, `expect`: 18, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 19)
- [ ] Harden `src/auth/external.rs` (`unwrap`: 19, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 19)
- [ ] Harden `src/provider/gemini.rs` (`unwrap`: 7, `expect`: 10, `panic!`: 0, `todo!`: 0, `unimplemented!`: 1, total: 18)
- [ ] Harden `src/restart_snapshot.rs` (`unwrap`: 0, `expect`: 17, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 17)
- [ ] Harden `src/server/client_state.rs` (`unwrap`: 0, `expect`: 14, `panic!`: 1, `todo!`: 0, `unimplemented!`: 1, total: 16)
- [ ] Harden `src/replay.rs` (`unwrap`: 11, `expect`: 2, `panic!`: 3, `todo!`: 0, `unimplemented!`: 0, total: 16)
- [ ] Harden `src/goal.rs` (`unwrap`: 0, `expect`: 16, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 16)
- [ ] Harden `src/server/client_actions.rs` (`unwrap`: 3, `expect`: 9, `panic!`: 2, `todo!`: 0, `unimplemented!`: 1, total: 15)
- [ ] Harden `src/tui/app/remote.rs` (`unwrap`: 0, `expect`: 13, `panic!`: 0, `todo!`: 0, `unimplemented!`: 1, total: 14)
- [ ] Harden `src/memory_graph.rs` (`unwrap`: 12, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 14)
- [ ] Harden `src/mcp/protocol.rs` (`unwrap`: 11, `expect`: 2, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 14)
- [ ] Harden `src/cli/selfdev.rs` (`unwrap`: 1, `expect`: 12, `panic!`: 0, `todo!`: 0, `unimplemented!`: 1, total: 14)
- [ ] Harden `src/setup_hints/macos_launcher.rs` (`unwrap`: 0, `expect`: 13, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 13)
- [ ] Harden `src/server/client_lifecycle.rs` (`unwrap`: 0, `expect`: 10, `panic!`: 3, `todo!`: 0, `unimplemented!`: 0, total: 13)
- [ ] Harden `src/registry.rs` (`unwrap`: 0, `expect`: 13, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 13)
- [ ] Harden `src/tool/batch.rs` (`unwrap`: 12, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 12)
- [ ] Harden `src/server/swarm_mutation_state.rs` (`unwrap`: 0, `expect`: 8, `panic!`: 4, `todo!`: 0, `unimplemented!`: 0, total: 12)
- [ ] Harden `src/provider_catalog.rs` (`unwrap`: 0, `expect`: 12, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 12)
- [ ] Harden `src/prompt.rs` (`unwrap`: 11, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 12)
- [ ] Harden `src/tool/agentgrep.rs` (`unwrap`: 0, `expect`: 11, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 11)
- [ ] Harden `src/tool/ambient.rs` (`unwrap`: 10, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 10)
- [ ] Harden `src/soft_interrupt_store.rs` (`unwrap`: 0, `expect`: 9, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 9)
- [ ] Harden `src/server/provider_control.rs` (`unwrap`: 3, `expect`: 6, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 9)
- [ ] Harden `src/platform.rs` (`unwrap`: 0, `expect`: 9, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 9)
- [ ] Harden `src/cli/login.rs` (`unwrap`: 0, `expect`: 8, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 9)
- [ ] Harden `src/cli/commands/restart.rs` (`unwrap`: 0, `expect`: 9, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 9)
- [ ] Harden `src/tool/side_panel.rs` (`unwrap`: 0, `expect`: 8, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/tool/browser.rs` (`unwrap`: 6, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/stdin_detect.rs` (`unwrap`: 0, `expect`: 8, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/sidecar.rs` (`unwrap`: 0, `expect`: 8, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/runtime_memory_log.rs` (`unwrap`: 0, `expect`: 8, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/message.rs` (`unwrap`: 4, `expect`: 1, `panic!`: 3, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/gateway.rs` (`unwrap`: 1, `expect`: 7, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/ambient.rs` (`unwrap`: 8, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 8)
- [ ] Harden `src/server/swarm.rs` (`unwrap`: 0, `expect`: 6, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 7)
- [ ] Harden `src/server/debug_testers.rs` (`unwrap`: 0, `expect`: 7, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 7)
- [ ] Harden `src/provider/cursor.rs` (`unwrap`: 4, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 7)
- [ ] Harden `src/dictation.rs` (`unwrap`: 0, `expect`: 7, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 7)
- [ ] Harden `src/browser.rs` (`unwrap`: 2, `expect`: 5, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 7)
- [ ] Harden `src/tui/app/helpers.rs` (`unwrap`: 0, `expect`: 6, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 6)
- [ ] Harden `src/tool/session_search.rs` (`unwrap`: 1, `expect`: 5, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 6)
- [ ] Harden `src/tool/open.rs` (`unwrap`: 6, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 6)
- [ ] Harden `src/setup_hints.rs` (`unwrap`: 0, `expect`: 6, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 6)
- [ ] Harden `src/server/swarm_persistence.rs` (`unwrap`: 0, `expect`: 6, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 6)
- [ ] Harden `src/provider/antigravity.rs` (`unwrap`: 0, `expect`: 6, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 6)
- [ ] Harden `src/logging.rs` (`unwrap`: 0, `expect`: 6, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 6)
- [ ] Harden `src/tool/mcp.rs` (`unwrap`: 4, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 5)
- [ ] Harden `src/tool/conversation_search.rs` (`unwrap`: 5, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 5)
- [ ] Harden `src/telegram.rs` (`unwrap`: 5, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 5)
- [ ] Harden `src/server/debug_command_exec.rs` (`unwrap`: 0, `expect`: 4, `panic!`: 0, `todo!`: 0, `unimplemented!`: 1, total: 5)
- [ ] Harden `src/provider/pricing.rs` (`unwrap`: 0, `expect`: 5, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 5)
- [ ] Harden `src/tui/ui.rs` (`unwrap`: 0, `expect`: 4, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 4)
- [ ] Harden `src/transport/windows.rs` (`unwrap`: 0, `expect`: 4, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 4)
- [ ] Harden `src/tool/skill.rs` (`unwrap`: 4, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 4)
- [ ] Harden `src/safety.rs` (`unwrap`: 2, `expect`: 1, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 4)
- [ ] Harden `src/login_qr.rs` (`unwrap`: 3, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 4)
- [ ] Harden `src/channel.rs` (`unwrap`: 4, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 4)
- [ ] Harden `crates/jcode-tui-workspace/src/workspace_map.rs` (`unwrap`: 0, `expect`: 4, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 4)
- [ ] Harden `src/tui/ui_messages.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/tui/ui_header.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 1, total: 3)
- [ ] Harden `src/tui/login_picker.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/tui/keybind.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/tui/app/auth.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/session.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/server/comm_plan.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/cli/terminal.rs` (`unwrap`: 2, `expect`: 0, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/bin/tui_bench.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `crates/jcode-provider-openrouter/src/lib.rs` (`unwrap`: 0, `expect`: 3, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 3)
- [ ] Harden `src/video_export.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/tui/ui_animations.rs` (`unwrap`: 0, `expect`: 0, `panic!`: 2, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/tui/backend.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/tui/account_picker.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/tool/mod.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 1, total: 2)
- [ ] Harden `src/server/debug_server_state.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/server/client_disconnect_cleanup.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/server/client_comm_channels.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/provider/openrouter_sse_stream.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/provider/jcode.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/perf.rs` (`unwrap`: 2, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/memory/activity.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/mcp/pool.rs` (`unwrap`: 0, `expect`: 0, `panic!`: 2, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/mcp/manager.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/copilot_usage.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/cache_tracker.rs` (`unwrap`: 2, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/auth/antigravity.rs` (`unwrap`: 0, `expect`: 2, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 2)
- [ ] Harden `src/ambient/runner.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 1, total: 2)
- [ ] Harden `src/tui/workspace_client.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/ui_prepare.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/ui_diagram_pane.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/test_harness.rs` (`unwrap`: 1, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/color_support.rs` (`unwrap`: 0, `expect`: 0, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/app/remote/reconnect.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/app/remote/input_dispatch.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/app/dictation.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/app/debug_bench.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tui/app/commands.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tool/todo.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tool/selfdev/reload.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/tool/memory.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/telemetry.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/server/headless.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/server/debug_swarm_read.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/server/debug_session_admin.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/server/comm_sync.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/server/client_comm_message.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/server/client_comm_context.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/server.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/provider/claude.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/provider/anthropic.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/protocol.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/plan.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/memory/pending.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/gmail.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/config.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/background.rs` (`unwrap`: 0, `expect`: 1, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `src/ambient/scheduler.rs` (`unwrap`: 1, `expect`: 0, `panic!`: 0, `todo!`: 0, `unimplemented!`: 0, total: 1)
- [ ] Harden `crates/jcode-tui-workspace/src/color_support.rs` (`unwrap`: 0, `expect`: 0, `panic!`: 1, `todo!`: 0, `unimplemented!`: 0, total: 1)
### Suppression cleanup backlog
- [ ] Remove or justify suppressions in `src/agent/turn_loops.rs` (unused_variables)
- [ ] Remove or justify suppressions in `src/auth/mod.rs` (unused_mut)
- [ ] Remove or justify suppressions in `src/cli/dispatch.rs` (deprecated, unused_mut, unused_mut)
- [ ] Remove or justify suppressions in `src/main.rs` (non_upper_case_globals, non_upper_case_globals)
- [ ] Remove or justify suppressions in `src/perf.rs` (non_snake_case)
- [ ] Remove or justify suppressions in `src/server.rs` (unused_mut, unused_mut)
- [ ] Remove or justify suppressions in `src/server/client_actions.rs` (clippy::too_many_arguments)
- [ ] Remove or justify suppressions in `src/server/client_lifecycle.rs` (clippy::too_many_arguments, clippy::too_many_arguments)
- [ ] Remove or justify suppressions in `src/server/client_session.rs` (clippy::too_many_arguments, clippy::too_many_arguments, clippy::too_many_arguments, clippy::too_many_arguments, clippy::too_many_arguments)
- [ ] Remove or justify suppressions in `src/server/comm_await.rs` (clippy::too_many_arguments)
- [ ] Remove or justify suppressions in `src/server/comm_session.rs` (clippy::too_many_arguments, clippy::too_many_arguments)
- [ ] Remove or justify suppressions in `src/server/comm_sync.rs` (clippy::too_many_arguments)
- [ ] Remove or justify suppressions in `src/server/debug_swarm_write.rs` (clippy::too_many_arguments)
- [ ] Remove or justify suppressions in `src/server/startup_tests.rs` (unused_mut)
- [ ] Remove or justify suppressions in `src/tui/app/remote.rs` (unused_imports, unused_imports)
- [ ] Remove or justify suppressions in `src/tui/app/state_ui.rs` (unused_mut)
- [ ] Remove or justify suppressions in `src/tui/info_widget.rs` (deprecated)
### Production `todo!` / `unimplemented!` backlog
- [ ] Remove `todo!` / `unimplemented!` from `src/tui/ui_header.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/tui/app/remote.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/tool/mod.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/server/debug_command_exec.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/server/debug.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/server/client_state.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/server/client_comm.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/server/client_actions.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/provider/gemini.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/cli/selfdev.rs` (1 occurrences)
- [ ] Remove `todo!` / `unimplemented!` from `src/ambient/runner.rs` (1 occurrences)
### Test `todo!` / `unimplemented!` backlog
- [ ] Replace test `todo!` / `unimplemented!` in `src/tui/app/tests.rs` (7 occurrences)
- [ ] Replace test `todo!` / `unimplemented!` in `src/server/startup_tests.rs` (1 occurrences)
- [ ] Replace test `todo!` / `unimplemented!` in `src/server/queue_tests.rs` (1 occurrences)
- [ ] Replace test `todo!` / `unimplemented!` in `src/server/client_session_tests.rs` (1 occurrences)
### TODO / FIXME / HACK marker backlog
- [ ] Resolve markers in `docs/CODE_QUALITY_AUDIT_2026-04-18.md` (9 markers)
- [ ] Resolve markers in `src/tui/ui_tests/prepare.rs` (5 markers)
- [ ] Resolve markers in `src/tui/ui_tests/tools.rs` (4 markers)
- [ ] Resolve markers in `src/stdin_detect.rs` (1 markers)
- [ ] Resolve markers in `docs/MEMORY_ARCHITECTURE.md` (1 markers)
- [ ] Resolve markers in `docs/IOS_CLIENT.md` (1 markers)
docs/COMPILE_PERFORMANCE_PLAN.md
+1150
View File
File diff suppressed because it is too large Load Diff
docs/COMPILE_TIME_ISOLATION_REFACTOR.md
+220
View File
@@ -0,0 +1,220 @@
# Compile-Time Isolation Refactor
This is the active migration plan for making full-feature debug/selfdev builds faster without removing features from the developer binary.
## Goal
Keep the normal debug/selfdev binary production-like, including PDF, embeddings, providers, update/selfdev tooling, and other integrations, while reducing the amount of Rust code that must be recompiled after common edits.
The target is not just "more crates". The target is a wider dependency DAG with smaller serial front-end units and cleaner invalidation boundaries.
## Current diagnosis
The workspace already has many crates, but the critical path is dominated by a small number of large crates stacked linearly:
```mermaid
graph LR
base["jcode-base\n~100k+ LOC"] --> appcore["jcode-app-core\n~100k LOC"]
appcore --> tui["jcode-tui\n~100k+ LOC"]
tui --> rootlib["jcode lib"]
rootlib --> bin["jcode bin"]
small["50+ smaller crates"] -. mostly parallel .-> base
```
From the last available Cargo timing report parsed with `scripts/compile_time_probe.sh --skip-build`:
- Cargo timing wall: **16.00s**
- Known jcode serial stack span: **14.72s**
- Known jcode serial stack summed unit time: **17.36s**
- Known jcode serial stack frontend time: **11.99s**
Slowest units from that timing report:
| Unit | Total | Frontend | Codegen |
|---|---:|---:|---:|
| `jcode-app-core` | 4.73s | 3.82s | 0.91s |
| `jcode-base` | 4.34s | 3.63s | 0.71s |
| `jcode-tui` | 4.18s | 3.14s | 1.04s |
| `jcode` bin | 2.34s | n/a | n/a |
| `jcode` lib | 1.77s | 1.40s | 0.37s |
This means the main bottleneck is rustc front-end serialization in a few mega-crates, not linker choice or third-party cold compile.
## Measurement
Use the focused timing probe for each phase:
```bash
scripts/compile_time_probe.sh --json target/compile-time-probe.json
scripts/compile_time_probe.sh --touch crates/jcode-tui/src/tui/app/input.rs
scripts/compile_time_probe.sh --touch crates/jcode-app-core/src/server.rs
scripts/compile_time_probe.sh --touch crates/jcode-base/src/provider/mod.rs
```
For broader repeated measurements, continue using:
```bash
scripts/bench_compile.sh selfdev-jcode --runs 3 --touch <path> --json
scripts/bench_selfdev_checkpoints.sh --skip-cold --touch <path> --runs 1
```
Track at least:
1. Full-feature selfdev build wall time.
2. Cargo timing wall time.
3. `jcode-base -> jcode-app-core -> jcode-tui -> jcode lib -> jcode bin` stack span.
4. Sum of frontend time in the serial stack.
5. Incremental rebuild after touching representative high-churn files.
6. Static report drift from `scripts/compile_isolation_report.py`: LOC, inline tests, `async_trait`, and target-state dependency advisories.
## Target architecture
```mermaid
graph TD
bin["jcode binary\ntiny composition root"] --> cli["jcode-cli"]
bin --> tui["jcode-tui"]
bin --> server["jcode-server"]
bin --> providers["provider leaf crates"]
bin --> tools["tool leaf crates"]
cli --> api["jcode-client-api / app-api"]
tui --> api
server --> api
api --> protocol["protocol + view models"]
protocol --> types["small stable type crates"]
server --> agent["jcode-agent"]
server --> registry["jcode-tool-registry"]
server --> auth["jcode-auth-core"]
server --> session["jcode-session-core"]
server --> memory["jcode-memory-core"]
providers --> provider_core["jcode-provider-core"]
tools --> tool_core["jcode-tool-core"]
```
Rules:
- TUI and CLI depend on client API, protocol, view models, and small type crates, not full server/provider/tool implementations.
- Provider implementations are leaf crates. AWS/Bedrock dependencies live only in the Bedrock provider crate.
- Tool implementations are leaf crates. Heavy tools like PDF/browser/Gmail/search are isolated behind tool-core interfaces.
- Shared bottom crates are small and stable. Avoid putting high-churn behavior in protocol/type crates.
- Avoid broad `pub use whole_crate::*` compatibility ladders in final architecture.
## Migration sequence
### Phase 0: measurement and guardrails
Status: started.
Deliverables:
- `scripts/compile_time_probe.sh`
- `scripts/compile_isolation_report.py`
- this document
- dependency boundary checks/advisory reports
Success criteria:
- Every structural phase has before/after timing.
- The timing report makes the serial stack visible.
### Phase 1: widen the god-crate critical path
Split the three long-pole crates into sibling domain crates. Priority is widening the graph, not extracting more tiny type crates.
Likely first splits:
- From `jcode-base`:
- `jcode-auth-core`
- `jcode-session-core`
- `jcode-memory-core`
- provider implementation crates, especially Bedrock/AWS as a leaf
- From `jcode-app-core`:
- `jcode-server`
- `jcode-agent`
- `jcode-tool-registry`
- service crates for background/swarm/update/selfdev as needed
- From `jcode-tui`:
- `jcode-client-api` / view-model boundary first
- then move reusable client-side state logic out of the terminal rendering crate only when it creates a real parallel unit
Success criteria:
- Touching common TUI code no longer recompiles app-core/provider/server implementation crates.
- Touching a provider implementation no longer recompiles TUI or broad base code.
- Cargo timing shows multiple medium-sized Jcode crates running in parallel instead of one 4-deep mega-crate ladder.
### Phase 2: kill glob re-export ladders
Current compatibility layering preserves the old monolith shape:
```rust
pub use jcode_base::*;
pub use jcode_app_core::*;
pub use jcode_tui::*;
```
Migration approach:
1. Keep compatibility re-exports temporarily while moving code.
2. Convert high-churn modules to explicit imports from leaf crates.
3. Remove glob re-exports once downstream imports are explicit.
Success criteria:
- New code does not rely on whole-layer prelude-style re-exports.
- Dependency direction is visible in imports and Cargo manifests.
### Phase 3: move inline tests out of hot crates
Problem:
- Inline `#[cfg(test)]` modules make `cargo test` compile large production crates plus large test bodies as one rustc unit.
Target:
- Integration tests or dedicated `*-test-support` crates for broad behavior tests.
- Keep tiny unit tests inline only when they are genuinely local and cheap.
Success criteria:
- Targeted tests no longer require monolithic test cfg builds for unrelated domains.
### Phase 4: reduce front-end macro tax
Targets:
- Replace `async_trait` with native `async fn` in traits where the trait is not used as `dyn`.
- Keep `async_trait` only at object-safe plugin/interface boundaries where boxed futures are intentional.
- Avoid adding derive-heavy types to broad shared crates unless the type is stable and necessary.
Success criteria:
- Fewer proc-macro expansions in the hot crates.
- No object-safety regressions.
## Anti-goals
- Do not make fast debug builds incomplete by default.
- Do not split code into tiny crates unless the split creates a real invalidation or parallelism boundary.
- Do not move high-churn behavior into low-level type/protocol crates.
- Do not do a single giant rewrite. Each phase should build and be measurable.
## Validation checklist per phase
Before committing a phase:
```bash
scripts/compile_time_probe.sh --skip-build
scripts/compile_isolation_report.py
scripts/check_dependency_boundaries.py
cargo check --profile selfdev -p jcode --bin jcode
```
For code-moving phases, also run the relevant targeted tests for the moved domain, plus one full selfdev build through the coordinated selfdev path when practical:
```bash
selfdev build target=tui
```
docs/CRATE_OWNERSHIP_BOUNDARIES.md
+259
View File
@@ -0,0 +1,259 @@
# Crate Ownership and Modularization Boundaries
This document defines the target structure for keeping `jcode` modular without turning shared crates into a dumping ground. It is intentionally practical: use it when deciding whether to move a type, helper, or behavior out of the root crate.
## Goals
Primary goal: make normal development and selfdev builds faster by shrinking the root crate's recompilation surface. Structural cleanliness is valuable because it supports that compile-time goal.
- Move stable DTOs and protocol-safe state into small crates so changes in root behavior do not recompile those contracts, and changes in contracts recompile only focused dependents.
- Keep dependency-light crates dependency-light so they compile quickly and do not pull large runtime/TUI/provider graphs into unrelated builds.
- Keep root-only behavior, storage, process, TUI, server, and provider runtime logic in the root crate until a full dependency boundary can move without increasing dependency fan-out.
- Avoid cyclic dependencies and hidden coupling through broad `jcode-core` re-exports.
- Preserve serde compatibility and root re-exports during migrations unless all call sites are intentionally updated.
- Measure success by compile impact: fewer root edits, fewer root-owned DTOs, smaller dependency fan-out, and faster `cargo check --profile selfdev` / `selfdev build` after common changes.
## Ownership rules
### Type crates own stable data contracts
A `*-types` crate should contain:
- Plain data structures used by multiple crates or protocol layers.
- Serialization shape and small pure helper methods tied to the data contract.
- No filesystem, network, process, TUI, provider client, global state, or storage access.
- Dependencies limited to serde, chrono, and other type crates where necessary.
Examples: `jcode-session-types`, `jcode-side-panel-types`, `jcode-selfdev-types`, `jcode-background-types`.
### Domain behavior modules own root runtime behavior
Root modules should keep behavior when it needs:
- `crate::storage`, `crate::config`, `crate::logging`, `crate::server`, or process spawning.
- Provider HTTP clients and auth managers.
- Tokio runtime, background tasks, channels, global caches, file locks, or PID registries.
- TUI rendering and crossterm/ratatui state.
If a type has inherent methods that need these APIs, either leave the type in root or move behavior and dependencies together into a domain crate. Do not move only the struct if that forces illegal inherent impls in root.
### `jcode-core` is for genuinely shared primitives
`jcode-core` should contain:
- Cross-domain primitives that do not have an obvious domain crate yet.
- Very small, dependency-light helpers used by many crates.
- Temporary DTO staging only when creating a new domain type crate would be premature.
`jcode-core` should not accumulate every extracted DTO indefinitely. Once a cluster grows, split it into a focused domain crate.
### Compile-speed decision rule
Prefer a split when it reduces root crate churn or dependency fan-out. Do not split just to make files look tidier if the new crate adds dependencies, increases rebuild fan-out, or forces frequent cross-crate edits. A good split has at least one of these compile-time benefits:
- Common root behavior edits no longer touch stable type definitions.
- A type-only change can be checked by compiling a small type crate plus focused dependents.
- Heavy dependencies stay out of DTO crates.
- Multiple downstream crates can use a small contract without depending on the root crate.
### Re-export policy
During migrations:
1. Move the type to the target crate.
2. Keep the old root path as `pub use ...` to preserve call sites.
3. Validate focused tests and selfdev build/reload.
4. Later, remove obsolete root re-exports only after downstream crates can depend directly on the domain crate.
## Move checklist
Use this checklist for every type or pure-helper migration. Copy it into the PR/commit notes when a move is non-trivial.
1. Classify the candidate.
- [ ] Is it a stable data contract or pure helper rather than root runtime behavior?
- [ ] Does it have inherent methods?
- [ ] Do those methods require root-only APIs such as storage, network clients, TUI state, process management, or globals?
- [ ] If behavior must move too, can the full dependency boundary move without increasing fan-out?
2. Check compatibility.
- [ ] Does its serde representation stay identical?
- [ ] Are defaults, skips, renames, and enum discriminants preserved?
- [ ] Are all field visibilities still appropriate?
- [ ] Can root keep a compatibility re-export?
3. Check crate health.
- [ ] Does the target crate already have the needed dependency policy?
- [ ] Are new dependencies limited to type-crate-appropriate libraries, usually `serde`, `serde_json`, `chrono`, or sibling type crates?
- [ ] Is the target crate still acyclic?
- [ ] Did `cargo metadata`/`cargo check` avoid pulling root, TUI, provider, storage, server, or process dependencies into the type crate?
4. Validate.
- [ ] Is there a focused test filter that covers the moved type?
- [ ] Did `cargo check --profile selfdev -p <type-crate> -p jcode --bin jcode` pass?
- [ ] Did relevant focused root tests pass?
- [ ] Did `cargo fmt` pass?
- [ ] Did selfdev build and reload pass from a clean committed HEAD?
## Dependency boundary guard
Run this guard after adding or changing any type crate dependency:
```sh
python3 scripts/check_dependency_boundaries.py
```
The guard blocks direct dependencies from `jcode-*-types` crates to root/runtime-heavy internal crates such as `jcode`, `jcode-core`, provider crates, TUI crates, protocol/runtime crates, and desktop/mobile crates. Type crates may depend on external lightweight libraries and other type crates. If a new internal dependency is needed, first decide whether it should itself be a type crate.
## Test policy
Prefer focused filters for validation. Broad filters often select unrelated stateful, timing-sensitive, or benchmark tests.
Known broad-filter hazards observed during modularization:
- `side_panel` selects unrelated pinned UI/layout and latency benchmark tests.
- `usage` selects app-display tests in addition to pure usage tests.
- `session::` selects live-attach server tests and picker behavior beyond session persistence.
- `ambient` selects TUI/helper integration tests with config and schedule state beyond ambient module persistence/runtime tests.
Document precise filters next to each domain crate/module. Broad filters are still useful for periodic sweeps, but they should not block a DTO-only extraction when precise tests and compile checks pass.
Focused validation matrix after the current DTO splits:
| Area | Fast compile check | Focused root tests used during split | Notes |
| --- | --- | --- | --- |
| Usage DTOs | `cargo check --profile selfdev -p jcode-usage-types -p jcode --bin jcode` | Prefer exact tests under usage/copilot usage modules. Avoid bare `usage` as a required gate because it selects display/UI tests too. | DTO crate owns report and local counter contracts. Runtime fetch/cache/display stay root. |
| Gateway DTOs | `cargo check --profile selfdev -p jcode-gateway-types -p jcode --bin jcode` | Focus gateway persistence/auth tests by exact test names when available. | Pairing/token HTTP/WebSocket behavior stays root. |
| Ambient DTOs | `cargo check --profile selfdev -p jcode-ambient-types -p jcode --bin jcode` | Scheduler/type consumers only. | Ambient DTO crate owns usage records only. Queue/runtime/prompt behavior stays root. |
| Ambient behavior modules | `cargo check --profile selfdev -p jcode --bin jcode` | `cargo test --profile selfdev -p jcode ambient::ambient_tests --lib`; `cargo test --profile selfdev -p jcode ambient::scheduler::tests --lib`; `cargo test --profile selfdev -p jcode ambient::runner::runner_tests --lib` | Avoid bare `ambient` as a required gate for module-only refactors because it selects cross-module TUI/config state tests. |
| Memory activity DTOs | `cargo check --profile selfdev -p jcode-memory-types -p jcode-core -p jcode --bin jcode` | `cargo test --profile selfdev -p jcode runtime_memory_log --lib`; `cargo test --profile selfdev -p jcode tui::info_widget::tests --lib` | `memory::activity` currently matches no tests, so use consumer tests. |
| Goal/todo/catchup core DTOs | `cargo check --profile selfdev -p jcode-core -p jcode --bin jcode` | Exact goal/todo/catchup filters if behavior changes. | Currently small/stable enough to leave in `jcode-core`; revisit if churn grows. |
## Compile baseline observations
Measured on 2026-04-30 with `scripts/dev_cargo.sh check --profile selfdev -p jcode --bin jcode` after the compile-speed boundary doc commit. This is a coarse mtime-touch benchmark, not a full statistical study, but it is enough to guide priorities.
| Scenario | Observed time | Interpretation |
| --- | ---: | --- |
| No-op check after recent doc-only commit | ~65.8s | Environment/cache state can dominate a first check. Treat as warmup/noise baseline, not pure no-op steady state. |
| Touch root behavior module `src/usage.rs` | ~6.25s | A root-only behavior edit can be relatively cheap when dependencies are already built. |
| Touch `crates/jcode-core/src/usage_types.rs` | ~65.35s | Editing `jcode-core` invalidates broad downstream dependents. Avoid adding high-churn domain DTOs to `jcode-core`. |
Implication: the compile-speed target is not simply "move things out of root". Moving stable, low-churn contracts out of root is good, but putting many high-churn domain DTOs into `jcode-core` can be counterproductive because `jcode-core` has high fan-out. Prefer focused leaf crates such as `jcode-usage-types`, `jcode-gateway-types`, and `jcode-ambient-types` for domain DTOs that are likely to change.
## `jcode-core` fan-out audit
At this checkpoint, the root crate is the only direct Cargo dependency on `jcode-core`, but root re-exports many `jcode-core` modules and root is the high-cost recompilation target. A touch to `jcode-core` invalidated broad downstream checks in the baseline above. Therefore `jcode-core` should be treated as a high-fan-out crate even if Cargo.toml direct dependents are currently few.
Observed root re-export/use paths:
- `src/catchup.rs` -> `catchup_types`
- `src/goal.rs` -> `goal_types`
- `src/todo.rs` -> `todo_types`
- `src/env.rs`, `src/id.rs`, `src/stdin_detect.rs`, `src/util.rs`, and panic UI helpers -> general utilities
Compile-speed priority from this audit:
1. Move clustered, likely-changing domain DTOs from `jcode-core` to focused leaf crates.
2. Keep stable general utilities in `jcode-core`.
3. Avoid adding new domain DTOs to `jcode-core` unless they are very stable or temporary staging.
| Module | Current contents | Preferred long-term home | Notes |
| --- | --- | --- | --- |
| `ambient_usage_types` | Ambient scheduler usage records/rate limit DTOs | moved to `jcode-ambient-types` | Compatibility re-export remains in root module. |
| `catchup_types` | Catch-up persisted state and rendered brief DTOs | `jcode-catchup-types` or stay in core | Small and low churn. Split only if catch-up grows. |
| `copilot_usage_types` | Local Copilot usage counters | moved to `jcode-usage-types` | Compatibility re-export remains in root module. |
| `gateway_types` | Paired device and pairing code persisted records | moved to `jcode-gateway-types` | Pairing/token behavior remains root. |
| `goal_types` | Goal state, milestones, status, updates | `jcode-goal-types` or `jcode-task-types` | Larger domain. Worth splitting if goal/tool work grows. |
| `memory_types` | Memory activity DTOs | moved to `jcode-memory-types` | Memory has enough domain weight for its own type crate. |
| `todo_types` | Todo item DTO | `jcode-task-types`, `jcode-todo-types`, or core | Tiny. Could join goal/catchup task-state crate. |
| `usage_types` | Provider usage report DTOs | moved to `jcode-usage-types` | Runtime fetch/cache/display remain root. |
| `env` | Environment variable helpers | stay in core | General utility, no domain crate needed. |
| `id` | ID helpers | stay in core | General utility. |
| `panic_util` | Panic formatting helpers | stay in core | General runtime utility. |
| `stdin_detect` | stdin detection helpers | stay in core | General platform/runtime utility. |
| `util` | Misc utilities | audit later | Should not become a catch-all. |
## Target domain type crates
Completed/high-value domain type splits:
1. `jcode-usage-types`
- `usage_types`
- `copilot_usage_types`
- pure account usage DTOs if/when separated from root formatting/runtime helpers
2. `jcode-gateway-types`
- `gateway_types`
- possibly `GatewayConfig` after deciding whether config owns it
- mobile gateway protocol-safe DTOs if needed by mobile crates
3. `jcode-ambient-types`
- `ambient_usage_types`
- ambient state/request/result DTOs, but only after root-only `AmbientState::load/save/record_cycle` methods are separated into root free functions or a persistence layer
4. `jcode-memory-types`
- `memory_types`
- any memory protocol/activity DTOs used across server/TUI/tools
5. Optional task-state crate
- `goal_types`
- `todo_types`
- `catchup_types` if the product model wants these grouped
## Big module refactor targets
These are not simple DTO moves. Refactor behavior boundaries first.
### `src/session.rs`
Target split:
- metadata/session model
- persistence and journal replay
- startup stubs and remote startup snapshots
- memory profiling/cache attribution
- rendering lives in existing `session/render.rs`
- crash recovery lives in existing `session/crash.rs`
### `src/ambient.rs`
Target split:
- visible cycle context I/O
- state persistence
- directive persistence
- schedule queue and locking
- prompt building
- manager/runtime orchestration
Do not move `AmbientState` as a DTO until load/save/record behavior is separated from the struct.
### `src/usage.rs`
Target split:
- API fetch providers
- provider response parsing
- local caches/sync
- display formatting
- account selection/guidance
- public report DTOs in `jcode-usage-types`
### `src/gateway.rs`
Target split:
- registry persistence
- pairing/token auth
- HTTP route handling
- WebSocket auth/extraction
- WebSocket relay
- public gateway DTOs in `jcode-gateway-types`
## Definition of “optimal enough”
The structure is good enough when:
- Each type crate has a clear domain and minimal dependency set.
- `jcode-core` contains only true primitives or documented temporary staging modules.
- Root modules no longer mix large DTO blocks, persistence, runtime orchestration, and rendering in one file.
- Every domain has focused validation commands.
- Selfdev build/reload works cleanly after every structural change.
docs/DESKTOP_APP_ARCHITECTURE.md
+486
View File
@@ -0,0 +1,486 @@
# Jcode Desktop Architecture Direction
Status: Proposed
Updated: 2026-04-25
This document captures the initial direction for a desktop application for Jcode under these constraints:
- no Electron/Tauri/web-app shell
- no general UI framework
- very high performance
- low idle resource use
- very custom product UI
- primary developer machine may be Linux
- most early users are expected to be on macOS
The goal is to make the desktop client a first-class Jcode surface without forking the Jcode runtime or turning the app into a heavyweight IDE clone.
See also:
- [`DESKTOP_STABLE_HOST_RELOAD_STARTUP.md`](./DESKTOP_STABLE_HOST_RELOAD_STARTUP.md)
- [`DESKTOP_SUPERAPP_WORKSPACE.md`](./DESKTOP_SUPERAPP_WORKSPACE.md)
- [`DESKTOP_CODEBASE_ARCHITECTURE.md`](./DESKTOP_CODEBASE_ARCHITECTURE.md)
- [`CLIENT_CORE_PRESENTATION_SPLIT_PLAN.md`](./CLIENT_CORE_PRESENTATION_SPLIT_PLAN.md)
- [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md)
- [`SERVER_ARCHITECTURE.md`](./SERVER_ARCHITECTURE.md)
- [`MEMORY_BUDGET.md`](./MEMORY_BUDGET.md)
## Executive summary
Build Jcode Desktop as a small Rust desktop client with a custom GPU-rendered UI. The app should connect to a local Jcode server/daemon that owns sessions, tools, agent execution, persistence, and permissions.
The frontend should be optimized as a render/input surface:
- Linux should be a first-class development platform.
- macOS should be the first-class product/distribution platform.
- The UI should not depend on Linux-only desktop concepts.
- The UI should not be a web view.
- The UI should not embed the agent runtime directly.
- Rendering should be on-demand, virtualized, and measurable from day one.
Recommended initial stack:
| Area | Decision |
|---|---|
| Frontend language | Rust |
| Backend/runtime | Existing Rust Jcode server/session runtime |
| Process model | Desktop frontend + local Jcode daemon/server |
| Window/input layer | Thin platform layer, likely `winit` initially |
| Rendering | `wgpu` with a custom 2D renderer |
| UI architecture | Retained UI tree with dirty tracking |
| Layout | Small custom layout system, not CSS/DOM |
| Text | Dedicated text layout/raster cache, likely `cosmic-text`/`swash` or platform-backed text later |
| Protocol | Versioned typed local event protocol |
| Persistence | Server-owned session/event persistence |
| Product identity | Agent operating console / mission control |
## Product stance
Jcode Desktop should not start as a full IDE and should not look like a conventional chatbot.
The differentiated product is a **keyboard-driven, Niri-like agent workspace superapp** for local development. The first-class object is not a chat window, but a workspace containing many navigable surfaces:
- agent sessions
- activity/task views
- diffs and changed files
- file/diff/tool surfaces
- optional future surfaces
- settings/debug/tool surfaces
The app should help users:
- supervise autonomous coding work
- inspect tool activity
- manage background tasks
- review changed files
- respond to permission prompts
- resume and coordinate sessions
- navigate many related surfaces spatially
The desktop client should complement the TUI/CLI, not replace it.
## Platform strategy
### Development host: Linux
Linux should support the fastest inner loop:
- launch the desktop client locally
- run renderer stress tests
- run protocol integration tests
- benchmark memory/frame/layout/text performance
- debug the UI engine without a Mac in the loop
The Linux build should be real, not a fake simulator. It should render through the same UI engine and exercise the same protocol/view-model paths as macOS.
### Product target: macOS first
Most early users are expected to be on macOS, so macOS polish should be a product requirement even if day-to-day development happens on Linux.
Mac-specific work that should not be postponed too long:
- native `.app` bundle
- app icon and menu bar integration
- command-key shortcuts
- system light/dark appearance
- Retina rendering correctness
- trackpad scrolling quality
- native clipboard behavior
- file/open-with integration
- code signing and notarization path
- good behavior under Mission Control, Spaces, and full-screen windows
### Avoid Linux-shaped product assumptions
Because the developer may use Linux, the architecture should explicitly avoid baking in assumptions that work well only with a Linux window manager.
Do not make these hard dependencies:
- Niri-style external spatial window management
- X11-specific APIs
- Wayland-only behavior
- terminal-first session workflows
- Linux notification semantics
- global shortcuts that are unavailable or hostile on macOS
The existing Linux/Niri workflow should remain excellent, but desktop product quality should be judged primarily against macOS expectations.
## Process architecture
Use a split process architecture:
```text
Jcode Desktop Frontend
- window/input
- custom rendering
- local view model
- transient UI state
- surface-local state
- protocol client
Jcode Server/Daemon
- sessions
- agent runtime
- tool runtime
- background tasks
- persistence
- permissions
- model/provider configuration
```
The server remains the source of truth for:
- canonical session history
- streaming events
- tool execution
- file edits
- background tasks
- permission state
- persisted configuration
The desktop frontend owns only surface-local state:
- selected session/surface
- draft input
- cursor and text selection
- scroll offsets
- pane sizes
- focused panel
- local command palette state
- render caches
This aligns with the multi-session model where a server-owned session can be shown by different clients or surfaces over time.
## Local protocol direction
The desktop app should consume a versioned, typed event stream rather than periodically fetching complete session snapshots.
Early protocol properties:
- local-first transport
- explicit protocol version
- capability negotiation
- append-only session events
- streaming deltas for assistant/tool output
- resumable subscriptions by event cursor
- compact events for high-volume tool output
- server-owned permission requests
Possible transports:
1. Existing Jcode server channel, if compatible with desktop needs.
2. Unix domain socket on Linux/macOS and named pipe on Windows.
3. Stdio JSON protocol for early prototypes and test harnesses.
Avoid localhost HTTP as the default unless there is a strong reason. It creates a larger local security surface than a user-owned socket/pipe.
Example event families:
```text
session.created
session.updated
surface.attached
message.created
message.delta
message.completed
tool.started
tool.output.delta
tool.completed
task.started
task.progress
task.completed
workspace.changed
git.changed
permission.requested
permission.resolved
error
```
## Rendering architecture
Use a custom renderer rather than a native widget hierarchy or web view.
Recommended layers:
```text
Platform window/input
-> input normalizer
-> app state/view model
-> retained UI tree
-> layout
-> text layout/cache
-> display list
-> GPU renderer
```
Core rules:
- no continuous render loop when idle
- render only on input, data events, animations, or explicit invalidation
- virtualize every unbounded list
- separate layout cost from paint cost
- cache shaped text by content/font/width
- use stable IDs for dirty tracking
- make debug/performance counters visible in-app
The renderer should initially support:
- rectangles
- rounded rectangles
- borders
- solid fills
- clipping
- scroll containers
- text runs
- monospaced blocks
- simple icons or vector-like primitives
- image support later
Defer:
- blur effects
- complex shadows
- animation framework
- SVG-heavy rendering
- full markdown renderer
- full terminal emulator
- embedded code editor
## UI architecture
Use a retained UI tree with immediate-style builder ergonomics.
Rationale:
- transcripts are long-lived and streamed incrementally
- tool outputs can be large
- panes need stable focus/selection state
- dirty tracking matters for resource use
- accessibility will eventually need stable semantic nodes
- multi-session surfaces need stable identity
The model should not imitate the DOM/CSS stack. A small product-specific layout system is enough:
- row
- column
- stack
- split pane
- fixed size
- flex fill
- scroll container
- virtual list
- overlay/modal
- intrinsic text measurement
## Text strategy
Text is one of the hardest parts of this project and should be treated as a core system, not a detail.
The desktop client needs:
- Unicode shaping
- font fallback
- monospace code/tool output
- wrapping
- incremental append layout
- selection/copy
- input cursor behavior
- command palette text input
- markdown-ish transcript styling
- ANSI-like tool output styling eventually
Initial recommendation:
- use a Rust text stack such as `cosmic-text`/`swash` if dependency review is acceptable
- maintain a GPU glyph atlas
- cache shaped lines/runs by stable block ID and available width
- specialize streamed append paths so new output does not re-layout the whole transcript
Mac-specific text quality should be evaluated early. If Rust text rendering is not good enough on macOS, consider platform-backed text for macOS while preserving the same higher-level text layout API.
## Performance and resource budgets
Initial budgets should be measured on both Linux development machines and representative macOS hardware.
| Metric | MVP target | Long-term target |
|---|---:|---:|
| Cold launch to visible window | < 500 ms | < 150 ms |
| Frontend idle CPU | ~0% | ~0% |
| Frontend idle RSS | < 100 MiB | < 50 MiB |
| Input-to-paint latency | < 32 ms | < 16 ms |
| Scrolling | 60 fps | 120 fps-capable |
| Fake transcript stress case | 100k blocks usable | 100k blocks smooth |
| Full transcript re-layout on append | forbidden | forbidden |
| Unbounded retained visible nodes | forbidden | forbidden |
| Renderer frame when idle | forbidden | forbidden |
Required early instrumentation:
- frame time
- layout time
- text shaping time
- display-list build time
- GPU submit time
- visible node count
- total retained node count
- glyph atlas size
- text cache size
- protocol event backlog
- daemon round-trip latency
- frontend RSS if available
A debug HUD should exist in the prototype before real Jcode integration is considered complete.
Example HUD:
```text
frame 1.8ms | layout 0.3ms | text 0.6ms | gpu 0.4ms
nodes 812 | visible 47 | glyph atlas 12.4 MiB | events 0 | daemon 2ms
```
## MVP scope
The first UI milestone should prove the engine before proving every product workflow.
### Milestone 1: custom shell with fake data
Success criteria:
- launches a native desktop window from Linux
- renders through the custom GPU pipeline
- shows session sidebar, transcript, composer, and activity panel
- handles mouse, keyboard, focus, and scrolling
- renders fake streamed transcript data
- virtualizes a 100k-block transcript
- idles at near-zero CPU
- exposes performance/debug HUD
- has screenshot or golden-state tests where practical
### Milestone 2: protocol connection
Success criteria:
- connects to local Jcode server/daemon
- lists sessions
- attaches to a session/surface
- subscribes to event stream
- sends a user prompt
- streams assistant/tool events into the transcript
- can stop/cancel an active run
- recovers from daemon restart or disconnect gracefully enough for development use
### Milestone 3: useful agent console
Success criteria:
- activity center for background tasks/tool calls
- permission request overlay
- workspace/git status panel
- changed-file list
- open external editor/diff action
- session search/filter
- macOS app bundle prototype
## Crate layout proposal
Do not put the whole desktop app in the root crate.
Suggested structure:
```text
crates/
jcode-desktop-protocol/ # shared protocol/event types if not already covered by server types
jcode-desktop-ui/ # UI tree, layout, text/cache abstractions, renderer-agnostic pieces
jcode-desktop-renderer/ # wgpu renderer and GPU resources
jcode-desktop/ # app shell, platform window, protocol client, product UI
```
If compile time becomes a problem, keep protocol/UI crates lightweight and gate GPU/window dependencies behind the final app crate.
## Dependency policy
“No frameworks” does not have to mean “no libraries.” It should mean no heavyweight app framework and no web-shell product architecture.
Likely acceptable dependencies:
- `wgpu` for rendering abstraction
- a very thin window/input layer such as `winit` for bootstrapping
- `cosmic-text`/`swash` or equivalent for text shaping/rasterization
- small serialization/protocol crates already consistent with Jcode
Avoid:
- Electron
- Tauri
- Qt
- Flutter
- GTK as the app framework
- WebView UI shell
- React/Vue/Svelte-style UI stack
- CSS/DOM-based architecture
If `winit` becomes limiting for macOS polish, the platform layer can grow direct AppKit support while preserving the renderer and UI model.
## macOS validation checklist
Because macOS is the primary user target, validate these early even if development happens on Linux:
- Retina scale factor correctness
- trackpad inertial scrolling
- text clarity compared with native apps
- keyboard shortcuts use Command rather than Control where appropriate
- system dark/light mode follows user preference
- window resizing and full-screen behavior feels native
- app menu and close/minimize/quit semantics are correct
- clipboard round-trips rich enough for code and transcripts
- local socket permissions are safe
- app bundle can launch/find the daemon reliably
## Open decisions
These should be resolved before implementation moves past the fake-data prototype:
1. Use `winit` initially or write direct platform shells from the start?
2. Use `wgpu` or direct Metal-first rendering?
3. Use `cosmic-text`/`swash` or platform text APIs?
4. Reuse the existing Jcode server protocol or introduce a desktop-specific event protocol crate?
5. Should the first desktop binary support multi-surface mode or only one active surface?
6. What is the minimum macOS version to support?
7. What is the first distribution path: local `.app`, Homebrew cask, or signed/notarized DMG?
## Recommended immediate next step
Create a fake-data desktop prototype that runs on Linux but measures the exact performance characteristics required by the eventual macOS product.
The prototype should not wait for a perfect daemon API. It should validate the expensive UI systems first:
- window creation
- renderer startup
- retained tree
- layout
- text cache
- virtualized transcript
- on-demand repaint
- debug HUD
Only after that should the real Jcode event stream be connected.
docs/DESKTOP_BUILD_OUT_PLAN.md
+155
View File
@@ -0,0 +1,155 @@
# Jcode Desktop Build-Out Plan
Status: Active planning note
Updated: 2026-05-25
## Current read
Jcode Desktop has moved past the original blank-canvas prototype. The strongest current thread is a single-session desktop surface with a local GitHub issue browser/cache and background `gh` sync. Recent work added:
- issue-browser renderer extraction
- issue-browser interactions
- local GitHub issue cache
- background issue sync UI state
- tests for issue navigation, investigation prompt injection, cache ranking, and sync failure behavior
The product docs still point at the long-term goal: a keyboard-driven, Niri-like local AI development superapp. The right next move is not to jump directly to a full IDE or complete workspace manager. Build outward from the proven single-session surface into issue-driven agent workflow and reusable surface primitives.
## Recommended product sequence
### 1. Finish the GitHub issue workflow until it is genuinely useful
This is the most valuable near-term build-out because it connects desktop UI, repo context, agent launch, and an actionable developer workflow.
Build next:
1. Issue sync polish
- visible auth/install guidance when `gh` is missing or unauthenticated
- sync timestamp and cache path surfaced in the browser
- manual refresh shortcut help in the UI
- bounded background sync status that cannot look stuck
2. Issue filtering and triage
- filter by label/state/text
- quick priority override persisted in `local_overrides`
- hide/done/local-dismiss issue affordance
3. Issue-to-agent workflow
- start a new session from selected issue
- inject structured issue context, comments, labels, and repo facts
- optionally spawn an implementation agent and a review/check agent
4. Issue activity loop
- show running sessions associated with issue numbers
- show changed files/test status once an agent acts
- jump from issue browser to the relevant session
Verification:
- `cargo test -p jcode-desktop issue --no-default-features`
- fake `gh` runner tests for auth/missing CLI/comment failures
- UI snapshot/layout tests for empty, syncing, error, and narrow-width states
### 2. Extract generic surface/workspace primitives from single-session state
Do this after the issue workflow has one concrete non-chat side surface. Avoid speculative architecture, but create the minimum model needed to compose session plus issue/activity surfaces.
Build next:
1. `SurfaceId`, `SurfaceKind`, `SurfaceState`
2. one lane with horizontal columns
3. focus movement left/right
4. zoom/unzoom focused surface
5. persistent layout per repo/workspace
6. surface command routing independent of renderer
Start with these surface kinds:
- `AgentSession`
- `GitHubIssues`
- `Activity`
- `DiffSummary` or `ChangedFiles`
Verification:
- pure state-machine tests for focus, move, close, zoom, and persistence
- renderer tests proving existing single-session surface still composes without regressions
- keyboard-routing tests separating navigation mode from text-entry mode
### 3. Build an activity/mission-control surface
The desktop app should excel at supervising autonomous work. Activity should become a first-class surface rather than a small status panel.
Build next:
1. stream current session status, background tasks, tool runs, and build/test jobs
2. filter by workspace/session/issue/tool type
3. select activity item to jump to session/tool output
4. show blocked permission prompts prominently
5. show recently completed validation results
Verification:
- synthetic event batch tests for high-volume updates
- no busy render loop while idle
- activity jump tests into session transcript/tool cards
### 4. Add Level-1 files/diff surfaces, not a full editor
Desktop needs review affordances before it needs editing.
Build next:
1. changed-files list
2. read-only file preview
3. read-only unified diff preview
4. open path/diff in external editor
5. copy path/snippet
6. link changed files back to sessions/issues
Verification:
- large diff/file virtualization tests
- binary/large-file fallbacks
- external-open command tests behind safe mocks
### 5. Command palette and shared command registry
Once there are multiple surfaces, the command palette should be the universal router.
Build next:
1. command registry for desktop commands
2. fuzzy palette over commands, sessions, issues, files, and surfaces
3. keyboard shortcut discovery from the same registry
4. palette actions with safe confirmation classification
Verification:
- registry uniqueness tests
- fuzzy ranking tests
- all visible shortcuts generated from registered commands
## Explicit non-goals for the next tranche
Do not build these yet:
- full code editor
- plugin/extension system
- general desktop window manager behavior
- elaborate multi-window support
- large settings/auth UI beyond what unblocks issue sync and model/session workflow
- webview/Electron/Tauri shell
## Next concrete implementation slice
The best immediate slice is:
> Make `/issues` a robust issue-to-agent launcher.
Definition of done:
1. Opening `/issues` loads cache immediately and starts a background sync when appropriate.
2. Empty/error states explain how to install/authenticate `gh`.
3. Pressing the investigate/start key opens or spawns a real session with full structured issue context.
4. The issue browser shows sync status, last sync, comment fetch failures, and selected issue context clearly.
5. Tests cover success, missing `gh`, auth failure, sync partial failure, empty cache, and narrow layout.
This slice is small enough to ship, but it moves the desktop app toward its core product identity: issue-driven agent mission control.
docs/DESKTOP_CODEBASE_ARCHITECTURE.md
+778
View File
@@ -0,0 +1,778 @@
# Desktop Codebase Architecture from the Existing TUI
Status: Proposed
Updated: 2026-04-25
This document translates the current Jcode TUI architecture into a concrete codebase plan for a future custom desktop app.
The desktop app is expected to have roughly the same product capabilities as the TUI, but it should not be a direct port of the TUI implementation. The TUI is terminal/cell-oriented and has accumulated a large amount of terminal-specific rendering, input, layout, scrolling, and cache logic. The desktop app should reuse the runtime/protocol/session concepts and some presentation models, but it should have a separate custom UI and rendering architecture.
See also:
- [`DESKTOP_APP_ARCHITECTURE.md`](./DESKTOP_APP_ARCHITECTURE.md)
- [`DESKTOP_SUPERAPP_WORKSPACE.md`](./DESKTOP_SUPERAPP_WORKSPACE.md)
- [`CLIENT_CORE_PRESENTATION_SPLIT_PLAN.md`](./CLIENT_CORE_PRESENTATION_SPLIT_PLAN.md)
- [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md)
- [`SERVER_ARCHITECTURE.md`](./SERVER_ARCHITECTURE.md)
## Current TUI observations
The current TUI is feature-rich and should be treated as the product reference implementation.
Approximate current size:
```text
src/tui/*.rs and submodules: 144 Rust files, ~115k lines
src/tui/app.rs: ~800 lines
src/tui/ui.rs: ~3.8k lines
src/tui/ui_prepare.rs: ~1.6k lines
src/tui/ui_viewport.rs: ~750 lines
src/tui/ui_messages.rs: ~2.4k lines
src/tui/markdown.rs: ~1.4k lines
src/protocol.rs: ~1.4k lines
```
Important existing pieces:
- `src/protocol.rs`
- newline-delimited JSON over Unix socket
- `Request`
- `ServerEvent`
- session subscribe/resume/history/message/cancel/tool/status events
- `src/server/`
- multi-client server/session runtime
- reconnect support
- session lifecycle
- client events
- background tasks/swarm/comm state
- `src/tui/app.rs` and `src/tui/app/*`
- TUI app state root
- input handling
- remote connection handling
- command parsing
- server event reducer
- local mode support
- copy/selection/navigation/session picker/debug overlays
- `src/tui/ui.rs` and `src/tui/ui_*`
- ratatui renderer
- terminal/cell layout
- viewport and scroll behavior
- side panes
- overlays
- visual debug capture
- `src/tui/ui_prepare.rs`
- frame preparation
- wrapped line maps
- full prep cache
- body/streaming/batch preparation
- `src/tui/ui_messages.rs`
- message-to-terminal-line rendering
- tool/system/background/swarm usage rendering
- line caches
- `src/tui/markdown.rs`
- terminal markdown rendering
- syntax highlighting cache
- mermaid integration hooks
## Key lesson
The TUI already has the right **feature set** and many correct **domain concepts**, but it does not yet have the right boundaries for a custom desktop UI.
The desktop should not import `ratatui::Line`, terminal-width wrapping, global renderer caches, or terminal input assumptions into core app state.
The desktop should instead use this split:
```text
Jcode server/runtime/protocol
-> client-core reducer and view model
-> desktop product views
-> custom UI tree/layout
-> display list
-> wgpu renderer
```
## What to reuse versus not reuse
### Reuse directly or almost directly
- server process architecture
- session ownership model
- reconnect semantics
- request/event protocol as the starting point
- server-side session history and tool execution
- model/provider/session metadata
- command concepts
- background task concepts
- swarm/goal/activity concepts
- permission concepts
- debug/diagnostic philosophy
### Reuse after extracting away terminal types
- server event reduction logic from `src/tui/app/remote/server_events.rs`
- message display block construction
- tool call summaries
- activity/status models
- markdown block parsing decisions
- copy target semantics
- session picker data model
- login/account picker data model
- command registry and command metadata
- info widget data models
- memory/debug summary models
### Do not reuse directly
- `ratatui::Line` as a cross-surface representation
- terminal cell layout
- terminal-specific scroll offsets as the primary desktop model
- global renderer state such as `LAST_MAX_SCROLL`-style globals
- terminal key protocol code
- terminal-specific markdown wrapping
- terminal-specific image/mermaid display code
- the giant `TuiState` trait as the desktop boundary
- monolithic `App` state with runtime, transport, UI, and render concerns mixed together
## The main architectural risk
If desktop development copies the TUI structure directly, the result will likely be:
- another very large `DesktopApp` state object
- rendering caches mixed with domain state
- platform input handling mixed with session reducers
- duplicated command logic
- duplicated server event handling
- hard-to-test UI behavior
- difficulty sharing behavior between TUI and desktop
The desktop app should avoid repeating this by creating a real client-core boundary before implementing too many features.
## Proposed crate/module architecture
The exact crate names can change, but the dependency direction should not.
```text
crates/
jcode-protocol/ # eventually extracted from src/protocol.rs
jcode-client-core/ # surface-independent client state/reducers/view models
jcode-desktop-ui/ # custom UI tree, layout, input routing, style tokens
jcode-desktop-renderer/ # wgpu renderer, display list, glyph/image atlases
jcode-desktop-platform/ # winit/AppKit/Linux shell abstraction, menus, clipboard
jcode-desktop/ # product app: windows, panels, protocol client, composition
```
Initial implementation may keep some of these as modules inside `crates/jcode-desktop` to reduce early friction, but the boundaries should be designed as if they were separate crates.
## Dependency direction
Allowed dependency flow:
```text
jcode-desktop
-> jcode-desktop-platform
-> jcode-desktop-renderer
-> jcode-desktop-ui
-> jcode-client-core
-> jcode-protocol
```
`jcode-client-core` must not depend on:
- `wgpu`
- `winit`
- AppKit
- Wayland/X11
- `ratatui`
- `crossterm`
- terminal markdown rendering
`jcode-desktop-ui` should not depend on the Jcode server runtime. It can depend on client-core view models and generic UI types.
`jcode-desktop-renderer` should not know what a Jcode session is. It renders display lists, text runs, images, clips, and primitives.
## `jcode-protocol`
The existing `src/protocol.rs` is already the natural starting point.
Long-term, extract it into a crate so both TUI and desktop consume the same wire types:
```text
crates/jcode-protocol/src/lib.rs
Request
ServerEvent
HistoryMessage
SessionActivitySnapshot
FeatureToggle
SwarmMemberStatus
protocol version/capability handshake types
```
Short-term, desktop can import the root crate types, but the protocol should be treated as shared API.
Desktop-specific protocol needs may include:
- session list with metadata optimized for a sidebar
- event cursors for resumable subscriptions
- richer activity snapshots
- workspace/git summary snapshots
- permission request snapshots
- changed-file summaries
- surface/window attachment metadata
Avoid making a second unrelated desktop protocol unless the existing protocol becomes a blocker.
## `jcode-client-core`
This is the most important shared layer.
It should own behavior and state that are independent of the terminal and independent of desktop rendering.
Suggested modules:
```text
jcode-client-core/
src/
lib.rs
app_model.rs
actions.rs
reducer.rs
protocol_reducer.rs
command_registry.rs
session_list.rs
transcript.rs
transcript_blocks.rs
composer.rs
activity.rs
permissions.rs
workspace.rs
git.rs
settings.rs
overlays.rs
selection.rs
status.rs
diagnostics.rs
view_model.rs
```
### Core state slices
```rust
struct ClientCore {
sessions: SessionListState,
active_surface: Option<SurfaceId>,
surfaces: SurfaceMap,
connection: ConnectionState,
commands: CommandRegistry,
activity: ActivityState,
permissions: PermissionState,
workspace: WorkspaceState,
diagnostics: DiagnosticsState,
}
```
Each surface owns local UI state:
```rust
struct SurfaceState {
session_id: SessionId,
transcript: TranscriptState,
composer: ComposerState,
selection: SelectionState,
scroll: ScrollState,
focused_region: FocusRegion,
overlays: OverlayStack,
pane_layout: PaneLayoutState,
}
```
The important rule:
> Server-owned session state and surface-local UI state must remain separate.
This matches the existing multi-session architecture docs and makes desktop multi-window/multi-surface possible later.
### Actions and reducers
Desktop and TUI should eventually share reducer logic through typed actions:
```rust
pub enum ClientAction {
Platform(PlatformAction),
User(UserAction),
Protocol(ServerEvent),
Tick(TickKind),
Command(CommandId),
}
```
Examples:
```rust
pub enum UserAction {
SubmitPrompt,
EditComposer(ComposerEdit),
ScrollTranscript { delta_px: f32 },
SelectSession(SessionId),
CancelRun,
ToggleActivityPanel,
OpenCommandPalette,
}
```
Reducers should return effects rather than performing side effects directly:
```rust
pub enum ClientEffect {
SendRequest(Request),
StartDaemon,
OpenExternalEditor(PathBuf),
CopyToClipboard(String),
ShowNotification(Notification),
RequestRender,
}
```
This is the clean boundary that the current TUI mostly lacks.
## Transcript model
The TUI currently reduces history/events into `DisplayMessage` and then terminal lines. Desktop needs a richer block model.
Suggested model:
```rust
struct TranscriptState {
blocks: Vec<TranscriptBlock>,
block_index: HashMap<BlockId, usize>,
streaming_block: Option<BlockId>,
version: u64,
}
enum TranscriptBlock {
User(UserBlock),
Assistant(AssistantBlock),
Tool(ToolBlock),
System(SystemBlock),
BackgroundTask(TaskBlock),
Swarm(SwarmBlock),
Usage(UsageBlock),
Memory(MemoryBlock),
Compaction(CompactionBlock),
}
```
This becomes the common semantic representation.
The TUI can continue rendering terminal lines from this model later. The desktop will render custom cards/rows from it.
### Desktop rendering path
```text
TranscriptBlock
-> DesktopTimelineItem
-> UI nodes
-> layout boxes
-> text layout runs
-> display list
```
Do not use terminal wrapped lines as the desktop source of truth.
## Feature mapping from TUI to desktop
| TUI feature | Current TUI shape | Desktop architecture |
|---|---|---|
| Chat transcript | `DisplayMessage` + wrapped `Line`s | `TranscriptBlock` + virtualized timeline |
| Streaming assistant text | `streaming_text` + incremental markdown | append-aware block text cache |
| Tool calls | `ToolCall` display messages and streaming tool calls | tool cards with compact/expanded states |
| Batch progress | `BatchProgress` in status/message prep | activity item + inline timeline block |
| Composer | terminal input string/cursor | custom text input model, IME-aware later |
| Queued messages | app queue fields | composer/session queue state in client-core |
| Soft interrupts | protocol events and pending queue | visible interruption banner/queue chip |
| Header/status | `ui_header`, `info_widget` | top bar + status/activity regions |
| Side pane | pinned diff/content/diagram pane | inspector panel with tabs |
| Mermaid/diagrams | terminal/image pane | image/vector surface, later side inspector |
| Diffs | terminal inline/pinned/file modes | changed files panel, diff cards, later hunk UI |
| Session picker | modal overlay | command palette/session switcher modal |
| Login/account picker | terminal overlays | settings/account modal views |
| Commands | slash commands/key handlers | shared command registry + palette + menus |
| Copy selection | line/cell ranges | semantic block/text selection |
| Workspace map | TUI workspace widget | session/workspace sidebar, optional spatial mode |
| Debug overlays | visual debug/frame capture | performance HUD + UI tree/render inspector |
| Reconnect | remote loop state machine | protocol client state machine in client-core/app |
| Replay | replay mode | event-log replay harness for desktop UI tests |
## Desktop product module layout
`jcode-desktop` should be product composition, not renderer internals.
Suggested modules:
```text
crates/jcode-desktop/src/
main.rs
app.rs # top-level DesktopApp orchestration
config.rs
protocol_client.rs # socket connection, read/write tasks
daemon.rs # start/connect/find bundled daemon
views/
root.rs
top_bar.rs
session_sidebar.rs
timeline.rs
timeline_blocks.rs
composer.rs
activity_panel.rs
workspace_panel.rs
inspector_panel.rs
command_palette.rs
permission_modal.rs
settings.rs
debug_hud.rs
reducers/
platform_events.rs
commands.rs
view_actions.rs
macos/
bundle.rs # build/package helpers if needed
appkit_hooks.rs # menus/lifecycle if winit is insufficient
```
## Custom UI crate layout
`jcode-desktop-ui` is the framework-like internal layer, but it is product-owned and small.
```text
crates/jcode-desktop-ui/src/
lib.rs
id.rs
geometry.rs
color.rs
style.rs
theme.rs
input.rs
focus.rs
accessibility.rs # semantic tree placeholder, not full impl initially
tree.rs # retained node tree
widget.rs # view builder traits/types
layout/
mod.rs
flex.rs
stack.rs
split.rs
scroll.rs
virtual_list.rs
text/
mod.rs
buffer.rs
selection.rs
shaping.rs
cache.rs
display_list.rs
invalidation.rs
animation.rs # minimal timers only, no full animation system initially
debug.rs
```
This crate should expose primitives such as:
```rust
pub enum UiNodeKind {
Row,
Column,
Stack,
SplitPane,
Scroll,
VirtualList,
Text,
TextInput,
Button,
CustomPaint,
}
```
But product views should mostly build specialized surfaces rather than generic widget soup.
## Renderer crate layout
`jcode-desktop-renderer` should know nothing about Jcode.
```text
crates/jcode-desktop-renderer/src/
lib.rs
gpu.rs
surface.rs
pipeline.rs
primitives.rs
text_renderer.rs
glyph_atlas.rs
image_atlas.rs
clips.rs
stats.rs
screenshot.rs
```
Input:
```rust
struct DisplayList {
commands: Vec<DrawCommand>,
}
enum DrawCommand {
Rect(RectPaint),
Border(BorderPaint),
Text(TextPaint),
Image(ImagePaint),
ClipBegin(Rect),
ClipEnd,
}
```
Output:
- frame rendered
- renderer stats
- optional screenshot/debug capture
The renderer should be testable with deterministic display lists and should support headless/golden rendering later if practical.
## Platform crate layout
Start with `winit`, but avoid spreading `winit` types through the product.
```text
crates/jcode-desktop-platform/src/
lib.rs
event.rs
window.rs
clipboard.rs
menus.rs
dialogs.rs
appearance.rs
shortcuts.rs
macos.rs
linux.rs
```
Normalize platform differences:
```rust
enum PlatformEvent {
WindowResized { size: PhysicalSize, scale: f64 },
ScaleFactorChanged { scale: f64 },
RedrawRequested,
Keyboard(KeyboardEvent),
Pointer(PointerEvent),
Scroll(ScrollEvent),
FilesDropped(Vec<PathBuf>),
AppearanceChanged(Appearance),
AppShouldQuit,
}
```
Keyboard shortcuts should use platform semantic modifiers:
```rust
enum ShortcutModifier {
Primary, // Cmd on macOS, Ctrl elsewhere
Alt,
Shift,
Control,
Command,
}
```
## Render/update loop
The TUI uses a redraw interval and `needs_redraw`. Desktop should keep the same spirit but be stricter.
```text
wait for platform/protocol/timer event
-> normalize event
-> reducer updates client-core/app state
-> collect effects
-> mark dirty UI nodes
-> if render requested:
layout dirty/visible nodes
shape dirty text
build display list
submit wgpu frame
publish debug stats
```
Rules:
- no continuous render loop when idle
- no full transcript re-layout on token append
- no unbounded visible node count
- protocol events may coalesce before rendering
- animations must explicitly schedule the next frame
- frame stats should be available before real feature integration is considered done
## Reuse path for existing TUI behavior
Do not stop all TUI work to extract everything first. Use an incremental route.
### Phase 1: desktop prototype independent of TUI internals
Build:
- desktop crates/modules
- fake transcript/activity data
- virtualized timeline
- debug HUD
- protocol-shaped fake events
Avoid depending on `src/tui`.
### Phase 2: protocol reuse
Use the existing server protocol:
- connect to `jcode serve`
- subscribe/resume session
- receive `ServerEvent`
- send `Request::Message`, `Request::Cancel`, etc.
Implement a desktop protocol reducer that mirrors the important behavior in `src/tui/app/remote/server_events.rs`, but writes to `ClientCore`/`TranscriptBlock`, not `DisplayMessage`.
### Phase 3: extract client-core
Once the desktop reducer shape is clear, extract shared pieces from TUI and desktop into `jcode-client-core`:
- transcript block model
- server event reducer
- command registry metadata
- activity model
- status model
- session list model
- permission model
At that point the TUI can gradually become another presentation of `jcode-client-core`, but it does not have to be converted all at once.
### Phase 4: feature parity
Add desktop versions of TUI features in priority order:
1. sessions, transcript, composer, send/cancel
2. tool cards and streaming output
3. activity panel and background tasks
4. command palette and core slash commands
5. permission prompts
6. session picker/resume/search
7. workspace/git/changed files
8. settings/login/account surfaces
9. diff/diagram inspector
10. debug/replay/profiling surfaces
## Desktop should be server-first
The TUI still supports local mode and remote/server mode. The desktop should start server-first.
Recommended desktop rule:
> Desktop always connects to a local Jcode server/daemon. It does not embed the agent runtime in-process.
Reasons:
- avoids UI freezes from runtime work
- keeps CLI/TUI/desktop as peers
- reuses reconnect/session lifecycle
- simpler crash isolation
- easier macOS bundle helper model
- avoids another local-mode runtime path
## Differences from the TUI model
### Scrolling
TUI scroll is line/cell based. Desktop scroll should be pixel based with fractional offsets.
```rust
struct ScrollState {
offset_px: f32,
velocity_px: f32,
anchor: Option<ScrollAnchor>,
auto_scroll: bool,
}
```
Virtualization should happen by pixel range and estimated/measured block heights.
### Text
TUI text is terminal spans and display widths. Desktop text should be shaped runs and glyph positions.
Desktop text caches should be keyed by:
- block ID
- content version/hash
- style
- available width
- font scale
- platform scale factor
### Selection
TUI selection is line/cell based. Desktop should use semantic selection:
- block ID
- text range within block
- optional structured copy target
### Layout
TUI layout is frame-sized terminal rects. Desktop should use a retained layout tree with dirty flags.
### Rendering caches
The TUI has several global caches. Desktop caches should be instance-owned and attributable:
```rust
struct RenderCaches {
text: TextLayoutCache,
glyphs: GlyphAtlas,
images: ImageAtlas,
timeline_measurements: MeasurementCache,
}
```
No process-global renderer state unless it is explicitly immutable/static.
## Testing strategy
Desktop should borrow the TUI's debug-first mentality, but use desktop-appropriate tests.
Required early tests:
- protocol reducer tests from `ServerEvent` sequences to `TranscriptBlock` state
- transcript virtualization tests with 100k fake blocks
- scroll anchor tests during streaming append
- layout tests for split panes and timeline rows
- text cache invalidation tests
- command registry tests
- display-list snapshot tests for stable fake UI states
- replay tests using captured protocol event logs
Avoid depending on GPU tests for basic correctness. Most UI behavior should be validated before `wgpu` submission.
## Implementation recommendation
Start by adding desktop code without touching the TUI too much:
```text
crates/jcode-desktop-ui # pure-ish UI/layout model
crates/jcode-desktop-renderer # wgpu display-list renderer
crates/jcode-desktop # fake-data app shell
```
Then connect to the server protocol.
Only after the desktop reducer/view model shape is proven should shared `jcode-client-core` extraction begin. This avoids prematurely extracting the wrong abstraction from the current TUI.
## Summary decision
The desktop app should be architected as a new custom presentation stack over shared client/runtime concepts, not as a ratatui port.
The TUI remains the feature reference. The server/protocol remains the runtime foundation. The new shared layer should be `client-core`, which owns surface-independent app behavior and view models. The desktop-specific code should focus on platform integration, retained UI, custom layout, text shaping, virtualization, and `wgpu` rendering.
docs/DESKTOP_FIRST_PROTOTYPE.md
+150
View File
@@ -0,0 +1,150 @@
# Desktop First Prototype Target
Status: Proposed
Updated: 2026-04-25
The first implementation step for Jcode Desktop should be **Phase 0: a fullscreen blank white canvas**.
Do not start with:
- fake workspace surfaces
- real server integration
- a full editor
- any browser work
- settings/auth flows
- packaging
- perfect text rendering
Start by proving the absolute foundation:
> a native fullscreen window with a custom GPU-rendered white canvas.
## Phase 0 visual target
```text
┌──────────────────────────────────────────────────────────────────────────────┐
│ │
│ │
│ │
│ │
│ blank white canvas │
│ │
│ │
│ │
│ │
└──────────────────────────────────────────────────────────────────────────────┘
```
## What Phase 0 must prove
1. A native window opens on Linux.
2. The app supports fullscreen or borderless fullscreen mode via `--fullscreen`.
3. The app creates a GPU surface.
4. The app clears the surface to white.
5. The app handles resize/scale-factor changes without crashing.
6. The app exits cleanly with `Esc` or close-window.
7. The app uses an on-demand event loop rather than a busy render loop.
8. The app can be built and run independently from the TUI.
## Why this comes before the spatial workspace
A blank canvas is intentionally tiny. It validates the platform/rendering foundation before adding product complexity.
It answers:
- Can we create the desktop crate cleanly?
- Does `winit` work as the initial platform shell?
- Does `wgpu` initialize on the Linux dev machine?
- Can we render a frame without a web view or UI framework?
- Can fullscreen behavior be tested early?
## Linux desktop entry
The repository includes an install-oriented desktop entry at:
```text
packaging/linux/jcode-desktop.desktop
```
It expects a `jcode-desktop` binary to be available on `PATH`. For local testing after installing or copying the binary somewhere your desktop launcher can execute, copy the entry to your user applications directory:
```bash
mkdir -p ~/.local/share/applications
cp packaging/linux/jcode-desktop.desktop ~/.local/share/applications/
update-desktop-database ~/.local/share/applications 2>/dev/null || true
```
## Phase 1 target after this
Once Phase 0 works, the next prototype is the fake-data spatial workspace. The first slice should prove the core Niri/Vim-style interaction model before real sessions or text rendering:
```text
Navigation mode:
h/l focus columns within the current workspace
j/k move to the workspace below/above
H/L move the focused column left/right
J/K move the focused column to the workspace below/above
n create a fake session surface
Ctrl+; create a fake session surface
Ctrl+? open/focus hotkey help
Ctrl+1 prefer 25%-screen-wide panels
Ctrl+2 prefer 50%-screen-wide panels
Ctrl+3 prefer 75%-screen-wide panels
Ctrl+4 prefer 100%-screen-wide panels
x close the focused surface
z zoom/unzoom the focused surface
i or Enter enter insert mode
Esc quit the prototype
Insert mode:
typing captured as draft input
Esc return to navigation mode
```
The initial renderer may use primitive colored/rounded primitives plus a tiny built-in bitmap font for early status and panel labels. Proper font rendering can follow after the workspace behavior feels right. The visual direction should put the color in a soft static blue/lavender/mint gradient background, with transparent panels on top, muted status colors, a very thin gray focus ring, and visible but subdued unfocused borders. A compact status bar should sit at the top, not the bottom. Individual panels should not have their own top header bars until real text/chrome is useful. Panels should fill most of the available space with only narrow gutters and slightly rounded corners. Panel count should adapt to both the current desktop app window size and the user-selected preferred panel size: `Ctrl+1` prefers 25%-screen-wide panels, `Ctrl+2` prefers 50%, `Ctrl+3` prefers 75%, and `Ctrl+4` prefers 100%. A fullscreen app with `Ctrl+1` can show four columns, while fullscreen with `Ctrl+4` shows one column. A 25%-screen-width app window shows one column regardless of preset because only one preferred quarter-screen panel fits. The layout direction is Niri-like: each workspace is a vertical lane containing a horizontally scrollable strip of full-height columns. Columns should never be stacked within the same workspace. The top status bar should include a left-side active workspace number, a centered flattened Waybar-like preview strip, and right-side mode/panel-size text. In the preview strip, nearby workspaces are shown as compact horizontal groups, each panel is a colored tick/block, inactive workspaces are dimmed, the active workspace group is highlighted, the focused panel is strongest, and the visible horizontal viewport is outlined. Smooth viewport/camera animations should make focus, workspace, spawn/close, and panel-size changes legible instead of teleporting instantly: `h/l` and `Ctrl+1` through `Ctrl+4` animate horizontally, `j/k` workspace changes slide vertically, and focused panel changes briefly pulse the outline for spawn, close, and focus handoff cues.
The target shape is:
```text
┌────────────────────────────────────────────────────────────────────────────────────┐
│ workspace 0 · NAV │
├────────────────────┬────────────────────┬────────────────────┬────────────────────┤
│ ● fox/coordinator │ wolf/impl │ owl/review │ activity │
│ │ │ │ │
│ full-height column │ full-height column │ full-height column │ full-height column │
│ │ │ │ │
│ │ │ │ │
│ │ │ │ │
├────────────────────┴────────────────────┴────────────────────┴────────────────────┤
│ h/l columns · j/k workspaces · Ctrl+; new · Ctrl+? help · n new · z zoom │
└────────────────────────────────────────────────────────────────────────────────────┘
```
Phase 1 proves the actual product bet:
- multiple visible agent sessions
- Niri-like spatial layout
- `h/l` column navigation and `j/k` workspace navigation
- move/close/zoom surfaces
- independent fake transcripts
- activity surface
- custom rendering performance
- near-zero idle CPU
## Initial Phase 1 surface kinds
```rust
enum SurfaceKind {
AgentSession,
Activity,
WorkspaceFiles,
Diff,
Debug,
}
```
No browser preplanning. No full editor yet.
## Phase 1 success bar
The fake workspace prototype is successful when a user can launch it, see multiple fake sessions, move between them with leader+`h/j/k/l`, create/move/close/zoom surfaces, and confirm the app remains smooth and idle-efficient.
docs/DESKTOP_SINGLE_SESSION_DESIGN.md
+72
View File
@@ -0,0 +1,72 @@
# Desktop Single Session Design
This document describes the visual target for the default `jcode-desktop` single-session mode.
## Layering
The single-session view is the primitive desktop surface. The Niri/workspace mode should later compose multiple single-session views rather than redefining what a session looks like.
```mermaid
flowchart TD
SingleSession[SingleSessionView\nspawn/connect/render one Jcode session]
Workspace[Niri Workspace Wrapper\narrange many sessions]
SessionA[SingleSessionView]
SessionB[SingleSessionView]
SessionC[SingleSessionView]
Workspace --> SessionA
Workspace --> SessionB
Workspace --> SessionC
```
## Typography
Primary font target:
- Family: `JetBrainsMono Nerd Font`
- Weight: `Light`
- Preferred fontconfig match: `JetBrainsMonoNerdFont-Light.ttf`
- Fallback family: `JetBrainsMono Nerd Font Mono`, then `JetBrains Mono`, then `monospace`
Rationale:
- Mono fits code, transcripts, tool output, and terminals.
- Light weight makes a dense agent session feel less heavy than the current blocky bitmap prototype.
- Nerd Font coverage gives us room for subtle icons/status glyphs later without switching fonts.
## Type scale
Initial target scale for a single session window:
| Role | Size | Weight | Notes |
| --- | ---: | --- | --- |
| Session title | 18 px | Light | Top left, preserves original case |
| Message body | 15 px | Light | Main transcript and assistant text |
| Metadata/status | 12 px | Light | Muted status, model, cwd, token/debug hints |
| Inline code/tool output | 14 px | Light | Same family, tighter line-height |
Line-height targets:
- Body: 1.45
- Code/tool output: 1.35
- Metadata: 1.25
## Rendering note
The current prototype uses a custom 5x7 bitmap text path in `render_helpers.rs`. That path is acceptable for layout exploration only. The next rendering pass should replace single-session text with a real font renderer that can:
1. Load `JetBrainsMonoNerdFont-Light.ttf` from fontconfig/system font paths.
2. Preserve casing and punctuation.
3. Shape/rasterize UTF-8 text, including Nerd Font glyphs.
4. Support alpha text over the existing WGPU surface.
5. Allow the workspace wrapper to reuse the same text renderer for each composed session.
## First visual goal
The default single-session window should read as one calm, focused coding conversation:
- No workspace lane/status strip.
- One content column.
- Large breathing room around the transcript.
- JetBrains Mono Light Nerd for every text element.
- Muted graphite text over the existing soft pastel background until a more final theme is chosen.
docs/DESKTOP_STABLE_HOST_RELOAD_STARTUP.md
+458
View File
@@ -0,0 +1,458 @@
# Desktop Stable Host, Hot Reload, and Fast Startup Plan
Status: In progress
Updated: 2026-05-24
## Why this exists
The current desktop hot reload path starts a replacement `jcode-desktop` process. The replacement process creates a new `winit` window, waits behind a reload handoff marker, then the old process exits. This is enough to preserve session continuity, but it cannot reliably preserve the physical OS window.
On Wayland in particular, a client cannot count on restoring global window position, and a top-level window cannot be transferred from one process to another. That means a process-level desktop reload is inherently allowed to close/reopen or move the window.
The goal of this plan is to make the desktop architecture support:
- hot reload without replacing the OS window
- fast warm reloads during self-development
- better perceived cold starts
- a clean long-term split between platform/window/rendering, app logic, and server/session runtime
## Current desktop ownership model
Today `crates/jcode-desktop/src/main.rs` owns almost everything in one process:
```text
jcode-desktop process
run()
winit EventLoop<DesktopUserEvent>
winit Window
Canvas
wgpu Surface / Device / Queue
glyphon FontSystem / text atlases / render caches
primitive and text render caches
DesktopApp
SingleSessionApp or Workspace
local UI state
session handles / async event channels
DesktopHotReloader
watches binary mtime
spawns replacement process
```
Current reload flow:
1. `DesktopHotReloader` notices the binary changed or receives force reload.
2. It captures `window.outer_position()` and `window.inner_size()`.
3. It spawns a replacement process with reload placement and handoff marker env vars.
4. The new process creates a hidden window and initializes its canvas.
5. The new process writes a `ready` marker.
6. The old process writes a `release` marker and exits.
7. The new process makes its new window visible.
This is visually better than an immediate exit, but it is still a new OS window.
## Current startup critical path
The current startup trace marks these milestones:
```text
args parsed
winit event loop created
window created
app state initialized
font loader spawned
wgpu instance created
wgpu surface created
wgpu adapter selected
wgpu device ready
surface configured
canvas ready
first frame presented
first content frame presented
```
Important observations:
- The process does not enter the event loop until after `Canvas::new(...)` completes.
- `Canvas::new(...)` performs WGPU instance/surface/adapter/device setup.
- Font loading is already moved to a background thread, but text rendering still depends on the host-side font/text renderer once real content is drawn.
- Workspace session-card loading is asynchronous after startup, which is good.
- Server/session reloads already reconnect over sockets without closing the desktop window. The problematic reload is the desktop UI binary itself.
## Constraints
### Window identity
A persistent window requires a persistent process that owns that window. The TUI can `exec()` because the terminal emulator owns the OS window. The desktop process owns its own OS window, so `exec()` or spawning a replacement destroys the top-level window.
### Wayland
Wayland does not provide reliable global window positioning or cross-process top-level window transfer. Any solution that depends on setting `x,y` on a new window will remain best-effort.
### Rust dynamic plugins
Loading desktop UI code as a Rust dynamic library could keep the process/window alive, but it is high risk as a first step:
- Rust has no stable plugin ABI for rich internal types.
- Unloading code safely is difficult with threads, statics, allocators, GPU handles, and panic/runtime state.
- Compiler/version mismatches can crash rather than fail gracefully.
- The boundary would need to be C-ABI-like or explicitly serialized anyway.
Dynamic plugins can be revisited later, but a process boundary is safer and easier to test.
## Architecture options
| Option | Window survives UI reload? | Cold-start benefit | Risk | Notes |
|---|---:|---:|---:|---|
| Keep current process handoff | No | None | Low | Cannot solve Wayland movement. |
| Desktop `exec()` like TUI | No | None | Low | Process image changes, but the OS window still dies. |
| Dynamic library plugin | Yes | Medium | High | Fastest theoretical reload, but ABI/unload hazards. |
| Stable host + reloadable app worker + display-list protocol | Yes | High perceived benefit | Medium | Recommended. |
| Stable host + child renders pixels offscreen | Yes | Medium | High | More bandwidth/GPU sharing complexity, less flexible. |
## Recommended design
Introduce a small stable desktop host that owns platform and rendering resources. The reloadable desktop app code runs behind a protocol boundary.
```mermaid
flowchart LR
Launcher[jcode-desktop launcher] --> Host[jcode-desktop-host]
Host --> Window[winit window]
Host --> Renderer[wgpu renderer + font/text atlases]
Host <-->|input, display updates, snapshots| Worker[jcode-desktop-app worker]
Worker <-->|session protocol| Server[jcode server/daemon]
```
### Stable host responsibilities
The host should own the things that must not disappear during a UI reload:
- OS window and event loop
- WGPU instance/device/queue/surface
- font system and glyph/text atlases
- image atlases and primitive buffers
- clipboard/platform adapters where practical
- app-worker lifecycle and restart policy
- last known UI snapshot for reload handoff
- immediate boot/splash/error UI
The host should be intentionally small and stable. Most product UI behavior should not live here.
### Reloadable app worker responsibilities
The worker should own product logic that we want to iterate on quickly:
- workspace/single-session reducers
- command handling
- layout decisions
- markdown/transcript shaping decisions at a logical level
- session/server protocol client
- view-model construction
- surface-local UI state, with snapshot export/import
The worker does **not** own the OS window. If it exits or is replaced, the host keeps showing the last committed frame, an overlay like “reloading UI…”, or a host-rendered fallback.
### Host/worker protocol
Start with a typed, versioned, local-only protocol over a Unix domain socket or inherited stdio pipes. JSON is fine for the first prototype because debuggability matters; switch to bincode/postcard only if measurements require it.
Minimum message families:
```text
Host -> Worker
Init { protocol_version, window_size, scale_factor, restored_snapshot }
Input { key/mouse/scroll/text/clipboard/drop }
Window { resized, scale_factor_changed, focus_changed }
Tick { now, animation_budget }
Command { reload_requested, shutdown_requested }
Worker -> Host
Ready { capabilities }
Frame { display_list, title, cursor, animation_state }
Patch { display_list_delta }
Snapshot { opaque_versioned_state }
Request { clipboard_read, clipboard_write, open_url, spawn_terminal }
Log / Error / Panic
```
The first implementation can send full frames. Add diffing once frame sizes are known.
### Display-list boundary
The host should render a semantic display list rather than pixels.
A useful first display list can include:
- rectangles, rounded rectangles, borders, rules
- text boxes with font family, size, color, wrap width, selection ranges
- image references
- clips/layers/transforms
- hit-test IDs/accessibility labels later
This keeps GPU, glyph, and image caches in the persistent host while letting worker code change layout and UI behavior. Text shaping can remain host-side for cache reuse: the worker says “draw this text in this box with these styles”, and the host shapes/renders it.
## Startup strategy
The same host architecture can improve startup if we treat the host as a fast shell, not merely as a reload wrapper.
### Cold start from no running processes
A true cold start still must pay for process launch and window creation. It should **not** have to block window visibility on WGPU device initialization.
The host should have a staged startup model:
```text
WindowHost phase
create event loop/window
make the OS window visible quickly
optionally draw a native/CPU/blank splash
start WGPU initialization asynchronously
start app-worker/server/session loading asynchronously
GpuHost phase
WGPU surface/device/queue ready
swap in the real renderer
draw cached snapshot or latest worker scene
LiveApp phase
worker ready
server/session state caught up
normal display-list rendering
```
This separates three different metrics that are currently coupled:
- **time to window visible**: should not wait for WGPU
- **time to first host pixels**: can use a no-GPU/native/CPU fallback if worthwhile
- **time to first real GPU content**: still depends on WGPU init
We can improve perceived and measured startup by:
1. Keeping the host binary small.
2. Creating and showing the window before loading product/session state.
3. Not blocking the host event loop on `Canvas::new(...)` / WGPU setup.
4. Starting WGPU init, app-worker launch, server connection, font loading, and session-card loading in parallel.
5. Drawing a host-owned GPU boot frame as soon as the WGPU surface is configured.
6. Showing a cached workspace/session snapshot immediately while live data catches up.
7. Lazily creating expensive text/image renderer resources on first real use.
8. Avoiding synchronous disk scans on the UI thread.
Target cold-start sequence:
```text
host process starts
parse minimal host args
create event loop/window
show window immediately
enter event loop
start worker process asynchronously
start/attach server asynchronously
start WGPU init asynchronously
optionally show no-GPU/native/CPU splash
WGPU ready -> paint host GPU boot frame
load cached UI snapshot
receive worker Ready/Frame
replace boot frame with live UI
```
### No-GPU boot phase
The host does not need to construct WGPU before it owns a window. Treat WGPU as a renderer backend that can appear later.
Implementation shape:
```rust
enum HostRendererState {
NoGpuBoot,
GpuInitializing,
GpuReady(Canvas),
GpuFailed { message: String },
}
```
In `NoGpuBoot`, the host can choose one of three fallback strategies:
1. **Blank visible window**: fastest and simplest, but may look unfinished.
2. **Native/platform splash**: use platform APIs where practical. Fast, but less portable.
3. **CPU framebuffer splash**: use a tiny software path such as `softbuffer` for a basic solid background/logo/status. More portable than native drawing, but adds another rendering backend.
The first implementation should probably start with a blank or minimal native-color visible window and metrics. Add a CPU splash only if measurements show a real user-visible gap worth covering.
The important rule is that WGPU readiness should upgrade the renderer in place; it should not recreate the OS window.
### User-perceived cold starts with a resident host
For the fastest user-visible launch, add an optional linger/autostart host:
- `jcode-desktop-host --daemon` starts at login or after first launch.
- It keeps WGPU device, font DB, and maybe recent workspace snapshot warm.
- Opening the app asks the resident host to show/create a window.
- The daemon exits after an idle timeout if low idle resource use is preferred.
This is not a true machine-cold start improvement, but it is the common “I opened the app and it appeared instantly” path.
### Warm reload
On app-worker reload:
1. Host asks old worker for a snapshot.
2. Host keeps the existing frame visible and overlays “Reloading UI…”.
3. Host starts the new worker binary.
4. New worker receives the snapshot and reconnects to the server/session.
5. New worker sends `Ready` and then a new `Frame`.
6. Host swaps to the new frame without moving or hiding the OS window.
If the new worker fails, the host keeps the old frame or a host-rendered error UI and can retry.
## Incremental migration plan
### Phase 0: measurement first
Add or preserve metrics for:
- process start to window created
- process start to window visible / event loop entered
- process start to WGPU init started
- process start to surface configured / WGPU ready
- process start to first no-GPU host pixels, if a fallback renderer exists
- process start to first GPU host frame
- process start to first live worker content frame
- reload blackout duration
- worker restart duration
- display-list bytes per frame
- input event to presented frame latency
The existing `--startup-log`, `--startup-benchmark`, and frame profiler are a good base. Extend them instead of inventing a second measurement system.
### Phase 1: display-list extraction in-process
Before adding IPC, make the current renderer consume a display-list-like structure instead of directly reading `DesktopApp` everywhere.
Expected outcome:
```text
DesktopApp + layout code -> DesktopScene / DisplayList -> Canvas::render_scene(...)
```
This is the safest first refactor because tests can compare current rendering behavior while creating the boundary the host/worker protocol will need.
### Phase 2: app-driver trait in-process
Introduce an internal boundary like:
```rust
trait DesktopAppDriver {
fn handle_input(&mut self, input: DesktopInputEvent) -> Vec<DesktopHostCommand>;
fn handle_session_events(&mut self, events: Vec<DesktopSessionEvent>) -> Vec<DesktopHostCommand>;
fn build_scene(&mut self, viewport: DesktopViewport) -> DesktopScene;
fn snapshot(&self) -> DesktopUiSnapshot;
fn restore(&mut self, snapshot: DesktopUiSnapshot);
}
```
Initially, the existing `DesktopApp` implements this in the same process.
### Phase 3: host mode and worker mode
Add two modes, possibly in the same binary first:
- `jcode-desktop --host`
- `jcode-desktop --app-worker --ipc <fd-or-socket>`
The host owns the current `run()` window/event loop/canvas path. The worker owns the app driver and session protocol. Use a socket/stdio protocol to pass inputs and display-list frames.
### Phase 4: worker hot reload
Change `DesktopHotReloader` so the host watches the worker binary and restarts only the worker. The host process and OS window remain alive.
Host binary reload can remain the current process handoff path because host changes should be rarer.
### Phase 5: split crates/binaries
Once the protocol is real, split into clearer crates/binaries:
```text
crates/jcode-desktop-protocol # host/worker messages and scene types
crates/jcode-desktop-renderer # Canvas/rendering/display-list renderer
crates/jcode-desktop-host # window/event loop/worker lifecycle
crates/jcode-desktop-app # product UI worker
crates/jcode-desktop # launcher or compatibility wrapper
```
This also supports cold-start optimization because the host can stay small and stable while app code grows.
## Risks and mitigations
### Display-list size and IPC overhead
Mitigation: start with full frames and measure. Add scene hashes, retained nodes, and patches only after we have real data.
### State loss on reload
Mitigation: define `DesktopUiSnapshot` early. Include draft text, focus, scroll offsets, selected session/surface, workspace placement, and pending local UI state. Keep it versioned and best-effort so incompatible snapshots fail gracefully.
### Protocol skew
Mitigation: explicit protocol version and capabilities. If a new worker is incompatible, the host shows a clear error and offers to restart the whole desktop app.
### Too much logic in the host
Mitigation: host owns platform/rendering/lifecycle only. If a behavior is product-specific and safe to reload, it belongs in the worker.
### Startup daemon resource use
Mitigation: make resident host optional, idle-timeout based, and measurable. Keep default behavior on-demand until data shows the daemon is worth it.
## Recommended MVP
The smallest useful MVP is not a separate process yet. It is:
1. Define `DesktopScene` / display-list types.
2. Refactor `Canvas::render(&DesktopApp, ...)` into `Canvas::render_scene(&DesktopScene, ...)` while still building the scene in-process.
3. Add startup/reload metrics around scene building and first host frame.
4. Add `DesktopUiSnapshot` types for the state we need to preserve.
After that, moving the scene builder into a worker process becomes a mechanical host/worker transport problem instead of a giant UI rewrite.
## Bottom line
For reloads that never move the window, we need a stable window owner. For faster starts, that stable owner should also be a small fast shell that can draw immediately, keep expensive renderer resources warm, and load product/session logic asynchronously.
The recommended path is a stable `jcode-desktop-host` plus a reloadable `jcode-desktop-app` worker communicating via a typed display-list protocol. It avoids Wayland window-position limitations, avoids Rust plugin ABI hazards, and creates a natural path to both smooth hot reloads and better perceived cold-start performance.
## Current implementation snapshot
As of 2026-05-24, the first stable-host slice exists in the current `jcode-desktop` binary:
- WGPU/canvas initialization is spawned after the `winit` window is created, so the event loop can enter before GPU setup finishes.
- `DesktopScene`, `DesktopUiSnapshot`, host/worker protocol messages, JSON-lines IPC framing, and protocol-version validation are in place.
- `--desktop-process-role stable-host` owns the OS window and renderer and starts a headless `app-worker` child process.
- Stable-host reload uses `DesktopReloadStrategy::AppWorkerRestart`, killing/restarting only the app worker while the host window stays alive.
- The app worker initializes a real `DesktopAppRuntime`, restores the host snapshot, applies key/session IPC events, and emits updated scenes back to the host.
- The host renders worker-produced scenes through `Canvas::render_scene(...)` when available.
Validation performed for this slice:
- `cargo test -p jcode-desktop desktop_`
- `cargo check -p jcode-desktop`
- `selfdev build target=desktop`
- runtime smoke: `cargo run -p jcode-desktop --bin jcode-desktop -- --desktop-process-role stable-host --startup-benchmark --startup-log`
## Crate-split and cold-start assessment
The next split should not be a separate binary yet. The code is functionally split, but `crates/jcode-desktop/src/main.rs` is still a large 10k+ line compilation unit that imports both heavy renderer dependencies (`wgpu`, `glyphon`) and app/session logic. Splitting crates before moving more code out of `main.rs` would add build-system complexity without materially reducing cold-start work.
Recommended order:
1. **Module split inside `jcode-desktop` first.** Move stable-host/window/reload code, worker-process loop, and canvas/rendering code out of `main.rs` into focused modules. This reduces merge conflicts and makes crate boundaries obvious without changing packaging.
2. **Promote protocol and scene types to a small crate.** `desktop_protocol`, `desktop_scene`, and IPC framing are already mostly independent. A future `jcode-desktop-protocol` crate can compile quickly and be shared by host and worker without pulling WGPU.
3. **Create a host binary only after the host module no longer depends on app reducers.** The host binary should depend on `winit`, `wgpu`, `glyphon`, protocol/scene types, and worker lifecycle only. It should not depend on session reducers except for versioned snapshots and host fallback/error UI.
4. **Create an app-worker binary after product UI state is behind `DesktopAppDriver`.** The worker binary should own `SingleSessionApp`, `Workspace`, session/server integration, and scene construction. It should not link WGPU or glyph atlases.
5. **Measure before adding a resident daemon.** A resident host can make user-perceived launch nearly instant, but it also adds idle memory/GPU cost. Add it only after we have startup metrics for cold host process, WGPU readiness, worker ready, and first live scene.
Expected cold-start impact:
- **On-demand cold start:** splitting crates/binaries mainly helps by keeping host initialization small and allowing worker/app loading to happen in parallel. It does not remove WGPU cost for first GPU pixels.
- **Perceived cold start:** the stable host can show the window immediately and keep a boot/cached frame visible while WGPU and worker startup race in parallel.
- **Warm/reload start:** this is where the architecture already wins. The host keeps the window and renderer alive, and only the worker restarts.
Current recommendation: defer physical crate split until the host/worker modules are cleaner and the debug-socket E2E reload test is in place. The next best engineering step is stronger end-to-end validation, not more crate boundaries.
docs/DESKTOP_SUPERAPP_WORKSPACE.md
+515
View File
@@ -0,0 +1,515 @@
# Desktop Superapp Workspace Direction
Status: Proposed
Updated: 2026-04-25
This document refines the Jcode Desktop product direction from a single chat-like app into a **Niri-like agent workspace superapp**.
The app should eventually host multiple kinds of surfaces:
- agent sessions
- task/activity views
- code/file surfaces
- diffs
- terminals or command output surfaces
- settings/auth/tools/debug surfaces
The goal is not to clone a general-purpose window manager. The goal is to give Jcode users a fast, keyboard-driven, spatial environment for supervising multiple agent sessions and related development tools inside one custom desktop app.
See also:
- [`DESKTOP_APP_ARCHITECTURE.md`](./DESKTOP_APP_ARCHITECTURE.md)
- [`DESKTOP_CODEBASE_ARCHITECTURE.md`](./DESKTOP_CODEBASE_ARCHITECTURE.md)
- [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md)
## Product thesis
Jcode Desktop should be a **local AI development superapp**:
```text
one native app
many sessions
many surfaces
fast spatial navigation
strong keyboard workflow
agent-first activity visibility
custom rendering and layout
```
The key UX is closer to:
- Niri / scrollable tiling workspace
- Vim-like keyboard navigation
- command palette
- agent mission control
And less like:
- a single chat window
- a conventional IDE clone
- a web-app shell
- a generic desktop window manager
## Why Niri-like
Niri's useful idea for Jcode is not the compositor implementation. It is the mental model:
- surfaces are arranged spatially
- focus moves predictably
- users navigate with keyboard-first commands
- new work appears in a structured place
- layout is persistent enough to build muscle memory
- many parallel tasks can be monitored without losing context
Jcode Desktop can bring that workflow to macOS users who do not have a Niri-like environment, while still working well on Linux.
## Workspace model
The desktop app should be built around these concepts:
```text
Workspace
-> Rows / Workspaces / Lanes
-> Columns
-> Surfaces
```
Terminology can be adjusted, but the core model should support:
- multiple agent sessions visible or quickly reachable
- spatial navigation with `h/j/k/l`-style movement
- opening related surfaces next to a session
- moving surfaces between columns/lanes
- zooming/focusing one surface temporarily
- preserving layout per project/workspace
Suggested initial terms:
| Term | Meaning |
|---|---|
| Workspace | A project/repo-level desktop environment |
| Lane | A vertical grouping or Niri-like workspace row |
| Column | A horizontal focus/navigation unit |
| Surface | A visible app panel: session, file/code view, diff, activity, settings, debug, etc. |
| Session surface | A surface attached to a server-owned Jcode session |
| Tool surface | File/code/diff/activity/settings/debug/etc. |
## Surface types
The app should be architected around a generic surface registry from the start.
```rust
enum SurfaceKind {
AgentSession,
Activity,
WorkspaceFiles,
CodeView,
Diff,
TerminalOutput,
Settings,
Debug,
Extension,
}
```
A surface should have:
```rust
struct SurfaceState {
id: SurfaceId,
kind: SurfaceKind,
title: String,
workspace_id: WorkspaceId,
lane_id: LaneId,
column_id: ColumnId,
focus_state: FocusState,
local_state: SurfaceLocalState,
}
```
The surface model should be independent from the renderer so it can support:
- one window with many surfaces
- multiple windows later
- pop-out surfaces later
- session surfaces and non-session utility surfaces using the same navigation model
## Agent sessions as first-class surfaces
An agent session should be one surface type, not the whole app.
```text
AgentSessionSurface
- transcript timeline
- composer
- inline tool cards
- session status
- session-local queue/interrupts
```
This allows layouts like:
```text
[Session A] [Session B] [Diff ]
[Activity ] [Files ] [CodeView ]
```
Or:
```text
Lane 1: main task
Column 1: coordinator session
Column 2: implementation agent session
Column 3: diff/editor
Lane 2: review
Column 1: changed files
Column 2: notes/session
```
## Navigation model
The app should have a modal/command-oriented keyboard model inspired by Vim, but adapted for macOS and desktop expectations.
### Important macOS constraint
Do not rely on plain `Cmd+H` for left navigation.
On macOS:
- `Cmd+H` hides the app
- `Cmd+M` minimizes
- `Cmd+Q` quits
- `Cmd+W` closes the current window/surface depending on app convention
- `Cmd+,` opens settings
Overriding these will make the app feel hostile to Mac users.
### Recommended navigation approach
Use one or both of these:
1. **Leader/command mode**
- Press a leader key, then `h/j/k/l`.
- Example: `Space h`, `Space j`, `Space k`, `Space l` when focus is not in text input.
- Or `Cmd+K h/j/k/l` as a command chord.
2. **Direct advanced shortcuts**
- `Cmd+Option+H/J/K/L` for focus movement on macOS.
- `Ctrl+Alt+H/J/K/L` or `Super+Alt+H/J/K/L` on Linux.
The leader model is safer because it avoids macOS reserved shortcuts and works well with Vim muscle memory.
### Suggested initial keymap
```text
Focus movement:
leader h focus left
leader j focus down / next lane
leader k focus up / previous lane
leader l focus right
Surface movement:
leader H move surface left
leader J move surface down
leader K move surface up
leader L move surface right
Workspace/session:
leader n new agent session
leader s session switcher
leader a activity center
leader e editor/files surface
leader d diff surface
leader / command palette
leader z zoom focused surface
leader x close focused surface
Agent control:
leader Enter focus composer / submit depending mode
leader Esc cancel/stop focused agent run, with confirmation if risky
```
The exact leader key should be configurable. Reasonable defaults:
- `Space` when focus is not in a text input
- `Cmd+K` as a universal command chord
- `Ctrl+Space` as an alternate for users who prefer explicit mode entry
## Input modes
The app should distinguish between navigation mode and text-entry mode.
```text
Navigation mode
- hjkl controls focus/layout
- keys trigger commands
- typing can open command palette or focused composer depending setting
Text-entry mode
- keys edit composer/editor/input
- Escape returns to navigation mode
- platform shortcuts still work: copy/paste/select all
```
This is critical once the app has text-entry surfaces. Without explicit input modes, global Vim-like keys will conflict with text entry.
## Layout behavior
The first implementation does not need full Niri behavior. It should start with a simpler model that can evolve.
### MVP layout
```text
single app window
left sidebar: workspaces/sessions
central surface grid: columns with focused surface
right activity/inspector panel optional
```
MVP navigation:
- focus next/previous surface
- move focus left/right between columns
- open new session to the right
- close surface
- zoom focused surface
- activity panel toggle
### Later layout
Niri-like scrollable layout:
- horizontal columns per lane
- vertical lane/workspace movement
- smooth animated focus movement
- persistent surface positions
- per-workspace layout restoration
- drag surfaces with mouse, but keyboard remains primary
- pop-out surface into native window
- dock pop-out surface back into workspace
## Surface lifecycle
Surface commands should be consistent across surface kinds.
```text
new-surface(kind)
close-surface(id)
focus-surface(direction)
move-surface(direction)
zoom-surface(id)
split-surface(kind, direction)
pop-out-surface(id)
dock-surface(id)
```
Agent session-specific commands become specialized actions on an `AgentSession` surface:
```text
send-message
cancel-run
soft-interrupt
background-tool
resume-session
fork-session
```
Non-session surfaces can add specialized commands later without changing generic surface lifecycle commands.
## Optional future surfaces
Do not preplan any large embedded app right now. The workspace model should stay generic enough to host future surface kinds, but the implementation plan should focus on agent sessions, activity, files, diffs, and command routing.
If a major embedded surface is considered later, it should go through its own design decision rather than being assumed by the initial desktop architecture.
## Built-in code editor direction
A built-in editor is a large system and should remain optional until the workspace/session workflow is strong.
Suggested levels:
### Level 1: file viewer and external editor
MVP-friendly:
- file tree / changed files
- read-only file preview
- open in external editor
- open diff externally
- copy paths/snippets
### Level 2: lightweight code viewer/diff editor
Useful and realistic:
- syntax-highlighted file view
- search within file
- inline diff viewer
- accept/reject generated changes later
- simple text selection/copy
### Level 3: real code editor
Large but possible later:
- rope text buffer
- multi-cursor maybe
- undo/redo
- syntax highlighting
- LSP integration
- diagnostics
- completion
- file save/reload conflict handling
- large-file performance
### Recommendation
Start with Level 1, then Level 2. Do not build a full editor before the agent workspace, transcript, activity, and diff workflow are excellent.
The architecture should support file/code surfaces generically, but should not commit to a full editor implementation early.
## Activity as a persistent surface
For a superapp, activity should not be just a small panel.
Activity should be a surface type that can be:
- pinned to the side
- opened as a full surface
- filtered by workspace/session/tool type
- navigated with the same surface commands
- used to jump to the relevant session/tool output
This is important because Jcode users may run many agents/tasks concurrently.
## Command palette as the universal router
The command palette should be the universal way to access everything:
- sessions
- surfaces
- files
- commands
- settings
- tools
- files and code views
- background tasks
- debug views
It should be backed by a shared command registry in `jcode-client-core`, not hardcoded separately per UI.
## Data model additions
`jcode-client-core` should include a workspace layout state model:
```rust
struct WorkspaceLayoutState {
workspaces: Vec<WorkspaceNode>,
active_workspace: WorkspaceId,
active_surface: Option<SurfaceId>,
}
struct WorkspaceNode {
id: WorkspaceId,
name: String,
lanes: Vec<LaneNode>,
}
struct LaneNode {
id: LaneId,
columns: Vec<ColumnNode>,
}
struct ColumnNode {
id: ColumnId,
surfaces: Vec<SurfaceId>,
active_surface_index: usize,
}
```
Surface-local data should be separated by kind:
```rust
enum SurfaceLocalState {
AgentSession(AgentSessionSurfaceState),
Activity(ActivitySurfaceState),
WorkspaceFiles(WorkspaceFilesSurfaceState),
CodeView(CodeViewSurfaceState),
Diff(DiffSurfaceState),
TerminalOutput(TerminalOutputSurfaceState),
Settings(SettingsSurfaceState),
Debug(DebugSurfaceState),
Extension(ExtensionSurfaceState),
}
```
This preserves the core rule:
> A session is server-owned runtime state. A surface is client-owned UI state.
## Renderer implications
A Niri-like superapp increases the importance of the custom UI engine.
The UI engine must support:
- nested split/column/lane layout
- animated or smooth focus movement later
- virtualized surfaces
- focus rings and active-surface indicators
- surface chrome/title bars that do not waste space
- zoom/focus mode
- drag-to-rearrange later
- stable IDs for accessibility/debugging
- cheap offscreen/inactive surface representation
Do not keep every surface fully rendered at all times. Inactive surfaces should keep state but avoid expensive layout/text/render work unless visible or prewarmed.
## Suggested first superapp milestone
Update the earlier fake-data desktop prototype to prove the superapp model, not only one transcript.
### Milestone: fake-data spatial workspace
Success criteria:
- one native window on Linux
- custom `wgpu` rendering
- workspace layout with multiple fake agent session surfaces
- focus movement with leader + `h/j/k/l`
- open/close/move/zoom fake surfaces
- activity surface with fake running tasks
- command palette can create session/activity/file/diff/debug placeholder surfaces
- transcript surfaces are virtualized independently
- debug HUD shows per-surface layout/render stats
- idle CPU remains near zero
Optional non-session surfaces can be placeholders at this stage. The important part is proving that the workspace model can host multiple surface kinds without committing to specific future apps.
## Product guardrails
Because “superapp” can explode in scope, keep these guardrails:
1. Agent sessions and activity are the core product.
2. Non-session surfaces are supporting tools, not the first milestone.
3. External integrations should come before embedded implementations.
4. Keyboard navigation must work before mouse drag layout polish.
5. Surface architecture must be generic from day one.
6. Do not build large embedded apps before diff/review workflows are excellent.
7. Keep the server as the source of truth for sessions and agents.
## Summary decision
Jcode Desktop should become a **keyboard-driven, Niri-like agent workspace superapp**.
The initial desktop app should prove:
- many session surfaces
- spatial navigation
- generic surface lifecycle
- command palette routing
- activity visibility
- performance under multiple visible surfaces
Then additional file/diff/tool surfaces can be added without changing the fundamental app model.
docs/GMAIL_COMPOSIO_BACKEND.md
+96
View File
@@ -0,0 +1,96 @@
# Gmail Tool: Composio Managed Backend
The native `gmail` tool can source credentials and transport from one of two
backends. The tool interface, confirmation gating, access-tier logic, and
token-lean output formatting are identical across backends; only the
auth/transport layer changes.
## Backends
| Backend | Auth | Pros | Cons |
|---|---|---|---|
| `direct` (default) | Local Google OAuth tokens (`jcode login google`) | No third party in the loop | Unverified-app warning; 7-day refresh-token expiry in Google "Testing" mode |
| `composio` | Composio-managed OAuth (Google-verified app) | No unverified-app warning, no 7-day expiry, no per-user Google Cloud project | Composio brokers Gmail token custody; external dependency/cost |
Both backends call the *same* Gmail REST endpoints
(`https://gmail.googleapis.com/gmail/v1/users/me/...`). The Composio backend
routes those calls through Composio's
[`proxy-execute`](https://docs.composio.dev/reference/api-reference/tools/postToolsExecuteProxy)
endpoint, which attaches the managed Gmail credentials. Because the upstream
response shape is unchanged, all existing typed parsing and output formatting
is reused.
## Selecting the backend
The backend is resolved from environment at `GmailClient::new()`:
- `JCODE_GMAIL_BACKEND=direct` (or unset) -> direct Google backend.
- `JCODE_GMAIL_BACKEND=composio` -> Composio backend (requires `COMPOSIO_API_KEY`).
If `composio` is requested but `COMPOSIO_API_KEY` is missing, jcode warns and
falls back to `direct`.
### Composio environment variables
| Variable | Required | Description |
|---|---|---|
| `COMPOSIO_API_KEY` | Yes | Project API key from <https://platform.composio.dev> |
| `COMPOSIO_BASE_URL` | No | Override API base (default `https://backend.composio.dev/api/v3.1`) |
| `COMPOSIO_GMAIL_AUTH_CONFIG_ID` | For `connect` | Gmail auth config id (`ac_...`) from the Composio dashboard. Defines the OAuth blueprint/scopes used by the connect flow. |
| `COMPOSIO_GMAIL_CONNECTED_ACCOUNT_ID` | No | Pin a specific connected account (`ca_...`). Normally set automatically after `connect`. |
| `COMPOSIO_GMAIL_USER_ID` / `COMPOSIO_USER_ID` | No | End-user id for multi-user connected accounts (defaults to `default`) |
## Connecting a Gmail account (in-agent OAuth)
Once `COMPOSIO_API_KEY` and `COMPOSIO_GMAIL_AUTH_CONFIG_ID` are set, the user
(or the agent) runs the gmail tool with `action: "connect"`:
1. jcode calls Composio's `POST /connected_accounts/link` (hosted "Connect
Link" flow) to start an OAuth session.
2. The returned `redirect_url` is opened in the system browser (printed to
stderr as a fallback, e.g. over SSH).
3. The user approves Gmail access on Google's consent screen. Because Composio
owns a Google-verified app, there is no "unverified app" warning.
4. jcode polls `GET /connected_accounts/{id}` until the connection is `ACTIVE`,
then persists it to `~/.jcode/composio_gmail.json`.
Future sessions load the persisted `connected_account_id`, so the connect step
is a one-time action per account. Tool calls before a connection exists return
a hint telling the agent to run `action: "connect"` first.
> Note: Composio is retiring `initiate()` for managed OAuth in favor of the
> Connect Link `link()` flow used here, so this path is the supported one going
> forward.
## One-time Composio setup
1. Sign in at <https://platform.composio.dev> and copy your project API key.
2. Connect a Gmail account (Composio's hosted OAuth, no unverified-app warning).
Note the resulting `connected_account_id` if you want to pin it.
3. Export the variables:
```bash
export JCODE_GMAIL_BACKEND=composio
export COMPOSIO_API_KEY="ck_..."
# optional:
export COMPOSIO_GMAIL_CONNECTED_ACCOUNT_ID="ca_..."
export COMPOSIO_GMAIL_USER_ID="me"
```
4. Ensure the `gmail` tool is enabled in `config.toml`:
```toml
[tools]
enabled = ["*"]
```
## Access tiers
- `direct`: honors the access tier chosen at `jcode login google`
(Read & Draft Only logins cannot send/trash, enforced at the OAuth scope level).
- `composio`: connections request full Gmail scopes, so send/trash are
available. The tool still requires explicit `confirmed: true` for send,
send_draft, and trash.
## Trust note
With the Composio backend, Composio holds your Gmail OAuth grant and sees API
traffic. This is the core tradeoff versus the direct backend. Disclose this to
users before enabling it as a default.
docs/HOOKS.md
+147
View File
@@ -0,0 +1,147 @@
# Lifecycle Hooks
jcode can run external commands at well-defined lifecycle points so other
programs can observe or gate agent behavior without forking jcode. Hooks
complement the [spawn hook](SPAWN_HOOK.md) (which controls *where headed
sessions appear*); lifecycle hooks tell you *what is happening inside them*.
## Configuration
```toml
# ~/.jcode/config.toml
[hooks]
turn_end = "~/bin/jcode-turn-notify" # observer
session_start = "" # observer
session_end = "" # observer
pre_tool = "~/bin/jcode-tool-policy" # gate
post_tool = "" # observer
pre_tool_timeout_ms = 5000
```
Env overrides (always win; empty value disables a config hook):
`JCODE_HOOK_TURN_END`, `JCODE_HOOK_SESSION_START`, `JCODE_HOOK_SESSION_END`,
`JCODE_HOOK_PRE_TOOL`, `JCODE_HOOK_POST_TOOL`, `JCODE_HOOK_PRE_TOOL_TIMEOUT_MS`.
## Common contract
- The hook command line is parsed shell-style (quotes and backslash escapes
work) but executed **directly**, not through a shell. A leading `~/` in the
program path is expanded.
- The hook runs in the session working directory when known.
- Every hook receives:
| Variable | Meaning |
| --- | --- |
| `JCODE_HOOK_EVENT` | `turn_end`, `session_start`, `session_end`, `pre_tool`, `post_tool` |
| `JCODE_HOOK_SESSION_ID` | Session the event belongs to |
| `JCODE_HOOK_CWD` | Session working directory |
| `JCODE_HOOK_PAYLOAD` | JSON object mirroring all fields (capped at 16 KB) |
| `JCODE_HOOKS_DISABLED` | Always `1`; suppresses hooks in nested jcode calls (recursion guard) |
## Observer hooks
`turn_end`, `session_start`, `session_end`, and `post_tool` are
**observers**: spawned detached, fire-and-forget. They can never block or slow
the agent; failures are only logged.
### `turn_end`
Fires when an agent turn completes (streaming turn path, which covers TUI,
desktop, swarm workers, and headless sessions).
Extra fields: `JCODE_HOOK_STATUS` (`ok`/`error`), `JCODE_HOOK_DURATION_MS`,
`JCODE_HOOK_MODEL`, `JCODE_HOOK_LAST_ASSISTANT_TEXT` (first 4000 chars),
`JCODE_HOOK_ERROR` (on failure).
### `session_start` / `session_end`
`session_start` fires when an agent session becomes active, with
`JCODE_HOOK_SOURCE` = `create` (brand new), `attach` (existing session object
attached), or `resume` (restored by id). `session_end` fires on normal close
(`JCODE_HOOK_SOURCE=close`).
### `post_tool`
Fires after every tool call. Extra fields: `JCODE_HOOK_TOOL_NAME`,
`JCODE_HOOK_STATUS`, `JCODE_HOOK_DURATION_MS`, `JCODE_HOOK_OUTPUT_BYTES` (on
success), `JCODE_HOOK_ERROR` (on failure).
## Gate hook: `pre_tool`
`pre_tool` runs **synchronously before every tool call** and can block it:
- The hook receives `JCODE_HOOK_TOOL_NAME` plus the full tool input JSON on
**stdin** (and a 16 KB-truncated copy in `JCODE_HOOK_TOOL_INPUT`).
- **Exit 0**: allow the call.
- **Exit 2**: block the call. The hook's stderr (trimmed, capped at 2000
chars) is returned to the model as the tool error, so the model can adapt.
- **Anything else fails open** with a logged warning: other exit codes,
timeout (`pre_tool_timeout_ms`, default 5s), missing binary, spawn errors.
Fail-open is deliberate: a broken policy script should degrade to "no policy"
rather than brick every session. If you need fail-closed semantics, make the
hook itself robust (it is your trust boundary, not jcode).
### Example policy script
```bash
#!/usr/bin/env bash
# ~/bin/jcode-tool-policy
# stdin: tool input JSON. Env: JCODE_HOOK_TOOL_NAME, JCODE_HOOK_SESSION_ID...
input=$(cat)
case "$JCODE_HOOK_TOOL_NAME" in
bash)
if grep -qE 'rm -rf /([^a-zA-Z]|$)|mkfs|dd if=' <<<"$input"; then
echo "blocked: destructive shell command" >&2
exit 2
fi
;;
write|edit)
if grep -q '"file_path":"/etc/' <<<"$input"; then
echo "blocked: writes to /etc are not allowed" >&2
exit 2
fi
;;
esac
exit 0
```
## Example: tmux status + desktop notification on turn end
```bash
#!/usr/bin/env bash
# ~/bin/jcode-turn-notify
if [ "$JCODE_HOOK_STATUS" = ok ]; then icon=✅; else icon=❌; fi
tmux display-message "jcode $icon ${JCODE_HOOK_SESSION_ID:0:12}" 2>/dev/null
notify-send "jcode turn $JCODE_HOOK_STATUS" \
"${JCODE_HOOK_LAST_ASSISTANT_TEXT:0:120}" 2>/dev/null
exit 0
```
## Example: JSON event log of all hook activity
Point several hooks at one script and fan out on `JCODE_HOOK_EVENT`:
```bash
#!/usr/bin/env bash
# ~/bin/jcode-event-log
echo "$JCODE_HOOK_PAYLOAD" >> ~/.local/state/jcode-events.jsonl
```
```toml
[hooks]
turn_end = "~/bin/jcode-event-log"
session_start = "~/bin/jcode-event-log"
session_end = "~/bin/jcode-event-log"
post_tool = "~/bin/jcode-event-log"
```
## Design notes
- Hook lookups are config-driven and re-read on config reload; you can add or
change hooks without restarting jcode.
- Hot paths (`pre_tool`/`post_tool`) check whether a hook is configured before
building any payload, so unconfigured hooks cost ~nothing.
- The recursion guard (`JCODE_HOOKS_DISABLED=1`) means a hook may safely call
`jcode` CLI commands without re-triggering hooks in that nested process.
docs/IOS_APP.md
+121
View File
@@ -0,0 +1,121 @@
# jcode iOS App
> Status: v2 rebuild. Pure Swift. This replaces the earlier prototype and the
> Rust-mobile-core/simulator direction, both removed in the `ios-app-restart`
> branch history.
## Product definition
A native iOS remote control for jcode servers running on your own machines.
The phone renders conversations and drives sessions; all heavy lifting (LLM
calls, tools, git, files, MCP) stays on the server. Reachability is assumed to
be Tailscale (or LAN); the app never talks to LLM providers directly.
Design identity: dark, calm, terminal-native without being retro. Mint accent
(`#4DD9A6`) for live/connected state. Dense information in touchable cards.
## Architecture decision: pure Swift
The app is a thin client over an existing, server-owned protocol. Behavior
worth sharing lives server-side already. A shared Rust core would duplicate
the server protocol a third time and require an FFI bridge, custom renderer
work, and a parallel simulator to stay honest. Instead:
- **Swift owns the client.** SwiftUI for UI, Swift 6 concurrency, `@Observable`.
- **The server protocol is the single source of truth.** The Swift codec is
validated by fixture tests against real wire JSON; drift fails tests.
- **Testability without devices** comes from layering, not a simulator:
everything below the view layer builds and tests on macOS via `swift test`.
## Layering
```
ios/
Package.swift SPM package, builds on macOS + iOS
Sources/JCodeKit/ platform-free client core (no UIKit)
Gateway.swift endpoints: /health, /pair, /ws
Pairing.swift pairing code -> auth token
Wire.swift Request/ServerEvent codecs (NDJSON over WS)
Transport.swift WebSocketTransport protocol + URLSession impl
Connection.swift actor: connect/auth/reconnect, AsyncStream<ServerEvent>
SessionReducer.swift pure state machine: events -> transcript/app state
CredentialStore.swift Keychain-backed server credentials (protocol + impl)
Sources/JCodeMobile/ SwiftUI app shell (iOS only)
JCodeMobileApp.swift entry
AppModel.swift @Observable glue: Connection + Reducer -> views
Views/ Pairing, Chat, Transcript, Sessions, Settings
QRScannerView.swift camera pairing
Theme.swift colors/typography tokens
Tests/JCodeKitTests/ swift test on macOS: codec fixtures, reducer, pairing
project.yml XcodeGen spec for the app target
```
Rules:
- `JCodeKit` never imports UIKit/SwiftUI. It must compile for macOS so the
whole behavior layer is testable headlessly by agents on this machine.
- Views contain no protocol or state-transition logic. `AppModel` only
forwards actions and publishes reducer output.
- `SessionReducer` is a pure function `(State, ServerEvent) -> State`
(plus local user intents). All streaming/tool/session edge cases are unit
tested there, replacing the old Rust simulator's role.
## Protocol
Server side (already shipped, unchanged):
- `jcode pair` CLI generates a 6-digit code (5 min TTL) and QR with
`jcode://pair?host=H&port=P&code=C`.
- `POST http://host:7643/pair` with `{code, device_id, device_name}` returns
`{token, server_name, server_version}`. Token is stored hashed server-side.
- `GET /health` for reachability checks.
- `ws://host:7643/ws?token=...` upgrades to a WebSocket carrying the same
newline-delimited JSON protocol as Unix-socket TUI clients
(`crates/jcode-protocol/src/wire.rs`, `#[serde(tag = "type")]`).
Client v1 requests: `subscribe`, `message`, `cancel`, `soft_interrupt`,
`ping`, `get_history`, `resume_session`, `set_model`, `rename_session`,
`clear`.
Client v1 events consumed: `ack`, `text_delta`, `reasoning_delta`,
`reasoning_done`, `text_replace`, `tool_start`, `tool_input`, `tool_exec`,
`tool_done`, `message_end`, `done`, `error`, `pong`, `state`, `session`,
`session_renamed`, `history`, `model_changed`, `available_models_updated`,
`tokens`, `interrupted`, `status_detail`, `notification`, `compaction`.
Unknown event types are ignored by design (forward compatibility).
## Feature scope
v1 (this rebuild):
- Pair via QR scan or manual host/port/code; multiple saved servers in Keychain
- Connect/disconnect lifecycle with automatic reconnect + backoff
- Chat: send, streamed assistant text, markdown rendering, reasoning indicator
- Tool calls rendered as collapsible cards with live status
- Interrupt (cancel) and soft-interrupt (queue a message mid-run)
- Session list (from history payload), switch via `resume_session`, rename
- Model picker from `available_models`
- Token usage + connection status surfaces
- Error and disconnect banners that never silently drop user input
Later (explicitly out of v1): push notifications (APNs), Live Activities,
voice input, image attachments, tool approval UX, ambient/swarm dashboards,
widgets, Mac Catalyst polish.
## Testing strategy
1. `swift test` (macOS, no Xcode project needed):
- codec round-trips against fixture JSON captured from the real server
- `SessionReducer` streaming scenarios (deltas, tool lifecycle, history
replay, interrupts, errors, reconnect resubscribe)
- pairing client against a stubbed URLProtocol
- connection actor against a fake `WebSocketTransport`
2. `xcodebuild build` of the app target (XcodeGen) keeps the UI compiling.
3. Manual/automated device pass via `xcrun simctl` once a simulator runtime is
installed; not a CI gate.
## CI
A macOS job runs `swift test` in `ios/` plus `xcodegen` + `xcodebuild build`
for the app target. TestFlight delivery can be re-added later (the previous
Codemagic pipeline was removed with the prototype).
docs/KEYMAP_CONFLICTS.md
+88
View File
@@ -0,0 +1,88 @@
# Keybinding conflict detection
jcode runs inside a terminal, which runs inside an OS. Both layers can claim a
key chord before it ever reaches jcode (for example Ghostty binding `Ctrl+Tab`
to "next tab", or macOS binding `Cmd+Space` to Spotlight). When that happens, a
jcode keybinding silently does nothing and it is not obvious why.
This feature discovers the key bindings that exist on the machine, compares them
against jcode's own configured bindings, and warns about overlaps.
## What it can and cannot detect
**Can detect (config-declared intercepts):**
- **macOS system shortcuts** read from `com.apple.symbolichotkeys` (Spotlight,
Mission Control, screenshots, input-source switching, etc.). Only shortcuts
that are *enabled* are considered.
- **Terminal emulator bindings.** Currently Ghostty, via
`ghostty +list-keybinds`, which reports the *effective* binding set (built-in
defaults merged with the user's config). This also catches rewrites such as
Ghostty mapping `Alt+Left`/`Alt+Right` to word-navigation escape sequences.
**Cannot detect:**
- Ad-hoc remappers (Karabiner-Elements, BetterTouchTool), window managers, or
global launcher hotkeys that are not stored in a file we read.
- Terminals other than Ghostty (yet). Adding one is a self-contained adapter
(see "Adding a terminal adapter" below).
It is a snapshot taken at startup, not a live hook, so changes made while jcode
is running are not seen until the snapshot is refreshed.
## How it surfaces
- **`/keys`** prints a full report: detected terminal, discovered binding
counts, and each conflict tied to the exact `[keybindings]` config field that
owns it. `/keys refresh` forces a rescan of the machine (otherwise a cached
snapshot up to a day old is reused).
- **Startup notice.** On launch, if the set of conflicts has *changed* since the
last time we warned, jcode shows a one-time heads-up pointing at `/keys`. It is
debounced by a signature of the conflict set, so users are warned once per
distinct set of conflicts and never nagged on every launch.
## Resolving a conflict
The report names the conflicting jcode action and its config field, e.g.:
```
⚠ Ctrl+Tab
jcode: Switch to next model (keybindings.model_switch_next = "ctrl+tab")
taken by terminal: next_tab
```
To fix, either:
- rebind the jcode action in `~/.jcode/config.toml` under `[keybindings]`
(e.g. `model_switch_next = "ctrl+shift+m"`), or
- change the conflicting shortcut in your terminal or OS settings.
## Implementation
All logic lives in `crates/jcode-setup-hints/src/keymap/`:
- `chord.rs` - `KeyChord`, a normalized `(cmd/ctrl/alt/shift + key)` that unifies
the different key spellings each source uses, plus `KeyChord::parse` for
jcode's own binding-string grammar.
- `macos_hotkeys.rs` - decode `com.apple.symbolichotkeys`
`[ascii, keycode, modmask]` triples (pure logic + a thin subprocess wrapper).
- `terminal.rs` - parse `ghostty +list-keybinds` output (pure logic + wrapper).
- `source.rs` - `DiscoveredBinding` and its `KeySource`.
- `conflicts.rs` - enumerate `KeybindingsConfig` as chords, diff against a
snapshot, and produce `Conflict`s keyed to config fields. `conflict_signature`
produces the stable signature used for startup debounce.
- `report.rs` - render the human-readable report and the compact status line.
- `mod.rs` - `collect_snapshot` / `refresh_and_save` / `snapshot_cached_or_refresh`
(persisted to `~/.jcode/keymap-snapshot.json`).
The pure parsing/decoding/diffing functions are unit-tested and do not touch the
machine; only the `read_*` wrappers shell out.
## Adding a terminal adapter
1. Add a `read_<terminal>_keybinds()` in `terminal.rs` (or a sibling module) that
produces `Vec<DiscoveredBinding>` with `source: KeySource::Terminal`, keeping
the parser pure and the subprocess/file read thin.
2. Call it from `collect_snapshot()` in `mod.rs`, ideally gated on the detected
terminal so we do not shell out to tools that are not present.
3. Add unit tests for the parser using sample config/output text.
docs/MEMORY_ARCHITECTURE.md
+875
View File
@@ -0,0 +1,875 @@
# Memory Architecture Design
> **Status:** Implemented (Core), Planned (Graph-Based Hybrid)
> **Updated:** 2026-01-27
Local embeddings + lightweight sidecar (GPT-5.3 Codex Spark) are implemented and running in production. This document describes both the current implementation and the planned graph-based hybrid architecture.
## Overview
See also: [Memory Regression Budget](./MEMORY_BUDGET.md) for the current measurable guardrails and review expectations.
A multi-layered memory system for cross-session learning that mimics how human memory works - relevant memories "pop up" when triggered by context rather than requiring explicit recall.
**Key Design Decisions:**
1. **Fully async and non-blocking** - The main agent never waits for memory; results from turn N are available at turn N+1
2. **Graph-based organization** - Memories form a connected graph with tags, clusters, and semantic links
3. **Cascade retrieval** - Embedding hits trigger BFS traversal to find related memories
4. **Hybrid grouping** - Combines explicit tags, automatic clusters, and semantic links
---
## Architecture Overview
```mermaid
graph TB
subgraph "Main Agent"
MA[TUI App]
MP[build_memory_prompt]
TP[take_pending_memory]
end
subgraph "Memory Agent"
CH[Context Handler]
EMB[Embedder<br/>all-MiniLM-L6-v2]
SR[Similarity Search]
CR[Cascade Retrieval]
HC[Sidecar<br/>GPT-5.3 Codex Spark]
end
subgraph "Memory Graph"
MG[(petgraph<br/>DiGraph)]
MS[Memory Nodes]
TN[Tag Nodes]
CN[Cluster Nodes]
end
MA -->|mpsc channel| CH
CH --> EMB
EMB --> SR
SR -->|initial hits| CR
CR -->|BFS traversal| MG
MG --> MS
MG --> TN
MG --> CN
CR -->|candidates| HC
HC -->|verified| TP
TP -->|next turn| MA
```
---
## Graph-Based Data Model
### Node Types
```mermaid
graph LR
subgraph "Node Types"
M((Memory))
T[Tag]
C{Cluster}
end
M -->|HasTag| T
M -->|InCluster| C
M -.->|RelatesTo| M
M ==>|Supersedes| M
M -.->|Contradicts| M
style M fill:#e1f5fe
style T fill:#fff3e0
style C fill:#f3e5f5
```
| Node Type | Description | Storage |
|-----------|-------------|---------|
| **Memory** | Core memory entry (fact, preference, procedure) | Content, metadata, embedding |
| **Tag** | Explicit label (user-defined or inferred) | Name, description, count |
| **Cluster** | Automatic grouping via embedding similarity | Centroid embedding, member count |
### Edge Types
| Edge Type | From → To | Description |
|-----------|-----------|-------------|
| `HasTag` | Memory → Tag | Memory has this explicit tag |
| `InCluster` | Memory → Cluster | Memory belongs to auto-discovered cluster |
| `RelatesTo` | Memory → Memory | Semantic relationship (weighted) |
| `Supersedes` | Memory → Memory | Newer memory replaces older |
| `Contradicts` | Memory → Memory | Conflicting information |
| `DerivedFrom` | Memory → Memory | Procedural knowledge derived from facts |
### Rust Implementation
```rust
use petgraph::graph::DiGraph;
/// Node in the memory graph
#[derive(Debug, Clone)]
pub enum MemoryNode {
Memory(MemoryEntry),
Tag(TagEntry),
Cluster(ClusterEntry),
}
/// Edge relationships
#[derive(Debug, Clone)]
pub enum EdgeKind {
HasTag,
InCluster,
RelatesTo { weight: f32 },
Supersedes,
Contradicts,
DerivedFrom,
}
/// The memory graph
pub struct MemoryGraph {
graph: DiGraph<MemoryNode, EdgeKind>,
// Indexes for fast lookup
memory_index: HashMap<String, NodeIndex>,
tag_index: HashMap<String, NodeIndex>,
cluster_index: HashMap<String, NodeIndex>,
}
```
---
## Hybrid Grouping System
The memory system uses three complementary organization methods:
```mermaid
graph TB
subgraph "Explicit: Tags"
T1["rust"]
T2["auth-system"]
T3["user-preference"]
end
subgraph "Automatic: Clusters"
C1[("Error Handling<br/>Cluster")]
C2[("API Patterns<br/>Cluster")]
end
subgraph "Semantic: Links"
L1["relates_to"]
L2["supersedes"]
L3["contradicts"]
end
M1((Memory 1)) --> T1
M1 --> C1
M1 -.-> L1
L1 -.-> M2((Memory 2))
M2 --> T1
M2 --> C2
M3((Memory 3)) --> T2
M3 ==> L2
L2 ==> M4((Memory 4))
```
### 1. Tags (Explicit)
User-defined or automatically inferred labels.
**Sources:**
- User explicitly tags: `memory { action: "remember", tags: ["rust", "auth"] }`
- Inferred from context (file paths, topics, entities)
- Extracted by sidecar during end-of-session processing
**Examples:**
- `#project:jcode` - Project-specific
- `#rust`, `#python` - Language-specific
- `#auth`, `#database` - Domain-specific
- `#preference`, `#correction` - Category tags
### 2. Clusters (Automatic)
Automatically discovered groupings based on embedding similarity.
**Algorithm:**
1. Periodically run HDBSCAN on memory embeddings
2. Create/update cluster nodes for dense regions
3. Assign `InCluster` edges to nearby memories
4. Track cluster centroids for fast lookup
**Benefits:**
- Discovers hidden patterns user didn't explicitly tag
- Groups related memories even without shared tags
- Enables "find similar" queries
### 3. Links (Semantic Relationships)
Explicit relationships between memories.
**Types:**
- **RelatesTo**: General semantic connection (weighted 0.0-1.0)
- **Supersedes**: Newer information replaces older
- **Contradicts**: Conflicting information (both kept, flagged)
- **DerivedFrom**: Procedural knowledge derived from facts
**Discovery:**
- Contradiction detection on write
- Sidecar identifies relationships during verification
- User can explicitly link memories
---
## Cascade Retrieval
When context triggers memory search, cascade retrieval finds related memories through graph traversal.
```mermaid
sequenceDiagram
participant C as Context
participant E as Embedder
participant S as Similarity Search
participant G as Graph BFS
participant H as Sidecar (Codex Spark)
participant R as Results
C->>E: Current context
E->>S: Context embedding
S->>S: Find top-k similar memories
S->>G: Initial hits (seed nodes)
loop BFS Traversal depth 2
G->>G: Follow HasTag edges
G->>G: Follow InCluster edges
G->>G: Follow RelatesTo edges
end
G->>H: Candidate memories
H->>H: Verify relevance to context
H->>R: Filtered, ranked memories
```
### Algorithm
```rust
pub fn cascade_retrieve(
&self,
context_embedding: &[f32],
max_depth: usize,
max_results: usize,
) -> Vec<(MemoryEntry, f32)> {
// Step 1: Embedding similarity search
let initial_hits = self.similarity_search(context_embedding, 10);
// Step 2: BFS traversal from hits
let mut visited: HashSet<NodeIndex> = HashSet::new();
let mut candidates: Vec<(NodeIndex, f32, usize)> = Vec::new();
let mut queue: VecDeque<(NodeIndex, usize)> = VecDeque::new();
for (node, score) in initial_hits {
queue.push_back((node, 0));
candidates.push((node, score, 0));
}
while let Some((node, depth)) = queue.pop_front() {
if depth >= max_depth || visited.contains(&node) {
continue;
}
visited.insert(node);
// Traverse edges
for edge in self.graph.edges(node) {
let neighbor = edge.target();
if visited.contains(&neighbor) {
continue;
}
let edge_weight = match edge.weight() {
EdgeKind::HasTag => 0.8, // Strong signal
EdgeKind::InCluster => 0.6, // Medium signal
EdgeKind::RelatesTo { weight } => *weight,
EdgeKind::Supersedes => 0.9, // Very relevant
_ => 0.3,
};
// Decay score by depth
let decayed_score = edge_weight * (0.7_f32).powi(depth as i32 + 1);
if let MemoryNode::Memory(_) = &self.graph[neighbor] {
candidates.push((neighbor, decayed_score, depth + 1));
}
queue.push_back((neighbor, depth + 1));
}
}
// Step 3: Dedupe, sort, and return top results
candidates.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
candidates.into_iter()
.filter_map(|(node, score, _)| {
if let MemoryNode::Memory(entry) = &self.graph[node] {
Some((entry.clone(), score))
} else {
None
}
})
.take(max_results)
.collect()
}
```
### Retrieval Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `similarity_threshold` | 0.4 | Minimum embedding similarity for initial hits |
| `max_initial_hits` | 10 | Number of embedding search results |
| `max_depth` | 2 | BFS traversal depth limit |
| `max_results` | 10 | Final results to return |
| `edge_decay` | 0.7 | Score decay per traversal step |
---
## Memory Entry Schema
```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MemoryEntry {
// Identity
pub id: String,
pub content: String,
pub category: MemoryCategory,
// Classification
pub memory_type: MemoryType, // Fact, Preference, Procedure, Correction
pub scope: MemoryScope, // Global, Project, Session
// Source tracking
pub session_id: Option<String>,
pub message_range: Option<(u32, u32)>,
pub file_paths: Vec<String>,
pub provenance: Provenance, // UserStated, Observed, Inferred
// Lifecycle
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub last_accessed: DateTime<Utc>,
pub access_count: u32,
pub strength: u32, // Consolidation count
// Trust & status
pub confidence: f32, // 0.0-1.0, decays over time
pub trust_score: f32, // Source-based trust
pub active: bool,
pub superseded_by: Option<String>,
// Embedding
pub embedding: Option<Vec<f32>>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum MemoryType {
Fact, // "This project uses PostgreSQL"
Preference, // "User prefers 4-space indentation"
Procedure, // "To deploy: run make deploy"
Correction, // "Don't use deprecated API"
Negative, // "Never commit .env files"
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum Provenance {
UserStated, // User explicitly said it
UserCorrected, // User corrected agent behavior
Observed, // Agent observed from behavior
Inferred, // Agent inferred from context
Extracted, // Extracted from session summary
}
```
---
## Advanced Features
### 1. Temporal Awareness
Memories have temporal context:
```rust
pub struct TemporalContext {
pub session_scope: bool, // Only relevant in session
pub recency_weight: f32, // Recent access boost
pub seasonal: Option<String>, // "end-of-sprint", "release-week"
}
```
**Recency boost formula:**
```
boost = 1.0 + (0.5 * e^(-hours_since_access / 24))
```
### 2. Confidence Decay
Confidence decays over time based on memory type:
| Memory Type | Half-life | Rationale |
|-------------|-----------|-----------|
| Correction | 365 days | User corrections are high value |
| Preference | 90 days | Preferences may evolve |
| Fact | 30 days | Codebase facts can become stale |
| Procedure | 60 days | Procedures change less often |
| Inferred | 7 days | Low-confidence inferences |
**Decay formula:**
```
confidence = initial_confidence * e^(-age_days / half_life)
* (1 + 0.1 * log(access_count + 1))
* trust_weight
```
### 3. Negative Memories
Things the agent should avoid doing:
```rust
MemoryEntry {
content: "Never use println! for logging in production code",
memory_type: MemoryType::Negative,
trigger_patterns: vec!["println!", "print!", "dbg!"],
...
}
```
**Surfacing:** Negative memories are surfaced when trigger patterns match current context.
### 4. Procedural Memories
How-to knowledge with structured steps:
```rust
pub struct Procedure {
pub name: String,
pub trigger: String, // "deploy to production"
pub steps: Vec<String>,
pub prerequisites: Vec<String>,
pub warnings: Vec<String>,
}
```
### 5. Provenance Tracking
Every memory tracks its source:
```rust
pub struct ProvenanceChain {
pub source: Provenance,
pub session_id: String,
pub timestamp: DateTime<Utc>,
pub context_snippet: String, // What was being discussed
pub confidence_reason: String, // Why this confidence level
}
```
### 6. Feedback Loops
Memories strengthen or weaken based on use:
```rust
impl MemoryEntry {
pub fn on_used(&mut self, helpful: bool) {
self.access_count += 1;
self.last_accessed = Utc::now();
if helpful {
self.strength = self.strength.saturating_add(1);
self.confidence = (self.confidence + 0.05).min(1.0);
} else {
self.confidence = (self.confidence - 0.1).max(0.0);
}
}
}
```
### 7. Post-Retrieval Maintenance
After serving memories to the main agent, the memory agent has valuable context it can use for background maintenance. This "opportunistic maintenance" happens asynchronously without blocking.
```mermaid
graph LR
subgraph "Retrieval Phase"
R1[Context Embedding]
R2[Similarity Search]
R3[Cascade BFS]
R4[Sidecar Verify]
R5[Serve to Agent]
end
subgraph "Maintenance Phase (Background)"
M1[Link Discovery]
M2[Cluster Update]
M3[Confidence Boost]
M4[Gap Detection]
end
R5 --> M1
R5 --> M2
R5 --> M3
R5 --> M4
style M1 fill:#1f6feb
style M2 fill:#1f6feb
style M3 fill:#1f6feb
style M4 fill:#1f6feb
```
**Available Context:**
- Current context embedding
- All memories that were retrieved (initial hits + BFS expansion)
- Which memories passed sidecar verification (actually relevant)
- Which were rejected (retrieved but not relevant)
- Co-occurrence patterns (memories that appear together)
**Maintenance Tasks:**
| Task | Trigger | Action |
|------|---------|--------|
| **Link Discovery** | 2+ memories verified relevant | Create/strengthen `RelatesTo` edges between co-relevant memories |
| **Cluster Refinement** | Retrieved memories span clusters | Update cluster centroids, consider merging nearby clusters |
| **Confidence Boost** | Memory verified relevant | Increment access count, boost confidence |
| **Confidence Decay** | Memory retrieved but rejected | Slightly decay confidence (may be stale) |
| **Gap Detection** | Context has no relevant memories | Log potential memory gap for later extraction |
| **Tag Inference** | Multiple memories share context | Infer common tag from context if none exists |
**Implementation:**
```rust
impl MemoryAgent {
/// Called after serving memories, runs maintenance in background
async fn post_retrieval_maintenance(&self, ctx: RetrievalContext) {
// Don't block - spawn maintenance tasks
tokio::spawn(async move {
// 1. Strengthen links between co-relevant memories
if ctx.verified_memories.len() >= 2 {
self.discover_links(&ctx.verified_memories, &ctx.embedding).await;
}
// 2. Boost confidence for verified memories
for mem_id in &ctx.verified_memories {
self.boost_confidence(mem_id).await;
}
// 3. Decay confidence for rejected memories
for mem_id in &ctx.rejected_memories {
self.decay_confidence(mem_id, 0.02).await; // Gentle decay
}
// 4. Detect gaps (context had no relevant memories)
if ctx.verified_memories.is_empty() && ctx.initial_hits > 0 {
self.log_memory_gap(&ctx.embedding, &ctx.context_snippet).await;
}
// 5. Periodic cluster update (every N retrievals)
if self.retrieval_count.fetch_add(1, Ordering::Relaxed) % 50 == 0 {
self.update_clusters().await;
}
});
}
}
```
**Gap Detection for Future Learning:**
When retrieval finds no relevant memories but the context seems important, log it:
```rust
struct MemoryGap {
context_embedding: Vec<f32>,
context_snippet: String,
timestamp: DateTime<Utc>,
session_id: String,
}
```
These gaps can be reviewed during end-of-session extraction to create new memories for topics the system didn't know about.
### 8. Scope Levels
Memories exist at different scopes:
```mermaid
graph TB
subgraph "Scope Hierarchy"
G[Global<br/>User-wide preferences]
P[Project<br/>Codebase-specific]
S[Session<br/>Current conversation]
end
G --> P
P --> S
style G fill:#e8f5e9
style P fill:#e3f2fd
style S fill:#fff3e0
```
| Scope | Lifetime | Examples |
|-------|----------|----------|
| Global | Permanent | "User prefers vim keybindings" |
| Project | Until deleted | "This project uses async/await" |
| Session | Current session | "Working on auth refactor" |
---
## Async Processing Pipeline
```mermaid
sequenceDiagram
participant MA as Main Agent<br/>TUI App
participant CH as mpsc Channel
participant MEM as Memory Agent<br/>Background Task
participant EMB as Embedder
participant GR as Graph Store
participant HC as Sidecar (Codex Spark)
Note over MA,MEM: Turn N
MA->>MA: build_memory_prompt()
MA->>MA: take_pending_memory()
Note right of MA: Returns Turn N-1 results
MA->>CH: try_send(ContextUpdate)
Note right of CH: Non-blocking
MA->>MA: Continue with LLM call
CH->>MEM: update_context_sync()
MEM->>EMB: Embed context
EMB-->>MEM: Context embedding
MEM->>GR: Similarity search
GR-->>MEM: Initial hits
MEM->>GR: BFS traversal
GR-->>MEM: Related memories
MEM->>HC: Verify relevance
HC-->>MEM: Filtered results
MEM->>MEM: Topic change detection
Note right of MEM: Clear surfaced if sim < 0.3
MEM->>MEM: set_pending_memory()
Note right of MEM: Available at Turn N+1
```
**Key Points:**
- Memory agent is a **singleton** (OnceCell) - only one instance ever runs
- Communication is **non-blocking** via `try_send()` on mpsc channel
- Results arrive **one turn behind** (processed in background)
- **Topic change detection** resets surfaced set when conversation shifts
- **Cascade retrieval** traverses graph for related memories
---
## Storage Layout
```
~/.jcode/memory/
├── graph.json # Serialized petgraph
├── projects/
│ └── <project_hash>.json # Per-directory memories
├── global.json # User-wide memories
├── embeddings/
│ └── <memory_id>.vec # Embedding vectors
├── clusters/
│ └── cluster_metadata.json # Cluster centroids and metadata
└── tags/
└── tag_index.json # Tag → memory mappings
```
---
## Memory Tools
Available to the main agent:
```
memory { action: "remember", content: "...", category: "fact|preference|correction",
scope: "project|global", tags: ["tag1", "tag2"] }
memory { action: "recall" } # Get relevant memories for context
memory { action: "search", query: "..." } # Semantic search
memory { action: "list", tag: "..." } # List by tag
memory { action: "forget", id: "..." } # Deactivate memory
memory { action: "link", from: "id1", to: "id2", relation: "relates_to" }
memory { action: "tag", id: "...", tags: ["new", "tags"] }
```
---
## Implementation Status
### Phase 1: Basic Memory Tools ✅
- [x] Memory store with file persistence
- [x] Basic memory tool
- [x] Integration with agent
### Phase 2: Embedding Search ✅
- [x] Local all-MiniLM-L6-v2 via tract-onnx
- [x] Background embedding process
- [x] Similarity search with cosine distance
### Phase 3: Memory Agent ✅
- [x] Async channel communication
- [x] Lightweight sidecar for relevance verification (currently GPT-5.3 Codex Spark)
- [x] Topic change detection
- [x] Surfaced memory tracking
### Phase 4: Graph-Based Architecture ✅
- [x] HashMap-based graph structure (simpler than petgraph for JSON serialization)
- [x] Tag nodes and HasTag edges
- [x] Cluster discovery and InCluster edges
- [x] Semantic link edges (RelatesTo)
- [x] Cascade retrieval algorithm with BFS traversal
### Phase 5: Post-Retrieval Maintenance ✅
- [x] Link discovery (co-relevant memories)
- [x] Confidence boost/decay on retrieval
- [x] Gap detection for missing knowledge
- [x] Periodic cluster refinement
- [x] Tag inference from context
### Phase 6: Advanced Features ✅
- [x] Confidence decay system (time-based with category-specific half-lives)
- [ ] Negative memories and trigger patterns
- [ ] Procedural memory support
- [x] Provenance tracking
- [x] Feedback loops (boost on use, decay on rejection)
- [ ] Temporal awareness
### Phase 7: Full Integration ✅
- [x] End-of-session extraction
- [x] Sidecar consolidation on write (see below)
- [x] User control CLI (`jcode memory` commands)
- [x] Memory export/import
### Phase 7.5: Sidecar Consolidation (Inline, Per-Turn) ✅
Lightweight consolidation that runs in the memory sidecar after returning results to the main agent. Only operates on memories already retrieved — no extra lookups, zero added latency.
`extract_from_context()` now performs inline write-time consolidation:
- [x] **Duplicate detection on write** — semantically similar memories are reinforced instead of duplicated.
- [x] **Contradiction detection on write** — contradictory memories are superseded during incremental extraction.
- [x] **Reinforcement provenance** — `MemoryEntry` tracks `Vec<Reinforcement>` breadcrumbs (`session_id`, `message_index`, `timestamp`).
### Phase 8: Deep Memory Consolidation (Ambient Garden) 📋
Full graph-wide consolidation that runs during ambient mode background cycles. See [AMBIENT_MODE.md](./AMBIENT_MODE.md) for the ambient mode design.
- [ ] Graph-wide similarity-based memory merging
- [ ] Redundancy detection and deduplication (beyond sidecar's local scope)
- [ ] Contradiction resolution (across full graph, not just retrieved set)
- [ ] Fact verification against codebase (check if factual memories are still true)
- [ ] Retroactive session extraction (crashed/missed sessions)
- [ ] Cluster reorganization
- [ ] Weak memory pruning (confidence < 0.05 AND strength <= 1)
- [ ] Relationship discovery across sessions
- [ ] Embedding backfill for memories missing embeddings
- [ ] Knowledge graph optimization
---
## Privacy & Security
### Do Not Remember
- API keys, secrets, credentials
- Passwords or tokens
- Personal identifying information
- File contents marked sensitive
### Filtering
Before storing any memory, scan for:
- Regex patterns for secrets (API keys, passwords)
- Files in `.gitignore` or `.secretsignore`
- Content from `.env` files
### User Control
- All memories stored in human-readable JSON
- CLI for viewing/editing/deleting
- Option to disable memory entirely
- Export/import for backup
---
## Future: Memory Consolidation (Sleep-Like Processing)
> **Status:** TODO - Design pending
Similar to how humans consolidate memories during sleep, jcode can run background consolidation to optimize the memory graph:
### Concept
```mermaid
graph LR
subgraph "Active Use"
A[Raw Memories]
B[Redundant Facts]
C[Weak Links]
D[Scattered Tags]
end
subgraph "Consolidation"
E[Merge Similar]
F[Detect Contradictions]
G[Prune Weak]
H[Reorganize Clusters]
end
subgraph "Optimized"
I[Unified Facts]
J[Resolved Conflicts]
K[Strong Connections]
L[Clean Taxonomy]
end
A --> E --> I
B --> E
B --> F --> J
C --> G --> K
D --> H --> L
```
### Potential Features
| Feature | Description |
|---------|-------------|
| **Similarity Merge** | Combine memories with >0.95 embedding similarity |
| **Redundancy Detection** | Find memories that express the same fact differently |
| **Contradiction Resolution** | Surface conflicting memories for user decision |
| **Weak Pruning** | Remove memories with low confidence + low access |
| **Cluster Optimization** | Re-run clustering, merge small clusters |
| **Link Strengthening** | Increase weights on frequently co-accessed pairs |
| **Tag Cleanup** | Merge similar tags, remove orphans |
### Architecture Options (TBD)
1. **Periodic daemon** - Run consolidation every N hours
2. **On-idle trigger** - Run when no active sessions for M minutes
3. **Capacity-based** - Run when memory count exceeds threshold
4. **Manual command** - User-triggered via `/consolidate`
### Open Questions for Consolidation
- How to handle user confirmation for destructive merges?
- Should consolidation be reversible?
- What's the right frequency/trigger?
- How to balance between "perfect organization" and "keep everything"?
---
## Open Questions
1. **Multi-machine sync:** Should memories sync across devices via encrypted backup?
2. **Team sharing:** Should some memories be shareable across a team?
3. **Cluster algorithm:** HDBSCAN vs k-means vs hierarchical clustering?
4. **Graph persistence:** JSON serialization vs SQLite for larger graphs?
---
*Last updated: 2026-01-27*
docs/MEMORY_BUDGET.md
+150
View File
@@ -0,0 +1,150 @@
# Memory Regression Budget
Status: active guardrail
Updated: 2026-04-18
This document defines the current memory regression budget for jcode.
The goal is not to freeze memory usage forever. The goal is to make memory changes:
- measurable
- reviewable
- intentionally justified
Where possible, budgets below are tied to counters and caps already exposed by the codebase rather than guessed RSS numbers.
## How to collect the metrics
Use existing debug surfaces instead of ad hoc instrumentation:
- TUI aggregate memory profile: `:debug memory`
- TUI memory sample history: `:debug memory-history`
- Markdown cache profile: `:debug markdown:memory`
- Mermaid cache profile: `:debug mermaid:memory`
- Agent/session memory profile via debug socket: `agent:memory`
Primary sources in code:
- `src/tui/app/debug_cmds.rs`
- `src/tui/memory_profile.rs`
- `src/session.rs`
- `src/tui/markdown.rs`
- `src/tui/mermaid.rs`
- `src/runtime_memory_log.rs`
## Budget model
We use two kinds of budgets:
1. Hard caps
- These are explicit limits already enforced by caches.
- Regressions here mean the code changed its bound or bypassed it.
2. Ratchet expectations
- These are expected relationships between memory counters.
- Regressions here are allowed only with explanation and updated docs/tests.
## Hard caps
### Markdown cache budget
Source: `src/tui/markdown.rs`
| Metric | Budget | Why |
|---|---:|---|
| `highlight_cache_entries` | `<= 256` | Explicit cache cap (`HIGHLIGHT_CACHE_LIMIT`) |
Required review action if violated:
- explain why the cache limit changed
- update this doc
- update any affected tests or benchmarks
### Mermaid cache budget
Sources:
- `src/tui/mermaid.rs`
- `src/tui/mermaid_cache_render.rs`
| Metric | Budget | Why |
|---|---:|---|
| `render_cache_entries` | `<= 64` | Explicit render-cache cap (`RENDER_CACHE_MAX`) |
| `image_state_entries` | `<= 12` | Explicit protocol-state cap (`IMAGE_STATE_MAX`) |
| `source_cache_entries` | `<= 8` | Explicit decoded-source cap (`SOURCE_CACHE_MAX`) |
| `active_diagrams` | `<= 128` | Explicit active-diagram cap (`ACTIVE_DIAGRAMS_MAX`) |
| `cache_disk_png_bytes` | `<= 50 MiB` | Explicit on-disk cache cap (`CACHE_MAX_SIZE_BYTES`) |
| `cache_disk_max_age_secs` | `<= 259200` | 3-day expiry (`CACHE_MAX_AGE_SECS`) |
Required review action if violated:
- document the new limit and reason
- verify eviction still works
- verify no unbounded growth path was introduced
## Ratchet expectations
### Session and transcript memory
Source: `src/session.rs`, `src/tui/memory_profile.rs`
These are not strict caps yet, but they are expected relationships.
| Metric relationship | Expectation |
|---|---|
| `provider_messages_cache.count` vs `messages.count` | Should remain in the same order of magnitude for a single session, and normally track the transcript closely |
| `session_provider_cache_json_bytes` vs `canonical_transcript_json_bytes` | Should remain comparable for normal chat flows, not explode independently |
| `transient_provider_materialization_json_bytes` | Should return to zero or near-zero outside active materialization-heavy paths |
| `display_large_tool_output_bytes` | Large values require explanation because they usually mean raw tool output is being retained too aggressively in the UI |
Required review action if violated:
- show before/after memory profiles
- explain which retention path grew
- prefer fixing duplication before raising any budget
### Runtime memory log expectations
Source: `src/runtime_memory_log.rs`
Runtime memory logs are the regression detection mechanism, not just a debug feature.
Expected behavior:
- server/client logs should be sufficient to explain large changes in:
- session/transcript totals
- provider cache totals
- TUI display totals
- side panel totals
- new large memory owners should emit attributable signals instead of appearing only as unexplained RSS growth
Required review action if violated:
- add or improve attribution before accepting the memory increase
## Review checklist for memory-affecting changes
When changing memory-heavy code, capture and include:
1. Which counters changed?
- aggregate `:debug memory`
- targeted `:debug markdown:memory` / `:debug mermaid:memory`
- `agent:memory` when session/provider cache behavior changes
2. Was a hard cap changed?
- if yes, explain why the old cap was insufficient
3. Did duplication increase?
- canonical transcript
- provider cache
- materialized provider view
- display copy
- side-panel copy
4. Did observability remain adequate?
- if memory grew, can logs/profiles explain where?
## Current initial budget summary
These are the concrete enforced limits today:
- Markdown highlight cache entries: 256
- Mermaid render cache entries: 64
- Mermaid protocol image-state entries: 12
- Mermaid decoded source-cache entries: 8
- Mermaid active diagrams: 128
- Mermaid on-disk PNG cache: 50 MiB, max age 3 days
Any intentional change to those limits must update this document in the same PR.
docs/MEMORY_GRAPH_PLAN.md
+284
View File
@@ -0,0 +1,284 @@
# Plan: Making the Memory Graph Earn Its Keep
> Status: Proposal. Companion to MEMORY_ARCHITECTURE.md. Goal: best recall accuracy.
## Current reality (verified in code)
- **Live automatic recall** (`memory_agent::process_context`) uses
`find_similar_with_embedding` -> `score_and_filter`: flat cosine over all
active memories, threshold 0.5, gap filter, top 10, then per-candidate binary
sidecar relevance, max 5 surfaced.
- The **graph traversal** (`cascade_retrieve`, BFS over tags/clusters/RelatesTo)
is only called by the manual `memory { action: search }` tool
(`tool/memory.rs:216`). It contributes **nothing** to per-turn surfacing.
- Maintenance (`post_retrieval_maintenance`) WRITES graph structure every turn
(RelatesTo links, auto-clusters + sidecar naming, inferred tags, confidence
boost/decay, pruning) but the live path never READS most of it back.
So today the graph's edges/clusters/tags are write-mostly. The parts that
matter (supersede/contradiction -> `active`, reinforcement strength, confidence
gating the active set) help data hygiene, not ranking.
## Goal: maximize recall accuracy in BOTH modes
Both modes are first-class targets. They share Stage 1 (candidate generation)
and the graph layer, but diverge at the quality/judgment stage. Strategy:
push as much accuracy as possible into the **shared local stack** (better
embedder, hybrid, query construction, graph rerank, priors) so Mode 1 gets
strong on its own, then let Mode 2's LLM add a final precision layer on top of
an already-good candidate set rather than compensating for a weak one.
### Shared local stack (lifts both modes)
- recall-2 better embedder + asymmetric prefixes
- recall-3 focused query construction (current intent, not 8k blob)
- recall-4 hybrid dense + BM25 + RRF
- recall-6 recency/confidence/strength/scope priors in the score
- Phases A-C graph: supersede-authoritative, 1-hop expansion, dedup
- A local cross-encoder reranker (small, on-device) as the Mode 1 top stage
### Mode 1 (no LLM) - get it as close to Mode 2 as possible
- Replace raw-cosine top-5 with: hybrid recall -> graph expansion -> local
cross-encoder rerank -> priors -> calibrated cutoff. No LLM needed for any of
this; a cross-encoder is the single biggest precision lever available offline.
- Tune the calibrated cutoff per-mode (Mode 1 can afford slightly higher recall
since there's no LLM filter downstream; rely on the cross-encoder for precision).
- Optional: cheap local query expansion (synonyms/identifier splitting) since
there's no LLM to do query rewriting.
### Mode 2 (LLM) - add precision, don't redo recall
- Feed the SAME strong candidate set (hybrid + graph + cross-encoder top ~15-20)
into one **listwise** LLM rerank, replacing today's independent binary calls
(binary calls can't compare candidates and waste the LLM's judgment).
- Use the LLM for query rewriting / HyDE at Stage 0 to fix hard recall misses
that the local stack can't reach.
- Keep the LLM as the final arbiter for contradictions and ambiguous relevance.
### Why this converges both
Mode 1 ceiling rises to "best offline retriever + cross-encoder" (very high).
Mode 2 starts from that same ceiling and adds LLM rewriting + listwise judgment,
so it's strictly >= Mode 1. The harness (recall-1) tracks both columns each
change so neither regresses.
## Two operating modes (gate: `agents.memory_sidecar_enabled`, env `JCODE_MEMORY_SIDECAR_ENABLED`, default off)
The system runs in two distinct modes; recall behavior differs substantially.
### Mode 1 - Sidecar OFF (embedding-only, no LLM)
- Surfacing (`evaluate_candidates`): skips LLM, takes top candidates by **raw
cosine** (up to 5). No relevance verification, no listwise judgment.
- Extraction: `extract_from_context` + final extraction are **skipped**. Memories
are created ONLY via the explicit `memory` tool, not auto-learned.
- Cluster naming: falls back to `infer_candidate_tag` (heuristic, no LLM).
- Net: fully local, zero LLM cost; weakest precision (no filtering) and no auto
memory growth.
### Mode 2 - Sidecar ON (LLM-assisted)
- Surfacing: per-candidate binary relevance check by sidecar LLM (parallel, max 5).
- Extraction: auto-extract on topic change + every 12 turns + session end, with
LLM dedup/contradiction checks.
- Maintenance: LLM-named clusters, contradiction detection.
### Implications for this plan
- Every recall improvement must be evaluated in BOTH modes (recall-1 harness
should report two columns).
- Phases A-C (graph as reranker/dedup/expansion) are **pure local** and benefit
Mode 1 the most, since Mode 1 currently has no quality layer beyond cosine.
- recall-5 (rerank) has two implementations: a local cross-encoder path for
Mode 1, and the listwise LLM rerank for Mode 2 (replacing today's binary calls).
- Phase D maintenance trimming primarily affects Mode 2 cost (cluster naming is
the LLM line item); Mode 1 already uses the heuristic fallback.
## Edge types and what each is good for
| Edge | Source of truth | Use in recall |
|---------------|-----------------|----------------|
| `Supersedes` | contradiction/dedup on write | Keep ONLY newest version in results; demote/hide superseded |
| `Contradicts` | sidecar on write | Surface both + flag conflict; never silently pick one |
| `RelatesTo` | co-relevance maintenance | 1-hop expansion to rescue near-misses |
| `DerivedFrom` | co-extraction | 1-hop expansion (procedures <-> facts) |
| `HasTag` | user + inference | Lexical/filter signal, scope narrowing |
| `InCluster` | auto clustering | Weakest; diversity/dedup at best |
## Design principle
Use the graph as a **structural reranker / recall-rescue layer**, NOT as the
primary retriever. Embeddings (+ future hybrid) generate candidates; the graph
re-scores and expands them. This is where graphs reliably help in RAG: relating,
deduping, and rescuing, not first-stage recall.
## Target live pipeline
```
Stage 1 Candidate generation (existing + future hybrid)
dense cosine (and later BM25 + RRF), generous top-N (~40)
Stage 2 Graph expansion (NEW, 1-hop only)
for each seed, pull neighbors via Supersedes / RelatesTo / DerivedFrom
score_neighbor = seed_score * edge_weight * depth_decay
this rescues relevant memories that embedding alone missed
Stage 3 Graph-aware dedup/canonicalize (NEW)
collapse Supersedes chains -> keep newest active only
group near-duplicate cluster members -> representative + count
Stage 4 Rerank + priors (ties into recall-5 / recall-6)
listwise rerank, then fold confidence / strength / recency / scope
apply calibrated cutoff
```
## Phased plan
### Phase A - Make supersede/contradiction authoritative in live recall (cheap, high value)
- In `score_and_filter` / `process_context`, post-filter results through the
graph: drop any memory whose `superseded_by` is set or that has an incoming
`Supersedes` edge from an active memory.
- Surface `Contradicts` pairs together with a conflict flag instead of letting
raw cosine arbitrarily pick one.
- Verifiable: unit test with a superseded chain; assert only newest surfaces.
### Phase B - Wire 1-hop graph expansion into the live path
- Add a `cascade=true` mode to the live retrieval (reuse `cascade_retrieve` but
cap depth=1 and restrict edges to Supersedes/RelatesTo/DerivedFrom; exclude
InCluster/HasTag fan-out which explode candidate count).
- Feed expanded set into the reranker, not directly to output.
- Verifiable (needs recall-1 harness): recall@5 with vs without expansion.
### Phase C - Graph-aware dedup before surfacing
- Collapse Supersedes/near-dup cluster members so the 5 surfaced slots aren't
wasted on restatements of one fact. Improves effective precision and recall.
### Phase D - Decide the fate of expensive maintenance
- Auto-clustering + sidecar cluster-naming + tag inference currently cost LLM
calls + full graph save per cycle and feed nothing into live recall.
- Options:
1. Repurpose clusters for Phase C dedup/diversity (keeps them, drops naming).
2. Cut cluster-naming + tag-inference entirely, redirect budget to embedder
upgrade + hybrid + rerank (recall-2/4/5).
- Recommended: cut naming + tag-inference now; keep cluster centroids only if
Phase C uses them. Keep confidence boost/decay, supersede, reinforcement.
### Phase E - Feedback loop closes via graph
- On inject + actual use, reinforce surfaced memories and strengthen the
RelatesTo edges among co-used memories (already partly there). Once Phase B
reads those edges, this feedback finally affects future recall.
## Cost to quantify first (before Phase D decision)
- Per maintenance cycle: # sidecar LLM calls (cluster naming), # graph load+save
round-trips, bytes rewritten. Add a counter / log and measure on the real
`~/.jcode/memory` graphs.
## Dependencies / ordering
- recall-1 (eval harness) gates B/C/D measurement.
- Phase A is independent and safe to do first (pure correctness win).
- Phases B/C should be measured against the harness; otherwise we're guessing.
```mermaid
graph LR
A[A: supersede authoritative] --> B[B: 1-hop expansion]
H[recall-1 harness] --> B
B --> C[C: graph dedup]
H --> D[D: trim/repurpose maintenance]
C --> E[E: feedback via edges]
```
## Implementation status (2026-06-14)
Benchmark (Mode 1, private ~/jcode-memory-bench, Sonnet judge):
- DONE: harness `memory_recall_bench` (queries/pool/judge/metrics), committed.
- Baseline: production dense (0.5 thr) = 0.0 recall@5; hybrid = 0.53.
Shipped to live path:
- DONE recall-0 + recall-4: memory agent uses `find_similar_hybrid`
(dense + BM25 + RRF, no cosine floor). Removed the recall-killing 0.5 threshold
and added lexical signal. Unit tests added. Bench: 0.0 -> 0.53 recall@5.
Evaluated, NOT shipped:
- recall-6 priors: roughly neutral (+1.8pt r@5 / -1.8pt r@10). Held back; bench
config `hybrid_priors` retained for re-evaluation after embedder upgrade.
Next (high value, larger change):
- recall-2: embedder upgrade (dense half is weak at 0.17 unthresholded).
- recall-3: focused query construction (window concatenates up to 12 msgs +
tool output; ~19% carry system-reminder boilerplate).
- recall-5: rerank stage. graph A-D: graph utilization.
## Update 2026-06-14 (rerank breakthrough, multi-agent)
Benchmark-driven results (Sonnet judge, 28 judged queries, jcode self-dev corpus):
| Config | recall@5 | recall@10 | precision@5 | MRR |
|--------------|----------|-----------|-------------|-------|
| baseline (prod dense, 0.5 thr) | 0.000 | 0.000 | 0.000 | 0.000 |
| hybrid (SHIPPED) | 0.530 | 0.679 | 0.229 | 0.504 |
| ce_rerank (local CE, rejected) | 0.325 | 0.420 | 0.129 | 0.322 |
| llm_rerank (listwise Sonnet) | 0.754 | 0.832 | 0.346 | 0.762 |
| oracle ceiling | 0.990 | 1.000 | 0.443 | 1.000 |
- Hybrid (dense+BM25+RRF) shipped: 0.0 -> 0.53 recall@5.
- Local cross-encoder REJECTED (out-of-distribution, 0.325).
- Listwise LLM reranker over the hybrid top-50 with a FOCUSED query: 0.53 -> 0.75
recall@5, captures most of the oracle headroom. This is the Mode-2 path.
- Embedder upgrade de-prioritized (pool recall already ~99%; bge anisotropic).
Implementation split (turtle + crocodile):
- Shared: jcode-base/src/memory_rerank.rs (prompt + parse + rerank_candidates),
used by both bench and memory_agent (single source of truth).
- memory_agent process_context: Mode-2 reranks hybrid candidates with the focused
query before surfacing; Mode-1 unchanged (no adequate local reranker).
- Focused query builder (focus_query_text) lands in memory_prompt.rs.
## Deferred follow-ups (2026-06-14, after the rerank pipeline shipped)
The two-stage pipeline (hybrid retrieve -> focused-query listwise LLM rerank ->
top-5) is live and committed; production recall@5 went 0.0 -> 0.53 -> ~0.75.
These remain as future work, each blocked or deliberately deprioritized:
1. **Remote embedding adapter** (low value, measure first). EmbeddingBackend
trait + LocalOnnxBackend scaffolding is shipped (embedding_backend.rs). A
remote openai/openai-compatible adapter + auto-select-on-embeddings-key +
re-embed migration would plug in via `active_backend()`. Deprioritized because
the oracle-ceiling analysis showed the embedder is a *capped* lever (the
candidate pool already contains ~99% of relevant memories; ranking, not
recall of the pool, was the bottleneck). Only revisit if a future change makes
the base embedder the bottleneck again, and A/B it in the bench first.
2. **Live reload + Mode-2 verification** (user action). Build+reload onto the new
binary and confirm the rerank fires in a real session (memory logs should show
the single listwise rerank instead of per-candidate sidecar checks). Pending
only because the shared worktree currently has an unrelated agent's
uncommitted changes; not a code issue.
3. **GPT-5.5 judge re-run** (blocked ~18 days). Re-run the bench LLM judge with
GPT-5.5 (`--backend=openai --reasoning=none`) once the OpenAI account quota
resets, and compare judge agreement against the current Claude-Sonnet gold
labels. Infrastructure is already in place (Sidecar::with_openai_model).
## Fork-the-judge / KV-reuse reranker (validated design, future)
Idea (user, 2026-06-14): instead of a separate tiny sidecar call for the memory
rerank, reuse the main agent's warm transcript KV cache and run the reranker as
a branch off it, so the judge's marginal cost is just the rerank suffix.
Benchmark findings (claude-sonnet-4-6, 28 judged queries, see
~/jcode-memory-bench/results/BASELINE_SUMMARY.md):
- Naive (full transcript as the rerank query): QUALITY REGRESSION. recall@5
0.81 -> 0.58, precision@5 0.34 -> 0.25. Noise dilutes even a frontier model.
- prefix_suffix (full transcript as prefix + focused intent appended as a
suffix with a "focus on THIS" marker): FULLY RECOVERS quality. recall@5 0.811,
precision@5 0.351, MRR 0.784 (>= the shipped focused-query rerank).
Conclusion:
- The cache-friendly structure (transcript-as-prefix for KV reuse) does NOT cost
accuracy *if* the focused rerank instruction is appended as a suffix.
- SELF-HOSTED (vLLM/SGLang/Ollama): viable + high-quality. Fork the rerank
sequence off the agent's warm transcript KV (SGLang fork / RadixAttention),
append the focused rerank suffix + candidate list, decode a short ranked list.
Near-free, full-model-quality reranking. Good basis for a local/premium memory
path. Requires a server that exposes prefix sharing/forking.
- PROVIDER APIs (default): NOT a cost win (cached-read on a ~50k-token transcript
prefix still costs ~10-20x a ~1k focused sidecar prompt, because the big
model's per-token rate dominates), but no longer a quality regression. Could be
exposed as an opt-in config "rerank with main model + prompt caching" for users
who prioritize rerank quality and have caching enabled. Default stays the cheap
focused-query sidecar, which wins on both cost and quality on the API path.
Bench repro: `memory_recall_bench metrics --config=llm_rerank
--query_view=focused|full|prefix_suffix --model=<model>`.
docs/MERMAID_RENDERING_REDESIGN.md
+173
View File
@@ -0,0 +1,173 @@
# ADR: Mermaid Rendering Redesign
Date: 2026-05-08
Status: Proposed
## Problem
The current Mermaid path is difficult to reason about because rendering, caching, UI placement, active diagram registration, deferred work, debug stats, and terminal image protocol state are coupled through global state and side effects.
Observed pain points:
- `jcode-tui-mermaid/src/lib.rs` is still a state hub despite the crate split.
- Markdown rendering decides Mermaid behavior directly, including streaming/deferred/side-only registration rules.
- Active diagrams are registered as a side effect of render calls, so simply preparing markdown mutates pinned-pane state.
- `with_preferred_aspect_ratio` uses thread-local state, so cache keys and render sizing depend on ambient context.
- The same diagram can be rendered in multiple contexts: chat inline placeholder, side panel image, pinned pane, streaming preview, debug probe. These contexts need different behavior but share low-level functions.
- Deferred rendering has its own dedupe/epoch/global queue and also performs active registration, increasing race risk.
- Image protocol rendering, PNG generation, image-state caches, and viewport rendering are mixed into the same public surface.
## Size API direction
The renderer now has an `mmdr-size-api` path guarded by the `mmdr-size-api` feature plus `JCODE_MMDR_SIZE_API_AVAILABLE=1`. That should become the primary path for the redesign:
- Renderer should ask Mermaid/layout for measured SVG/canvas dimensions instead of relying on source text complexity estimates for final PNG sizing.
- `calculate_render_size` should become a request target hint, not the source of truth for output dimensions.
- The fallback SVG-retargeting path should remain only as compatibility code until the patched renderer is always available.
- Debug stats should report `render_size_backend` and fail loudly in tests when the size API path is expected but unavailable.
- Cache keys should include normalized target/profile inputs, while artifacts should store measured output dimensions returned by the size API.
This reduces bugs from aspect-ratio retargeting, blurry upscaling, placeholder height mismatch, and pane resize oscillation.
## Target design
Use an explicit, staged pipeline with pure data between stages:
```mermaid
flowchart TD
A[Markdown/Event source] --> B[Diagram extraction]
B --> C[DiagramRegistry update]
C --> D[RenderScheduler]
D --> E[RenderCache]
E --> F[Renderer: Mermaid AST/Layout/SVG/PNG]
E --> G[Placement planner]
G --> H[Terminal image presenter]
C --> I[Pinned/side panel selectors]
```
### 1. Diagram extraction
Markdown renderers should only extract fenced Mermaid blocks into immutable descriptors:
```rust
struct DiagramBlock {
id: DiagramId,
source_hash: u64,
source: Arc<str>,
origin: DiagramOrigin,
ordinal: usize,
}
```
They should not directly mutate active diagrams or synchronously render unless a caller explicitly asks for a blocking fallback.
### 2. Explicit render request
Replace ambient `with_preferred_aspect_ratio` and boolean parameters with one request object:
```rust
struct RenderRequest {
diagram_id: DiagramId,
source_hash: u64,
source: Arc<str>,
target: RenderTarget,
profile: RenderProfile,
priority: RenderPriority,
mode: RenderMode,
}
struct RenderProfile {
width_cells: Option<u16>,
preferred_aspect_per_mille: Option<u16>,
theme: MermaidTheme,
}
enum RenderMode {
CacheOnly,
EnqueueIfMissing,
Blocking,
}
```
Cache keys should be built only from `source_hash + normalized RenderProfile`, never from thread-local context.
### 3. Registry owns active state
Introduce a `DiagramRegistry` owned by TUI app/session state, not a global Mermaid crate vector.
Responsibilities:
- Track diagrams visible in the current prepared transcript/side panel.
- Track streaming preview separately with a generation id.
- Publish the ordered list for pinned pane selection.
- Clear/update atomically per prepare pass.
Rendering should return `RenderArtifact`; it should never register active diagrams as a side effect.
### 4. Scheduler owns async/deferred behavior
A scheduler receives explicit requests and returns one of:
```rust
enum RenderStatus {
Ready(RenderArtifact),
Pending { request_id: RenderRequestId },
Failed(RenderError),
ProtocolUnavailable,
}
```
Rules:
- Deduplication is by full cache key.
- Workers do not mutate active registry.
- Worker completion only publishes `MermaidRenderCompleted` plus artifact metadata.
- Epoch invalidation is scoped to request generations, not one global counter unless truly necessary.
### 5. Placement planner is separate from rendering
Markdown/side-panel preparation should insert placeholders based on `RenderStatus` and desired placement:
- Inline image placeholder lines for chat/side panel.
- Sidebar marker for side-only mode.
- Error block for failed render.
- Pending placeholder for deferred/streaming render.
Image widget rendering should consume `RenderArtifact` plus `PlacementPlan`, not know Mermaid source or render scheduling.
### 6. Public module boundaries
Recommended crate modules:
- `model.rs`: `DiagramId`, `DiagramBlock`, `RenderProfile`, `RenderTarget`, `RenderArtifact`, `RenderStatus`, errors.
- `extract.rs`: Markdown Mermaid block extraction helpers.
- `cache.rs`: disk and memory artifact metadata cache.
- `renderer.rs`: Mermaid parse/layout/SVG/PNG conversion only.
- `scheduler.rs`: request queue, worker, dedupe, completion events.
- `registry.rs`: active/streaming diagram state, ideally app-owned.
- `placement.rs`: placeholder/image-region planning.
- `presenter.rs`: ratatui-image/Kitty/Sixel/iTerm viewport rendering.
- `debug.rs`: stats collected from explicit events.
## Migration plan
1. Add explicit model types and cache key normalization tests.
2. Add a new scheduler API while keeping old wrappers.
3. Convert `render_mermaid_sized_internal` into pure-ish `renderer::render_to_png(request) -> RenderArtifact`.
4. Move active diagram writes out of render functions into markdown prepare/app registry updates.
5. Replace `with_preferred_aspect_ratio` call sites with explicit `RenderProfile` plumbing.
6. Split presenter/image-state code from PNG rendering code.
7. Delete old boolean wrapper APIs and thread-local render profile.
## Validation criteria
- Unit tests for cache key normalization and filename parsing.
- Unit tests for registry update ordering, streaming preview replacement, and atomic clear/update.
- Scheduler tests for dedupe, cache-hit, cache-miss pending, worker completion, and no active-state mutation.
- Markdown renderer tests that Mermaid blocks produce deterministic placeholders without global side effects.
- Existing scroll/pinned-pane tests still pass.
- A debug probe can render a diagram with explicit profile and report the exact cache key used.
## Near-term safe refactor
Before a full migration, the highest ROI change is to introduce explicit request/status types and make old public functions thin compatibility wrappers. That lets us migrate call sites one at a time while reducing new bugs from additional boolean/thread-local behavior.
docs/MODULAR_ARCHITECTURE_RFC.md
+924
View File
@@ -0,0 +1,924 @@
# Modular Architecture RFC
Status: Draft
This RFC describes a modular target architecture for jcode that matches the current codebase, preserves the existing product model, and gives us a safe migration path from today's mostly-monolithic root crate to a layered workspace.
It is intentionally aligned with:
- [`REFACTORING.md`](./REFACTORING.md)
- [`COMPILE_PERFORMANCE_PLAN.md`](./COMPILE_PERFORMANCE_PLAN.md)
- [`SERVER_ARCHITECTURE.md`](./SERVER_ARCHITECTURE.md)
- [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md)
## Goals
- Document the architecture that exists today, not an idealized version.
- Define a target layered and crate architecture that improves maintainability and compile times.
- Establish dependency rules that prevent the workspace from collapsing back into a monolith.
- Provide a phased migration plan that fits the refactoring roadmap and compile-performance plan.
- Preserve runtime behavior: one shared server, reconnecting clients, session-local self-dev capability, and stable tool/provider flows.
## Non-Goals
- A big-bang rewrite.
- Renaming every module or crate immediately.
- Forcing every subsystem into a separate crate before its boundaries are ready.
- Changing the core product architecture from single-server, multi-client.
## Executive Summary
Today, jcode is best described as a **modular monolith with a growing workspace shell**:
- The root `jcode` crate still owns most runtime orchestration and product behavior.
- Several heavy or relatively self-contained subsystems have already moved into workspace crates.
- The codebase has strong module-level separation in some areas, but several broad root modules still act as architectural chokepoints.
The target architecture is a **layered workspace**:
1. **Foundation layer** for stable shared types and runtime primitives.
2. **Domain/runtime layer** for session, agent, provider, and server logic.
3. **Interface layer** for CLI, TUI, self-dev, and optional heavy integrations.
4. **Composition layer** where the top-level `jcode` package wires the product together.
The most important design rule is this:
> High-churn orchestration code must depend on stable lower layers, while stable lower layers must never depend back on runtime/UI/product-specific code.
That rule serves both architecture quality and compile-speed goals.
## Current Architecture
### Current runtime model
At the product level, the runtime architecture is already clear:
- `jcode` is a **single-server, multi-client** application.
- The server owns sessions, swarm state, background tasks, provider state, and shared services.
- Clients are primarily TUI frontends that attach to server-owned sessions.
- Self-dev is session-local capability on the shared server, not a separate architecture.
That model should stay intact.
### Current code organization
The current code organization is mixed:
- **Root crate `jcode`** still contains most product logic.
- **Workspace crates** already isolate several heavy or stable seams.
- **Subdirectories under `src/`** increasingly reflect domain boundaries, especially for `agent`, `cli`, `server`, `tool`, and `tui`.
Current workspace members from `Cargo.toml` are grouped roughly as follows:
- root package: `jcode`
- foundation/runtime support: `jcode-agent-runtime`, `jcode-core`, `jcode-storage`, `jcode-terminal-launch`, `jcode-tool-core`
- data-contract crates: `jcode-ambient-types`, `jcode-auth-types`, `jcode-background-types`, `jcode-batch-types`, `jcode-config-types`, `jcode-gateway-types`, `jcode-memory-types`, `jcode-message-types`, `jcode-selfdev-types`, `jcode-session-types`, `jcode-side-panel-types`, `jcode-task-types`, `jcode-tool-types`, `jcode-usage-types`
- protocol and planning: `jcode-protocol`, `jcode-plan`
- heavy or optional integrations: `jcode-embedding`, `jcode-pdf`, `jcode-notify-email`
- auth and providers: `jcode-azure-auth`, `jcode-provider-core`, `jcode-provider-metadata`, `jcode-provider-openrouter`, `jcode-provider-gemini`
- TUI extraction seams: `jcode-tui-core`, `jcode-tui-markdown`, `jcode-tui-mermaid`, `jcode-tui-render`, `jcode-tui-workspace`
- product surfaces outside the main TUI binary: `jcode-desktop`
### What the root crate still owns
The root crate still directly owns most of the following concerns:
- CLI parsing and dispatch
- server orchestration and socket lifecycle
- session state and persistence
- agent turn execution and tool orchestration
- provider implementation composition and runtime provider wiring; the shared `Provider` trait now lives in `jcode-provider-core`
- protocol/message/config types
- tool registry and many tool implementations
- TUI application state and rendering
- auth, memory, safety, ambient mode, and product glue
This is why the root crate is still the primary compile and architecture hotspot.
### Existing extracted workspace seams
These splits already exist and should be treated as real architectural footholds, not temporary accidents:
| Crate | Current role |
|---|---|
| `jcode-agent-runtime` | shared interrupt and lightweight runtime primitives for agent execution |
| `jcode-ambient-types` | usage and rate-limit records shared by ambient/background flows |
| `jcode-auth-types` | provider-neutral auth state and credential metadata |
| `jcode-background-types` | background-task status and progress DTOs |
| `jcode-batch-types` | batch tool progress DTOs, currently depending only on message types internally |
| `jcode-config-types` | stable configuration data contracts |
| `jcode-core` | low-level utilities such as IDs, env helpers, fs helpers, stdin detection, and formatting |
| `jcode-gateway-types` | gateway-facing data contracts |
| `jcode-memory-types` | memory subsystem data contracts |
| `jcode-message-types` | message content and transport-adjacent data contracts |
| `jcode-protocol` | client/server protocol surface built from stable type crates and provider-core values |
| `jcode-plan` | plan/task graph data model shared across coordination flows |
| `jcode-selfdev-types` | self-development request/status data contracts |
| `jcode-session-types` | session DTOs, currently depending only on message types internally |
| `jcode-side-panel-types` | side-panel page and update data contracts |
| `jcode-task-types` | task/tool scheduling data contracts |
| `jcode-tool-core` | runtime tool contracts such as the `Tool` trait and execution context |
| `jcode-tool-types` | stable tool output/image DTOs |
| `jcode-usage-types` | usage accounting data contracts |
| `jcode-storage` | storage helpers layered on `jcode-core` |
| `jcode-embedding` | ONNX/tokenizer-based embedding implementation and heavy inference deps |
| `jcode-pdf` | PDF text extraction |
| `jcode-azure-auth` | Azure bearer token retrieval |
| `jcode-notify-email` | SMTP/IMAP/mail transport |
| `jcode-provider-metadata` | provider/login catalog and profile metadata |
| `jcode-provider-core` | shared provider contract (`Provider`/`EventStream`), value types, route/cost/model helpers, shared HTTP client, schema helpers |
| `jcode-provider-openrouter` | OpenRouter-specific catalog/cache/support helpers |
| `jcode-provider-gemini` | Gemini schema/model/support helpers |
| `jcode-tui-core` | low-level terminal UI primitives that do not need full app state |
| `jcode-tui-markdown` | markdown wrapping/rendering, layered on mermaid/workspace support |
| `jcode-tui-mermaid` | mermaid parsing, rendering, caching, viewport, and widget support |
| `jcode-tui-render` | reusable TUI layout/render helpers |
| `jcode-tui-workspace` | workspace-map data/model/widget rendering |
| `jcode-terminal-launch` | terminal process launch helpers |
| `jcode-desktop` | desktop app surface and session/workspace rendering experiments |
These are already aligned with the compile-performance plan's strategy: isolate heavy dependencies and stable helper surfaces first.
### Current chokepoints
The root crate still has several broad, high-fanout modules that make both maintenance and incremental compilation harder. Current sizes observed from the tree:
- `src/server.rs`: ~1731 lines
- `src/provider/mod.rs`: ~2283 lines
- `src/session.rs`: ~2730 lines
- `src/protocol.rs`: ~1198 lines
- `src/main.rs`: ~55 lines
This supports the current plan direction:
- CLI decomposition is already mostly underway and should continue.
- Server, provider, session, and TUI state boundaries remain the most important structural work.
- The top-level binary entrypoint is already close to the desired thin composition shape.
### Current architecture in one picture
```mermaid
flowchart TD
J[jcode root crate]
J --> CLI[CLI and startup]
J --> Server[Server orchestration]
J --> Session[Session and persistence]
J --> Agent[Agent turn loop and tools]
J --> Provider[Provider trait and runtime impls]
J --> TUI[TUI app and rendering]
J --> Coreish[Protocol, message, config, ids]
J --> Product[Auth, memory, safety, ambient, notifications]
J --> AR[jcode-agent-runtime]
J --> Emb[jcode-embedding]
J --> PDF[jcode-pdf]
J --> Azure[jcode-azure-auth]
J --> Mail[jcode-notify-email]
J --> PMeta[jcode-provider-metadata]
J --> PCore[jcode-provider-core]
J --> POR[jcode-provider-openrouter]
J --> PGem[jcode-provider-gemini]
J --> TW[jcode-tui-workspace]
```
## Architectural Problems To Solve
### 1. The root crate is still the product and the platform
Today the root crate acts as all of the following at once:
- domain model holder
- runtime orchestrator
- UI host
- provider abstraction layer
- integration shell
- compile boundary for unrelated edits
That makes it hard to reason about ownership and easy to create accidental coupling.
### 2. Stable types and high-churn orchestration still live together
Broadly reused types like protocol structures, message forms, IDs, route metadata, and config types should be more stable than server, TUI, or provider orchestration logic. Today many of these still live in the same crate and sometimes in the same dependency fanout path.
### 3. Some boundary slices exist, but the center remains too wide
The existing workspace crates are good first splits, but they mostly isolate leaves. The center of gravity is still inside the root crate, especially around:
- session state
- provider runtime behavior and concrete provider composition
- server lifecycle
- tool registry wiring
- TUI app state and reducers
### 4. Compile-speed and architecture incentives are the same problem
The compile-performance plan is correct that crate boundaries matter most. The same boundaries that reduce invalidation pressure also improve ownership and testability.
## Target Architecture
### Layered model
The target is a layered workspace with a thin composition root. Arrows below mean
"depends on".
```mermaid
flowchart TD
App[jcode top-level package]
subgraph L2[Layer 2: interfaces and product surfaces]
TUI[jcode-tui]
SelfDev[jcode-selfdev]
CLI[jcode-cli or root CLI modules]
end
subgraph L1[Layer 1: domain/runtime]
Server[jcode-server]
Agent[jcode-agent]
Provider[jcode-provider]
Session[jcode-session]
end
subgraph L0[Layer 0: foundation and support]
Core[jcode-core]
AR[jcode-agent-runtime]
Emb[jcode-embedding]
PDF[jcode-pdf]
Azure[jcode-azure-auth]
Mail[jcode-notify-email]
PMeta[jcode-provider-metadata]
PCore[jcode-provider-core]
POR[jcode-provider-openrouter]
PGem[jcode-provider-gemini]
TW[jcode-tui-workspace]
end
App --> Server
App --> TUI
App --> SelfDev
App --> CLI
CLI --> Server
CLI --> TUI
CLI --> Core
TUI --> Core
TUI --> TW
SelfDev --> Server
SelfDev --> Core
Server --> Agent
Server --> Provider
Server --> Session
Server --> Core
Server --> Mail
Agent --> Provider
Agent --> Session
Agent --> Core
Agent --> AR
Provider --> Core
Provider --> PCore
Provider --> PMeta
Provider --> POR
Provider --> PGem
Provider --> Azure
Session --> Core
Session --> Emb
Session --> PDF
```
The exact crate names can evolve, but the dependency direction should not.
## Optimal compile-oriented workspace shape
The optimal crate structure is not "one crate per folder". The target should optimize for three forces at the same time:
1. **Invalidation boundaries:** high-churn edits should not rebuild unrelated stable subsystems.
2. **Dependency weight boundaries:** heavy dependencies should sit behind leaf crates or opt-in features.
3. **Ownership boundaries:** each crate should have one reason to change and a small public API.
The current root-crate size distribution makes the main opportunity clear: `src/tui`, `src/server`, `src/tool`, `src/provider`, `src/cli`, and `src/auth` dominate root-crate lines. Splitting only tiny helpers is useful as a safe staging tactic, but the long-term win is moving these high-churn domains behind stable lower-layer contracts.
### Desired final crate families
#### 1. Contract/type crates
These crates should be small, low-dependency, and slow-changing. They are allowed to be depended on broadly.
Existing examples:
- `jcode-message-types`
- `jcode-tool-types`
- `jcode-session-types`
- `jcode-config-types`
- `jcode-protocol`
- `jcode-provider-core`
- `jcode-plan`
- `jcode-*-types`
Target direction:
- Keep these crates boring and DTO-heavy.
- Prefer `serde`, `chrono`, and small utility dependencies only.
- Avoid `tokio`, `reqwest`, `ratatui`, provider SDKs, storage paths, and product orchestration.
- If a type requires a service handle, task runtime, channel sender, or filesystem layout, it is probably not a pure contract type.
Compile-time reason:
- These crates will be rebuilt whenever public contracts change, so they must change rarely.
- They allow `server`, `tui`, `agent`, and `provider` crates to talk without depending on the root crate.
#### 2. Domain/runtime crates
These own product behavior but should depend only downward on contracts/support crates.
Target crates:
- `jcode-provider`: provider composition, provider routing, streaming contract adapters, and concrete runtime implementations layered on the `jcode-provider-core` trait.
- `jcode-agent`: turn loop, compaction orchestration, provider/tool interaction, recovery logic.
- `jcode-session`: session model, state transitions, persistence-facing session operations.
- `jcode-server`: daemon lifecycle, client attachment, swarm/background coordination, service registries.
- `jcode-tools` or narrower `jcode-tool-core` plus `jcode-tool-impl`: tool registry contracts and tool implementations.
- `jcode-auth`: root auth orchestration after provider-neutral data lives in `jcode-auth-types` and heavy leaf SDKs stay separate.
- `jcode-memory`: memory graph/log/search orchestration once its contracts are stable enough.
Compile-time reason:
- These are the main root invalidation hotspots.
- They should become independent enough that an edit in TUI rendering does not rebuild provider implementations, and an edit in provider routing does not rebuild server socket lifecycle.
#### 3. Interface/product crates
These are high-churn application surfaces and should sit above runtime/domain crates.
Target crates:
- `jcode-cli`: parsing and command dispatch if CLI keeps growing.
- `jcode-tui`: app state, reducers, key handling, command/input handling, UI orchestration.
- `jcode-desktop`: already a separate surface.
- `jcode-selfdev`: self-dev build/reload/customization workflows if they remain a substantial product surface.
Compile-time reason:
- UI and CLI are edited frequently. Their churn should not force recompilation of stable server/provider/session internals.
- TUI should depend on protocol/service contracts, not on concrete server internals.
#### 4. Heavy leaf adapter crates
These should remain isolated and often feature-gated.
Existing examples:
- `jcode-embedding`
- `jcode-pdf`
- `jcode-azure-auth`
- `jcode-notify-email`
- `jcode-tui-mermaid`
- provider support crates such as `jcode-provider-openrouter` and `jcode-provider-gemini`
Target direction:
- Keep heavy dependencies out of the root crate and out of broadly shared contracts.
- Prefer opt-in features when the product can degrade gracefully.
- Keep a thin root/domain facade when runtime integration still belongs at a higher layer.
Compile-time reason:
- Heavy crates are fine when cached, but terrible when dragged into unrelated rebuilds.
- Feature-gated leaves make local inner loops cheaper without removing full-product builds.
#### 5. Composition package
The top-level `jcode` package should eventually become mostly:
- binary entrypoints
- feature defaults
- runtime graph assembly
- compatibility re-exports/facades during migration
- product configuration and packaging defaults
It should not be the long-term home of large implementation modules.
### Recommended dependency direction
A healthy final graph should look like this:
```text
jcode binary/composition
-> jcode-cli, jcode-tui, jcode-server, jcode-selfdev
jcode-cli / jcode-tui
-> jcode-protocol, jcode-*-types, jcode-server-client contracts
jcode-server
-> jcode-agent, jcode-session, jcode-provider, jcode-tools, jcode-storage
jcode-agent
-> jcode-provider, jcode-tools, jcode-session, jcode-agent-runtime
jcode-provider
-> jcode-provider-core, jcode-provider-* leaves, jcode-auth-types
jcode-session
-> jcode-session-types, jcode-message-types, jcode-storage, optional leaf adapters
contract/type crates
-> serde and small support crates only
```
The forbidden direction is just as important:
- contract crates must not depend on runtime/domain crates
- provider crates must not depend on TUI or server crates
- TUI crates must not depend on concrete server internals when protocol/client contracts are sufficient
- leaf adapter crates must not become backdoors into the root crate
- the root crate should not be required by workspace peers except temporarily during migration
### Split readiness checklist
A root module is ready to become a crate when most of these are true:
- Its public API can be described in less than a page.
- It does not need to call back into arbitrary root modules.
- Its dependencies are either lower-layer contracts or intentionally owned leaf adapters.
- Tests can run at the crate level without booting the full product.
- A touched-file benchmark shows it is on a meaningful invalidation path.
- It has a stable facade in the root crate for compatibility during migration.
If these are not true yet, keep decomposing internally first.
### What not to do
Avoid these tempting but harmful structures:
- **One mega `jcode-common` crate.** It becomes the new root crate and invalidates everything.
- **One crate per source directory.** This creates noisy APIs and dependency cycles without compile wins.
- **Moving high-churn traits too early.** A poorly stabilized trait crate can become worse than the monolith.
- **Moving UI-adjacent state into core.** This contaminates lower layers with `ratatui`/terminal concepts.
- **Provider leaf crates depending on root.** That prevents the root from ever becoming a composition shell.
- **Splitting by dependency weight only.** Heavy leaf isolation is good, but ownership and API stability matter too.
### Highest-ROI next crate seams from the current tree
Based on the current root size and existing footholds, the best next work is probably:
1. **Provider contracts:** keep shrinking `src/provider/mod.rs` until a `jcode-provider` trait/runtime crate can depend only on `jcode-message-types`, `jcode-provider-core`, and small runtime primitives.
2. **Server core:** extract protocol-independent pieces of `src/server/` such as client lifecycle state machines, swarm/background coordination DTOs, and reload/update policies behind server-local contracts.
3. **TUI reducer/state core:** extract non-rendering app state transitions from `src/tui/app/*` before moving the whole TUI crate.
4. **Tool contracts and registry shape:** separate tool definitions, schemas, execution context, and registry metadata from individual tool implementations.
5. **Session domain:** isolate session state transitions and persistence-facing operations from server/TUI/provider orchestration.
6. **Auth facade:** keep provider-neutral auth data in `jcode-auth-types`, heavy SDKs in leaf crates, and move root auth orchestration only after provider contracts stabilize.
A useful near-term policy: every time a large root file is touched, ask whether some pure table, DTO, parser, reducer, classifier, or state transition can move downward into an existing support crate without pulling runtime dependencies with it.
### Compile-time success metrics
Each structural phase should record at least:
- touched-file `cargo check` for the edited hotspot
- touched-file selfdev build for the edited hotspot
- `cargo tree -p jcode --edges normal --depth 1` before/after for dependency surprises
- crate-level test coverage for newly extracted crates
A split is successful if it either:
- lowers warm touched-file times for common edits, or
- prevents unrelated heavy crates from rebuilding when the root changes, or
- makes the next larger extraction materially safer.
A split should be reconsidered if it adds public API churn, creates cycles, or requires broad root re-exports that hide the actual dependency direction.
## Target crate responsibilities
### `jcode-core`
Purpose: stable shared types and utilities with minimal dependencies.
Should contain:
- IDs and naming primitives
- protocol DTOs that are not server-implementation-specific
- message/content/tool-definition types shared across runtime layers
- config primitives and enums that do not require runtime services
- small shared utility types with high reuse
Should not contain:
- TUI code
- server lifecycle code
- provider network code
- tokio task orchestration unless truly unavoidable
- product-specific wiring
Notes:
- This is the most important future extraction because it enables the rest.
- `src/protocol.rs`, `src/id.rs`, and carefully selected parts of `config.rs` and `message.rs` are the likely first feeders.
### `jcode-session`
Purpose: session domain model, persistence, and state transitions.
Should contain:
- session model and persisted metadata
- session storage/loading/snapshot logic
- reducer-like state transitions for session-owned data
- memory extraction hooks that are session-domain concerns
Should not contain:
- socket handling
- TUI state
- provider HTTP details
- direct server daemon lifecycle logic
Notes:
- This crate is not explicitly named in the current compile-performance plan, but the current size and fanout of `src/session.rs` make session extraction a natural stabilizing move.
- If introducing `jcode-session` feels too early, the same boundary should still be established internally first and extracted later.
### `jcode-provider`
Purpose: provider contracts and runtime-facing provider orchestration.
Should contain:
- the `Provider` trait once it depends only on lower-layer types
- provider routing abstractions
- runtime-facing provider composition
- shared streaming abstractions for provider results
Should not contain:
- provider-specific heavy catalogs and schema helpers that already live well in leaf crates
- server or TUI logic
Notes:
- Existing crates `jcode-provider-core`, `jcode-provider-metadata`, `jcode-provider-openrouter`, and `jcode-provider-gemini` remain useful under this layer.
- The key migration step is shrinking the `Provider` trait's dependency surface so it no longer depends on root-crate-only message/runtime types.
### `jcode-agent`
Purpose: agent turn engine and tool orchestration.
Should contain:
- turn-loop engine
- stream handling and response recovery
- tool execution orchestration
- compaction integration
- prompt assembly inputs that are agent-domain concerns
Should not contain:
- server socket lifecycle
- TUI state
- provider-specific leaf implementations
Notes:
- This aligns directly with the refactoring roadmap's "Agent Turn-Loop Unification" phase.
- `jcode-agent-runtime` remains the low-level runtime primitive crate below it.
### `jcode-server`
Purpose: daemon lifecycle and multi-client coordination.
Should contain:
- socket listeners and debug socket handling
- client attach/detach lifecycle
- swarm coordination
- reload/update server behaviors
- server-owned registries and shared service wiring
Should not contain:
- TUI rendering
- provider implementation details beyond service interfaces
- session persistence internals that belong in `jcode-session`
Notes:
- The current `src/server/` submodule tree is already the right shape for this extraction.
- `src/server.rs` should continue shrinking into a facade/composition module.
### `jcode-tui`
Purpose: client UI state, reducers, and rendering.
Should contain:
- app state and reducers
- remote client behavior and reconnect logic
- renderer/widget orchestration
- TUI-specific command/input handling
Should not contain:
- server daemon code
- session persistence internals
- provider network logic
Notes:
- This aligns directly with the refactoring roadmap's "TUI State/Reducer Split" phase.
- `jcode-tui-workspace` can remain a leaf crate or become a child dependency of `jcode-tui`.
### `jcode-selfdev`
Purpose: self-dev workflows, customization records, reload/build productization.
Should contain:
- self-dev state and tooling policy
- build/reload orchestration specific to self-dev workflows
- customization record and migration logic as it lands
Should not contain:
- generic server lifecycle not specific to self-dev
- general TUI rendering
Notes:
- This aligns with the compile-performance plan's issue-#32 direction and with the already-unified shared-server model.
### `jcode` top-level package
Purpose: composition root and shipping product package.
Should eventually be responsible for:
- binary entrypoints
- feature/default selection
- wiring the runtime graph together
- packaging and product defaults
It should not remain the long-term home of most implementation logic.
## Dependency Rules
These rules are the core of the RFC.
### Rule 1: Dependencies flow downward only
A higher layer may depend on a lower layer. A lower layer may not depend on a higher layer.
- foundation cannot depend on domain/runtime, interfaces, or product crates
- domain/runtime cannot depend on TUI or self-dev UI/product layers
- leaf adapters must not pull UI or server concerns downward
### Rule 2: No TUI types below the interface layer
- `ratatui`, `crossterm`, renderer state, viewport state, widget models, and clipboard/image/UI helper types must stay out of server, agent, provider, and core crates
- server-to-client data crosses the boundary via protocol/event types, not TUI structs
### Rule 3: No server daemon types in core or provider-support crates
- socket/session attachment state, fanout senders, debug socket helpers, and daemon lifecycle code must not appear in `jcode-core`, `jcode-provider-core`, or provider leaf crates
### Rule 4: Provider implementation crates depend on contracts, not on the server or TUI
- provider leaf crates may depend on `jcode-core`, `jcode-provider`, and `jcode-provider-core`
- they must not depend on `jcode-server` or `jcode-tui`
### Rule 5: Async/network-heavy dependencies do not belong in `jcode-core`
`jcode-core` should stay cheap to compile and highly reusable.
Avoid putting these there unless absolutely necessary:
- `reqwest`
- provider SDKs
- UI crates
- ONNX/tokenizer stacks
- mail/PDF dependencies
### Rule 6: Stable contracts should change more slowly than orchestration
Before extracting a crate, first shrink and stabilize its public surface.
Examples:
- move pure data types before moving stateful runtime code
- move pure helper functions before moving integration shells
- keep facades in the root crate during transitions if they reduce churn
### Rule 7: Avoid cross-cutting "utils" crates
Do not create a dumping-ground crate.
If code has a clear owner, it belongs with that owner:
- protocol/data types -> `jcode-core`
- session persistence -> `jcode-session`
- provider route/schema helpers -> provider crates
- rendering helpers -> `jcode-tui`
### Rule 8: The root package may compose many crates, but peer crates should stay narrow
The top-level `jcode` package can wire multiple domains together. Peer crates should not casually depend on each other sideways when a lower-level contract would do.
### Rule 9: New crate boundaries should follow both ownership and invalidation logic
A crate split is worth doing when it improves at least one of these substantially, and ideally both:
- clearer ownership and testability
- lower compile invalidation for common edits
### Rule 10: Preserve behavior with facades during migration
During migration, it is acceptable for the root crate to keep temporary facade modules that re-export or forward into extracted crates. That is preferable to risky behavior changes.
## Recommended Target Mapping From Today's Code
This is the recommended direction from the current tree, not a one-shot move list.
| Current area | Likely target |
|---|---|
| `src/id.rs`, protocol/message/config primitives | `jcode-core` |
| `src/session.rs`, parts of `storage`, restart snapshot concerns | `jcode-session` |
| `src/agent/*`, parts of `compaction`, tool orchestration seams | `jcode-agent` |
| `src/server/` + shrinking `src/server.rs` facade | `jcode-server` |
| `src/provider/mod.rs` trait/contracts plus provider composition seams | `jcode-provider` |
| existing provider helper crates | remain leaf/provider support crates |
| `src/tui/*` + `jcode-tui-workspace` | `jcode-tui` + leaf workspace widget crate |
| `src/cli/*` | stay in root initially or become `jcode-cli` later if justified |
| `src/tool/selfdev/*`, self-dev workflow/productization | `jcode-selfdev` |
## Phased Migration Plan
This migration is intentionally incremental and aligned with existing docs.
### Phase 0: Codify the architecture now
Deliverables:
- this RFC
- cross-links from refactoring and compile-performance docs
- dependency rules documented before more splits land
Why now:
- the repo already has enough workspace structure that undocumented drift is becoming more expensive
### Phase 1: Finish internal module decomposition in the root crate
Aligns with `REFACTORING.md` phases 2 through 6.
Focus areas:
- continue CLI decomposition until `main()` stays parse + runtime bootstrap only
- continue shrinking `src/server.rs` into a thin facade over `src/server/*`
- unify agent turn-loop variants behind one engine
- continue TUI state/reducer separation
- continue provider state isolation and pure helper extraction
Exit criteria:
- root modules are organized by ownership, not by convenience
- candidate extraction seams are obvious and lower-risk
### Phase 2: Extract `jcode-core`
This is the highest-leverage shared boundary.
First moves should be narrow and stable:
- IDs
- small protocol DTOs
- tool definition and message content forms that are broadly shared
- config enums/primitives that do not need runtime services
Avoid moving unstable orchestration APIs too early.
Exit criteria:
- server, agent, provider, and TUI code can all depend on the same lower-level shared types without depending on the root crate
### Phase 3: Extract runtime/domain crates
Primary targets:
1. `jcode-provider`
2. `jcode-agent`
3. `jcode-server`
4. `jcode-session`
Recommended order:
- start with whichever boundary is already most internally modular after Phase 1
- in practice, provider and server look like the strongest current candidates because they already have meaningful submodule trees and leaf support crates
- session may remain internal slightly longer if its public surface is still too entangled
Exit criteria:
- the root crate no longer defines the main provider, server, and agent contracts directly
### Phase 4: Extract `jcode-tui`
Focus:
- move client app/reducer/rendering code out of the root crate once protocol and runtime service boundaries are stable
- keep server events and client view-state concerns separated by protocol types
This phase should happen after enough shared contract extraction exists to avoid TUI depending back on root implementation details.
Exit criteria:
- TUI can evolve rapidly without dragging broad server/provider recompilation
### Phase 5: Extract `jcode-selfdev`
Focus:
- isolate self-dev workflow code and future customization/productization work
- keep shared-server runtime behavior intact
- move issue-#32 style no-rebuild customization logic here when it becomes concrete
Exit criteria:
- self-dev product behavior is explicit and no longer scattered across server/CLI/tool glue
### Phase 6: Shrink the root package into a composition shell
Desired end state:
- `src/main.rs` remains thin
- `jcode::run()` is mostly wiring
- the top-level package primarily assembles runtime services and default product configuration
### Continuous work across all phases
These should continue throughout the migration:
- keep carving heavy leaf dependencies into workspace crates where boundaries are safe
- measure touched-file compile timings after structural changes
- protect behavior with facades, tests, and refactor verification scripts
- prefer data-driven customization over source edits where issue #32 applies
## Migration Priorities
If we must prioritize, use this order:
1. stabilize and extract shared lower-level types
2. keep shrinking server/provider/session/agent hotspots internally
3. extract runtime contracts and orchestration crates
4. extract TUI
5. extract self-dev productization
This ordering gives the best overlap between architecture safety and compile-speed payoff.
## Acceptance Criteria
We should consider this RFC materially implemented when most of the following are true:
- the root package is primarily a composition shell
- shared cross-cutting types live in a lower-level crate rather than the root crate
- server, agent, provider, and TUI have clear ownership boundaries
- provider support crates no longer need root-crate-only types
- TUI depends on protocol/service contracts rather than runtime internals
- common self-dev edits avoid recompiling unrelated heavy subsystems whenever possible
- architecture docs match the actual crate graph
## Practical Guidance For Future Changes
When deciding where new code should go:
1. Ask who owns the behavior.
2. Ask which layers should be allowed to know about it.
3. Ask whether putting it in the root crate will increase invalidation for unrelated edits.
4. Prefer the narrowest stable owner that does not create an artificial abstraction.
Short version:
- if it is shared data, push downward
- if it is orchestration, keep it above stable contracts
- if it is UI, keep it out of runtime crates
- if it is heavy and isolated, make it a leaf crate
## Open Questions
These do not block the RFC, but they should be revisited as migration proceeds:
- Should `jcode-session` become an explicit crate, or remain an internal boundary until later?
- Should CLI remain in the top-level package permanently, or eventually become `jcode-cli`?
- Should `message` and `protocol` remain together in `jcode-core`, or split into separate contract crates if they evolve at different rates?
- Should `jcode-tui-workspace` remain a separate leaf crate long-term, or fold into `jcode-tui` once the larger TUI extraction lands?
## Recommendation
Adopt this RFC as the architectural north star for refactors and crate splits.
In practice that means:
- keep following the current refactoring roadmap
- keep using the compile-performance plan's measured, crate-boundary-first strategy
- treat every new extraction as part of one layered architecture, not as an isolated cleanup
docs/MULTI_SESSION_CLIENT_ARCHITECTURE.md
+617
View File
@@ -0,0 +1,617 @@
# Multi-Session Client Architecture (Proposed)
Status: Proposed
This document describes a proposed evolution of jcode's UI architecture from the
current **single-session-per-client** model to a **multi-session-capable client**
model with built-in session workspace management.
The goal is to support a built-in spatial/multi-session UI for users on all
platforms, while preserving the current external-window workflow used with tools
like Niri.
See also:
- [`SERVER_ARCHITECTURE.md`](./SERVER_ARCHITECTURE.md)
- [`SWARM_ARCHITECTURE.md`](./SWARM_ARCHITECTURE.md)
- [`WINDOWS.md`](./WINDOWS.md)
## Summary
Today, jcode is effectively organized like this:
- **Server** owns many sessions.
- **Each client** usually attaches to one session.
- **Each terminal window/process** usually hosts one client.
That gives a practical mapping of:
- `session ≈ client ≈ process`
The proposed architecture changes the client model to:
- **Server** still owns many sessions.
- **Many clients** may still exist at once.
- **Each client may host one or many session surfaces**.
That changes the mapping to:
- `session = server-owned runtime`
- `surface = client-side attachment/view of a session`
- `client = container for one or many surfaces`
This preserves independent windows while enabling a built-in multi-session
workspace.
## Goals
- Add a built-in multi-session workspace UI.
- Preserve the current independent-client workflow.
- Preserve interoperability with external window managers like Niri.
- Make macOS and other platforms first-class for spatial multi-session use.
- Avoid duplicating the entire TUI stack into separate "independent" and
"workspace" apps.
- Keep the server as the source of truth for sessions.
## Non-Goals
- Replacing OS-level window managers.
- Building a general-purpose terminal multiplexer for arbitrary applications.
- Requiring all users to adopt workspace mode.
- Supporting fully concurrent editing from multiple interactive attachments to the
same session in the first version.
## Current Architecture
Current high-level model:
```text
Server
├── Session A
├── Session B
└── Session C
Client 1 -> Session A
Client 2 -> Session B
Client 3 -> Session C
```
In practice, each client is typically its own terminal window/process, so users
who want a spatial layout today rely on an external window manager.
This works well on Linux with tools like Niri, but is not portable enough for a
cross-platform built-in workspace experience.
## Proposed Architecture
### Core idea
The server continues to own sessions, but the client evolves from a
single-session UI into a **multi-session shell**.
```text
Server
├── Session A
├── Session B
├── Session C
└── Session D
Client 1 (workspace)
├── Surface A -> Session A
├── Surface B -> Session B
└── Surface C -> Session C
Client 2 (independent)
└── Surface D -> Session D
```
A independent window becomes just a client hosting one surface. A workspace
becomes a client hosting many surfaces.
## Terminology
### Session
A server-owned runtime containing:
- conversation history
- provider/model state
- tool execution state
- session persistence
- background task state
- memory extraction state
A session is **not** fundamentally a window or process.
### Surface (or Attachment)
A client-side interactive or passive view of a session.
Examples:
- a session shown inside the built-in workspace
- a independent jcode window attached to one session
A surface is the UI representation of a session in a specific client.
### Client
A TUI process that hosts one or many surfaces.
Examples:
- current independent jcode window
- future multi-session workspace client
## Key Design Rule
The architecture must separate:
### Shared session state
Owned by the server:
- messages
- streaming/tool events
- model/provider selection
- persisted metadata
- background execution state
- server-side session lifecycle
### Surface-local UI state
Owned by a specific client surface:
- input draft
- cursor position
- scroll position
- selection/copy state
- local pane focus
- pane zoom/fullscreen state
- local viewport and layout placement
This separation is required to support:
- one session shown in different places over time
- popping a session out into a independent window
- docking a independent session back into a workspace
- different local view state for the same underlying session
## Client Modes
The same client binary should support two primary modes.
### Single-surface mode
Equivalent to today's independent client:
- one client
- one surface
- one session attached
This should remain the default/simple mental model for many users.
### Multi-surface mode
Workspace mode:
- one client
- many surfaces
- spatial navigation and session management built in
This mode provides the in-app session manager and workspace UI.
## Interoperability with External Window Managers
Preserving interop with Niri and similar tools is a core requirement.
The built-in workspace must not replace independent clients. Instead, both should
remain first-class.
### Required workflow support
- attach a session inside the in-app workspace
- pop a session out into its own independent client/window
- optionally dock a independent session back into a workspace
- allow multiple independent clients to coexist with a workspace client
### Resulting model
- many clients may exist at once
- each client may host one or many session surfaces
- the server still owns the underlying sessions
## Interaction Ownership
For an initial implementation, a session should have **one active interactive
surface** at a time.
That means:
- if a workspace surface is popped out into a independent window, the independent
surface becomes the active interactive owner
- the workspace surface should either disappear or become passive
- docking reverses that ownership
This avoids synchronization problems with:
- multiple input drafts
- racing submissions
- cursor/focus conflicts
- duplicate interactive ownership of the same session
A future design may allow richer mirroring or passive previews, but v1 should
prefer a single active controller per session.
## Niri-Style Workspace UX
The preferred first version is **not** a tiled multi-pane dashboard where many
sessions are all visible at once.
Instead, the built-in workspace should behave like a Niri-style spatial session
manager:
- the main viewport shows **one full-size session at a time**
- each session occupies a full-screen logical cell in the workspace
- moving left/right/up/down moves the **camera** through the workspace
- each workspace row behaves like a Niri horizontal strip of sessions
- moving up/down switches workspace rows and restores that row's remembered focus
- new sessions are inserted to the **right of the focused session** in the
current workspace row
Conceptually:
```text
workspace +1: [session C]
workspace 0: [session A] [session B]
workspace -1: [session D] [session E] [session F]
```
This is intentionally **not** a fixed matrix with fake empty cells.
## Workspace Map / Info Widget
The built-in info widget should act as a **workspace map**, not a text-heavy
status list.
### Role
The widget should let the user understand at a glance:
- which workspace row is current
- which session is focused in the current row
- what sessions exist to the left/right
- what sessions exist in nearby rows above/below
- which session was last focused in each non-current row
- which sessions are running, completed, errored, waiting, or detached
### Layout model
The widget should render a **vertical stack of horizontal strips**.
- each row represents one workspace
- each rectangle in a row represents one session
- only sessions that actually exist are shown
- non-current workspaces still remember their last-focused session
This preserves the Niri mental model much better than a synthetic grid.
### Visual language
The widget should be shape-first and text-light.
Each session is represented as a rectangle.
Suggested encoding:
- **idle** → dim outlined rectangle
- **focused** → bright or double-outlined rectangle
- **running** → animated rectangle border / spinner-like perimeter motion
- **completed** → green rectangle
- **waiting** → yellow rectangle
- **error** → red rectangle
- **detached** → distinct outline style (for example dashed or external marker)
The widget should avoid verbose labels inside the map itself. Session names and
full details belong in the main header/status area, not in the map.
### Example shape progression
One session:
```text
╔══════╗
╚══════╝
```
Add one to the right:
```text
┌──────┐ ╔══════╗
└──────┘ ╚══════╝
```
Move up and add one there:
```text
╔══════╗
╚══════╝
┌──────┐ ┌──────┐
└──────┘ └──────┘
```
The real TUI version should use color and animation rather than text markers.
## Client-Side Architecture
The current single `App` object is too monolithic to scale cleanly to many
sessions. The client should be split into layers.
### `ClientShell`
Global process/UI state:
- terminal event loop
- workspace layout
- camera/viewport position for workspace movement
- focus management
- keyboard mode (normal/insert/command)
- surface management
- pop-out / dock orchestration
- global commands and notifications
### `SessionController`
Per-session live controller:
- subscribe/resume session
- submit message
- cancel current turn
- apply model/session commands
- receive and apply server events
- reconnect logic
### `SessionSurfaceState`
Per-surface local UI state:
- input buffer
- cursor position
- scroll state
- selection/copy state
- side pane local viewport
- local focus and zoom state
### Shared session renderer
A reusable rendering layer that can render a session surface into an arbitrary
rect. This is the key step for making both independent and workspace modes reuse
one UI stack.
## Suggested Internal Model
```rust
struct ClientShell {
surfaces: Vec<SessionSurface>,
focused_surface: Option<SurfaceId>,
mode: ClientMode,
layout: LayoutState,
}
struct SessionSurface {
surface_id: SurfaceId,
session_id: SessionId,
controller: SessionController,
ui: SessionSurfaceState,
}
struct SessionController {
// v1: dedicated remote connection per surface
// v2: multiplexed session handle
}
struct SessionSurfaceState {
input: String,
cursor_pos: usize,
scroll_offset: usize,
side_pane_focus: bool,
zoomed: bool,
}
```
This enables:
- independent mode = one-surface client
- workspace mode = many-surface client
## Transport / Protocol Strategy
### Phase 1: dedicated connection per active surface
Fastest practical path:
- one client process
- one remote connection per live session surface
Pros:
- minimal protocol changes
- reuses the current session-oriented client behavior
- easiest way to prove out workspace UX
Cons:
- more overhead per hosted surface
- duplicate connection/reconnect machinery inside one process
- not the cleanest long-term abstraction
### Phase 2: multiplexed client protocol
Longer-term architecture:
- one client connection can subscribe to many sessions
- requests and events are explicitly tagged by `session_id`
Examples:
```rust
Request::SendMessage { session_id, ... }
Request::Cancel { session_id, ... }
ServerEvent::TextDelta { session_id, text }
ServerEvent::Done { session_id, ... }
```
Pros:
- cleaner workspace-native design
- lower connection overhead
- clearer event routing for multi-session clients
Cons:
- larger protocol and server refactor
Recommendation: do not block v1 on protocol multiplexing.
## Keybindings and Navigation
A good default workspace binding set is:
- `Alt+h/j/k/l` for workspace movement
- configurable remapping for users who already use those bindings in an external
WM (for example remapping to `Super+h/j/k/l`)
The client should support a modal split like:
- **normal mode** → workspace navigation and layout actions
- **insert mode** → focused session receives typed input
This avoids conflicts between text entry and spatial movement.
## Pop-Out / Dock Workflows
### Pop out to independent window
1. User selects a workspace surface.
2. Client spawns a independent jcode client attached to the same session.
3. Independent surface becomes the active interactive owner.
4. Workspace surface is removed or downgraded to passive.
### Dock into workspace
1. User requests dock for a independent session.
2. Workspace client creates a surface for that session.
3. Workspace surface becomes active interactive owner.
4. Independent client exits or detaches.
## Interop API Surface
The architecture should expose a small control surface for external and internal
interop.
Potential operations:
- `list_sessions`
- `list_surfaces`
- `workspace_state`
- `focus_session(session_id)`
- `open_session_in_window(session_id)`
- `dock_session(session_id)`
- `undock_session(session_id)`
- `move_session_to_workspace(session_id, position)`
This can initially be provided through existing jcode control channels such as:
- CLI commands
- the main server protocol
- debug/control socket
The exact public API shape is less important than preserving a clean internal
model for these operations.
## Recommended UI Direction
For a first version, prefer a **full-screen, camera-style workspace** over a
true many-pane dashboard.
Reasons:
- much closer to the Niri mental model
- keeps each session full-size and fully readable
- makes smooth movement between sessions more feasible in a terminal UI
- simplifies rendering because only the current session needs full live focus
- still allows richer overview modes later
This can later grow into optional resizeable session surfaces or richer
multi-visible workspace views, but the first version should optimize for a
smooth Niri-like experience.
## Migration Plan
### Phase 0: renderer extraction
- Extract a reusable session rendering layer from the current TUI.
- Stop assuming one `App` owns the entire terminal surface.
### Phase 1: surface/controller split
- Split current monolithic client state into shell/controller/surface layers.
- Keep single-surface behavior unchanged.
### Phase 2: workspace model + map widget
- Introduce a Niri-style workspace row model.
- Add the workspace-map info widget with rectangle-only state rendering.
- Track remembered focus per workspace row.
### Phase 3: full-screen camera navigation
- Allow one client process to host multiple session surfaces.
- Show one full-size session at a time.
- Move the viewport between neighboring sessions/workspaces.
### Phase 4: pop-out support
- Add commands to open a hosted session in a independent client.
- Preserve current `jcode --resume <session>` workflow.
### Phase 5: dock support
- Allow a independent session to be reattached into a workspace client.
- Keep one interactive owner per session.
### Phase 6: protocol cleanup
- Evaluate session-multiplexed protocol support.
- Replace dedicated per-surface connections if and when it is clearly beneficial.
## Open Questions
- Should passive mirrored surfaces exist in v1, or should a session exist in only
one visible place at a time?
- Which pieces of side-panel state are session-scoped vs surface-scoped?
- Should workspace mode be a new command (`jcode workspace`) or a runtime mode of
the normal client?
- How should dock/undock be exposed: command palette, slash commands, CLI, debug
socket, or all of the above?
- How much workspace layout state should be persisted across launches?
- How much offscreen session state should be pre-rendered for smooth animation?
## Recommendation
Adopt the following design direction:
1. **Expand the client to support multiple session surfaces.**
2. **Keep the server as the owner of sessions.**
3. **Preserve independent clients as first-class.**
4. **Treat workspace panes and independent windows as different surfaces for the
same session model.**
5. **Start with one active interactive surface per session.**
6. **Use a Niri-style full-screen workspace with a rectangle-only workspace map
widget as the primary UX.**
7. **Prototype with one connection per active surface before attempting protocol
multiplexing.**
This gives jcode a portable built-in multi-session workspace without sacrificing
existing workflows or external window-manager interop.
docs/ONBOARDING_SANDBOX.md
+184
View File
@@ -0,0 +1,184 @@
# Onboarding sandbox
If you want to iterate on onboarding repeatedly without touching your real auth state, use a separate sandbox rooted under `JCODE_HOME` and `JCODE_RUNTIME_DIR`.
This repo already supports that isolation:
- `JCODE_HOME` redirects jcode-owned state such as `~/.jcode` into a sandbox directory.
- `JCODE_HOME` also redirects app config into `JCODE_HOME/config/jcode`.
- `JCODE_RUNTIME_DIR` redirects sockets and other ephemeral runtime files.
- External auth trust decisions are stored in the sandbox config, so a fresh sandbox starts with no trusted external auth imports.
## Fast start
```bash
scripts/onboarding_sandbox.sh fresh
```
That gives you a clean jcode launch with isolated state.
## Test with your REAL logins (import them in the sandbox)
A clean sandbox is fully isolated, so the onboarding "import existing logins"
step has nothing to import. To exercise the import + "continue where you left
off" steps against your actual accounts, seed copies of your real credential and
transcript files into the sandbox:
```bash
# Copy real external logins (Codex/Claude/Gemini/Copilot/Cursor/OpenCode/pi)
scripts/onboarding_sandbox.sh seed-real-logins
# Also copy your real Codex/Claude transcripts so the "continue a session"
# step has real history to resume:
scripts/onboarding_sandbox.sh seed-real-logins --with-transcripts
# Or do it all in one shot: reset, seed, and launch jcode
scripts/onboarding_sandbox.sh fresh-real --with-transcripts
```
How it works: when `JCODE_HOME` is set, jcode resolves every external credential
and transcript lookup to `$JCODE_HOME/external/<same-relative-path-as-$HOME>`.
`seed-real-logins` copies your real files there, so detection and import behave
exactly as they would on a first-run machine that already has those tools
installed. The copies are real tokens, so the sandbox stays local-only; your
original `$HOME` files are never moved, rewritten, or deleted.
Once seeded, just launch the sandbox and walk onboarding; it will detect and
offer to import each real login:
```bash
scripts/onboarding_sandbox.sh jcode
```
## Common commands
```bash
# Show the exact env vars and sandbox paths
scripts/onboarding_sandbox.sh env
scripts/onboarding_sandbox.sh status
# Start over from a blank onboarding state
scripts/onboarding_sandbox.sh reset
scripts/onboarding_sandbox.sh fresh
# Log into a provider without touching your normal jcode config
scripts/onboarding_sandbox.sh login openai
scripts/onboarding_sandbox.sh login claude
scripts/onboarding_sandbox.sh auth-status
# Save the resulting logged-in sandbox as a reusable local fixture
scripts/onboarding_sandbox.sh fixture-save normal-openai
# Later, restore that exact auth state without repeating browser login
scripts/onboarding_sandbox.sh fixture-load normal-openai
scripts/onboarding_sandbox.sh auth-status
# Or load and run one command in the fixture-backed sandbox
scripts/onboarding_sandbox.sh fixture-run normal-openai -- auth-test --provider openai --no-smoke
# Run arbitrary jcode commands in the sandbox
scripts/onboarding_sandbox.sh jcode auth status
scripts/onboarding_sandbox.sh jcode pair
```
## Reusable local auth fixtures
For repeated login testing, use local auth fixtures. A fixture is a copy of a
sandbox `JCODE_HOME` after you have put it into an interesting state, for
example a typical logged-in OpenAI user, an expired token state, or an external
auth import approval state.
The fixture store defaults to `.tmp/auth-fixtures`, which is intentionally local
developer state. Fixtures may contain real OAuth tokens or API-key references, so
do not commit or share them.
Recommended workflow:
```bash
# One-time setup for a realistic logged-in state
scripts/onboarding_sandbox.sh reset
scripts/onboarding_sandbox.sh login openai
scripts/onboarding_sandbox.sh auth-status
scripts/onboarding_sandbox.sh fixture-save normal-openai
# Fast repeat loop after that
scripts/onboarding_sandbox.sh fixture-load normal-openai
scripts/onboarding_sandbox.sh auth-status
scripts/onboarding_sandbox.sh jcode auth-test --provider openai
```
The lower-level helper can also be used directly:
```bash
scripts/auth_fixture.sh list
scripts/auth_fixture.sh save normal-openai
scripts/auth_fixture.sh load normal-openai
scripts/auth_fixture.sh run normal-openai -- auth status
```
Useful environment overrides:
- `JCODE_ONBOARDING_SANDBOX`: select which sandbox receives the fixture.
- `JCODE_ONBOARDING_DIR`: use an explicit sandbox directory.
- `JCODE_AUTH_FIXTURE_DIR`: use a fixture store outside the repo, for example
`~/.local/share/jcode-auth-fixtures`.
Suggested fixture names:
- `normal-openai`
- `normal-claude`
- `expired-openai`
- `api-key-openrouter`
- `external-opencode-approved`
## Mobile onboarding simulator
The repo also has a resettable headless mobile simulator with predefined onboarding scenarios.
```bash
# Start the simulator in the background
scripts/onboarding_sandbox.sh mobile-start onboarding
# Inspect it
scripts/onboarding_sandbox.sh mobile-status
scripts/onboarding_sandbox.sh mobile-state
scripts/onboarding_sandbox.sh mobile-log
# Reset it back to the scenario start
scripts/onboarding_sandbox.sh mobile-reset
```
Supported scenarios today:
- `onboarding`
- `pairing_ready`
- `connected_chat`
## Why this is safer
A fresh sandbox means:
- no real jcode config files are reused
- no real runtime sockets are reused
- no previously trusted external auth sources are reused
- you can blow it away with one `reset`
When using fixtures, the sandbox is still isolated from your normal jcode state,
but the loaded fixture may intentionally contain copied auth state from an earlier
sandbox login.
## Recommended workflow
For tight onboarding iteration, use this loop:
1. `scripts/onboarding_sandbox.sh reset`
2. `scripts/onboarding_sandbox.sh fresh`
3. walk the onboarding flow
4. adjust code
5. repeat
If you are iterating specifically on mobile onboarding UX, keep the simulator running and use `mobile-reset` between passes.
## Caveat
This sandbox is designed to isolate jcode-owned state and trusted external-import state. If you later decide to test explicit import/reuse flows from external tools, do that intentionally and treat it as a separate test case from first-run onboarding.
docs/OPENAI_COMPATIBLE_PROFILE_RUNTIME_PLAN.md
+119
View File
@@ -0,0 +1,119 @@
# OpenAI-Compatible Profile Runtime Migration Plan
## Problem
`OpenRouterProvider` currently represents two distinct concepts:
1. Standard OpenRouter, with OpenRouter-specific routing, provider pinning, endpoint metadata, and an `openrouter` catalog namespace.
2. Direct OpenAI-compatible providers such as NVIDIA NIM, Groq, Cerebras, Chutes, and custom endpoints, which reuse the same HTTP transport but have distinct credentials, API bases, catalogs, and model IDs.
Because `MultiProvider` stores only one `openrouter` runtime slot, switching from standard OpenRouter to a direct profile replaces the active runtime/catalog view. This caused issue #274: after switching from `openrouter/owl-alpha` to NVIDIA NIM, `/model` no longer exposed standard OpenRouter and could mis-associate OpenRouter models with NVIDIA.
## Target architecture
Separate transport, profile identity, and route aggregation.
```rust
struct OpenAiCompatibleClient {
api_base: String,
api_key_env: String,
env_file: String,
auth_header: AuthHeaderConfig,
}
struct OpenAiCompatibleProfileRuntime {
profile_id: String, // "openrouter", "nvidia-nim", "groq", ...
display_name: String, // "OpenRouter", "NVIDIA NIM", ...
cache_namespace: String, // usually profile_id
default_model: Option<String>,
provider_routing: bool, // true for standard OpenRouter features
client: OpenAiCompatibleClient,
}
```
`MultiProvider` should eventually move from:
```rust
openrouter: RwLock<Option<Arc<openrouter::OpenRouterProvider>>>,
```
to something like:
```rust
openai_compatible: RwLock<BTreeMap<String, Arc<OpenAiCompatibleProfileRuntime>>>,
active_openai_compatible_profile: RwLock<Option<String>>,
```
Standard OpenRouter becomes one profile in this map, not the container for every compatible provider.
## Route aggregation rule
`/model` should aggregate routes from every configured profile:
```rust
for profile in configured_openai_compatible_profiles() {
routes.extend(profile.model_routes());
}
```
Switching active runtime to NVIDIA NIM should only update active selection:
```rust
active_openai_compatible_profile = Some("nvidia-nim".into());
```
It should not remove or relabel `openai_compatible["openrouter"]`.
## Compatibility requirements
Keep existing user-facing forms working:
- `openrouter:<model>` targets standard OpenRouter.
- `nvidia-nim:<model>` targets NVIDIA NIM.
- `openai-compatible:<model>` targets the configured custom endpoint.
- `--provider openrouter` remains standard OpenRouter.
- `--provider openai-compatible` remains the generic/custom profile.
- Existing `OpenRouterProvider` type can remain as a compatibility wrapper while internals move.
## Incremental migration slices
1. **Route aggregation slice, completed in `b1272ae`**
- Standard OpenRouter cached routes are scoped to the `openrouter` namespace.
- Direct profiles can be active without hiding standard OpenRouter from `/model`.
- Regression: OpenRouter `owl-alpha` -> NVIDIA NIM -> `/model` keeps OpenRouter route and does not relabel it as NVIDIA.
2. **Profile runtime struct**
- Introduce `OpenAiCompatibleProfileRuntime` around current OpenRouter provider settings.
- Keep `OpenRouterProvider` as a type alias/wrapper initially.
3. **Runtime registry**
- Add a map of configured compatible profiles to `MultiProvider`.
- Populate it from configured/saved credentials at startup and auth-change time.
4. **Active profile selection**
- Replace implicit environment mutation as the only active-profile state with explicit profile IDs.
- Use env application only as a compatibility/bootstrap layer.
5. **Picker and server snapshots**
- Emit profile-scoped routes and available-model snapshots.
- Include profile ID/api method in debug output so mislabeling is testable.
6. **Rename cleanup**
- Rename generic internals from OpenRouter to OpenAI-compatible where accurate.
- Keep public commands and config stable.
## Validation matrix
For each configured profile pair, verify:
- Active profile A, inactive profile B: `/model` shows both A and B routes.
- Selecting a B route switches to B and keeps A visible.
- Models with slash IDs are not automatically treated as standard OpenRouter unless the route/profile says so.
- OpenRouter provider-pinning remains available only for the standard OpenRouter profile.
- Direct-profile static and live catalogs remain namespace-scoped.
Key regression scenarios:
- `openrouter/owl-alpha` -> `nvidia-nim:nvidia/llama-...` -> OpenRouter still selectable.
- Cerebras active with Groq configured -> no relabeling of Cerebras models as Groq.
- Chutes active with stale legacy OpenRouter cache -> no stale OpenRouter models under Chutes.
docs/PROVIDER_DOCTOR.md
+166
View File
@@ -0,0 +1,166 @@
# Provider Doctor
`jcode provider-doctor` is a user-facing diagnostic that answers one question:
> Why isn't my provider/model (or the model picker) working?
It walks the same strict end-to-end checkpoints that the live coverage ledger
tracks (`jcode provider-test-coverage`), but as an interactive command you can run
yourself, with clear pass/fail output and a "what to try next" hint on the first
failure.
It works with **OpenAI-compatible providers** (cerebras, fpt, nvidia-nim,
comtegra, deepseek, groq, openrouter, and other `openai-compatible` profiles).
## Quick start
```bash
# Validate jcode's own wiring for a provider, no API key, no spend:
jcode provider-doctor cerebras --tier offline
# Validate the key + live model catalog (needs a key, negligible spend):
jcode provider-doctor cerebras --tier catalog
# Full readiness, including real chat, streaming, and tool calls (spends balance):
jcode provider-doctor cerebras --tier full
# Pin a specific model and emit JSON for scripting/CI:
jcode provider-doctor cerebras --model gpt-oss-120b --tier full --json
```
The model defaults to the provider's default model (or the first live catalog
model). Use the global `--model` flag to pin a specific one.
## Tiers
Pick how much to exercise. Each tier validates as much as is possible given its
constraints, so you can debug cheaply and escalate only when needed.
| Tier | Needs key? | Spends balance? | What it adds | Catches |
| --- | --- | --- | --- | --- |
| `offline` | no | no | jcode-side wiring against a synthetic catalog | catalog reload, picker rendering, fallback labeling, and model-switch routing bugs for this provider |
| `catalog` (default) | yes | ~none | live `GET /models` | bad/missing key, dead endpoint, model not in the live catalog |
| `full` | yes | yes | non-streaming chat, streaming, tool-call loop | the model actually chats, streams, and supports tool-calling |
Only the `full` tier can earn strict ("READY") coverage. The lighter tiers
intentionally record the API-dependent checkpoints as skipped, so nothing is
over-credited in the coverage ledger.
## Checkpoints
Every run reports these strict checkpoints in order. A pair is fully ready only
when all of them pass on the `full` tier.
1. `auth_credential_loaded` - a credential was found for the provider
2. `model_catalog_live_endpoint` - the live `/models` endpoint returned models
3. `catalog_hot_reload_current_session` - the catalog reloaded into the session
4. `picker_live_models` - the picker shows the live models, including the selected one
5. `picker_fallback_labeling` - routes are live-catalog backed, not static fallback
6. `model_switch_route` - switching models produces a provider-explicit route
7. `non_streaming_chat_completion` - a basic chat reply came back (full tier)
8. `streaming_chat_completion` - a streamed reply came back (full tier)
9. `tool_call_parse` - the model emitted a parseable tool call (full tier)
10. `tool_execution_loop` - the tool-call loop ran (full tier)
11. `tool_result_followup` - the tool result was fed back (full tier)
12. `real_jcode_tool_smoke` - an end-to-end tool smoke passed (full tier)
(Checkpoints 1-2 plus the auth-lifecycle stages are pre-flight; 7-12 are the
API-dependent ones gated behind `--tier full`.)
## Reading the output
```
Provider doctor: Cerebras / gpt-oss-120b
Tier: catalog (API key, ~no spend: adds live catalog fetch)
...
[ PASS] Credential loaded Loaded credential from CEREBRAS_API_KEY
[ PASS] Live model catalog endpoint 2 live model(s) returned
[ PASS] Catalog hot reload in current session 2 catalog route(s) reloaded
[ PASS] Picker shows live models 2 model(s) in picker, selected `gpt-oss-120b`
[ PASS] Picker fallback labeling all routes backed by live catalog (no static fallback)
[ PASS] Model switch route switch request `cerebras:...` routed via `openai-compatible:cerebras`
[ skip] Non-streaming chat completion catalog tier: requires --tier full (spends balance)
...
Verdict: tier `catalog` passed. Run `--tier full` to confirm full readiness (spends balance).
```
- `PASS` / `FAIL` - the checkpoint ran and passed/failed.
- `skip` - the current tier does not run this checkpoint (use `--tier full`).
- The verdict line tells you whether the tier passed, fully passed (`READY`), or
failed, and on failure points at the first failing checkpoint with a next step.
The command exits non-zero when the chosen tier did not fully pass, so it can be
used as a CI/scripting gate.
## Spend tracking (how much does a run cost?)
Balance-spending tiers (`catalog` makes a catalog call, `full` makes several
chat/stream/tool calls) report exactly what they consumed so you can budget:
```
Spend this run: 3 billable API calls, 554 tokens (289 in + 265 out), cost not reported by provider
```
- **billable API calls** - how many requests actually hit the provider.
- **tokens** - prompt + completion totals summed across those calls, when the
provider returns a `usage` block. Streaming probes request
`stream_options.include_usage` so streamed calls are counted too.
- **cost** - shown as a USD figure only when the provider reports a `cost`
field; many providers (e.g. cerebras) only return tokens, so you'll see
"cost not reported by provider" and can multiply tokens by your plan's rate.
A full cerebras run is roughly 550-620 tokens (about $0.0003).
`--json` includes the same data under a `spend` object
(`billable_calls`, `prompt_tokens`, `completion_tokens`, `total_tokens`,
`has_token_data`, `reported_cost_usd`).
This spend is **persisted** into the coverage ledger alongside the run, so
`jcode provider-test-coverage` shows a cumulative "Recorded spend" footer
summing the latest run per pair. That gives you a durable, at-a-glance answer to
"how much has exercising this coverage cost me so far?"
## Typical debugging flow
1. **"My picker is broken / shows the wrong models."**
Run `--tier offline`. If `picker_live_models`, `picker_fallback_labeling`, or
`model_switch_route` fail, it's a jcode-side routing bug for that provider:
capture the output and file an issue.
2. **"It won't connect / says auth failed."**
Run `--tier catalog`. If `auth_credential_loaded` or
`model_catalog_live_endpoint` fail, the key/endpoint is the problem. Run
`jcode login --provider <provider>`.
3. **"It connects but the model behaves badly."**
Run `--tier full`. If `non_streaming_chat_completion` /
`streaming_chat_completion` / the `tool_*` checkpoints fail, the model itself
is the issue; try another model from the live catalog.
## Relationship to coverage
Every doctor run records a live-verification event into the coverage ledger,
tagged with the tier (`doctor_tier`). A `full`-tier pass that clears all 11
strict checkpoints flips the pair to strict ("READY") in
`jcode provider-test-coverage`. Lighter tiers record the API-dependent
checkpoints as skipped, so they never over-credit a pair.
`jcode provider-test-coverage` renders the same 11 checkpoints as an 11-stage
pipeline. Each observed pair gets one compact line: a status token (`READY`, or
`N/11` = how many stages it cleared) followed by `provider / model`, and then,
for any pair that is not yet READY, the first blocker plus the exact
`provider-doctor` command to push it past that blocker. So the two commands are
two views of one pipeline: the coverage report shows where every pair is stuck
and hands you the doctor command to advance it.
Each line ends with a freshness note, e.g.:
```
READY cerebras / gpt-oss-120b last tested 9 minutes ago (2026-05-30) by developer (dev build)
6/11 nvidia-nim / gemma-4-31b failed at `streaming reply`; run `jcode provider-doctor nvidia-nim --model gemma-4-31b --tier full`; last tested 2 days ago ...
```
- **how long ago** the most recent run was, in plain English plus the absolute
date, so you can tell at a glance whether the evidence is stale.
- **who ran it**: a clean release build is labeled `user (release build)` (real
user evidence), a dirty/dev build is `developer (dev build)`. This is derived
durably from the build flag recorded with each run, not guessed.
docs/PROVIDER_SESSION_SHARED_CONTRACT_AUDIT.md
+371
View File
@@ -0,0 +1,371 @@
# Provider, Session, and Shared-Contract Boundary Audit
Status: 2026-04-16 audit note
This document audits the current provider, session, and shared-contract seams in the jcode workspace and recommends the next **realistic** crate moves that improve modularity without creating high-churn dependency cycles.
It is intentionally conservative. The goal is to identify boundaries that are both:
- structurally useful
- low enough churn to be worth turning into workspace crates now
See also:
- [`COMPILE_PERFORMANCE_PLAN.md`](./COMPILE_PERFORMANCE_PLAN.md)
- [`REFACTORING.md`](./REFACTORING.md)
- [`SERVER_ARCHITECTURE.md`](./SERVER_ARCHITECTURE.md)
- [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md)
## Executive summary
The next clean workspace moves are **not** a full `Provider` trait extraction and **not** a full `session.rs` split.
The best next steps are:
1. **Add a small `jcode-shared-contracts` crate** for the serde-only protocol/session overlap types that already act like shared contracts.
2. **After that, add a narrow `jcode-session-contracts` crate** for session metadata/replay/view structs that are widely reused but do not need the full `Session` runtime.
3. **If we want one more provider-side move before a larger provider refactor, extract the pure provider identity/selection layer** into `jcode-provider-core` or a small `jcode-provider-selection` crate.
The main things to avoid for now:
- extracting `Provider` / `EventStream` into a shared crate
- extracting all of `protocol.rs`
- extracting all of `session.rs`
- moving `provider_catalog.rs` wholesale into a crate
Those look tempting, but today they would mostly convert existing high-churn coupling into workspace-crate churn.
## Current workspace boundary state
Already landed and directionally good:
- `crates/jcode-provider-metadata`
- `crates/jcode-provider-core`
- `crates/jcode-provider-openrouter`
- `crates/jcode-provider-gemini`
A useful property of the current extracted crates is that they are still **leaf-like support crates**.
Current local workspace dependency picture for those crates:
- `jcode-provider-core`: no local workspace deps
- `jcode-provider-metadata`: no local workspace deps
- `jcode-provider-openrouter`: no local workspace deps
- `jcode-provider-gemini`: no local workspace deps
That is the right pattern to preserve. The next crate moves should keep producing small, leaf-ish crates instead of creating new central hubs that everything recompiles through.
## Hotspots and coupling observed
Relevant file sizes in the main crate:
- `src/session.rs`: 2730 lines
- `src/provider/mod.rs`: 2283 lines
- `src/protocol.rs`: 1198 lines
- `src/provider/openrouter.rs`: 1132 lines
- `src/provider/gemini.rs`: 1117 lines
- `src/provider_catalog.rs`: 775 lines
- `src/plan.rs`: 17 lines
High-level coupling observed during the audit:
- `src/provider/mod.rs` directly references `auth`, `logging`, `bus`, `message`, and `usage`
- `src/session.rs` directly references `message`, `protocol`, `plan`, `storage`, and support modules
- `src/protocol.rs` directly references `bus`, `config`, `message`, `plan`, `provider`, `session`, and `side_panel`
- `src/provider_catalog.rs` is especially tied to `env`, `storage`, and `logging`
That means the biggest blockers are not the already-extracted support crates. They are the remaining mixed runtime/facade modules in the main crate.
## Dependency shape
```mermaid
flowchart LR
P[provider/mod.rs] --> AUTH[auth]
P --> MSG[message]
P --> BUS[bus]
P --> USAGE[usage]
S[session.rs] --> MSG
S --> PROTO[protocol.rs]
S --> PLAN[plan.rs]
S --> STORE[storage]
PROTO --> BUS
PROTO --> CFG[config]
PROTO --> MSG
PROTO --> PLAN
PROTO --> PROVIDER_TYPES[provider types]
PROTO --> SESSION_TYPES[session types]
```
The key architectural smell is that some types that are effectively **shared contracts** still live inside large mixed-responsibility modules.
## Provider boundary audit
### What is already in a good state
The existing provider crate moves were well chosen:
- `jcode-provider-metadata` holds stable login/profile catalog data
- `jcode-provider-core` holds route/cost/shared HTTP client/core value types
- `jcode-provider-openrouter` holds OpenRouter-specific catalog/cache/ranking/model-spec support
- `jcode-provider-gemini` holds Gemini Code Assist schema/types/support helpers
These are all relatively pure support surfaces.
### What is not a good next move yet
### Do not extract `Provider` / `EventStream` yet
`src/provider/mod.rs` is still deeply entangled with:
- `crate::message::{Message, StreamEvent, ToolDefinition}`
- auth-driven behavior
- runtime selection/failover
- logging and bus notifications
- provider-specific compaction and transport behavior
Moving the trait now would likely create a new shared crate that still changes whenever runtime/provider behavior changes.
That would improve directory layout, but not boundary quality.
### Do not move `provider_catalog.rs` wholesale yet
`src/provider_catalog.rs` is not just metadata. It currently mixes:
- catalog/profile values
- env mutation
- auth probing helpers
- config-file lookup
- logging/warnings
That facade is still too runtime-aware to become a clean leaf crate as-is.
### Best realistic provider move
### Option A: extract provider identity + pure selection
Most realistic provider-side move after the current support crates:
- move the provider identity enum currently represented by `ActiveProvider`
- move `src/provider/selection.rs`
- optionally move pure fallback ordering helpers that do not depend on auth/runtime state
Target:
- either a new `crates/jcode-provider-selection`
- or a small `provider_identity` / `selection` module inside `jcode-provider-core`
Why this is realistic:
- `selection.rs` is already pure logic
- it does not need `Message`, `EventStream`, auth state, or storage
- it would shave some policy code out of `src/provider/mod.rs`
- it creates a stable place for provider-order and provider-name normalization rules
Why this should stay narrow:
- once the code starts touching account failover, auth checks, runtime availability, or logging, it stops being a good crate boundary
## Session boundary audit
### Why `session.rs` should not be extracted wholesale yet
`src/session.rs` is large, but it is not one thing.
It currently mixes:
- persisted session data structures
- runtime session state
- journaling / file persistence helpers
- replay-event persistence
- startup/remote snapshot helpers
- image rendering helpers
A whole-file crate extraction would drag in more coupling than it removes.
Current blockers:
- `StoredMessage` depends on `crate::message::{ContentBlock, Message, Role, ToolCall}`
- replay-event types currently depend on `crate::protocol::SwarmMemberStatus`
- replay-event plan snapshots currently depend on `crate::plan::PlanItem`
- the session module also owns persistence and storage concerns
So the next move should be a **session-contract slice**, not a full session crate.
### Best realistic session move
### Option B: narrow `jcode-session-contracts`
After shared contracts are extracted first, move the session types that are:
- serde-only
- reused outside `session.rs`
- not tied to `storage` or the full `Session` runtime
Good first candidates:
- `SessionStatus`
- `SessionImproveMode`
- `StoredDisplayRole`
- `StoredTokenUsage`
- `StoredCompactionState`
- `StoredMemoryInjection`
- `RenderedImageSource`
- `RenderedImage`
- `StoredReplayEvent` and `StoredReplayEventKind` once their swarm/plan payloads stop pointing back into `protocol.rs`
What should stay in the main crate for now:
- `Session`
- `StoredMessage`
- session journaling/file IO
- session startup/load/save orchestration
- message-to-image rendering functions
Why this is realistic:
- these contract structs already have broad fanout across agent, server, replay, and TUI code
- they are semantically session-level contracts, not session-runtime behavior
- the move becomes much cleaner once shared swarm/protocol payloads are extracted first
## Shared-contract boundary audit
This is the highest-leverage next seam.
There are several small, serde-only types that are clearly shared contracts already, but they currently live inside large modules:
- `PlanItem` in `src/plan.rs`
- `TranscriptMode` in `src/protocol.rs`
- `CommDeliveryMode` in `src/protocol.rs`
- `FeatureToggle` in `src/protocol.rs`
- `SessionActivitySnapshot` in `src/protocol.rs`
- `SwarmMemberStatus` in `src/protocol.rs`
- `AgentInfo` in `src/protocol.rs`
- `ContextEntry` in `src/protocol.rs`
- `SwarmChannelInfo` in `src/protocol.rs`
- `AwaitedMemberStatus` in `src/protocol.rs`
- `NotificationType` in `src/protocol.rs`
These are used across server, tool, TUI, replay, and session persistence flows, but they do not need the rest of `protocol.rs`.
### Best overall next move
### Option C: add `jcode-shared-contracts`
Recommended contents for the first pass:
- `PlanItem`
- `TranscriptMode`
- `CommDeliveryMode`
- `FeatureToggle`
- `SessionActivitySnapshot`
- swarm-related status/info structs:
- `SwarmMemberStatus`
- `AgentInfo`
- `ContextEntry`
- `SwarmChannelInfo`
- `AwaitedMemberStatus`
- `NotificationType`
Why this is the best next move:
- it breaks the `session.rs -> protocol.rs / plan.rs` dependency knot at the contract layer
- it gives replay/session persistence a clean dependency for swarm and plan snapshots
- it trims `protocol.rs` without trying to extract `Request` and `ServerEvent` yet
- it preserves the current successful pattern of a small, leaf-ish support crate with mostly `serde` types
Minimal dependency goal:
- `serde`
- nothing else, if possible
## Recommended sequencing
### Phase 1
Create `crates/jcode-shared-contracts`.
Expected immediate moves:
- `src/plan.rs` contents
- the small shared structs/enums listed above from `src/protocol.rs`
Keep in main crate for now:
- `Request`
- `ServerEvent`
- `encode_event` / `decode_request`
### Phase 2
Create `crates/jcode-session-contracts`.
Do this only after Phase 1, so session replay types can point at `jcode_shared_contracts::*` instead of `crate::protocol::*` or `crate::plan::*`.
### Phase 3
If a provider-side move is still desired before a larger provider refactor, extract only:
- provider identity enum
- pure selection/fallback ordering helpers
Do **not** include:
- `Provider` trait
- `EventStream`
- account failover
- auth state inspection
- runtime provider availability
- logging/bus side effects
## Moves to explicitly defer
These should be treated as later-stage refactors, not next-step crate moves.
### Defer: full `protocol.rs` crate
Reason:
- `Request` and `ServerEvent` still pull in `message`, `provider`, `session`, `side_panel`, and `bus`
- extracting the whole file now would create a broad, high-fanout crate instead of a clean contract crate
### Defer: full `session.rs` crate
Reason:
- the file mixes contracts, runtime state, rendering, journaling, and persistence
- `StoredMessage` still anchors the session layer to `message.rs`
### Defer: full provider trait / impl crate split
Reason:
- the trait seam is still mixed with runtime behavior and provider-specific execution policy
- moving it now would likely centralize churn rather than reduce it
### Defer: full `provider_catalog.rs` extraction
Reason:
- the file is still a runtime facade around env/config/auth probing, not just metadata
## Why this order avoids dependency-cycle mistakes
The sequence matters:
1. extract small shared contracts first
2. then extract session contracts that depend on those shared contracts
3. only then revisit deeper provider or protocol extraction
That order avoids creating crates that need to point back into the main crate for basic DTOs, which is exactly how high-churn dependency cycles usually start.
## Recommended concrete next actions
1. Add `crates/jcode-shared-contracts` with serde-only types from `plan.rs` and the small protocol/session overlap set.
2. Update `session.rs`, `protocol.rs`, server, tool, replay, and TUI imports to point at that crate.
3. Re-measure touched-file compile times for:
- `src/session.rs`
- `src/protocol.rs`
- `src/provider/mod.rs`
4. If the new seam stays clean, follow with a narrow `jcode-session-contracts` extraction.
5. Revisit provider trait extraction only after message/runtime/provider-execution seams are thinner.
docs/REFACTORING.md
+77
View File
@@ -0,0 +1,77 @@
# Refactoring Roadmap
This document defines the safe, incremental path for refactoring jcode while preserving behavior.
See also:
- [`docs/CODE_QUALITY_10_10_PLAN.md`](CODE_QUALITY_10_10_PLAN.md) for the code-quality target, phased uplift program, and initial hotspot refactor list.
- [`docs/COMPILE_PERFORMANCE_PLAN.md`](COMPILE_PERFORMANCE_PLAN.md) for compile-speed baselines, tactical build workflow, and the workspace/crate split roadmap.
## Goals
- Keep existing sessions and user workflows stable during refactors.
- Make regressions visible early with repeatable checks.
- Reduce architectural coupling in stages (not big-bang rewrites).
## Non-Negotiable Safety Rules
1. Use an isolated environment for refactor runs:
- `scripts/refactor_shadow.sh serve`
- `scripts/refactor_shadow.sh run`
- `scripts/refactor_shadow.sh build --release`
2. Before each refactor merge, run the phase-1 verification suite:
- `scripts/refactor_phase1_verify.sh`
3. Warning count may not increase above baseline:
- `scripts/check_warning_budget.sh`
4. Run security preflight before merges:
- `scripts/security_preflight.sh`
5. Prefer behavior-preserving moves first (extract/rename/split), then logic changes.
## Phase Plan
### Phase 1: Safety + Hygiene (current)
- Add isolated dev/run workflow for refactors.
- Add repeatable verification script.
- Add warning-budget guard to prevent warning drift.
- Clean low-risk warning debt without functional changes.
### Phase 2: CLI Decomposition
- Move `main.rs` subcommand handlers into focused `src/cli/*` modules.
- Keep top-level `main()` as parse + dispatch.
### Phase 3: Server Decomposition
- Split `server.rs` by responsibility (session lifecycle, debug API, swarm coordination, reload/update).
- Replace stringly states with typed enums where practical.
### Phase 4: Agent Turn-Loop Unification
- Consolidate duplicated turn-loop variants into one shared engine with pluggable event sink.
### Phase 5: TUI State/Reducer Split
- Separate app state, command parsing, remote-event reduction, and rendering control.
### Phase 6: Provider State Isolation
- Reduce global mutable state by moving caches into explicit state holders.
## Verification Matrix
- Compile: `cargo check -q`
- Compile timing: `scripts/bench_compile.sh check --runs 3 --touch <hot-file>` and `scripts/bench_compile.sh release-jcode --runs 3`
- Warnings: `scripts/check_warning_budget.sh`
- Security: `scripts/security_preflight.sh`
- Unit+integration tests: `cargo test -q`
- E2E tests: `cargo test --test e2e -q`
- Combined: `scripts/refactor_phase1_verify.sh`
docs/RENDER_PARITY_ACCEPTANCE_CRITERIA.md
+117
View File
@@ -0,0 +1,117 @@
# Render-Core Parity: Acceptance Criteria, Thresholds, and Statistical Reporting
Status: adopted for the `jcode-render-core` -> TUI switchover.
Harness: `crates/jcode-tui-markdown/src/render_core_adapter_tests.rs`
(differential tests comparing `render_markdown_via_core*` against the legacy
`render_markdown*` pipeline).
## 1. Parity levels (what is measured)
Each level is a distinct, machine-checkable comparison function. A level only
counts as covered when a test asserts it directly.
| Level | Definition | Comparator | Tests |
|-------|------------|------------|-------|
| L1: Content parity | Whitespace-collapsed visible text is identical | `flattened()` | `parity_*`, `fuzz_visible_text_parity`, `fuzz_random_documents_parity` |
| L2: Line-structure parity | Per-line trimmed non-blank visible text is identical (catches line-break divergence) | `nonblank_texts()` | `fuzz_random_documents_line_structure` |
| L3: Wrapped-layout parity | L2 comparison after production wrapping at widths {20, 40, 80} | `nonblank_texts()` on wrapped output | `fuzz_random_documents_wrapped_parity` |
| L4: Style invariants | Targeted styling equivalence: math fg spans identical, bold carries BOLD, inline code carries bg fill, headings colored by level, display math framed | span-level predicates | `probe_math_divergence`, `core_marks_bold_and_code_styling`, `core_renders_display_math_frame`, etc. |
Deliberately out of scope (documented divergence, not failure): blank-line
padding counts, decorative marker glyph choices, and exact `Style` equality on
non-invariant spans. Any new intentional divergence must be listed here.
## 2. Acceptance thresholds
All differential tests are **zero-tolerance**: the acceptance criterion is
`mismatches == 0` for every level at every tier. There is no "acceptable
mismatch rate"; the statistics below quantify what a passing run *proves*
about the residual mismatch probability, not what we tolerate.
| Tier | Iterations per fuzz suite | Gate | Residual mismatch rate bound (95%, rule of three: p < 3/N) |
|------|---------------------------|------|--------------------------------------------------|
| CI (default) | 5000 (L1, L2), 3000×3 widths = 9000 renders (L3) | must pass on every PR touching `jcode-tui-markdown`, `jcode-render-core` | p < 6.0e-4 per generated document (L1/L2); p < 1.0e-3 per doc per width (L3) |
| Pre-switchover deep run | `JCODE_MD_FUZZ_ITERS=100000` | must pass once, on the exact commit proposed for switchover | p < 3.0e-5 per generated document |
| Nightly (optional soak) | `JCODE_MD_FUZZ_ITERS=25000`, rotating `JCODE_MD_FUZZ_SEED` (e.g. epoch-day) | failures file an issue with repro seed | accumulates coverage across seeds over time |
Rationale for the bound: with N i.i.d. generated documents and 0 observed
mismatches, the one-sided 95% upper confidence bound on the mismatch
probability is `1 - 0.05^(1/N) ≈ 3/N` (the "rule of three").
Fixed-corpus criteria (non-statistical, exhaustive):
- Every entry in the `fuzz_visible_text_parity` corpus (currently 43 cases)
passes L1. Adding a construct to the generator **requires** adding at least
one fixed-corpus case for it, so regressions localize.
- Every `parity_*` unit case passes L1. Every L4 invariant test passes.
Switchover gate (all required):
1. CI tier green on the switchover commit.
2. Deep run (100k iters) green for L1, L2, and L3.
3. Deep run repeated with a second seed (any value differing from the
defaults) green, to reduce seed-specific blind spots.
4. Generator coverage checklist (section 4) has no unchecked construct that
the legacy renderer supports.
## 3. Statistical reporting requirements
Every fuzz test already implements, and must preserve, this reporting
contract on failure:
- **Reproducibility:** each failure reports the iteration index `i`; the
per-iteration RNG is derived as
`seed = base_seed + i * 0x100000001B3`, so any single failure is
reproducible with `JCODE_MD_FUZZ_SEED=<base_seed> JCODE_MD_FUZZ_ITERS=<i+1>`
(or by re-deriving the single seed). `base_seed` defaults are fixed
constants per suite and overridable via `JCODE_MD_FUZZ_SEED`.
- **Bounded failure dump:** collect up to 5 failing cases before aborting the
loop, then report all of them (input document, core output, legacy output)
in one assertion message. Never fail on only the first case; multiple
examples are needed to classify a divergence.
- **Full-input echo:** the raw markdown input is printed verbatim so a
failing case can be promoted directly into the fixed corpus.
Required additions when reporting results for a switchover decision (manual
or scripted summary, e.g. in the PR description):
- `iters`, `base_seed`, suite name, and pass/fail per level (L1-L4).
- The implied 95% upper bound `3/N` for each passed fuzz suite.
- For any failure found during development: a one-line classification
(parser divergence, adapter styling, wrap divergence, generator artifact)
and the corpus case added to pin the fix.
## 4. Generator coverage checklist
The statistical bound only covers the generator's distribution. The
generator (`gen_block`/`gen_inline`) must cover, and the checklist is audited
whenever a construct is added to either renderer:
- [x] headings (1-3), paragraphs, hard/soft breaks
- [x] bold, italic, strikethrough, inline code, links
- [x] inline math, display math, currency-dollar disambiguation
- [x] ordered/unordered/nested/task lists, definition lists
- [x] blockquotes (nested, multiline), thematic breaks
- [x] fenced code blocks (with/without language), tables (1-3 cols)
- [x] footnotes, CJK/emoji text
- [ ] setext headings (fixed corpus only, not in generator)
- [ ] images, autolinks, HTML fragments (fixed corpus only)
- [ ] reference-style links, indented code blocks (uncovered)
Unchecked items must either be added to the generator or explicitly waived in
the switchover PR with a fixed-corpus case demonstrating parity.
## 5. Running
```sh
# CI-equivalent (all levels, default iterations)
cargo test -p jcode-tui-markdown --lib render_core_adapter
# Deep run (switchover gate; ~80s on an XPS 13)
JCODE_MD_FUZZ_ITERS=100000 \
cargo test -p jcode-tui-markdown --lib render_core_adapter::tests::fuzz
# Alternate-seed confirmation
JCODE_MD_FUZZ_ITERS=100000 JCODE_MD_FUZZ_SEED=20260713 \
cargo test -p jcode-tui-markdown --lib render_core_adapter::tests::fuzz
```
docs/RESUME_BEHAVIOR.md
+39
View File
@@ -0,0 +1,39 @@
# `/resume` behavior
`/resume`, `/session`, and `/sessions` open the interactive session picker. They are local UI commands and must not be sent as chat prompts.
## Default action
- `Enter` resumes the highlighted session in the current terminal.
- `Ctrl+Enter` opens the highlighted session in a new terminal.
- `Esc`, `q`, or `Ctrl+C` closes the picker without changing sessions.
The default is controlled by:
```toml
[keybindings]
session_picker_enter = "current-terminal"
```
Set `session_picker_enter = "new-terminal"` to swap `Enter` and `Ctrl+Enter`.
## Current-terminal resume
When resuming in the current terminal:
- Jcode sessions switch the current workspace/client to that session.
- Importable external sessions are first converted to a Jcode session, then resumed in place.
- If multiple sessions are selected, only the first selected target is resumed in place and the UI tells the user this.
- The picker closes after queueing the in-place resume.
## New-terminal resume
When opening in a new terminal:
- Each selected target opens in its own terminal when possible.
- If terminal launch is unavailable, the UI prints manual `jcode --resume <id>` commands.
- The picker remains open after launching and clears the current multi-selection so more sessions can be opened.
## Saved sessions
`/save [label]` bookmarks the current session so it appears in the saved section of the picker. `/unsave` removes that bookmark.
docs/SAFETY_SYSTEM.md
+543
View File
@@ -0,0 +1,543 @@
# Safety System
> **Status:** Design
> **Updated:** 2026-02-08
A human-in-the-loop safety layer for unmonitored agent operations. Designed as an independent subsystem that any jcode feature can integrate with. Currently the only consumer is ambient mode, but the system is intentionally decoupled so it can be reused for future features.
## Overview
When an agent operates without direct user supervision (e.g. ambient mode), it needs a way to:
1. **Know what it's allowed to do** without asking
2. **Request permission** for actions that require human approval
3. **Notify the user** that a request is pending
4. **Wait or move on** while the user reviews
5. **Report what it did** after each session
The safety system provides all of this. There are only two tiers: auto-allowed and requires-permission. There is no "always denied" — if the user explicitly approves something, the agent can do it. The core principle is that **anything that communicates with another human or leaves a trace outside the local sandbox requires permission.**
---
## Architecture
```mermaid
graph TB
subgraph "Agent (e.g. Ambient Mode)"
A[Agent wants to take action]
AC{Action classification}
AUTO[Auto-allowed<br/>execute immediately]
PERM[Requires permission<br/>call request_permission tool]
end
subgraph "Safety System"
RQ[(Review Queue<br/>persistent)]
CL[Action Classifier]
NF[Notification Dispatcher]
TL[Transcript Logger]
SR[Session Reporter]
end
subgraph "Notification Channels"
EM[Email]
SM[SMS / Text]
DN[Desktop Notification]
WH[Webhook]
TUI[TUI Widget]
end
subgraph "User Review"
PH[Phone / Email]
CLI[jcode safety review]
TW[TUI Review Panel]
end
A --> CL
CL --> AC
AC -->|safe| AUTO
AC -->|needs review| PERM
PERM --> RQ
RQ --> NF
NF --> EM
NF --> SM
NF --> DN
NF --> WH
NF --> TUI
PH -->|approve/deny| RQ
CLI -->|approve/deny| RQ
TW -->|approve/deny| RQ
RQ -->|decision| A
AUTO --> TL
PERM --> TL
TL --> SR
style AUTO fill:#e8f5e9
style PERM fill:#fff3e0
style RQ fill:#e3f2fd
```
---
## Action Classification
Every action an agent wants to take is classified into one of two tiers. There is no "always denied" tier — if the user approves it, the agent can do it. The safety system's job is to make sure the user is asked, not to prevent actions entirely.
### Tier 1: Auto-Allowed (no permission needed)
Actions that are local, reversible, and don't affect anything outside the project sandbox.
| Action | Rationale |
|--------|-----------|
| Read files in project | Read-only, no side effects |
| Read git history / status | Read-only |
| Run tests (read-only) | Verification, no mutations |
| Memory operations (within per-cycle caps) | Local data, reversible |
| Create local branches / git worktrees | Local only, easily deleted |
| Write to ambient's own log/state files | Internal bookkeeping |
| Embed / similarity search | Computation only |
| Analyze sessions for extraction | Read-only analysis |
### Tier 2: Requires Permission (ask user)
Actions that leave a trace outside the local sandbox, affect shared state, or can't be easily undone. **The general rule: anything that communicates directly with another human always requires permission — no exceptions.**
| Action | Rationale |
|--------|-----------|
| **Communication with humans (always Tier 2)** | |
| Send emails | Irreversible, visible to others |
| Submit assignments | Academic consequences |
| Post to Slack / Discord / chat | Visible to others |
| Create GitHub issues / PR comments | Publicly visible |
| Any form of direct human communication | Cannot be unsent |
| **Code modifications** | |
| Modify code in a repo (must use worktree + PR) | Requires review before merge |
| Push to remote | Visible to collaborators |
| Create pull requests | Visible to collaborators |
| Modify CI/CD pipelines | Affects shared infrastructure |
| **System changes** | |
| Install system packages | Modifies system state |
| Modify dotfiles / system config | Affects other tools |
| Start network services / open ports | Security implications |
| **Deployment** | |
| Deploy to any environment | Affects users/services |
| **Data** | |
| Delete files outside project sandbox | May not be recoverable |
| Drop databases / clear non-trivial caches | Data loss risk |
| **Financial / Account** | |
| Purchases / billing changes | Financial consequences |
| Change passwords / API keys / auth | Security consequences |
| Revoke tokens / modify permissions | Access consequences |
### Custom Rules
Users can configure custom classification rules to promote or demote actions:
```toml
[safety.rules]
# Promote: allow ambient to create PRs without asking
allow_without_permission = ["create_pull_request"]
# Demote: always ask before running any tests (e.g. expensive integration tests)
require_permission = ["run_tests"]
# Override: allow push to specific remotes
allow_push_to = ["origin"]
```
---
## Permission Request Flow
```mermaid
sequenceDiagram
participant AG as Agent
participant CL as Classifier
participant RQ as Review Queue
participant NF as Notifier
participant US as User
AG->>CL: "I want to create a PR"
CL->>CL: Classify action → Tier 2
CL->>AG: Permission required
AG->>RQ: request_permission({action, context, rationale})
RQ->>RQ: Store pending request
RQ->>NF: Dispatch notification
NF->>US: Email: "jcode ambient wants to create a PR"
NF->>US: Desktop notification (if available)
Note over AG: Agent decides: wait or move on?
alt Wait for approval
AG->>AG: Block on this action, continue other work
else Move on
AG->>AG: Skip this action, continue with next task
end
US->>RQ: Approve (via email link / CLI / TUI)
RQ->>AG: Permission granted
alt Agent waited
AG->>AG: Execute the action
else Agent moved on
AG->>AG: Execute on next ambient cycle
end
```
### The `request_permission` Tool
Available to any agent operating under the safety system:
```rust
// Tool: request_permission
{
"action": "create_pull_request",
"description": "Create PR for ambient/fix-auth-tests branch with 3 test fixes",
"rationale": "Found 3 failing tests in auth module. Fixed them on a worktree branch.",
"urgency": "low", // "low" | "normal" | "high"
"wait": false // should the agent block until approved?
}
```
**Response:**
```rust
// If wait=true and user responds:
{ "approved": true, "message": "looks good" }
// If wait=true and timeout:
{ "approved": false, "reason": "timeout", "timeout_minutes": 60 }
// If wait=false:
{ "queued": true, "request_id": "req_abc123" }
```
### Agent Behavior While Waiting
When the agent requests permission with `wait: true`:
- It doesn't block the entire cycle — it moves on to other ambient tasks
- When the user approves, the action is queued for the next cycle (or current cycle if still running)
- If the user doesn't respond within a configurable timeout, the request expires and is logged
When the agent requests permission with `wait: false`:
- The request is queued for user review
- The agent continues without the action
- If approved later, it's picked up on the next ambient cycle
---
## Notification System
### Channels
```mermaid
graph LR
subgraph "Notification Dispatcher"
ND[Dispatcher]
end
subgraph "Channels"
EM[Email<br/>SMTP / SendGrid / etc]
SM[SMS<br/>Twilio / similar]
DN[Desktop<br/>notify-send / Wayland]
WH[Webhook<br/>custom HTTP POST]
TUI[TUI Widget<br/>in-app badge]
end
ND --> EM
ND --> SM
ND --> DN
ND --> WH
ND --> TUI
style ND fill:#e3f2fd
style EM fill:#fff3e0
style SM fill:#fff3e0
style DN fill:#fff3e0
style WH fill:#fff3e0
style TUI fill:#e8f5e9
```
**Channel priority:** Users configure which channels to use and in what order. Notifications are sent to all enabled channels simultaneously.
**Notification content:**
- What the agent wants to do (action + description)
- Why it wants to do it (rationale)
- How to approve/deny (link or instructions)
- Urgency level
### Configuration
```toml
[safety.notifications]
# Enable/disable channels
email = true
sms = false
desktop = true
webhook = false
# Email settings
[safety.notifications.email]
to = "jeremy@example.com"
# Provider: "smtp", "sendgrid", "ses"
provider = "smtp"
smtp_host = "smtp.gmail.com"
smtp_port = 587
# SMS settings (if enabled)
[safety.notifications.sms]
to = "+1234567890"
provider = "twilio"
# Webhook (if enabled)
[safety.notifications.webhook]
url = "https://example.com/jcode-safety"
secret = "..."
# Desktop notification (uses notify-send or similar)
[safety.notifications.desktop]
enabled = true
# Notification preferences
[safety.notifications.preferences]
# Only notify for these urgency levels and above
min_urgency = "low" # "low" | "normal" | "high"
# Batch notifications (don't spam)
batch_interval_seconds = 60 # Collect notifications for 60s before sending
# Quiet hours (no notifications except high urgency)
quiet_start = "23:00"
quiet_end = "07:00"
```
---
## Session Transcript & Summary
After every ambient cycle (or any unmonitored agent session), the safety system generates a report.
### Transcript
Full log of every action taken, with context:
```json
{
"session_id": "ambient-2026-02-08-143022",
"started_at": "2026-02-08T14:30:22Z",
"ended_at": "2026-02-08T14:35:18Z",
"provider": "openai",
"model": "5.2-codex-xhigh",
"token_usage": { "input": 12400, "output": 3200 },
"actions": [
{
"type": "memory_consolidation",
"description": "Merged 2 duplicate memories about dark mode preference",
"tier": "auto_allowed",
"details": { "merged": ["mem_abc", "mem_def"], "into": "mem_ghi" }
},
{
"type": "memory_prune",
"description": "Deactivated 1 memory with confidence 0.02",
"tier": "auto_allowed",
"details": { "pruned": ["mem_xyz"] }
},
{
"type": "permission_request",
"description": "Create PR for 3 auth test fixes",
"tier": "requires_permission",
"status": "pending",
"request_id": "req_abc123"
}
],
"pending_permissions": 1,
"scheduled_next": "2026-02-08T15:05:00Z"
}
```
### Summary
A human-readable summary sent via configured notification channels:
```
Ambient cycle completed (4m 56s)
Done:
- Merged 2 duplicate memories (dark mode preference)
- Pruned 1 stale memory (confidence: 0.02)
- Extracted 3 memories from crashed session jcode-red-fox-1234
- Verified 5 facts against codebase (all still valid)
Needs your review:
- [Approve/Deny] Create PR for auth test fixes (3 files changed)
Next cycle: ~35 minutes
Budget: 62% remaining today
```
### Delivery
- **Always:** Written to `~/.jcode/ambient/transcripts/YYYY-MM-DD-HHMMSS.json`
- **If email enabled:** Summary sent after each cycle (respecting batch interval)
- **If TUI open:** Summary shown in ambient info widget
- **CLI:** `jcode ambient log` to view recent transcripts
---
## Review Queue
### Storage
```
~/.jcode/safety/
├── queue.json # Pending permission requests
├── history.json # Past decisions (for learning patterns)
└── config.json # Cached safety configuration
```
### Review Interfaces
**1. TUI (when jcode is open)**
A review panel showing pending requests:
```
╭─ Safety Review (1 pending) ──────────────────╮
│ │
│ [HIGH] Create PR for auth test fixes │
│ Branch: ambient/fix-auth-tests │
│ Files: 3 changed (+42 -18) │
│ Rationale: Found 3 failing tests in auth │
│ module during ambient scout. │
│ │
│ [a] Approve [d] Deny [v] View diff │
╰───────────────────────────────────────────────╯
```
**2. CLI**
```bash
jcode safety review # Interactive review of pending requests
jcode safety list # List all pending requests
jcode safety approve <id> # Approve a specific request
jcode safety deny <id> # Deny a specific request
jcode safety log # View decision history
```
**3. Email / Remote**
Notification emails include approve/deny links. These hit a local webhook (or use a relay service) to record the decision.
### Decision History
Past decisions are stored so the system can learn patterns:
```json
{
"request_id": "req_abc123",
"action": "create_pull_request",
"decision": "approved",
"decided_at": "2026-02-08T14:42:00Z",
"decided_via": "tui",
"response_time_seconds": 420
}
```
This history could eventually feed into smarter classification — if the user always approves a certain type of action, suggest promoting it to auto-allowed.
---
## Integration API
The safety system exposes a simple API for any jcode feature to use:
```rust
pub struct SafetySystem {
classifier: ActionClassifier,
queue: ReviewQueue,
notifier: NotificationDispatcher,
logger: TranscriptLogger,
}
impl SafetySystem {
/// Check if an action is allowed without permission
pub fn is_auto_allowed(&self, action: &Action) -> bool;
/// Request permission for an action
/// Returns immediately if wait=false, blocks until decision if wait=true
pub async fn request_permission(&self, request: PermissionRequest) -> PermissionResult;
/// Log an action that was taken (for transcript)
pub fn log_action(&self, action: &ActionLog);
/// Generate session summary
pub fn generate_summary(&self) -> SessionSummary;
/// Get pending requests
pub fn pending_requests(&self) -> Vec<PermissionRequest>;
/// Record a decision (from TUI, CLI, or remote)
pub fn record_decision(&self, request_id: &str, decision: Decision) -> Result<()>;
}
pub struct PermissionRequest {
pub id: String,
pub action: String,
pub description: String,
pub rationale: String,
pub urgency: Urgency,
pub wait: bool,
pub context: Option<serde_json::Value>,
}
pub enum PermissionResult {
Approved { message: Option<String> },
Denied { reason: Option<String> },
Queued { request_id: String },
Timeout,
}
pub enum Urgency {
Low,
Normal,
High,
}
```
---
## Implementation Phases
### Phase 1: Foundation
- [ ] Action classifier (tier 1/2/3 lookup)
- [ ] Review queue (persistent storage)
- [ ] `request_permission` tool for agents
- [ ] Transcript logger
- [ ] Basic session summary generation
### Phase 2: Notification Channels
- [ ] Desktop notifications (notify-send / Wayland)
- [ ] Email notifications (SMTP)
- [ ] Webhook support
- [ ] Notification batching and quiet hours
- [ ] SMS (Twilio or similar)
### Phase 3: Review Interfaces
- [ ] TUI review panel
- [ ] CLI commands (`jcode safety review/list/approve/deny/log`)
- [ ] Email approve/deny links (relay service)
### Phase 4: Configuration
- [ ] `[safety]` config section
- [ ] Custom classification rules (promote/demote actions)
- [ ] Per-project overrides
- [ ] Notification channel configuration
### Phase 5: Intelligence
- [ ] Decision history tracking
- [ ] Pattern detection (auto-suggest promotions)
- [ ] Urgency inference from context
---
*Last updated: 2026-02-08*
docs/SECURITY_DEPENDENCIES.md
+43
View File
@@ -0,0 +1,43 @@
# Dependency Security Triage
Last reviewed: 2026-05-14
This file tracks the current `cargo audit` findings for jcode and the intended remediation path.
It is not an allowlist. It is a triage record so advisories are visible and actionable.
## Current advisories
| Advisory | Crate | Dependency path | Affected area in jcode | Triage | Planned action |
|---|---|---|---|---|---|
| `RUSTSEC-2025-0141` | `bincode` | `syntect -> bincode` | Markdown/code highlighting in the TUI | Unmaintained transitive dependency. No direct exposure in the provider/auth flow. | Track `syntect` upgrades or replace `syntect` if upstream does not move off `bincode` soon. |
| `RUSTSEC-2024-0436` | `paste` | `ratatui -> paste`, `tokenizers -> paste`, `tract-* -> paste` | TUI rendering, tokenizers, embedding/model support | Widely transitive. Not isolated to one module. | Prefer upstream dependency upgrades before any local workaround. Re-evaluate after bumping `ratatui`, `tokenizers`, and `tract-*`. |
| `RUSTSEC-2026-0002` | `lru` | `ratatui -> lru` | TUI rendering/cache internals | Unsoundness warning in a UI dependency. Not in auth/provider logic, but still ships in-process. | Upgrade `ratatui` / `ratatui-image` together once compatible. |
| `RUSTSEC-2026-0097` | `rand` | `azure_core`, `tungstenite`, `tract-*`, `ratatui-image`, and others | Azure auth, websocket, embedding, and UI transitive paths | Unsoundness warning involving custom loggers using `rand::rng()`. Jcode does not intentionally use that pattern, but the crate is broad in the graph. | Prefer upstream upgrades to `rand` 0.9-compatible dependency stacks. |
| `RUSTSEC-2026-0141` | `lettre` | `jcode-notify-email -> lettre` | Notification email sending | Vulnerability applies to the Boring TLS backend hostname verification path. Jcode's `lettre` dependency uses rustls/native-tls features, not `boring-tls`, so this is not believed exploitable in the current build. | Keep ignored in `scripts/security_preflight.sh`; remove ignore after `lettre` ships a patched release or if feature use changes. |
| `RUSTSEC-2026-0098` | `rustls-webpki` | `rustls` dependency stack | TLS certificate validation in rustls consumers | Name constraints for URI names incorrectly accepted. Transitive via TLS libraries. | Upgrade rustls/webpki stack when compatible releases are available. |
| `RUSTSEC-2026-0099` | `rustls-webpki` | `rustls` dependency stack | TLS certificate validation in rustls consumers | Name constraints accepted for wildcard certificates. Transitive via TLS libraries. | Upgrade rustls/webpki stack when compatible releases are available. |
| `RUSTSEC-2026-0104` | `rustls-webpki` | `rustls` dependency stack | TLS certificate revocation list parsing | Reachable panic in CRL parsing. Transitive via TLS libraries. | Upgrade rustls/webpki stack when compatible releases are available. |
| `RUSTSEC-2026-0049` | `rustls-webpki` | `rustls` dependency stack (`aws-smithy` rustls 0.21, `imap`/`rustls-connector` rustls 0.22) | TLS certificate revocation list handling | CRLs not considered authoritative by Distribution Point due to faulty matching logic. Transitive via the older rustls stacks; fix needs rustls-webpki >=0.103.10, which requires major bumps of the `aws-sdk`/`imap` stacks. | Upgrade rustls/webpki stack when compatible releases are available. |
| `RUSTSEC-2026-0187` | `lopdf` | `jcode-pdf -> pdf-extract 0.8.2 -> lopdf 0.34` | PDF text extraction (`/pdf`, image/PDF reads) | Stack overflow parsing deeply nested PDF objects. Only reached when extracting text from a (potentially malicious) PDF the user opens; not in the auth/provider/network path. `pdf-extract 0.8.2` pins `lopdf 0.34`, so it cannot be bumped to the fixed `>=0.42` without an upstream `pdf-extract` release. | Upgrade once `pdf-extract` ships a release depending on `lopdf >=0.42`; remove the ignore then. |
| `RUSTSEC-2023-0086` | `lexical-core` | `imap -> imap-proto -> lexical-core` | Gmail/IMAP support path | Old unsound transitive dependency in the mail stack. Higher priority than the UI-only findings because it touches network-parsed data. | Investigate upgrading or replacing `imap` / `imap-proto`. If no maintained path exists, isolate or remove the IMAP dependency. |
## Priority order
1. `rustls-webpki` TLS advisories via rustls stack
2. `lexical-core` via `imap-proto`
3. `lettre` if Jcode ever enables `boring-tls`
4. `lru` via `ratatui`
5. `bincode` via `syntect`
6. `paste` / `rand` via multiple transitive dependencies
## Notes
- None of the advisories above were introduced by the provider-auth refactor.
- The provider/auth hardening work should continue independently of these dependency upgrades.
- `RUSTSEC-2024-0320` (`yaml-rust`) was removed from the dependency graph on 2026-03-05 by trimming `syntect` features to built-in syntax/theme dumps instead of YAML loading.
- `RUSTSEC-2026-0194` / `RUSTSEC-2026-0195` (`quick-xml` 0.39.2): reached only through `wayland-scanner`, a build-time proc-macro in the desktop crate's winit stack. It parses trusted, vendored Wayland protocol XML during compilation and never touches untrusted input at runtime. Remediation is upstream: `wayland-scanner` needs to move to `quick-xml >= 0.41`. Triaged and ignored in `scripts/security_preflight.sh` on 2026-07-04.
- `scripts/security_preflight.sh` ignores the vulnerability advisories that are explicitly triaged above (`lettre` and `rustls-webpki`) so CI can remain actionable. New vulnerabilities still fail CI by default.
- Before changing dependency versions, run:
- `cargo check`
- `cargo test -j 1`
- `scripts/security_preflight.sh`
docs/SERVER_ARCHITECTURE.md
+157
View File
@@ -0,0 +1,157 @@
# Server Architecture
See also:
- [`SERVER_SERVICE_SPLIT_PLAN.md`](./SERVER_SERVICE_SPLIT_PLAN.md)
- [`SWARM_ARCHITECTURE.md`](./SWARM_ARCHITECTURE.md)
- [`MULTI_SESSION_CLIENT_ARCHITECTURE.md`](./MULTI_SESSION_CLIENT_ARCHITECTURE.md)
## Overview
jcode uses a **single-server, multi-client** architecture. One server process
manages all sessions and state; TUI clients connect over a Unix socket and
can reconnect transparently after disconnects or server reloads.
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ SERVER (🔥 blazing) │
│ │
│ jcode serve │
│ ├── Unix socket: /run/user/$UID/jcode.sock │
│ ├── Debug socket: /run/user/$UID/jcode-debug.sock │
│ ├── Registry: ~/.jcode/servers.json │
│ ├── Provider (Claude/OpenAI/OpenRouter) │
│ ├── MCP pool (shared across sessions) │
│ └── Sessions: │
│ ├── 🦊 fox (active) → "🔥 blazing 🦊 fox" │
│ ├── 🐻 bear (active) → "🔥 blazing 🐻 bear" │
│ └── 🦉 owl (idle) → "🔥 blazing 🦉 owl" │
└─────────────────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Client 1│ │ Client 2│ │ Client 3│
│ 🦊 fox │ │ 🐻 bear │ │ 🦉 owl │
└─────────┘ └─────────┘ └─────────┘
```
## Naming
```
SERVER = Adjective/Verb modifier SESSIONS = Animal nouns
──────────────────────────── ────────────────────────
🔥 blazing ❄️ frozen ⚡ swift 🦊 fox 🐻 bear 🦉 owl
🌀 rising 🍂 falling 🌊 rushing 🌙 moon ⭐ star 🔥 fire
✨ bright 🌑 dark 💫 spinning 🐺 wolf 🦁 lion 🐋 whale
Combined: "🔥 blazing 🦊 fox" = server + session
```
The server gets a random adjective/verb name on startup (e.g., "blazing").
Each session gets an animal noun (e.g., "fox"). Together they form a natural
phrase displayed in the UI: "🔥 blazing 🦊 fox".
The server name persists across reloads via the registry (`~/.jcode/servers.json`).
When the server execs into a new binary on `/reload`, the new process registers
with a fresh name. Stale entries are cleaned up automatically.
## Lifecycle
```
START CONNECT RELOAD
───── ─────── ──────
jcode (first run) jcode (subsequent) /reload
│ │ │
├─▶ No server? Spawn daemon ├─▶ Server exists? ├─▶ Server execs into
├─▶ Wait for socket │ Connect directly │ new binary (same PID)
├─▶ Connect as client │ ├─▶ All clients disconnect
└─▶ Create session └─▶ Create/resume session └─▶ Clients auto-reconnect
```
### Server Startup
When you run `jcode`, it checks if a server is already running:
1. **Server exists**: connect directly as a client
2. **No server**: spawn `jcode serve` as a detached daemon (with `setsid`),
wait for the socket, then connect
The server is fully detached from the spawning client via `setsid()`, so killing
any client never affects the server or other clients.
Long-lived deployments can give the daemon a stable client-visible identity with
`jcode serve --server-name <name>` or the `JCODE_SERVER_NAME` environment
variable. The optional `JCODE_SERVER_DISPLAY_NAME` environment variable is also
accepted for service managers that prefer a display-oriented name. CLI input wins
over environment input. Names are normalized to registry-safe lowercase labels,
so `mount-cloud/fabian` displays as `mount-cloud-fabian`.
### Server Shutdown
The server shuts down when:
- **Idle timeout**: no clients connected for 5 minutes (configurable)
- **Manual**: server process is killed
- **Reload**: server execs into a new binary (same socket path)
### Remote Client Working Directory
By default, a client sends its current working directory to the server when it
subscribes, and the server uses that as the session working directory. Socket
forwarding wrappers for remote daemons can keep the client and server paths
separate with `--remote-working-dir`:
```bash
jcode --socket /tmp/jcode.sock -C /local/checkout --remote-working-dir /remote/checkout
```
`-C` must exist on the client. `--remote-working-dir` must be an absolute path
that exists on the server.
### Client Reconnection
Clients have a built-in reconnect loop. When the connection drops (server
reload, network issue, etc.):
1. Client shows "Connection lost - reconnecting..."
2. Retries with exponential backoff (1s, 2s, 4s... up to 30s)
3. On reconnect, resumes the same session (session state persists on disk)
4. If server was reloaded, client may also re-exec itself if a newer
client binary is available
### Hot Reload (`/reload`)
1. Client sends `Request::Reload` to server
2. Server sends `Reloading` event to the requesting client
3. Server calls `exec()` into the new binary with `serve` args
4. New server process starts on the same socket
5. All clients auto-reconnect
6. The initiating client also re-execs if its binary is outdated
## Socket Paths
```
/run/user/$UID/
├── jcode.sock # Main communication socket
└── jcode-debug.sock # Debug/testing socket
```
## Self-Dev Mode
When running `jcode` inside the jcode repository:
1. Auto-detects the repo and enables self-dev mode
2. Connects to the normal shared jcode server
3. Marks that session as canary/self-dev via subscribe metadata
4. Enables selfdev prompt/tooling only for that session
5. `/reload` still hot-reloads the shared server and clients reconnect
## Key Behaviors
| Scenario | Behavior |
|----------|----------|
| First `jcode` run | Spawns server daemon, connects |
| Subsequent `jcode` | Connects to existing server |
| Kill a client | Server + other clients unaffected |
| `/reload` | Server execs new binary, clients reconnect |
| All clients close | Server idle-timeout after 5 min |
| Resume session | `jcode --resume fox` reconnects to existing session |
docs/SERVER_SERVICE_SPLIT_PLAN.md
+598
View File
@@ -0,0 +1,598 @@
# Server Service Split Plan
Status: Audit-based plan
Scope: `src/server*.rs` and `src/server/**/*.rs` in the current shared-server architecture.
This document audits the current server stack and proposes an incremental split into five in-process services:
- session
- client
- swarm
- debug
- maintenance
The intent is to improve ownership boundaries and reduce argument fanout without changing the single-process runtime model.
See also:
- [`SERVER_ARCHITECTURE.md`](./SERVER_ARCHITECTURE.md)
- [`SWARM_ARCHITECTURE.md`](./SWARM_ARCHITECTURE.md)
- [`UNIFIED_SELFDEV_SERVER_PLAN.md`](./UNIFIED_SELFDEV_SERVER_PLAN.md)
## Executive Summary
Today the server is already logically split by file, but not by ownership boundary.
The dominant pattern is:
- `Server` owns nearly all shared state in one struct.
- `ServerRuntime` clones that full state bag into connection handlers.
- `handle_client()` and `handle_debug_client()` receive very wide dependency lists.
- maintenance loops in `server.rs` mutate the same raw maps used by client, session, swarm, and debug paths.
That means the main extraction seam is **not** transport or process boundaries. The main seam is introducing **service-owned state + service APIs** inside the existing process.
The safest path is:
1. keep one server process
2. keep current modules and behavior
3. introduce service handle structs around existing state
4. move mutation behind service methods
5. reduce `handle_client()` and `handle_debug_client()` to a few service/context arguments
Do **not** start with crates, traits, or IPC splits. The code is not ready for that yet, and the current pain is mostly ownership fanout, not runtime topology.
## Current Stack Audit
### Top-level runtime shape
Current runtime flow:
```mermaid
flowchart TD
Server[server.rs::Server] --> Runtime[server/runtime.rs::ServerRuntime]
Runtime --> MainAccept[main socket accept loop]
Runtime --> DebugAccept[debug socket accept loop]
Runtime --> GatewayAccept[gateway accept loop]
MainAccept --> ClientLifecycle[client_lifecycle.rs::handle_client]
DebugAccept --> DebugRouter[debug.rs::handle_debug_client]
GatewayAccept --> ClientLifecycle
Server --> Maintenance[reload, bus monitor, idle timeout, registry, memory, ambient]
ClientLifecycle --> SessionModules[session/actions/provider/session-state handlers]
ClientLifecycle --> SwarmModules[comm_* and swarm handlers]
DebugRouter --> DebugModules[debug_* handlers]
```
### Shared state concentration
`src/server.rs` owns one large `Server` struct with state spanning all concerns, including:
- sessions and default session id
- client count and client connection map
- swarm membership, plans, shared context, coordinator map
- file touch tracking and reverse indexes
- channel subscriptions and reverse indexes
- debug client routing and debug jobs
- swarm event history and event bus
- ambient runner, shared MCP pool
- shutdown signals and soft interrupt queues
- await-members runtime
This is a service container in practice, but it is represented as one broad state owner.
### Existing positive seams
The code already contains a few useful seams we should preserve:
- `runtime.rs` already isolates accept-loop orchestration from bootstrap.
- `state.rs` already centralizes shared types and delivery helpers.
- `swarm.rs` is already the closest thing to a stateful domain service.
- `reload.rs` is already separate from bootstrap, even though `server.rs` still owns most maintenance wiring.
- `debug_*` modules are already split by debug command domain.
These are good extraction points. The plan below leans on them instead of fighting them.
## Module Heat Map
Largest server-side modules at the time of audit:
| File | Lines | Primary concern today | Future service |
|---|---:|---|---|
| `src/server/client_lifecycle.rs` | 1767 | client request loop and router | client |
| `src/server/client_comm.rs` | 1492 | swarm communication requests | swarm |
| `src/server/client_actions.rs` | 1249 | session-local actions | session |
| `src/server/swarm.rs` | 1202 | swarm state mutation and fanout | swarm |
| `src/server/comm_control.rs` | 1183 | swarm control / await-members / client debug bridge | swarm + debug |
| `src/server/client_session.rs` | 1091 | subscribe, resume, clear, reload | session + client boundary |
| `src/server/comm_session.rs` | 987 | spawn/stop session flows | session + swarm boundary |
| `src/server/debug.rs` | 980 | debug socket command router | debug |
| `src/server/reload.rs` | 826 | reload and graceful shutdown | maintenance |
| `src/server/debug_server_state.rs` | 748 | debug snapshots across all stores | debug |
Interpretation:
- The architecture is not blocked on missing modules.
- It is blocked on **cross-service state access** and **router width**.
## Where Coupling Is Highest
### 1. `ServerRuntime` is a full-state courier
`runtime.rs` clones almost every shared field into the runtime and forwards them into:
- main client handling
- debug client handling
- gateway client handling
This makes transport code depend on internal service storage details.
### 2. `handle_client()` is both connection loop and application router
`client_lifecycle.rs::handle_client()` currently combines:
- stream read loop
- per-connection state
- session attach / resume / clear
- provider control
- swarm communication dispatch
- debug bridge requests
- message processing lifecycle
- disconnect cleanup
That is the clearest signal that client, session, swarm, and debug responsibilities are crossing in one place.
### 3. session flows directly mutate swarm state
`client_session.rs` does real session work, but also directly touches:
- swarm member registration
- channel subscription cleanup
- plan participant rename/removal
- status updates
- event sender registration
- interrupt queue rename/removal
That makes session lifecycle hard to extract cleanly because it owns both agent state and swarm membership side effects.
### 4. maintenance loops reach into domain maps directly
`server.rs` maintenance tasks currently touch shared state directly for:
- reload handling
- background task wakeup / notification delivery
- bus monitoring and file touch conflict detection
- idle timeout
- runtime memory logging
- registry publishing
- ambient scheduling
This makes background jobs depend on storage layout instead of service APIs.
### 5. debug paths bypass future boundaries
`debug.rs` and `debug_*` modules inspect or mutate many raw stores directly.
That is fine for now, but it will block extraction unless debug becomes a consumer of service snapshots and public mutation methods.
## Proposed Service Split
The target split is still one process and one Tokio runtime.
The change is ownership and APIs, not deployment.
### 1. Session Service
**Owns:**
- `sessions`
- `session_id` default/global session tracking
- `shutdown_signals`
- `soft_interrupt_queues`
- session event sender registration and fanout
- session-local agent actions and provider/session mutation
- headless session creation primitives
**Primary modules after split:**
- `state.rs` delivery pieces
- `client_session.rs` session-only parts
- `client_actions.rs`
- `provider_control.rs`
- `headless.rs`
- parts of `reload.rs` for graceful shutdown helpers
**Public API examples:**
- `attach_client(...)`
- `resume_session(...)`
- `clear_session(...)`
- `spawn_headless_session(...)`
- `queue_soft_interrupt(...)`
- `fanout_session_event(...)`
- `rename_session(...)`
- `shutdown_session(...)`
- `session_snapshot(...)`
**Boundary rule:** session service should not directly own swarm membership rules.
It can expose lifecycle events or return session metadata that another layer uses to update swarm state.
### 2. Client Service
**Owns:**
- socket, debug socket, gateway transport accept loops
- client connection registry
- client count / attachment count
- connection-scoped state and request routing
- subscribe / reconnect orchestration across services
- client API wrappers
**Primary modules after split:**
- `runtime.rs`
- `socket.rs`
- `client_api.rs`
- `client_lifecycle.rs` connection loop and router only
- `client_disconnect_cleanup.rs`
- client-facing parts of `client_state.rs`
**Public API examples:**
- `spawn_accept_loops(...)`
- `run_client_connection(stream)`
- `register_connection(...)`
- `cleanup_connection(...)`
- `connected_clients_snapshot()`
**Boundary rule:** client service routes requests, but does not own business state for sessions, swarms, or debug jobs.
### 3. Swarm Service
**Owns:**
- `swarm_members`
- `swarms_by_id`
- `shared_context`
- `swarm_plans`
- `swarm_coordinators`
- channel subscriptions and reverse indexes
- swarm event history and event broadcast
- file touch tracking and reverse indexes
- await-members runtime
- status broadcasting, plan broadcasting, conflict notifications
**Primary modules after split:**
- `swarm.rs`
- `client_comm.rs`
- `comm_plan.rs`
- `comm_control.rs` swarm portions
- `comm_session.rs` swarm coordination portions
- `comm_sync.rs`
- file-touch portions of `server.rs::monitor_bus`
- `await_members_state.rs`
**Public API examples:**
- `join_swarm(...)`
- `leave_swarm(...)`
- `set_member_status(...)`
- `assign_role(...)`
- `update_plan(...)`
- `subscribe_channel(...)`
- `publish_notification(...)`
- `record_file_touch(...)`
- `detect_conflicts(...)`
- `await_members(...)`
- `snapshot_swarm(...)`
**Boundary rule:** swarm service can request message delivery through the session service, but should not reach into raw session maps.
### 4. Debug Service
**Owns:**
- debug socket request router
- client debug bridge state
- debug job registry
- testers and debug command execution helpers
- server and swarm snapshots for inspection
**Primary modules after split:**
- `debug.rs`
- `debug_command_exec.rs`
- `debug_events.rs`
- `debug_help.rs`
- `debug_jobs.rs`
- `debug_server_state.rs`
- `debug_session_admin.rs`
- `debug_swarm_read.rs`
- `debug_swarm_write.rs`
- `debug_testers.rs`
- `debug_ambient.rs`
**Public API examples:**
- `run_debug_connection(stream)`
- `submit_debug_job(...)`
- `server_snapshot()`
- `swarm_snapshot(...)`
- `route_transcript_injection(...)`
**Boundary rule:** debug service should read snapshots from other services and mutate them only through explicit service methods.
It should not be a privileged backdoor around normal APIs except where intentionally documented.
### 5. Maintenance Service
**Owns:**
- reload monitor and reload-state plumbing
- registry publish / cleanup background tasks
- idle timeout monitor
- runtime memory logging loop
- embedding preload and idle unload
- ambient loop startup/wiring
- background task completion delivery orchestration
- bus subscription loops that translate infra events into service calls
**Primary modules after split:**
- `reload.rs`
- `reload_state.rs`
- background-task delivery logic from `server.rs`
- registry and idle-timeout pieces from `server.rs`
- runtime memory logging pieces from `server.rs`
- `monitor_bus()` after it is narrowed to service calls
**Public API examples:**
- `start_background_loops(...)`
- `handle_reload_signal(...)`
- `deliver_background_task_completion(...)`
- `publish_registry_metadata(...)`
- `run_idle_monitor(...)`
- `run_bus_monitor(...)`
**Boundary rule:** maintenance service should orchestrate services, not own their domain maps.
## Recommended Dependency Direction
```mermaid
flowchart LR
Client[Client Service] --> Session[Session Service]
Client --> Swarm[Swarm Service]
Client --> Debug[Debug Service]
Swarm --> Session
Debug --> Session
Debug --> Swarm
Maintenance --> Session
Maintenance --> Swarm
Maintenance --> Client
```
Rules:
- `Server` becomes bootstrap and wiring only.
- `ServerRuntime` becomes transport runtime only.
- session and swarm are the main domain services.
- debug and maintenance depend on domain services, not the other way around.
## Concrete Extraction Seams
### Seam A: turn `state.rs` into the session-delivery foundation
`state.rs` already contains the best low-risk shared seam:
- session event sender registration
- session event fanout
- soft interrupt queue registration and enqueue
Make this the initial backbone of the session service instead of leaving it as generic helpers.
Why this is safe:
- logic is already centralized
- heavily reused by swarm, debug, and maintenance
- extraction reduces duplication of `SessionAgents` and queue plumbing without changing behavior
### Seam B: separate connection routing from business handlers
Split `client_lifecycle.rs` into:
- `ClientConnection` or `ClientLoop` for stream handling and per-client state
- `ClientRequestRouter` for mapping `Request` variants to service calls
The router should depend on `SessionService`, `SwarmService`, and `DebugService`, not raw `Arc<RwLock<HashMap<...>>>` fields.
Why this is safe:
- no protocol change
- no state ownership change yet
- mostly signature narrowing and file movement
### Seam C: move swarm membership side effects out of session lifecycle code
Today subscribe/resume/clear paths do both session and swarm work.
That should become:
- session service: attach/resume/rename session
- swarm service: join/update/leave member state
- client service: orchestrate the sequence for a request
This is likely the most important semantic seam for future maintainability.
Why this is safe:
- it clarifies ownership without changing the shared-server model
- it removes the hardest cross-domain coupling first
### Seam D: make maintenance loops call service APIs only
`monitor_bus()`, reload orchestration, idle timeout, and background-task wakeup should stop mutating shared maps directly.
They should call:
- `session_service.queue_soft_interrupt(...)`
- `session_service.fanout_session_event(...)`
- `swarm_service.record_file_touch(...)`
- `swarm_service.broadcast_status(...)`
- `swarm_service.detect_conflicts(...)`
Why this is safe:
- behavior stays the same
- background logic becomes testable in isolation
- future refactors no longer require editing `server.rs`
### Seam E: make debug consume snapshots, not storage
The debug stack currently knows too much about internal maps.
Introduce service snapshot methods so debug code reads pre-shaped data:
- `session_service.snapshot_sessions()`
- `client_service.snapshot_connections()`
- `swarm_service.snapshot_state()`
- `maintenance_service.snapshot_runtime_health()`
Why this is safe:
- debug stays powerful
- domain internals become easier to change
- read-only inspection stops blocking storage changes
## First Safe Moves
These are the first changes I would recommend landing in order.
### Move 1: docs and ownership rules
Land this plan and treat it as the contract for future refactors.
**Why first:** it prevents accidental partial extractions that worsen coupling.
### Move 2: introduce service handle structs with zero behavior change
Add thin wrappers such as:
- `SessionServiceHandle`
- `ClientServiceHandle`
- `SwarmServiceHandle`
- `DebugServiceHandle`
- `MaintenanceServiceHandle`
Initially these can just wrap the current `Arc` fields.
No logic movement is required yet.
**Payoff:** stops the spread of 20+ argument lists.
### Move 3: change `ServerRuntime` to hold service handles, not raw maps
`runtime.rs` is the cleanest place to narrow dependencies because it already acts as the server’s execution runtime.
**Payoff:** connection accept code no longer needs to know the storage layout of every subsystem.
### Move 4: change `handle_client()` and `handle_debug_client()` signatures
Replace wide argument lists with a few typed contexts:
- `ClientRequestContext`
- `DebugRequestContext`
- service handles
**Payoff:** largest readability win with limited behavioral risk.
### Move 5: extract swarm membership orchestration from `client_session.rs`
Create explicit swarm membership methods and have client/session flows call them.
**Payoff:** this is the first real domain split and removes one of the biggest architecture knots.
### Move 6: move `monitor_bus()` behind the swarm/session API boundary
Keep behavior, but stop direct map access from the maintenance loop.
**Payoff:** background infrastructure becomes modular and easier to test.
## Moves To Avoid Early
Avoid these until the service-handle layer exists:
- splitting into separate processes
- creating new crates for each service
- introducing async traits for every domain call
- changing the on-the-wire protocol
- changing session persistence format
- merging debug and normal sockets into one transport path as part of the refactor
These are higher-risk and do not solve the present problem as directly as state/API narrowing.
## Suggested File Landing Plan
### Phase 1: no behavior change
- add service handle types
- make `Server` store those handles or construct them centrally
- thread handles through `runtime.rs`
- narrow `handle_client()` and `handle_debug_client()` inputs
### Phase 2: move ownership boundaries
- move session delivery helpers under session service
- move swarm membership/status/channel/plan mutation fully under swarm service
- move debug readers to service snapshots
- move maintenance loops to service APIs
### Phase 3: clean module layout
Possible end-state layout:
```text
src/server/
bootstrap.rs # current server.rs bootstrap pieces
runtime.rs # accept loops and transport runtime
services/
session.rs
client.rs
swarm.rs
debug.rs
maintenance.rs
session/
actions.rs
lifecycle.rs
provider.rs
delivery.rs
swarm/
comm.rs
plan.rs
control.rs
sync.rs
state.rs
debug/
router.rs
jobs.rs
snapshots.rs
testers.rs
maintenance/
reload.rs
bus.rs
idle.rs
memory.rs
registry.rs
```
This can be reached gradually. It does not need to happen in one PR.
## Decision Record
### Recommended first code extraction
If one tiny extraction is desired after docs, the safest one is:
- introduce **service handle structs only**, with no behavior change
That is the highest-leverage low-risk move because it narrows dependency surfaces immediately and creates a place to move methods later.
### Recommended non-goal for now
Do not split the server into separate OS services. The current architecture benefits from shared MCP pool, shared embedding lifecycle, shared reload handling, and shared in-memory coordination. The code should first be made modular **inside** the existing process.
docs/SOFT_INTERRUPT.md
+372
View File
@@ -0,0 +1,372 @@
# Soft Interrupt: Seamless Message Injection
## Overview
Soft interrupt allows users to inject messages into an ongoing AI conversation without cancelling the current generation. Instead of the disruptive cancel-and-restart flow, messages are queued and naturally incorporated at safe points where the model provider connection is idle.
## Current Behavior (Hard Interrupt)
```
User types message during AI processing
│
▼
ToolDone event
│
▼
remote.cancel() ← Cancels current generation
│
▼
Wait for Done event
│
▼
Send user message as new request
│
▼
AI restarts fresh
```
**Problems:**
- Loses any partial work the AI was doing
- Delay while cancellation completes
- Full context re-send on new API call
- Jarring user experience
## New Behavior (Soft Interrupt)
```
User types message during AI processing
│
▼
Message stored in soft_interrupt queue
│
▼
AI continues processing...
│
▼
Safe injection point reached
│
▼
Message appended to conversation history
│
▼
AI naturally sees it on next loop iteration
```
**Benefits:**
- No cancellation, no lost work
- No delay
- AI naturally incorporates user input
- Smooth user experience
## Safe Injection Points
The key constraint is: **we can only inject when not actively streaming from the model provider**. The agent loop has several natural pause points:
### Agent Loop Structure (src/agent.rs)
```rust
loop {
// 1. Build messages and call provider.stream()
// === PROVIDER OWNS THE CONNECTION HERE ===
// Stream events: TextDelta, ToolStart, ToolInput, ToolUseEnd
// 2. Stream ends
// 3. Add assistant message to history
// (MUST happen before injection to preserve cache and conversation order)
// 4. Check if tool calls exist
if tool_calls.is_empty() {
// ═══════════════════════════════════════════════
// ✅ INJECTION POINT B: No tools, turn complete
// ═══════════════════════════════════════════════
break;
}
// 5. Execute tools and add tool_results
for tc in tool_calls {
// Execute single tool...
// Add result to history...
// ═══════════════════════════════════════════════
// ✅ INJECTION POINT C: Between tool executions
// (only for urgent aborts - must add skipped tool_results first)
// ═══════════════════════════════════════════════
}
// ═══════════════════════════════════════════════
// ✅ INJECTION POINT D: All tools done, before next API call
// ═══════════════════════════════════════════════
// Loop continues → next provider.stream() call
}
```
### Critical API Constraint
**The Anthropic API requires that every `tool_use` block must be immediately followed by
its corresponding `tool_result` block.** No user text messages can be injected between
a `tool_use` and its `tool_result`.
This means we CANNOT inject messages:
- After the assistant message with tool_use blocks
- Before all tool_results have been added
### Injection Point Details
| Point | Location | Timing | Use Case |
|-------|----------|--------|----------|
| **B** | Turn complete | No tools requested | Safe: no tool_use blocks to pair |
| **C** | Inside tool loop | Urgent abort only | Must add stub tool_results first |
| **D** | After all tools | Before next API call | **Default**: safest point for injection |
**Important**: We do NOT inject between tools for non-urgent interrupts. Doing so would
place user text between tool_results, which could violate API constraints. All non-urgent
injection is deferred to Point D.
### Point B: Turn Complete (No Tools)
```
Timeline:
Provider: TextDelta... [stream ends, no tool calls]
Agent: ──► INJECT HERE ◄──
Agent: Would exit loop, but instead continues with user message
AI sees: "I finished my response, user has follow-up"
```
**Best for:** Quick follow-ups when AI is just responding with text.
### Point C: Between Tools
```
Timeline:
Agent: Execute tool 1 → result 1
Agent: ──► INJECT HERE ◄──
Agent: Execute tool 2 → result 2 (or skip if user said "stop")
Agent: Next API call
AI sees: "Tool 1 result, user interjection, tool 2 result (or skip message)"
```
**Best for:**
- Urgent abort: "wait, don't do the other tools"
- Mid-execution guidance: "for the next file, also check X"
### Point D: After All Tools
```
Timeline:
Agent: Execute all tools → all results collected
Agent: ──► INJECT HERE ◄──
Agent: Next API call includes: [all tool results] + [user message]
AI sees: "All my tools completed, and user added context"
```
**Best for:** Default behavior. Cleanest, most predictable.
## Implementation
### Protocol Changes
Add new request type for soft interrupt:
```rust
// src/protocol.rs
#[serde(rename = "soft_interrupt")]
SoftInterrupt {
id: u64,
content: String,
/// If true, can abort remaining tools at point C
urgent: bool,
}
```
### Agent Changes
Add soft interrupt queue and check at each injection point:
```rust
// src/agent.rs
pub struct Agent {
// ... existing fields
soft_interrupt_queue: Vec<SoftInterruptMessage>,
}
struct SoftInterruptMessage {
content: String,
urgent: bool,
}
impl Agent {
/// Check and inject any pending soft interrupt messages
fn inject_soft_interrupts(&mut self) -> Option<String> {
if self.soft_interrupt_queue.is_empty() {
return None;
}
let messages: Vec<String> = self.soft_interrupt_queue
.drain(..)
.map(|m| m.content)
.collect();
let combined = messages.join("\n\n");
// Add as user message to conversation
self.add_message(Role::User, vec![ContentBlock::Text {
text: combined.clone(),
cache_control: None,
}]);
self.session.save().ok();
Some(combined)
}
/// Check for urgent interrupt that should abort remaining tools
fn has_urgent_interrupt(&self) -> bool {
self.soft_interrupt_queue.iter().any(|m| m.urgent)
}
}
```
### Injection Point Implementation
```rust
// In run_turn_streaming / run_turn_streaming_mpsc
loop {
// ... stream from provider ...
// ... add assistant message to history ...
// NOTE: We CANNOT inject here if there are tool calls!
// The API requires tool_use → tool_result with no intervening messages.
if tool_calls.is_empty() {
// Point B: No tools, turn complete - safe to inject
if let Some(msg) = self.inject_soft_interrupts() {
let _ = event_tx.send(ServerEvent::SoftInterruptInjected {
content: msg,
point: "B".to_string(),
});
// Don't break - continue loop to process the injected message
continue;
}
break;
}
// ... tool execution loop ...
for (i, tc) in tool_calls.iter().enumerate() {
// Check for urgent abort before each tool (except first)
if i > 0 && self.has_urgent_interrupt() {
// Point C: Urgent abort - MUST add skipped tool_results first
for skipped in &tool_calls[i..] {
self.add_message(Role::User, vec![ContentBlock::ToolResult {
tool_use_id: skipped.id.clone(),
content: "[Skipped: user interrupted]".to_string(),
is_error: Some(true),
}]);
}
// Now safe to inject user message
if let Some(msg) = self.inject_soft_interrupts() {
let _ = event_tx.send(ServerEvent::SoftInterruptInjected {
content: msg,
point: "C".to_string(),
});
}
break;
}
// ... execute tool and add tool_result ...
}
// Point D: After all tools done, safe to inject
if let Some(msg) = self.inject_soft_interrupts() {
let _ = event_tx.send(ServerEvent::SoftInterruptInjected {
content: msg,
point: "D".to_string(),
});
}
}
```
### TUI Changes
Update interleave handling to use soft interrupt:
```rust
// src/tui/app.rs
// Instead of:
// remote.cancel() → wait → send message
// Do:
// remote.soft_interrupt(message, urgent)
// The message will be injected at the next safe point
// No cancellation, no waiting
```
### Server Event for Feedback
```rust
// src/protocol.rs
ServerEvent::SoftInterruptInjected {
content: String,
point: String, // "A", "B", "C", or "D"
}
```
This allows the TUI to show feedback like "Message injected after tool X".
## User Experience
### Default Mode (queue_mode = false)
```
User presses Enter during processing:
→ Message queued for soft interrupt
→ Status shows: "⏳ Will inject at next safe point"
→ AI continues working...
→ [ToolDone] → Message injected
→ Status shows: "✓ Message injected"
→ AI naturally incorporates it
```
### Urgent Mode (Shift+Enter or special flag)
```
User presses Shift+Enter during processing:
→ Message queued as urgent soft interrupt
→ Status shows: "⚡ Will inject ASAP (may skip tools)"
→ AI continues current tool...
→ [ToolDone] → Remaining tools skipped, message injected
→ AI sees: tool 1 result + "user interrupted, skipped tools 2-3" + user message
```
## Comparison
| Aspect | Hard Interrupt (current) | Soft Interrupt (new) |
|--------|-------------------------|---------------------|
| Cancels generation | Yes | No |
| Loses partial work | Yes | No |
| Delay | Yes (wait for cancel) | No |
| API calls | Wastes partial call | Efficient |
| User experience | Jarring | Smooth |
| Complexity | Simple | Moderate |
## Edge Cases
1. **Multiple soft interrupts**: Combine into single message with `\n\n` separator
2. **Soft interrupt during text-only response**: Inject at Point B, continue loop
3. **Provider handles tools internally** (Claude CLI): Still works, injection happens in our loop
4. **Urgent interrupt with no tools**: Treated as normal (nothing to skip)
5. **Stream error**: Clear soft interrupt queue, report error normally
## Testing
1. Send message while AI is streaming text (no tools) → should inject at Point B
2. Send message while AI is executing tools → should inject at Point D (after all tools)
3. Send urgent message while multiple tools queued → should skip remaining tools at Point C
4. Send multiple messages rapidly → should combine into one injection
5. Verify no API errors about tool_use/tool_result pairing
docs/SPAWN_HOOK.md
+184
View File
@@ -0,0 +1,184 @@
# Spawn Hook: External Control of Headed Session Spawns
jcode opens new terminal windows in several flows: swarm agent spawning
(`swarm spawn` with `spawn_mode=visible`), resume-in-new-terminal, self-dev
sessions, restart restores, and jade relay launches. By default jcode detects
an installed terminal emulator (kitty, wezterm, alacritty, gnome-terminal, ...)
and opens a new OS window.
The **spawn hook** lets an external program take over this spawn so it can
decide *where and how* the session appears: a tmux pane, a kitty tab, a zellij
pane, a tab in a wrapper app like herd, a specific monitor/workspace, etc.
## Configuration
```toml
# ~/.jcode/config.toml
[terminal]
spawn_hook = "tmux new-window"
```
Or per-environment:
```bash
export JCODE_SPAWN_HOOK="tmux new-window"
# An empty value disables a config-file hook:
export JCODE_SPAWN_HOOK=
```
Env always wins over the config file.
## Contract
When a headed spawn happens and a hook is configured, jcode runs:
```
<spawn_hook> <jcode-binary> <args...>
```
- The hook command is parsed shell-style (quotes and backslash escapes work),
but it is executed directly, not through a shell.
- The jcode binary and its full argument list are appended as extra argv
entries (the familiar `$TERMINAL -e <cmd>` convention).
- The hook's working directory is the session working directory.
- The hook process is detached; jcode does not wait for it.
- If the hook fails to start (binary missing, parse error), jcode logs a
warning and falls back to its built-in terminal detection.
### Metadata environment
The hook (and any terminal spawned by the built-in fallback) receives:
| Variable | Meaning |
| --- | --- |
| `JCODE_SPAWN_KIND` | Why the spawn happened: `swarm-agent`, `resume`, `selfdev`, `restart`, `jade-relay` |
| `JCODE_SPAWN_SESSION_ID` | The jcode session the window will run |
| `JCODE_SPAWN_TITLE` | Suggested window/tab title (includes session icon + name) |
| `JCODE_SPAWN_CWD` | Session working directory |
| `JCODE_SPAWN_PROGRAM` | Path of the jcode binary to execute |
| `JCODE_SPAWN_COMMAND` | Full command line, shell-escaped, for hooks that take one shell string |
| `JCODE_SPAWN_SWARM_ID` | (swarm spawns) The swarm the agent joins |
| `JCODE_SPAWN_COORDINATOR_SESSION_ID` | (swarm spawns) The coordinator session that requested the spawn |
| `JCODE_FRESH_SPAWN` | `1` when the spawn is a fresh window handoff |
### Client terminal environment (multi-terminal routing)
The jcode server process is long-lived: it captures terminal-identifying env
vars (`ZELLIJ_SESSION_NAME`, `TMUX`, `DISPLAY`, `KITTY_WINDOW_ID`, ...) once at
startup. When you later open a *new* terminal/tmux/zellij session and connect a
client to the same server, the server's copies are stale, so a spawn hook run by
the server would otherwise target the *old* terminal.
To fix this (see issue #405), each connecting client snapshots its own
terminal-identifying env and sends it to the server. When a spawn hook runs, the
server re-exports the requesting client's values so the hook follows the
terminal the user is actually attached to:
- The native variable (e.g. `ZELLIJ_SESSION_NAME`) is overridden with the
client's value, so hooks that read it directly target the right session.
- A `JCODE_CLIENT_<NAME>` alias (e.g. `JCODE_CLIENT_ZELLIJ_SESSION_NAME`,
`JCODE_CLIENT_TMUX`, `JCODE_CLIENT_DISPLAY`) is also exported so a hook can
explicitly distinguish the client's terminal from the server's.
Covered keys include the terminal multiplexers (zellij, tmux, screen), terminal
emulators (kitty, wezterm, ghostty, alacritty, iTerm, Windows Terminal,
handterm), and the display server (`DISPLAY`, `WAYLAND_DISPLAY`). Only vars that
the client actually has set are forwarded.
## Examples
### tmux: one window per agent
```toml
[terminal]
spawn_hook = "tmux new-window"
```
`tmux new-window <jcode> --resume ses_x` runs the command in a new window of
the current tmux server. For panes instead:
```toml
[terminal]
spawn_hook = "tmux split-window -h"
```
### kitty: one tab per agent (remote control)
```toml
[terminal]
spawn_hook = "kitty @ --to unix:/tmp/kitty.sock launch --type=tab --"
```
### Custom router script
For full control (placement, titles, swarm vs resume routing), point the hook
at a script:
```toml
[terminal]
spawn_hook = "~/bin/jcode-spawn-router"
```
```bash
#!/usr/bin/env bash
# ~/bin/jcode-spawn-router
# argv: the jcode command to run ("$@"). Env: JCODE_SPAWN_* metadata.
case "$JCODE_SPAWN_KIND" in
swarm-agent)
# Swarm workers as tmux panes in a window named after the swarm.
tmux new-window -n "swarm:${JCODE_SPAWN_SWARM_ID:0:8}" "$@" 2>/dev/null \
|| tmux split-window "$@"
;;
*)
# Everything else as a normal terminal window.
kitty --title "$JCODE_SPAWN_TITLE" -e "$@" &
;;
esac
```
A hook that exits non-zero after launching nothing will NOT trigger the
built-in fallback (jcode only falls back when the hook process cannot be
started), so a router script should handle its own fallback like the example
above.
### Single-shell-string consumers
Some launchers want one shell command string instead of argv. Use
`$JCODE_SPAWN_COMMAND`:
```bash
#!/usr/bin/env bash
zellij action new-pane -- bash -lc "$JCODE_SPAWN_COMMAND"
```
## Programmatic discovery
Programs that wrap jcode (e.g. herd-style session managers) can set
`JCODE_SPAWN_HOOK` in the environment of the `jcode` server process they
launch. Every headed spawn the server performs, including swarm agents
requested by coordinators over the socket protocol, will then route through
the wrapper's hook.
## Focus hook
When jcode wants to bring an existing session window to the foreground (e.g.
after launching a self-dev window), it normally does a best-effort
wmctrl/xdotool title search on X11. That doesn't work under Wayland or inside
multiplexers, and a wrapper that owns placement should also own focus:
```toml
[terminal]
spawn_hook = "tmux new-window"
focus_hook = "~/bin/jcode-focus" # env: JCODE_FOCUS_SESSION_ID, JCODE_FOCUS_TITLE
```
```bash
#!/usr/bin/env bash
# ~/bin/jcode-focus
tmux select-window -t "$(tmux list-windows -F '#{window_id} #{window_name}' \
| grep -F "$JCODE_FOCUS_TITLE" | head -1 | cut -d' ' -f1)"
```
Env override: `JCODE_FOCUS_HOOK` (empty value disables a config-file hook).
If the hook fails to start, jcode falls back to the built-in focus path.
docs/SPONSORED_DISCOVERY_SPONSOR_ONBOARDING.md
+304
View File
@@ -0,0 +1,304 @@
# Sponsored discovery sponsor onboarding
This runbook is the source of truth for adding a tool sponsor to jcode's
`discover_tools` catalog. It covers product approval, catalog data, service
behavior, client validation, rollout, and rollback.
Sponsors pay for placement in a discovery category, not for recommendations.
The agent must still choose the best tool for the user's task and may choose a
non-sponsored alternative. Do not onboard a sponsor whose agreement requires
preferential recommendations, hidden placement, or weaker user safeguards.
## What usually changes
Adding a sponsor to an existing category is a discovery-service catalog change.
It should not require a jcode release.
A jcode code change is required when the sponsor needs:
- a new category in `DISCOVERY_CATEGORIES`;
- a response field the current client does not support;
- a new setup or provenance mechanism; or
- different disclosure, privacy, telemetry, or safety behavior.
The hosted catalog and discovery service are not stored in this repository.
Coordinate that change with the service owner. The client-side contract is in:
- `crates/jcode-app-core/src/tool/discover.rs`;
- `crates/jcode-base/src/sponsors.rs`;
- `crates/jcode-base/src/sponsors/provenance.rs`;
- `crates/jcode-tui/src/tui/app/sponsor_disclosure.rs`; and
- `TELEMETRY.md`.
## 1. Intake and approval
Record the following before editing the catalog:
- sponsor's legal and public product names;
- canonical tool name and URL;
- requested category;
- concise, factual product description;
- supported platforms and prerequisites;
- exact installation or connection steps;
- whether setup uses MCP and, if so, its exact command and arguments;
- permissions, credentials, network access, and data the tool receives;
- pricing, trial, account, payment, and other consequential requirements;
- support contact, technical owner, start date, and end or review date; and
- rollback contact and maximum acceptable disable time.
The discovery owner must verify that:
1. The product is real, reachable, and relevant to its category.
2. The description is factual rather than comparative or promotional.
3. Setup instructions use an official, versioned, or otherwise auditable
distribution channel where possible.
4. No credential, API key, referral secret, user identifier, or environment
value is embedded in catalog data.
5. The setup does not bypass jcode's confirmation requirements. Signups,
payments, destructive operations, and other consequential actions still
require the normal user confirmation and sponsorship disclosure.
6. The commercial agreement buys discoverability only. Editorial ranking and
agent selection remain independent.
Reject or pause onboarding if any item cannot be verified.
## 2. Choose or add a category
Current categories are defined by `DISCOVERY_CATEGORIES` in
`crates/jcode-base/src/sponsors.rs`. Category values are lowercase slugs.
Use an existing category whenever it accurately describes the capability. To
add a category:
1. Add its slug to `DISCOVERY_CATEGORIES`.
2. Add the same value to the discovery service's category allowlist.
3. Update any public category documentation.
4. Run the client tests listed below.
5. Ship the jcode change before publishing entries that rely on the category.
Do not create a sponsor-specific category. Categories describe user needs, not
vendors.
## 3. Create the catalog entry
Use a stable, lowercase tool slug for `name`. A complete internal catalog entry
should contain enough data for both API phases:
```json
{
"name": "example-tool",
"category": "databases",
"blurb": "Managed PostgreSQL with branching and connection pooling",
"url": "https://example.com/product",
"setup": "Run `npx -y example-tool-mcp@1.2.3`, then connect the resulting MCP server.",
"mcp": {
"command": "npx",
"args": ["-y", "example-tool-mcp@1.2.3"]
},
"active": true
}
```
Client-visible fields are:
| Field | Required | Rules |
|-------|----------|-------|
| `name` | Yes | Stable canonical slug. It is also the sponsor key used by coarse MCP usage metering. |
| `blurb` | Yes | Short, factual capability description. Do not claim it is "best" or imply endorsement. |
| `url` | Recommended | HTTPS product or setup page controlled by the vendor. |
| `setup` | Select phase | Complete instructions returned only after the agent selects the tool. Never include secrets. |
| `mcp.command` | For MCP provenance | Executable used by `mcp connect`. Must exactly match the eventual connection command. |
| `mcp.args` | For MCP provenance | Ordered string array. Must exactly match the eventual connection arguments. |
The service may keep operational fields such as `category`, `active`, campaign
dates, or ordering metadata, but it must not expose private commercial data to
the client.
### Setup safety review
Run the setup in a clean test environment before publishing it. Review the
package owner, source repository, install scripts, transitive behavior, required
permissions, and credential flow. Prefer a pinned version in catalog setup
instructions. If an intentionally floating version is used, document who owns
continuous monitoring and emergency disablement.
For MCP entries, `mcp.command` and `mcp.args` are security- and
measurement-sensitive. jcode records discovery provenance only when a later MCP
connection exactly matches both values. A prose `setup` string alone does not
enable provenance tagging or coarse usage metering.
## 4. Implement the two API phases
The default client sends `GET https://api.jcode.sh/v1/discovery` with a
three-second timeout and a 64 KiB maximum response. It sends a
`User-Agent: jcode/<version>` header and a random
`x-jcode-discovery-request-id` correlation header.
The request query parameters are:
| Parameter | Phase | Meaning |
|-----------|-------|---------|
| `category` | Both | Required category slug. |
| `q` | Both | Optional short capability query. |
| `reason` | Both | Agent-stated need or selection rationale. Store and handle it as potentially sensitive text even though the client instructs the agent not to include private data. |
| `tool` | Select | Canonical name chosen from a previous browse response. Its presence selects the second phase. |
### Browse response
Browse returns eligible tools without setup instructions or MCP launch data:
```json
{
"tools": [
{
"name": "example-tool",
"blurb": "Managed PostgreSQL with branching and connection pooling",
"url": "https://example.com/product"
}
]
}
```
Return `{"tools": []}` when no entry is eligible. Do not return `setup` or
`mcp` during browse. The two-phase design requires a specific selection and
reason before setup is revealed.
### Select response
Select looks up `tool` within `category` and returns one canonical entry:
```json
{
"tool": {
"name": "example-tool",
"blurb": "Managed PostgreSQL with branching and connection pooling",
"url": "https://example.com/product",
"setup": "Run `npx -y example-tool-mcp@1.2.3`, then connect the resulting MCP server.",
"mcp": {
"command": "npx",
"args": ["-y", "example-tool-mcp@1.2.3"]
}
}
}
```
Use a non-2xx response for an unknown category, unknown tool, invalid request,
or service failure. Never silently substitute another sponsor. Keep total JSON
below 64 KiB and avoid redirects because they make behavior harder to audit.
## 5. Validate before production
First validate the service directly. Use generic test text because `q` and
`reason` are sent to and may be stored by the discovery service.
```bash
DISCOVERY_URL=https://staging.example.com/v1/discovery
curl --fail-with-body --get "$DISCOVERY_URL" \
--data-urlencode 'category=databases' \
--data-urlencode 'q=managed postgres for a test application' \
--data-urlencode 'reason=validate the databases discovery listing'
curl --fail-with-body --get "$DISCOVERY_URL" \
--data-urlencode 'category=databases' \
--data-urlencode 'tool=example-tool' \
--data-urlencode 'reason=selected for staging validation after reviewing the listed database options'
```
Verify all of the following:
- browse includes the sponsor exactly once in the intended category;
- browse omits `setup`, `mcp`, credentials, and private campaign metadata;
- select returns the same canonical `name`, plus reviewed setup instructions;
- an unknown tool and category fail rather than returning a default entry;
- response bodies remain under 64 KiB;
- logged query and reason text follows the service's retention and access policy;
- the request ID appears in service logs and can be correlated for reliability
debugging without a persistent user identifier; and
- disabling the catalog entry removes it from browse and prevents selection.
Then validate through jcode by pointing a test config at staging:
```toml
[sponsors]
enabled = true
endpoint = "https://staging.example.com/v1/discovery"
```
In a disposable jcode session, browse the category, select the sponsor, and, if
applicable, connect the advertised MCP server. Confirm:
- the first discovery use displays `(sponsored discovery)`;
- the browse output says placement does not imply preference;
- setup appears only after selection;
- consequential next actions still request confirmation and mention the
sponsorship;
- an MCP connection using the exact structured command and arguments displays
discovery provenance; and
- the tool works without requiring undisclosed permissions or data.
For client-side changes, run at minimum:
```bash
cargo test -p jcode-app-core tool::discover
cargo test -p jcode-base sponsors
cargo test -p jcode-base discovery_provenance
cargo test -p jcode-tui sponsor_disclosure
cargo check -p jcode
```
If Cargo's filters change, run the containing crate's tests instead of skipping
the check.
## 6. Roll out and monitor
1. Publish to staging and complete the validation checklist.
2. Obtain sign-off from the discovery owner and the setup security reviewer.
3. Publish the entry disabled or outside its campaign window, if supported.
4. Enable it in production without changing unrelated catalog entries.
5. Repeat one browse and one select request against production.
6. Monitor discovery success/failure rates, response size and latency, browse to
select behavior, and coarse provenance usage. See `TELEMETRY.md` for the
client telemetry boundary.
7. Re-review setup and destination URLs whenever the vendor changes its package,
ownership, permissions, or authentication flow.
Do not use selection or usage counts to make the agent prefer a sponsor. Those
signals are for reliability, aggregate reporting, and catalog quality.
## 7. Roll back
The primary rollback is to disable or remove the sponsor's service-side catalog
entry. This must stop both browse placement and direct select lookup. Use it for
security concerns, misleading copy, broken setup, expired agreements, service
abuse, or vendor request.
After disabling:
1. Verify browse no longer returns the entry.
2. Verify direct selection of its name fails.
3. Preserve only the logs and aggregate records required by the applicable
retention policy.
4. Notify the technical and commercial owners.
5. Open a post-incident issue if users could have installed unsafe or incorrect
software.
A client release is necessary only if catalog disablement cannot contain the
problem, for example a compromised category-wide response or a flaw in client
setup handling.
## Definition of done
A sponsor is onboarded only when every box is checked:
- [ ] Intake, ownership, campaign dates, and rollback contact are recorded.
- [ ] Placement-only policy and independent recommendations are accepted.
- [ ] Category and factual copy are approved.
- [ ] Setup and destination URL pass security review in a clean environment.
- [ ] Browse and select responses match the documented schemas.
- [ ] Browse does not expose setup or MCP launch data.
- [ ] MCP command and arguments exactly match the tested connection, if used.
- [ ] Staging jcode validation passes, including disclosure and confirmation.
- [ ] Unknown and disabled entries fail closed.
- [ ] Production smoke tests pass and monitoring has an owner.
- [ ] Rollback has been tested or demonstrated by disabling the staging entry.
docs/SWARM_ARCHITECTURE.md
+313
View File
@@ -0,0 +1,313 @@
# Swarm Architecture
Status: Largely implemented (see `SWARM_TASK_GRAPH.md` for the DAG-first model
that supersedes the agent-first framing here; its staged comm migration is in
progress)
This document captures the swarm coordination design. It describes how agents
coordinate, plan, communicate, and integrate work with optional git worktrees.
## Goals
- Parallel work across many agents without locks.
- A comprehensive initial plan, but allowed to evolve as work progresses.
- Plan distribution is out-of-band (not stored in the repo).
- Swarm runtime state survives reloads and crash recovery via daemon snapshots.
- Explicit coordination via broadcast updates, DMs, and channels.
- Optional git worktrees used only when they make sense.
- Integration handled by worktree managers, not the coordinator.
## Roles
### Recursive spawning (unbounded-depth tree)
Spawning is recursive. Any swarm member can spawn child agents, and those
children can spawn their own children, forming a spawn tree. There is no depth
cap: growth is bounded only by the total swarm member cap. The root session
that first spawns in a repo is depth 0.
The spawn/parent edge is encoded by `report_back_to_session_id`: a child spawned
by `P` reports back to `P`. Walking that chain reconstructs ancestry and depth,
so each agent "owns" the subtree it spawned. An agent may stop any agent in its
own subtree (itself or a transitive descendant); `force=true` is still required
to stop sessions outside the requester's subtree (e.g. user-created peers).
When a mid-tree member leaves (stop, crash, disconnect, feature-off), its direct
children are reparented rather than orphaned: they attach to their live
grandparent, falling back to the current coordinator, else they become roots.
Session renames (resume) rewrite children's report-back edges to the new id.
This keeps ownership, stop permissions, subtree broadcast scope, and completion
report-back coherent across member churn.
The single per-swarm "coordinator" slot still exists, but only for shared,
swarm-level plan operations (propose/approve/assign/task-control on the one
shared plan). Only a root session claims that slot, and only when it is empty or
stale. A live coordinator no longer blocks anyone else from spawning.
Nested owners coordinate their own subtree through spawn prompts, direct
messages, and stop, not through the shared plan. Plan/task operations
(`assign_task`, `assign_next`, `task_control`, `approve_plan`, `reject_plan`)
deliberately stay gated to the root coordinator because there is exactly one
`VersionedPlan` per `swarm_id`; allowing multiple coordinators to mutate it
concurrently would make the shared plan incoherent.
### Coordinator
- Owns the shared swarm-level plan: creates it, assigns scopes, approves updates.
- Reviews plan update proposals and broadcasts approved updates.
- Can issue plan updates directly when it discovers a plan issue.
- Decides if a git worktree is needed and groups agents per worktree.
- Holds the per-swarm coordinator slot for shared plan operations
(propose/approve/assign/task-control). This is a root-session role, not a
prerequisite for spawning.
- Does not perform merges or integration.
### Worktree Manager
- Owns a single worktree scope.
- Knows the full plan and the worktree scope.
- Coordinates work inside that worktree.
- Responsible for integration when that worktree scope is done.
### Agents
- Execute tasks in parallel.
- Receive the full plan plus their scoped instructions on spawn.
- Propose plan updates when they discover issues or new requirements.
- Coordinate directly with other agents via DM or channels.
- Emit lifecycle events when they start, finish, or stop unexpectedly.
- May spawn their own child agents (no depth cap; bounded only by the total
swarm member cap) and stop any
agent in the subtree they spawned. Stopping agents outside their own subtree
still requires `force=true`.
## Agent Lifecycle States
- spawned: session created, not yet ready.
- ready: plan and scope received, waiting for work.
- running: actively executing a task or tool.
- blocked: cannot proceed (dependency, conflict, or missing info).
- completed: assigned scope done, waiting for new assignment.
- failed: unrecoverable error, needs coordinator decision.
- stopped: intentionally shut down by coordinator.
- crashed: unexpected exit (no clean shutdown).
## Agent Lifecycle Notifications
- Each agent emits a completion event when its assigned scope is done.
- Each agent emits a stop event when it cannot continue or exits unexpectedly.
- The coordinator receives these events and decides next steps (respawn, rescope,
shutdown, or mark complete).
- Lifecycle updates drive the swarm info widget status indicators.
## Completion Report Policy
- Spawned or assigned agents owned by a coordinator (`report_back_to_session_id`) must
finish each prompted work turn with a useful final assistant response. The server
automatically forwards that final response to the owning coordinator as the
completion report.
- A completion report should include outcome/status, changes or findings, validation
performed, and blockers or follow-ups. It should not be just `done`, a lifecycle
status change, or a tool transcript.
- Reports are required for spawn prompts, assigned plan tasks, and explicit
start/wake/resume/retry task-control runs. If a worker fails before producing a
final response, the coordinator still receives the failure lifecycle notification.
- Reports are not required for idle spawn-without-prompt sessions, user-created peers
that have no report-back owner, ordinary status broadcasts while work is still
running, or intentional cleanup/stop of an idle worker.
- Agents should avoid sending a separate final-report DM unless they need interactive
coordination before finishing; the automatic forwarded report is the default path.
## User Interaction
- The user primarily interacts with the coordinator.
- Other agents do not surface directly to the user unless the coordinator routes
updates or requests.
## Plan Distribution and Updates
- Swarm plan is a server-level object scoped by `swarm_id` (not a session todo list).
- Session todos remain private to each session and are not used as swarm plan storage.
- Plan v1 is created/owned by the coordinator.
- Plan updates are proposed by agents and must be reviewed by the coordinator.
- Plan updates are propagated to plan participants, not every agent in the swarm.
- Plan participation is explicit (coordinator assignment/spawn policy or resync attach).
- The plan is not stored in a repo file.
- Agents can explicitly request plan attachment/resync when needed.
Plan update flow:
```mermaid
flowchart LR
Agent[Agent] -->|propose update| Coordinator
Coordinator -->|approve update| Plan[Swarm Plan]
Coordinator -->|direct update| Plan
Plan --> Participants[Plan Participants]
```
## Worktree Usage
- Worktrees are optional and used only when isolation helps (large refactors,
risky changes, or divergent dependencies).
- Most work should remain in the main workspace unless a worktree is justified.
- Many agents can share a single worktree.
- Each worktree has a Worktree Manager who owns integration.
- Each worktree is assigned a logical `swarm_id` so communication, plan updates,
and UI views span all worktrees in the same swarm.
Worktree grouping:
```mermaid
flowchart TB
Coordinator --> Plan
Plan --> A1[Agent 1]
Plan --> A2[Agent 2]
Plan --> A3[Agent 3]
Plan --> A4[Agent 4]
Coordinator --> WTM1[Worktree Manager 1]
Coordinator --> WTM2[Worktree Manager 2]
WTM1 --> WT1[Worktree Group 1]
WT1 --> A1
WT1 --> A2
WTM2 --> WT2[Worktree Group 2]
WT2 --> A3
WT2 --> A4
```
Integration:
```mermaid
flowchart LR
WTM1 -->|integrate| Integration[Integration Branch]
WTM2 -->|integrate| Integration
Integration --> Main[Main Branch]
```
## Communication
Explicit agent-to-agent communication is required for coordination and conflict
resolution. The system supports:
- Direct messages (DMs) - the preferred exception channel
- Subtree broadcast (reaches only the sender's spawned subtree; the swarm
coordinator keeps whole-swarm reach as an escape hatch)
- Topic channels (group chats) - discouraged; prefer DMs and task-graph artifacts
- Shared context keys (set/read/append) - discouraged; prefer the repo and
typed node artifacts. Share notifications are subtree-scoped like broadcasts.
- Channel discovery and member inspection
All agents can send DMs and subtree broadcasts.
All inter-agent communication is delivered as notifications (DMs, channel messages,
broadcasts, plan updates, intent notices, and lifecycle events). Notifications are
queued as soft interrupts and injected into running agents at safe points, so
messages can be interleaved during a turn without starting a new turn.
Completed or idle agents do not resume automatically when notifications arrive.
They only resume when the coordinator assigns new work, explicitly starts or wakes an
assigned task, or respawns them. Recovery handoffs are explicit too: retry keeps the
same assignee, reassign moves work to another existing agent, replace swaps to a new
assignee after safe state checks, and salvage reassigns with preserved task-progress
context.
Status snapshot, summary read, and full context read are separate operations:
- Status snapshot: lock-free member metadata plus current processing/tool snapshot. This
must stay available even while the target agent is busy.
- Summary read: short activity feed (tool calls with intent, brief results, and
optionally exposed thoughts).
- Full context read: explicit, heavy read of an agent's full context and tool
outputs. This should be used sparingly to avoid context bloat.
Communication topology:
```mermaid
flowchart LR
A1[Agent 1] -->|DM| Comms[Comms Router]
A2[Agent 2] -->|DM| Comms
A3[Agent 3] -->|DM| Comms
A1 -->|channel| Comms
A2 -->|channel| Comms
A3 -->|swarm| Comms
Comms --> A1
Comms --> A2
Comms --> A3
A1 --> Summary[Summary Feed]
A2 --> Summary
A3 --> Summary
A1 --> Full[Full Context Store]
A2 --> Full
A3 --> Full
```
## UI (TUI)
Two real-time widgets accompany the swarm system: a swarm info widget and a plan
info widget. Both update continuously from event streams.
### Swarm info widget
- Graph view of agents, worktree managers, coordinator, and channels.
- Edges represent communication paths: DM, channel, and swarm broadcast.
- Nodes show status (idle, running, blocked) and current task or intent.
- Updates in real time based on communication events, lifecycle events, and tool intent events.
Swarm graph view:
```mermaid
flowchart LR
Coord[Coordinator] -->|broadcast| A1[Agent 1]
Coord -->|broadcast| A2[Agent 2]
A1 -->|DM| A2
A2 -->|channel:#parser| Chan[Channel]
A1 -->|channel:#parser| Chan
WTM[Worktree Manager] --> A1
WTM --> A2
```
### Plan info widget
- Graph view of the task DAG with dependencies.
- Nodes show owner, scope, and status (queued, running, running_stale, done, blocked, failed).
- Checkpoints are shown as node badges or subnodes.
- Coordinators can inspect durable per-task progress, including assignment metadata, heartbeat age, and last checkpoint summary.
- Progress is visible through completed node count, critical path status, and persisted checkpoint/heartbeat data after reloads.
- Updates in real time from plan broadcasts and task status events.
Plan graph view:
```mermaid
flowchart TB
T1[Define API] --> T2[Refactor Parser]
T1 --> T3[Update Tests]
T2 --> T4[Integrate]
T3 --> T4
```
## File Touch and Intent
- File touch notifications are used for conflict detection.
- An optional short `intent` field on tool calls is planned to provide a
preemptive summary of what a tool is trying to do.
- Intent should be brief and is used to build the summary activity feed.
## Conflict Handling (No Locks)
- The system is optimistic by default (no locks).
- Conflicts should prompt the involved agents to communicate directly.
- Coordination happens via DM or channel, not through the coordinator.
## Summary
This design emphasizes parallelism, explicit communication, and optional worktree
isolation. The coordinator is responsible for planning and plan updates; worktree
managers are responsible for integration; agents collaborate directly to resolve
conflicts.
docs/SWARM_TASK_GRAPH.md
+603
View File
@@ -0,0 +1,603 @@
# Swarm as a Task DAG (Design)
Status: Being implemented (supersedes the agent-first framing in
`SWARM_ARCHITECTURE.md`). The DAG engine, deep/light modes, gates, growth
mechanics, and comm migration steps 1-2 (artifact dataflow, subtree-scoped
broadcast) are live; channel/shared-context deprecation (steps 3-4) is pending.
This document captures the planned reframe of the swarm module from an
agent-centric model into a **task DAG (directed acyclic graph)**. The DAG becomes
the primary object; agents become fungible workers that execute, decompose, and
verify nodes. It records the architecture, the data model, the completion/coverage
guarantees, the bias budget, and the tool surface, based on the design discussion.
---
## 1. Motivation and core reframe
Today swarm is **agent-first**: you drive work by spawning agents and talking to
them (DMs, channels, roles), with a `VersionedPlan` of `PlanItem`s bolted on the
side. The dependency graph already exists under the hood (`PlanItem.blocked_by`
edges, `summarize_plan_graph`, `next_runnable_item_ids`, `run_plan`/`fill_slots`),
but it is an implementation detail. Coverage and thoroughness are left to whoever
happens to be driving.
The reframe makes the **task DAG the primary abstraction**:
- You declare a graph of tasks with dependency edges and per-node specs.
- The scheduler walks the DAG: a node becomes runnable when its dependencies
complete, is assigned to a worker (reuse-or-spawn), and on completion unblocks
its dependents automatically.
- Agents are workers pulled from a pool, not entities you micromanage.
- Coordinator / worktree-manager roles demote to **scheduler policy**, not
user-facing concepts.
```mermaid
flowchart LR
subgraph Now[Now: agent-first]
C[Coordinator] --> A1[Agent] --> P1[plan item]
end
subgraph Next[Reframe: DAG-first]
T1[Task A] --> T2[Task B]
T1 --> T3[Task C]
T2 --> T4[Task D]
T3 --> T4
W[(worker pool)] -.executes.-> T1
end
```
The existing `jcode-plan` graph code is the foundation; this is an evolution of
it, not a rewrite.
---
## 1a. Two modes: deep (comprehensive) vs light (fan-out)
The DAG engine runs in one of two **modes**. This is deliberately **one engine,
two presets**, not two separate systems. Both modes use the same DAG data model,
scheduler, dataflow-on-edges, and member-cap mechanism. The only difference is
whether the rigor machinery (mandatory decomposition + critique/verify gates +
recursion) is engaged, and how large the member cap is.
Key framing: **light mode is just deep mode with the structural pressures turned
off and a small cap.** Build them as a single engine with a mode knob; do not fork
the scheduler or dataflow.
- **Deep mode (comprehensive):** everything in this document. Goal is to leave no
nook unexplored. Recursive, self-deepening tree; decomposition is mandatory
(composite by default); a critique/verify gate is required before any node
closes; recursion is encouraged with no depth limit; the full typed handoff
schema is enforced, including `what_i_did_not_check`. Scales up to the 1000-agent
member cap. High cost and latency, used deliberately. Examples: "explore
multimonitor support in scrollwm", large/risky refactors.
- **Light mode (fan-out):** the cheaper preset for parallelizing work for speed and
a modest quality bump, without going extreme. Goal is just to run independent
units in parallel. Mostly flat (one level of fan-out); decomposition is optional
(agent's choice); the critique/verify gate is off, or at most a single optional
final check; recursion is discouraged/disabled; the handoff artifact is
lightweight and may be free-form. Small worker cap (e.g. 4-16). Low cost and
latency. This is essentially today's spawn-and-fan-out behavior kept cheap.
Examples: "run these 5 independent edits in parallel".
| Dimension | Deep (comprehensive) | Light (fan-out) |
| -------------------- | ------------------------------------------ | -------------------------------- |
| Goal | Leave no nook unexplored | Parallelize for speed/quality |
| Shape | Recursive, self-deepening tree | Mostly flat, one level of fan-out|
| Decomposition | Mandatory (composite by default) | Optional, agent's choice |
| Critique/verify gate | Required before any node closes | Off (or one optional final check)|
| Recursion | Encouraged, depth unbounded | Discouraged/disabled |
| Handoff artifact | Full typed schema, `what_i_did_not_check` | Lightweight, free-form ok |
| Member cap | up to 1000 agents | small (e.g. 4-16 workers) |
| Cost / latency | High, deliberate | Low, fast |
Shared across both modes: the DAG data model, the scheduler, dataflow-on-edges
(typed-or-light artifacts), and the member-cap mechanism (only the ceiling
differs). The rigor sections of this document (6, 7) describe **deep mode**; light
mode simply disables those gates.
---
## 2. Ownership tree over a dependency graph
The trap in "everyone edits one shared graph" is that there is a single shared
`VersionedPlan`; concurrent free-form edits make it incoherent, which is why plan
mutation is currently gated to one coordinator. The fix is to change **what a
mutation is**, not to add locks.
Model it as a **tree of ownership laid over a graph of dependencies**. The unit of
mutation is *expanding a node you own*, never editing arbitrary nodes.
- Writes are **partitioned by owner**: you only ever add children under your own
node, so two owners never write the same region. This removes the coordinator
bottleneck without locks while keeping one global graph.
- The graph stays a single **server-owned, versioned** source of truth (reuse
`VersionedPlan`), but mutations become **append-style ops** (add nodes, add
edges, complete node), validated server-side for acyclicity + ownership, instead
of last-write-wins blobs.
- New edges may only point at **already-existing upstream** nodes, which preserves
acyclicity by construction.
---
## 3. Node kinds: atomic vs composite
Every node has one of two fates when an agent picks it up. The status flips at
runtime, not at draft time. A node does **not** have to be declared composite up
front; any agent at any depth can choose to expand its assigned node.
- **Atomic**: the worker executes the task directly and writes a handoff artifact.
- **Composite**: the assigned agent decides the task is too big. Instead of
executing, it **decomposes** the node into a child sub-DAG that *it now owns*.
The original node becomes a **join / synthesis point**: it stays in progress
until all children complete, then the owner re-wakes, reads the children's
artifacts, and writes one synthesized output for whatever depends on it.
A composite node's owner is a **planner + integrator only** (map then reduce): it
decomposes, the children do the work, it synthesizes. It does not execute leaf
work itself. This keeps each node's responsibility clean and ownership boundaries
crisp.
```mermaid
flowchart TB
A[Task A - atomic] --> D[Task D - composite]
B[Task B - atomic] --> D
D --> D1[child D.1]
D --> D2[child D.2]
D1 --> D2
D2 --> Dj[D join / synthesize]
Dj --> E[Task E downstream of D]
```
Recursion is bounded only by a single **total-member cap** (1000 agents per swarm,
section 10). There is intentionally no depth limit and no per-node fan-out limit:
the spawn tree may nest and fan out freely until the swarm hits the member cap.
---
## 4. Node kinds by terminal action
The DAG is task-type-agnostic. The structure (decompose, gate, typed handoff) is
identical regardless of task; only the **artifact type and "done" criteria**
change. Most real tasks are **explore-then-act**, and that ordering is just a
dependency edge: exploration nodes feed implementation nodes.
| Kind | Artifact (output) | "Done" contract |
| ----------- | --------------------------------------- | ------------------------------ |
| `explore` | findings (the deliverable for research) | survives a critique gate |
| `implement` | diff / commit ref + what changed | survives a verify gate |
| `verify` | pass/fail + concrete failures | checks executed |
| `fix` | patch | re-verify passes |
- **Verify and fix** are what make the system *act and self-correct* rather than
only describe. A failing verify spawns fix nodes (more graph), exactly like a
critique spawns gap nodes.
- The final `synthesize` node is **optional**: for a pure exploration task it is
the deliverable; for an implementation task the deliverable is the merged,
verified code and the report is a thin rollup of what shipped.
```mermaid
flowchart LR
E[explore: how does X work] -. findings .-> P[plan changes]
P --> I1[implement: module A]
P --> I2[implement: module B]
I1 --> V{verify: build + tests}
I2 --> V
V -->|fails| Fix[fix node]
Fix --> V
V -->|passes| Done[working feature, committed]
```
---
## 5. Dataflow: how a finished dependency passes off information
Key insight: **the dependency edge IS the data channel.** Today a completion
report flows back to the *spawner* (control flow). In a DAG it must also flow
forward to *dependents* (data flow).
- On completion, each node stores a structured **handoff artifact** attached to
the node.
- When a downstream task becomes runnable (all deps done), the scheduler
**assembles its input** = the task's own prompt + the merged artifacts of all
its dependencies, injected into the worker's starting context. Fan-out (one dep
unblocks many) and fan-in (many deps feed one) both fall out naturally.
- Artifacts default to **by-reference, not by-value**: "I built the API in
`crates/foo/api.rs`, types X/Y, commit `abc123`." The repo + git are the shared
medium; the downstream agent reads the files itself. Embed by-value only for
things not in the repo (a decision, a design, an analysis). This keeps context
small, which matters at depth.
- For composite nodes the handoff is **decompose-then-synthesize** (map-reduce):
when children finish, the owner re-wakes with the children's artifacts and
writes one integration/synthesis report. A parent never just concatenates child
noise; it produces a clean summary for the next layer.
---
## 6. Completion and coverage: comprehensiveness as structure
Goal: completion should be so comprehensive that it is very unlikely any nook or
cranny of a task was missed. Comprehensiveness must be a **structural property of
the graph**, not a request in a prompt. Three reinforcing pressures:
### 6.1 Mandatory decomposition (breadth)
Exploration-style nodes are **composite by default**: the agent's first job is not
to answer but to **enumerate the surface area** into child nodes. Coverage becomes
visible and auditable in the graph (was there a node for hotplug? for DPI?) rather
than buried in prose you must trust.
### 6.2 Critique / verify gate (adversarial gap-finding)
Certain nodes get an auto-inserted **critique** (for explore) or **verify** (for
code) dependent before their parent can synthesize/close.
- The gate is adversarial: "what nook/cranny did this miss?" / "does it actually
work?"
- If it finds gaps or failures, it emits **new child nodes** back into the graph
(the recursion), and the parent cannot close until those drain.
- This is how you get "very unlikely we missed anything": the graph literally will
not let a parent complete with an open gap or failing check.
```mermaid
flowchart LR
Q[explore X] --> E[enumerate facets]
E --> F1[facet: hotplug]
E --> F2[facet: DPI]
F1 --> Cr{critique}
F2 --> Cr
Cr -->|gap found| F3[facet: cursor crossing]
F3 --> Cr
Cr -->|no gaps| S[synthesize report]
```
### 6.3 Typed artifact with explicit "what I did not check" (makes thinness visible)
The handoff artifact has a **required schema** so shallow output is structurally
detectable:
- `findings`
- `evidence` (file:line refs / commit refs, not bare claims)
- `edge_cases_considered`
- `validation` (verify results for code)
- `open_questions`
- `confidence`
- `what_i_did_not_check`
`what_i_did_not_check` is the cheat code: forcing an agent to list what it did
*not* explore surfaces unexplored crannies, which the critique/scheduler converts
into new nodes. Empty `open_questions` + `what_i_did_not_check` on a complex task
is itself a red flag the auditor checks.
So comprehensiveness now means two things, both enforced as gates: did we cover
the surface (critique), and does it actually work (verify). Both convert
gaps/failures into new nodes.
### 6.4 Implemented enforcement (2026-07: growth mechanics)
The pressures above are implemented as hard engine rules in `jcode-plan`'s
`dag` module, not prompt requests:
- **Root gate (plan-wide audit).** Every deep-mode `seed` auto-inserts a
parent-less gate (`plan::gate`) depending on every root-level node. A flat
seed whose nodes all execute atomically still cannot reach a terminal state
without a final adversarial pass, and that pass can `inject_gap` new
top-level nodes (growth at the top of the tree). Re-seeding widens the root
gate's scope and re-opens it if it had already passed.
- **Enumerated gate coverage.** A passing deep gate artifact must address
EVERY done node in its audit scope by id (scope = the gate's non-gate
`depends_on`, one rule for composite and root gates), up to an enumeration
cap of 20. Above the cap, enumeration relaxes only for HIGH-confidence
nodes; every medium/low/unparseable-confidence node must still be addressed
by id. "All good,
no gaps" is structurally rejected (`UncoveredSiblings`). A stale-scope rule
(`StaleGateScope`) rejects a pass when nodes entered the scope after the
gate was dispatched.
- **Artifact-or-nothing turn ends.** Deep mode has no auto-complete: a worker
turn that ends with its node still running gets the node re-queued once to a
fresh worker (`no_artifact_requeues`) and failed on repeat. The only ways a
deep node closes are `expand_node` (decompose) or `complete_node` (validated
artifact).
- **Growth accounting.** Every node records an origin (`seed`/`expand`/`gap`/
`gate`); `PlanGraphStatus` carries `seeded_count`/`grown_count` and
`plan_status`/`run_plan` print a growth line, so a deep plan that never
outgrew its seed is visibly under-explored.
---
## 7. Bias budget: what is fixed vs emergent
Central tension: too much pre-bias and you have hardcoded a brittle workflow; too
little and you lose the coverage guarantees. The split:
### What the first agent decides (the seed, deliberately small)
1. The root task framing (inherited from the user prompt).
2. The first-level decomposition: the initial facet/child nodes and their edges.
Even that first decomposition is **provisional**: every child can re-decompose,
and gates can inject siblings the first agent never imagined. The first agent sets
the *seed*, not the *shape*.
### What the system fixes (structural, not the first agent's choice)
- **Gate discipline**: every composite node gets a critique/verify dependent
before it can close. No opting out of being audited.
- **Handoff contract**: typed artifact with `what_i_did_not_check` /
`open_questions`, forced on every node.
- **Recursion right**: any descendant can expand its own node. The first agent
cannot "lock" the shape it drafted.
- **Gap/failure -> new nodes**: critique and verify convert misses into graph
regardless of the original plan.
The comprehensiveness guarantee therefore does **not** depend on the first agent
being smart. A mediocre first decomposition still gets caught and expanded.
```mermaid
flowchart LR
A[fully scripted: system dictates facets] --- B[seeded: first agent drafts, gates+recursion correct it] --- C[fully emergent: agent decides everything, no gates]
```
We sit at **B (seeded + structural gates)**:
- **Content / domain knowledge**: ~all from agents (first one seeds, descendants
refine). The system knows nothing about the domain (e.g. scrollwm).
- **Process / rigor / coverage**: ~none is the first agent's choice; it is
structural and uniform across the whole tree.
- **Final shape**: mostly emergent. The first agent's draft is typically a small
fraction of the final node count; most nodes are born from re-decomposition and
gate-spawned gaps/fixes.
**Invariant to protect:** do not leak domain assumptions into the structural
layer. Gates must stay **domain-agnostic**: critique asks "what is unexplored
*given this task's own stated scope and artifacts*"; verify runs "this task's
declared acceptance checks." Bias toward thoroughness is intentional; bias toward
specific content must be near zero. Re-running the same task should yield similar
first-level facets (stable seed) but different deep structure (adaptive
exploration).
---
## 8. Interface: enforced graph API, not an agent script
The interface choice determines how much rigor can actually be enforced. Options
considered:
- **A. Reframed swarm tool (graph ops).** Server owns the graph and *enforces*
invariants (acyclicity, ownership, mandatory gates, typed handoff) on every
mutation. Only this option makes "comprehensiveness is structural" true.
- **B. Agent writes a script.** Feels powerful, but a script that runs to
completion up front cannot express a graph that grows from runtime discovery.
It would have to block/await on node results and re-enter, becoming an
imperative driver around the same API, except now rigor lives in unvalidated
agent code and the gates are bypassable. This is the under-biased failure mode.
- **C. Tool primitive + optional declarative sugar.** Tool is the validated
substrate; allow a one-shot spec for the static part while runtime growth still
goes through tool calls.
**Decision: A as the foundation, with C's sugar.** The graph is a server-side
object mutated through validated ops, not an agent-side script. The agent keeps
full freedom in *deciding* the graph (arbitrary reasoning/tool use) and zero
freedom to skip the structural gates when *enacting* it.
```mermaid
flowchart TB
Agent[agent reasoning - arbitrary] -->|submit spec / expand_node| Tool[swarm tool = graph ops]
Tool -->|validate: acyclic, owned, gated, typed| Graph[(server-owned DAG)]
Graph --> Sched[scheduler: fill_slots/run_plan]
Sched -->|hydrate input from upstream artifacts| Workers
Workers -->|complete_node + artifact| Tool
```
### Proposed tool surface (evolution of `swarm`)
- `swarm task_graph {nodes:[...], edges:[...]}` - seed the initial DAG in one call
(the first agent's draft). Batch form of the ops, validated identically.
- `swarm expand_node {node_id, children:[...], edges:[...]}` - runtime
decomposition (the recursion). Ownership- and acyclicity-checked.
- `swarm complete_node {node_id, artifact:{findings, evidence, validation,
edge_cases_considered, open_questions, confidence, what_i_did_not_check}}` -
typed handoff that the gates inspect.
- `swarm run` - hand off to the scheduler.
- `spawn` / `dm` / `channel` remain as low-level escape hatches.
The "more control" agents actually want is per-node prompts, computed fan-out, and
conditional expansion. Those are served by (a) the agent computing the node list
however it likes *then* submitting it as a validated spec, and (b) runtime
`expand_node` calls, not by a scripting language that bypasses enforcement.
---
## 8a. Communication rework: dataflow first, chat second
The current swarm tool gives agents a rich human-chat surface: DMs, swarm-wide
broadcast, topic channels (subscribe/members), a shared key-value context store,
plus delivery modes (`notify`/`interrupt`/`wake`) and `await_members`. That is a
human-collaboration metaphor bolted onto agents. For the DAG model it is both too
much and the wrong shape. The rework is **by subtraction, not addition**.
### Why the current model is misfit
1. **It is chat, not dataflow.** Every existing channel is push-notification
messaging between *agents*. But in the DAG, the primary information transfer is
**node -> dependent via the artifact on the edge**, which does not exist as a
comm primitive yet. The most important "communication" in the new model is the
one thing the current toolset cannot express, so agents would have to simulate
dataflow by DMing each other - exactly the lossy coordination we are replacing.
2. **Too many overlapping primitives.** DM vs broadcast vs channel vs
shared-context-fanout are four ways to push text at other agents, and `message`
already auto-routes among three of them. The codebase already carries an
action-synonym normalization layer because models keep inventing verbs; that is
a smell that the surface is too large. More actions means more model error.
3. **Broadcasts must not scale to the member cap.** Whole-swarm fanout at the
1000-member cap (section 10) would be a 1000-way notification storm per send.
This is why broadcast-style sends are subtree-scoped (migration step 2,
implemented in `handle_comm_message`/`handle_comm_share`): a sender reaches
only its spawned subtree, and only the coordinator retains whole-swarm reach.
### The two-tier target model
Keep two tiers and drop the middle:
- **Tier 1 - structural dataflow (new, primary).** The handoff artifact on edges.
On completion, a node's typed artifact flows forward to its dependents
automatically via the scheduler, which hydrates each newly-runnable node's input
from its upstream artifacts. This replaces the bulk of what DMs are used for
today ("here is what I found, now you go"). Unlike a fire-and-forget DM it is
typed, durable, by-reference, and survives reloads.
- **Tier 2 - exception channel (keep, slim).** Direct agent-to-agent contact only
for genuinely unstructured cases the graph cannot model: conflict resolution
("we are both editing `foo.rs`") and clarifying questions up the ownership tree.
That is **DM + a subtree-scoped broadcast**, nothing more.
### What is demoted or cut
- **Shared-context key-value store**: largely redundant with the repo (the real
shared medium) plus typed artifacts. Keep only for a concrete non-repo
shared-state need; otherwise it is a second source of truth and should go.
- **Swarm-wide broadcast**: replaced with **subtree-scoped broadcast** that reaches
only an agent's owned descendants, so it cannot become a member-cap-sized storm.
Whole-swarm broadcast becomes a rare coordinator-only operation.
- **Generic topic channels**: unnecessary once dataflow is structural. Channels are
how *humans* organize ad hoc collaboration; agents should collaborate through
graph edges, not freeform rooms.
### Alignment with the DAG model
The dependency edge becomes the main communication channel (typed artifacts), and
agent-to-agent messaging shrinks to a small exception path (DM + subtree
broadcast). This aligns comm with the DAG, removes the broadcast-storm risk at the
member cap, and shrinks the error-prone tool surface.
### Staged migration (do not rip out up front)
Cutting channels/shared-context is a real behavior change for existing swarm flows.
Stage it:
1. **Done.** Artifact dataflow: completion artifacts flow to dependents and
hydrate their input.
2. **Done.** Broadcast scoped to the sender's spawned subtree (including the
no-subscriber channel fallback and shared-context notifications); whole-swarm
broadcast remains only as a coordinator escape hatch.
3. Migrate existing flows off channels/shared-context (tool schema now
discourages them).
4. Deprecate, then remove, the redundant chat primitives once flows have migrated.
---
## 9. Worked example: graph evolution over time
Task: "explore multimonitor support in scrollwm." Status legend: queued, running,
composite (decomposing/awaiting children), critique, done.
### T0 - First agent drafts the top-level skeleton
The root agent does not answer; it lays a seed: explore, gate, synthesize.
```mermaid
flowchart LR
R[explore multimonitor / composite] --> Cr{critique}
Cr --> S[synthesize report]
```
### T1 - Root expands explore into facets (composite -> children)
```mermaid
flowchart LR
F1[geometry/layout] --> Cr
F2[hotplug/disconnect] --> Cr
F3[DPI/scaling] --> Cr
F4[focus/cursor crossing] --> Cr
F5[workspace to output map] --> Cr
F6[existing code touchpoints] --> Cr
Cr{critique} --> S[synthesize]
```
### T2 - Scheduler dispatches ready facets (fan-out, parallel workers)
### T3 - A facet self-decomposes (recursion)
`w2` finds hotplug is deep, expands its own node, now owns a sub-DAG.
```mermaid
flowchart LR
F2[hotplug owner:w2 / composite] --> Cr
F2 --> H1[udev/event source]
F2 --> H2[reflow on remove]
F2 --> H3[restore on re-add]
H1 --> Hj[hotplug synth]
H2 --> Hj
H3 --> Hj
Hj --> Cr{critique}
```
### T4 - Atomic facets finish; edges now carry artifacts
`F1,F3,F4,F6` complete with typed artifacts; critique is blocked on `F2`.
### T5 - Hotplug children finish; owner re-wakes to synthesize (reduce)
`w2` reads H1/H2/H3 and writes one clean hotplug report; composite closes.
### T6 - Critique finds a gap and spawns new graph
Auditor reads every facet's `what_i_did_not_check`; nobody covered
fullscreen-on-one-output or mixed refresh rate. It injects new nodes and a
re-critique; synthesize stays blocked.
```mermaid
flowchart LR
Cr{critique: gaps found} --> G1[fullscreen on one output]
Cr --> G2[mixed refresh rate]
G1 --> Cr2{re-critique}
G2 --> Cr2
Cr2 --> S[synthesize]
```
### T7 - Gap nodes finish, re-critique passes, synthesize runs
Synthesize assembles ALL upstream artifacts (by reference) into the final report.
What the example demonstrates: breadth (facets as visible coverage), recursion
(hotplug self-decomposes), dataflow on edges (artifacts hydrate dependents),
map-reduce per composite (owner synthesizes), and comprehensiveness as a gate
(critique converts misses into graph; parent cannot close with open gaps). The
graph is never drafted once; it grows wherever depth or gaps are found and shrinks
in attention as subtrees collapse into synthesized artifacts.
---
## 10. Data model changes (against `jcode-plan`)
Reuse `VersionedPlan` / `PlanItem` (already has `blocked_by` edges,
`summarize_plan_graph`, `next_runnable_item_ids`, `newly_ready_item_ids`,
`run_plan`/`fill_slots`). Add:
- `PlanItem`: `owner_session`, `kind: atomic | composite` (plus terminal-action
kind: `explore | implement | verify | fix`), `parent_node`, and
`output: Option<HandoffArtifact>`.
- `HandoffArtifact`: typed schema from section 6.3.
- New op-based mutations: `expand_node(node_id, children, edges)` and
`complete_node(node_id, artifact)`, ownership-checked, acyclicity-checked,
versioned. `task_graph` is the batch-seed form.
- Scheduler: on dispatch, hydrate worker input from upstream `output`s; on
composite-join, re-wake owner to synthesize; auto-insert critique/verify
dependents per gate discipline.
- Roles (coordinator / worktree-manager) become scheduler policy, not user-facing
entities.
### Runaway prevention: a single total-member cap
Runaway prevention is one cap, not a matrix of limits. A swarm may hold at most
**`MAX_SWARM_MEMBERS` = 1000** live members (agents). There is deliberately **no
depth cap and no per-node breadth/fan-out cap**: the spawn tree may nest and fan
out freely until the swarm reaches 1000 members, at which point further spawns are
refused with a clear error. This is implemented in `ensure_spawn_coordinator_swarm`
(`server/comm_session.rs`) by counting live members of the swarm and rejecting the
spawn once the count reaches the cap. The older `MAX_SWARM_SPAWN_DEPTH` depth limit
is removed.
### Honest tradeoffs / limits
- The single member cap is the only throttle. It bounds total concurrency/cost but
does not prevent a lopsided tree (e.g. one greedy node consuming much of the
budget); that is left to agent judgment and the gate discipline.
- The graph **orders** work but does **not** do mutual exclusion. Two subtrees
editing the same files is still the "no-locks, talk it out via DM" case,
unchanged from today.
- Domain bias must be kept out of the structural/gate layer (section 7 invariant).
---
## 11. Suggested build order
1. Land the typed `HandoffArtifact` schema + `PlanItem` field additions in
`jcode-plan`.
2. Add validated `expand_node` / `complete_node` / `task_graph` ops (ownership +
acyclicity + gate auto-insertion).
3. Extend the scheduler to hydrate downstream input from upstream artifacts and to
re-wake composite owners for synthesis.
4. Build a text-based simulator to watch a graph evolve (like section 9) and
verify scheduler/critique/verify mechanics before wiring into the live swarm.
5. Reframe the tool surface (`task_graph`/`expand_node`/`complete_node`/`run`) and
the TUI to a graph-first view; keep `spawn`/`dm` as escape hatches.
6. Update `SWARM_ARCHITECTURE.md` to point at this DAG-first model.
docs/TERMINAL_BENCH.md
+119
View File
@@ -0,0 +1,119 @@
# Terminal-Bench 2.0 with jcode
This document describes the cleanest currently-working path for running jcode on Terminal-Bench 2.0 through Harbor.
## What is in the repo
- `scripts/jcode_harbor_agent.py`
- Harbor custom agent adapter for jcode
- `scripts/run_terminal_bench_harbor.sh`
- helper that wires Harbor to the adapter and a Linux-compatible jcode binary
- `scripts/run_terminal_bench_campaign.py`
- sequential campaign runner that preserves small batches in a stitchable layout
- `scripts/build_linux_compat.sh`
- builds a Linux jcode artifact against an older glibc baseline for TB-style containers
## Why the compat binary matters
Many Terminal-Bench task containers use an older glibc than a locally-built host binary. The Harbor adapter should use a Linux binary produced by:
```bash
scripts/build_linux_compat.sh /tmp/jcode-compat-dist
```
The helper script will build it for you automatically if it is missing.
## Auth and model assumptions
The current adapter is designed for:
- OpenAI OAuth auth file at `~/.jcode/openai-auth.json`
- `gpt-5.4`
- high reasoning effort
- priority service tier
Those defaults can be overridden with environment variables.
## Sequential campaign mode
If you want to run only a few tasks at a time but keep a coherent artifact set, use the campaign runner.
Example:
```bash
python scripts/run_terminal_bench_campaign.py \
--campaign-dir ~/tb2-jcode-campaign \
--task regex-log \
--task largest-eigenval \
--task cancel-async-tasks
```
What it does:
- runs tasks sequentially with `--n-concurrent 1`
- preserves Harbor jobs under `campaign-dir/harbor-jobs/`
- writes a pinned `campaign.json`
- refuses to mix runs if key settings drift
- appends per-task outcomes to `results.jsonl`
This is the recommended path when you want to batch tasks gradually and stitch them together later.
## Quick start
Assuming Terminal-Bench is already available at `/tmp/terminal-bench-2`:
```bash
scripts/run_terminal_bench_harbor.sh \
--include-task-name regex-log \
--n-tasks 1 \
--n-concurrent 1 \
--jobs-dir /tmp/jcode-tb2 \
--job-name regex-log-pilot \
--yes
```
Or point Harbor directly at the remote dataset:
```bash
scripts/run_terminal_bench_harbor.sh \
--dataset terminal-bench@2.0 \
--include-task-name regex-log \
--n-tasks 1 \
--n-concurrent 1 \
--jobs-dir /tmp/jcode-tb2 \
--job-name regex-log-pilot \
--yes
```
## Useful environment variables
- `JCODE_HARBOR_BINARY`
- path to the Linux-compatible jcode binary to upload into the task container
- `JCODE_HARBOR_BINARY_DIR`
- output directory used when auto-building the compat binary
- `JCODE_HARBOR_OPENAI_AUTH`
- path to the OpenAI OAuth file
- `JCODE_HARBOR_CA_BUNDLE`
- optional host CA bundle path to upload into the task container
- `JCODE_TB_MODEL`
- Harbor model string, default `openai/gpt-5.4`
- `JCODE_TB_PATH`
- default local Terminal-Bench path, default `/tmp/terminal-bench-2`
- `JCODE_OPENAI_REASONING_EFFORT`
- default `high`
- `JCODE_OPENAI_SERVICE_TIER`
- default `priority`
## Notes on fairness and state isolation
The adapter gives each trial a fresh in-container jcode home directory under `/tmp/jcode-home`, so memories and auth state are isolated per trial container.
## Current validation status
This path has already been validated with real Harbor task runs using:
- `regex-log`
- `largest-eigenval`
- `cancel-async-tasks`
All three passed in-container with verifier reward `1.0` during the initial pilot.
docs/TUISTATE_TRAIT_DECOMPOSITION.md
+156
View File
@@ -0,0 +1,156 @@
# TuiState Trait Decomposition Plan
Status: Analysis + proposed plan
This document audits the `TuiState` trait (`crates/jcode-tui/src/tui/mod.rs`) and
proposes a safe, incremental decomposition. It is the Phase 1.5 follow-on to the
`App` god-object decomposition (see `CLIENT_CORE_PRESENTATION_SPLIT_PLAN.md`).
## Current state
- `pub trait TuiState` exposes **114 methods**.
- Implementors: 2 (`App` in `tui/app/tui_state.rs`, and `TestState` in
`tui/ui_tests/mod.rs`).
- Consumers: ~95 usages across 29 files, almost all as `&dyn TuiState` (50
render-function signatures take `app: &dyn TuiState`).
It is the presentation-layer counterpart to the `App` god-object: a single wide
interface that couples every render module to the entire client surface.
## Why a naive sub-trait split has limited value
Two structural facts constrain the refactor:
1. **`App` implements the whole surface regardless.** Splitting `TuiState` into
`TuiTranscriptState + TuiInputState + ...` does not reduce what `App` must
implement, and (because the trait is presentation-only data access) it does
not change crate-level compile coupling. The win is intent/navigability, not
decoupling of `App`.
2. **`&dyn TuiState` does not compose.** Render functions take trait objects.
Rust has no stable `&dyn (A + B)`, so any consumer that needs methods from
more than one domain must take a supertrait that re-aggregates them. The two
central renderers (`ui.rs`, `ui_viewport.rs`) use methods from nearly every
domain, so they would keep the full supertrait bound.
Measured: of the ~28 `&dyn TuiState` render modules, only **2** are
multi-category (`ui.rs`, `ui_viewport.rs`); the other ~26 each use a single
domain. So a sub-trait split *does* narrow the declared surface for the majority
of render modules, but the headline god-interface (driven by the 2 central
renderers) stays wide via the supertrait.
Conclusion: the split is worthwhile for readability and for narrowing leaf
render-module bounds, but it is **not** a compile-coupling win and should be done
incrementally to avoid a high-conflict big-bang across 29 files.
## Proposed target shape
```
trait TuiState:
TuiTranscriptState + TuiInputState + TuiScrollState + TuiStreamStatusState
+ TuiProviderState + TuiSessionServerState + TuiWorkspaceState
+ TuiDiagramPaneState + TuiDiffPaneState + TuiSidePanelState
+ TuiInlineState + TuiOverlayState + TuiCopySelectionState
+ TuiOnboardingState + TuiMiscState
{}
```
`App` and `TestState` keep a single `impl` per sub-trait (mechanical move). The 2
central renderers take `&dyn TuiState` (the supertrait). Each leaf render module
narrows to the one sub-trait it needs.
## Method categorization (all 114)
### TuiTranscriptState
display_messages, display_user_message_count, compacted_hidden_user_prompts,
has_display_edit_tool_messages, side_pane_images, display_messages_version,
render_streaming_markdown
### TuiInputState
input, cursor_pos, queued_messages, interleave_message,
pending_soft_interrupts, has_stashed_input, command_suggestions,
command_suggestion_selected, suggestion_prompts, queue_mode,
next_prompt_new_session_armed, dictation_key_label
### TuiScrollState
scroll_offset, auto_scroll_paused, chat_overscroll_active,
copy_selection_edge_autoscroll_active, chat_native_scrollbar,
has_pending_mouse_scroll_animation
### TuiStreamStatusState
streaming_text, is_processing, streaming_tokens, streaming_cache_tokens,
output_tps, streaming_tool_calls, elapsed, status, active_skill,
subagent_status, batch_progress, time_since_activity, stream_message_ended,
status_notice, status_detail, rate_limit_remaining, animation_elapsed
### TuiProviderState
provider_name, provider_model, upstream_provider, connection_type,
mcp_servers, available_skills, auth_status, update_cost,
total_session_tokens, session_compaction_count, context_info,
context_snapshot, context_limit, cache_ttl_status
### TuiSessionServerState
is_remote_mode, is_canary, is_replay, current_session_id,
session_display_name, server_display_name, server_display_icon,
server_sessions, connected_clients, remote_startup_phase_active,
client_update_available, server_update_available, info_widget_data,
active_experimental_feature_notice
### TuiWorkspaceState
workspace_mode_enabled, workspace_map_rows, workspace_animation_tick
### TuiDiagramPaneState
diagram_mode, diagram_focus, diagram_index, diagram_scroll,
diagram_pane_ratio, diagram_pane_ratio_user_adjusted, diagram_pane_animating,
diagram_pane_enabled, diagram_pane_position, diagram_zoom
### TuiDiffPaneState
diff_mode, diff_pane_scroll, diff_pane_scroll_x, diff_pane_focus,
diff_line_wrap
### TuiSidePanelState
side_panel, side_panel_image_zoom_percent, side_panel_native_scrollbar,
pin_images, pinned_images_auto_hide_remaining_secs
### TuiInlineState
inline_interactive_state, inline_view_state, inline_ui_state
### TuiOverlayState
changelog_scroll, help_scroll, model_status_overlay, session_picker_overlay,
login_picker_overlay, account_picker_overlay, usage_overlay
### TuiCopySelectionState
copy_badge_ui, copy_selection_mode, copy_selection_range,
copy_selection_status
### TuiOnboardingState
onboarding_preview_mode, onboarding_welcome_active, onboarding_welcome_kind
### TuiMiscState
working_dir, now_millis, has_notification, centered_mode
## Incremental, low-conflict migration
Do **not** split all 15 sub-traits at once across 29 files. Recommended order:
1. Land the documented section headers in the trait definition (done; pure
comments, single file). Gives the categorization a canonical home.
2. Extract one leaf sub-trait with a single-file consumer as a proof of pattern
(e.g. `TuiCopySelectionState` or `TuiDiagramPaneState`). Verify with
`cargo check -p jcode-tui`.
3. Extract remaining leaf sub-traits one per commit, narrowing the corresponding
leaf render module's bound in the same commit.
4. Keep `ui.rs` and `ui_viewport.rs` on the `TuiState` supertrait throughout.
Each step is behavior-preserving (data accessors only) and compiles
independently, so it can be merged between other agents' work without a
big-bang conflict.
## Verification
- `cargo check -p jcode-tui` after each sub-trait extraction (TMPDIR must point
at real disk, not the RAM-backed tmpfs, or ring/aws-lc-sys build scripts fail
with "Disk quota exceeded").
- `cargo test -p jcode-tui --lib` once at the end. Note: the lib test suite has
pre-existing flaky parallel-order failures unrelated to this trait (verify any
failing test in isolation with `--test-threads=1`).
docs/UNIFIED_SELFDEV_SERVER_PLAN.md
+179
View File
@@ -0,0 +1,179 @@
# Unified Self-Dev / Normal Server Plan
> Status: **Implemented.**
>
> This document is preserved as a historical design/rollout plan. The current
> architecture uses a single shared server, with self-dev handled as a
> session-local canary capability rather than a separate dedicated daemon/socket.
> Any references below to `/tmp/jcode-selfdev.sock`, `canary-wrapper`, or
> `JCODE_SELFDEV_MODE` describe the pre-merge architecture or transition steps,
> not the current runtime design.
## Goal
Reduce RAM usage by removing the dedicated self-dev daemon/socket pair and treating self-dev as a **session capability** on the normal shared server.
Today, normal sessions and self-dev sessions can end up with separate long-lived server processes, which duplicates:
- Tokio runtime overhead
- allocator heap / fragmentation footprint
- MCP pool state
- embedding/model lifecycle machinery
- event buffers, registries, session maps, swarm maps
- general server baseline RSS
## Current Architecture
### Normal mode
- Main socket: runtime `jcode.sock`
- Debug socket: runtime `jcode-debug.sock`
- Startup path: `jcode` -> default client flow -> spawn `jcode serve` if needed
### Self-dev mode
- Main socket: `/tmp/jcode-selfdev.sock`
- Debug socket: `/tmp/jcode-selfdev-debug.sock`
- Startup path:
- repo auto-detection or `jcode self-dev`
- `cli/selfdev.rs::run_self_dev()`
- exec into `canary-wrapper`
- wrapper ensures self-dev server exists on dedicated socket
- wrapper launches TUI client against that socket
## Key Finding From Code Inspection
The runtime already supports **per-session self-dev state**:
- protocol `Subscribe { working_dir, selfdev }`
- server subscribe handling can mark only that session as canary/self-dev
- `selfdev` tool availability is already gated on `session.is_canary`
- prompt additions are already gated on `session.is_canary`
- clear/resume/headless flows already preserve or infer canary state per session
This means the main remaining split is not the session model, but the **startup / reload / wrapper plumbing**.
## Target Architecture
### One shared server
- Main socket: runtime `jcode.sock`
- Debug socket: runtime `jcode-debug.sock`
- Self-dev sessions connect to the same server as normal sessions
### Self-dev becomes session-local
A client is self-dev if any of the following are true:
- explicit `jcode self-dev`
- current working directory is the jcode repo (auto-detected)
- resumed session is already canary
That client connects to the shared server and sends:
- `working_dir`
- `selfdev: true`
The server then:
- marks the session canary
- registers selfdev tools for that session
- includes selfdev prompt additions for that session only
### Debug socket
With one shared server, there is one shared debug socket.
Consequences:
- no dedicated self-dev debug socket
- debug tooling sees both normal and self-dev sessions from the same server
- selfdev-sensitive actions remain gated by target session canary state
## Important Policy Decision
If a self-dev session triggers a reload, it reloads the **shared server**.
That means all clients reconnect.
This is the cleanest design for RAM savings.
The binary chosen for reload should depend on the **triggering session**, not a server-global self-dev mode flag:
- normal session reload -> stable / launcher candidate
- canary session reload -> repo / canary candidate
## Implementation Phases
### Phase 1 - Client-side self-dev on shared server path
**Goal:** stop repo auto-detection from forcing a separate self-dev daemon.
Changes:
- do not auto-divert repo startup into `canary-wrapper`
- introduce a client-only self-dev signal (separate from server self-dev env)
- keep using normal server spawn/connect path
- continue sending `Subscribe { selfdev: true }`
- prevent the shared server child process from inheriting the client-only self-dev env
- stop server self-dev detection from inferring self-dev based on current working directory
Expected result:
- opening jcode inside the repo uses the shared server path by default
- session still becomes canary/self-dev
- explicit `jcode self-dev` command may still use legacy wrapper temporarily
### Phase 2 - Move explicit `jcode self-dev` onto shared server path
**Goal:** make explicit self-dev command use the same shared-server flow.
Changes:
- simplify `cli/selfdev.rs::run_self_dev()`
- keep optional `cargo build --release`
- set client-only self-dev mode
- connect through normal client/server startup path
- remove need for `canary-wrapper` in standard usage
Expected result:
- both auto-detected self-dev and explicit `jcode self-dev` share one server
### Phase 3 - Session-targeted reload selection
**Goal:** remove server-global self-dev assumptions from reload/update behavior.
Changes:
- include triggering session context in reload handling
- choose server exec target based on triggering session canary state
- always run reload monitor on the shared server, but authorize via session state / request policy
Expected result:
- one shared server can still reload into the right binary
### Phase 4 - Remove dedicated self-dev socket assumptions
**Goal:** fully retire the separate socket model.
Changes:
- deprecate `/tmp/jcode-selfdev.sock` and `/tmp/jcode-selfdev-debug.sock`
- update docs, tests, and scripts that probe self-dev via separate sockets
- simplify debug/test tooling to use the shared debug socket
## Risks / Tradeoffs
### Shared reload impact
A self-dev-triggered reload affects all clients on the shared server.
This is the main behavior change and the key tradeoff for RAM savings.
### Legacy tooling assumptions
Some scripts and tests currently prefer the self-dev debug socket path and will need updating.
### Scattered env-based logic
There are multiple `JCODE_SELFDEV_MODE` checks across startup, hot reload, and server behavior; these need to be separated into:
- client self-dev request
- server self-dev mode (legacy / compatibility)
- session canary capability
## Files Likely To Change
- `src/cli/dispatch.rs`
- `src/cli/selfdev.rs`
- `src/cli/hot_exec.rs`
- `src/server.rs`
- `src/server/reload.rs`
- `src/server/client_session.rs`
- `src/tui/mod.rs`
- `src/tui/backend.rs`
- `docs/SERVER_ARCHITECTURE.md`
- debug/test scripts that assume separate self-dev sockets
## Recommended Order
1. Land Phase 1 foundations and shared-path client self-dev
2. Land explicit `jcode self-dev` shared-path behavior
3. Refactor reload/update selection to be session-targeted
4. Remove legacy wrapper/socket assumptions and update tests/docs
docs/WINDOWS.md
+137
View File
@@ -0,0 +1,137 @@
# Windows Support Architecture
This document describes how jcode achieves cross-platform support for Linux, macOS, and Windows.
## Status
- **Transport layer**: Implemented (`src/transport/`)
- **Platform module**: Implemented (`src/platform.rs`)
- **Windows transport**: Implemented but untested (`src/transport/windows.rs`)
- **Windows platform**: Implemented (`src/platform.rs` has `#[cfg(windows)]` branches)
- **Windows CI**: Not yet set up
## Design Principle
**Zero cost on Unix.** The abstraction layer uses `#[cfg]` compile-time gates and type aliases so that Linux and macOS code paths compile to the exact same binary as before. Windows gets its own implementations behind `#[cfg(windows)]`. No traits, no dynamic dispatch, no runtime branching.
## Install Paths
Current Windows install paths from `scripts/install.ps1`:
- Launcher: `%LOCALAPPDATA%\\jcode\\bin\\jcode.exe`
- Stable channel binary: `%LOCALAPPDATA%\\jcode\\builds\\stable\\jcode.exe`
- Immutable versioned binaries: `%LOCALAPPDATA%\\jcode\\builds\\versions\\<version>\\jcode.exe`
Unlike the current Unix self-dev/local-build flow, the PowerShell installer currently installs the stable channel rather than a separate `current` channel.
## Transport Layer (`src/transport/`)
The transport layer abstracts IPC (Inter-Process Communication). On Unix, jcode uses Unix domain sockets. On Windows, jcode uses named pipes.
### Module Structure
```
src/transport/
mod.rs - conditional re-exports (cfg-gated)
unix.rs - type aliases wrapping tokio Unix sockets (zero-cost)
windows.rs - named pipe Listener/Stream with split support
```
### Unix (Linux + macOS)
Unix transport is a thin re-export of existing types:
```rust
pub use tokio::net::UnixListener as Listener;
pub use tokio::net::UnixStream as Stream;
pub use tokio::net::unix::OwnedWriteHalf as WriteHalf;
pub use tokio::net::unix::OwnedReadHalf as ReadHalf;
pub use std::os::unix::net::UnixStream as SyncStream;
```
The compiled binary is byte-for-byte identical to what it was before the abstraction.
### Windows
Windows transport provides custom types wrapping `tokio::net::windows::named_pipe`:
- **`Listener`**: Wraps `NamedPipeServer` with an accept loop that creates new pipe instances for each connection (named pipes are single-client, so a new instance is created after each accept)
- **`Stream`**: Enum over `NamedPipeServer` (accepted) or `NamedPipeClient` (connected), implementing `AsyncRead + AsyncWrite`
- **`ReadHalf` / `WriteHalf`**: Created via `stream.into_split()` using `Arc<Mutex<Stream>>` since named pipes don't support native kernel-level splitting
- **`SyncStream`**: Opens the named pipe as a regular file for blocking I/O
Socket paths are converted to pipe names: `/run/user/1000/jcode.sock` becomes `\\.\pipe\jcode`.
### API Surface
Both platforms export the same interface:
| Export | Unix | Windows |
|--------|------|---------|
| `Listener` | `tokio::net::UnixListener` | Custom struct wrapping `NamedPipeServer` |
| `Stream` | `tokio::net::UnixStream` | Enum over `NamedPipeServer`/`NamedPipeClient` |
| `ReadHalf` | `tokio::net::unix::OwnedReadHalf` | `Arc<Mutex<Stream>>` wrapper |
| `WriteHalf` | `tokio::net::unix::OwnedWriteHalf` | `Arc<Mutex<Stream>>` wrapper |
| `SyncStream` | `std::os::unix::net::UnixStream` | `std::fs::File` wrapper |
## Platform Module (`src/platform.rs`)
Centralizes all non-IPC OS-specific operations:
| Function | Unix | Windows |
|----------|------|---------|
| `symlink_or_copy(src, dst)` | `std::os::unix::fs::symlink()` | Try `symlink_file/dir`, fall back to copy |
| `atomic_symlink_swap(src, dst, temp)` | Create temp symlink + rename | Remove + copy (best effort) |
| `set_permissions_owner_only(path)` | `chmod 600` | No-op |
| `set_permissions_executable(path)` | `chmod 755` | No-op |
| `is_process_running(pid)` | `kill(pid, 0)` | Returns `true` (stub) |
| `replace_process(cmd)` | `exec()` (replaces process) | `spawn()` + `exit()` |
## Files Migrated
All OS-specific code has been moved out of application files into the transport and platform modules:
| File | What was migrated |
|------|------------------|
| `src/server.rs` | `UnixListener`, `UnixStream`, `OwnedReadHalf`, `OwnedWriteHalf` |
| `src/tui/backend.rs` | `UnixStream`, `OwnedWriteHalf`, `OwnedReadHalf` |
| `src/tui/client.rs` | `UnixStream`, `OwnedWriteHalf` |
| `src/tui/app.rs` | `UnixListener`, `OwnedWriteHalf`, file permissions |
| `src/tool/communicate.rs` | `std::os::unix::net::UnixStream` |
| `src/tool/debug_socket.rs` | `tokio::net::UnixStream` |
| `src/main.rs` | `UnixStream` (health checks), all `exec()` calls, file permissions |
| `src/build.rs` | Symlinks, executable permissions |
| `src/update.rs` | Symlinks, permissions, atomic swap |
| `src/auth/oauth.rs` | Credential file permissions |
| `src/skill.rs` | Symlink creation |
| `src/video_export.rs` | Frame symlinks |
| `src/ambient.rs` | Process liveness check |
| `src/registry.rs` | Process liveness check |
| `src/session.rs` | Process liveness check |
## Dependencies
```toml
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.59", features = ["Win32_Foundation", "Win32_System_Threading"] }
```
The `tokio` dependency already includes named pipe support on Windows (part of `features = ["full"]`).
## What Doesn't Change
The vast majority of the codebase is platform-agnostic:
- All provider code (HTTP-based)
- All tool implementations (except bash tool's shell selection)
- TUI rendering (crossterm + ratatui already cross-platform)
- Agent logic, memory, sessions, config
- MCP client/server protocol
- JSON serialization, protocol handling
## Remaining Work
1. **Windows CI** - Add GitHub Actions Windows runner, test compilation and basic IPC
2. **Shell tool** - Detect platform and use `cmd.exe` or `pwsh.exe` on Windows
3. **Self-update** - Handle Windows exe replacement (can't overwrite running binary)
4. **Testing** - Run full test suite on Windows
docs/WRAPPERS.md
+134
View File
@@ -0,0 +1,134 @@
# jcode wrapper / scripting guide
This document describes the non-interactive CLI surface intended for wrappers, scripts, and other tools that invoke `jcode`.
## Recommended flags
Use these flags by default in wrappers:
```bash
jcode --quiet --no-update --no-selfdev ...
```
- `--quiet` suppresses non-error CLI/status chatter
- `--no-update` avoids update-check noise/work
- `--no-selfdev` avoids repository auto-detection changing runtime behavior
## Discover available models
List model names that can be passed to `-m/--model`:
```bash
jcode --quiet model list
jcode --quiet model list --json
jcode --quiet --provider openai model list --json
```
## Discover providers and current selection
List provider IDs you can pass to `-p/--provider`:
```bash
jcode --quiet provider list
jcode --quiet provider list --json
```
Inspect the currently requested and resolved provider/model selection:
```bash
jcode --quiet provider current
jcode --quiet --provider openai --model gpt-5.4 provider current --json
```
Verbose human summary:
```bash
jcode --quiet model list --verbose
```
## Run one prompt and return JSON
```bash
jcode --quiet run --json "Reply with exactly OK"
```
## Stream one prompt as NDJSON
```bash
jcode --quiet run --ndjson "Reply with exactly OK"
```
Typical event types:
- `start`
- `connection_phase`
- `connection_type`
- `text_delta`
- `text_replace`
- `tool_start`
- `tool_input`
- `tool_exec`
- `tool_done`
- `tokens`
- `done`
- `error`
The final `done` event includes the assembled text and usage summary.
Example shape:
```json
{
"session_id": "session_...",
"provider": "OpenAI",
"model": "gpt-5.4",
"text": "OK",
"usage": {
"input_tokens": 123,
"output_tokens": 7,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": null
}
}
```
## Inspect authentication state
```bash
jcode --quiet auth status
jcode --quiet auth status --json
```
JSON output includes:
- `any_available`
- `providers[]`
- `id`
- `display_name`
- `status`
- `method`
- `auth_kind`
- `recommended`
## Inspect build/version details
```bash
jcode --quiet version
jcode --quiet version --json
```
JSON output includes:
- `version`
- `git_hash`
- `git_tag`
- `build_time`
- `git_date`
- `release_build`
## Notes
- JSON commands are designed so the intended machine-readable result is printed to `stdout`
- With `--quiet`, wrapper-oriented commands should keep `stderr` empty unless there is a real warning/error
- `jcode model list` and `jcode run --json` do not require the TUI
- `jcode model list` does not require an already-running shared server
docs/dev/crate-splitting-plan.md
+58
View File
@@ -0,0 +1,58 @@
# Compile-time crate splitting plan
## Goal
Minimize the amount of code that must be rechecked or rebuilt when iterating on
Jcode. The root `jcode` crate is still the integration shell, but stable leaf
code should live in small crates with one-way dependencies.
## Principles
1. Extract stable leaves first: filesystem/storage, protocol/types, parsers,
provider request/stream codecs, and TUI render primitives.
2. Avoid cyclic domain crates. Root `jcode` may depend on leaf crates, but leaf
crates must not call back into root logging/config/runtime directly. Use data
types, callbacks, or explicit events at boundaries.
3. Split by recompilation volatility, not by directory names. Code edited often
should not force heavy provider/TUI/server modules to rebuild unless needed.
4. Keep heavy optional dependencies behind crates/features. Embeddings, PDF,
desktop/mobile, browser, and image/render pipelines should remain isolated.
5. Preserve compatibility facades during migration. `crate::storage::*` can
re-export `jcode-storage::*` while callers move gradually.
## Current first step
`jcode-storage` is now a leaf crate for app paths, permission hardening, atomic
JSON writes, and append-only JSONL helpers. The root `src/storage.rs` module is a
thin compatibility facade that preserves existing logging behavior for backup
recovery.
Measured after extraction on this machine:
- `cargo check -p jcode-storage`: ~0.9s after initial dependencies were built.
- `cargo check -p jcode --lib`: ~14s in the current warm-cache state.
## Recommended next extractions
1. `jcode-provider-anthropic`: move Anthropic request/stream translation out of
root `src/provider/anthropic.rs` and depend only on `jcode-provider-core`,
`jcode-message-types`, and serde/reqwest primitives.
2. `jcode-provider-openai`: same for OpenAI request/stream handling. This
reduces rebuilds when editing server/TUI code and makes provider tests cheap.
3. `jcode-session-core`: move session storage paths, journal metadata, and
memory-profile pure transforms once dependencies on root prompt/logging are
cut behind callbacks.
4. `jcode-tui-app-state`: split key/input/navigation state transitions from
rendering. Keep ratatui rendering in `jcode-tui-render`/root while state tests
compile without the whole root crate.
5. `jcode-server-protocol-runtime`: split websocket/client event fanout glue from
agent execution so server tests do not rebuild TUI/provider internals.
## Anti-patterns to avoid
- Extracting crates that depend on root `jcode`. That preserves the compile-time
bottleneck and creates dependency cycles.
- Tiny crates for every file. Too many crates increase metadata overhead and make
refactors painful.
- Moving only type aliases while leaving implementations in root. The expensive
compile units remain expensive.
docs/images/high-level.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 65 KiB

docs/images/memory-arch.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 144 KiB

docs/images/memory.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 89 KiB

docs/images/openclaw.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 234 KiB

docs/images/swarm.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 76 KiB

docs/jcode_reddit_dashboard.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 947 KiB

docs/proposals/CROSS_DEVICE_WORKTREE_SYNC.md
+174
View File
@@ -0,0 +1,174 @@
# Cross-Device Worktree and Build Sync (Proposed)
Status: Proposed (design only, not implemented)
## Problem
A user develops jcode (and other repos) on multiple machines, e.g. a MacBook
(aarch64-darwin) and a Linux laptop. On a single machine, `selfdev build` +
reload means every local jcode instance runs the new version, and multiple
agents can share one worktree because the server mediates edits and tracks
conflicts (`FileTouchService`, `file_activity.rs`).
Split across machines, this breaks:
- A `selfdev build` on the Linux laptop does not update the MacBook's jcode,
and vice versa.
- There is no shared worktree, so changes made on one machine are invisible
to agents/sessions on the other until manually pushed/pulled.
Goal: make multiple machines behave like one logical worktree + one logical
build channel, the same way multiple agents already share one worktree on a
single machine.
## Existing building blocks (verified in code)
| Building block | Where | Why it matters |
| --- | --- | --- |
| Arch-independent version identity | `jcode-build-support/src/source_state.rs` (`SourceState::version_label`, `fingerprint`) | Fingerprint hashes full commit hash + status + `diff --binary HEAD` + untracked contents. Identical source trees on two machines produce the same label, even though binaries differ per-arch. |
| Auto-reload on newer binary | `server/util.rs::server_has_newer_binary`, `reload_exec_target` | Mtime-based channel scan. A peer-triggered local build that publishes to `builds/current` triggers the existing reload flow with zero changes. |
| Pull → build → install → exec | `session_rebuild.rs` | Already implements the receiving side's pipeline shape. |
| Network door, same protocol | `jcode-base/src/gateway.rs` (WS + plain HTTP on :7643) | Remote clients speak the identical newline-JSON protocol as Unix-socket clients. Plain HTTP handler (`/pair`, `/health`) is a natural place for `/peer/*` endpoints. |
| NAT-friendly device event bus | `server/jade_relay.rs` | Device IDs, heartbeats, long-polled command events. Works when machines cannot reach each other directly. |
| Server-side tool execution | server architecture | Tools (bash, edit) run in the server process; a remote client attaching to another machine's server gets the full multi-agent-one-worktree behavior, including conflict warnings. |
| Remote build precedent | `scripts/remote_build.sh` | rsync + ssh + sync-back pattern. |
### Known gap: repo identity across machines
`repo_scope_key` / `worktree_scope_key` hash the *local canonical path* of the
git common dir / worktree. These will never match across machines
(`/Users/jeremy/...` vs `/home/jeremy/...`). Cross-device features must key
repos by something portable: normalized origin URL, or an explicit repo name
in config (`[sync] repo_id = "jcode"`), falling back to origin URL hash.
## Design tensions
1. **Binaries cannot be shared** across darwin-aarch64 and linux-x86_64.
"Same version everywhere" must mean *same source state, built per machine*,
with `version_label` as the cross-machine equality check.
2. **Not all writes are server-mediated.** Tool edits flow through the server,
but `bash`, editors, and `cargo` mutate the worktree invisibly. Cross-device
sync therefore needs a byte-level capture mechanism (git plumbing snapshot
and/or fs watcher), not just tool-event forwarding.
3. **Laptops go offline.** Pure live-sync (mutagen/syncthing style) has poor
conflict semantics. Git-based convergence (per-device sync refs, merges)
handles offline divergence honestly.
## Mechanism: shipping a worktree state without touching HEAD
To capture a possibly-dirty worktree atomically without moving the user's
HEAD or index:
```
GIT_INDEX_FILE=$tmp git add -A # tracked + untracked into temp index
tree=$(GIT_INDEX_FILE=$tmp git write-tree)
commit=$(git commit-tree $tree -p HEAD -m "jcode sync: <device> <fingerprint>")
git push origin $commit:refs/jcode/sync/<device>
```
- Atomic, content-addressed, includes untracked files, excludes gitignored.
- Receiver fetches the ref, applies it (checkout into worktree or materialize
diff), then verifies `current_source_state().fingerprint` matches the
beacon's fingerprint, guaranteeing byte-exact reproduction.
- The sender's worktree/index/HEAD never move.
## Phased plan
### Phase B - selfdev build parity (do first; kills the stated pain)
After a successful `selfdev build` + publish on machine X:
1. Compute `SourceState` (already done by the build pipeline).
2. Snapshot the worktree to `refs/jcode/sync/<device>` (mechanism above) and
push to the shared git remote. Clean trees can skip the snapshot and use
the existing commit.
3. Announce a **version beacon** `{repo_id, version_label, fingerprint,
full_hash, sync_ref, device, timestamp}`:
- Fast path: HTTP POST to peer gateways (`/peer/version-beacon`) over
Tailscale.
- Fallback: jade relay device event (works through NAT).
- Slow path: peer polls sync refs on the git remote.
Machine Y's server runs a small peer-sync task (same shape as
`jade_relay::spawn_if_configured`):
1. Receives beacon; ignores if `version_label` matches what it already runs
or recently applied (**echo suppression**, prevents rebuild ping-pong).
2. Policy gate, default conservative:
- Worktree clean AND local HEAD is an ancestor of the beacon commit
→ auto-apply: fetch, advance, build via the selfdev build queue
(native arch), publish. The existing `server_has_newer_binary()` poll
then auto-reloads.
- Otherwise → do not clobber. Surface in TUI/status:
`peer build available: <label> from <device> (blocked: local changes)`
with a one-keystroke accept.
3. After publish, Y's `version_label` equals X's. Cross-device parity is
verifiable by comparing labels (e.g. in `selfdev status` and the beacon
acks).
Config sketch:
```toml
[sync]
enabled = true
repo_id = "jcode" # portable repo identity
peers = ["macbook.tail-net.ts.net:7643"]
auto_apply = "clean-ff-only" # off | clean-ff-only | always-notify
```
### Phase A - hub attach (one authoritative worktree when both online)
`jcode attach <host>`: TUI connects to the peer machine's server through the
existing gateway WS. Because tools execute server-side, the attached client
participates fully in that machine's worktree, conflict tracking included.
Work items:
- Client transport: WS stream in place of Unix socket (bridge already exists
server-side; needs a client-side counterpart).
- Pairing/auth UX for a trusted personal device (DeviceRegistry exists).
- Audit client-side local-disk reads that assume the session's filesystem,
e.g. `jcode-tui/src/tui/ui_file_diff.rs:270` (`std::fs::read_to_string` of
the diffed file). These need server RPCs (a `read_file` control request) or
graceful degradation.
### Phase C - true worktree federation (long-term)
Generalize Phase B's snapshot + Phase A's peering into continuous two-way
sync:
- **Data plane:** throttled auto-snapshots to per-device sync refs, triggered
by (a) server-mediated tool edits, (b) an fs watcher for bash/editor/cargo
writes, (c) timers. Peers fetch refs directly (ssh/Tailscale) or via the
shared remote.
- **Convergence:** if local HEAD/state is an ancestor → fast-forward apply.
If diverged → keep both snapshots, mark the repo "split", and let the
harness spawn an agent to perform the merge (the cross-device analog of
the server managing same-machine conflicts).
- **Coordination plane:** federate `FileTouchService` events over the peer
link so agents on both machines see "another agent edited lines 40-60"
warnings across devices. Optionally add advisory write leases for hot
files.
## Alternatives considered
- **Syncthing/mutagen for the worktree:** simple, but byte-level conflicts
(`.sync-conflict` files), no atomicity across multi-file edits, and target
dirs / build artifacts need careful exclusion. Git-plumbing snapshots give
atomic, content-addressed, mergeable states using semantics git users
already understand.
- **Always build on one machine + copy binaries:** broken by arch mismatch;
cross-compiling darwin from linux (and vice versa) is not worth the
toolchain cost given both machines have working local toolchains.
- **NFS/SSHFS shared worktree:** punishes offline use and IDE/file-watcher
performance; a non-starter for laptops.
## Suggested order of implementation
1. Phase B beacon + receiver with `auto_apply = "clean-ff-only"`, git-remote
polling only (no new ports), `selfdev status` showing peer parity.
2. Add gateway `/peer/version-beacon` fast path + TUI notification for the
blocked case.
3. Phase A `jcode attach` (WS client transport + pairing + file-read RPC).
4. Phase C federation, reusing the beacon snapshot machinery and the peer
link from A.
docs/proposals/computer-use-maximal-control.md
+229
View File
@@ -0,0 +1,229 @@
# Roadmap: maximal macOS control for the `computer` tool
Goal: give the agent as much reliable control over macOS as the platform allows,
including **background control that does not disturb what the user is looking at**.
This builds on the v1 `computer` tool (PR #345): screenshot, coordinate
mouse/keyboard, scroll, AX-tree read, cursor, permission check.
Everything below is implementable in Rust with crates that are already in the
lockfile or available on crates.io (`accessibility-sys`, `screencapturekit`,
`objc2-app-kit`, `core-graphics`). No Swift/ObjC build step.
---
## The one hard constraint
macOS has **one HID cursor and one keyboard-focus** shared by the whole login
session. Synthetic *coordinate* input (CGEvent) is therefore always visible: it
moves the real cursor and types into the focused app.
**Background / not-in-view control must avoid CGEvent** and instead go through:
1. **Accessibility (AX) actions** - act on a specific element by reference.
2. **Apple Events / scripting** - drive scriptable apps with no UI.
3. **Per-window capture** - "see" a window without raising it.
True simultaneous "you work + I work independently" needs a **separate display
or login session** (see Tier 4).
---
## Tier 0 - done (v1, PR #345)
- `screenshot` (main display, point/pixel scale aware)
- `move` / `click` / `double_click` / `right_click` / `drag` / `scroll`
- `type` / `key` (chords)
- `ui` (AX tree read via osascript)
- `cursor`, `check_permissions`
## Tier 1 - AX semantic actions ← highest leverage for background control
Read + act on elements by reference, no cursor movement, target app need not be
frontmost. Uses `accessibility-sys` (`AXUIElementPerformAction`,
`AXUIElementSetAttributeValue`, `AXUIElementCopyElementAtPosition`,
`AXUIElementCopyAttributeValue`).
- `find_element { role?, title?, value?, pid?, app? }` -> stable element handles
- `element_at { x, y }` -> element under a point (AXUIElementCopyElementAtPosition)
- `press { element }` -> `AXPress` (click a button in a background window)
- `set_value { element, value }` -> type into a field without focus
- `get_value { element }`
- `perform_action { element, ax_action }` -> any advertised AX action
- `select_menu { app, path: ["File","Export…"] }` -> drive the menu bar of any app
Handle format: `pid` + AX path (index chain) or a session-scoped element id cache,
so the model can act structurally instead of by pixels.
Why it matters: this is the actual "click things you're not looking at" capability.
## Tier 2 - app / window / system management
Mostly `objc2-app-kit` (`NSWorkspace`, `NSRunningApplication`) + AX window
attributes + CoreGraphics window list.
- `list_apps` / `activate_app { app }` / `hide_app` / `quit_app`
- `list_windows { pid? }` (CGWindowList) with ids, titles, bounds, on/off-screen
- `focus_window` / `move_window` / `resize_window` / `minimize_window` / `close_window`
(AX window actions - can target background windows)
- `window_screenshot { window_id }` -> capture a specific window even if occluded
(`CGWindowListCreateImage` now, ScreenCaptureKit later)
- `spaces` awareness (which Space an app is on; activating may switch Spaces - visible)
## Tier 3 - clipboard, input fidelity, observation
- `get_clipboard` / `set_clipboard { text }` (`NSPasteboard` via objc2-app-kit)
- `key_down` / `key_up` (hold modifiers, game-style input)
- `type_into { element, text }` (AX set value + confirm) for reliability over blind typing
- `wait_for { element|condition, timeout }` using `AXObserver*` notifications
(e.g. wait for a sheet to appear) instead of sleep-and-poll
- `paste_type { text }` - set clipboard + Cmd-V for fast/large text entry
## Tier 4 - true background / parallel operation (advanced)
These give genuinely off-screen, non-interfering control. Higher setup cost.
- **Apple Events scripting bridge**: `run_applescript { script }` / `run_jxa`.
Fully headless for scriptable apps (Mail, Notes, Safari, Finder, Music, System
Settings panes, Terminal, many pro apps). No cursor, no focus. Per-app
Automation permission (prompts on first use).
- **Virtual / headless display**: route the agent's cursor+windows to a second
(virtual) display the user isn't looking at. Options: a virtual display driver
(e.g. BetterDisplay/`CGVirtualDisplay` private API) or a real unused monitor.
Lets the agent move windows there and use coordinate input without touching the
user's screen.
- **Separate login / Screen Sharing session**: a second macOS session has its own
cursor and focus; the agent drives that one. Strongest isolation, most setup.
- **Shortcuts integration**: invoke the user's `Shortcuts` automations
(`shortcuts run …`) as high-level, sanctioned actions.
## Tier 5 - sensors / extras (optional, opt-in)
- `ocr { region|window }` via Vision framework (read text in images / non-AX apps).
- `screen_record { seconds }` short clips via ScreenCaptureKit.
- Audio in/out control, notifications, `do_not_disturb` toggling via scripting.
- Camera/mic are separate TCC permissions; keep strictly opt-in.
---
## Permissions (TCC) - the gatekeeping reality
| Permission | Unlocks | Auto-grantable? |
|---|---|---|
| **Accessibility** | CGEvent input, all AX read/act, window control | No - user toggles once (we can prompt + deep-link) |
| **Screen Recording** | screenshots, window/ocr capture | Request API exists (`CGRequestScreenCaptureAccess`) |
| **Automation (Apple Events)** | scripting each app | Prompts per target app on first send |
| **Input Monitoring** | reading global input stream (only if we add capture) | Request API exists |
Plan: a `request_permissions` action that calls
`AXIsProcessTrustedWithOptions(prompt=true)` (adds jcode to the list + shows the
dialog) and deep-links to the exact System Settings pane, then polls
`AXIsProcessTrusted()`. One prompt + one toggle; never zero-touch for Accessibility
(Apple's anti-malware boundary).
Important: the permission attaches to the **host binary/terminal** running jcode.
For a stable experience we likely want a signed jcode.app with a fixed bundle id so
the grant persists across updates (otherwise each new binary path re-prompts).
## Safety model (high blast radius)
- Gated like `bash`: refuses early if required permission missing.
- `dry_run` on mutating actions: resolve + report target without acting.
- Prefer AX semantic actions over blind coordinate clicks (auditable, robust).
- Screenshot/element echo on destructive coordinate clicks.
- No global input *capture* unless explicitly enabled (keeps us out of Input
Monitoring by default).
- Per-action audit log; optional allowlist/denylist of target apps.
## Suggested build order
1. **Tier 1 (AX actions)** - biggest capability jump, enables background control.
2. **Tier 2 window mgmt + per-window screenshot** - "see and act on hidden windows".
3. **Tier 3 clipboard + AXObserver waits** - reliability.
4. **`run_applescript`/JXA bridge (Tier 4)** - headless scripting for many apps.
5. **Virtual-display / second-session (Tier 4)** - true parallel, non-interfering.
6. Signed jcode.app bundle for durable permissions.
7. Vision OCR (Tier 5) as needed.
## Crates
- `accessibility-sys` 0.2 (AX read/act/observe) - on crates.io
- `screencapturekit` 7 (modern capture) - on crates.io; `core-graphics` window list as fallback
- `objc2-app-kit` / `objc2-foundation` 0.3 - already in lockfile (NSWorkspace, NSPasteboard)
- `core-graphics` 0.23 - already a direct dep (CGEvent, CGWindowList, CGDisplay)
---
## Tool interface design (decided)
### Single tool, progressive disclosure
One `computer` tool, `action`-dispatched (like `browser`). To keep always-on
context flat regardless of how many tiers exist, the schema uses **progressive
disclosure**:
- **Always-on core (~370 tokens, measured with tiktoken cl100k_base):**
`screenshot, ui, ocr, click, type, key, press, set_value, run_applescript,
setup, discover`.
- **`discover { category }`** returns full specs for advanced actions on demand
(`mouse|keyboard|ax|windows|apps|clipboard|scripting|displays|system|all`),
~130 tokens per category, paid only when used.
- **Shared handle types** (`element`, `window_id`, `region`) defined once and
reused, so params do not multiply with actions.
Measured always-on cost:
| Design | Actions visible | Always-on tokens |
|---|---|---|
| Current v1 tool | 12 | ~720 |
| Flat, all tiers (~46 actions) | 46 | ~1,020 |
| **Progressive core** | 11 | **~370** |
Background control is a property of the *mechanism*, not the tier: CGEvent =
visible; **AX actions (press/set_value/select_menu) + Apple Events = background**.
### `setup` / `check_permissions` action
A first-class `setup` action that:
1. **Reports** status of every requirement: Accessibility (`AXIsProcessTrusted`),
Screen Recording (`CGPreflightScreenCaptureAccess`), Automation (per-app, via
first Apple Event), plus install/bundle health.
2. **Requests** what it can programmatically:
- `AXIsProcessTrustedWithOptions(prompt=true)` — shows the Accessibility dialog
and pre-adds jcode to the list (toggled off).
- `CGRequestScreenCaptureAccess()` — prompts for Screen Recording.
- First Apple Event to a target app — triggers its Automation prompt.
3. **Deep-links** to the exact System Settings pane for anything still missing:
- `x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility`
- `…?Privacy_ScreenCapture`
- `…?Privacy_Automation`
4. **Polls** `AXIsProcessTrusted()` until granted, then reports "ready".
**Hard limit:** the Accessibility *toggle itself cannot be flipped by any API*
(Apple anti-malware boundary). `tccutil` can only reset, not grant. So the best
achievable UX is **"one or two prompts + one toggle,"** never zero-touch.
### Durable permissions: signed app bundle
TCC permissions attach to the **running binary's identity**. A bare dev/cli binary
changes path/signature across updates, so macOS re-prompts every time. To make a
grant stick:
- Ship/install jcode as a **signed `.app` bundle with a stable bundle id**
(e.g. `com.jcode.app`) and a Designated Requirement, so the Accessibility /
Screen Recording grant persists across updates.
- `setup` should detect "running from an unstable/unsigned path" and offer to
install the proper bundle, so the user grants **once**.
### Build order (updated)
1. Progressive-disclosure refactor of the v1 tool (core + `discover`).
2. `setup` action (check + request + deep-link + poll).
3. Tier 1 AX actions (background control).
4. Tier 2 window/app management + per-window screenshot.
5. Tier 3 clipboard + AXObserver waits.
6. `run_applescript`/JXA bridge (Tier 4 headless scripting).
7. Signed app bundle for durable permissions.
8. Tier 5 OCR (Vision). (Camera/audio intentionally excluded.)
9. Virtual-display / second-session for true parallel work (advanced).
docs/proposals/computer-use-tool.md
+149
View File
@@ -0,0 +1,149 @@
# Proposal: native `computer` tool for macOS computer use
## Summary
Add a single native tool, **`computer`**, that lets the agent observe and control
the macOS GUI — screenshots, the accessibility (AX) tree, mouse/keyboard input,
window/app management, and clipboard — through one `action`-dispatched interface.
This mirrors the existing **`browser`** tool (`crates/jcode-app-core/src/tool/browser.rs`):
one registered tool, an `action: String` that selects a sub-operation, with optional
typed params. It gives jcode a closed control loop (*see screen → decide → act*)
without depending on a browser or external automation tooling.
## Motivation
- The agent can already drive a browser; it cannot drive native macOS apps, system
UI, or anything outside the browser sandbox.
- "Computer use" agents need exactly three primitives: **read the screen**, **read
UI structure**, and **synthesize input**. macOS exposes all three through the
Accessibility / Quartz Event Services / ScreenCaptureKit stack.
- A single, well-scoped tool keeps the tool surface small and the permission story
in one place.
## Architecture
```
crates/jcode-macos-control/ (new) cfg(target_os = "macos") platform crate
└─ AX (accessibility-sys), CGEvent (core-graphics),
CoreFoundation (core-foundation), screenshots (ScreenCaptureKit / CGDisplay),
app/window control (objc2 + objc2-app-kit), clipboard (objc2 NSPasteboard)
crates/jcode-app-core/src/tool/computer.rs (new) ComputerTool
└─ thin dispatch layer: parse input -> call jcode-macos-control -> ToolOutput
└─ registered in crates/jcode-app-core/src/tool/mod.rs base_tools()
```
- All native APIs are reached through existing Rust bindings (`objc2`,
`accessibility-sys`, `core-graphics`, `core-foundation`) — **no Swift/ObjC build
step**.
- On non-macOS targets the tool still registers but every action returns a clean
`unsupported on this platform` error, so the tool list stays stable across OSes.
- `screenshot` returns its image via `ToolOutput::with_image` (base64), matching how
`browser` returns screenshots today.
## Permissions (the important part)
macOS splits this across **four** TCC permissions. Programmatic *request* support
differs per permission:
| Permission | Used for | Programmatic request |
|---|---|---|
| **Accessibility** | drive other apps' UI, inject `CGEvent` input | ⚠️ prompt + deep-link only; user must toggle |
| **Screen Recording** | screenshots / `get_ui_tree` of some apps | ✅ `CGRequestScreenCaptureAccess()` |
| **Input Monitoring** | reading the global input stream | ✅ `IOHIDRequestAccess(...)` |
| **Automation** (Apple Events) | scripting cooperating apps | ✅ prompts on first send, per target app |
**Accessibility is the one that cannot be auto-granted** (Apple's anti-malware
boundary), and it is required for input injection. Best achievable flow, exposed via
the `request_permissions` action:
1. `AXIsProcessTrustedWithOptions([kAXTrustedCheckOptionPrompt: true])` — shows the
system dialog *and auto-adds jcode to the Accessibility list* (toggled off).
2. Deep-link to the exact pane:
`open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"`.
3. Poll `AXIsProcessTrusted()` until granted, then report ready.
So the experience becomes **one prompt + one toggle**, not "go hunt in Settings" —
but never zero-touch for Accessibility.
`check_permissions` reports the current state of all four so the agent can tell the
user precisely what is missing before attempting control.
## Actions
`action` (required) selects the operation. Params below are optional and validated
per action.
**Permissions**
- `check_permissions` → status of accessibility / screen-recording / input-monitoring
- `request_permissions` → prompt + deep-link flow above
**Observe**
- `screenshot` — `{ display?, window_id?, region? }` → image
- `get_ui_tree` — `{ pid? | frontmost, max_depth? }` → serialized AX tree (role, title, value, position, size, actions)
- `find_element` — `{ role?, title?, value?, pid? }` → matching elements + their identifiers
- `element_at` — `{ x, y }` → element under the point
**Mouse**
- `move` — `{ x, y }`
- `click` / `double_click` / `right_click` — `{ x?, y? }` (current position if omitted)
- `drag` — `{ from: [x,y], to: [x,y] }`
- `scroll` — `{ x?, y?, dx, dy }`
**Keyboard**
- `type` — `{ text }`
- `key` — `{ keys: "cmd+shift+4" }` (chord)
- `key_down` / `key_up` — `{ key }`
**Semantic AX (preferred over raw input when available)**
- `press` — `{ element }` (AXPress)
- `set_value` — `{ element, value }`
- `get_value` — `{ element }`
- `perform_action` — `{ element, ax_action }`
- `select_menu` — `{ app, path: ["File", "Export…"] }`
**Window / app**
- `list_apps`, `activate_app` `{ app }`
- `list_windows` `{ pid? }`, `focus_window` `{ window }`
- `move_window` `{ window, x, y }`, `resize_window` `{ window, w, h }`
- `minimize_window` / `close_window` `{ window }`
**Clipboard**
- `get_clipboard`, `set_clipboard` `{ text }`
> Element identifiers: `find_element` / `get_ui_tree` return stable-enough handles
> (e.g. `pid` + AX path or a session-scoped element id) that semantic actions accept,
> so the agent can act structurally instead of by pixel coordinates when possible.
## Safety
Computer use is high-blast-radius, so:
- **Permission-gated** like other powerful tools; refuses early with a clear message
if Accessibility/Screen Recording is missing.
- **`dry_run` param** on mutating actions — resolves and reports the target without
acting.
- **Screenshot-assisted confirmation** for destructive coordinate clicks (return the
region/element being targeted).
- **No global input *capture*** in v1 (we synthesize input but do not log the user's
keystrokes), keeping us out of Input Monitoring unless a future feature needs it.
- Prefer **semantic AX actions** over blind coordinate input wherever the element is
resolvable — more robust and more auditable.
## Implementation plan
1. `jcode-macos-control` crate: permissions, screenshot, AX read, AX action,
CGEvent input, window/app control, clipboard. Unit-test the pure parts
(input parsing, chord parsing, tree serialization).
2. `ComputerTool` in `tool/computer.rs`: input struct + `action` dispatch +
schema + description; register `"computer"` in `tool/mod.rs` `base_tools()`.
3. Default-off / gated rollout + docs in `docs/`.
4. Follow-up: Windows/Linux backends behind the same tool surface.
## Open questions
- Element handle format — `pid`+AX-path vs an opaque session-scoped id cache?
- Should `request_permissions` block-and-poll, or return immediately with status and
let the agent re-check?
- Default enablement: opt-in flag vs always-registered-but-gated?
docs/provider-model-catalog-audit.md
+61
View File
@@ -0,0 +1,61 @@
# OpenAI-compatible provider `/models` audit
Scope: built-in `OpenAiCompatibleProfile` entries in `crates/jcode-provider-metadata/src/lib.rs`.
Legend:
- `verified data[]`: docs show or explicitly reference OpenAI-compatible `GET /models` shape `{ object: "list", data: [...] }`, or the provider says it implements the OpenAI Models API.
- `verified top-level array`: docs show a top-level model array.
- `supported endpoint, shape not shown`: docs say `/models` exists or OpenAI compatibility includes it, but do not show the response body.
- `catalog/static only`: docs point to a static catalog/models page, not a live `/models` endpoint.
- `unknown`: could not find provider docs proving `/models`.
## Audited providers
| Provider | Evidence | Expected parser support | Notes |
|---|---|---:|---|
| OpenCode Zen | OpenCode docs and models.dev catalog | static/bootstrap, endpoint unverified | OpenCode itself uses models.dev. `/models` on Zen not independently verified. |
| OpenCode Go | OpenCode docs and models.dev catalog | static/bootstrap, endpoint unverified | Same as Zen. |
| Z.AI | Search did not find `/models`; provider docs URL 404 from current metadata path | unknown | Need direct current docs URL or live key-backed test. |
| Kimi Code | Kimi third-party-agent docs found, OpenAI Compatible config only | unknown | No `/models` response shape found. |
| 302.AI | Official docs contain `Models（列出模型）GET` page | likely OpenAI-compatible data[] | Dedicated list-models page exists. Response body in fetched text was truncated before example. |
| Baseten | Official docs say public OpenAI-compatible endpoint `https://inference.baseten.co/v1` | supported endpoint, shape not shown | No dedicated `/models` response found. |
| Cortecs | Official docs overview only plus OpenCode provider entry | catalog/static only | No `/models` endpoint docs found. |
| DeepSeek | Official `GET /models` docs show `{ object, data[] }` | verified data[] | Covered by parser. |
| Comtegra | Official docs list supported `/v1/models` linking OpenAI Models API | supported endpoint, shape OpenAI | Covered by parser. |
| FPT AI Marketplace | Official docs show chat/completions through LiteLLM/OpenAI, no models endpoint | unknown/no evidence | Live `/models` may fail. |
| Firmware/FrogBot | OpenCode provider docs only | catalog/static only | No direct provider API docs found. |
| Hugging Face | General Inference Providers docs, OpenAI-compatible API | supported endpoint, shape not shown | No dedicated `/models` page verified. |
| Moonshot AI | Search/current URL did not expose `/models` docs | unknown | Kimi API search hints model list endpoint, but no official Moonshot page fetched. |
| Nebius | Quickstart docs OpenAI-compatible endpoint | supported endpoint, shape not shown | Dedicated `/models` page not verified. |
| Scaleway | Official “Using Models API” docs found | supported endpoint, shape likely OpenAI | Covered by parser if OpenAI shape. |
| STACKIT | Official integration docs say OpenAI-compatible API and model picker fetches `/models` | supported endpoint, shape not shown | Covered if OpenAI shape. |
| Groq | API reference has Models/List models | verified data[] | Covered. |
| Mistral | API reference has Models/List Available Models | verified data[] style | Covered. |
| Perplexity | API docs fetch/search did not find list-models endpoint | unknown | May not support `/models`; static docs list models. |
| Together AI | Official `GET /models` docs show top-level array | verified top-level array | Parser was fixed for this. |
| DeepInfra | Official OpenAI-compatible docs point to static model catalog, no `/models` page found | catalog/static only | Live `/models` unverified. |
| Fireworks | Official list-models docs found for account model API `{ models: [...] }`; OpenAI compat endpoint also exists | verified models[] variant for account API | Parser supports `models[]` and `name`. Need live base endpoint shape still unverified. |
| MiniMax | Official text generation docs show OpenAI-compatible base and static supported-models table | catalog/static only | No `/models` endpoint found. |
| xAI | API reference includes Models section | verified data[] likely | Covered. |
| LM Studio | Official OpenAI compatibility docs list `GET /v1/models` | supported endpoint, shape not shown | OpenAI local server expected data[]. |
| Ollama | Official OpenAI compatibility blog/docs cover chat; `/v1/models` docs not found in fetched page | unknown | Need raw docs/source or live local test. |
| Chutes | Live user response showed `{ object:"list", data:[...] }` with numeric pricing | verified data[] plus numeric pricing | Parser fixed and stale default removed. |
| Cerebras | Official `GET /v1/models` docs show `{ object, data[] }` | verified data[] | Covered. |
| Alibaba Coding Plan | Official docs show OpenAI-compatible base URL but warn Coding Plan is for coding tools only; no `/models` docs | unknown/no evidence | Static default likely needed; live `/models` may fail. |
| Generic openai-compatible | User-supplied endpoint | parser contract | We support `{data[]}`, top-level array, `{models[]}`, id/name identifiers. |
## Parser coverage after `f291f0e`
Supported response forms:
- `{ "data": [{ "id": "..." }] }`
- top-level `[{ "id": "..." }]`
- `{ "models": [{ "id" or "name": "..." }] }`
- numeric or string pricing fields
- context fields: `context_length`, `contextLength`, `max_context_length`, `maxModelLength`, `max_model_len`, `trainingContextLength`
## Gaps identified
No additional parser shape is proven necessary yet. The remaining issue is provider capability/profile accuracy:
- Some providers are OpenAI-compatible for chat but do not document live `GET /models`.
- For those, live catalog refresh should remain best-effort and must gracefully fall back to static catalog.
- Long-term, `OpenAiCompatibleProfile` should probably carry a `model_catalog` capability/strategy so providers known not to support `/models` do not emit noisy refresh failures.
docs/reddit_dashboard.py
+310
View File
@@ -0,0 +1,310 @@
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from matplotlib.gridspec import GridSpec
from matplotlib.patches import Patch
sns.set_theme(style="darkgrid")
plt.rcParams.update({
'figure.facecolor': '#0d1117',
'axes.facecolor': '#161b22',
'text.color': '#e6edf3',
'axes.labelcolor': '#e6edf3',
'xtick.color': '#8b949e',
'ytick.color': '#8b949e',
'grid.color': '#21262d',
'axes.edgecolor': '#30363d',
'font.family': 'sans-serif',
'font.size': 11,
})
# === ALL SUBREDDITS (consistent across every chart) ===
data = {
'r/ClaudeAI': {'subs': 509, 'avg_up': 8, 'avg_com': 5.2, 'max_up': 113, 'relevance': 10, 'tier': 1},
'r/ChatGPTCoding': {'subs': 357, 'avg_up': 46, 'avg_com': 25.0, 'max_up': 883, 'relevance': 9, 'tier': 1},
'r/LocalLLaMA': {'subs': 628, 'avg_up': 3, 'avg_com': 4.2, 'max_up': 29, 'relevance': 9, 'tier': 1},
'r/cursor': {'subs': 122, 'avg_up': 12, 'avg_com': 13.1, 'max_up': 129, 'relevance': 8, 'tier': 2},
'r/rust': {'subs': 388, 'avg_up': 19, 'avg_com': 8.7, 'max_up': 133, 'relevance': 8, 'tier': 2},
'r/commandline': {'subs': 115, 'avg_up': 13, 'avg_com': 4.4, 'max_up': 230, 'relevance': 8, 'tier': 2},
'r/opensource': {'subs': 326, 'avg_up': 18, 'avg_com': 6.1, 'max_up': 185, 'relevance': 7, 'tier': 3},
'r/neovim': {'subs': 148, 'avg_up': 28, 'avg_com': 11.9, 'max_up': 278, 'relevance': 7, 'tier': 3},
'r/selfhosted': {'subs': 698, 'avg_up': 18, 'avg_com': 9.0, 'max_up': 248, 'relevance': 6, 'tier': 3},
'r/ollama': {'subs': 101, 'avg_up': 9, 'avg_com': 5.3, 'max_up': 84, 'relevance': 6, 'tier': 3},
'r/programming': {'subs': 6841,'avg_up': 125, 'avg_com': 26.4, 'max_up': 2566, 'relevance': 5, 'tier': 4},
'r/linux': {'subs': 1821,'avg_up': 161, 'avg_com': 32.0, 'max_up': 890, 'relevance': 5, 'tier': 4},
'r/artificial': {'subs': 1224,'avg_up': 60, 'avg_com': 24.0, 'max_up': 569, 'relevance': 5, 'tier': 4},
'r/MachineLearning':{'subs':3024,'avg_up': 23, 'avg_com': 11.7, 'max_up': 208, 'relevance': 5, 'tier': 4},
'r/SideProject': {'subs': 629, 'avg_up': 1, 'avg_com': 0.6, 'max_up': 4, 'relevance': 4, 'tier': 5},
}
tier_palette = {1: '#58a6ff', 2: '#3fb950', 3: '#d2a8ff', 4: '#f0883e', 5: '#f85149'}
tier_names = {1: 'Tier 1: Perfect Fit', 2: 'Tier 2: Strong Fit', 3: 'Tier 3: Good Fit',
4: 'Tier 4: Broad Reach', 5: 'Tier 5: Skip'}
subs_list = list(data.keys())
colors = [tier_palette[data[s]['tier']] for s in subs_list]
# Hour data (Pacific) — ALL subs
hour_data = {
'r/ClaudeAI': [4,5,8,8,5,11,13,11,12,10,11,2,0,0,0,0,0,0,0,0,0,0,0,0],
'r/ChatGPTCoding': [1,4,9,4,4,3,3,5,4,5,8,7,7,5,4,2,7,4,4,2,2,1,2,3],
'r/LocalLLaMA': [0,3,2,5,6,6,5,12,11,6,7,8,0,0,0,0,3,6,7,0,6,4,3,0],
'r/cursor': [2,1,5,1,5,5,8,8,8,7,9,4,9,2,5,2,2,4,3,1,4,2,2,1],
'r/rust': [4,5,8,6,5,7,6,9,8,7,8,4,5,2,2,2,3,2,2,1,1,2,1,0],
'r/commandline': [1,3,6,3,7,9,8,3,4,6,3,4,4,8,6,5,1,3,2,4,4,2,1,3],
'r/opensource': [1,3,7,3,6,1,4,4,9,5,6,3,7,10,5,7,6,3,1,0,3,2,2,2],
'r/neovim': [4,5,5,3,3,6,8,1,4,3,8,8,3,4,7,2,4,7,2,1,3,4,3,2],
'r/selfhosted': [3,3,5,4,7,5,4,6,8,11,8,7,3,3,2,5,2,2,2,2,3,3,1,1],
'r/ollama': [2,6,6,6,3,2,4,7,10,6,5,5,8,5,2,3,6,3,2,2,2,1,2,2],
'r/programming': [3,3,4,7,7,11,8,6,6,6,2,10,4,1,1,2,5,1,2,0,2,4,3,2],
'r/linux': [3,4,6,2,4,3,3,1,4,7,7,10,6,7,5,6,5,5,5,1,1,2,2,1],
'r/artificial': [2,4,8,5,5,0,5,7,7,5,4,6,3,8,3,3,2,3,6,2,4,3,3,2],
'r/MachineLearning':[7,3,2,3,4,4,7,1,11,6,3,7,4,5,6,3,3,6,3,1,2,2,4,3],
'r/SideProject': [0,0,0,0,0,0,0,0,13,11,18,13,11,8,10,11,5,0,0,0,0,0,0,0],
}
# Day data
day_names = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
day_data = {
'r/ChatGPTCoding': [19,17,15,12,12,9,16],
'r/cursor': [26,21,12,0,0,21,20],
'r/commandline': [19,13,15,12,13,11,17],
'r/opensource': [14,20,11,11,17,13,14],
'r/neovim': [19,17,16,9,7,9,23],
'r/rust': [23,39,20,0,0,0,18],
'r/ollama': [11,20,18,15,16,8,12],
'r/linux': [8,24,25,11,13,11,8],
'r/artificial': [18,22,16,10,13,11,10],
'r/MachineLearning':[21,23,11,7,14,12,12],
}
# ======================== FIGURE ========================
fig = plt.figure(figsize=(24, 36))
fig.suptitle('jcode Reddit Strategy Dashboard', fontsize=28, fontweight='bold',
color='#58a6ff', y=0.985)
fig.text(0.5, 0.979, 'Complete analysis of 15 subreddits for promoting jcode (Rust AI coding agent CLI) | All times Pacific',
ha='center', fontsize=13, color='#8b949e')
gs = GridSpec(6, 2, figure=fig, hspace=0.32, wspace=0.28,
top=0.965, bottom=0.02, left=0.09, right=0.95)
# ---- LEGEND (shared) ----
legend_elements = [Patch(facecolor=tier_palette[t], label=tier_names[t]) for t in [1,2,3,4,5]]
# ========== 1. COMPOSITE RANKING ==========
ax1 = fig.add_subplot(gs[0, :])
composite = []
for s in subs_list:
d = data[s]
composite.append(d['relevance'] * (d['avg_up'] + d['avg_com'] * 2))
sort_idx = np.argsort(composite)[::-1]
sorted_subs = [subs_list[i] for i in sort_idx]
sorted_composite = [composite[i] for i in sort_idx]
sorted_colors = [colors[i] for i in sort_idx]
bars = ax1.barh(range(len(sorted_subs)), sorted_composite, color=sorted_colors, edgecolor='none', height=0.7)
ax1.set_yticks(range(len(sorted_subs)))
ax1.set_yticklabels(sorted_subs, fontsize=11, fontweight='bold')
ax1.invert_yaxis()
ax1.set_xlabel('Composite Score = Relevance x (Avg Upvotes + 2 x Avg Comments)', fontsize=11)
ax1.set_title('OVERALL RANKING', fontsize=18, fontweight='bold', pad=15)
for i, (bar, val) in enumerate(zip(bars, sorted_composite)):
ax1.text(val + max(sorted_composite)*0.01, i, f'{val:.0f}', va='center', fontsize=10, color='#e6edf3')
ax1.legend(handles=legend_elements, loc='lower right', fontsize=9, facecolor='#161b22', edgecolor='#30363d')
# ========== 2. SUBSCRIBERS vs ENGAGEMENT ==========
ax2 = fig.add_subplot(gs[1, 0])
subs_k = [data[s]['subs'] for s in subs_list]
avg_up = [data[s]['avg_up'] for s in subs_list]
relevances = [data[s]['relevance'] for s in subs_list]
sizes = [r*35 for r in relevances]
ax2.scatter(subs_k, avg_up, s=sizes, c=colors, alpha=0.85, edgecolors='white', linewidth=0.5, zorder=5)
for i, s in enumerate(subs_list):
ax2.annotate(s.replace('r/', ''), (subs_k[i], avg_up[i]), fontsize=7.5, color='#c9d1d9',
xytext=(6, 4), textcoords='offset points')
ax2.set_xlabel('Subscribers (K)', fontsize=11)
ax2.set_ylabel('Avg Upvotes per Post', fontsize=11)
ax2.set_title('SUBSCRIBERS vs ENGAGEMENT', fontsize=14, fontweight='bold')
ax2.set_xscale('log')
# ========== 3. AVG COMMENTS ==========
ax3 = fig.add_subplot(gs[1, 1])
avg_com = [data[s]['avg_com'] for s in subs_list]
sort_c = np.argsort(avg_com)[::-1]
ax3.barh(range(len(subs_list)), [avg_com[i] for i in sort_c],
color=[colors[i] for i in sort_c], edgecolor='none', height=0.65)
ax3.set_yticks(range(len(subs_list)))
ax3.set_yticklabels([subs_list[i] for i in sort_c], fontsize=10)
ax3.invert_yaxis()
ax3.set_xlabel('Avg Comments per Post', fontsize=11)
ax3.set_title('DISCUSSION DEPTH', fontsize=14, fontweight='bold')
for i, idx in enumerate(sort_c):
ax3.text(avg_com[idx] + 0.3, i, f'{avg_com[idx]:.1f}', va='center', fontsize=9, color='#8b949e')
# ========== 4. HEATMAP — ALL SUBS ==========
ax4 = fig.add_subplot(gs[2, :])
heat_subs = list(hour_data.keys())
heat_matrix = np.array([hour_data[s] for s in heat_subs], dtype=float)
# Normalize each row
row_sums = heat_matrix.sum(axis=1, keepdims=True)
row_sums[row_sums == 0] = 1
heat_norm = heat_matrix / row_sums * 100
im = ax4.imshow(heat_norm, cmap='YlOrRd', aspect='auto', interpolation='nearest')
ax4.set_yticks(range(len(heat_subs)))
ax4.set_yticklabels(heat_subs, fontsize=10)
ax4.set_xticks(range(24))
ax4.set_xticklabels([f'{h}' for h in range(24)], fontsize=9)
ax4.set_xlabel('Hour of Day (Pacific Time)', fontsize=12)
ax4.set_title('POSTING ACTIVITY HEATMAP BY HOUR', fontsize=16, fontweight='bold', pad=12)
cbar = plt.colorbar(im, ax=ax4, shrink=0.5, pad=0.02)
cbar.set_label('% of posts in that hour', color='#8b949e', fontsize=10)
cbar.ax.yaxis.set_tick_params(color='#8b949e')
plt.setp(plt.getp(cbar.ax.axes, 'yticklabels'), color='#8b949e')
# Mark peak hour per sub
for i in range(len(heat_subs)):
row = heat_norm[i]
if row.max() > 0:
peak_h = np.argmax(row)
ax4.text(peak_h, i, '*', ha='center', va='center', fontsize=16, color='black', fontweight='bold')
# Add morning/afternoon/evening labels
ax4.axvline(x=5.5, color='#58a6ff', linewidth=0.5, alpha=0.4, linestyle='--')
ax4.axvline(x=11.5, color='#f0883e', linewidth=0.5, alpha=0.4, linestyle='--')
ax4.axvline(x=17.5, color='#d2a8ff', linewidth=0.5, alpha=0.4, linestyle='--')
ax4.text(2.5, -0.8, 'Late Night', ha='center', fontsize=8, color='#8b949e')
ax4.text(8.5, -0.8, 'Morning', ha='center', fontsize=8, color='#58a6ff')
ax4.text(14.5, -0.8, 'Afternoon', ha='center', fontsize=8, color='#f0883e')
ax4.text(20.5, -0.8, 'Evening', ha='center', fontsize=8, color='#d2a8ff')
# ========== 5. DAY OF WEEK ==========
ax5 = fig.add_subplot(gs[3, 0])
day_subs_list = list(day_data.keys())
x = np.arange(7)
n = len(day_subs_list)
width = 0.8 / n
cmap_day = plt.cm.Set2
for i, sub in enumerate(day_subs_list):
vals = day_data[sub]
total = sum(vals)
if total == 0: continue
pcts = [v/total*100 for v in vals]
c = tier_palette[data[sub]['tier']]
ax5.bar(x + i*width - n*width/2, pcts, width, label=sub.replace('r/', ''), color=c, alpha=0.7, edgecolor='none')
ax5.set_xticks(x)
ax5.set_xticklabels(day_names, fontsize=11, fontweight='bold')
ax5.set_ylabel('% of posts', fontsize=11)
ax5.set_title('DAY OF WEEK DISTRIBUTION', fontsize=14, fontweight='bold')
ax5.legend(fontsize=6.5, facecolor='#161b22', edgecolor='#30363d', loc='upper right', ncol=2)
# ========== 6. VIRAL POTENTIAL ==========
ax6 = fig.add_subplot(gs[3, 1])
max_up = [data[s]['max_up'] for s in subs_list]
sort_m = np.argsort(max_up)[::-1]
ax6.barh(range(len(subs_list)), [max_up[i] for i in sort_m],
color=[colors[i] for i in sort_m], edgecolor='none', height=0.65)
ax6.set_yticks(range(len(subs_list)))
ax6.set_yticklabels([subs_list[i] for i in sort_m], fontsize=10)
ax6.invert_yaxis()
ax6.set_xlabel('Max Upvotes (Recent Posts)', fontsize=11)
ax6.set_title('VIRAL POTENTIAL', fontsize=14, fontweight='bold')
for i, idx in enumerate(sort_m):
ax6.text(max_up[idx] + 20, i, f'{max_up[idx]:,}', va='center', fontsize=9, color='#8b949e')
# ========== 7. BEST TIME TO POST (visual timeline) ==========
ax_time = fig.add_subplot(gs[4, :])
best_times = {
'r/ClaudeAI': (6, 9, 'Tue-Wed'),
'r/ChatGPTCoding': (10, 12, 'Monday'),
'r/LocalLLaMA': (7, 9, 'Tue-Wed'),
'r/cursor': (10, 12, 'Monday'),
'r/rust': (7, 10, 'Tuesday'),
'r/commandline': (5, 6, 'Mon/Sun'),
'r/opensource': (8, 13, 'Tue/Fri'),
'r/neovim': (6, 11, 'Sun/Mon'),
'r/selfhosted': (8, 10, 'Tuesday'),
'r/ollama': (8, 12, 'Tuesday'),
'r/programming': (5, 8, 'Monday'),
'r/linux': (9, 12, 'Tue-Wed'),
'r/artificial': (7, 9, 'Tuesday'),
'r/MachineLearning':(8, 11, 'Tuesday'),
'r/SideProject': (9, 12, 'Wed'),
}
y_positions = list(range(len(best_times)))
for i, (sub, (start, end, day)) in enumerate(best_times.items()):
c = tier_palette[data[sub]['tier']]
ax_time.barh(i, end - start, left=start, height=0.6, color=c, alpha=0.85, edgecolor='white', linewidth=0.5)
ax_time.text(end + 0.3, i, f'{start}am-{end}{"pm" if end >= 12 else "am"} ({day})',
va='center', fontsize=9, color='#c9d1d9')
ax_time.set_yticks(y_positions)
ax_time.set_yticklabels(list(best_times.keys()), fontsize=10)
ax_time.invert_yaxis()
ax_time.set_xlim(0, 24)
ax_time.set_xticks(range(0, 25, 2))
ax_time.set_xticklabels([f'{h}:00' for h in range(0, 25, 2)], fontsize=9)
ax_time.set_xlabel('Pacific Time', fontsize=12)
ax_time.set_title('BEST TIME TO POST (Pacific)', fontsize=16, fontweight='bold', pad=12)
ax_time.axvline(x=8, color='#3fb950', linewidth=1, alpha=0.3, linestyle='--')
ax_time.axvline(x=12, color='#f0883e', linewidth=1, alpha=0.3, linestyle='--')
ax_time.legend(handles=legend_elements, loc='upper right', fontsize=8, facecolor='#161b22', edgecolor='#30363d')
# ========== 8. STRATEGY TABLE ==========
ax7 = fig.add_subplot(gs[5, :])
ax7.axis('off')
schedule = [
['Subreddit', 'Subs', 'Best Day', 'Best Time', 'Approach', 'Score'],
['r/ClaudeAI', '509K', 'Tue-Wed', '6-9am', '"Built with Claude" flair', '675'],
['r/ChatGPTCoding', '357K', 'Monday', '10am-12pm','Demo video, compare to Cursor', '675'],
['r/LocalLLaMA', '628K', 'Tue-Wed', '7-9am', 'Technical deep-dive, OSS angle', '103'],
['r/cursor', '122K', 'Monday', '10am-12pm','CLI alternative to Cursor', '305'],
['r/rust', '388K', 'Tuesday', '7-10am', 'Project flair, Rust internals', '297'],
['r/commandline', '115K', 'Mon/Sun', '5am / 1pm','GIF demo, CLI showcase', '174'],
['r/neovim', '148K', 'Sunday', '6am/10am', 'Terminal-first, vim integration', '363'],
['r/opensource', '326K', 'Tue/Fri', '8am / 1pm','OSS launch announcement', '212'],
['r/selfhosted', '698K', 'Tuesday', '8-10am', 'Self-hostable AI coding agent', '216'],
['r/ollama', '101K', 'Tuesday', '8am-12pm', 'Local model integration angle', '95'],
['r/programming', '6.8M', 'Monday', '5-8am', 'Blog post / deep technical', '888'],
['r/linux', '1.8M', 'Tue-Wed', '9am-12pm', 'Linux-native CLI tool angle', '1125'],
['r/artificial', '1.2M', 'Tuesday', '7-9am', 'AI agent capabilities showcase', '540'],
['r/MachineLearning','3M', 'Tuesday', '8-11am', 'Technical architecture post', '233'],
['r/SideProject', '629K', 'Any', '9am-3pm', 'SKIP - zero engagement', '9'],
]
table = ax7.table(cellText=schedule[1:], colLabels=schedule[0],
cellLoc='center', loc='center',
colColours=['#21262d']*6)
table.auto_set_font_size(False)
table.set_fontsize(10)
table.scale(1, 1.6)
# Color the table
tier_for_sub = {s: data[s]['tier'] for s in data}
for key, cell in table.get_celld().items():
row, col = key
cell.set_edgecolor('#30363d')
if row == 0:
cell.set_facecolor('#21262d')
cell.set_text_props(color='#58a6ff', fontweight='bold', fontsize=11)
else:
sub_name = schedule[row][0]
if sub_name in tier_for_sub:
tier = tier_for_sub[sub_name]
cell.set_facecolor('#0d1117')
if col == 0:
cell.set_text_props(color=tier_palette[tier], fontweight='bold')
else:
cell.set_text_props(color='#e6edf3')
else:
cell.set_facecolor('#0d1117')
cell.set_text_props(color='#e6edf3')
ax7.set_title('COMPLETE POSTING STRATEGY', fontsize=18, fontweight='bold', pad=20, color='#58a6ff')
plt.savefig('/tmp/jcode_reddit_dashboard.png', dpi=150, bbox_inches='tight',
facecolor='#0d1117', edgecolor='none')
print("Saved to /tmp/jcode_reddit_dashboard.png")