nexu-io--open-design/specs/current/ai-native-observability-loop.md

AI Native Observability Loop

Purpose

Define the target observability loop that lets Open Design improve agent quality, reliability, latency, and cost from production evidence.

This is a planning spec. It does not implement runtime behavior. It starts from the existing Langfuse trace forwarding and PostHog run analytics, then defines the target loop that adds datasets, experiments, annotation, and release gates around those signals.

Issue: #3713

Current State

Open Design already reports completed runs to Langfuse and PostHog. The current implementation captures useful operational facts:

• trace identity: run_id == langfuse_trace_id == traceId;
• run status, error code, failure category, failure detail, retryability, and user action;
• timing fields for queue, prompt build, spawn, first token, generation, tool aggregate, finalize, and total duration;
• token and cache fields for provider input, effective input, cache read/write, uncached input, estimated context tokens, and cache source;
• tags and metadata for agent, model, skill, design system, runtime, app channel, operating system, and client type;
• user feedback scores such as user_rating and user_rating_reason.

The current Langfuse tree is still mostly runtime-oriented. A typical trace is shaped like:

open-design-turn
  agent-run
    llm
    tool:read
    tool:write
    tool:todowrite

This answers "what did the subprocess and tools do?" It does not consistently answer "which product task stage made the run succeed, fail, become slow, become expensive, or require human judgment?"

The current project also has far more traces than durable evaluation assets. That means Open Design can inspect individual runs, but the system does not yet turn production evidence into enough datasets, experiments, annotation queues, and release gates to drive continuous improvement.

The storage foundation is partly in place but not yet connected to this observability boundary:

• project files, uploads, pasted content, generated files, and live artifacts currently live under the local project filesystem rooted at <dataDir>/projects/<projectId>;
• saved standalone HTML artifacts live under <dataDir>/artifacts and are served through the daemon's static /artifacts/* route;
• media generation writes its output into the same project file directory;
• ProjectStorage, LocalProjectStorage, and S3ProjectStorage already define a read/write/list/delete/stat adapter contract for S3-compatible stores, but the current project file and artifact routes still call the local filesystem helpers directly;
• Cloudflare R2 / S3-compatible storage is already used for release artifacts, visual-regression assets, landing-page static assets, and mock recordings, so the repository has S3-compatible operational precedent outside the product runtime.

The gap is therefore not "does Open Design know how to talk to object storage?" The gap is productizing that substrate for user attachments, generated artifacts, access control, manifest indexing, retention, signed inspection URLs, and Langfuse-safe observability payloads.

Target Model

Open Design should treat observability as a closed loop:

production trace
  -> semantic task observations
  -> automatic scores
  -> agent triage and clustering
  -> dataset or annotation candidate
  -> experiment and regression gate
  -> optimization proposal
  -> human approval when risk is high
  -> release
  -> online observation

Langfuse is the trace, score, dataset, experiment, and annotation surface. It is the observability index, not the durable storage layer for user attachments or generated artifacts. PostHog remains the aggregate product analytics and alerting surface. Open Design owns the domain model that maps a design-agent run onto task stages, quality signals, storage references, and release decisions.

Semantic Trace Shape

The target trace tree should expose product stages in addition to low-level runtime spans:

open-design-turn
  brief-intake
    attachment-manifest
  route-task-kind
  resolve-context
    resolve-skill
    resolve-design-system
    resolve-memory
    resolve-plugin
  build-prompt-stack
  spawn-agent
  agent-work
    plan
    generate-artifact
    edit-artifact
    tool-call
  verify-artifact
    preview-render
    artifact-manifest
    export-check
  critique
  repair
  evaluator
  finalize

The exact implementation can preserve the existing agent-run, llm, and tool:* observations, but those observations should sit inside or alongside semantic task stages so humans and agents can diagnose by product intent.

Heavy Input and Artifact Boundary

Open Design is a design product, so the input and output surface is heavier than a short chat prompt. A run can include long briefs, reference images, PDFs, brand files, prior project files, generated HTML, images, decks, documents, preview screenshots, export bundles, and reproducibility materials. These objects should not be copied wholesale into Langfuse input/output fields.

The target boundary is:

Layer	Responsibility
Open Design object storage	Stores original user attachments, parsed text, thumbnails, OCR or embedding outputs, generated artifacts, preview screenshots, export bundles, and reproducibility files.
Langfuse	Stores trace-safe manifests, summaries, hashes, stage status, storage references, evaluator scores, usage, latency, and cost fields.
PostHog	Stores aggregate product analytics and user/workspace-level trend metrics derived from trace-safe fields.

Langfuse should retain enough information for analysis, triage, and replay without becoming the source of truth for sensitive or large user data. The default policy is:

• do not upload original attachments or artifact packages to Langfuse by default;
• always emit an attachment_manifest and artifact_manifest with stable ids, MIME type, byte size, SHA-256 hash, parse/build status, truncation status, redaction status, and Open Design storage references;
• include short summaries and product-analysis fields that are safe to index in Langfuse;
• use short-lived signed URLs only for interactive inspection, not as the durable reference stored in Langfuse;
• store durable od://-style references, object ids, or artifact ids that Open Design can resolve after authorization;
• only upload a raw attachment to Langfuse Media when the file type is supported, the data is low sensitivity or explicitly allowed, and replaying the trace requires Langfuse-native media resolution;
• run masking/redaction before any input, output, or metadata leaves Open Design.

The minimum manifest shape should be stable across traces:

{
  "attachment_manifest": [
    {
      "attachment_id": "att_123",
      "role": "reference_image",
      "mime_type": "image/png",
      "size_bytes": 2481032,
      "sha256": "sha256:...",
      "parse_status": "ok",
      "summary": "Brand reference image with dark UI styling.",
      "redacted": false,
      "truncated": false,
      "stored_in_open_design": true,
      "langfuse_media_id": null,
      "storage_ref": "od://objects/workspaces/ws_1/projects/proj_1/runs/run_1/attachment/att_123"
    }
  ],
  "artifact_manifest": [
    {
      "artifact_id": "art_456",
      "type": "html_prototype",
      "mime_type": "text/html",
      "size_bytes": 482193,
      "sha256": "sha256:...",
      "build_status": "ok",
      "preview_status": "ok",
      "export_status": "ok",
      "summary": "Generated HTML prototype for the requested dashboard flow.",
      "redacted": false,
      "truncated": false,
      "storage_ref": "od://objects/workspaces/ws_1/projects/proj_1/runs/run_1/artifact/art_456",
      "thumbnail_ref": "od://objects/workspaces/ws_1/projects/proj_1/runs/run_1/preview/prev_thumb_789"
    }
  ]
}

This boundary resolves the conflict between Langfuse truncation and incomplete product analytics: Langfuse gets complete trace-safe semantics, while Open Design keeps complete raw data behind product authorization and retention policies.

Attachment and Artifact Registry

Open Design should add a registry layer above ProjectStorage instead of letting traces, filesystem paths, and UI URLs each invent their own storage semantics. The registry is the product source of truth for durable input and output objects; Langfuse stores only registry-derived manifest fields.

Recommended object classes:

Object class	Examples	Registry owner
attachment	Uploaded images, PDFs, brand files, reference assets, pasted files.	User/project input pipeline.
parsed_attachment	OCR text, extracted PDF text, thumbnails, embeddings, image descriptors.	Attachment ingest worker.
artifact	HTML prototypes, images, decks, documents, audio/video, code packages.	Agent/tool output pipeline.
preview	Rendered preview HTML, screenshots, thumbnails, validation captures.	Preview/verification pipeline.
export	Downloadable bundles, handoff packages, deployment artifacts.	Export/finalize pipeline.
replay	Minimal reproduction bundle, prompt stack snapshot, selected logs.	Observability/evaluation pipeline.

Each registry record should have a stable id, object class, workspace/project/run scope, storage backend, storage key, content hash, MIME type, byte size, origin, retention policy, sensitivity class, redaction state, parse/build status, preview/export status when applicable, and provenance back to the source trace or run.

The storage reference in Langfuse should be durable and non-secret:

od://objects/workspaces/<workspaceId>/projects/<projectId>/runs/<runId>/<objectClass>/<objectId>

The daemon should resolve that reference through product authorization and then, only for interactive viewing or replay, issue a short-lived signed URL or stream the object through a guarded daemon endpoint. Long-lived signed URLs should not be written to Langfuse, PostHog, datasets, or PR comments.

The registry should use the existing ProjectStorage contract as the first storage backend boundary, but the product routes need one more layer:

HTTP / CLI / tool route
  -> AttachmentRegistry or ArtifactRegistry
  -> ProjectStorage backend (local by default, s3 when configured)
  -> object manifest event
  -> Langfuse trace-safe metadata

This keeps local-first desktop usage intact while making cloud-backed storage an environment choice rather than a product model fork.

Score Model

A score-eligible run is a terminal production agent run or accepted-dataset experiment/evaluation run with enough trace metadata to identify trace id, dataset item id when applicable, task kind, agent, model, final status, and timing/token baselines. Local smoke tests and harness-only runs are excluded unless they are recorded as accepted-dataset evaluation runs. Each eligible run should receive automatic scores where the signal can be computed safely:

Score	Meaning
task_success	The run produced a useful terminal result for the requested task.
artifact_valid	Required artifact files and manifest entries exist and are readable.
preview_ok	The primary preview renders without fatal runtime errors.
user_request_covered	The output appears to address the explicit user request.
design_quality	Automated or rubric-based critique result for visual/product quality.
stability_risk	Failure, retry, timeout, or brittle-runtime risk bucket.
cost_bucket	Cost/token use relative to task kind and model baseline.
latency_bucket	Latency relative to task kind and agent/model baseline.
user_rating	Human end-user feedback, already reported from the UI.

Score applicability must be explicit so dashboards and datasets do not confuse not-applicable scores with failures or evaluator gaps. When a score does not apply, Open Design should not write a score value; it should record score_applicability.<score> = "not_applicable". When a score applies but cannot be computed because required trace fields or evaluator inputs are missing, it should record score_applicability.<score> = "insufficient_signal" and treat the missing score as an observability gap, not as a failed run.

Accepted-dataset experiment and evaluation runs must reuse the same score fields, score applicability values, and latency/cost bucket definitions as production runs. Fixed datasets are the release-gate baseline; other accepted datasets use the same fields for candidate improvement analysis instead of a separate QA-only scoring schema.

Score	Applies to
task_success	All score-eligible production and accepted-dataset experiment/evaluation runs with a terminal result.
artifact_valid	Artifact-producing score-eligible production and accepted-dataset experiment/evaluation tasks that create, edit, export, or inspect project files with a required manifest or file contract.
preview_ok	Artifact-producing score-eligible production and accepted-dataset experiment/evaluation tasks whose primary output has a supported preview renderer.
user_request_covered	Production and accepted-dataset experiment/evaluation tasks with a natural-language request or expected outcome and terminal output to compare.
design_quality	Score-eligible production and accepted-dataset experiment/evaluation outputs for visual, product, deck, prototype, image, video, audio, or document tasks with an applicable rubric.
stability_risk	All eligible runs.
cost_bucket	All score-eligible runs with model and token metadata.
latency_bucket	All score-eligible runs with timing metadata.
user_rating	Runs where the user submitted explicit feedback.

Automatic scores should start conservative. Deterministic evaluators are preferred before LLM-as-judge. Human feedback and annotation can later calibrate or replace weak automatic scores.

Dataset and Experiment Model

Production traces should become evaluation assets through an explicit promotion path:

• failed traces with clear root cause become regression dataset candidates;
• low-score traces become quality dataset candidates;
• high-cost or high-latency traces become efficiency dataset candidates;
• ambiguous traces enter annotation queues before promotion;
• accepted dataset items retain source trace id, task kind, agent, model, skill, design system, prompt stack fingerprint, trace-safe attachment/artifact manifests, and expected outcome.

Experiments should run the same dataset against candidate changes such as:

• prompt stack edits;
• skill or design-system changes;
• agent/model routing changes;
• retry and runtime policy changes;
• context compression or cacheability changes;
• evaluator or guardrail changes.

An experiment is successful only when quality, reliability, latency, and cost are considered together. A lower-cost run that fails the task is not an improvement.

Fixed Dataset Trust Contract

Fixed datasets are the trust anchor for human-agent collaboration. Agents can discover issues, propose prompt/skill/model/runtime changes, and run the experiments, but humans should not have to accept those proposals on narrative confidence alone. A fixed dataset gives the team a shared, repeatable evidence surface before a change reaches production.

For every optimization proposal, the agent should report:

• which fixed dataset or approved provisional baseline was used;
• which candidate changed, including git sha, agent/model routing, prompt stack fingerprint, skill/design-system versions, and evaluator version;
• pass/fail status for each blocking quality, reliability, latency, and cost metric;
• the concrete traces, dataset items, artifacts, and annotations behind any failure or tradeoff;
• whether any sensitive attachment or generated artifact entered the evaluation fixture set, and who approved that promotion.

The default trust rule is conservative:

• quality must not regress on fixed task datasets;
• stability must not regress;
• latency and cost must improve or stay flat, unless a human explicitly accepts a quality tradeoff;
• sensitive artifact or attachment cases cannot become long-lived fixtures without human approval.

This contract turns observability into a collaboration mechanism. The agent does the repetitive measurement work; the human reviews evidence, calibrates judgment-heavy evaluators, and approves only the risk that cannot be reduced to data.

Human and Agent Responsibilities

The loop is AI Native because agents should operate the routine diagnosis and validation work, while humans intervene at high-leverage judgment points.

Agent-owned work:

• scan recent traces and dashboards on a schedule;
• rank failures by category, detail, stage, agent, model, OS, and task kind;
• cluster low-score or high-cost traces into failure modes;
• propose dataset and annotation candidates;
• run experiments for candidate prompt, skill, model, or runtime changes;
• summarize pass/fail evidence and tradeoffs across quality, reliability, latency, and cost;
• draft optimization PRs or prompt/skill changes when the risk is low.

Human-owned work:

• approve production releases and high-risk behavior changes;
• approve provisional baselines and sensitive fixture promotion;
• label ambiguous or subjective quality cases;
• approve major system prompt, skill, or design-system changes;
• decide quality versus cost tradeoffs when metrics conflict;
• review annotation queues and calibrate evaluators.

The intended working loop is:

agent finds a signal
  -> agent explains the likely cause
  -> agent proposes data/evaluator/prompt/runtime changes
  -> fixed-dataset experiment proves or rejects the proposal
  -> human reviews the pass/fail evidence and approves only the meaningful risk

Metrics

The loop should report progress through a small set of durable metrics.

Quality:

• task_success by task kind, agent, model, skill, and design system;
• preview_ok and artifact_valid for artifact-producing tasks;
• user rating rate and negative-reason distribution;
• annotation agreement or evaluator drift where available.

Reliability:

• terminal failure rate;
• unknown or unattributable failure share;
• retryable failure share and retry success rate;
• failure category/detail/stage distribution.

Latency:

• P50/P90/P99 total duration by task kind;
• slowest segment by queue, prompt build, spawn, first token, generation, tool aggregate, verify, and finalize;
• preview and artifact verification duration once those stages are observable.

Cost:

• provider input tokens, effective input tokens, output tokens, and total tokens;
• estimated context tokens and uncached input tokens;
• cache hit ratio and cache source;
• calculated cost when Langfuse pricing supports the model;
• fallback cost bucket when a model has no pricing mapping.

Loop health:

• dataset item count by source and task kind;
• experiment count and pass/fail trend;
• annotation queue throughput and aging;
• number of shipped changes linked to observed trace evidence.

Data completeness:

• share of artifact-producing runs with a valid artifact_manifest;
• share of attachment-bearing runs with a valid attachment_manifest;
• share of traces with truncated input/output and no compensating summary;
• share of manifests with missing storage references, hashes, or parse/build status;
• masked or redacted attachment share by task kind and data class.

Rollout Slices

Slice 1: Spec and Baseline

• Land this spec.
• Keep runtime behavior unchanged.
• Document current trace coverage, score coverage, dataset count, and known gaps from Langfuse/PostHog.

Slice 2: Semantic Stage Observations

• Add task-stage observations around existing run lifecycle boundaries.
• Preserve existing trace ids, tags, and low-level observations.
• Emit stage status, duration, and failure metadata so failed traces can be grouped by product stage.
• Emit trace-safe attachment and artifact manifests during brief-intake and verify-artifact.

Slice 3: Object Storage and Manifest Contract

• Define the Open Design object storage contract for attachments, parsed derivatives, generated artifacts, preview screenshots, export bundles, and reproducibility files.
• Define the registry schema and od:// reference format for attachment, parsed_attachment, artifact, preview, export, and replay objects.
• Store original heavy inputs and outputs in Open Design controlled storage, not in Langfuse by default.
• Store durable object references, hashes, summaries, redaction state, truncation state, and parse/build status in Langfuse.
• Wire user upload, project files, standalone artifact save, live artifact, media generation, preview verification, and export/finalize flows through the registry instead of writing unindexed files only.
• Reuse ProjectStorage as the backend adapter boundary: local remains the default, while S3-compatible storage is enabled by environment configuration once product authorization, retention, and signed inspection URLs are in place.
• Add masking before Langfuse ingestion so raw sensitive content cannot leak through input, output, or metadata fields.

Slice 4: Automatic Evaluators

• Add deterministic scores for artifact validity, preview success, task success proxy, latency bucket, and cost bucket.
• Keep evaluator failures non-blocking.
• Write scores back to Langfuse and mirror aggregate fields to PostHog where dashboarding needs them.

Slice 5: Dataset and Annotation Promotion

• Add an agent-operable workflow that proposes dataset items from failed, low-score, high-cost, and high-latency traces.
• Route ambiguous quality cases to Langfuse annotation queues.
• Preserve provenance from dataset item back to source trace and accepted human annotation.
• Preserve trace-safe manifest references for attachments and artifacts; require human approval before promoting sensitive raw attachments or artifacts into durable evaluation fixtures.

Slice 6: Experiment and Release Gates

• Define fixed task datasets for core task kinds.
• Run experiments before prompt, skill, model-routing, retry, or context changes are treated as durable wins.
• Gate releases on quality not regressing while reliability, latency, or cost improves.
• Treat fixed-dataset pass/fail evidence as the default trust artifact for product and engineering review.

The release gate should use a fixed comparator so the same experiment result produces the same pass/fail decision across implementations:

Gate field	Rule
Baseline comparator	Compare the candidate against the latest approved release on the fixed dataset for each affected task kind. If no fixed dataset exists yet, compare against the last 14 days of production traces promoted into the baseline window and block shipping until a maintainer approves that provisional baseline.
Blocking quality metrics	task_success, user_request_covered, and applicable artifact_valid / preview_ok / design_quality. Any blocking quality metric regressing by more than 1 percentage point, or producing one additional critical artifact failure in a fixed regression dataset, blocks the release.
Blocking reliability metrics	Terminal failure rate, unknown failure share, retryable failure share, and retry success rate. A candidate fails if terminal or unknown failures increase by more than 1 percentage point, or if retry success drops by more than 2 percentage points.
Blocking latency and cost metrics	P90 total duration and total token or calculated cost by task kind. A candidate fails if either P90 latency or cost increases by more than 5% without an explicit human-approved quality tradeoff.
Advisory metrics	P50 latency, P99 latency, cache hit ratio, annotation throughput, dataset growth, and user rating volume. These must be reported with the gate result but do not block by themselves.
Improvement threshold	A candidate qualifies as an improvement only when all blocking metrics pass and at least one blocking reliability, latency, or cost metric improves by 5% or more, or one documented failure category is eliminated on the fixed regression dataset.

Slice 7: Agent-Operated Optimization Loop

• Schedule an agent report that ranks the most important quality, reliability, latency, and cost opportunities.
• Let agents draft low-risk improvement PRs with linked trace, dataset, and experiment evidence.
• Require human approval for high-risk prompts, guardrails, model-routing, and product-quality decisions.

Relationship to Existing Plans

specs/current/run-reliability-optimization-plan.md remains the concrete plan for reducing failures and optimizing latency/token cost after the first observability slice. This spec is the higher-level loop that makes that work repeatable and connects it to quality evaluation, datasets, experiments, annotation, and human-agent collaboration.

specs/current/automation-self-evolution.md describes how successful runs and sources can promote durable memory, skills, design systems, and automation templates. This observability loop provides the evidence and gates for deciding which proposed evolutions are trustworthy.

Non-goals

• This spec does not replace Langfuse or PostHog.
• This spec does not make Langfuse the storage layer for original user attachments or generated artifact packages.
• This spec does not add a new model router.
• This spec does not require all evaluators to be LLM judges.
• This spec does not implement runtime changes in the spec-only PR.
• This spec does not make agents auto-ship high-risk changes without human approval.

Acceptance Criteria for Follow-up Implementation

The loop is working when a representative production regression can be handled end to end:

1. A bad run produces a trace with a clear failing task stage.
2. Attachment and artifact manifests include trace-safe summaries, hashes, storage references, truncation state, and redaction state when applicable.
3. Automatic or human scores identify the quality, stability, latency, or cost problem.
4. The trace is promoted to a dataset item or annotation queue with provenance.
5. An experiment compares a proposed fix against the accepted dataset.
6. The result summarizes quality, reliability, latency, cost, and data completeness tradeoffs.
7. A human approves high-risk changes or the agent proceeds with a low-risk improvement.
8. Online dashboards confirm the improvement after release.