stablyai--orca/docs/reference/new-user-parallel-work-telemetry.md

# New User Parallel Work Telemetry

## Goal

Measure whether the new parallel-work tours and the eight-step `Getting started` flow improve new-user activation and retention.

The product question is not "did the tour render?" It is:

- Did users who were eligible for the new parallel-work education retain better at D1, D3, and D7?
- Did seeing or completing the tour increase the behaviors that the activation plan expects: terminal splits, two agents in one worktree, two worktrees within 72 hours, task/review source use, setup script configuration, notifications, and agent capability setup?
- Where do users leave the education flow: before the tour starts, during a tour, after skipping a tour, after opening `Getting started`, or partway through the eight setup steps?

This document implements the telemetry portion of [`new-user-parallel-work-activation.md`](./new-user-parallel-work-activation.md).

Validation mode: the current branch logs the would-be event name and payload to the console instead of sending these new feature-education events through product telemetry. Keep the logged payload shape identical to the event payloads below so reviewers can verify what would be emitted before product telemetry is re-enabled.

## Product Decisions

Question: Should Orca keep, revise, or disable the automatic `workspace-agent-sessions` tour for new users?

Decision owner/use: Product and growth review the retention and activation read after rollout. If exposed users retain and activate better than eligible-but-unexposed users, keep or expand the tour. If exposed users underperform or skip/cancel heavily, revise copy, timing, or gating.

Question: Which part of the education flow is weak?

Decision owner/use: If users usually stop on a specific tour step, revise that step or shorten the tour. If users complete the tour but do not progress through `Getting started`, improve the handoff. If users open `Getting started` but stall before the first two parallel-work steps, revise the checklist actions.

Dashboard: `New User Parallel Work Activation`

Action: Use the dashboard to decide whether to continue rollout, adjust the tour, reorder checklist steps, or add missing downstream action telemetry.

## Existing Telemetry To Reuse

Keep the existing contextual-tour events as the primary tour exposure and outcome contract:

- `contextual_tour_shown`
- `contextual_tour_outcome`

`contextual_tour_outcome` already carries:

- `tour_id`
- `source`
- `outcome`: `completed`, `skipped`, or `cancelled`
- `steps_seen`
- `total_steps`

For the current branch, `workspace-agent-sessions` is the activation tour. Its source should distinguish automatic first-workspace exposure from checklist-triggered education:

- `workspace_agent_sessions_visible` for automatic surface-driven tour exposure
- `setup_guide_parallel_work` for the explicit checklist `Try it out` path

Do not add one telemetry event per tour opportunity or step render. Opportunity checks are driven by local UI eligibility, retries, target visibility, modal state, and session guards; emitting every "not shown" reason would turn tour gating into passive eligibility telemetry. One shown event plus one outcome event keeps volume low and still answers whether users saw, skipped, cancelled, completed, and how far they got.

## Tour Depth Improvement

Add stable step-depth fields to `contextual_tour_outcome` so the analysis can tell whether a user reached the actual late steps, not only how many visible steps happened to render.

Additive schema fields:

- `furthest_step_index`: integer `1..8`
- `defined_step_count`: integer `1..8`

Keep `steps_seen` and `total_steps`:

- `steps_seen` remains the count of unique rendered visible steps.
- `total_steps` remains the visible-step denominator for the user session.
- `furthest_step_index` is the highest 1-based index from the tour definition reached by the user.
- `defined_step_count` is the number of steps defined for that tour, regardless of target skipping.

This handles target-skipping correctly. A user who sees tour steps 1, 4, and 5 should report `steps_seen: 3`, `total_steps: 3`, `furthest_step_index: 5`, and `defined_step_count: 5`.

## New Setup Guide Events

The eight `Getting started` steps need a small telemetry contract because they are not contextual-tour steps. They are durable activation milestones.

### `setup_guide_opened`

Emit once when the modal opens.

Payload:

- `source`: `sidebar`, `contextual_tour`, `settings`, `feature_wall`, `help_menu`, or `unknown`
- `initial_completed_count`: integer `0..8`
- `total_steps`: integer `8`
- `first_incomplete_step_id`: one of the eight setup step ids, or `none`

Use:

- Measures whether users reach the checklist after the tour.
- Establishes starting depth so existing progress does not look like checklist-driven progress.

### `setup_guide_closed`

Emit once when a shown setup guide closes or unmounts.

Payload:

- `source`: same value used by `setup_guide_opened`
- `outcome`: `completed`, `dismissed`, or `interrupted`
- `initial_completed_count`: integer `0..8`
- `final_completed_count`: integer `0..8`
- `total_steps`: integer `8`
- `active_step_id`: one of the eight setup step ids, or `none`

Use:

- Measures whether users treat the checklist as useful or dismiss it without progress.
- Gives a compact "how far through the 8 steps" read without uploading a progress snapshot on every render.

### `setup_guide_step_completed`

Emit once per setup step when that step first transitions from incomplete to complete while the setup guide is visible. Hidden/background progress is persisted locally as baseline state but not emitted, because async setup refresh cannot reliably distinguish old completions from new user action.

Payload:

- `step_id`: one of `two-worktrees`, `browser`, `notifications`, `default-agent`, `task-sources`, `setup-script`, `add-two-repos`, `agent-capabilities`
- `section_id`: `parallel-work` or `setup`
- `completed_count`: integer `1..8`
- `total_steps`: integer `8`
- `setup_guide_visible`: boolean

Use:

- Measures the in-guide depth through the eight steps without backfilling old setup state on launch.
- Lets retention analysis bucket users by `0`, `1`, `2`, `3-4`, `5-7`, and `8` completed setup steps.

Implementation detail: persist a local set of emitted setup-guide step ids, similar in spirit to `contextualToursSeenIds`, so inferred durable state does not re-emit on every launch. Do not upload that local set.

## Downstream Activation Events

Reuse existing downstream events whenever possible:

- `workspace_created` for worktree/workspace creation depth
- `agent_started` for agent activity and follow-up actions
- `settings_changed` for supported settings such as notifications
- setup script prompt events for setup-script configuration paths
- task/source-specific events where they already exist

Add action-specific downstream telemetry only when the retention question cannot be answered from existing events. The first likely gap is terminal splitting. The activation plan names first-session terminal split as a primary success criterion, but the branch currently records `terminal-pane-split` as local feature-interaction state, not product telemetry.

If no existing telemetry captures terminal split, add:

### `terminal_pane_split`

Emit when a split succeeds, capped to the first split for each `source` + `direction` pair per UTC day on the local profile. This keeps the event useful for activation cohorts without making heavy terminal users dominate telemetry volume.

Payload:

- `source`: `contextual_tour`, `keyboard`, `context_menu`, `command`, `unknown`
- `direction`: `vertical` or `horizontal`

Do not include pane ids, worktree ids, paths, commands, prompts, repo names, branch names, hostnames, or terminal content.

## Retention Analysis

Use new-user eligibility as the least-biased baseline:

1. Eligible baseline: new-user cohorts represented by first-run/onboarding events rather than uploading local tour eligibility state.
2. Exposed cohort: users with `contextual_tour_shown` where `tour_id = workspace-agent-sessions`.
3. Tour outcome cohorts: users with `contextual_tour_outcome` broken down by `outcome`.
4. Tour depth cohorts: `furthest_step_index / defined_step_count` and `steps_seen / total_steps`.
5. Checklist depth cohorts: latest `setup_guide_step_completed.completed_count`, bucketed as `0`, `1`, `2`, `3-4`, `5-7`, and `8`.

Primary retention read:

- D1, D3, and D7 return to `app_opened`
- Compare new-user baseline, shown, completed, skipped, cancelled, and checklist-depth cohorts

Primary activation read:

- terminal split in first workspace session
- two agent sessions in one worktree
- two worktrees within 72 hours
- follow-up/manual agent action
- task/review source use
- setup script configured
- notifications configured
- agent capabilities completed

Interpretation rules:

- A completed tour with no downstream activation means the tour is educational but not converting.
- A skipped tour with strong downstream activation means the feature may be discoverable without the tour.
- A high cancelled rate usually points to target loss, modal conflicts, or timing problems.
- Strong checklist-depth retention with weak tour retention means the checklist is the value driver, not the automatic tour.

## Volume Estimate

Telemetry volume estimate:

- Trigger: contextual tour shown/outcome, setup guide open/close, setup step first completion, optional terminal split
- Expected events/user/day: 2-6 for new users who encounter the flow
- Max events/user/day: about 14 on a heavy first day: 2 tour events, 2 setup guide events, 8 setup step completions, 1-2 split events
- At 1,000 DAU: expected 2,000-6,000 events/day
- Monthly at 1,000 DAU: expected 60,000-180,000 events/month
- Approval note: The events answer the rollout decision directly and are bounded by explicit user actions or one-time durable setup completions. There are no timers, render-loop events, polling snapshots, or passive heartbeats.

If observed volume exceeds expectation, remove or sample `setup_guide_opened` / `setup_guide_closed` before removing completion events. Completion events are the durable activation-depth read.

## Privacy

All payloads must be low-cardinality and schema-bounded.

Never include:

- prompts
- terminal commands
- paths
- URLs
- hostnames
- repo names
- branch names
- task titles
- user-authored text
- raw errors
- tokens
- pane ids, worktree ids, repo ids, or other persistent product identifiers

## Implementation Plan

1. Extend `contextual_tour_outcome` schema in `src/shared/telemetry-events.ts` with optional `furthest_step_index` and `defined_step_count`.
2. Track the highest reached defined tour step in `ContextualTourOverlay.tsx` alongside the existing `steps_seen` set.
3. Add setup-guide event schemas and bounded enums in `src/shared/telemetry-events.ts`.
4. Add a setup-guide telemetry helper in `src/renderer/src/lib/feature-education-telemetry.ts` or a dedicated setup-guide telemetry module.
5. Emit `setup_guide_opened` and `setup_guide_closed` from `SetupGuideModal.tsx`.
6. Emit `setup_guide_step_completed` from a small hook that observes visible setup-guide progress transitions, persists emitted step ids locally, and ignores hidden/background completions except as local baseline state.
7. Add `terminal_pane_split` only if no existing telemetry captures successful terminal splits.
8. Add schema tests for accepted payloads, rejected raw strings, count bounds, and enum bounds.
9. Add renderer tests that prove each modal open emits one open event, each close emits one close event, each step completion emits once, rerenders do not re-emit, and pre-completed persisted steps are not backfilled repeatedly.

## Dashboard Tiles

Create a PostHog dashboard named `New User Parallel Work Activation`.

Tiles:

- Tour exposure: `contextual_tour_shown` by `tour_id` and `source`
- Tour outcome: `contextual_tour_outcome` by `outcome`
- Tour depth: median `furthest_step_index / defined_step_count`
- Setup guide reach: `setup_guide_opened` by `source`
- Setup guide close outcome: `setup_guide_closed` by `outcome`
- Eight-step depth: latest `setup_guide_step_completed.completed_count` bucket
- Activation funnel: tour shown -> tour outcome -> setup guide opened -> first two setup steps complete -> D1/D3/D7 return
- Downstream actions: tour shown -> terminal split -> two agents in one worktree -> two worktrees within 72 hours

Do not publish dashboard tiles that require raw user content or identifiers.