wehub-resource-sync
394 lines
40 KiB
Markdown
394 lines
40 KiB
Markdown
# Telemetry Availability
|
|
|
|
Append-only reference for dashboard authors. Use this file to answer: "From what date is this event or property trustworthy enough to chart?"
|
|
|
|
This file records operational availability. Event contracts still live in `src/shared/telemetry-events.ts`, and dashboard/query examples live with the dashboard work that introduced them.
|
|
|
|
## How To Use
|
|
|
|
- Prefer `dashboard_ready_at_utc` as the lower bound for saved PostHog retention, funnel, and cohort tiles.
|
|
- If `dashboard_ready_at_utc` is `TBD`, do not save production cohort or retention tiles. QA/exploratory charts may proceed only after checking first-seen evidence, and their descriptions must say the boundary is not final.
|
|
- Use the latest required first-valid timestamp when a dashboard depends on multiple signals. An earlier timestamp may prove one event exists while a join key or depth field is still absent, which silently biases cohorts.
|
|
- Do not save a cohort chart from one isolated first-seen row.
|
|
- Mention rollout boundaries in saved insight descriptions when querying across pre-rollout data.
|
|
- Do not add these timestamps to telemetry payloads. They are repo-side interpretation metadata.
|
|
|
|
## Boundary Fields
|
|
|
|
| Field | Meaning |
|
|
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
|
|
| `code_merged_at_utc` | When the implementation reached `main`. Useful for source history only. |
|
|
| `first_released_at_utc` | First release commit/build that contains the code. This is the earliest possible user exposure. |
|
|
| `first_seen_at_utc` | First observed PostHog row for the event/property. This proves at least one event flowed. |
|
|
| `dashboard_ready_at_utc` | Earliest lower bound for saved dashboards after all required signals have first-valid rows and pass any validation window. |
|
|
|
|
First-valid means the released app version has been observed, the required event/property fields are populated, joins used by the dashboard are available, and any validation window listed in the rollout entry has passed.
|
|
|
|
D1+/D3+/D7+ retention means the user fired `app_opened` at least once after 24/72/168 hours from cohort start, matching the existing dashboards' mature-denominator logic. Do not read D3/D7 as exactly-on-day returns, and do not include users whose cohort start is too recent to satisfy the relevant window.
|
|
|
|
## Cross-Cutting Rules
|
|
|
|
- `app_opened` is the return marker. It fires once per app session after telemetry consent/gating resolves.
|
|
- `nth_repo_added = 0` means the user had no repos at emit time. It is not a missing-value sentinel.
|
|
- Missing `nth_repo_added` means the classifier failed soft or the event predates the property. Do not bucket it with `0`.
|
|
- Main injects `nth_repo_added` only into schemas that declare the field, and injects onboarding `cohort` only into schemas that declare `cohort`.
|
|
- `onboarding_started { cohort: 'fresh_install' }` is the fresh-install anchor. `upgrade_backfill` rows should not be mixed into new-user onboarding funnels.
|
|
- Classify a user's fresh-install onboarding cohort from the first `onboarding_started { cohort: 'fresh_install' }` and carry that forward. Existing-user classification can flip after live completion is persisted, so later event-level `cohort` values are not a replacement for the start anchor.
|
|
- Numeric onboarding step values are rollout-specific. Prefer `value_kind` when the relevant event/property is available.
|
|
- Current `onboarding_step_skipped` means an optional preference/setup step was skipped toward required project setup. It does not mean the user abandoned repo setup.
|
|
- `source = 'unknown'` is not a real product surface. It means the caller omitted a source or the value failed schema validation.
|
|
- `workspace_created` means create-worktree IPC succeeded. It is not a general "usable workspace exists" or "workspace revealed" marker.
|
|
- `add_repo_setup_step_action` is a historical post-add choice-screen event. Current Add Project flows auto-open the default checkout or reveal the project row, so absence of this event after the default-checkout handoff rollout is expected and should not be read as setup abandonment.
|
|
- `add_repo_existing_workspaces_detected` is a detection signal for migration opportunity, not proof that a user chose an existing workspace. Current Add Project flows may emit it before automatically opening the default checkout.
|
|
- `add_repo_default_checkout_handoff.result = 'revealed_project'` means Orca could not confidently open the default checkout and instead revealed the newly added project row. It is the direct fallback metric for the 2026-06-03 Add Project handoff rollout.
|
|
- `agent_started` means PTY spawn succeeded with agent telemetry attached. It is not first-repo activation and does not prove the user sent a prompt.
|
|
- `agent_prompt_sent` means a live agent hook observed an explicit non-empty user prompt. It excludes hydrated/replayed status, agent auto-start, bare shells, draft prefill, and hookless sessions; missing rows mean no hook-confirmed interaction was observed, not proof the user never typed.
|
|
- Workspace-outcome joins are native Electron coverage unless the query explicitly proves remote/web instrumentation. Remote runtime and web paths can bypass native repo/worktree telemetry, so do not interpret missing workspace outcome rows as product drop-off for SSH, remote, or web users.
|
|
|
|
## Rollouts
|
|
|
|
### 2026-05-08 - Repo Cohort Property
|
|
|
|
Scope: `nth_repo_added` on repo/activation/retention events. Current schemas declare it on `app_opened`, `repo_added`, `add_repo_setup_step_action`, `add_repo_existing_workspaces_detected`, `add_repo_default_checkout_handoff`, `workspace_created`, `workspace_create_failed`, `setup_script_prompt_shown`, `setup_script_prompt_action`, `agent_started`, `agent_prompt_sent`, and `agent_error`. The original rollout covered `app_opened`, `repo_added`, `add_repo_setup_step_action`, `workspace_created`, `workspace_create_failed`, `agent_started`, and `agent_error`; later events have their own first-seen timestamps below.
|
|
|
|
| Field | Value |
|
|
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| PR | `#1591` |
|
|
| Merge commit | `002e2acc38cad262d694049cc86c0962ffd381ce` |
|
|
| `code_merged_at_utc` | `2026-05-08T17:50:44Z` |
|
|
| First release | `v1.3.42-rc.1` |
|
|
| First release commit | `b6dab85aaa0e9b600ec93846824ba815542a1e1e` |
|
|
| `first_released_at_utc` | `2026-05-08T17:56:35Z` |
|
|
| Earliest `first_seen_at_utc` | `2026-05-08T18:40:00.354Z` on `app_opened` |
|
|
| `dashboard_ready_at_utc` | Event-dependent. Use `2026-05-08T18:40:00.354Z` for general `app_opened` repo-count cuts; use `2026-05-08T20:01:06.364Z` or later for first-repo activation cuts requiring `repo_added`. |
|
|
|
|
PostHog evidence checked at `2026-05-23T23:34:32Z`:
|
|
|
|
| Event | First seen with `nth_repo_added` |
|
|
| --------------------------------------- | ------------------------------------------ |
|
|
| `app_opened` | `2026-05-08T18:40:00.354Z` (`1.3.42-rc.1`) |
|
|
| `workspace_created` | `2026-05-08T19:15:06.913Z` |
|
|
| `agent_started` | `2026-05-08T19:15:07.130Z` |
|
|
| `agent_prompt_sent` | `TBD` |
|
|
| `repo_added` | `2026-05-08T20:01:06.364Z` (`1.3.42`) |
|
|
| `add_repo_setup_step_action` | `2026-05-08T20:01:20.897Z` |
|
|
| `workspace_create_failed` | `2026-05-08T23:04:32.315Z` |
|
|
| `agent_error` | `2026-05-13T14:13:16.096Z` |
|
|
| `add_repo_existing_workspaces_detected` | `2026-05-20T05:33:20.160Z` |
|
|
| `add_repo_default_checkout_handoff` | `TBD` |
|
|
| `setup_script_prompt_shown` | `2026-05-21T04:09:04.231Z` |
|
|
| `setup_script_prompt_action` | `2026-05-21T04:10:31.616Z` |
|
|
|
|
Dashboard caveats:
|
|
|
|
- Earlier rows do not have `nth_repo_added`; show them as pre-rollout or exclude them from repo-count cohorts.
|
|
- A dashboard that depends on a specific event must use that event's first-seen timestamp, not the earliest `app_opened` timestamp.
|
|
- `repo_added nth_repo_added = 1` is the first-repo activation marker.
|
|
- `add_repo_setup_step_action` is only representative of the old post-add choice-screen funnel. After the 2026-06-03 default-checkout handoff change below, Add Project no longer shows that choice screen during the normal Git add/clone/create/onboarding paths.
|
|
|
|
### 2026-05-09 - Onboarding Cohort Injection
|
|
|
|
Scope: `cohort` on onboarding events. Current schemas declare it on `onboarding_started`, `onboarding_step_viewed`, `onboarding_step_completed`, `onboarding_step_skipped`, `onboarding_tour_outcome`, `onboarding_step4_path_clicked`, `onboarding_step4_path_failed`, `onboarding_task_sources_snapshot`, `onboarding_windows_terminal_snapshot`, `onboarding_completed`, `onboarding_dismissed`, `onboarding_agent_picked`, onboarding import/setup events, `onboarding_feature_setup_toggled`, `onboarding_feature_setup_run`, `onboarding_feature_setup_terminal_opened`, and `onboarding_feature_setup_terminal_interacted`. See `src/shared/telemetry-events.ts` for the exact current roster.
|
|
|
|
The original `#1608` rollout covered `onboarding_started`, `onboarding_step_viewed`, `onboarding_step_completed`, `onboarding_step_skipped`, `onboarding_step4_path_clicked`, `onboarding_step4_path_failed`, `onboarding_completed`, `onboarding_dismissed`, `onboarding_agent_picked`, and onboarding import events. Later onboarding events joined the roster by declaring `cohort` in their schemas.
|
|
|
|
| Field | Value |
|
|
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| PR | `#1608` |
|
|
| Merge commit | `318e2b4c2c501b50280dc9e5b11ed34bd899bdd9` |
|
|
| `code_merged_at_utc` | `2026-05-09T02:57:18Z` |
|
|
| First release | `v1.3.44` |
|
|
| First release commit | `e602decf1d35dc54c84eea516fc2b9afdc065cce` |
|
|
| `first_released_at_utc` | `2026-05-09T08:05:39Z` |
|
|
| Earliest `first_seen_at_utc` | `2026-05-09T09:54:03.038Z` on `onboarding_started { cohort: 'fresh_install' }` (`1.3.45`) |
|
|
| `dashboard_ready_at_utc` | `2026-05-09T09:54:03.038Z` for fresh-install onboarding-start cohorts; use later event-specific first-seen timestamps where a tile requires another event. |
|
|
|
|
Later onboarding cohort events:
|
|
|
|
| Event | PR | First release | `first_seen_at_utc` |
|
|
| ---------------------------------------------- | ---------------------------------------------------- | --------------------------------------- | ----------------------------------------- |
|
|
| `onboarding_feature_setup_run` | `#1853` (`2e5ac1c8ebdc93f6281622d9e4ef60ab14c71096`) | `v1.4.2-rc.1` at `2026-05-14T22:23:46Z` | `2026-05-15T05:55:37.248Z` (`1.4.2-rc.4`) |
|
|
| `onboarding_feature_setup_terminal_opened` | `#1853` (`2e5ac1c8ebdc93f6281622d9e4ef60ab14c71096`) | `v1.4.2-rc.1` at `2026-05-14T22:23:46Z` | `2026-05-15T05:55:37.261Z` (`1.4.2-rc.4`) |
|
|
| `onboarding_feature_setup_terminal_interacted` | `#1853` (`2e5ac1c8ebdc93f6281622d9e4ef60ab14c71096`) | `v1.4.2-rc.1` at `2026-05-14T22:23:46Z` | `2026-05-15T11:47:28.022Z` (`1.4.2-rc.6`) |
|
|
| `onboarding_feature_setup_toggled` | `#1853` (`2e5ac1c8ebdc93f6281622d9e4ef60ab14c71096`) | `v1.4.2-rc.1` at `2026-05-14T22:23:46Z` | `2026-05-17T03:03:54.535Z` (`1.4.3`) |
|
|
| `onboarding_task_sources_snapshot` | `#2275` (`ececd27fd8203a7acb3e430fa016fb4653aaa93c`) | `v1.4.7-rc.1` at `2026-05-19T00:47:11Z` | `2026-05-19T04:18:23.872Z` (`1.4.7`) |
|
|
|
|
Dashboard caveats:
|
|
|
|
- Use `onboarding_started { cohort: 'fresh_install' }` as the new-user denominator.
|
|
- Keep the single observed `upgrade_backfill` rows out of fresh-install funnels.
|
|
- `onboarding_step_completed` had semantic `value_kind` before the tour instrumentation, but `onboarding_step_viewed` and `onboarding_step_skipped` only became semantic-step-safe with the tour telemetry rollout below.
|
|
- Later onboarding events joined the `cohort` roster by declaring `cohort` in their schema. Use event-specific first-seen timestamps for tiles that depend on those later events.
|
|
- `onboarding_task_sources_snapshot` fires only when the integrations step exits through Continue or Skip to project setup. Viewing integrations and leaving does not emit the snapshot.
|
|
- Old numeric step analysis must be scoped to the flow version being analyzed.
|
|
|
|
### 2026-05-14 - Feature Wall Base Telemetry
|
|
|
|
Scope: first feature-wall tour event family before the inline onboarding tour. This added base open/close and tile activity telemetry. Historical `feature_wall_closed` rows from this rollout have `dwell_ms`; the source/depth fields needed for tour cohort retention only arrive in the later tour telemetry rollout.
|
|
|
|
| Field | Value |
|
|
| ------------------------ | ---------------------------------------------------------------------------------------------------------------- |
|
|
| PR | `#1772` |
|
|
| Merge commit | `fdf7d9e97a46f27b3667d9bae6c8aca5d11345d8` |
|
|
| `code_merged_at_utc` | `2026-05-14T22:20:50Z` |
|
|
| First release | `v1.4.2-rc.1` |
|
|
| First release commit | `c66e9801fe5f159cac23564ec7c028957973692a` |
|
|
| `first_released_at_utc` | `2026-05-14T22:23:46Z` |
|
|
| `first_seen_at_utc` | `2026-05-15T06:15:30.263Z` on `feature_wall_opened { source: 'popup' }` (`1.4.2-rc.4`) |
|
|
| `dashboard_ready_at_utc` | Event-dependent. Useful for base feature-wall usage after first-seen rows, not for inline-tour cohort retention. |
|
|
|
|
PostHog evidence checked at `2026-05-23T23:34:32Z`:
|
|
|
|
| Event/property | `first_seen_at_utc` |
|
|
| ------------------------------------------------- | ----------------------------------------- |
|
|
| `feature_wall_opened { source: 'popup' }` | `2026-05-15T06:15:30.263Z` (`1.4.2-rc.4`) |
|
|
| `feature_wall_tile_focused` | `2026-05-15T06:15:31.073Z` (`1.4.2-rc.4`) |
|
|
| `feature_wall_closed` with legacy `dwell_ms` only | `2026-05-15T06:18:22.677Z` |
|
|
| `feature_wall_tile_clicked` | `2026-05-17T04:35:04.142Z` (`1.4.3`) |
|
|
| `feature_wall_opened { source: 'help_menu' }` | `2026-05-17T07:02:23.278Z` (`1.4.3`) |
|
|
|
|
Dashboard caveats:
|
|
|
|
- Do not infer inline onboarding tour cohorts from this rollout.
|
|
- Historical `feature_wall_closed` null `source`/depth fields are pre-rollout absence, not product behavior.
|
|
|
|
### 2026-05-23 - Inline Tour Surface
|
|
|
|
Scope: optional "Explore Orca" tour during onboarding and the Help menu entry point. This shipped the product surface, not the full retention cohort instrumentation. It also added `source = 'onboarding'` to the feature-wall open source enum and added feature-wall group/feature/docs click events with source.
|
|
|
|
| Field | Value |
|
|
| -------------------------- | ---------------------------------------------------------------------------- |
|
|
| PR | `#2652` |
|
|
| Merge commit | `669ade23133b9905fff1b6ed22755ee2911ea517` |
|
|
| `code_merged_at_utc` | `2026-05-23T19:21:14Z` |
|
|
| First release | `v1.4.23-rc.0` |
|
|
| First release commit | `1f1171e0dee5d0fbf969de56845882bf8deb9037` |
|
|
| `first_released_at_utc` | `2026-05-23T20:14:38Z` |
|
|
| First app version observed | `1.4.23-rc.0` at `2026-05-23T20:54:12.343Z` |
|
|
| `dashboard_ready_at_utc` | `TBD`; do not use this surface-only rollout for tour cohort retention tiles. |
|
|
|
|
Dashboard caveats:
|
|
|
|
- `feature_wall_opened { source: 'onboarding' }` is a release-availability signal for the inline surface, not the canonical tour cohort assignment.
|
|
- Use this boundary only for exploratory checks that do not require the new outcome/depth fields.
|
|
- Tour cohorts are observational and self-selected. Do not use causal "lift" or "impact" language unless a separate experiment supports it.
|
|
|
|
### 2026-05-23 - Onboarding Tour Retention Telemetry
|
|
|
|
Scope: low-cardinality telemetry that makes inline-tour cohort retention dashboards possible.
|
|
|
|
Current status: the inline first-run onboarding tour was later removed from active onboarding after PR #2734 moved these education/setup moments to contextual feature tours and the Getting started with Orca guide. The historical schemas still accept seven-step `tour`/`agent_setup` onboarding rows for compatibility, but current active onboarding should not emit new `onboarding_step_* { value_kind: 'tour' }` rows.
|
|
|
|
Added/changed signals:
|
|
|
|
- `onboarding_tour_outcome` with `outcome`, intro/tour duration fields, optional tour depth fields, `advanced_via`, and injected onboarding `cohort`.
|
|
- `onboarding_step_viewed.value_kind` and `onboarding_step_skipped.value_kind`, including `value_kind = 'tour'`.
|
|
- `feature_wall_closed` optional `source`, `exit_action`, `furthest_step`, `last_group_id`, and bounded visited/completed depth counts.
|
|
- `feature_wall_opened { source: 'onboarding' }` through the inline tour surface.
|
|
|
|
| Field | Value |
|
|
| ------------------------ | ------------------------------------------ |
|
|
| PR | `#2713` |
|
|
| Branch commit | `9115143927dce38547545a07e58b85d17796fd7a` |
|
|
| Merge commit | `b9c55bb07127f799528761ec31d7e2b7f8598c07` |
|
|
| `code_merged_at_utc` | `2026-05-23T21:15:11Z` |
|
|
| First release | `v1.4.23-rc.1` |
|
|
| First release commit | `c74765922e770c6540496735a5a2be838457256a` |
|
|
| `first_released_at_utc` | `2026-05-23T21:16:38Z` |
|
|
| `first_seen_at_utc` | `TBD` |
|
|
| `dashboard_ready_at_utc` | `TBD` |
|
|
|
|
Required readiness signals:
|
|
|
|
| Signal | `first_seen_at_utc` |
|
|
| -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
|
| `onboarding_tour_outcome` with valid `outcome` and joinable fresh-install start | `TBD` |
|
|
| `onboarding_step_viewed { value_kind: 'tour' }` | Historical only; no new active-onboarding rows after the PR #2734 education handoff. |
|
|
| `feature_wall_closed` with non-null `source`, `exit_action`, and expected depth fields | `TBD` |
|
|
|
|
QA/opportunistic signals:
|
|
|
|
| Signal | `first_seen_at_utc` |
|
|
| ------------------------------------------------ | ------------------------------------------------------------------------- |
|
|
| `onboarding_step_skipped { value_kind: 'tour' }` | Historical only; do not gate dashboard readiness on natural skip traffic. |
|
|
| `feature_wall_opened { source: 'onboarding' }` | `TBD`; surface availability only. |
|
|
|
|
PostHog evidence checked at `2026-05-23T23:34:32Z`:
|
|
|
|
- `1.4.23-rc.0` had app traffic from `2026-05-23T20:54:12.343Z` through `2026-05-23T22:59:47.599Z`.
|
|
- No `1.4.23-rc.1` or `1.4.23-rc.2` app traffic was observed in the checked query window.
|
|
- No rows were observed for `onboarding_tour_outcome`, `onboarding_step_viewed { value_kind: 'tour' }`, `onboarding_step_skipped { value_kind: 'tour' }`, `feature_wall_opened { source: 'onboarding' }`, or `feature_wall_closed` with the new source/depth fields.
|
|
- No rows were observed for `feature_wall_group_selected`, `feature_wall_feature_selected`, or `feature_wall_docs_clicked` in the checked query window.
|
|
|
|
Dashboard readiness rule:
|
|
|
|
Set this rollout's `dashboard_ready_at_utc` to the latest first-valid timestamp across the required readiness signals, then use that as the lower bound. If a query uses a local variable, name it `tour_telemetry_ready_at_utc` and assign it to the same value. After first-seen rows exist, keep saved cohort tiles in QA/exploratory mode until there is at least a 24-hour validation window where reached-tour users reconcile to exactly one primary inline cohort and closed tour sessions include depth fields.
|
|
|
|
Invalid before this rollout's `dashboard_ready_at_utc`:
|
|
|
|
- Tour cohort onboarding completion/progress.
|
|
- Corrected onboarding outcome split by tour cohort.
|
|
- Tour depth before abandonment.
|
|
- Skipped-tour later Help recovery.
|
|
- D1/D3/D7 retention by tour cohort, until cohorts are also old enough for each maturity window.
|
|
- Numeric-step max-progress charts unless segmented by rollout or app version.
|
|
|
|
Primary cohort interpretation:
|
|
|
|
- Historical denominator for inline completed/partial/skipped rates: `onboarding_step_viewed { value_kind: 'tour' }` after this rollout's `dashboard_ready_at_utc`. Do not use this as a current active-onboarding readiness signal after the PR #2734 education handoff.
|
|
- `onboarding_tour_outcome.outcome = 'completed_inline'` means the user completed the inline tour during fresh onboarding.
|
|
- `onboarding_tour_outcome.outcome = 'started_partial'` means the user started the inline tour but the fresh onboarding session resolved without inline completion.
|
|
- `onboarding_tour_outcome.outcome = 'skipped_intro'` means the user reached the tour intro and skipped without starting the inline tour.
|
|
- Users who started onboarding but never reached the tour intro are `pre_tour_abandoned`, not `unresolved_inline_outcome`.
|
|
- Users who reached the tour intro but have no outcome event are `unresolved_inline_outcome`; monitor this as a telemetry QA gap.
|
|
- Later Help menu tour usage is an overlay on the user's inline cohort. It does not replace the primary inline outcome.
|
|
- `completed_inline` wins over earlier partial state if the user returns and completes during the fresh onboarding session.
|
|
- `started_partial` resolves only after the user started the inline tour and the fresh onboarding session resolves without inline completion.
|
|
- `skipped_intro` intentionally has no `tour_dwell_ms` or depth fields.
|
|
|
|
Feature-wall close interpretation:
|
|
|
|
- `feature_wall_closed.dwell_ms` is session-close dwell time.
|
|
- `feature_wall_closed.source` is the entry point (`onboarding`, `help_menu`, `popup`, or `unknown`).
|
|
- `exit_action = 'onboarding_continue'` means the inline onboarding tour ended through Continue to project setup.
|
|
- `exit_action = 'done'` means a non-onboarding tour session ended through Done.
|
|
- `exit_action = 'dismissed'` is the default close/unmount path.
|
|
- Depth fields are per explicit tour session. Persisted completion state can affect UI progress but must not be interpreted as current-session depth.
|
|
- Missing/null `feature_wall_closed.source` or depth fields mean the row predates the expanded schema, came from an older app version, or failed field coverage. Do not bucket null with `source = 'unknown'`; exclude it from source/depth cohort tiles or show it as a telemetry coverage gap.
|
|
|
|
### 2026-06-02 - Active Onboarding Step Removal
|
|
|
|
Scope: active first-run onboarding no longer emits the `agent_setup` or `tour` semantic steps. The removed "Set up Orca for agents" and "Explore Orca" education/setup moments are covered by PR #2734 through contextual feature tours and the Getting started with Orca guide.
|
|
|
|
This is a product-flow and telemetry-interpretation boundary, not a new event rollout. Historical onboarding schemas still accept seven-step `agent_setup` and `tour` rows so old data remains queryable, but dashboard authors should not expect new active-onboarding rows for those semantic steps after this rollout.
|
|
|
|
| Field | Value |
|
|
| ------------------------ | ------------------------------------------------------------------------------- |
|
|
| PR | `#4445` |
|
|
| Merge commit | `TBD` |
|
|
| `code_merged_at_utc` | `TBD` |
|
|
| First release | `TBD` |
|
|
| First release commit | `TBD` |
|
|
| `first_released_at_utc` | `TBD` |
|
|
| `first_seen_at_utc` | N/A |
|
|
| `dashboard_ready_at_utc` | Event-dependent; use this as a cutoff only after the PR is merged and released. |
|
|
|
|
Dashboard caveats:
|
|
|
|
- Treat `onboarding_step_* { value_kind: 'agent_setup' }`, `onboarding_step_* { value_kind: 'tour' }`, and `onboarding_tour_outcome` as historical first-run onboarding signals after this rollout.
|
|
- Do not use absence of new `agent_setup` or `tour` onboarding rows as a drop-off signal; those steps no longer exist in active onboarding.
|
|
- Segment numeric onboarding step analysis across this boundary. The active final step changed from seven-step onboarding to the five-step active flow.
|
|
- Continue using `contextual_tour_shown` and `contextual_tour_outcome` from PR #2734 for current feature-education exposure and outcome analysis.
|
|
|
|
### 2026-06-03 - Final Code Onboarding Step Removal
|
|
|
|
Scope: active first-run onboarding no longer emits the final code/project picker step. The notifications step is now the final step, and completing it opens the Add Project modal.
|
|
|
|
This is a product-flow and telemetry-interpretation boundary, not a new event rollout. Historical onboarding schemas still accept five-step rows so old data remains queryable, but dashboard authors should not expect new active-onboarding rows for the removed final code/project picker step after this rollout.
|
|
|
|
| Field | Value |
|
|
| ------------------------ | ------------------------------------------------------------------------------- |
|
|
| PR | `#4524` |
|
|
| Merge commit | `TBD` |
|
|
| `code_merged_at_utc` | `TBD` |
|
|
| First release | `TBD` |
|
|
| First release commit | `TBD` |
|
|
| `first_released_at_utc` | `TBD` |
|
|
| `first_seen_at_utc` | N/A |
|
|
| `dashboard_ready_at_utc` | Event-dependent; use this as a cutoff only after the PR is merged and released. |
|
|
|
|
Dashboard caveats:
|
|
|
|
- Treat `onboarding_step_*` rows for the removed final code/project picker step as historical first-run onboarding signals after this rollout.
|
|
- Segment numeric onboarding step analysis across this boundary. At this boundary, the active final step changed from the five-step active flow to `ONBOARDING_FINAL_STEP = 4`; later onboarding step rollouts may supersede that final-step value.
|
|
- Do not use absence of new final code/project picker rows as a drop-off signal; that step no longer exists in active onboarding.
|
|
|
|
### 2026-06-03 - Add Project Default Checkout Handoff
|
|
|
|
Scope: Add Project handoff telemetry and interpretation change. The Git add/clone/create/onboarding flows no longer show the post-add setup-choice screen; they close the add modal and open the project/default checkout when available. If no default checkout is available, they reveal the project row instead.
|
|
|
|
`repo_added` remains the add/import marker. `add_repo_existing_workspaces_detected` remains a low-cardinality detection event for pre-existing non-main workspaces on local folder, runtime server path, and SSH server path adds. `add_repo_default_checkout_handoff` is the direct rollout-health event for the automatic handoff decision.
|
|
|
|
| Field | Value |
|
|
| ------------------------ | ------------------------------------------------------------------------------------- |
|
|
| PR | `#4530` |
|
|
| Merge commit | `TBD` |
|
|
| `code_merged_at_utc` | `TBD` |
|
|
| First release | `TBD` |
|
|
| First release commit | `TBD` |
|
|
| `first_released_at_utc` | `TBD` |
|
|
| `first_seen_at_utc` | `TBD` on `add_repo_default_checkout_handoff` |
|
|
| `dashboard_ready_at_utc` | Use the first release boundary once known for charts that compare setup-choice usage. |
|
|
|
|
Dashboard caveats:
|
|
|
|
- Treat `add_repo_setup_step_action` rows after this rollout as historical compatibility or stale modal cleanup only, not the normal Add Project funnel.
|
|
- Do not build current-flow conversion funnels that require `repo_added -> add_repo_setup_step_action`; the normal next step is an automatic workspace reveal/open, not a tracked user choice.
|
|
- Use `add_repo_existing_workspaces_detected` to estimate how often added projects had non-main existing workspaces, but do not infer the user selected "use existing worktrees" because that choice no longer exists in the normal flow.
|
|
- Use `add_repo_default_checkout_handoff` for the current handoff outcome. `result = 'opened_default_checkout'` is the expected path; `result = 'revealed_project'` is the graceful fallback. Break down fallback rows by `source` and `reason`.
|
|
|
|
### 2026-06-10 - Repo Added Git-vs-Folder Signal
|
|
|
|
Scope: `repo_added.is_git_repo` replaces the retired `onboarding_completed.is_git_repo` split for git-vs-folder analysis. Project selection moved out of onboarding in the 1.4.46 flow, so `onboarding_completed` now fires before any repo is chosen. After that boundary, the old `onboarding_completed.is_git_repo` value is not a valid git-vs-folder signal.
|
|
|
|
`repo_added.is_git_repo` is sourced from git detection at the add point. It is optional so SSH/remote paths that genuinely cannot determine git-ness can omit the property instead of defaulting to `false`.
|
|
|
|
| Field | Value |
|
|
| ------------------------ | ------------------------------------------------------------------------------------------- |
|
|
| PR | `#5121` |
|
|
| Merge commit | `TBD` |
|
|
| `code_merged_at_utc` | `TBD` |
|
|
| First release | `TBD` |
|
|
| First release commit | `TBD` |
|
|
| `first_released_at_utc` | `TBD` |
|
|
| `first_seen_at_utc` | `TBD` on `repo_added.is_git_repo` |
|
|
| `dashboard_ready_at_utc` | `TBD`; use only after first-seen rows exist and field coverage has been checked in PostHog. |
|
|
|
|
PostHog evidence checked at `2026-06-10T19:00:00Z`:
|
|
|
|
- Dashboard tile "Fresh-install onboarding completion over time" (`JlIt5J1N`, insight id `9076383`, project `406068`) showed the git-repo share collapse to about 4% while plain-folder completions spiked to about 88% on 2026-06-05.
|
|
- Raw `onboarding_completed.is_git_repo` counts by `app_version` showed a version cliff: versions through `1.4.45` were about 80% true, while `1.4.46`, `1.4.47`, and `1.4.48` had zero true rows in the sampled data.
|
|
|
|
Dashboard caveats:
|
|
|
|
- Treat `onboarding_completed.is_git_repo` as historical only after app version `1.4.45`.
|
|
- Do not stitch historical `onboarding_completed.is_git_repo` and new `repo_added.is_git_repo` series without an explicit version boundary and label change; they are emitted at different funnel moments.
|
|
- Repoint dashboard tile `JlIt5J1N` to use `repo_added.is_git_repo` once the new field is observed in release telemetry.
|
|
- Omitted `repo_added.is_git_repo` means unknown/degraded detection, not plain folder. Only explicit `false` means plain folder.
|
|
|
|
### 2026-06-16 - Windows Terminal Preferences Onboarding Step
|
|
|
|
Scope: Windows first-run onboarding adds a terminal preferences step before notifications. The step lets users choose the default Windows terminal shell and right-click paste/menu behavior before their first project handoff.
|
|
|
|
`onboarding_step_*` rows can now emit `value_kind = 'windows_terminal'` at step `4`. Notifications move to step `5`, so `ONBOARDING_FINAL_STEP = 5` for current active onboarding. Non-Windows clients skip the Windows terminal step but still persist through the skipped step so resumed onboarding lands on notifications. `onboarding_windows_terminal_snapshot` records the low-cardinality selected shell bucket, right-click behavior, exit action, duration, and advance method when the visible Windows terminal step exits.
|
|
|
|
| Field | Value |
|
|
| ------------------------ | -------------------------------------------------------------------------------------- |
|
|
| PR | `#5488` |
|
|
| Merge commit | `68abadba8198c627fb642c41e54937c04ccddfe8` |
|
|
| `code_merged_at_utc` | `2026-06-16T21:07:26Z` |
|
|
| First release | `TBD` |
|
|
| First release commit | `TBD` |
|
|
| `first_released_at_utc` | `TBD` |
|
|
| `first_seen_at_utc` | `TBD` on `onboarding_step_viewed { value_kind: 'windows_terminal' }` and `onboarding_windows_terminal_snapshot` |
|
|
| `dashboard_ready_at_utc` | `TBD`; use only after first-seen rows exist and Windows/non-Windows split plus snapshot coverage are verified. |
|
|
|
|
Dashboard caveats:
|
|
|
|
- Segment numeric onboarding step analysis across this boundary. Step `4` is Windows terminal preferences in the current flow, but was notifications in the previous active flow.
|
|
- Use `value_kind` rather than numeric `step` when comparing notifications or Windows terminal setup across releases.
|
|
- Non-Windows users can have persisted `lastCompletedStep` values that include the skipped Windows step; do not treat that as evidence they viewed the Windows terminal page.
|
|
- `onboarding_windows_terminal_snapshot.default_shell = 'other'` means Orca could not bucket the persisted setting. It is not a raw shell path and should be monitored as telemetry quality, not a product choice.
|
|
|
|
## Updating This File
|
|
|
|
When adding or changing telemetry that dashboard authors will depend on:
|
|
|
|
1. Add or update one rollout entry in this file.
|
|
2. Record PR, merge commit, first release, release commit, `first_released_at_utc`, `first_seen_at_utc`, and `dashboard_ready_at_utc`.
|
|
3. Include the app version/build on first-seen rows when it is available.
|
|
4. Add `PostHog evidence checked at ...` with the UTC query time.
|
|
5. If dashboard readiness depends on multiple signals, add a required readiness signal table and set `dashboard_ready_at_utc` only after all of them are first-valid.
|
|
6. Add dashboard caveats for pre-rollout rows, missing/null fields, and maturity windows.
|
|
7. Keep product-specific facts here, not in agent instructions or telemetry payloads.
|