tracer-cloud--opensre/docs/DEVELOPMENT.md

# Development guide

Contributor-focused workflows: local setup details stay in [SETUP.md](https://github.com/Tracer-Cloud/opensre/blob/main/SETUP.md) at the repo root (Windows, troubleshooting, MCP/OpenClaw).

## Clone and install

```bash
git clone https://github.com/Tracer-Cloud/opensre.git
cd opensre
make install
```

[`make install`](https://github.com/Tracer-Cloud/opensre/blob/main/Makefile) runs `uv sync --frozen --extra dev` and the analytics install helper. Use **`uv run opensre …`** from the repo root so you always hit this checkout’s `.venv`, not another `opensre` on your `PATH`.

```bash
opensre onboard
opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json
```

## Quality gates (same as CI)

From the repo root:

```bash
make lint          # ruff check
make format-check  # ruff format --check (CI-enforced)
make typecheck     # mypy config core gateway integrations platform surfaces tools
make test-cov      # pytest + coverage (default unit suite)
```

One-shot (includes heavier `test-full`): `make check`.

Before a PR, run at least `make lint`, `make format-check`, `make typecheck`, and `make test-cov` (see [CONTRIBUTING.md](https://github.com/Tracer-Cloud/opensre/blob/main/CONTRIBUTING.md)).

## Interactive shell action policy

Action-planner behavior, postprocessing transforms, compatibility seams, and the rule-extension checklist are documented in [`docs/interactive-shell-action-policy.md`](https://github.com/Tracer-Cloud/opensre/blob/main/docs/interactive-shell-action-policy.md).

## Package architecture

The seven first-party packages, the four-tier layering (which package may
import which), the folder diagram, per-layer responsibilities, and cross-layer
flows are documented in [`docs/ARCHITECTURE.md`](ARCHITECTURE.md).

## Tool registry — surface-scoped, lazy loading

Loading every vendor tool at startup was slow. A static index
(`tools/registry_index.py`) reads tool metadata by scanning the source, without importing executors, so a turn loads only the tools it needs.

- `get_registered_tools(surface)` imports only that surface's tool modules.
- `get_tool_descriptors(surface)` returns metadata with no executor import.
- `load_tool(descriptor)` imports the executor, only when a tool runs.

Adding a vendor tool is a `@tool`/`BaseTool` module; the index finds it and no other vendor is imported. `tests/tools/test_registry_index.py` checks the index matches the imported registry exactly, so they cannot drift.

## Investigation pipeline architecture

The six-stage investigation pipeline (resolve integrations → extract alert → plan → ReAct evidence loop → diagnose → deliver), the loop's guardrails (tool cap, stagnation breaker, context budget, duplicate detection), and diagrams are documented in [`docs/investigation-pipeline-architecture.md`](investigation-pipeline-architecture.md).

## Investigation tool calling

Tool schemas, provider adapters (`transports/sdk/agent_clients.py`), and investigation message shapes are documented in [`docs/investigation-tool-calling.md`](investigation-tool-calling.md) (all LLM providers, not vendor-specific).

## Interactive shell: REPL watchdog demo

PR reviewers expect a **visible demo** (terminal log or screenshot) in the PR under **Demo/Screenshot**, not only tests. Copy the exact steps from this section into your PR description, then attach your terminal output or recording.

1. `uv run opensre` (TTY).
2. `/trust on` (or confirm the elevated-action prompt when running `/watch`).
3. `/watch <pid> --max-cpu 80` — expect `task … started.` (use a real PID, e.g. the shell’s Python process).
4. `/watches` — table columns include id, pid, kind, status, thresholds, last sample.
5. `/unwatch <task_id>` or `/cancel <task_id>` — then `/watches` again; status should show **cancelled**.
6. Optional: lower `--max-cpu` so a threshold trips; after Telegram sends, the REPL prints one line: `[task …] alarm fired: … (telegram delivered)`.

Automated equivalent (runs in `make test-cov`):  
`uv run pytest tests/interactive_shell/test_watchdog_repl_e2e_demo.py -v --tb=short`

Longer transcript (optional): [tests/interactive_shell/repl_watchdog_demo.md](https://github.com/Tracer-Cloud/opensre/blob/main/tests/interactive_shell/repl_watchdog_demo.md).

## VS Code dev container

The dev container is defined under [`.devcontainer/`](https://github.com/Tracer-Cloud/opensre/tree/main/.devcontainer). It builds from [`.devcontainer/Dockerfile`](https://github.com/Tracer-Cloud/opensre/blob/main/.devcontainer/Dockerfile) (Python **3.13**), then **`postCreateCommand`** creates `.venv-devcontainer` and runs **`pip install -e '.[dev]'`** (not `uv`). Docker Desktop, OrbStack, Colima, or another compatible runtime must be available on the host.

## Benchmark

```bash
make benchmark
```

To refresh README benchmark copy from cached results (no LLM calls): `make benchmark-update-readme`.

## Deployment

Full deployment instructions, prerequisites, and environment variable reference:
**[DEPLOYMENT.md](../DEPLOYMENT.md)**

Quick reference:

| Path | Commands |
| ---- | -------- |
| EC2 (Docker/ECR — web + gateway) | `make build-image` → `make deploy` / `make destroy` |
| Gateway (AMI + systemd — gateway only) | `make bake-gateway` → `make deploy-gateway` / `make destroy-gateway` |
| Hosted (Railway / ECS / Vercel) | Deploy with repo `Dockerfile`; set `LLM_PROVIDER` + API key |

### Hosted runtime (Railway / ECS / Vercel)

1. Deploy this repository as a standard Python/FastAPI app using the repo `Dockerfile` or your host's native Python workflow.
2. Set `LLM_PROVIDER` and the matching API key (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` — see [`.env.example`](https://github.com/Tracer-Cloud/opensre/blob/main/.env.example)).
3. Add `DATABASE_URI` and `REDIS_URI` for hosted layouts that need persistence.
4. Add integration and storage env vars your deployment needs.

Minimal LLM env:

```bash
export LLM_PROVIDER=anthropic
export ANTHROPIC_API_KEY=...
```

For Railway: ensure the project has Postgres and Redis services and that the OpenSRE
service has `DATABASE_URI` and `REDIS_URI` set before deploying. Set
`OPENSRE_DEPLOYMENT_METHOD=railway` for telemetry labeling.

## Telemetry and privacy

`opensre` ships with two telemetry stacks, both opt-out:

- **PostHog** — anonymous product analytics (commands used, success/failure, rough runtime, CLI/Python/OS/arch, and limited command metadata).
- **Sentry** — crashes and errors (stack traces, environment, release).

Events are tagged with `entrypoint`, `opensre.runtime`, and `deployment_method`. Sensitive headers, paths, and secret-shaped keys are scrubbed before send.

A random install ID is stored under `~/.opensre/anonymous_id`. PostHog `distinct_id` is scoped to that ID. Telemetry is off in GitHub Actions and pytest.

### First-launch GitHub login

On the first interactive launch (all platforms, except CI/CD and test harnesses), OpenSRE requires a GitHub device-flow sign-in before the REPL prompt. On success it sets `github_username` as a PostHog **person property** (via `$identify`/`$set`, which forces `$process_person_profile: True` for that one event — this is the only intentional PII OpenSRE sends) and emits a `github_login_completed` event. A configured GitHub integration suppresses re-prompting on later launches.

The existing kill-switches still apply: `OPENSRE_NO_TELEMETRY` / `DO_NOT_TRACK` make the `$identify` and `github_login_completed` calls no-ops, but the login itself still runs. Set `OPENSRE_SKIP_GITHUB_LOGIN=1` to bypass the login gate entirely (also auto-bypassed in CI — `CI=true`, `GITHUB_ACTIONS=true` — and in pytest).

### Kill-switch matrix

| Env var                        | PostHog    | Sentry     |
| ------------------------------ | ---------- | ---------- |
| `OPENSRE_NO_TELEMETRY=1`       | disabled   | disabled   |
| `DO_NOT_TRACK=1`               | disabled   | disabled   |
| `OPENSRE_ANALYTICS_DISABLED=1` | disabled   | unaffected |
| `OPENSRE_SENTRY_DISABLED=1`    | unaffected | disabled   |
| `OPENSRE_SENTRY_LOGGING_DISABLED=1` | unaffected | disables `logger.error`/`logger.exception` forwarding to Sentry; `capture_exception` unaffected |

Full opt-out:

```bash
export OPENSRE_NO_TELEMETRY=1
```

### Sentry DSN

Self-hosted users can set `SENTRY_DSN` to their project; unset uses the bundled default. `SENTRY_DSN=` (empty) drops events in `before_send`.

### Deployment tagging

Set `OPENSRE_DEPLOYMENT_METHOD` to `railway`, `ec2`, `vercel`, or `local` (default `local`) to label Sentry events.

### Local PostHog event log

By default, outbound PostHog payloads are also appended to `~/.opensre/posthog_events.txt` (rotates at 1000 lines). Disable:

```bash
export OPENSRE_ANALYTICS_LOG_EVENTS=0
```

We do not collect alert contents, file contents, hostnames, credentials, raw CLI arguments, or PII by design.