learningcircuit--local-deep-research/docs/RELEASE_GUIDE.md

# Release Guide

## 🚀 Automated Release Process

Releases are **fully automated** end-to-end whenever a PR that bumps
`src/local_deep_research/__version__.py` is merged to `main`. No
separate tag push, manual workflow trigger, or release-page click is
required — the workflow detects the new version, runs all gates, cuts
the GitHub release, and (after one approval click) publishes to PyPI
and Docker Hub.

PRs that don't touch `__version__.py` merge normally but skip the
release pipeline (the version-check job sees the tag already exists
and short-circuits everything downstream).

## 📋 How Releases Work

### 1. **Automatic Release Creation**
- **Trigger**: Push to `main` whose `__version__.py` resolves to a tag
  that does not yet exist as a GitHub release. In practice this means
  "merge a PR that bumps `__version__.py`". Tag pushes (`v*.*.*`) and
  manual `workflow_dispatch` runs also trigger the pipeline and bypass
  the version-exists check.
- **Version**: Read from `src/local_deep_research/__version__.py` by
  the `version-check` job; the tag is `v<version>`.
- **Release body**: Composed by `.github/workflows/release.yml` from three sources:
  1. **AI narrative** generated by OpenRouter (`vars.AI_MODEL`, default
     `moonshotai/kimi-k2-thinking`). The model receives the rendered
     hand-written notes, the auto-generated PR list, every PR's title +
     body (batched via one GraphQL call), and the diff between the
     previous release tag and this one (filtered to drop lockfiles,
     generated docs, SBOM, static assets, and binary patches; capped at
     700k chars).
  2. **Hand-written notes** from `docs/release_notes/<version>.md` —
     rendered from contributor-supplied `changelog.d/*.md` fragments by
     the workflow itself at release time (see
     [Release-notes flow](#-release-notes-flow-towncrier-news-fragments)
     below). No manual `pdm run towncrier build` is required before
     merging the bump.
  3. **Auto-generated PR list** from GitHub's generate-notes API,
     label-categorized.
- **No duplicates**: If a release for `v<version>` already exists, the
  `version-check` job sets `should_release=false` and every downstream
  job (security gate, CI gate, build, publish) is skipped.

### 2. **Approval and Publishing**

The release pipeline uses the `release` GitHub environment to gate the
publish steps. `DOCKER_USERNAME` / `DOCKER_PASSWORD` are scoped to that
environment, so any job that pushes to Docker Hub must declare
`environment: release` and therefore goes through the approval gate.

When you merge to `main` (or push a tag), the pipeline runs in this
order:

1. Security gates + CI gates run automatically.
2. `build` job runs (version pin, SBOM, Sigstore bundles), then
   `provenance` job generates SLSA provenance for those artifacts.
3. **One `release` env approval prompt** in `release.yml`. Approving
   unlocks all release-env jobs in the same run, which then execute
   sequentially:
   1. `prerelease-docker` — canonical multi-arch Docker build, cosign
      sign, SBOM/SLSA attestations, push as `prerelease-v<ver>-<sha>`
      and re-point the floating `:prerelease` tag.
   2. `publish-docker` — retags the prerelease manifest as `:1.6.9`,
      `:1.6`, `:latest` (no rebuild, digest-preserving), then re-verifies
      digest + cosign + Trivy on the promoted tag.
   3. `trigger-pypi` — dispatches `publish.yml` via `repository_dispatch`
      (PyPI Trusted Publishing requires the publish step to run in a
      top-level workflow, so this can't be a reusable workflow_call).
   4. `monitor-pypi` — polls `publish.yml` for completion. The inner
      polling loop times out at 40 minutes (after which the job fails);
      the surrounding GH Actions `timeout-minutes` is 90 to leave a
      safety margin around the poll budget.
   5. `create-release` — publishes the GitHub Release with
      SBOM/sig/provenance assets. Runs **last**, gated on all of the
      above succeeding, so the public Release never points at missing
      Docker tags or a missing PyPI version.

If any of `prerelease-docker`, `publish-docker`, or `monitor-pypi` fails,
`create-release` is skipped and no public GitHub Release is created. The
`cleanup-on-rejection` job then handles failure-mode cleanup:

- If `publish-docker` failed mid-retag (e.g., `:1.6.9` landed but
  `:latest` failed), it rolls back any landed release tags BEFORE
  deleting prerelease tags and cosign artifacts (deleting cosign
  artifacts while release tags share the manifest digest would invalidate
  release-tag signatures).
- If `publish-docker` succeeded but a later step (PyPI or
  create-release) failed, `cleanup-on-rejection` does NOT fire — Docker
  release tags exist and their cosign artifacts must stay. See
  "Recovery from PyPI failure" below.

### Recovery from PyPI failure (atomicity hole)

The one orphan state the pipeline cannot fully clean up: `publish-docker`
succeeded, PyPI failed. At this point Docker `:1.6.9` / `:1.6` /
`:latest` exist and are signed; PyPI has nothing; no GitHub Release.
`monitor-pypi` opens a tracking issue labeled `ci-cd`. To recover:

1. Inspect the `publish.yml` workflow run, fix the underlying cause.
2. Manually re-dispatch PyPI publish. `client_payload[sha]` is REQUIRED
   (publish.yml fails closed without it) and must be the full 40-hex
   release commit — the commit the failed `release.yml` run built, NOT
   current main HEAD (main may have moved during the approval wait).
   Get it with `gh run view <release-run-id> --json headSha`:
   ```bash
   gh api repos/LearningCircuit/local-deep-research/dispatches \
     -f event_type=publish-pypi \
     -F 'client_payload[tag]=v<X.Y.Z>' \
     -F 'client_payload[sha]=<40-hex release commit>'
   ```
3. Once PyPI publishes successfully, manually create the GitHub Release
   from the existing tag (the SBOM/sig/provenance artifacts are still
   uploaded as workflow artifacts on the failed `release.yml` run; you
   can download them and attach manually, or re-run `create-release`
   manually if the run is still re-runnable in the Actions UI).

> Earlier iterations of this refactor described a single approval gate
> with a pre-approval testing window. That design required
> `DOCKER_USERNAME` / `DOCKER_PASSWORD` to be repo-level secrets so the
> canonical build could run without env approval. They are env-scoped to
> `release` instead, so the gate sits in front of the build. The
> atomicity refactor preserves this single-approval model — one click
> unlocks the whole chain, and create-release runs last so the
> "published Release with broken artifacts" failure mode is closed.

## 👥 Who Can Release

Code owners (defined in `.github/CODEOWNERS`):
- `@LearningCircuit`
- `@hashedviking`
- `@djpetti`

## 📝 Release Workflow

### For Regular Releases:

1. **Bump version** in `src/local_deep_research/__version__.py` (or
   merge the auto-bump PR opened by `.github/workflows/version_check.yml`).
2. **Merge to main** → Release automatically created. The release
   workflow renders fragments from `changelog.d/*.md` into
   `docs/release_notes/<X.Y.Z>.md`, composes the body (AI narrative +
   rendered changelog + auto PR list), and publishes the GitHub release.
3. **Approve publishing** in GitHub Actions (PyPI/Docker).
4. **Merge the cleanup PR** opened automatically by the
   `cleanup-changelog` job (titled
   `chore: clear changelog fragments for <X.Y.Z>`). It persists
   `docs/release_notes/<X.Y.Z>.md` and removes the consumed fragments
   from `changelog.d/`. Squash-merge — the diff has no review value
   beyond a sanity check that the rendered notes look right.

To preview the rendered notes locally before merging the bump:

```bash
pdm run towncrier build --draft --version <X.Y.Z>
```

(`--draft` writes nothing.) Skip if no fragments exist for this release
— the workflow tolerates a missing per-version file (warns and proceeds
with auto-notes plus AI summary only).

### For Hotfixes:

1. **Create hotfix branch** from main
2. **Make minimal fix**
3. **Bump patch version** (e.g., 0.4.3 → 0.4.4)
4. **Fast-track review** by code owners
5. **Merge to main** → Automatic release

## 🔧 Manual Release Options

### Option A: Manual Trigger
- Go to Actions → "Create Release" → "Run workflow"
- No inputs are required: the workflow reads the version from
  `src/local_deep_research/__version__.py` at HEAD. To release an
  older or different version, use Option B (push a version tag).

### Option B: Version Tags
- `git tag v0.4.3 && git push origin v0.4.3`
- Automatically creates release; the workflow uses the tag's commit
  SHA (not `main` HEAD), so this is the correct path for backporting.

## 🛡️ Branch Protection

- **Main branch** is protected
- **Required reviews** from code owners
- **No direct pushes** - only via approved PRs
- **Status checks** must pass (CI tests)

## 📦 Version Numbering

Follow [Semantic Versioning](https://semver.org/):
- **Major** (X.0.0): Breaking changes
- **Minor** (0.X.0): New features, backward compatible
- **Patch** (0.0.X): Bug fixes, backward compatible

## 🚨 Emergency Procedures

If automation fails, do NOT create a GitHub release through the UI as
the first recovery step — under the atomicity refactor, a manually
created GitHub release does NOT trigger `publish.yml` (it listens only
on `repository_dispatch`) and does NOT trigger `docker-publish.yml`
(workflow_call only). The downstream `release:` listeners that DO fire
(`backwards-compatibility.yml`, `sbom.yml`) are observability-only.

Recovery, in order of preference:
1. **Check workflow logs** in GitHub Actions to identify which job
   failed, and use the targeted recovery for that failure mode:
   - PyPI failure with Docker already promoted: see
     [Recovery from PyPI failure](#recovery-from-pypi-failure-atomicity-hole) above.
   - Any other failure: re-run the failed job via the Actions UI if
     it's still re-runnable (typically within 30 days).
2. **Re-trigger the full pipeline** via `workflow_dispatch` if
   re-running individual jobs isn't possible. Safe for digest-keyed
   cosign verification — old digests remain valid because their cosign
   artifacts persist; the new run produces a new digest with its own
   signatures.
3. **Contact code owners** if recovery requires manual Docker Hub or
   PyPI intervention.

## 📝 Release-notes flow (towncrier news fragments)

Hand-written release notes are assembled from per-PR fragments using
[towncrier](https://towncrier.readthedocs.io). This replaces the older
shared `docs/release_notes/<version>.md` model, which broke down at
LDR's PR throughput (multiple PRs/day racing for the same file).

### Contributor side

Each PR with user-visible behavior change drops one tiny markdown file:

```
changelog.d/<PR-number>.<category>.md
```

Categories: `breaking`, `security`, `feature`, `bugfix`, `removal`,
`misc` (canonical list lives in `[[tool.towncrier.type]]` entries in
`pyproject.toml`; the pre-commit hook reads it from there). Orphan
fragments (no PR/issue number) use `changelog.d/+<slug>.<category>.md`.
The pre-commit hook (`recommend-release-notes`) nudges contributors who
add ≥20 source lines without a fragment, and validates filenames so a
typo'd category doesn't silently vanish at render time. See
[`changelog.d/README.md`](../changelog.d/README.md) for the full
convention.

### Maintainer side

The release workflow handles the render. There is nothing to run
manually before merging the version bump.

`.github/workflows/release.yml` (`create-release` job):
1. Sparse-checks-out `changelog.d/` + `pyproject.toml`, installs
   `towncrier~=24.8`.
2. Runs `towncrier build --yes --version <X.Y.Z>` against the runner's
   throwaway workspace. Towncrier writes the rendered output to
   `docs/release_notes/<X.Y.Z>.md` (per the `{version}`-templated
   `filename`; `single_file = false` makes this a per-release file
   rather than appending to a master CHANGELOG) and removes the
   consumed fragments locally.
3. Reads the rendered file as input to the AI summary and as the
   "hand-written notes" section of the published GitHub release body.

Persistence to `main` happens in the `cleanup-changelog` job, which
re-runs the same render against the release commit (`github.sha`) and
opens a `chore/post-release-cleanup-<X.Y.Z>` PR with the deletions and
the rendered file. Squash-merge it.

If `changelog.d/` is empty (maintenance release with no fragments), the
render step is skipped cleanly and the release proceeds with the AI
narrative + auto PR list only — no hard failure.

### Preview without committing

```bash
pdm run towncrier build --draft --version <X.Y.Z>
```

`--draft` renders to stdout without touching any files or fragments.
Useful while iterating on a fragment locally.

## 📊 Release Checklist

- [ ] Version updated in `__version__.py` (or auto-bump PR merged)
- [ ] Code owner approval received
- [ ] CI tests passing
- [ ] Merge to main completed
- [ ] Release automatically created
- [ ] PyPI/Docker publishing approved
- [ ] `chore: clear changelog fragments for <X.Y.Z>` PR merged