learningcircuit--local-deep-research/docs/RELEASE_GUIDE.md

Release Guide

🚀 Automated Release Process

Releases are fully automated end-to-end whenever a PR that bumps src/local_deep_research/__version__.py is merged to main. No separate tag push, manual workflow trigger, or release-page click is required — the workflow detects the new version, runs all gates, cuts the GitHub release, and (after one approval click) publishes to PyPI and Docker Hub.

PRs that don't touch __version__.py merge normally but skip the release pipeline (the version-check job sees the tag already exists and short-circuits everything downstream).

📋 How Releases Work

1. Automatic Release Creation

• Trigger: Push to main whose __version__.py resolves to a tag that does not yet exist as a GitHub release. In practice this means "merge a PR that bumps __version__.py". Tag pushes (v*.*.*) and manual workflow_dispatch runs also trigger the pipeline and bypass the version-exists check.
• Version: Read from src/local_deep_research/__version__.py by the version-check job; the tag is v<version>.
• Release body: Composed by .github/workflows/release.yml from three sources:
  1. AI narrative generated by OpenRouter (vars.AI_MODEL, default moonshotai/kimi-k2-thinking). The model receives the rendered hand-written notes, the auto-generated PR list, every PR's title + body (batched via one GraphQL call), and the diff between the previous release tag and this one (filtered to drop lockfiles, generated docs, SBOM, static assets, and binary patches; capped at 700k chars).
  2. Hand-written notes from docs/release_notes/<version>.md — rendered from contributor-supplied changelog.d/*.md fragments by the workflow itself at release time (see Release-notes flow below). No manual pdm run towncrier build is required before merging the bump.
  3. Auto-generated PR list from GitHub's generate-notes API, label-categorized.
• No duplicates: If a release for v<version> already exists, the version-check job sets should_release=false and every downstream job (security gate, CI gate, build, publish) is skipped.

2. Approval and Publishing

The release pipeline uses the release GitHub environment to gate the publish steps. DOCKER_USERNAME / DOCKER_PASSWORD are scoped to that environment, so any job that pushes to Docker Hub must declare environment: release and therefore goes through the approval gate.

When you merge to main (or push a tag), the pipeline runs in this order:

1. Security gates + CI gates run automatically.
2. build job runs (version pin, SBOM, Sigstore bundles), then provenance job generates SLSA provenance for those artifacts.
3. One release env approval prompt in release.yml. Approving unlocks all release-env jobs in the same run, which then execute sequentially:
   1. prerelease-docker — canonical multi-arch Docker build, cosign sign, SBOM/SLSA attestations, push as prerelease-v<ver>-<sha> and re-point the floating :prerelease tag.
   2. publish-docker — retags the prerelease manifest as :1.6.9, :1.6, :latest (no rebuild, digest-preserving), then re-verifies digest + cosign + Trivy on the promoted tag.
   3. trigger-pypi — dispatches publish.yml via repository_dispatch (PyPI Trusted Publishing requires the publish step to run in a top-level workflow, so this can't be a reusable workflow_call).
   4. monitor-pypi — polls publish.yml for completion. The inner polling loop times out at 40 minutes (after which the job fails); the surrounding GH Actions timeout-minutes is 90 to leave a safety margin around the poll budget.
   5. create-release — publishes the GitHub Release with SBOM/sig/provenance assets. Runs last, gated on all of the above succeeding, so the public Release never points at missing Docker tags or a missing PyPI version.

If any of prerelease-docker, publish-docker, or monitor-pypi fails, create-release is skipped and no public GitHub Release is created. The cleanup-on-rejection job then handles failure-mode cleanup:

• If publish-docker failed mid-retag (e.g., :1.6.9 landed but :latest failed), it rolls back any landed release tags BEFORE deleting prerelease tags and cosign artifacts (deleting cosign artifacts while release tags share the manifest digest would invalidate release-tag signatures).
• If publish-docker succeeded but a later step (PyPI or create-release) failed, cleanup-on-rejection does NOT fire — Docker release tags exist and their cosign artifacts must stay. See "Recovery from PyPI failure" below.

Recovery from PyPI failure (atomicity hole)

The one orphan state the pipeline cannot fully clean up: publish-docker succeeded, PyPI failed. At this point Docker :1.6.9 / :1.6 / :latest exist and are signed; PyPI has nothing; no GitHub Release. monitor-pypi opens a tracking issue labeled ci-cd. To recover:

1. Inspect the publish.yml workflow run, fix the underlying cause.
2. Manually re-dispatch PyPI publish. client_payload[sha] is REQUIRED (publish.yml fails closed without it) and must be the full 40-hex release commit — the commit the failed release.yml run built, NOT current main HEAD (main may have moved during the approval wait). Get it with gh run view <release-run-id> --json headSha:

   gh api repos/LearningCircuit/local-deep-research/dispatches \
     -f event_type=publish-pypi \
     -F 'client_payload[tag]=v<X.Y.Z>' \
     -F 'client_payload[sha]=<40-hex release commit>'
   

3. Once PyPI publishes successfully, manually create the GitHub Release from the existing tag (the SBOM/sig/provenance artifacts are still uploaded as workflow artifacts on the failed release.yml run; you can download them and attach manually, or re-run create-release manually if the run is still re-runnable in the Actions UI).

Earlier iterations of this refactor described a single approval gate with a pre-approval testing window. That design required DOCKER_USERNAME / DOCKER_PASSWORD to be repo-level secrets so the canonical build could run without env approval. They are env-scoped to release instead, so the gate sits in front of the build. The atomicity refactor preserves this single-approval model — one click unlocks the whole chain, and create-release runs last so the "published Release with broken artifacts" failure mode is closed.

👥 Who Can Release

Code owners (defined in .github/CODEOWNERS):

• @LearningCircuit
• @hashedviking
• @djpetti

📝 Release Workflow

For Regular Releases:

1. Bump version in src/local_deep_research/__version__.py (or merge the auto-bump PR opened by .github/workflows/version_check.yml).
2. Merge to main → Release automatically created. The release workflow renders fragments from changelog.d/*.md into docs/release_notes/<X.Y.Z>.md, composes the body (AI narrative + rendered changelog + auto PR list), and publishes the GitHub release.
3. Approve publishing in GitHub Actions (PyPI/Docker).
4. Merge the cleanup PR opened automatically by the cleanup-changelog job (titled chore: clear changelog fragments for <X.Y.Z>). It persists docs/release_notes/<X.Y.Z>.md and removes the consumed fragments from changelog.d/. Squash-merge — the diff has no review value beyond a sanity check that the rendered notes look right.

To preview the rendered notes locally before merging the bump:

pdm run towncrier build --draft --version <X.Y.Z>

(--draft writes nothing.) Skip if no fragments exist for this release — the workflow tolerates a missing per-version file (warns and proceeds with auto-notes plus AI summary only).

For Hotfixes:

1. Create hotfix branch from main
2. Make minimal fix
3. Bump patch version (e.g., 0.4.3 → 0.4.4)
4. Fast-track review by code owners
5. Merge to main → Automatic release

🔧 Manual Release Options

Option A: Manual Trigger

• Go to Actions → "Create Release" → "Run workflow"
• No inputs are required: the workflow reads the version from src/local_deep_research/__version__.py at HEAD. To release an older or different version, use Option B (push a version tag).

Option B: Version Tags

• git tag v0.4.3 && git push origin v0.4.3
• Automatically creates release; the workflow uses the tag's commit SHA (not main HEAD), so this is the correct path for backporting.

🛡️ Branch Protection

• Main branch is protected
• Required reviews from code owners
• No direct pushes - only via approved PRs
• Status checks must pass (CI tests)

📦 Version Numbering

Follow Semantic Versioning:

• Major (X.0.0): Breaking changes
• Minor (0.X.0): New features, backward compatible
• Patch (0.0.X): Bug fixes, backward compatible

🚨 Emergency Procedures

If automation fails, do NOT create a GitHub release through the UI as the first recovery step — under the atomicity refactor, a manually created GitHub release does NOT trigger publish.yml (it listens only on repository_dispatch) and does NOT trigger docker-publish.yml (workflow_call only). The downstream release: listeners that DO fire (backwards-compatibility.yml, sbom.yml) are observability-only.

Recovery, in order of preference:

1. Check workflow logs in GitHub Actions to identify which job failed, and use the targeted recovery for that failure mode:
   • PyPI failure with Docker already promoted: see Recovery from PyPI failure above.
   • Any other failure: re-run the failed job via the Actions UI if it's still re-runnable (typically within 30 days).
2. Re-trigger the full pipeline via workflow_dispatch if re-running individual jobs isn't possible. Safe for digest-keyed cosign verification — old digests remain valid because their cosign artifacts persist; the new run produces a new digest with its own signatures.
3. Contact code owners if recovery requires manual Docker Hub or PyPI intervention.

📝 Release-notes flow (towncrier news fragments)

Hand-written release notes are assembled from per-PR fragments using towncrier. This replaces the older shared docs/release_notes/<version>.md model, which broke down at LDR's PR throughput (multiple PRs/day racing for the same file).

Contributor side

Each PR with user-visible behavior change drops one tiny markdown file:

changelog.d/<PR-number>.<category>.md

Categories: breaking, security, feature, bugfix, removal, misc (canonical list lives in [[tool.towncrier.type]] entries in pyproject.toml; the pre-commit hook reads it from there). Orphan fragments (no PR/issue number) use changelog.d/+<slug>.<category>.md. The pre-commit hook (recommend-release-notes) nudges contributors who add ≥20 source lines without a fragment, and validates filenames so a typo'd category doesn't silently vanish at render time. See changelog.d/README.md for the full convention.

Maintainer side

The release workflow handles the render. There is nothing to run manually before merging the version bump.

.github/workflows/release.yml (create-release job):

1. Sparse-checks-out changelog.d/ + pyproject.toml, installs towncrier~=24.8.
2. Runs towncrier build --yes --version <X.Y.Z> against the runner's throwaway workspace. Towncrier writes the rendered output to docs/release_notes/<X.Y.Z>.md (per the {version}-templated filename; single_file = false makes this a per-release file rather than appending to a master CHANGELOG) and removes the consumed fragments locally.
3. Reads the rendered file as input to the AI summary and as the "hand-written notes" section of the published GitHub release body.

Persistence to main happens in the cleanup-changelog job, which re-runs the same render against the release commit (github.sha) and opens a chore/post-release-cleanup-<X.Y.Z> PR with the deletions and the rendered file. Squash-merge it.

If changelog.d/ is empty (maintenance release with no fragments), the render step is skipped cleanly and the release proceeds with the AI narrative + auto PR list only — no hard failure.

Preview without committing

pdm run towncrier build --draft --version <X.Y.Z>

--draft renders to stdout without touching any files or fragments. Useful while iterating on a fragment locally.

📊 Release Checklist

• Version updated in __version__.py (or auto-bump PR merged)
• Code owner approval received
• CI tests passing
• Merge to main completed
• Release automatically created
• PyPI/Docker publishing approved
• chore: clear changelog fragments for <X.Y.Z> PR merged