learningcircuit--local-deep-research/docs/CI_CD_INFRASTRUCTURE.md

CI/CD and Infrastructure Documentation

This document describes the continuous integration, security scanning, and development infrastructure used by the Local Deep Research project.

Overview

The project uses many GitHub Actions workflows and 20+ pre-commit hooks to ensure code quality, security, and reliability.

At-a-glance health: see docs/ci/workflow-status.md — an auto-generated dashboard with live badges for every workflow, surfacing disabled, manual-only, and stale (silently-failing) ones at the top. Regenerate with pdm run python scripts/generate_workflow_status.py.

┌─────────────────────────────────────────────────────────────────┐
│                        Developer Workflow                        │
├─────────────────────────────────────────────────────────────────┤
│  Local Development          │  Pull Request        │  Main/Dev  │
│  ─────────────────          │  ────────────        │  ────────  │
│  • Pre-commit hooks         │  • All tests         │  • Deploy  │
│  • Unit tests               │  • Security scans    │  • Publish │
│  • Linting                  │  • Code review       │  • Release │
└─────────────────────────────────────────────────────────────────┘

Pre-Commit Hooks

Pre-commit hooks run locally before each commit. Install with:

pre-commit install
pre-commit install-hooks

Standard Hooks

Hook	Purpose
check-yaml	Validate YAML syntax
end-of-file-fixer	Ensure files end with newline
trailing-whitespace	Remove trailing whitespace
check-added-large-files	Block files >1MB
check-case-conflict	Prevent case-sensitivity issues
forbid-new-submodules	Prevent git submodules

Security Hooks

Hook	Purpose
gitleaks	Detect secrets, API keys, passwords in code
check-sensitive-logging	Prevent logging of passwords, tokens, keys
check-safe-requests	Enforce SSRF-safe HTTP functions (safe_get, safe_post)
check-url-security	Validate URL handling in JavaScript (XSS prevention)
file-whitelist-check	Only allow approved file types
check-image-pinning	Require SHA256 digests for Docker images

Code Quality Hooks

Hook	Purpose
ruff	Python linter (with auto-fix)
ruff-format	Python formatter (Black-compatible)
eslint	JavaScript linter
shellcheck	Shell script linter
actionlint	GitHub Actions workflow validator
custom-code-checks	Loguru usage, UTC datetime, raw SQL detection

Project-Specific Hooks

Hook	Purpose
check-env-vars	Environment variables must use SettingsManager
check-deprecated-db-connection	Enforce per-user database connections
check-ldr-db-usage	Prevent shared ldr.db usage
check-research-id-type	research_id must be string/UUID, not int
check-datetime-timezone	All DateTime columns (models and migrations) must use UtcDateTime from sqlalchemy_utc
check-session-context-manager	Require context managers for DB sessions
check-pathlib-usage	Use pathlib.Path instead of os.path
check-no-external-resources	No external CDN/resource references
check-css-class-prefix	CSS classes must have ldr- prefix

---

GitHub Actions Workflows

Test Workflows

Workflow	Trigger	Purpose
docker-tests.yml	PR, push	Consolidated Docker tests: pytest + coverage, UI tests (51 Puppeteer tests), LLM tests, infrastructure tests (single Docker build shared across all jobs). Includes tests previously in critical-ui-tests, extended-ui-tests, metrics-analytics-tests, library-ui-tests, mobile-ui-tests, and news-tests workflows.
e2e-research-test.yml	PR, push	End-to-end research flow
fuzz.yml	Schedule	Fuzzing tests

Security Scanning

Workflow	Trigger	Purpose
codeql.yml	PR, push, schedule	GitHub CodeQL analysis
semgrep.yml	PR, push	Semgrep static analysis
osv-scanner.yml	PR, push, schedule	OSV vulnerability scanning (Python + npm)
gitleaks.yml	PR, push	Secret detection
security-tests.yml	PR, push	Security-focused test suite
devskim.yml	PR, push	Microsoft DevSkim analysis
checkov.yml	PR, push	Infrastructure-as-code scanning
container-security.yml	PR, push	Container vulnerability scanning
hadolint.yml	PR, push	Dockerfile linting
owasp-zap-scan.yml	Schedule	OWASP ZAP dynamic scanning
retirejs.yml	PR, push	JavaScript vulnerability scanning
zizmor-security.yml	PR, push	Additional security checks
ossf-scorecard.yml	Schedule	OpenSSF Scorecard
security-headers-validation.yml	PR, push	HTTP security headers
security-file-write-check.yml	PR, push	File write security
npm-audit.yml	PR, push	npm audit for JS dependencies

Dependency Management

Workflow	Trigger	Purpose
dependency-review.yml	PR	Review dependency changes
update-dependencies.yml	Schedule	Auto-update Python deps
update-npm-dependencies.yml	Schedule	Auto-update npm deps
update-precommit-hooks.yml	Schedule	Update pre-commit hooks
validate-image-pinning.yml	PR, push	Verify Docker image pins

UI/Accessibility

Workflow	Trigger	Purpose
responsive-ui-tests-enhanced.yml	PR, push	Responsive design tests

Build & Deploy

Workflow	Trigger	Purpose
prerelease-docker.yml	workflow_call from release.yml	Canonical multi-arch Docker build, cosign sign, SBOM/SLSA attestations. Jobs declare environment: release so the first release env approval gates the build (env-scoped Docker Hub secrets).
docker-publish.yml	workflow_call from release.yml	Retag prerelease manifest as :1.6.9 / :1.6 / :latest (gated by release env). No rebuild — registry-side metadata only. Inlined as a reusable workflow so its result is visible to downstream jobs in release.yml (lets create-release block on Docker success, lets cleanup-on-rejection safely scope cosign artifact deletion).
docker-multiarch-test.yml	PR, push	Multi-architecture build test
publish.yml	repository_dispatch from release.yml	Publish to PyPI. Stays on repository_dispatch (not workflow_call) because PyPI Trusted Publishing rejects OIDC claims from reusable workflows — pypa/gh-action-pypi-publish#166, pypi/warehouse#11096.
release.yml	Push to main, tag v*.*.*, manual	Orchestrate release: gates → build → provenance → prerelease-docker → publish-docker → trigger-pypi → monitor-pypi → create-release (last)

Code Quality

Workflow	Trigger	Purpose
pre-commit.yml	PR, push	Run pre-commit hooks in CI
mypy-type-check.yml	PR, push	Python type checking
ai-code-reviewer.yml	PR	AI-assisted code review
claude-code-review.yml	PR	Claude-based code review

Repository Management

Workflow	Trigger	Purpose
sync-main-to-dev.yml	Push to main	Sync main branch to dev
label-fixed-in-dev.yml	Push to dev	Auto-label fixed issues
danger-zone-alert.yml	PR	Alert on sensitive file changes
check-env-vars.yml	PR, push	Environment variable validation
file-whitelist-check.yml	PR, push	File type validation
version_check.yml	PR, push	Version consistency check

---

Dependabot Configuration

Dependabot automatically creates PRs for dependency updates:

Ecosystem	Directories	Schedule
Python (pip)	/	Weekly (Monday 04:00)
npm	/, /tests/*	Weekly/Daily
GitHub Actions	/	Weekly
Docker	/	Daily

---

Coverage Reporting

Coverage reports are generated by the docker-tests.yml workflow (pytest-tests job):

• HTML Report: Deployed to GitHub Pages at https://learningcircuit.github.io/local-deep-research/coverage/
• PR Comments: Each PR receives a comment with coverage percentage
• Badge: Coverage badge updated via GitHub Gist

Configuration in pyproject.toml:

[tool.coverage.run]
source = ["src"]
omit = ["*/tests/*", "*/migrations/*"]

[tool.coverage.report]
exclude_lines = ["pragma: no cover", "if TYPE_CHECKING:"]

---

Security Architecture

Supply Chain Security

1. Dependency Pinning: All GitHub Actions use SHA256 digests
2. Docker Image Pinning: All base images use SHA256 digests
3. Lock Files: pdm.lock and package-lock.json committed
4. Vulnerability Scanning: OSV-Scanner, npm audit, RetireJS

Runtime Security

1. SSRF Protection: safe_get(), safe_post(), SafeSession wrappers
2. XSS Prevention: DOMPurify for HTML sanitization
3. SQL Injection: SQLAlchemy ORM (no raw SQL)
4. Secret Management: Environment variables via SettingsManager

Container Security

1. Non-root User: Containers run as ldruser:1000
2. Minimal Base Image: Python slim images
3. Health Checks: Docker health check endpoints
4. Read-only Where Possible: Minimal write permissions

---

Running Tests Locally

Quick Test (Unit Tests Only)

pdm run pytest tests/test_settings_manager.py tests/test_utils.py -v

Full Test Suite

pdm run pytest tests/ --ignore=tests/ui_tests --ignore=tests/fuzz -v

With Coverage

pdm run pytest tests/ --cov=src --cov-report=html -v
open coverage/htmlcov/index.html

UI Tests (Requires Server)

# Terminal 1: Start server
pdm run ldr-web

# Terminal 2: Run UI tests
cd tests/ui_tests && npm test

---

Docker Testing

Build and run tests in Docker:

# Build test image
docker build --target ldr-test -t ldr-test .

# Run tests
docker run --rm -v "$PWD":/app -w /app ldr-test \
  pytest tests/ --ignore=tests/ui_tests -v

---

Environment Variables for CI

Variable	Purpose
CI=true	Indicates CI environment
LDR_TESTING_WITH_MOCKS=true	Enable test mocks
LDR_DISABLE_RATE_LIMITING=true	Disable HTTP rate limits in tests (canonical name). The legacy DISABLE_RATE_LIMITING=true is still honored but emits a deprecation warning. Distinct from LDR_RATE_LIMITING_ENABLED, which controls the adaptive search-engine rate limiter — different subsystem.

---

Adding New Workflows

When adding a new workflow:

1. Use pinned action versions with SHA256 digests
2. Add permissions: {} at top level (minimal permissions)
3. Add job-level permissions as needed
4. Include step-security/harden-runner step
5. Add workflow to this documentation

Example template:

name: New Workflow

on:
  pull_request:
    branches: [main]

permissions: {}

jobs:
  example:
    runs-on: ubuntu-latest
    permissions:
      contents: read

    steps:
      - name: Harden the runner
        uses: step-security/harden-runner@... # pinned
        with:
          egress-policy: audit

      - uses: actions/checkout@... # pinned
        with:
          persist-credentials: false