chore: import upstream snapshot with attribution

.github/BRANCH_PROTECTION.md

@@ -0,0 +1,41 @@
+# Branch Protection Configuration
+
+Required settings for OSSF Scorecard compliance on `main` and `dev` branches.
+
+## Required Settings (GitHub UI: Settings -> Branches -> Add rule)
+
+### Protect matching branches
+- [x] Require a pull request before merging
+  - [x] Require at least 1 approving review
+  - [x] Dismiss stale PR approvals when new commits are pushed
+- [x] Require status checks to pass before merging
+  - [x] Require branches to be up to date before merging
+- [x] Do not allow bypassing the above settings
+
+### Status checks to require
+- `test` (unit tests)
+- `lint` (code quality)
+- `typecheck` (mypy)
+
+## Setup Instructions
+
+1. Go to repository **Settings** -> **Branches**
+2. Click **Add branch protection rule**
+3. Set "Branch name pattern" to `main`
+4. Configure the settings listed above
+5. Click **Create**
+6. Repeat for `dev` branch
+
+## Why This Matters
+
+OSSF Scorecard's Branch-Protection check verifies that:
+- Direct pushes to protected branches are prevented
+- Code review is required before merging
+- Status checks must pass before merging
+
+These settings help prevent accidental or malicious changes to production code.
+
+## References
+
+- [OSSF Scorecard Branch-Protection Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection)
+- [GitHub Branch Protection Documentation](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)

.github/CODEOWNERS

@@ -0,0 +1,134 @@
+# Code Owners for Local Deep Research
+# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
+
+# Global owners - these people are automatically requested for review on all PRs
+# and their approval is required for merging to protected branches.
+# NOTE: this list is mirrored in .github/workflows/pr-triage.yml (CODEOWNERS const).
+# Keep both in sync when the maintainer roster changes.
+* @LearningCircuit @hashedviking @djpetti
+
+# Critical infrastructure and release files require maintainer approval
+/Dockerfile @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/__version__.py @LearningCircuit @hashedviking @djpetti
+
+# Core API changes
+/src/local_deep_research/api/ @LearningCircuit @hashedviking @djpetti
+
+# Database migrations and metrics (sensitive for data integrity)
+/src/local_deep_research/metrics/ @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/migrate*.py @LearningCircuit @hashedviking @djpetti
+
+# Database and encryption - critical for security
+/src/local_deep_research/database/ @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/database/ @LearningCircuit @hashedviking @djpetti
+
+# Authentication and security
+/src/local_deep_research/web/auth/ @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/security/ @LearningCircuit @hashedviking @djpetti
+
+# Core application
+/src/local_deep_research/web/app.py @LearningCircuit @hashedviking @djpetti
+
+# Security configurations
+/docker-compose*.yml @LearningCircuit @hashedviking @djpetti
+
+# Pre-commit security hooks (contains security checks - do not tamper)
+/.pre-commit-config.yaml @LearningCircuit @hashedviking @djpetti
+/.pre-commit-hooks/ @LearningCircuit @hashedviking @djpetti
+
+# Security scan ignore files (lower risk - for managing false positives)
+/.trivyignore* @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.gitleaksignore @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.gitleaks.toml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.semgrepignore @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# Notifications
+/src/local_deep_research/notifications/ @tombii @LearningCircuit @hashedviking @djpetti
+/tests/notifications/ @tombii @LearningCircuit @hashedviking @djpetti
+
+# Scripts
+/scripts/ @tombii @LearningCircuit @hashedviking @djpetti
+
+# Tests and CI
+/tests/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# Test workflows (excluding security-tests.yml)
+/.github/workflows/docker-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/e2e-research-test.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/responsive-ui-tests-enhanced.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/playwright-webkit-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/puppeteer-e2e-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/fuzz.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# Linting and checking workflows
+/.github/workflows/mypy-type-check.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/hadolint.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/validate-image-pinning.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# CI utilities
+/.github/workflows/docker-multiarch-test.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/.github/workflows/label-fixed-in-dev.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# Documentation and examples
+/docs/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/examples/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/README.md @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @djpetti @hashedviking @LearningCircuit
+
+# UI templates (cosmetic changes)
+/src/local_deep_research/web/templates/pages/benchmark.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/benchmark_results.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/collection_details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/collections.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/cost_analytics.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/document_details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/download_manager.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/embedding_settings.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/history.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/library.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/metrics.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/news.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/progress.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/research.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/results.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/templates/pages/subscriptions.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# CSS styles
+/src/local_deep_research/web/static/css/styles.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/collections.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/collection_details.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/document_details.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/mobile-responsive.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/mobile-navigation.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/news.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/settings.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/web/static/css/subscriptions.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# LLM provider settings (schema only, no secrets)
+/src/local_deep_research/defaults/llm_providers/google_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/defaults/llm_providers/ionos_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/defaults/llm_providers/openrouter_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/defaults/llm_providers/xai_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# Search engine settings (schema only, no secrets)
+/src/local_deep_research/defaults/settings/search_engines/elasticsearch.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/defaults/settings/search_engines/paperless.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/defaults/settings/search_engines/scaleserp.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+/src/local_deep_research/defaults/settings/search_engines/serper.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
+
+# ──────────────────────────────────────────────────────────────────────
+# Repository integrity guardrails — maintainer-only approval
+#
+# These files control what is tracked in git and what passes CI/pre-commit
+# whitelist checks. Restricted to prevent re-introduction of broad
+# file-type exceptions (e.g. binary wildcards) without explicit maintainer
+# approval. Binary files bloat the repo and should not be added to main.
+#
+# IMPORTANT: these rules MUST remain the last entries in this file
+# (GitHub CODEOWNERS uses last-match-wins).
+# ──────────────────────────────────────────────────────────────────────
+/.gitignore @LearningCircuit
+/.file-whitelist.txt @LearningCircuit
+/.pre-commit-hooks/file-whitelist-check.sh @LearningCircuit
+/.github/scripts/file-whitelist-check.sh @LearningCircuit
+/.github/CODEOWNERS @LearningCircuit

.github/FUNDING.yml

@@ -0,0 +1,13 @@
+# These are supported funding model platforms
+
+# How donations are used:
+# - Ongoing development and maintenance
+# - Infrastructure costs (hosting, API keys for testing)
+# - Feature development and improvements
+# - Bug fixes and security updates
+# - Supporting contributors who need it most
+# - Building project reserves for future needs
+# - Exploring options to provide free public hosting of LDR
+
+github: [LearningCircuit, djpetti, hashedviking] # GitHub Sponsors for individual contributors
+ko_fi: localdeepresearch # Support the project with one-time donations

.github/FUZZING.md

@@ -0,0 +1,68 @@
+# Fuzzing Strategy
+
+This document explains our fuzzing approach and addresses OSSF Scorecard's
+Fuzzing check.
+
+## Current Implementation
+
+This project uses **Hypothesis** for property-based fuzz testing:
+
+- **Workflow**: `.github/workflows/fuzz.yml`
+- **Tests**: `tests/fuzz/test_fuzz_security.py`, `tests/fuzz/test_fuzz_utilities.py`
+- **Schedule**: Weekly on Sunday + on changes to security/utilities code
+
+### What We Test
+
+Our fuzz tests cover:
+- Input validation edge cases
+- Security-sensitive string handling
+- API boundary testing
+- File path validation
+- URL parsing and sanitization
+
+### Test Configuration
+
+```yaml
+# Regular CI runs
+--hypothesis-seed=0  # Reproducible tests
+
+# Extended scheduled runs
+HYPOTHESIS_PROFILE=extended  # More examples, deeper exploration
+```
+
+## Why Not OSS-Fuzz?
+
+OSSF Scorecard's Fuzzing check looks for integration with:
+- [OSS-Fuzz](https://github.com/google/oss-fuzz) - Google's continuous fuzzing
+- [ClusterFuzzLite](https://google.github.io/clusterfuzzlite/) - Lightweight alternative
+- [OneFuzz](https://github.com/microsoft/onefuzz) - Microsoft's fuzzing platform
+
+**These are not appropriate for this project because:**
+
+1. **OSS-Fuzz targets native code** - Designed for C/C++ vulnerabilities using
+   libFuzzer/AFL++. This project is primarily Python.
+
+2. **Hypothesis is the Python equivalent** - Property-based testing with
+   automatic example generation provides equivalent security value.
+
+3. **No native code attack surface** - Our security-sensitive code is Python,
+   where Hypothesis testing is the standard approach.
+
+## OSSF Scorecard Note
+
+Scorecard's "Fuzzing" check does not recognize Hypothesis-based testing.
+This is a known limitation of the check. Our fuzzing implementation provides
+equivalent security value for Python codebases.
+
+## Future Considerations
+
+If the project adds significant native code dependencies (C extensions, FFI),
+we would consider:
+- ClusterFuzzLite integration for continuous fuzzing
+- Native fuzz targets for critical C code paths
+
+## References
+
+- [Hypothesis Documentation](https://hypothesis.readthedocs.io/)
+- [OSSF Scorecard Fuzzing Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#fuzzing)
+- [Property-Based Testing in Python](https://hypothesis.works/)

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,34 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: 'bug'
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Run command '...'
+2. Enter query '....'
+3. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**System Information:**
+ - OS: [e.g. Ubuntu 22.04]
+ - Python Version: [e.g. 3.9.0]
+ - Model Used: [e.g. deepseek-r1:14b]
+ - Hardware Specs: [e.g. 32GB RAM, NVIDIA RTX 3080]
+
+**Additional context**
+Add any other context about the problem here.
+
+**Output/Logs**
+```
+If applicable, add relevant output or error logs here
+```

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,79 @@
+# Contributing to Local Deep Research
+
+We love your input! We want to make contributing to Local Deep Research as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+- Becoming a maintainer
+
+## Development Process
+
+We use GitHub to host code, to track issues and feature requests, as well as accept pull requests.
+
+1. Fork the repo and create your branch from `main`
+2. If you've added code that should be tested, add tests
+3. If you've changed APIs, update the documentation
+4. Ensure the test suite passes
+5. Make sure your code follows the existing style
+6. Issue that pull request!
+
+## Any Contributions You Make Will Be Under the MIT License
+
+When you submit code changes, your submissions are understood to be under the same [MIT License](../../LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
+
+## Report Bugs Using GitHub's [Issue Tracker](https://github.com/LearningCircuit/local-deep-research/issues)
+
+Report a bug by [opening a new issue](https://github.com/LearningCircuit/local-deep-research/issues/new); it's that easy!
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+  - Be specific!
+  - Give sample code if you can
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
+
+## Development Setup
+
+1. Clone your fork of the repository:
+```bash
+git clone https://github.com/YOUR_USERNAME/local-deep-research.git
+cd local-deep-research
+```
+
+2. Create a virtual environment and activate it:
+```bash
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+3. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+4. Install Ollama and set up your preferred model:
+```bash
+# Follow instructions at https://ollama.ai for installation
+ollama pull deepseek-r1:14b  # Or mistral:7b for lighter hardware
+```
+
+## Code Style
+
+- Follow PEP 8 guidelines
+- Use type hints where possible
+- Write descriptive docstrings for functions and classes
+- Keep functions focused and modular
+- Comment complex logic
+
+## License
+
+By contributing, you agree that your contributions will be licensed under its MIT License.
+
+## References
+
+This document was adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/master/CONTRIBUTING.md).

.github/PULL_REQUEST_TEMPLATE/first_time_contributor.md

@@ -0,0 +1,21 @@
+## Description
+
+Fixes #
+
+## Note on File Types
+
+This project uses a file whitelist for security (configured in `.gitignore`). Currently allowed: `.py`, `.js`, `.html`, `.css`, `.json`, `.md`, `.yml`, `.yaml`, `.sh`, `.cfg`, `.ipynb`, `.template`
+
+Need a different file type? We're happy to discuss it in the comments, and if approved, it can be added to `.gitignore`.
+
+## Pre-commit Hooks
+
+If you haven't already, consider setting up pre-commit hooks for automatic code formatting:
+```bash
+pre-commit install
+pre-commit install-hooks
+```
+
+---
+
+Thanks for your first contribution to Local Deep Research! Welcome to the community. We genuinely appreciate the time and effort you've put into this PR 🎉

.github/PULL_REQUEST_TEMPLATE/pull_request_template.md

@@ -0,0 +1,12 @@
+## Description
+
+Fixes #
+
+## CI test coverage
+
+By default, this PR runs the unit/lint checks. Heavy E2E suites are label-gated — add the label that matches what you touched so the relevant workflow runs:
+
+- `test:puppeteer` or `test:e2e` — Puppeteer E2E suite (~30–60 min; uses paid LLM/search API quotas).
+- `ldr_research` or `ldr_research_static` — LDR research integration workflow that posts findings as a PR comment.
+
+WebKit/Mobile Safari tests run on the daily 02:00 UTC schedule and at release; the responsive UI suite runs at release and on manual dispatch.

.github/SECURITY_ALERTS.md

@@ -0,0 +1,218 @@
+# Security Alert Assessment
+
+This document explains the security scanning alerts that have been assessed
+and determined to be false positives or intentionally suppressed.
+
+## DS162092 - Hardcoded URLs
+
+**Status:** Excluded in DevSkim workflow via `exclude-rules`
+
+### Explanation
+
+This rule is excluded because this research tool legitimately integrates with
+external APIs. All hardcoded URLs are intentional service endpoints:
+
+- **ArXiv** - Academic paper repository (`https://arxiv.org/`)
+- **PubMed** - Medical literature database
+- **Semantic Scholar** - AI-powered research tool
+- **OpenAlex** - Open catalog of scholarly works
+- **Archive.org** - Wayback Machine integration
+
+### Why Exclusion Is Safe
+
+1. **Legitimate API endpoints** - URLs are for real research services
+2. **SSRF protection** - Production code uses
+   `src/local_deep_research/security/ssrf_validator.py` to block dangerous URLs
+3. **No user-controlled URLs** - All URLs are hardcoded service endpoints
+4. **Test coverage** - URL handling is tested in `tests/fuzz/test_security_fuzzing.py`
+
+---
+
+## DS137138 - Hardcoded Credentials
+
+**Status:** Excluded via workflow configuration (no action needed)
+
+### Explanation
+
+All ~100+ hardcoded credential alerts are in the `tests/` directory and are
+intentional mock data for testing.
+
+### Why They Are Safe
+
+1. **All instances are test fixtures** - Named clearly as mock data:
+   - `api_key="test_key"`
+   - `password="testpass"`
+   - `sample_data_with_secrets()`
+
+2. **DevSkim already excludes tests** - The `.github/workflows/devskim.yml`
+   configuration includes:
+   ```yaml
+   ignore-globs: 'examples/**,tests/**'
+   ```
+
+3. **Never real credentials** - All test values are obviously fake
+
+4. **Gitleaks handles real secrets** - The `.github/workflows/gitleaks.yml`
+   workflow scans for actual leaked credentials
+
+### Example Test Fixtures
+
+- `tests/fixtures/mock_credentials.py`
+- `tests/unit/auth/test_login.py`
+- `tests/integration/api/test_authentication.py`
+
+---
+
+## DS172411 - JavaScript DOM (innerHTML)
+
+**Status:** Addressed with XSS protection infrastructure
+
+### Explanation
+
+The codebase has comprehensive XSS protection infrastructure in
+`src/local_deep_research/web/static/js/security/xss-protection.js`:
+
+| Protection Function | Purpose |
+|---------------------|---------|
+| `escapeHtml()` | HTML entity escaping for text content |
+| `sanitizeHtml()` | DOMPurify-based HTML sanitization |
+| `safeSetInnerHTML()` | Safe innerHTML wrapper |
+| `sanitizeUserInput()` | User input validation and sanitization |
+
+### Current Status
+
+| Category | Count | Status |
+|----------|-------|--------|
+| Using `escapeHtml()` | ~35 | Safe |
+| Using `textContent` | ~20 | Safe |
+| Static HTML only | ~15 | Safe |
+| Using `sanitizeHtml()` | ~5 | Safe |
+
+All innerHTML usages have been reviewed and appropriate sanitization applied.
+
+---
+
+## DS176209 - Suspicious Comments
+
+**Status:** Excluded in DevSkim workflow via `exclude-rules`
+
+### Explanation
+
+DevSkim flags comments containing words like `TODO`, `FIXME`, `HACK`, `BUG`,
+`XXX` as "suspicious". These are **standard development annotations** used
+to track technical debt and future work.
+
+### Why Exclusion Is Safe
+
+1. **Not a security rule** - This is a code quality check, not security
+2. **Standard practice** - TODO/FIXME comments are used in every codebase
+3. **No runtime impact** - Comments have no effect on application behavior
+4. **IDE support** - Development tools already track these annotations
+
+---
+
+## DS126858 - Weak/Broken Hash Algorithm
+
+**Status:** Excluded in DevSkim workflow via `exclude-rules`
+
+### Explanation
+
+DevSkim flags any literal occurrence of `sha1` as a "weak/broken hash
+algorithm". In this codebase the only matches are:
+
+1. **SLSA provenance JSON keys** in `.github/workflows/prerelease-docker.yml` —
+   the `"sha1"` key inside `digest` objects is part of the
+   [SLSA in-toto provenance schema](https://slsa.dev/spec/v0.2/provenance) and
+   identifies the algorithm Git itself uses for commit hashes. We are not
+   choosing SHA-1 as a cryptographic primitive — Git's commit identifier
+   format is fixed.
+
+2. **SQLCipher KDF/HMAC algorithm enums** in
+   `src/local_deep_research/settings/env_definitions/db_config.py`
+   (`PBKDF2_HMAC_SHA1`, `HMAC_SHA1`). These exist for backwards-compatibility
+   with existing user databases; the default is SHA-512. Each occurrence
+   carries an inline `# DevSkim: ignore DS126858` annotation with rationale.
+
+### Why Exclusion Is Safe
+
+1. **Not a cryptographic choice** - The `sha1` strings in SLSA provenance are
+   *protocol-mandated key names*, not crypto operations we control.
+2. **Git's commit hashing is SHA-1 by design** - The Linux kernel and every
+   git-backed project produces SHA-1 commit IDs; SLSA records them honestly.
+3. **Real SHA-1 misuse would be reviewed** - The SQLCipher backwards-compat
+   uses are documented and reviewed; new uses of SHA-1 as a cryptographic
+   primitive would be caught in code review and by CodeQL.
+4. **No password/signature SHA-1 in this codebase** - Authentication uses
+   `secrets`/Argon2-class KDFs and SQLCipher's SHA-512 default.
+
+---
+
+## Container Image CVEs
+
+**Status:** Documented, awaiting upstream fixes
+
+### No Fix Available
+
+The following CVEs are in the Debian base image packages with no upstream
+fixes currently available:
+
+| CVE | Package | Severity | Notes |
+|-----|---------|----------|-------|
+| CVE-2025-14104 | util-linux | Medium | No fix version |
+| CVE-2022-0563 | util-linux | Low | Debian won't fix |
+| CVE-2025-6141 | Various | Low | No fix version |
+
+These are monitored and will be addressed when fixes become available.
+
+---
+
+## DevSkim Rule Exclusions Summary
+
+The following rules are excluded in `.github/workflows/devskim.yml`:
+
+| Rule | Name | Reason |
+|------|------|--------|
+| DS162092 | Hardcoded URL | Legitimate API endpoints for research services |
+| DS176209 | Suspicious Comment | Standard TODO/FIXME annotations |
+| DS137138 | Hardcoded Credentials | All matches are test fixtures (mock data) |
+| DS148264 | Use cryptographic random | All `random` usages are non-security (ML shuffle, jitter) |
+| DS172411 | setTimeout code injection | All setTimeout calls pass function refs, never strings |
+| DS126858 | Weak/Broken Hash Algorithm | SLSA-schema-required `sha1` JSON key + SQLCipher backwards-compat enums |
+
+### Review Cadence
+
+These exclusions should be reviewed **quarterly** to ensure:
+- No new security-relevant URLs are being masked
+- Exclusions remain appropriate as the codebase evolves
+- New DevSkim rules are evaluated for applicability
+
+**Last reviewed:** May 2026
+
+---
+
+## GitHub Security Tab Dismissals
+
+Some security alerts can only be dismissed, or are very difficult to
+suppress, outside the
+[GitHub Security tab](https://docs.github.com/en/code-security/dependabot/dependabot-alerts/viewing-and-updating-dependabot-alerts).
+This is a
+[GitHub platform limitation](https://github.com/orgs/community/discussions/163277) —
+Dependabot alerts, code scanning alerts, and secret scanning alerts are
+managed primarily through the repository UI rather than via configuration
+files or inline annotations.
+
+Dismissals made through the Security tab include a reason (e.g.,
+"tolerable in this context", "no bandwidth to fix", "false positive") and
+an optional comment, but these are only visible to users with repository
+write access. GitHub provides no export or in-repo tracking mechanism, so
+unlike the other suppressions documented in this file, these dismissals
+cannot be tracked in version-controlled files.
+
+---
+
+## References
+
+- [DevSkim Configuration](workflows/devskim.yml)
+- [Gitleaks Configuration](workflows/gitleaks.yml)
+- [XSS Protection Module](../src/local_deep_research/web/static/js/security/xss-protection.js)
+- [SSRF Validator](../src/local_deep_research/security/ssrf_validator.py)

.github/SECURITY_SCORECARD.md

@@ -0,0 +1,203 @@
+# OSSF Scorecard Compliance Notes
+
+This document explains our compliance status with OSSF Scorecard checks
+and documents any accepted risks or false positives.
+
+## Current Score: 8/10 for Pinned-Dependencies
+
+**Summary:**
+- ✅ 196/196 GitHub Actions pinned by SHA
+- ✅ 4/4 Container images pinned by digest
+- ✅ 1/1 Security tools use official SHA-pinned actions (zizmor)
+- ⚠️ 2/27 pip commands pinned by hash (remainder version-pinned — accepted risk)
+- ⚠️ 21/24 npm commands pinned (3 are operational commands)
+- ⚠️ 1 false positive for downloadThenRun
+- ⚠️ APT packages intentionally unpinned (base image controls versions)
+
+## Pinned-Dependencies
+
+### GitHub Actions: COMPLIANT ✅
+
+All GitHub Actions use commit SHA pinning (40-character hex):
+
+```yaml
+# Examples from our workflows:
+actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3  # v6.0.0
+actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548  # v6
+step-security/harden-runner@df199fb7be9f65074067a9eb93f12bb4c5547cf2  # v2.13.3
+```
+
+### Docker Images: COMPLIANT ✅
+
+All Docker images use SHA256 digest pinning:
+
+```yaml
+# Examples from our workflows and docker-compose:
+python:3.13.9-slim@sha256:326df678c20c78d465db501563f3492d17c42a4afe33a1f2bf5406a1d56b0e86
+redis:alpine@sha256:8360960f5fb56a282d78686203dd875862cd4b52a4184c17ac753690252d6d31
+node:20-alpine@sha256:bcd88137d802e2482c9df3cdec71e0431857ebbbdba6973776b5593214056d86
+```
+
+### Official GitHub Actions: COMPLIANT ✅
+
+One security tool uses an official GitHub Action with SHA-pinning,
+which provides equivalent security to hash-pinned pip installs:
+
+| Tool | Action | SHA |
+|------|--------|-----|
+| zizmor | `zizmorcore/zizmor-action` | `135698455da5c3b3e55f73f4419e481ab68cdd95` |
+
+This official action runs the tool in a container with internal integrity verification,
+which OSSF Scorecard accepts as equivalent to hash pinning.
+
+> **Note:** pip-audit was removed because it duplicates OSV-Scanner's coverage (both
+> query the OSV database) and its internal use of pip's dependency resolver conflicts
+> with PDM's `[tool.pdm.resolution.overrides]`, causing persistent CI failures.
+
+> **Note:** Checkov previously used `bridgecrewio/checkov-action` but was reverted to CLI
+> installation (`pip install checkov==3.2.499`) due to a known bug with multiple
+> consecutive action calls. See [checkov-action#170](https://github.com/bridgecrewio/checkov-action/issues/170)
+> and [checkov#5866](https://github.com/bridgecrewio/checkov/issues/5866).
+
+### pip install: VERSION-PINNED (Accepted Risk) ⚠️
+
+Scorecard flags `pip install package==version` because it prefers hash pinning.
+The remaining pip commands use exact version pinning.
+
+**Flagged commands and their status:**
+
+| File | Line | Command | Status |
+|------|------|---------|--------|
+| Dockerfile | 49-50 | `pip3 install pip==26.0` `pip install pdm==2.26.2...` | Version-pinned |
+| Dockerfile | 194 | `pip3 install pip==26.0` | Version-pinned |
+| check-env-vars.yml | 36 | `pip install loguru==0.7.3 sqlalchemy==2.0.36...` | Version-pinned |
+| checkov.yml | 33 | `pip install checkov==3.2.499` | Version-pinned |
+| fuzz.yml | 51-52 | `pip install pip==25.0` `pip install pdm==2.26.2` | Version-pinned |
+| mypy-type-check.yml | 39,47-48 | `pip install pdm==2.26.2 mypy==1.14.1...` | Version-pinned |
+| publish.yml | 139,328 | `pip install pdm==2.26.2` `pip install wheel==0.46.2` | Version-pinned |
+| puppeteer-e2e-tests.yml | 65 | `pip install -e .` | Local package |
+| responsive-ui-tests-enhanced.yml | 97-98 | `pip install pip==25.0` `pip install -e .` | Version-pinned + local (Alert #5688) |
+| semgrep.yml | 36 | `pip install semgrep==1.87.0` | Version-pinned |
+| update-precommit-hooks.yml | 35-38 | `pip install pip==25.0` `pip install pre-commit-update==0.6.1` | **Hash-pinned** |
+| validate-image-pinning.yml | 67 | `pip install pyyaml==6.0.2` | Version-pinned |
+| backwards-compatibility.yml | 66-68 | `pip install --upgrade pip` `pip install pytest` `pip install -e .` | Intentionally unpinned |
+| backwards-compatibility.yml | 106-107 | `pip install --upgrade pip` `pip install pytest` | Intentionally unpinned |
+| backwards-compatibility.yml | 154-155 | `pip install --upgrade pip` `pip install "local-deep-research==..."` | Intentionally unpinned |
+| backwards-compatibility.yml | 169 | `pip install -e .` | Local package |
+
+> **Note:** `backwards-compatibility.yml` intentionally uses unpinned pip commands
+> because it tests compatibility with prior PyPI releases of local-deep-research.
+> Pinning these commands would defeat the purpose of the compatibility tests.
+
+**Why we don't use hash pinning:**
+
+1. **Platform-specific hashes**: pip package hashes vary by Python version, OS, and architecture.
+   A single hash won't work across different CI runners.
+
+2. **Maintenance burden**: Every version update requires regenerating hashes for all platforms.
+
+3. **Marginal security benefit**: These are dev/CI tools running in hardened CI environments
+   (step-security/harden-runner) with egress auditing. Supply chain attacks on PyPI packages
+   are mitigated by version pinning and short execution windows.
+
+4. **Industry practice**: Version pinning (`==`) is the standard for CI tool installation.
+   Hash pinning is typically reserved for production dependencies.
+
+5. **Local packages**: `pip install -e .` installs the local source code and cannot be hash-pinned.
+
+6. **Bootstrap commands**: `pip install pip==X.Y.Z` cannot be hash-pinned because pip is the
+   tool performing the verification — it cannot verify its own integrity during a self-upgrade.
+
+### Specific OSSF Scorecard Alerts
+
+| Alert | File | Lines | Description | Status |
+|-------|------|-------|-------------|--------|
+| #5688 | responsive-ui-tests-enhanced.yml | 97-98 | `pip install pip==25.0` (bootstrap) and `pip install -e .` (local source) | Accepted limitation |
+
+### npm Commands: MOSTLY COMPLIANT (21/24) ⚠️
+
+The 3 "unpinned" npm commands are operational commands, not package installations:
+
+| File | Line | Command | Reason Not Pinned |
+|------|------|---------|-------------------|
+| npm-audit.yml | 56 | `npm i --package-lock-only` | Generates lockfile only |
+| npm-audit.yml | 72 | `npm i --package-lock-only` | Generates lockfile only |
+| update-npm-dependencies.yml | 74 | `npm update` | Intentionally updates to latest |
+
+These commands don't install packages directly - they either generate lockfiles
+or intentionally update packages. They cannot and should not be "pinned".
+
+### downloadThenRun: FALSE POSITIVE ⚠️
+
+**Flagged:** `examples/elasticsearch/test_elasticsearch.sh:60`
+
+```bash
+curl -s http://localhost:9200 | python3 -m json.tool | head -10
+```
+
+**Why it's a false positive:**
+- This fetches JSON from localhost:9200 (local Elasticsearch)
+- Pipes to `python3 -m json.tool` (stdlib JSON formatter)
+- Shows first 10 lines of pretty-printed output
+
+This is NOT downloading and running a remote script. It's formatting local JSON output.
+The scorecard pattern-matches `curl | python` as potentially dangerous, but this
+is a safe operation on localhost data.
+
+**OSSF Scorecard Alert:** #4411
+
+### APT Packages: INTENTIONALLY UNPINNED ⚠️
+
+**Files affected:** `publish.yml`, `e2e-research-test.yml`, `responsive-ui-tests-enhanced.yml`, `Dockerfile`
+
+| File | Packages | Runner/Base |
+|------|----------|-------------|
+| publish.yml | libsqlcipher-dev, patchelf | ubuntu-22.04 |
+| e2e-research-test.yml | jq | ubuntu-22.04 |
+| responsive-ui-tests-enhanced.yml | wget, gnupg, ca-certificates, fonts-liberation, etc. | ubuntu-latest |
+| Dockerfile | curl, git, build-essential, etc. | python:3.13.9-slim@sha256:... |
+
+**Rationale for NOT pinning APT packages:**
+
+1. **Version availability**: Old APT package versions are removed from Ubuntu archives after 6-12 months.
+   Pinning to `package=1.2.3-1ubuntu1` causes builds to fail when that version is removed.
+
+2. **Base image controls versions**: Docker base images are SHA-pinned, which deterministically controls
+   which APT package versions are available. The combination of `python:3.13.9-slim@sha256:326df678...`
+   and `apt-get install curl` produces the same result every time that base image is used.
+
+3. **Runner stability**: GitHub workflow runners use pinned Ubuntu versions (e.g., `ubuntu-22.04`)
+   which provide consistent package versions throughout the runner's lifecycle.
+
+4. **Version variation**: APT package version strings vary between Ubuntu releases and architectures,
+   making cross-platform pinning impractical.
+
+5. **Industry consensus**: Security experts recommend pinning the base image/runner rather than
+   individual packages. Base image pinning provides stronger guarantees with lower maintenance burden.
+
+**Mitigations in place:**
+
+- ✅ Docker base images pinned to SHA256 digests (see Docker Images section above)
+- ✅ GitHub runner versions pinned where practical (ubuntu-22.04)
+- ✅ Dependabot configured to monitor for security updates
+- ✅ Step-security/harden-runner audits all egress traffic
+- ✅ Minimal package sets installed (only what's needed)
+
+### Enforcement
+
+We have automated verification for our pinning strategy:
+- `.github/workflows/validate-image-pinning.yml` - Validates Docker image digests
+- Pre-commit hooks verify action SHA pinning
+- All pip install commands use explicit version specifiers (except `backwards-compatibility.yml` which intentionally tests with unpinned versions)
+
+### Review Cadence
+
+These decisions are reviewed quarterly to ensure they remain appropriate:
+- **Next review:** Q2 2026
+- **Owner:** Security team
+
+## References
+
+- [OSSF Scorecard Pinned-Dependencies Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies)
+- [StepSecurity Harden Runner](https://github.com/step-security/harden-runner)
+- [pip Hash Checking Mode](https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode)

.github/actionlint.yaml

@@ -0,0 +1,4 @@
+self-hosted-runner:
+  # Custom runner labels used in this repository
+  labels:
+    - ubuntu-24.04-arm

.github/codeql/codeql-config.yml

@@ -0,0 +1,8 @@
+name: "CodeQL Config"
+
+# Paths to ignore - exclude test files and security-reviewed intentional features
+paths-ignore:
+  - '**/tests/**'
+  - '**/test_*.py'
+  - '**/*_test.py'
+  - 'src/local_deep_research/security/file_write_verifier.py'

.github/dependabot.yml

@@ -0,0 +1,95 @@
+# Dependabot configuration for automated dependency updates
+
+version: 2
+updates:
+  # Python dependencies
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "04:00"
+    open-pull-requests-limit: 5
+    labels:
+      - "dependencies"
+      - "python"
+    commit-message:
+      prefix: "chore"
+      include: "scope"
+    ignore:
+      # faiss-cpu >=1.14 wheels execute AVX2 instructions unconditionally at
+      # dlopen and SIGILL on CPUs without AVX2 (#4480). Do not propose bumping
+      # the constraint past 1.13.x until upstream ships an AVX2-safe wheel
+      # (facebookresearch/faiss#5296). Remove this ignore together with the
+      # pyproject pin when lifting it — exit conditions tracked in #4499.
+      - dependency-name: "faiss-cpu"
+        versions: [">=1.14"]
+
+  # JavaScript dependencies
+  - package-ecosystem: "npm"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "04:00"
+    open-pull-requests-limit: 5
+    labels:
+      - "dependencies"
+      - "javascript"
+    commit-message:
+      prefix: "chore"
+      include: "scope"
+
+  # GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "04:00"
+    labels:
+      - "dependencies"
+      - "github-actions"
+    commit-message:
+      prefix: "chore"
+      include: "scope"
+
+  - package-ecosystem: docker
+    directory: /
+    schedule:
+      interval: daily
+
+  - package-ecosystem: npm
+    directory: /tests/api_tests_with_login
+    schedule:
+      interval: daily
+
+  - package-ecosystem: npm
+    directory: /tests/infrastructure_tests
+    schedule:
+      interval: daily
+
+  - package-ecosystem: npm
+    directory: /tests
+    schedule:
+      interval: daily
+
+  - package-ecosystem: npm
+    directory: /tests/puppeteer
+    schedule:
+      interval: daily
+
+  - package-ecosystem: npm
+    directory: /tests/ui_tests
+    schedule:
+      interval: daily
+
+  - package-ecosystem: npm
+    directory: /tests/ui_tests/playwright
+    schedule:
+      interval: daily
+
+  - package-ecosystem: npm
+    directory: /tests/accessibility_tests
+    schedule:
+      interval: daily

.github/labels.yml

@@ -0,0 +1,58 @@
+# PR triage labels managed declaratively by .github/workflows/labels-sync.yml.
+# This file is an additive sync — labels not listed here are left untouched.
+# Persistent labels (applied at PR open by .github/workflows/pr-triage.yml):
+- name: external-contributor
+  color: 'cccccc'
+  description: PR from someone outside the maintainer team
+- name: first-time-contributor
+  color: 'c2e0c6'
+  description: First contribution to this repo
+- name: bot
+  color: 'ededed'
+  description: PR opened by an automated account
+- name: needs-rework
+  color: 'e99695'
+  description: PR needs substantive rework before it can be reviewed in depth
+# Lifecycle labels (toggled per-PR by .github/workflows/pr-triage.yml):
+- name: needs-codeowner-review
+  color: 'fbca04'
+  description: Awaiting first review from a global codeowner
+- name: awaiting-author
+  color: 'd4c5f9'
+  description: Codeowner requested changes; ball is with author
+- name: awaiting-codeowner
+  color: 'fef2c0'
+  description: Author has responded; needs codeowner re-review
+# LDR research trigger labels (used by .github/workflows/e2e-research-test.yml
+# and .github/workflows/issue-research.yml — applied manually by maintainers,
+# auto-removed by the workflow after the run completes).
+# Descriptions are phrased as direct instructions so the AI code reviewer
+# (.github/workflows/ai-code-reviewer.yml) can decide when to apply them.
+- name: ldr_research
+  color: '7057ff'
+  description: Apply for substantive logic/architecture changes — when in doubt, add it. Runs full LDR research.
+- name: ldr_research_static
+  color: '7057ff'
+  description: Cheaper variant of ldr_research for routine code changes — favor running this over no analysis.
+# E2E test trigger labels (used by .github/workflows/puppeteer-e2e-tests.yml).
+# Both labels trigger the same Puppeteer suite; descriptions guide the AI
+# reviewer toward running the suite for any PR that touches the web stack.
+- name: 'test:puppeteer'
+  color: '0e8a16'
+  description: Apply if PR touches UI, web routes, auth, or templates — when in doubt, add it. Runs Puppeteer E2E.
+- name: 'test:e2e'
+  color: '1d76db'
+  description: Alias for test:puppeteer (same Puppeteer workflow). Apply one; both trigger the same job.
+# Full sharded UI test trigger (used by .github/workflows/ui-full-shards.yml).
+# Invokes docker-tests.yml with strict-mode=true so the 14-way ui-tests matrix
+# runs. Heavy — re-runs the full docker-tests job graph including pytest.
+- name: 'test:ui-full-shards'
+  color: '0e8a16'
+  description: Apply if PR touches puppeteer test infra or you want all 14 UI shards — costs ~45 min CI time.
+# Human-only signal labels — must NOT be set by any workflow, bot, or AI
+# reviewer. These exist to communicate a human judgement that automation
+# cannot make. If you find a workflow or hook applying one of these,
+# that's a bug — remove the automation, don't extend the label list.
+- name: code-ready
+  color: '006b75'
+  description: Code looks technically ready; still needs CI + approver. Apply manually — never auto-applied.

.github/release.yml

@@ -0,0 +1,84 @@
+changelog:
+  categories:
+    - title: 🔒 Security Updates
+      labels:
+        - security
+    - title: 💥 Breaking Changes
+      labels:
+        - breaking-change
+        - critical-changes
+    - title: ✨ New Features
+      labels:
+        - feature
+        - enhancement
+        - dev-enhancement
+    - title: 🐛 Bug Fixes
+      labels:
+        - bug
+        - bugfix
+        - dev-bugfix
+    - title: ⚡ Performance Improvements
+      labels:
+        - performance
+    - title: 🗄️ Database Changes
+      labels:
+        - database
+    - title: 📚 Documentation
+      labels:
+        - documentation
+    - title: 🔧 CI/CD & Maintenance
+      labels:
+        - ci-cd
+        - maintenance
+        - github-actions
+    - title: ⬆️ Dependencies
+      labels:
+        - dependencies
+    - title: 🧹 Code Quality & Refactoring
+      labels:
+        - refactoring
+        - technical-debt
+        - developer-experience
+    - title: 🧪 Tests
+      labels:
+        - tests
+    - title: ⚙️ Configuration
+      labels:
+        - configuration
+    - title: 🐍 Python Changes
+      labels:
+        - python
+    - title: 🎨 Frontend Changes
+      labels:
+        - javascript
+        - visualization
+        - css
+        - ui-ux
+        - accessibility
+        - snappy
+    - title: 🐳 Docker & Deployment
+      labels:
+        - docker
+    - title: 🔍 Research Features
+      labels:
+        - research
+        - research-quality
+        - research-speed
+    - title: 📊 Metrics & Analytics
+      labels:
+        - metrics
+        - benchmark-results
+    - title: 🔄 Branch Syncs & Automation
+      labels:
+        - sync
+        - auto-merge
+        - automation
+        - ai_code_review
+    - title: 🚧 Review & Blocked Items
+      labels:
+        - merge-blocked
+        - security-review-needed
+        - conflicts
+    - title: Other Changes
+      labels:
+        - "*"

.github/scripts/check-file-writes.sh

@@ -0,0 +1,234 @@
+#!/bin/bash
+
+# Security check for potential unencrypted file writes to disk
+# This script helps prevent accidentally bypassing encryption at rest
+
+set -e
+
+echo "Checking for potential unencrypted file writes to disk..."
+echo "========================================="
+
+# Patterns that might indicate writing sensitive data to disk
+# Note: Using basic grep patterns without lookaheads
+SUSPICIOUS_PATTERNS=(
+  # Python patterns
+  "\.write\("
+  "\.save\("
+  "\.dump\("
+  "open\(.*['\"]w['\"].*\)"
+  "open\(.*['\"]wb['\"].*\)"
+  "with.*open\(.*['\"]w['\"]"
+  "with.*open\(.*['\"]wb['\"]"
+  "\.to_csv\("
+  "\.to_json\("
+  "\.to_excel\("
+  "\.to_pickle\("
+  "tempfile\.NamedTemporaryFile.*delete=False"
+  "Path.*\.write_text\("
+  "Path.*\.write_bytes\("
+  "shutil\.copy"
+  "shutil\.move"
+  "\.export_to_file\("
+  "\.save_to_file\("
+  "\.write_pdf\("
+  "\.savefig\("
+  # JavaScript patterns
+  "fs\.writeFile"
+  "fs\.writeFileSync"
+  "fs\.createWriteStream"
+  "fs\.appendFile"
+)
+
+# Directories to exclude from checks
+EXCLUDE_DIRS=(
+  "tests"
+  "test"
+  "__pycache__"
+  ".git"
+  "node_modules"
+  ".venv"
+  "venv"
+  "migrations"
+  "static"
+  "vendor"
+  "dist"
+  "build"
+  ".next"
+  "coverage"
+  "examples"
+  "scripts"
+  ".github"
+  "cookiecutter-docker"
+)
+
+# Files to exclude
+EXCLUDE_FILES=(
+  "*_test.py"
+  "test_*.py"
+  "*.test.js"
+  "*.spec.js"
+  "*.test.ts"
+  "*.spec.ts"
+  "setup.py"
+  "webpack.config.js"
+  "**/migrations/*.py"
+  "*.min.js"
+  "*.bundle.js"
+  "*-min.js"
+  "*.min.css"
+)
+
+# Safe keywords that indicate encrypted or safe operations
+# These patterns indicate that file writes have been security-verified
+SAFE_KEYWORDS=(
+  "write_file_verified"
+  "write_json_verified"
+)
+
+# Known safe usage patterns (logs, configs, etc.)
+SAFE_USAGE_PATTERNS=(
+  "security/file_write_verifier.py"
+  "import tempfile"
+  "tempfile\.mkdtemp"
+  "tmp_path"
+  "tmp_file"
+)
+
+# Build exclude arguments for grep
+EXCLUDE_ARGS=""
+for dir in "${EXCLUDE_DIRS[@]}"; do
+  EXCLUDE_ARGS="$EXCLUDE_ARGS --exclude-dir=$dir"
+done
+
+for file in "${EXCLUDE_FILES[@]}"; do
+  EXCLUDE_ARGS="$EXCLUDE_ARGS --exclude=$file"
+done
+
+# Track if we found any issues
+FOUND_ISSUES=0
+ALL_MATCHES=""
+
+echo "Scanning codebase for suspicious patterns..."
+
+# Search only in src/ directory to avoid .venv and other non-source directories
+SEARCH_PATHS="src/"
+
+# Single pass to collect all matches
+for pattern in "${SUSPICIOUS_PATTERNS[@]}"; do
+  # Use grep with binary files excluded and max line length to avoid issues with minified files
+  # shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
+  matches=$(grep -rn -I $EXCLUDE_ARGS -- "$pattern" $SEARCH_PATHS --include="*.py" --include="*.js" --include="*.ts" 2>/dev/null | head -1000 || true)
+  if [ -n "$matches" ]; then
+    ALL_MATCHES="$ALL_MATCHES$matches\n"
+  fi
+done
+
+# Also check for specific problematic patterns in one pass
+# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
+temp_matches=$(grep -rn -I $EXCLUDE_ARGS -E "tmp_path|tempfile|/tmp/" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
+if [ -n "$temp_matches" ]; then
+  ALL_MATCHES="$ALL_MATCHES$temp_matches\n"
+fi
+
+# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
+db_matches=$(grep -rn -I $EXCLUDE_ARGS -E "report_content.*open|report_content.*write|markdown_content.*open|markdown_content.*write" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
+if [ -n "$db_matches" ]; then
+  ALL_MATCHES="$ALL_MATCHES$db_matches\n"
+fi
+
+# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
+export_matches=$(grep -rn -I $EXCLUDE_ARGS -E "export.*Path|export.*path\.open|export.*\.write" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
+if [ -n "$export_matches" ]; then
+  ALL_MATCHES="$ALL_MATCHES$export_matches\n"
+fi
+
+# Now filter all matches at once
+if [ -n "$ALL_MATCHES" ]; then
+  echo "Filtering results for false positives..."
+
+  # Remove duplicates and sort (use tr to handle potential null bytes)
+  ALL_MATCHES=$(echo -e "$ALL_MATCHES" | tr -d '\0' | sort -u)
+
+  filtered_matches=""
+  while IFS= read -r line; do
+    [ -z "$line" ] && continue
+
+    # Check if line contains safe keywords
+    skip_line=0
+    for safe_pattern in "${SAFE_KEYWORDS[@]}"; do
+      if echo "$line" | grep -qE -- "$safe_pattern"; then
+        skip_line=1
+        break
+      fi
+    done
+
+    # Check if line contains safe usage patterns
+    if [ "$skip_line" -eq 0 ]; then
+      for usage_pattern in "${SAFE_USAGE_PATTERNS[@]}"; do
+        if echo "$line" | grep -qE -- "$usage_pattern"; then
+          skip_line=1
+          break
+        fi
+      done
+    fi
+
+    # Additional filters for test/mock files that might not be caught by path exclusion
+    if [ "$skip_line" -eq 0 ]; then
+      if echo "$line" | grep -qE "test|mock|stub" && ! echo "$line" | grep -q "#"; then
+        skip_line=1
+      fi
+    fi
+
+    # Allowlist of files that legitimately write to disk without encryption.
+    # These must NOT touch user data or secrets — only:
+    #   - web/app_factory.py            — Flask/framework config writes
+    #   - document_loaders/bytes_loader.py — in-memory → tmp for parsers
+    #   - journal_quality/downloader.py    — public OpenAlex/DOAJ/predatory/
+    #     JabRef/ROR snapshots downloaded to the user data dir (bibliographic
+    #     metadata only — journal names, ISSNs, h-indices; no PII/secrets)
+    #   - journal_quality/data_sources/*.py — same family, per-source adapters
+    #     that write the intermediate JSON manifests under the user data dir
+    # If you add an entry here, document WHY the file's writes are safe
+    # (public data, not user-specific, not encrypted at rest by design).
+    if [ "$skip_line" -eq 0 ]; then
+      if echo "$line" | grep -qE "web/app_factory\.py|document_loaders/bytes_loader\.py|journal_quality/downloader\.py|journal_quality/data_sources/.+\.py"; then
+        skip_line=1
+      fi
+    fi
+
+    # Filter safe temp files with proper cleanup
+    if [ "$skip_line" -eq 0 ]; then
+      if echo "$line" | grep -q "database/encrypted_db.py"; then
+        skip_line=1
+      fi
+    fi
+
+    if [ "$skip_line" -eq 0 ]; then
+      filtered_matches="$filtered_matches$line\n"
+      FOUND_ISSUES=1
+    fi
+  done <<< "$ALL_MATCHES"
+
+  if [ -n "$filtered_matches" ] && [ "$FOUND_ISSUES" -eq 1 ]; then
+    echo "⚠️  Found potential unencrypted file writes:"
+    echo "========================================="
+    echo -e "$filtered_matches"
+  fi
+fi
+
+echo "========================================="
+
+if [ $FOUND_ISSUES -eq 1 ]; then
+  echo "❌ Security check failed: Found potential unencrypted file writes"
+  echo ""
+  echo "Please review the above findings and ensure:"
+  echo "1. Sensitive data is not written to disk unencrypted"
+  echo "2. Temporary files are properly cleaned up"
+  echo "3. Use in-memory operations where possible"
+  echo "4. If file writes are necessary, ensure they're encrypted or add '# Safe: <reason>' comment"
+  echo ""
+  echo "For exports, use the in-memory pattern like in export_report_to_memory()"
+  exit 1
+else
+  echo "✅ Security check passed: No suspicious unencrypted file writes detected"
+fi

.github/scripts/combine-ai-reviews.sh

@@ -0,0 +1,155 @@
+#!/usr/bin/env bash
+#
+# Combine multiple per-model AI code reviews into a single sticky PR comment.
+#
+# This is the pure-assembly half of the AI Code Reviewer workflow
+# (.github/workflows/ai-code-reviewer.yml), split out so the non-trivial parts —
+# header/footer stripping, "Reviewer N" section assembly, label union, and the
+# pass/fail aggregation — can be unit-tested without any network or GitHub API
+# access. The workflow runs the models (network) and writes each reviewer's raw
+# stdout to resp_<i>.json and its exit code to code_<i>; this script turns those
+# into the comment body, label set, decision, and success count.
+#
+# Usage:
+#   combine-ai-reviews.sh <work_dir> <model_1> [<model_2> ...]
+#
+#   Reads, for i in 0..N-1 (N = number of model args):
+#     <work_dir>/resp_<i>.json  - raw stdout from ai-reviewer.sh for reviewer i
+#     <work_dir>/code_<i>        - that reviewer's integer exit code
+#     <work_dir>/err_<i>.log     - that reviewer's stderr (only shown in DEBUG)
+#   The model names are used only for the per-reviewer log-group labels; the
+#   posted comment stays anonymized ("Reviewer 1", "Reviewer 2", ...).
+#
+#   Environment:
+#     HEAD_SHA    - head commit sha for the "Last reviewed at commit" line (opt)
+#     DEBUG_MODE  - "true" to echo raw responses + stderr to this script's stderr
+#
+#   Writes (into <work_dir>):
+#     comment_body.md    - the combined sticky-comment body
+#     labels.txt         - deduped labels, one per line (may be empty)
+#     decision.txt       - "pass" or "fail"
+#     success_count.txt  - number of reviewers that produced a usable review
+#
+#   Logs ::group:: sections and debug output to stderr (kept off stdout so the
+#   output files are the only contract). Exit status is 0 even when every
+#   reviewer failed — the caller inspects success_count.txt and decides whether
+#   to fail the workflow. A non-zero exit means a usage error only.
+
+set -euo pipefail
+
+WORK_DIR="${1:-}"
+if [ -z "$WORK_DIR" ] || [ ! -d "$WORK_DIR" ]; then
+  echo "usage: $0 <work_dir> <model> [<model> ...]" >&2
+  exit 2
+fi
+shift
+MODELS=("$@")
+if [ "${#MODELS[@]}" -eq 0 ]; then
+  echo "error: at least one model name is required" >&2
+  exit 2
+fi
+
+DEBUG_MODE="${DEBUG_MODE:-false}"
+HEAD_SHA="${HEAD_SHA:-}"
+
+COMBINED_REVIEW=""
+ALL_LABELS=""
+ANY_FAIL="false"
+SUCCESS_COUNT=0
+
+# Assemble the combined comment in model order. Reviewers are anonymized as
+# "Reviewer N" — the model -> number mapping appears only in the (debug) logs
+# below, so readers judge the feedback, not the model.
+for i in "${!MODELS[@]}"; do
+  REVIEWER_NUM=$((i + 1))
+  EXIT_CODE="$(cat "$WORK_DIR/code_$i" 2>/dev/null || echo 1)"
+  # A missing/empty/garbage exit-code file counts as a failed reviewer rather
+  # than tripping the numeric comparison below (which would print "integer
+  # expression expected" and then fall through to wrongly treating the reviewer
+  # as a success).
+  [[ "$EXIT_CODE" =~ ^[0-9]+$ ]] || EXIT_CODE=1
+  AI_RESPONSE="$(cat "$WORK_DIR/resp_$i.json" 2>/dev/null || echo "")"
+  echo "::group::Reviewer #$REVIEWER_NUM (${MODELS[i]})" >&2
+
+  if [ "$DEBUG_MODE" = "true" ]; then
+    {
+      echo "=== RAW AI RESPONSE (reviewer #$REVIEWER_NUM, exit $EXIT_CODE) ==="
+      echo "$AI_RESPONSE"
+      echo "--- stderr ---"
+      cat "$WORK_DIR/err_$i.log" 2>/dev/null || true
+      echo "=== END RAW AI RESPONSE ==="
+    } >&2
+  fi
+
+  # One model failing (hard error, empty, or non-JSON) must not sink the others:
+  # record a short failure note and keep going. The caller fails the workflow
+  # only if *every* reviewer failed (success_count == 0). A reviewer that errors
+  # out never sets a verdict, so it can never flip the aggregate decision to
+  # "fail" — only a model that returns a valid "fail" verdict does. Sensitive raw
+  # output is never written to the comment.
+  # Require a JSON *object*: a bare `jq .` accepts any valid JSON, including a
+  # top-level string/array/number that a refusing or misbehaving model might
+  # emit ("I refuse to review this"). That would pass the gate and then crash
+  # the `.review` access below under `set -e`, sinking *every* reviewer and
+  # writing no output at all. `type == "object"` rejects non-objects so they
+  # degrade to a per-reviewer failure note, preserving the others.
+  if [ "$EXIT_CODE" -ne 0 ] || [ -z "$AI_RESPONSE" ] || ! echo "$AI_RESPONSE" | jq -e 'type == "object"' >/dev/null 2>&1; then
+    echo "⚠️ Reviewer #$REVIEWER_NUM did not return a usable review (exit $EXIT_CODE)" >&2
+    REVIEW_BODY="_This reviewer could not complete its review (see workflow logs for details)._"
+    LABELS=""
+  else
+    REVIEW_RAW=$(echo "$AI_RESPONSE" | jq -r '.review // "No review provided"')
+    # Trim whitespace with tr (the verdict is a single token). The previous
+    # `| xargs` aborted the entire script on an unbalanced quote in the value,
+    # which—inside this per-reviewer loop—would sink every reviewer.
+    DECISION=$(echo "$AI_RESPONSE" | jq -r '.fail_pass_workflow // "uncertain"' | tr -d '[:space:]')
+    LABELS=$(echo "$AI_RESPONSE" | jq -r '.labels_added[]? // empty')
+    SUCCESS_COUNT=$((SUCCESS_COUNT + 1))
+    [ "$DECISION" = "fail" ] && ANY_FAIL="true"
+
+    # Each review opens with a "## AI Code Review" H2 and closes with the
+    # singular Friendly AI Reviewer footer. Under a "### 👤 Reviewer N"
+    # subheading the H2 is redundant and the footer would repeat once per model,
+    # so strip both here; a single (plural) footer is added to the combined
+    # comment below. The footer match keys on the stable "Review by [Friendly AI
+    # Reviewer]" prefix and the \s*\z tail absorbs trailing whitespace; if the
+    # upstream markers ever change and aren't found, the text is left intact
+    # (nothing is dropped).
+    REVIEW_BODY=$(printf '%s' "$REVIEW_RAW" \
+      | perl -0pe 's/\A\s*##\s*AI Code Review\s*\n+//' \
+      | perl -0pe 's/\n*---\s*\n\*Review by \[Friendly AI Reviewer\][^\n]*\n?\s*\z//s')
+  fi
+
+  # Lead each section with a blank line so the "### Reviewer N" heading always
+  # renders (command substitution strips trailing newlines, so the separator
+  # must precede the heading, not follow the previous body).
+  COMBINED_REVIEW=$(printf '%s\n\n### 👤 Reviewer %s\n\n%s' "$COMBINED_REVIEW" "$REVIEWER_NUM" "$REVIEW_BODY")
+  [ -n "$LABELS" ] && ALL_LABELS=$(printf '%s\n%s' "$ALL_LABELS" "$LABELS")
+  echo "::endgroup::" >&2
+done
+
+# Aggregate labels (union across reviewers) and the pass/fail decision (request
+# changes if ANY reviewer did).
+LABELS=$(printf '%s\n' "$ALL_LABELS" | sed '/^[[:space:]]*$/d' | sort -u)
+if [ "$ANY_FAIL" = "true" ]; then
+  DECISION="fail"
+else
+  DECISION="pass"
+fi
+
+REVIEWER_WORD="reviewers"
+[ "${#MODELS[@]}" -eq 1 ] && REVIEWER_WORD="reviewer"
+STICKY_MARKER="<!-- ai-code-review:sticky -->"
+FOOTER="*Reviews by [Friendly AI Reviewer](https://github.com/LearningCircuit/Friendly-AI-Reviewer) - made with ❤️*"
+# COMBINED_REVIEW already starts with "\n\n", so it follows the title directly
+# (no extra newline) to yield exactly one blank line before "### Reviewer 1".
+# The backticks in `%s` are intentional literal Markdown (inline-code the SHA),
+# not a command substitution — single quotes keep them literal on purpose.
+# shellcheck disable=SC2016
+COMMENT_BODY=$(printf '%s\n\n## 🤖 AI Code Review (%s %s)%s\n\n---\n%s\n\n_Last reviewed at commit `%s`_' \
+  "$STICKY_MARKER" "${#MODELS[@]}" "$REVIEWER_WORD" "$COMBINED_REVIEW" "$FOOTER" "${HEAD_SHA:0:7}")
+
+printf '%s' "$COMMENT_BODY" > "$WORK_DIR/comment_body.md"
+printf '%s' "$LABELS" > "$WORK_DIR/labels.txt"
+printf '%s' "$DECISION" > "$WORK_DIR/decision.txt"
+printf '%s' "$SUCCESS_COUNT" > "$WORK_DIR/success_count.txt"

.github/scripts/file-whitelist-check.sh

@@ -0,0 +1,594 @@
+#!/bin/bash
+
+# File Whitelist Security Check Script
+# Enhanced security checks with comprehensive file type detection
+
+set -euo pipefail
+
+# Load allowed file patterns from shared whitelist (single source of truth)
+REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo ".")"
+WHITELIST_FILE="$REPO_ROOT/.file-whitelist.txt"
+
+if [ ! -f "$WHITELIST_FILE" ]; then
+  echo "❌ Missing .file-whitelist.txt — cannot run whitelist check."
+  exit 1
+fi
+
+ALLOWED_PATTERNS=()
+while IFS= read -r line; do
+  [[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
+  ALLOWED_PATTERNS+=("$line")
+done < "$WHITELIST_FILE"
+
+# Load per-check ignore lists (exact paths to skip for specific checks)
+IGNORE_ENV_FILES=()
+IGNORE_ENV_FILE="$REPO_ROOT/.github/security/ignore-env-files.txt"
+if [ -f "$IGNORE_ENV_FILE" ]; then
+  while IFS= read -r line; do
+    [[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
+    IGNORE_ENV_FILES+=("$line")
+  done < "$IGNORE_ENV_FILE"
+fi
+
+IGNORE_SUSPICIOUS_FILETYPES=()
+IGNORE_SUSPICIOUS_FILE="$REPO_ROOT/.github/security/ignore-suspicious-filetypes.txt"
+if [ -f "$IGNORE_SUSPICIOUS_FILE" ]; then
+  while IFS= read -r line; do
+    [[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
+    IGNORE_SUSPICIOUS_FILETYPES+=("$line")
+  done < "$IGNORE_SUSPICIOUS_FILE"
+fi
+
+# Get list of files to check
+if [ "${CHECK_ALL_FILES:-}" = "true" ]; then
+echo "🔍 Checking ALL tracked files (release gate mode)..."
+CHANGED_FILES=$(git ls-files)
+TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
+echo "📋 Found $TOTAL_FILE_COUNT tracked files to check"
+elif [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
+# For PRs: check all files that would be added/modified in the entire PR
+echo "🔍 Checking files in PR from $GITHUB_BASE_REF to HEAD..."
+
+CHANGED_FILES=$(git diff --name-only --diff-filter=AM origin/"$GITHUB_BASE_REF"..HEAD)
+FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
+echo "📋 Found $FILE_COUNT changed files with git diff"
+
+# Also get newly added files across all commits in the PR
+# Use a more robust approach that handles edge cases
+ALL_NEW_FILES=$(git log --name-only --pretty=format: --diff-filter=A origin/"$GITHUB_BASE_REF"..HEAD 2>/dev/null | grep -v '^$' | sort | uniq || echo "")
+NEW_FILE_COUNT=$(echo "$ALL_NEW_FILES" | wc -w)
+echo "📋 Found $NEW_FILE_COUNT newly added files with git log"
+
+# Combine both lists and remove duplicates - handle empty ALL_NEW_FILES
+if [ -n "$ALL_NEW_FILES" ]; then
+CHANGED_FILES=$(echo -e "$CHANGED_FILES\n$ALL_NEW_FILES" | sort | uniq | grep -v '^$')
+fi
+TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
+echo "📋 Total unique files to check: $TOTAL_FILE_COUNT"
+else
+# For direct pushes: check files in the current commit
+echo "🔍 Checking files in latest commit..."
+CHANGED_FILES=$(git diff --name-only --diff-filter=AM HEAD~1..HEAD)
+TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
+echo "📋 Found $TOTAL_FILE_COUNT files in direct push"
+fi
+
+echo "🔍 Running comprehensive security checks..."
+echo ""
+
+FILES_CHECKED=0
+WHITELIST_VIOLATIONS=()
+LARGE_FILES=()
+BINARY_FILES=()
+SUSPICIOUS_FILES=()
+RESEARCH_DATA_VIOLATIONS=()
+FLASK_SECRET_VIOLATIONS=()
+ENV_FILE_VIOLATIONS=()
+HIGH_ENTROPY_VIOLATIONS=()
+HARDCODED_PATH_VIOLATIONS=()
+HARDCODED_IP_VIOLATIONS=()
+SUSPICIOUS_FILETYPE_VIOLATIONS=()
+
+# Use improved file processing that handles spaces and special characters
+while IFS= read -r file; do
+[ -z "$file" ] && continue
+
+# Skip deleted files
+if [ ! -f "$file" ]; then
+continue
+fi
+
+FILES_CHECKED=$((FILES_CHECKED + 1))
+if [ $((FILES_CHECKED % 10)) -eq 0 ]; then
+printf "."
+fi
+
+# 1. Whitelist check
+ALLOWED=false
+for pattern in "${ALLOWED_PATTERNS[@]}"; do
+if echo "$file" | grep -qE "$pattern"; then
+ALLOWED=true
+break
+fi
+done
+
+if [ "$ALLOWED" = "false" ]; then
+WHITELIST_VIOLATIONS+=("$file")
+fi
+
+# 2. Large file check (>1MB)
+if [ -f "$file" ]; then
+FILE_SIZE=$(stat -c%s "$file" 2>/dev/null || echo 0)
+if [ "$FILE_SIZE" -gt 1048576 ]; then
+LARGE_FILES+=("$file ($(echo "$FILE_SIZE" | awk '{printf "%.1fMB", $1/1024/1024}'))")
+fi
+fi
+
+# 3. Binary file check
+if file "$file" | grep -q "binary"; then
+BINARY_FILES+=("$file")
+fi
+
+# 4. Secret pattern check - REMOVED: gitleaks workflow handles this more accurately
+
+# 5. Suspicious filename patterns - whitelist approach
+SAFE_FILENAME_PATTERNS=(
+".*token_counter.*\.py$"
+".*migrate.*token.*\.py$"
+".*enhanced.*token.*\.md$"
+"docs/.*token.*\.md$"
+"tests/.*\.py$"
+"docs/decisions/.*\.md$"
+".*session_passwords\.py$"
+".*change_password\.html$"
+"tests/ui_tests/.*password.*\.js$"
+".*password_validator\.py$"
+".*password_utils\.py$"
+)
+
+# Check if filename looks suspicious
+if echo "$file" | grep -iE "(secret|password|token|\.key$|\.pem$|\.p12$|\.pfx$|\.env$)" >/dev/null; then
+# Check if filename matches whitelist patterns
+FILENAME_WHITELISTED=false
+for pattern in "${SAFE_FILENAME_PATTERNS[@]}"; do
+if echo "$file" | grep -qE "$pattern"; then
+FILENAME_WHITELISTED=true
+break
+fi
+done
+
+if [ "$FILENAME_WHITELISTED" = "false" ]; then
+SUSPICIOUS_FILES+=("$file")
+fi
+fi
+
+# 6. LDR-specific security checks
+# Check for research data leakage
+if [ -f "$file" ] && [ -r "$file" ]; then
+# Check for hardcoded research queries in non-test files
+if ! echo "$file" | grep -qE "(test|mock|example)"; then
+if grep -E "(research_id|session_id|query_id).*=.*[\"'][0-9a-f]{8,}[\"']" "$file" >/dev/null 2>&1; then
+RESEARCH_DATA_VIOLATIONS+=("$file")
+fi
+fi
+
+# Check for Flask secret keys
+if grep -E "SECRET_KEY.*=.*[\"'][^\"']{16,}[\"']" "$file" >/dev/null 2>&1; then
+if ! grep -iE "(os\.environ|getenv|config\[|example|placeholder)" "$file" >/dev/null 2>&1; then
+FLASK_SECRET_VIOLATIONS+=("$file")
+fi
+fi
+
+# Check for environment files
+if echo "$file" | grep -E "\.(env|env\.[a-zA-Z]+)$" >/dev/null; then
+ENV_IGNORED=false
+for epath in "${IGNORE_ENV_FILES[@]+${IGNORE_ENV_FILES[@]}}"; do
+[ "$file" = "$epath" ] && ENV_IGNORED=true && break
+done
+if [ "$ENV_IGNORED" = "false" ]; then
+ENV_FILE_VIOLATIONS+=("$file")
+fi
+fi
+
+# Check for high-entropy strings (potential keys/secrets)
+if [ -f "$file" ] && [ -r "$file" ]; then
+# Skip HTML files and other safe file types for entropy checks
+if ! echo "$file" | grep -qE "\.(html|css|js|json|yml|yaml|md)$"; then
+# Skip news_strategy.py which contains example categories in prompts
+if ! echo "$file" | grep -qE "news_strategy\.py$"; then
+# Look for base64-like strings or hex strings that are suspiciously long
+if grep -E "[a-zA-Z0-9+/]{40,}={0,2}|[a-f0-9]{40,}" "$file" >/dev/null 2>&1; then
+# Exclude common false positives
+if ! grep -iE "(sha256|md5|hash|test|example|fixture|integrity)" "$file" >/dev/null 2>&1; then
+HIGH_ENTROPY_VIOLATIONS+=("$file")
+fi
+fi
+fi
+fi
+fi
+
+# Check for hardcoded paths (Unix/Windows)
+if ! echo "$file" | grep -qE "(test|mock|example|\.md$|docker|Docker|\.yml$|\.yaml$|config/paths\.py$|security/path_validator\.py$)"; then
+# Look for absolute paths and user home directories
+if grep -E "(/home/[a-zA-Z0-9_-]+|/Users/[a-zA-Z0-9_-]+|C:\\\\Users\\\\[a-zA-Z0-9_-]+|/opt/|/var/|/etc/|/usr/local/)" "$file" >/dev/null 2>&1; then
+# Exclude common false positives and Docker volume mounts
+if ! grep -iE "(example|sample|placeholder|TODO|FIXME|/usr/local/bin|/etc/hosts|documentation|/etc/searxng|volumes?:|docker)" "$file" >/dev/null 2>&1; then
+HARDCODED_PATH_VIOLATIONS+=("$file")
+fi
+fi
+fi
+
+# Check for hardcoded IP addresses
+if ! echo "$file" | grep -qE "(test|mock|example|\.md$)"; then
+# Look for IPv4 addresses (excluding common safe ones)
+if grep -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$file" >/dev/null 2>&1; then
+# Exclude localhost, documentation IPs, and common examples
+if ! grep -E "\b(127\.0\.0\.1|0\.0\.0\.0|localhost|192\.168\.|10\.|172\.(1[6-9]|2[0-9]|3[0-1])\.|255\.255\.255\.|192\.0\.2\.|198\.51\.100\.|203\.0\.113\.)" "$file" >/dev/null 2>&1; then
+# Additional check to exclude obvious non-IPs (version numbers, etc)
+if grep -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$file" | grep -vE "(version|v[0-9]+\.|release|tag)" >/dev/null 2>&1; then
+HARDCODED_IP_VIOLATIONS+=("$file")
+fi
+fi
+fi
+fi
+
+# 7. Suspicious file type check - detect potentially dangerous file types
+if [ -f "$file" ]; then
+# Check if file is in the suspicious-filetypes ignore list
+FILETYPE_IGNORED=false
+for fpath in "${IGNORE_SUSPICIOUS_FILETYPES[@]+${IGNORE_SUSPICIOUS_FILETYPES[@]}}"; do
+[ "$file" = "$fpath" ] && FILETYPE_IGNORED=true && break
+done
+
+if [ "$FILETYPE_IGNORED" = "false" ]; then
+# Check for suspicious file extensions
+if echo "$file" | grep -iE "\.(exe|dll|so|dylib|bin|deb|rpm|msi|dmg|pkg|app)$" >/dev/null; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (executable/binary)")
+elif echo "$file" | grep -iE "\.(zip|tar|gz|rar|7z|tar\.gz|tar\.bz2|tgz)$" >/dev/null; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (compressed archive)")
+elif echo "$file" | grep -iE "\.(log|tmp|temp|cache|bak|backup|swp|swo|DS_Store|thumbs\.db|desktop\.ini|~|\.orig|\.rej|\.patch)$" >/dev/null; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (temporary/cache)")
+elif echo "$file" | grep -iE "\.(png|jpg|jpeg|gif|bmp|tiff|svg|ico|webp)$" >/dev/null; then
+# Images are suspicious unless in specific directories
+if ! echo "$file" | grep -qE "(^docs/images/|^src/local_deep_research/web/static/favicon\.png$|^installers/.*\.ico$)"; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (image file)")
+fi
+elif echo "$file" | grep -iE "\.(mp3|mp4|wav|avi|mov|mkv|flv|wmv|webm|m4a|ogg)$" >/dev/null; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (media file)")
+elif echo "$file" | grep -iE "\.(csv|xlsx|xls|doc|docx|pdf|ppt|pptx)$" >/dev/null; then
+# Documents are suspicious unless in docs directory
+if ! echo "$file" | grep -qE "docs/"; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (document file)")
+fi
+elif echo "$file" | grep -iE "\.(db|sqlite|sqlite3)$" >/dev/null; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (database file)")
+elif echo "$file" | grep -iE "node_modules/|__pycache__/|\.pyc$|\.pyo$|\.egg-info/|dist/|build/|\.cache/" >/dev/null; then
+SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (build artifact/cache)")
+fi
+fi
+fi
+fi
+done <<< "$CHANGED_FILES"
+
+echo ""
+echo "✓ Checked $FILES_CHECKED files"
+echo ""
+
+# Report all violations with detailed explanations
+echo "📊 Security scan completed. Analyzing results..."
+echo "📋 Summary of findings:"
+echo "   - File type violations: ${#WHITELIST_VIOLATIONS[@]}"
+echo "   - Large files: ${#LARGE_FILES[@]}"
+echo "   - Binary files: ${#BINARY_FILES[@]}"
+echo "   - Suspicious filenames: ${#SUSPICIOUS_FILES[@]}"
+echo "   - Research data leaks: ${#RESEARCH_DATA_VIOLATIONS[@]}"
+echo "   - Hardcoded Flask secrets: ${#FLASK_SECRET_VIOLATIONS[@]}"
+echo "   - Environment files: ${#ENV_FILE_VIOLATIONS[@]}"
+echo "   - High-entropy strings: ${#HIGH_ENTROPY_VIOLATIONS[@]}"
+echo "   - Hardcoded paths: ${#HARDCODED_PATH_VIOLATIONS[@]}"
+echo "   - Hardcoded IPs: ${#HARDCODED_IP_VIOLATIONS[@]}"
+echo "   - Suspicious file types: ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]}"
+
+TOTAL_VIOLATIONS=0
+
+if [ ${#WHITELIST_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ WHITELIST VIOLATIONS - File types not allowed in repository:"
+echo "   These files don't match any pattern in .file-whitelist.txt."
+echo "   Binary files (images, audio, etc.) bloat the repo and should NOT be committed."
+echo "   Only a small set of explicitly listed binary files is allowed — store others externally."
+echo "   If this is a legitimate text/config file, add it to .file-whitelist.txt (requires maintainer approval)."
+echo ""
+for violation in "${WHITELIST_VIOLATIONS[@]}"; do
+echo "  🚫 $violation"
+
+# Show file type and extension
+FILE_EXT="${violation##*.}"
+if [ -f "$violation" ]; then
+FILE_TYPE=$(file -b "$violation" 2>/dev/null || echo "unknown")
+echo "     → File extension: .$FILE_EXT"
+echo "     → File type: $FILE_TYPE"
+echo "     → First few lines:"
+head -3 "$violation" 2>/dev/null | while read -r line; do
+echo "       $line"
+done
+fi
+
+echo "     → Issue: File extension/type not in .file-whitelist.txt"
+echo "     → Fix: For text/config files, add pattern to .file-whitelist.txt"
+echo "     → Note: Binary files should NOT be added to the repo — store them externally"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#WHITELIST_VIOLATIONS[@]}))
+fi
+
+if [ ${#LARGE_FILES[@]} -gt 0 ]; then
+echo ""
+echo "❌ LARGE FILES (>1MB) - Files too big for repository:"
+echo "   Large files should typically be stored externally or compressed."
+echo ""
+for violation in "${LARGE_FILES[@]}"; do
+echo "  📏 $violation"
+echo "     → Issue: File size exceeds 1MB limit"
+echo "     → Fix: Use Git LFS, external storage, or compress the file"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#LARGE_FILES[@]}))
+fi
+
+if [ ${#BINARY_FILES[@]} -gt 0 ]; then
+echo ""
+echo "⚠️  BINARY FILES DETECTED - Review these carefully:"
+echo "   Binary files may contain sensitive data and can't be easily reviewed."
+echo ""
+for violation in "${BINARY_FILES[@]}"; do
+echo "  🔒 $violation"
+echo "     → Issue: Binary file detected (contents not reviewable)"
+echo "     → Action: Verify this file doesn't contain sensitive data"
+echo ""
+done
+fi
+
+if [ ${#SUSPICIOUS_FILES[@]} -gt 0 ]; then
+echo ""
+echo "❌ SUSPICIOUS FILENAMES - Files with security-sensitive names:"
+echo "   These filenames contain words that often indicate sensitive files."
+echo ""
+for violation in "${SUSPICIOUS_FILES[@]}"; do
+echo "  🚨 $violation"
+
+# Show which keyword triggered the detection
+if echo "$violation" | grep -qi "secret"; then
+echo "     → Triggered by: 'secret' in filename"
+elif echo "$violation" | grep -qi "password"; then
+echo "     → Triggered by: 'password' in filename"
+elif echo "$violation" | grep -qi "token"; then
+echo "     → Triggered by: 'token' in filename"
+elif echo "$violation" | grep -qi "api"; then
+echo "     → Triggered by: 'api' in filename"
+elif echo "$violation" | grep -qi "key"; then
+echo "     → Triggered by: 'key' in filename"
+fi
+
+# Show file content preview if it exists
+if [ -f "$violation" ]; then
+FILE_TYPE=$(file -b "$violation" 2>/dev/null || echo "unknown")
+FILE_SIZE=$(stat -c%s "$violation" 2>/dev/null || echo "unknown")
+echo "     → File info: $FILE_TYPE (${FILE_SIZE} bytes)"
+echo "     → Content preview:"
+head -3 "$violation" 2>/dev/null | while read -r line; do
+echo "       $line"
+done
+fi
+
+echo "     → Issue: Filename contains suspicious keywords (secret/password/token/key)"
+echo "     → Fix: Rename file or add to SAFE_FILENAME_PATTERNS whitelist"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#SUSPICIOUS_FILES[@]}))
+fi
+
+# LDR-specific violation reports
+if [ ${#RESEARCH_DATA_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ RESEARCH DATA LEAKAGE - Hardcoded research session data found:"
+echo "   Research IDs and session data should never be hardcoded in production code."
+echo ""
+for violation in "${RESEARCH_DATA_VIOLATIONS[@]}"; do
+echo "  📊 $violation"
+
+# Show the specific lines with research data
+echo "     → Found hardcoded research data:"
+grep -n -E "(research_id|session_id|query_id).*=.*[\"'][0-9a-f]{8,}[\"']" "$violation" 2>/dev/null | head -3 | while read -r line; do
+echo "       $line"
+done
+
+echo "     → Issue: Hardcoded research/session IDs in non-test file"
+echo "     → Fix: Use environment variables or configuration files"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#RESEARCH_DATA_VIOLATIONS[@]}))
+fi
+
+if [ ${#FLASK_SECRET_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ FLASK SECRET KEY - Hardcoded Flask secret keys found:"
+echo "   Flask secret keys must never be hardcoded for security reasons."
+echo ""
+for violation in "${FLASK_SECRET_VIOLATIONS[@]}"; do
+echo "  🔐 $violation"
+
+# Show the specific lines with secret keys
+echo "     → Found hardcoded Flask secret key:"
+grep -n -E "SECRET_KEY.*=.*[\"'][^\"']{16,}[\"']" "$violation" 2>/dev/null | head -3 | while read -r line; do
+echo "       $line"
+done
+
+echo "     → Issue: Hardcoded Flask SECRET_KEY"
+echo "     → Fix: Use os.environ or load from secure config file"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#FLASK_SECRET_VIOLATIONS[@]}))
+fi
+
+if [ ${#ENV_FILE_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ ENVIRONMENT FILES - .env files detected:"
+echo "   Environment files contain sensitive configuration and should never be committed."
+echo ""
+for violation in "${ENV_FILE_VIOLATIONS[@]}"; do
+echo "  🌍 $violation"
+echo "     → Issue: Environment file in repository"
+echo "     → Fix: Add to .gitignore and use .env.example instead"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#ENV_FILE_VIOLATIONS[@]}))
+fi
+
+if [ ${#HIGH_ENTROPY_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ HIGH ENTROPY STRINGS - Potential secrets or keys detected:"
+echo "   Long random strings may be API keys, tokens, or other secrets."
+echo ""
+for violation in "${HIGH_ENTROPY_VIOLATIONS[@]}"; do
+echo "  🎲 $violation"
+
+# Show sample of high entropy strings
+echo "     → Found high-entropy strings:"
+grep -n -E "[a-zA-Z0-9+/]{40,}={0,2}|[a-f0-9]{40,}" "$violation" 2>/dev/null | head -3 | while read -r line; do
+# Truncate long lines for readability
+echo "       ${line:0:120}..."
+done
+
+echo "     → Issue: High-entropy strings that could be secrets"
+echo "     → Fix: Review and move to environment variables if sensitive"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HIGH_ENTROPY_VIOLATIONS[@]}))
+fi
+
+if [ ${#HARDCODED_PATH_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ HARDCODED PATHS - System-specific paths detected:"
+echo "   Absolute paths can expose system structure and break portability."
+echo ""
+for violation in "${HARDCODED_PATH_VIOLATIONS[@]}"; do
+echo "  📁 $violation"
+
+# Show the specific hardcoded paths
+echo "     → Found hardcoded paths:"
+grep -n -E "(/home/[a-zA-Z0-9_-]+|/Users/[a-zA-Z0-9_-]+|C:\\\\Users\\\\[a-zA-Z0-9_-]+|/opt/|/var/|/etc/|/usr/local/)" "$violation" 2>/dev/null | head -5 | while read -r line; do
+echo "       $line"
+done
+
+echo "     → Issue: Hardcoded absolute paths reduce portability"
+echo "     → Fix: Use relative paths, environment variables, or config files"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HARDCODED_PATH_VIOLATIONS[@]}))
+fi
+
+if [ ${#HARDCODED_IP_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ HARDCODED IP ADDRESSES - External IP addresses detected:"
+echo "   Hardcoded IPs can expose infrastructure and cause connectivity issues."
+echo ""
+for violation in "${HARDCODED_IP_VIOLATIONS[@]}"; do
+echo "  🌐 $violation"
+
+# Show the specific IP addresses
+echo "     → Found hardcoded IP addresses:"
+grep -n -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$violation" 2>/dev/null | grep -v -E "(127\.0\.0\.1|0\.0\.0\.0|localhost|192\.168\.|10\.|172\.(1[6-9]|2[0-9]|3[0-1])\.|255\.255\.255\.|192\.0\.2\.|198\.51\.100\.|203\.0\.113\.)" | head -5 | while read -r line; do
+echo "       $line"
+done
+
+echo "     → Issue: Hardcoded IP addresses (non-private/localhost)"
+echo "     → Fix: Use DNS names, environment variables, or config files"
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HARDCODED_IP_VIOLATIONS[@]}))
+fi
+
+if [ ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]} -gt 0 ]; then
+echo ""
+echo "❌ SUSPICIOUS FILE TYPES - Potentially dangerous file types detected:"
+echo "   These file types are commonly used for malware, data leaks, or bloat the repository."
+echo ""
+for violation in "${SUSPICIOUS_FILETYPE_VIOLATIONS[@]}"; do
+echo "  🚨 $violation"
+
+FILE_PATH="${violation%% (*}"
+FILE_CATEGORY="${violation##*\\(}"
+FILE_CATEGORY="${FILE_CATEGORY%\\)}"
+
+# Provide specific guidance based on file category
+case "$FILE_CATEGORY" in
+"executable/binary")
+echo "     → Issue: Executable/binary files can contain malware"
+echo "     → Fix: Remove executable files, use package managers instead"
+;;
+"compressed archive")
+echo "     → Issue: Compressed archives hide their contents from review"
+echo "     → Fix: Extract contents and commit individual files instead"
+;;
+"temporary/cache")
+echo "     → Issue: Temporary/cache files should not be committed"
+echo "     → Fix: Add to .gitignore and remove from repository"
+;;
+"image file")
+echo "     → Issue: Binary image files bloat the repo and should NOT be committed"
+echo "     → Fix: Store images externally. Only a few explicitly listed images in docs/images/ are allowed"
+;;
+"media file")
+echo "     → Issue: Media files are large and rarely needed in code repos"
+echo "     → Fix: Use external hosting or remove if unnecessary"
+;;
+"document file")
+echo "     → Issue: Office documents should be in docs/ directory if needed"
+echo "     → Fix: Move to docs/ directory or convert to markdown"
+;;
+"database file")
+echo "     → Issue: Database files contain data that shouldn't be in source control"
+echo "     → Fix: Add to .gitignore and use migrations/seeds instead"
+;;
+"build artifact/cache")
+echo "     → Issue: Build artifacts and cache files bloat the repository"
+echo "     → Fix: Add to .gitignore and remove from repository"
+;;
+esac
+
+# Show file info if available
+if [ -f "$FILE_PATH" ]; then
+FILE_SIZE=$(stat -c%s "$FILE_PATH" 2>/dev/null || echo "unknown")
+if [ "$FILE_SIZE" != "unknown" ]; then
+READABLE_SIZE=$(echo "$FILE_SIZE" | awk '{if($1>=1048576) printf "%.1fMB", $1/1048576; else if($1>=1024) printf "%.1fKB", $1/1024; else printf "%dB", $1}')
+echo "     → File size: $READABLE_SIZE"
+fi
+fi
+echo ""
+done
+TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]}))
+fi
+
+# Final result
+if [ $TOTAL_VIOLATIONS -eq 0 ]; then
+echo ""
+echo "✅ All security checks passed!"
+exit 0
+else
+echo ""
+echo "💡 To fix these issues:"
+echo "   - For text/config files: add pattern to .file-whitelist.txt (requires maintainer approval)"
+echo "   - For binary files (images, audio, video, archives): do NOT add to the repo"
+echo "     Binary files permanently bloat git history. Store them externally instead."
+echo "     Only a small set of explicitly listed binary files is permitted."
+echo "   - Use environment variables for secrets"
+echo "   - Never hardcode research data or session IDs"
+echo "   - Use .env.example files instead of .env"
+echo "   - Replace absolute paths with relative paths or configs"
+echo "   - Use DNS names instead of hardcoded IP addresses"
+echo ""
+echo "⚠️  SECURITY REMINDER: This is a public repository!"
+echo "   Never commit sensitive data, API keys, or personal information."
+exit 1
+fi

.github/scripts/npm_audit_allowlist.py

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+"""Filter ``npm audit --json`` output against an allowlist of advisories.
+
+Reads ``npm audit --json`` on stdin. Exits non-zero (1) if any
+moderate-or-higher vulnerability is NOT fully explained by allowlisted
+advisories, otherwise 0.
+
+npm reports a parent package as vulnerable when it (transitively) depends on
+a vulnerable child; the parent's ``via`` then contains a *string* reference to
+the child rather than an advisory object. We resolve those ``via`` chains so a
+package that is vulnerable *only* because of an allowlisted advisory is itself
+treated as handled — otherwise allowlisting one leaf (e.g. js-yaml) would still
+leave its dozen parents (lhci/*, jest/*) failing the gate.
+
+The allowlist is supplied via the ``AUDIT_ALLOWLIST`` env var as a
+space-separated list of GHSA IDs. Keep it tight and justified in the workflow
+that sets it — only advisories with **no available fix** in **dev/test-only**
+tooling belong here.
+"""
+
+import json
+import os
+import sys
+
+ALLOW = set(os.environ.get("AUDIT_ALLOWLIST", "").split())
+SEVERITIES = {"moderate", "high", "critical"}
+
+
+def main() -> int:
+    raw = sys.stdin.read().strip()
+    if not raw:
+        print(
+            "::error::empty npm audit output (audit did not run?)",
+            file=sys.stderr,
+        )
+        return 1
+    try:
+        data = json.loads(raw)
+    except json.JSONDecodeError as exc:
+        print(
+            f"::error::could not parse npm audit JSON: {exc}", file=sys.stderr
+        )
+        return 1
+
+    # A genuine npm audit report always carries these keys (an empty/clean
+    # audit still has "vulnerabilities": {}). Their absence means the audit
+    # did NOT actually run — e.g. a registry/network error returns valid JSON
+    # like {"message": ..., "error": ...} and npm exits non-zero, which the
+    # caller's `|| true` swallows. Fail safe (gate red) rather than pass a
+    # green gate on an audit that never happened.
+    if "vulnerabilities" not in data or "auditReportVersion" not in data:
+        print(
+            "::error::npm audit did not return a valid report (audit failed "
+            f"to run?): {data.get('message', 'unknown error')}",
+            file=sys.stderr,
+        )
+        return 1
+
+    vulns = data.get("vulnerabilities") or {}
+    if not isinstance(vulns, dict):
+        print(
+            "::error::npm audit 'vulnerabilities' is not an object — "
+            "treating as a failed audit",
+            file=sys.stderr,
+        )
+        return 1
+
+    def reachable_ghsas(name, seen):
+        """All advisory GHSAs reachable from a package's via-chain."""
+        if name in seen:
+            return set()
+        seen.add(name)
+        found = set()
+        for via in (vulns.get(name) or {}).get("via", []):
+            if isinstance(via, dict) and "GHSA-" in (via.get("url") or ""):
+                # Take only the GHSA token, not any trailing path/query the
+                # URL might carry (e.g. .../GHSA-xxxx/foo?bar).
+                tail = via["url"].split("GHSA-", 1)[1]
+                found.add("GHSA-" + tail.split("/")[0].split("?")[0])
+            elif isinstance(via, str):
+                found |= reachable_ghsas(via, seen)
+        return found
+
+    unhandled = 0
+    for name, info in vulns.items():
+        if info.get("severity") not in SEVERITIES:
+            continue
+        ghsas = reachable_ghsas(name, set())
+        if ghsas and ghsas <= ALLOW:
+            print(f"  allowlisted: {name} ({info['severity']}) {sorted(ghsas)}")
+        else:
+            unhandled += 1
+            detail = sorted(ghsas - ALLOW) if ghsas else "no-GHSA"
+            print(f"  UNHANDLED:   {name} ({info['severity']}) {detail}")
+
+    suffix = "y" if unhandled == 1 else "ies"
+    print(f"  -> {unhandled} non-allowlisted moderate+ vulnerabilit{suffix}")
+    return 1 if unhandled else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/scripts/validate-docker-compose-images.sh

@@ -0,0 +1,122 @@
+#!/bin/bash
+# Validates that all docker-compose image references use SHA256 digests
+# Prevents supply chain attacks by ensuring immutable image references
+
+set -euo pipefail
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+# Configuration - images that are allowed without SHA digests
+ALLOWED_EXCEPTIONS=(
+    "localdeepresearch/local-deep-research:latest"  # Own image, built by CI
+)
+
+# Check if an image reference is in the exceptions list
+is_exception() {
+    local image="$1"
+    for exception in "${ALLOWED_EXCEPTIONS[@]}"; do
+        if [[ "$image" == "$exception" ]]; then
+            return 0
+        fi
+    done
+    return 1
+}
+
+# Validate a single docker-compose file
+validate_compose_file() {
+    local file="$1"
+    local violations=0
+    local line_num=0
+
+    while IFS= read -r line; do
+        line_num=$((line_num + 1))
+
+        # Check if this line contains an image reference
+        if [[ "$line" =~ ^[[:space:]]*image:[[:space:]]*(.+)$ ]]; then
+            local image="${BASH_REMATCH[1]}"
+            image=$(echo "$image" | tr -d '"' | xargs)  # Remove quotes and whitespace
+
+            # Skip if it's an exception
+            if is_exception "$image"; then
+                echo -e "${YELLOW}  Line $line_num: $image (exception)${NC}"
+                continue
+            fi
+
+            # Check if image has SHA digest
+            if [[ ! "$image" =~ @sha256: ]]; then
+                echo -e "${RED}  ❌ Line $line_num: Missing SHA digest${NC}"
+                echo -e "${RED}     Image: $image${NC}"
+                violations=$((violations + 1))
+            else
+                echo -e "${GREEN}  ✓ Line $line_num: $image${NC}"
+            fi
+        fi
+    done < "$file"
+
+    return $violations
+}
+
+# Main validation logic
+main() {
+    local total_violations=0
+    local files_checked=0
+
+    echo "🔍 Validating docker-compose image pinning..."
+    echo ""
+
+    # Find all docker-compose files
+    while IFS= read -r compose_file; do
+        # Skip cookiecutter templates and examples (documentation only)
+        if [[ "$compose_file" =~ cookiecutter-docker/ ]] || [[ "$compose_file" =~ examples/ ]]; then
+            echo -e "${YELLOW}⏭  Skipping: $compose_file (template/example)${NC}"
+            continue
+        fi
+
+        echo "📄 Checking: $compose_file"
+        if validate_compose_file "$compose_file"; then
+            : # No violations
+        else
+            violations=$?
+            total_violations=$((total_violations + violations))
+        fi
+        files_checked=$((files_checked + 1))
+        echo ""
+    done < <(find . -name "docker-compose*.yml" -o -name "docker-compose*.yaml")
+
+    # Summary
+    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+    echo "📊 Summary:"
+    echo "   Files checked: $files_checked"
+    echo "   Violations: $total_violations"
+    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+
+    if [ $total_violations -gt 0 ]; then
+        echo ""
+        echo -e "${RED}❌ Found $total_violations unpinned images in docker-compose files${NC}"
+        echo ""
+        echo "Images must use SHA256 digests for security and reproducibility."
+        echo ""
+        echo "To fix:"
+        echo "  1. Pull the image: docker pull <image:tag>"
+        echo "  2. Get digest: docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
+        echo "  3. Update file: image: <image:tag>@sha256:..."
+        echo ""
+        echo "Example:"
+        echo "  # Bad"
+        echo "  image: ollama/ollama:latest"
+        echo ""
+        echo "  # Good"
+        echo "  image: ollama/ollama:latest@sha256:8850b8b33936b9fb..."
+        exit 1
+    fi
+
+    echo ""
+    echo -e "${GREEN}✅ All docker-compose images properly pinned${NC}"
+    exit 0
+}
+
+main "$@"

.github/scripts/validate-workflow-images.py

@@ -0,0 +1,164 @@
+#!/usr/bin/env python3
+"""
+Validates that all GitHub Actions workflow service containers and container images
+use SHA256 digests for supply chain security.
+
+Prevents tag tampering attacks by ensuring immutable image references.
+"""
+
+import sys
+from pathlib import Path
+from typing import List, Tuple
+
+try:
+    import yaml
+except ImportError:
+    print("❌ Error: PyYAML is required. Install with: pip install pyyaml")
+    sys.exit(1)
+
+
+# ANSI color codes
+class Colors:
+    RED = "\033[0;31m"
+    GREEN = "\033[0;32m"
+    YELLOW = "\033[1;33m"
+    BLUE = "\033[0;34m"
+    NC = "\033[0m"  # No Color
+
+
+def has_sha_digest(image_ref: str) -> bool:
+    """Check if image reference includes SHA256 digest."""
+    return "@sha256:" in image_ref
+
+
+def validate_workflow(workflow_path: Path) -> List[Tuple[str, str, str]]:
+    """
+    Validate a workflow file for unpinned images.
+
+    Returns:
+        List of (job_name, violation_type, image) tuples
+    """
+    violations = []
+
+    try:
+        with open(workflow_path, encoding="utf-8") as f:
+            data = yaml.safe_load(f)
+
+        if not isinstance(data, dict) or "jobs" not in data:
+            return violations
+
+        # Check each job
+        for job_name, job_def in data["jobs"].items():
+            if not isinstance(job_def, dict):
+                continue
+
+            # Check container: field
+            if "container" in job_def:
+                container = job_def["container"]
+
+                # Container can be a string or dict with 'image' key
+                if isinstance(container, str):
+                    if not has_sha_digest(container):
+                        violations.append((job_name, "container", container))
+                elif isinstance(container, dict) and "image" in container:
+                    image = container["image"]
+                    if not has_sha_digest(image):
+                        violations.append((job_name, "container", image))
+
+            # Check services: field
+            if "services" in job_def and isinstance(job_def["services"], dict):
+                for service_name, service_def in job_def["services"].items():
+                    if isinstance(service_def, dict) and "image" in service_def:
+                        image = service_def["image"]
+                        if not has_sha_digest(image):
+                            violations.append(
+                                (job_name, f"service '{service_name}'", image)
+                            )
+
+    except yaml.YAMLError as e:
+        print(f"{Colors.RED}❌ YAML parse error in {workflow_path}:{Colors.NC}")
+        print(f"   {e}")
+        # Return a violation to fail the check
+        violations.append(("parse_error", "error", str(e)))
+    except Exception as e:
+        print(
+            f"{Colors.RED}❌ Error processing {workflow_path}: {e}{Colors.NC}"
+        )
+        violations.append(("error", "error", str(e)))
+
+    return violations
+
+
+def main():
+    """Main validation logic."""
+    workflows_dir = Path(".github/workflows")
+
+    if not workflows_dir.exists():
+        print(
+            f"{Colors.RED}❌ .github/workflows directory not found{Colors.NC}"
+        )
+        sys.exit(1)
+
+    print("🔍 Validating GitHub Actions workflow images...")
+    print()
+
+    total_violations = 0
+    files_checked = 0
+
+    # Process all workflow files
+    for workflow_file in sorted(workflows_dir.glob("*.yml")) + sorted(
+        workflows_dir.glob("*.yaml")
+    ):
+        violations = validate_workflow(workflow_file)
+
+        if violations:
+            print(f"{Colors.RED}📄 {workflow_file.name}:{Colors.NC}")
+            for job_name, violation_type, image in violations:
+                print(
+                    f"{Colors.RED}   ❌ Job '{job_name}' {violation_type}: {image}{Colors.NC}"
+                )
+            print()
+            total_violations += len(violations)
+        else:
+            print(f"{Colors.GREEN}   ✓ {workflow_file.name}{Colors.NC}")
+
+        files_checked += 1
+
+    # Summary
+    print("━" * 50)
+    print("📊 Summary:")
+    print(f"   Files checked: {files_checked}")
+    print(f"   Violations: {total_violations}")
+    print("━" * 50)
+
+    if total_violations > 0:
+        print()
+        print(
+            f"{Colors.RED}❌ Found {total_violations} unpinned images in workflow files{Colors.NC}"
+        )
+        print()
+        print("Service container images must use SHA256 digests for security.")
+        print()
+        print("To fix:")
+        print("  1. Pull the image: docker pull <image:tag>")
+        print(
+            "  2. Get digest: docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
+        )
+        print("  3. Update workflow:")
+        print()
+        print("Example:")
+        print("  services:")
+        print("    redis:")
+        print(f"{Colors.RED}      image: redis:alpine  # Bad{Colors.NC}")
+        print(
+            f"{Colors.GREEN}      image: redis:alpine@sha256:...  # Good{Colors.NC}"
+        )
+        return 1
+
+    print()
+    print(f"{Colors.GREEN}✅ All workflow images properly pinned{Colors.NC}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

.github/security/ignore-env-files.txt

@@ -0,0 +1,4 @@
+# Exact file paths excluded from the environment-file check.
+# These are NOT actual .env files — they are templates/documentation.
+
+src/local_deep_research/defaults/.env.template

.github/security/ignore-suspicious-filetypes.txt

@@ -0,0 +1,5 @@
+# Exact file paths excluded from the suspicious-file-type check.
+# These are legitimate assets required by the application.
+
+src/local_deep_research/web/static/sounds/error.mp3
+src/local_deep_research/web/static/sounds/success.mp3

.github/sync-config.yml

@@ -0,0 +1,36 @@
+# Configuration for automated main-to-dev sync
+sync:
+  # Enable/disable automatic syncing
+  enabled: false
+
+  # Auto-merge PRs when there are no conflicts
+  auto_merge_clean: true
+
+  # Branches to sync from main to
+  target_branches:
+    - dev
+
+  # Ignore sync if these strings are in the commit message
+  ignore_patterns:
+    - "Merge pull request"
+    - "from LearningCircuit/local-deep-research:dev"
+    - "[skip sync]"
+    - "[no sync]"
+
+  # Labels to add to sync PRs
+  labels:
+    clean: ["sync", "auto-merge"]
+    conflicts: ["sync", "conflicts", "manual-review-required"]
+
+  # Reviewers to assign for conflict resolution
+  conflict_reviewers:
+    - "LearningCircuit"
+    - "djpetti"
+    - "HashedViking"
+  # Additional settings
+  settings:
+    # Delete sync branches after successful merge
+    delete_branch_after_merge: true
+
+    # Maximum number of sync attempts per day
+    max_daily_syncs: 10

.github/workflows/README.md

@@ -0,0 +1,50 @@
+# GitHub Actions Workflows
+
+This directory contains GitHub Actions workflows for automated development tasks.
+
+## Update NPM Dependencies Workflow
+
+**File**: `update-npm-dependencies.yml`
+
+### Purpose
+Automatically updates NPM dependencies across all package.json files in the project and fixes security vulnerabilities.
+
+### Triggers
+- **Scheduled**: Every Thursday at 08:00 UTC (day after PDM updates)
+- **Manual**: Can be triggered manually via GitHub Actions UI
+- **Workflow Call**: Can be called by other workflows
+
+### What it does
+1. **Security Audit**: Runs `npm audit` to identify security vulnerabilities
+2. **Security Fixes**: Automatically fixes moderate+ severity vulnerabilities with `npm audit fix`
+3. **Dependency Updates**: Updates all dependencies to latest compatible versions with `npm update`
+4. **Testing**: Runs relevant tests to ensure updates don't break functionality
+5. **Pull Request**: Creates automated PR with all changes
+
+### Directories Managed
+- `/` - Main web dependencies (Vite, Bootstrap, etc.)
+- `/tests/ui_tests` - UI test dependencies (Puppeteer)
+
+### Branch Strategy
+- Creates branch: `update-npm-dependencies-{run_number}`
+- Targets: `main` branch
+- Labels: `maintenance`
+- Reviewers: `djpetti,HashedViking,LearningCircuit`
+
+### Security Focus
+- Only auto-fixes moderate+ severity vulnerabilities
+- Preserves compatible version updates (no major version bumps)
+- Runs security audit before and after updates
+- Requires tests to pass before creating PR
+
+### Manual Usage
+You can manually trigger this workflow:
+1. Go to Actions tab in GitHub
+2. Select "Update NPM dependencies"
+3. Click "Run workflow"
+4. Optionally specify custom npm arguments
+
+### Troubleshooting
+- If tests fail, the PR won't be created
+- Check the workflow logs for specific error messages
+- Security issues that can't be auto-fixed will need manual intervention

.github/workflows/advanced-search-reminder.yml

@@ -0,0 +1,58 @@
+name: Advanced-search change reminder
+
+on:
+  # pull_request (not pull_request_target) — fork PRs get a read-only
+  # token, so the label call 403s; the script catches that and stays
+  # green (the original comment claimed the 403 was silent, but nothing
+  # caught it, so fork PRs got a red check — e.g. #4872). Intentional
+  # no-op on forks: the advisory label is only meaningful for
+  # internal-branch PRs anyway.
+  pull_request:
+    types: [opened, synchronize, reopened]
+    paths:
+      - 'src/local_deep_research/advanced_search_system/**'
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  label:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      pull-requests: write
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Apply benchmark-needed label
+      uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+      with:
+        script: |
+          // Advisory only — asks reviewers to consider running
+          // src/local_deep_research/benchmarks/compare_configurations()
+          // before merging changes that can affect research quality.
+          // The label's description (set on the label itself in repo
+          // settings) carries the reminder text so reviewers see it
+          // on hover without needing an inline comment.
+          try {
+            await github.rest.issues.addLabels({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+              labels: ['benchmark-needed']
+            });
+          } catch (err) {
+            // Fork PRs get a read-only token — the call 403s ("Resource
+            // not accessible by integration"). Advisory label only, so
+            // log and stay green (same pattern as pr-triage.yml).
+            if (err.status !== 403) throw err;
+            console.log('Label call returned 403 (read-only fork token). Skipping.');
+          }

.github/workflows/ai-code-reviewer.yml

@@ -0,0 +1,291 @@
+name: AI Code Reviewer
+
+# Reviews are opt-in per PR: add the ai_code_review label to request one
+# (the label is auto-removed after the run so re-adding it re-triggers).
+# The automatic opened/synchronize/ready_for_review triggers were removed —
+# reviewing every push burned reviewer-API spend on PRs nobody asked to have
+# reviewed, and a billing outage (HTTP 402) then showed a red check on every
+# PR in the repo.
+#
+# Residual risk accepted: the diff text goes into the LLM prompt, so a
+# hostile diff can try to steer the posted comment or labels (prompt
+# injection). The maintainer-applied label gate plus the advisory-only
+# decision (FAIL_ON_REQUESTED_CHANGES defaults to false) bound that
+# blast radius, and it exists for internal PRs too.
+
+on:
+  # zizmor: ignore[dangerous-triggers] — pull_request_target is required so
+  # the review also works on fork PRs: fork-triggered pull_request runs get
+  # no secrets (OPENROUTER_API_KEY) and a read-only token, so the reviewer
+  # could neither call the model nor post its comment. This is safe under
+  # two invariants — KEEP BOTH when editing this file:
+  #   1. Only the ai_code_review label triggers a run, and only users with
+  #      triage permission can add labels: every run on fork code is an
+  #      explicit maintainer action, never attacker-initiated.
+  #   2. Nothing from the PR head is ever executed. The checkout is the
+  #      trusted base branch (pull_request_target default), so
+  #      .github/scripts/combine-ai-reviews.sh runs from base; the PR head
+  #      is fetched as *data only* to compute the diff.
+  pull_request_target:
+    types: [labeled]
+
+# Per-PR concurrency with cancel-in-progress: re-adding the label while a
+# review is still running cancels the now-stale run. Keyed on the PR number
+# so runs for *different* PRs never cancel each other — that repo-wide
+# cancellation was the problem in the earlier attempt (#3554, reverted
+# #3599), not per-PR cancellation.
+#
+# The labeled trigger fires for *every* label added to the PR, but the job's
+# `if` only proceeds when the label is ai_code_review. Without the run_id
+# suffix below, a skip-only labeled event would still enter this group,
+# cancel a real in-flight review, and then skip itself — leaving the PR with
+# no review at all. Routing those into a unique group keyed on run_id means
+# they cancel nothing; the `if` discards them quietly.
+concurrency:
+  group: >-
+    ai-code-review-${{ github.event.pull_request.number }}${{
+      (github.event.action == 'labeled' && github.event.label.name != 'ai_code_review')
+      && format('-skip-{0}', github.run_id) || ''
+    }}
+  cancel-in-progress: true
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  comprehensive-review:
+    name: AI Code Review
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    environment: ci
+    # Only the ai_code_review label triggers a review (works on drafts too);
+    # every other labeled event is discarded here.
+    if: github.event.label.name == 'ai_code_review'
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      # SECURITY: this checks out the trusted BASE branch (the
+      # pull_request_target default), which is what makes running
+      # .github/scripts/combine-ai-reviews.sh below safe on fork PRs.
+      # Never add `ref: ${{ github.event.pull_request.head.sha }}` here —
+      # that would execute fork-controlled scripts with secrets in env.
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y jq
+
+      - name: Get PR diff
+        env:
+          BASE_REF: ${{ github.event.pull_request.base.ref }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          # The PR head — possibly fork code — is fetched as *data only*
+          # through the base repo's refs/pull/N/head mirror (present for
+          # fork and same-repo PRs alike); it is never checked out and
+          # nothing from it is executed. Env vars prevent template
+          # injection from malicious branch names.
+          git fetch origin "$BASE_REF" "+refs/pull/$PR_NUMBER/head:refs/remotes/origin/pr-head"
+          git diff "origin/$BASE_REF...origin/pr-head" --no-color > diff.txt
+
+      - name: Download AI reviewer script
+        run: |
+          curl -fsSL https://raw.githubusercontent.com/LearningCircuit/Friendly-AI-Reviewer/main/ai-reviewer.sh -o ai-reviewer.sh
+          chmod +x ai-reviewer.sh
+
+      - name: AI Code Review
+        id: ai-review
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+          # Comma-separated list of models. Each model produces one independent
+          # review, presented as "Reviewer 1", "Reviewer 2", ... in a single
+          # combined comment. Falls back to the shared AI_MODEL (also used by
+          # release.yml, which must stay a single model) when unset, so a
+          # one-entry list reproduces the original single-review behavior.
+          AI_REVIEW_MODELS: ${{ vars.AI_REVIEW_MODELS || vars.AI_MODEL || 'moonshotai/kimi-k2-thinking' }}
+          AI_TEMPERATURE: ${{ vars.AI_TEMPERATURE || '0.1' }}
+          AI_MAX_TOKENS: ${{ vars.AI_MAX_TOKENS || '64000' }}
+          MAX_DIFF_SIZE: ${{ vars.MAX_DIFF_SIZE || '800000' }}
+          EXCLUDE_FILE_PATTERNS: ${{ vars.EXCLUDE_FILE_PATTERNS || '*.lock,*.min.js,*.min.css,package-lock.json,yarn.lock' }}
+          STRUCTURED_OUTPUT: ${{ vars.STRUCTURED_OUTPUT || 'true' }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          REPO_FULL_NAME: ${{ github.repository }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          FAIL_ON_REQUESTED_CHANGES: ${{ vars.FAIL_ON_REQUESTED_CHANGES || 'false' }}
+          DEBUG_MODE: ${{ vars.DEBUG_MODE || 'false' }}
+        run: |
+          echo "Running AI code review..."
+
+          # Parse the comma-separated model list. Each entry becomes one
+          # independent reviewer ("Reviewer 1", "Reviewer 2", ...). A single
+          # entry reproduces the original one-review behavior.
+          # `read` returns non-zero at EOF; the here-string always appends a
+          # trailing newline so it returns 0 here, but `|| true` keeps a future
+          # edge case from tripping `set -e`. An empty/whitespace AI_REVIEW_MODELS
+          # yields only empty entries, which the filter below drops, correctly
+          # falling through to the "No models configured" error.
+          IFS=',' read -ra RAW_MODELS <<< "$AI_REVIEW_MODELS" || true
+          MODELS=()
+          for m in "${RAW_MODELS[@]}"; do
+            # Trim surrounding whitespace with parameter expansion. (xargs would
+            # choke on a model id containing a quote and word-collapse internal
+            # spaces — model ids shouldn't have either, but this is robust.)
+            m="${m#"${m%%[![:space:]]*}"}"   # strip leading whitespace
+            m="${m%"${m##*[![:space:]]}"}"   # strip trailing whitespace
+            [ -n "$m" ] && MODELS+=("$m")
+          done
+          if [ "${#MODELS[@]}" -eq 0 ]; then
+            echo "❌ No models configured in AI_REVIEW_MODELS"
+            exit 1
+          fi
+          echo "Configured ${#MODELS[@]} reviewer model(s)"
+
+          # Launch every model concurrently so the job's wall-clock is the
+          # slowest single review, not the sum of all of them. Each reviewer
+          # writes its JSON response and its diagnostics to its own index-keyed
+          # files, so the "Reviewer N" numbering stays stable and the parallel
+          # output never interleaves. AI_MODEL is overridden per call; all other
+          # settings (temperature, token limits, diff size, context fetching)
+          # are shared.
+          pids=()
+          for i in "${!MODELS[@]}"; do
+            AI_MODEL="${MODELS[i]}" bash ai-reviewer.sh < diff.txt \
+              > "resp_$i.json" 2> "err_$i.log" &
+            pids[i]=$!
+          done
+
+          # Record each reviewer's exit code to code_<i> (a failing model must
+          # not abort the others, so disable errexit around wait).
+          for i in "${!MODELS[@]}"; do
+            set +e
+            wait "${pids[i]}"
+            echo $? > "code_$i"
+            set -e
+          done
+
+          # Assemble the combined comment with the extracted, unit-tested helper
+          # (.github/scripts/combine-ai-reviews.sh). It reads resp_<i>.json /
+          # code_<i> from the working dir and writes the comment body, label set,
+          # decision, and success count back out. Keeping the assembly in a
+          # script is what lets tests/ci/test_combine_ai_reviews.py cover the
+          # parsing, footer-stripping, label union, and pass/fail logic without
+          # any network or GitHub API access.
+          bash .github/scripts/combine-ai-reviews.sh "$PWD" "${MODELS[@]}"
+          COMMENT_BODY="$(cat comment_body.md)"
+          LABELS="$(cat labels.txt)"
+          DECISION="$(cat decision.txt)"
+          SUCCESS_COUNT="$(cat success_count.txt)"
+
+          # Post (or update) the reviews as a single sticky PR comment. On each
+          # push we edit the same comment in place instead of stacking a new one
+          # per push. The hidden marker identifies our comment specifically, so
+          # we never touch comments from humans or other bots. (Review comments
+          # made before this change lack the marker and are left as-is.)
+          # A commit-sha line lets reviewers tell which push the (edited-in-place)
+          # comment reflects, since GitHub only shows a vague "edited" marker.
+          # This literal MUST match the marker the helper script prepends to the
+          # comment body.
+          STICKY_MARKER="<!-- ai-code-review:sticky -->"
+          # STICKY_MARKER is interpolated into the jq program below, so it must
+          # stay free of `"` and `\`. gh api's --jq takes no --arg, so we can't
+          # pass it as a jq variable; keeping the marker a literal constant is
+          # what keeps this safe. The filter yields a single id (we maintain
+          # exactly one sticky comment), so no post-filtering is needed.
+          EXISTING_COMMENT_ID=$(gh api "repos/$REPO_FULL_NAME/issues/$PR_NUMBER/comments" \
+            --paginate \
+            --jq "[.[] | select(.body | contains(\"$STICKY_MARKER\"))] | last | .id // empty")
+          if [ -n "$EXISTING_COMMENT_ID" ]; then
+            echo "Updating existing AI review comment ($EXISTING_COMMENT_ID)"
+            printf '%s' "$COMMENT_BODY" | gh api \
+              "repos/$REPO_FULL_NAME/issues/comments/$EXISTING_COMMENT_ID" \
+              --method PATCH -F body=@- >/dev/null
+          else
+            echo "Posting new AI review comment"
+            printf '%s' "$COMMENT_BODY" | gh pr comment "$PR_NUMBER" --repo "$REPO_FULL_NAME" -F -
+          fi
+
+          # Add labels to PR. Iterate line-by-line (labels.txt is one label per
+          # line) rather than word-splitting, so a multi-word label such as
+          # "good first issue" is treated as a single label, not three.
+          echo "Labels to add: $LABELS"
+          while IFS= read -r label; do
+            [ -n "$label" ] || continue
+            echo "Adding label: $label"
+            if gh label create "$label" --color "0366d6" --description "Auto-created by AI reviewer" 2>/dev/null; then
+              echo "Created new label: $label"
+            else
+              echo "Label already exists: $label"
+            fi
+            if gh issue edit "$PR_NUMBER" --add-label "$label" 2>/dev/null; then
+              echo "Successfully added label: $label"
+            else
+              echo "Failed to add label: $label"
+            fi
+          done <<< "$LABELS"
+
+          # Handle workflow decision (after posting review and labels)
+          echo "AI decision: $DECISION"
+
+          # Store the decision as a step output for the downstream
+          # "Fail Workflow if Requested" step. $GITHUB_OUTPUT is the modern
+          # mechanism — values are scoped to steps.<id>.outputs.<name> and
+          # cannot re-enter the runner's process environment, eliminating the
+          # $GITHUB_ENV injection class (zizmor github-env audit #8017).
+          #
+          # Defense-in-depth: an allowlist before the write blocks newline
+          # injection. If a manipulated model response put a newline in
+          # $DECISION, the bare echo above could write an additional output
+          # key. Outputs can't reach the process environment so impact is
+          # minimal, but the case statement closes the door cheaply and
+          # replaces the lost-on-rework intent of #4985.
+          case "$DECISION" in pass|fail) ;; *) DECISION="fail" ;; esac
+          echo "DECISION=$DECISION" >> "$GITHUB_OUTPUT"
+
+          # Remove the ai_code_review label so re-adding it re-triggers a review.
+          echo "Removing ai_code_review label to allow easy re-triggering"
+          if gh issue edit "$PR_NUMBER" --remove-label "ai_code_review" 2>/dev/null; then
+            echo "Successfully removed ai_code_review label"
+          else
+            echo "Failed to remove ai_code_review label (may have been already removed)"
+          fi
+
+          # If every reviewer failed, the posted comment carries only failure
+          # notes — surface that as a red workflow run rather than a silent pass,
+          # matching the original single-review behavior on an unusable response.
+          if [ "$SUCCESS_COUNT" -eq 0 ]; then
+            echo "❌ All ${#MODELS[@]} reviewer(s) failed to produce a usable review."
+            exit 1
+          fi
+
+          echo "✅ AI review completed successfully ($SUCCESS_COUNT/${#MODELS[@]} reviewers)"
+
+      - name: Fail Workflow if Requested
+        if: env.FAIL_ON_REQUESTED_CHANGES == 'true' && steps.ai-review.outputs.DECISION == 'fail'
+        run: |
+          echo "❌ AI requested changes and FAIL_ON_REQUESTED_CHANGES is enabled. Failing workflow."
+          echo "The review has been posted above. Please address the requested changes."
+          exit 1
+
+      - name: Cleanup
+        if: always()
+        run: |
+          rm -f diff.txt
+          # Per-reviewer response/log/exit-code files from the parallel fan-out,
+          # plus the assembly helper's output files.
+          rm -f resp_*.json err_*.log code_* comment_body.md labels.txt decision.txt success_count.txt
+          # Only remove ai_response.txt if it exists (only created in debug mode)
+          if [ -f "ai_response.txt" ]; then
+            rm -f ai_response.txt
+          fi

.github/workflows/backwards-compatibility.yml

@@ -0,0 +1,264 @@
+name: Backwards Compatibility
+
+on:
+  workflow_call:  # Allows this workflow to be called by other workflows
+  pull_request:
+    types: [opened, synchronize, reopened]
+    branches: [main]
+    paths:
+      # Core encryption and database management
+      - 'src/local_deep_research/database/encrypted_db.py'
+      - 'src/local_deep_research/database/sqlcipher_*.py'
+      - 'src/local_deep_research/database/auth_db.py'
+      - 'src/local_deep_research/database/session_*.py'
+      # Database initialization and migrations
+      - 'src/local_deep_research/database/initialize.py'
+      - 'src/local_deep_research/database/library_init.py'
+      - 'src/local_deep_research/database/thread_local_session.py'
+      # Database models (schema changes)
+      - 'src/local_deep_research/database/models/*.py'
+      # Encryption settings configuration
+      - 'src/local_deep_research/settings/env_definitions/db_config.py'
+      # Dependencies
+      - 'pyproject.toml'
+      - 'pdm.lock'
+      # Alembic migration infrastructure
+      - 'src/local_deep_research/database/alembic_runner.py'
+      - 'src/local_deep_research/database/migrations/**'
+      # Test files (ensure tests themselves are valid)
+      - 'tests/performance/database/test_backwards_compatibility.py'
+      - 'tests/database/test_encryption_constants.py'
+      - 'tests/database/test_alembic_migrations.py'
+      - 'tests/database/test_migration_0003_indexes.py'
+      - 'tests/database/test_migration_0004_app_settings.py'
+      - 'tests/performance/database/scripts/create_compat_db.py'
+  push:
+    branches: [main]
+  release:
+    types: [created]
+  schedule:
+    # Weekly on Sundays at 2am UTC - catch dependency drift
+    - cron: '0 2 * * 0'
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)
+permissions: {}
+
+jobs:
+  # Fast test - runs on every trigger including PRs.
+  # Provides quick feedback on encryption constant stability.
+  encryption-constants:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    name: Verify Encryption Constants
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Free up disk space
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
+          sudo docker image prune --all --force
+
+      - name: Install dependencies
+        run: pdm install --dev
+
+      - name: Run encryption constants tests
+        run: |
+          pdm run pytest tests/database/test_encryption_constants.py -v --tb=short
+
+  # Slow test - full PyPI version compatibility.
+  # Skipped on PRs because it installs previous PyPI versions which is slow
+  # and the encryption-constants job already provides fast PR feedback.
+  pypi-compatibility:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    name: PyPI Version Compatibility
+    permissions:
+      contents: read
+    # Run on main push, releases, schedule, manual dispatch, and workflow_call (release gate)
+    if: |
+      github.event_name == 'push' ||
+      github.event_name == 'release' ||
+      github.event_name == 'schedule' ||
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'workflow_call'
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Free up disk space
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
+          sudo docker image prune --all --force
+
+      - name: Install test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install pytest
+
+      - name: Get previous PyPI version
+        id: prev-version
+        run: |
+          # Get list of available versions from PyPI
+          VERSIONS_OUTPUT=$(pip index versions local-deep-research 2>&1) || true
+
+          if echo "$VERSIONS_OUTPUT" | grep -q "Available versions:"; then
+            VERSIONS=$(echo "$VERSIONS_OUTPUT" | grep "Available versions:" | cut -d: -f2 | tr ',' '\n' | tr -d ' ')
+          else
+            echo "::warning::Could not fetch versions from PyPI: ${VERSIONS_OUTPUT}"
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          if [ -z "$VERSIONS" ]; then
+            echo "::warning::No versions found on PyPI"
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          # Get current and previous versions
+          CURRENT=$(echo "$VERSIONS" | head -1)
+          PREVIOUS=$(echo "$VERSIONS" | head -2 | tail -1)
+
+          if [ "$CURRENT" = "$PREVIOUS" ] || [ -z "$PREVIOUS" ]; then
+            echo "No previous version available"
+            echo "skip=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          {
+            echo "current=$CURRENT"
+            echo "previous=$PREVIOUS"
+            echo "skip=false"
+          } >> "$GITHUB_OUTPUT"
+          echo "Previous version: $PREVIOUS, Current: $CURRENT"
+
+      - name: Create database with previous version
+        id: create-db
+        if: steps.prev-version.outputs.skip != 'true'
+        run: |
+          # Create isolated venv for previous version
+          python -m venv prev_venv
+          source prev_venv/bin/activate
+
+          # Install previous version — may fail if the published package
+          # has broken dependency metadata (e.g. requests version conflict).
+          # Upgrade pip for reliable dependency resolution in fresh venvs.
+          python -m pip install --upgrade pip
+          if ! pip install "local-deep-research==${{ steps.prev-version.outputs.previous }}" 2>&1; then
+            echo "::warning::Previous version ${{ steps.prev-version.outputs.previous }} has unresolvable dependencies — skipping compat test"
+            echo "install_failed=true" >> "$GITHUB_OUTPUT"
+            deactivate
+            exit 0
+          fi
+          echo "install_failed=false" >> "$GITHUB_OUTPUT"
+
+          # Create test database
+          mkdir -p test_db
+          python tests/performance/database/scripts/create_compat_db.py test_db compat_user "TestPass123!"
+
+          deactivate
+
+          echo "Database created with version ${{ steps.prev-version.outputs.previous }}"
+          ls -la test_db/
+
+      - name: Install current version
+        if: steps.prev-version.outputs.skip != 'true' && steps.create-db.outputs.install_failed != 'true'
+        run: pdm install --dev
+
+      - name: Test opening database with current version
+        if: steps.prev-version.outputs.skip != 'true' && steps.create-db.outputs.install_failed != 'true'
+        run: |
+          # Run the backwards compatibility test with RUN_SLOW_TESTS enabled
+          RUN_SLOW_TESTS=true pdm run pytest tests/performance/database/test_backwards_compatibility.py -v --tb=short
+
+      - name: Report skipped
+        if: steps.prev-version.outputs.skip == 'true' || steps.create-db.outputs.install_failed == 'true'
+        run: |
+          if [ "${{ steps.create-db.outputs.install_failed }}" = "true" ]; then
+            echo "::notice::Backwards compatibility test skipped - previous version has unresolvable dependency conflicts"
+          else
+            echo "::notice::Backwards compatibility test skipped - no previous PyPI version available"
+          fi
+
+  # Database migration tests - validates Alembic migration infrastructure.
+  # Runs on any trigger — including PRs that touch migration-relevant paths
+  # (gated by the top-level `pull_request: paths:` filter).
+  migration-tests:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    name: Database Migration Tests
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Free up disk space
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
+          sudo docker image prune --all --force
+
+      - name: Install dependencies
+        run: pdm install --dev
+
+      - name: Run Alembic migration tests
+        run: |
+          pdm run pytest tests/database/test_alembic_migrations.py -v --tb=short
+
+      - name: Run migration version tests
+        run: |
+          pdm run pytest tests/database/test_migration_0003_indexes.py tests/database/test_migration_0004_app_settings.py -v --tb=short
+
+      - name: Run database initialization tests
+        run: |
+          pdm run pytest tests/database/test_initialize_functions.py tests/test_database_initialization.py -v --tb=short
+
+      - name: Run encrypted DB integration tests
+        run: |
+          pdm run pytest tests/auth_tests/test_encrypted_db.py -v --tb=short

.github/workflows/bearer.yml

@@ -0,0 +1,117 @@
+name: Bearer Security Scan
+
+on:
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+  schedule:
+    # Run security scan daily at 4 AM UTC
+    - cron: '0 4 * * *'
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  bearer-scan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Run Bearer SAST Scanner
+      uses: bearer/bearer-action@828eeb928ce2f4a7ca5ed57fb8b59508cb8c79bc # v2
+      with:
+        scanner: sast,secrets
+        config-file: bearer.yml
+        format: sarif
+        output: bearer-results.sarif
+        # DO NOT change to exit-code: 1 — findings are enforced via SARIF alerts
+        # in the release gate (check-code-scanning-alerts job in release-gate.yml).
+        # Failing here would break CI without adding security value.
+        exit-code: 0
+
+    # Fail loudly if Bearer produced no SARIF — never fabricate an empty one.
+    # An empty-results SARIF uploaded under the bearer-security category would
+    # make GitHub mark every previously-open Bearer alert as fixed, silently
+    # clearing real findings. Mirror the Grype/Trivy jobs: error + exit 1, skip
+    # upload. bearer-action writes a SARIF even with zero findings, so a missing
+    # file means a real scan failure, not a clean run.
+    - name: Ensure SARIF file exists
+      id: check-sarif
+      if: always()
+      run: |
+        if [ -f bearer-results.sarif ]; then
+          echo "exists=true" >> "$GITHUB_OUTPUT"
+        else
+          echo "::error::Bearer did not produce a SARIF file — scan needs to be rerun"
+          exit 1
+        fi
+
+    - name: Upload Bearer results to GitHub Security tab
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      if: always() && steps.check-sarif.outputs.exists == 'true'
+      with:
+        sarif_file: bearer-results.sarif
+        category: bearer-security
+
+    - name: Upload Bearer results as artifact
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      if: always() && steps.check-sarif.outputs.exists == 'true'
+      with:
+        name: bearer-scan-results
+        path: bearer-results.sarif
+        retention-days: 7
+
+    - name: Display Bearer summary
+      if: always()
+      run: |
+        {
+          echo "## Bearer Security Scan Summary"
+          echo ""
+        } >> "$GITHUB_STEP_SUMMARY"
+
+        if [ -f bearer-results.sarif ]; then
+          # Count findings by level
+          ERRORS=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='error']))" 2>/dev/null || echo "0")
+          WARNINGS=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='warning']))" 2>/dev/null || echo "0")
+          NOTES=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='note']))" 2>/dev/null || echo "0")
+          TOTAL=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); print(len(data.get('runs',[{}])[0].get('results',[])))" 2>/dev/null || echo "0")
+
+          {
+            echo "📊 **Scan Results:**"
+            echo "- **Errors:** $ERRORS"
+            echo "- **Warnings:** $WARNINGS"
+            echo "- **Notes:** $NOTES"
+            echo "- **Total:** $TOTAL"
+            echo ""
+          } >> "$GITHUB_STEP_SUMMARY"
+
+          if [ "$ERRORS" -gt 0 ]; then
+            echo "⚠️ **Action Required:** Error-level issues found" >> "$GITHUB_STEP_SUMMARY"
+          elif [ "$WARNINGS" -gt 0 ]; then
+            echo "⚠️ **Review Recommended:** Warning-level issues found" >> "$GITHUB_STEP_SUMMARY"
+          else
+            echo "✅ **No Error or Warning level issues found**" >> "$GITHUB_STEP_SUMMARY"
+          fi
+
+          {
+            echo ""
+            echo "📋 **Details:**"
+            echo "- Bearer scans for sensitive data flow and secrets exposure"
+            echo "- Results uploaded to GitHub Security tab"
+            echo "- SARIF report available in artifacts"
+          } >> "$GITHUB_STEP_SUMMARY"
+        else
+          echo "❌ Bearer scan failed or no results generated" >> "$GITHUB_STEP_SUMMARY"
+        fi

.github/workflows/check-config-docs.yml

@@ -0,0 +1,32 @@
+name: Check Configuration Docs
+
+# Manual-only check. Config docs are auto-regenerated by the version bump
+# workflow (version_check.yml) on every push to main, so a blocking PR
+# check is unnecessary and creates manual work for contributors.
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  check-config-docs:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Check configuration docs are up to date
+        run: python scripts/generate_config_docs.py --check

.github/workflows/check-env-vars.yml

@@ -0,0 +1,43 @@
+name: Check Environment Variables
+
+on:
+  pull_request:
+  workflow_call:  # Called by ci-gate.yml for release pipeline
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions:
+  contents: read
+
+jobs:
+  check-env-vars:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Install dependencies
+        run: |
+          pip install loguru==0.7.3 sqlalchemy==2.0.36 sqlalchemy-utc==0.14.0 platformdirs==4.3.6 pydantic==2.10.4
+
+      - name: Run environment variable validation
+        run: |
+          python tests/settings/env_vars/test_env_var_usage.py

.github/workflows/check-workflow-status.yml

@@ -0,0 +1,52 @@
+name: Check Workflow Status Dashboard
+
+# Fails when a workflow file is added/renamed without a corresponding row
+# in docs/ci/workflow-status.md. Pure structural check — no GitHub API
+# calls, no live data — so it runs fast and doesn't need any auth.
+#
+# To fix a failure: regenerate the dashboard with
+#   `pdm run python scripts/generate_workflow_status.py`
+# This requires `gh` authenticated against the repo. If you can't run it
+# locally, ping a maintainer to regenerate, or add a temporary placeholder
+# `\`<your-new-workflow>.yml\`` mention in the file's manual-edit region
+# to unblock the PR.
+on:
+  pull_request:
+    paths:
+      - '.github/workflows/**'
+      - 'docs/ci/workflow-status.md'
+      - 'scripts/generate_workflow_status.py'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  check-structure:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Install PyYAML
+        # Pinned to match pdm.lock; the rest of the repo uses exact
+        # `==` pins for ad-hoc workflow installs (see e.g.
+        # validate-image-pinning.yml). Floating `~=` ranges can pick up
+        # yanked / replaced patch versions silently.
+        run: pip install pyyaml==6.0.3
+
+      - name: Verify dashboard structure
+        run: python scripts/generate_workflow_status.py --check-structure

.github/workflows/checkov.yml

@@ -0,0 +1,100 @@
+name: Checkov IaC Security Scan
+
+on:
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  checkov:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      security-events: write
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Install Checkov
+        run: pip install checkov==3.2.499
+
+      # Reverted to CLI approach due to known bug in bridgecrewio/checkov-action
+      # when running multiple consecutive action calls (heredoc delimiter overflow)
+      # See: https://github.com/bridgecrewio/checkov-action/issues/170
+      # See: https://github.com/bridgecrewio/checkov/issues/5866
+      - name: Run Checkov on Dockerfile
+        run: |
+          checkov -f Dockerfile \
+            --framework dockerfile \
+            --skip-check CKV_DOCKER_3 \
+            -o cli -o sarif \
+            --output-file-path console,checkov-docker.sarif
+
+      - name: Run Checkov on docker-compose files
+        run: |
+          checkov \
+            -f docker-compose.yml \
+            -f docker-compose.gpu.override.yml \
+            -f docker-compose.tts.yml \
+            -f docker-compose.unraid.yml \
+            --framework yaml \
+            -o cli -o sarif \
+            --output-file-path console,checkov-compose.sarif
+
+      - name: Run Checkov on GitHub Actions workflows
+        run: |
+          checkov -d .github/workflows \
+            --framework github_actions \
+            --skip-check CKV_GHA_7 \
+            -o cli -o sarif \
+            --output-file-path console,checkov-workflows.sarif
+
+      - name: Check SARIF files exist
+        id: check-sarif
+        if: always()
+        run: |
+          [ -f checkov-docker.sarif ] && echo "docker=true" >> "$GITHUB_OUTPUT"
+          [ -f checkov-compose.sarif ] && echo "compose=true" >> "$GITHUB_OUTPUT"
+          [ -f checkov-workflows.sarif ] && echo "workflows=true" >> "$GITHUB_OUTPUT"
+
+      - name: Upload Dockerfile SARIF results
+        if: always() && steps.check-sarif.outputs.docker == 'true'
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        with:
+          sarif_file: checkov-docker.sarif
+          category: checkov-dockerfile
+
+      - name: Upload docker-compose SARIF results
+        if: always() && steps.check-sarif.outputs.compose == 'true'
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        with:
+          sarif_file: checkov-compose.sarif
+          category: checkov-compose
+
+      - name: Upload GitHub Actions SARIF results
+        if: always() && steps.check-sarif.outputs.workflows == 'true'
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        with:
+          sarif_file: checkov-workflows.sarif
+          category: checkov-workflows
+
+      - name: Summary
+        if: always()
+        run: |
+          echo "=== Checkov IaC Security Scan Summary ==="
+          echo "SARIF results uploaded to GitHub Security tab"
+          echo "Scan completed (strict mode - failures block CI)"

.github/workflows/ci-gate.yml

@@ -0,0 +1,259 @@
+name: CI Gate
+
+# CI quality gate for the release pipeline.
+#
+# Ensures all PR-quality checks (linting, type checking, tests, validation)
+# run and pass before any release proceeds. Complements the security-focused
+# release-gate.yml with code quality and correctness checks.
+#
+# Architecture:
+#   release.yml → ci-gate.yml → {pre-commit, mypy, docker-tests, ...}
+#   Nesting depth: release.yml(1) → ci-gate.yml(2) → workflow(3) — safe limit.
+
+on:
+  workflow_call:  # Called by release.yml
+    secrets:
+      OPENROUTER_API_KEY:
+        required: false
+  workflow_dispatch:  # Manual trigger
+
+permissions: {}  # Minimal top-level for OSSF Scorecard
+
+jobs:
+  # ============================================
+  # Code Quality
+  # ============================================
+  pre-commit:
+    uses: ./.github/workflows/pre-commit.yml
+    permissions:
+      contents: read
+
+  mypy-type-check:
+    uses: ./.github/workflows/mypy-type-check.yml
+    permissions:
+      contents: read
+
+  # ============================================
+  # Test Suites
+  # ============================================
+  docker-tests:
+    uses: ./.github/workflows/docker-tests.yml
+    with:
+      strict-mode: true
+    permissions:
+      contents: write       # Needed by pytest-tests for gh-pages deployment
+      pull-requests: write  # Needed by pytest-tests for PR comments (no-ops in release context)
+    secrets:
+      OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+
+  # ============================================
+  # Validation
+  # ============================================
+  validate-image-pinning:
+    uses: ./.github/workflows/validate-image-pinning.yml
+    permissions:
+      contents: read
+
+  file-whitelist-check:
+    uses: ./.github/workflows/file-whitelist-check.yml
+    with:
+      check-all-files: true
+    permissions:
+      contents: read
+
+  check-env-vars:
+    uses: ./.github/workflows/check-env-vars.yml
+    permissions:
+      contents: read
+
+  security-file-write-check:
+    uses: ./.github/workflows/security-file-write-check.yml
+    permissions:
+      contents: read
+
+  # ============================================
+  # Summary job that reports overall status
+  # ============================================
+  ci-gate-summary:
+    name: CI Gate Summary
+    runs-on: ubuntu-latest
+    needs:
+      # Code Quality
+      - pre-commit
+      - mypy-type-check
+      # Test Suites
+      - docker-tests
+      # Validation
+      - validate-image-pinning
+      - file-whitelist-check
+      - check-env-vars
+      - security-file-write-check
+    if: always()
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Check CI scan results
+        env:
+          PRE_COMMIT_RESULT: ${{ needs.pre-commit.result }}
+          MYPY_RESULT: ${{ needs.mypy-type-check.result }}
+          DOCKER_TESTS_RESULT: ${{ needs.docker-tests.result }}
+          IMAGE_PINNING_RESULT: ${{ needs.validate-image-pinning.result }}
+          FILE_WHITELIST_RESULT: ${{ needs.file-whitelist-check.result }}
+          ENV_VARS_RESULT: ${{ needs.check-env-vars.result }}
+          FILE_WRITE_RESULT: ${{ needs.security-file-write-check.result }}
+        run: |
+          # Redirect all output to GITHUB_STEP_SUMMARY (fixes SC2129)
+          exec >> "$GITHUB_STEP_SUMMARY"
+
+          # Count results first
+          FAILED=""
+          PASS_COUNT=0
+          FAIL_COUNT=0
+
+          check_result() {
+            local result="$1"
+            if [ "$result" = "success" ]; then
+              PASS_COUNT=$((PASS_COUNT + 1))
+              return 0
+            else
+              FAIL_COUNT=$((FAIL_COUNT + 1))
+              FAILED="true"
+              return 1
+            fi
+          }
+
+          # Check all results silently first
+          check_result "$PRE_COMMIT_RESULT" || true
+          check_result "$MYPY_RESULT" || true
+          check_result "$DOCKER_TESTS_RESULT" || true
+          check_result "$IMAGE_PINNING_RESULT" || true
+          check_result "$FILE_WHITELIST_RESULT" || true
+          check_result "$ENV_VARS_RESULT" || true
+          check_result "$FILE_WRITE_RESULT" || true
+
+          TOTAL=$((PASS_COUNT + FAIL_COUNT))
+
+          # ============================================
+          # BIG STATUS BANNER
+          # ============================================
+          if [ -z "$FAILED" ]; then
+            echo "# :white_check_mark: CI GATE: PASSED"
+            echo ""
+            echo "> **All $TOTAL CI checks passed successfully.**"
+            echo "> This release is approved from a code quality perspective."
+          else
+            echo "# :x: CI GATE: FAILED"
+            echo ""
+            echo "> **$FAIL_COUNT of $TOTAL checks failed.** Release is blocked."
+            echo "> Review the failures below and fix before releasing."
+          fi
+
+          echo ""
+          echo "---"
+          echo ""
+          echo "## Detailed Results"
+          echo ""
+
+          # Reset for detailed output
+          FAILED=""
+
+          # ============================================
+          # Code Quality
+          # ============================================
+          echo "### Code Quality"
+
+          if [ "$PRE_COMMIT_RESULT" = "success" ]; then
+            echo ":white_check_mark: **Pre-commit (linting, formatting)**: Passed"
+          else
+            echo ":x: **Pre-commit (linting, formatting)**: $PRE_COMMIT_RESULT"
+            FAILED="true"
+          fi
+
+          if [ "$MYPY_RESULT" = "success" ]; then
+            echo ":white_check_mark: **Mypy Type Check**: Passed"
+          else
+            echo ":x: **Mypy Type Check**: $MYPY_RESULT"
+            FAILED="true"
+          fi
+
+          # ============================================
+          # Test Suites
+          # ============================================
+          echo ""
+          echo "### Test Suites"
+
+          if [ "$DOCKER_TESTS_RESULT" = "success" ]; then
+            echo ":white_check_mark: **Docker Tests (pytest + UI + LLM + infra + smoke)**: Passed"
+          else
+            echo ":x: **Docker Tests (pytest + UI + LLM + infra + smoke)**: $DOCKER_TESTS_RESULT"
+            FAILED="true"
+          fi
+
+          # ============================================
+          # Validation
+          # ============================================
+          echo ""
+          echo "### Validation"
+
+          if [ "$IMAGE_PINNING_RESULT" = "success" ]; then
+            echo ":white_check_mark: **Docker Image Pinning**: Passed"
+          else
+            echo ":x: **Docker Image Pinning**: $IMAGE_PINNING_RESULT"
+            FAILED="true"
+          fi
+
+          if [ "$FILE_WHITELIST_RESULT" = "success" ]; then
+            echo ":white_check_mark: **File Whitelist Security**: Passed"
+          else
+            echo ":x: **File Whitelist Security**: $FILE_WHITELIST_RESULT"
+            FAILED="true"
+          fi
+
+          if [ "$ENV_VARS_RESULT" = "success" ]; then
+            echo ":white_check_mark: **Environment Variables**: Passed"
+          else
+            echo ":x: **Environment Variables**: $ENV_VARS_RESULT"
+            FAILED="true"
+          fi
+
+          if [ "$FILE_WRITE_RESULT" = "success" ]; then
+            echo ":white_check_mark: **Security File Writes**: Passed"
+          else
+            echo ":x: **Security File Writes**: $FILE_WRITE_RESULT"
+            FAILED="true"
+          fi
+
+          # ============================================
+          # Final result with prominent summary
+          # ============================================
+          echo ""
+          echo "---"
+          echo ""
+
+          if [ -n "$FAILED" ]; then
+            echo "## :rotating_light: Action Required"
+            echo ""
+            echo "| Status | Result |"
+            echo "|--------|--------|"
+            echo "| **Gate** | :x: **BLOCKED** |"
+            echo "| **Passed** | $PASS_COUNT |"
+            echo "| **Failed** | $FAIL_COUNT |"
+            echo ""
+            echo "_Fix the failing checks above before releasing._"
+            exit 1
+          else
+            echo "## :tada: Ready for Release"
+            echo ""
+            echo "| Status | Result |"
+            echo "|--------|--------|"
+            echo "| **Gate** | :white_check_mark: **APPROVED** |"
+            echo "| **Passed** | $PASS_COUNT / $TOTAL |"
+            echo ""
+            echo "_All CI checks passed. Security scans run as separate gate in release pipeline._"
+          fi

.github/workflows/claude-code-review.yml

@@ -0,0 +1,57 @@
+name: Claude Code Review
+
+on:
+  pull_request:
+    types: [labeled]
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  claude-review:
+    # Only run when 'claude-review' label is added (opt-in)
+    if: github.event.label.name == 'claude-review'
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    environment: ci
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
+      id-token: write  # Required for OIDC authentication with claude-code-action
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 1
+
+      - name: Run Claude Code Review
+        id: claude-review
+        uses: anthropics/claude-code-action@558b1d6cab4085c7753fe402c10bef0fbb92ac7a # v1.0.165
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+
+          # CLI arguments for Claude (v1.0+ format)
+          claude_args: |
+            --model claude-opus-4-5-20251101
+
+          # Prompt for automated review (no @claude mention needed)
+          prompt: |
+            Please review this pull request and provide feedback on:
+            - Code quality and best practices
+            - Potential bugs or issues
+            - Performance considerations
+            - Security concerns
+            - Test coverage
+            Be constructive and helpful in your feedback.

.github/workflows/codeql.yml

@@ -0,0 +1,114 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL Advanced"
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  schedule:
+    - cron: '45 5 * * 0'
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  analyze:
+    name: Analyze (${{ matrix.language }})
+    # Runner size impacts CodeQL analysis time. To learn more, please see:
+    #   - https://gh.io/recommended-hardware-resources-for-running-codeql
+    #   - https://gh.io/supported-runners-and-hardware-resources
+    #   - https://gh.io/using-larger-runners (GitHub.com only)
+    # Consider using larger runners or machines with greater resources for possible analysis time improvements.
+    runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      packages: read
+      actions: read
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+        - language: javascript-typescript
+          build-mode: none
+        - language: python
+          build-mode: none
+        # CodeQL supports the following values keywords for 'language': 'actions', 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
+        # Use `c-cpp` to analyze code written in C, C++ or both
+        # Use 'java-kotlin' to analyze code written in Java, Kotlin or both
+        # Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
+        # To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
+        # see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
+        # If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
+        # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout repository
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        persist-credentials: false
+
+    # Add any setup steps before running the `github/codeql-action/init` action.
+    # This includes steps like installing compilers or runtimes (`actions/setup-node`
+    # or others). This is typically only required for manual builds.
+    # - name: Setup runtime (example)
+    #   uses: actions/setup-example@v1
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
+      with:
+        languages: ${{ matrix.language }}
+        build-mode: ${{ matrix.build-mode }}
+        config-file: ./.github/codeql/codeql-config.yml
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+
+        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
+        # queries: security-extended,security-and-quality
+
+    # If the analyze step fails for one of the languages you are analyzing with
+    # "We were unable to automatically build your code", modify the matrix above
+    # to set the build mode to "manual" for that language. Then modify this step
+    # to build your code.
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
+    - if: matrix.build-mode == 'manual'
+      shell: bash
+      run: |
+        echo 'If you are using a "manual" build mode for one or more of the' \
+          'languages you are analyzing, replace this with the commands to build' \
+          'your code, for example:'
+        echo '  make bootstrap'
+        echo '  make release'
+        exit 1
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
+      with:
+        category: "/language:${{matrix.language}}"

.github/workflows/compose-integration-test.yml

@@ -0,0 +1,195 @@
+name: Compose Integration Test
+
+# Brings up the bundled docker-compose.yml end-to-end in CI and verifies the
+# whole stack (searxng, ollama, local-deep-research) reaches a healthy state.
+# This is the test that would have caught #3874 (broken cap_drop on searxng)
+# before users hit it. See #3886 for the rationale.
+#
+# Cost: ~3-6 min per run depending on cache state. Runs only via the release
+# pipeline (`release.yml` includes it as `compose-integration-gate`) and on
+# manual dispatch. NOT inside `release-gate.yml`, because release-gate runs
+# daily and the failure modes here (compose/image changes) are tied to
+# actual release events, not time. NOT on pull_request — too expensive for
+# per-PR feedback latency.
+#
+# Do not move this into release-gate's daily cron. Ongoing compose drift
+# between releases is already covered by `compose-published-smoke.yml`,
+# which runs weekly against main's compose.yml + the *published* Docker Hub
+# image — the more meaningful drift to catch, since users pull the
+# published image, not main's build override. PR #3962 tried adding this
+# to the daily cron and was closed for that reason.
+#
+# Tests the docker-compose.yml as users actually run it — no test-only
+# overrides. The model pre-pull was removed from the compose itself in this
+# same PR so the stack starts in seconds rather than waiting on a multi-GB
+# download that the test doesn't need.
+
+on:
+  workflow_call:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  compose-up:
+    name: docker compose up + healthcheck
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      # Build the LDR image from the working tree and tag it as the name the
+      # bundled compose references. Compose then uses the local image instead
+      # of pulling the published one — so we test the current code path, not
+      # whatever's on Docker Hub.
+      - name: Build LDR image with the published tag
+        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        with:
+          context: .
+          # Same target as docker-tests.yml's `ldr-prod` build, so its cache
+          # layers (scope=ldr-prod, populated by ci-gate during the same
+          # release pipeline) are reusable here. Explicit target guards
+          # against future Dockerfile reorderings making the default stage
+          # something other than `ldr`.
+          target: ldr
+          load: true
+          tags: localdeepresearch/local-deep-research:latest
+          # Read from both our own scope and the shared `ldr-prod` scope —
+          # whichever has the layers wins. Falls back to a fresh build if
+          # neither does (e.g. on a brand-new branch). Only write to our
+          # own scope so we don't poison the cross-workflow cache.
+          cache-from: |
+            type=gha,scope=compose-integration
+            type=gha,scope=ldr-prod
+          cache-to: type=gha,mode=max,scope=compose-integration
+
+      - name: Bring up the stack
+        # `docker compose up -d` pulls any image it doesn't have locally
+        # (default pull_policy: missing), so ollama and searxng are pulled
+        # inline. The LDR image is the locally-built one from the previous
+        # step — compose sees it's already present and uses it as-is.
+        # `--no-build` defends against a future docker-compose.yml change
+        # adding a `build:` directive — we want the image we tagged above,
+        # not a fresh build that bypasses the cache strategy.
+        run: docker compose up -d --no-build
+
+      - name: Wait for the stack to be healthy and serving
+        # Budget: 6 min covers cold daemon startup + LDR migrations + flask
+        # boot + slow CI runners. Without the model pull this completes in
+        # ~1-2 min on a warm runner.
+        #
+        # Container resolution: always via `docker compose ps -q <service>`
+        # so we don't couple to compose's `container_name:` values. If those
+        # drift, this still works.
+        run: |
+          set -euo pipefail
+          deadline=$(( $(date +%s) + 360 ))
+
+          cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
+          status() {
+            local cid=$1
+            [ -n "$cid" ] || { echo missing; return; }
+            docker inspect -f "{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}" "$cid" 2>/dev/null || echo missing
+          }
+
+          while :; do
+            now=$(date +%s)
+            if [ "$now" -ge "$deadline" ]; then
+              echo "::error::Timed out after 6 min waiting for stack to be healthy"
+              docker compose ps
+              exit 1
+            fi
+
+            ollama_id=$(cid_for ollama)
+            searxng_id=$(cid_for searxng)
+            ldr_id=$(cid_for local-deep-research)
+
+            ollama_h=$(status "$ollama_id")
+            searxng_h=$(status "$searxng_id")
+            ldr_h=$(status "$ldr_id")
+
+            echo "[$(date -u +%H:%M:%S)] ollama=$ollama_h searxng=$searxng_h ldr=$ldr_h"
+
+            # All three services have healthchecks: ollama and searxng via
+            # docker-compose.yml, LDR via the Dockerfile (HEALTHCHECK at
+            # Dockerfile:306, probing /api/v1/health). status() returns the
+            # health status when one is defined, so require "healthy" for
+            # all three — strictly stronger signal than "running".
+            if [ "$ollama_h" = "healthy" ] && [ "$searxng_h" = "healthy" ] && [ "$ldr_h" = "healthy" ]; then
+              echo "All services healthy."
+              break
+            fi
+
+            # Fail fast on any container exiting non-zero rather than burning
+            # the whole 6 min budget.
+            for svc in ollama searxng local-deep-research; do
+              cid=$(cid_for "$svc")
+              [ -n "$cid" ] || continue
+              s=$(docker inspect -f '{{.State.Status}}' "$cid" 2>/dev/null || true)
+              if [ "$s" = "exited" ] || [ "$s" = "dead" ]; then
+                echo "::error::Container for service $svc has exited"
+                exit 1
+              fi
+            done
+
+            sleep 10
+          done
+
+      - name: Probe LDR HTTP endpoint
+        # Avoids `curl ... | grep` so we don't need pipefail to surface curl
+        # failures — the HTTP code is captured directly via -w and checked
+        # with a case statement. `|| echo "000"` is the sentinel for true
+        # network failures (connection refused, DNS, etc.) so we can log
+        # on retry.
+        #
+        # Deliberately no `-f`: with `-f`, curl exits non-zero on HTTP
+        # 4xx/5xx AND suppresses -w output — which would collapse "404",
+        # "503", and network-error all into "000" and erase the most
+        # interesting failure signal (LDR up but serving an error page).
+        # Without -f, every HTTP response gives us its real code; only
+        # true network failures fall through to "000".
+        run: |
+          set -euo pipefail
+          for i in $(seq 1 30); do
+            code=$(curl -sS -o /dev/null -w "%{http_code}" http://localhost:5000/ 2>/dev/null || echo "000")
+            case "$code" in
+              200|301|302|303|307|308)
+                echo "LDR is serving on :5000 (HTTP $code)"
+                exit 0
+                ;;
+            esac
+            echo "Waiting for LDR HTTP (attempt $i/30, last code: $code)..."
+            sleep 5
+          done
+          echo "::error::LDR HTTP probe failed after ~150s"
+          exit 1
+
+      - name: Dump compose state and logs
+        if: always()
+        run: |
+          echo "::group::docker compose ps"
+          docker compose ps || true
+          echo "::endgroup::"
+
+          for svc in ollama searxng local-deep-research; do
+            echo "::group::$svc logs"
+            docker compose logs --no-color --tail=500 "$svc" || true
+            echo "::endgroup::"
+          done
+
+      - name: Tear down
+        if: always()
+        run: docker compose down -v --remove-orphans

.github/workflows/compose-published-smoke.yml

@@ -0,0 +1,257 @@
+name: Compose Published-Image Smoke
+
+# Tests "main's docker-compose.yml + the currently-published Docker Hub image"
+# end-to-end. Catches drift between compose changes that have landed on main
+# and the image artefact that users actually pull when they follow the README
+# quickstart:
+#
+#   curl -O .../docker-compose.yml && docker compose up -d
+#
+# This complements the release-pipeline test in compose-integration-test.yml,
+# which builds the image from the working tree (validating "this code's
+# compose with this code's image") and runs only at release time + manual
+# dispatch. That test cannot catch the drift case — the published image
+# always lags main between releases.
+#
+# Triggers: weekly schedule + manual dispatch only. No pull_request trigger
+# because PRs don't change the published image — by definition this test
+# can only fail on drift that's already landed on main.
+#
+# On failure, opens a tracking issue (or comments on the existing one) so a
+# scheduled-job failure isn't lost in the noise.
+
+on:
+  workflow_dispatch:
+  schedule:
+    # 05:00 UTC every Monday — offset from release-gate's 02:00 daily cron
+    # to avoid runner contention. Weekly cadence is enough since the failure
+    # modes are slow-moving (compose-vs-published-image drift) and the test
+    # itself burns CI minutes.
+    - cron: '0 5 * * 1'
+
+permissions:
+  contents: read
+
+jobs:
+  compose-up:
+    name: docker compose up + healthcheck (published image)
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    permissions:
+      contents: read
+      issues: write  # for the failure-issue step
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout (for docker-compose.yml on default branch)
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      # Pull all three services from their registries, including the LDR
+      # image. This is the whole point of this workflow — exercise the
+      # exact artefact users get from `docker compose up -d`.
+      - name: Pull all images
+        run: docker compose pull
+
+      - name: Bring up the stack
+        # `--no-build` defends against a future docker-compose.yml change
+        # adding a `build:` directive — this workflow specifically tests the
+        # *published* image artifact, not a fresh build from source.
+        run: docker compose up -d --no-build
+
+      - name: Wait for the stack to be healthy and serving
+        # Same wait logic as compose-integration-test.yml. 6 min budget
+        # covers cold daemon startup + LDR migrations + flask boot.
+        # Container resolution: always via `docker compose ps -q <service>`
+        # so we don't couple to compose's `container_name:` values.
+        run: |
+          set -euo pipefail
+          deadline=$(( $(date +%s) + 360 ))
+
+          cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
+          status() {
+            local cid=$1
+            [ -n "$cid" ] || { echo missing; return; }
+            docker inspect -f "{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}" "$cid" 2>/dev/null || echo missing
+          }
+
+          while :; do
+            now=$(date +%s)
+            if [ "$now" -ge "$deadline" ]; then
+              echo "::error::Timed out after 6 min waiting for stack to be healthy"
+              docker compose ps
+              exit 1
+            fi
+
+            ollama_id=$(cid_for ollama)
+            searxng_id=$(cid_for searxng)
+            ldr_id=$(cid_for local-deep-research)
+
+            ollama_h=$(status "$ollama_id")
+            searxng_h=$(status "$searxng_id")
+            ldr_h=$(status "$ldr_id")
+
+            echo "[$(date -u +%H:%M:%S)] ollama=$ollama_h searxng=$searxng_h ldr=$ldr_h"
+
+            # All three services have healthchecks: ollama and searxng via
+            # docker-compose.yml, LDR via the Dockerfile (HEALTHCHECK at
+            # Dockerfile:306, probing /api/v1/health). status() returns the
+            # health status when one is defined, so require "healthy" for
+            # all three.
+            if [ "$ollama_h" = "healthy" ] && [ "$searxng_h" = "healthy" ] && [ "$ldr_h" = "healthy" ]; then
+              echo "All services healthy."
+              break
+            fi
+
+            for svc in ollama searxng local-deep-research; do
+              cid=$(cid_for "$svc")
+              [ -n "$cid" ] || continue
+              s=$(docker inspect -f '{{.State.Status}}' "$cid" 2>/dev/null || true)
+              if [ "$s" = "exited" ] || [ "$s" = "dead" ]; then
+                echo "::error::Container for service $svc has exited"
+                exit 1
+              fi
+            done
+
+            sleep 10
+          done
+
+      - name: Probe LDR HTTP endpoint
+        # Avoids `curl ... | grep` so we don't need pipefail to surface curl
+        # failures — the HTTP code is captured directly via -w and checked
+        # with a case statement. `|| echo "000"` is the sentinel for true
+        # network failures (connection refused, DNS, etc.) so we can log
+        # on retry.
+        #
+        # Deliberately no `-f`: with `-f`, curl exits non-zero on HTTP
+        # 4xx/5xx AND suppresses -w output — which would collapse "404",
+        # "503", and network-error all into "000" and erase the most
+        # interesting failure signal (LDR up but serving an error page).
+        # Without -f, every HTTP response gives us its real code; only
+        # true network failures fall through to "000".
+        run: |
+          set -euo pipefail
+          for i in $(seq 1 30); do
+            code=$(curl -sS -o /dev/null -w "%{http_code}" http://localhost:5000/ 2>/dev/null || echo "000")
+            case "$code" in
+              200|301|302|303|307|308)
+                echo "LDR is serving on :5000 (HTTP $code)"
+                exit 0
+                ;;
+            esac
+            echo "Waiting for LDR HTTP (attempt $i/30, last code: $code)..."
+            sleep 5
+          done
+          echo "::error::LDR HTTP probe failed after ~150s"
+          exit 1
+
+      - name: Dump compose state and logs
+        if: always()
+        run: |
+          echo "::group::docker compose ps"
+          docker compose ps || true
+          echo "::endgroup::"
+
+          for svc in ollama searxng local-deep-research; do
+            echo "::group::$svc logs"
+            docker compose logs --no-color --tail=500 "$svc" || true
+            echo "::endgroup::"
+          done
+
+          # Capture image digests for both the workflow log (every run, via
+          # the cat below) and the auto-failure-issue body (failure runs
+          # only, via the next step). Always-logging gives us audit and
+          # bisection signal — "what was the published image SHA when this
+          # passed on date X?" — without needing to re-run the workflow.
+          # Resolve every container by service name through `docker compose
+          # ps -q` so we don't depend on `container_name:` values.
+          cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
+          {
+            echo "## Image digests"
+            for svc in ollama searxng local-deep-research; do
+              cid=$(cid_for "$svc")
+              if [ -n "$cid" ]; then
+                img=$(docker inspect -f '{{.Image}}' "$cid" 2>/dev/null || echo missing)
+                echo "- $svc: image=$img container=$cid"
+              else
+                echo "- $svc: container=missing"
+              fi
+            done
+          } > /tmp/digests.md
+          echo "::group::Image digests"
+          cat /tmp/digests.md
+          echo "::endgroup::"
+
+      - name: Open or update failure-tracking issue
+        if: failure() && github.event_name == 'schedule'
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          REPO: ${{ github.repository }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          # Stable title prefix so we dedup across weeks. If an issue with
+          # this title is already open, comment on it; otherwise create a
+          # fresh one. We don't auto-close on success — let a human triage.
+          TITLE="[compose-published-smoke] published-image drift detected"
+
+          DATE=$(date -u +%Y-%m-%d)
+          DIGESTS=$(cat /tmp/digests.md 2>/dev/null || echo "(digests not captured)")
+
+          BODY=$(cat <<MARKDOWN
+          The weekly **published-image smoke test** failed on $DATE.
+
+          This usually means \`main\`'s \`docker-compose.yml\` and the currently-published
+          \`localdeepresearch/local-deep-research:latest\` on Docker Hub have drifted —
+          either a compose change landed without an image republish, or the
+          published image (or one of its dependencies) regressed.
+
+          **Workflow run:** $RUN_URL
+
+          **What to check first:**
+          - Compose changes since the last release: \`git log --oneline <last-release-tag>..main -- docker-compose.yml\`
+          - Whether the LDR image needs a republish (cut a release, or the image was repointed)
+          - Upstream image regressions: ollama, searxng image digests below
+
+          $DIGESTS
+
+          *This issue is auto-managed by \`.github/workflows/compose-published-smoke.yml\`.
+          Subsequent failures will add comments here. Close it when the underlying drift is fixed
+          and a successful run rolls in.*
+          MARKDOWN
+          )
+
+          # Find existing open issue with the same title
+          NUM=$(gh issue list -R "$REPO" \
+            --state open \
+            --search "in:title \"[compose-published-smoke] published-image drift detected\"" \
+            --json number,title \
+            --jq '.[0].number // empty' \
+            --limit 1)
+
+          if [ -n "$NUM" ]; then
+            echo "Commenting on existing issue #$NUM"
+            # `--body=` (= form) prevents gh from interpreting body content
+            # as flags if it ever starts with `-`. Same below.
+            gh issue comment "$NUM" -R "$REPO" --body="$BODY"
+          else
+            echo "Creating new issue"
+            gh issue create -R "$REPO" \
+              --title="$TITLE" \
+              --label="bug,docker,ci-cd" \
+              --body="$BODY"
+          fi
+
+      - name: Tear down
+        # `|| true` so a teardown flake (daemon hiccup, hung container) can't
+        # flip the job to failure() after the smoke test itself passed —
+        # which would falsely trigger the auto-issue step. CI runners are
+        # ephemeral, so any leftover containers/volumes vanish with the
+        # runner regardless.
+        if: always()
+        run: docker compose down -v --remove-orphans || true

.github/workflows/container-security.yml

@@ -0,0 +1,166 @@
+name: Container Security
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  trivy-scan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Free up disk space
+      run: |
+        # Remove unnecessary files to free up disk space for Docker image scanning
+        sudo rm -rf /usr/share/dotnet
+        sudo rm -rf /usr/local/lib/android
+        sudo rm -rf /opt/ghc
+        sudo rm -rf /opt/hostedtoolcache/CodeQL
+        sudo docker image prune -af
+        df -h
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        persist-credentials: false
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+    - name: Build Docker image for scanning
+      uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+      with:
+        context: .
+        push: false
+        load: true
+        tags: local-deep-research:scan
+        cache-from: type=gha,scope=trivy-scan
+        cache-to: type=gha,mode=max,scope=trivy-scan
+
+    - name: Run Trivy vulnerability scanner
+      uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
+      with:
+        image-ref: local-deep-research:scan
+        format: 'sarif'
+        output: 'trivy-results.sarif'
+        severity: 'CRITICAL,HIGH,MEDIUM'
+        ignore-unfixed: true
+        scan-type: 'image'
+        trivyignores: '.trivyignore'
+        version: 'v0.69.2'
+
+    - name: Check if SARIF file exists
+      id: check-sarif
+      if: always()
+      run: |
+        if [ -f trivy-results.sarif ]; then
+          echo "exists=true" >> "$GITHUB_OUTPUT"
+        else
+          echo "::error::Trivy did not produce trivy-results.sarif — scan needs to be rerun"
+          exit 1
+        fi
+
+    - name: Upload Trivy scan results to GitHub Security tab
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      if: always() && steps.check-sarif.outputs.exists == 'true'
+      with:
+        sarif_file: 'trivy-results.sarif'
+        category: container-security
+
+    - name: Upload Trivy scan results as artifact
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      if: always() && steps.check-sarif.outputs.exists == 'true'
+      with:
+        name: trivy-scan-results
+        path: trivy-results.sarif
+        retention-days: 7  # Reduced for security
+
+    - name: Display Trivy summary
+      if: always()
+      run: |
+        {
+          echo "## Container Security Scan Summary"
+          echo ""
+          if [ -f trivy-results.sarif ]; then
+            echo "✅ Trivy scan completed - Results available in Security tab"
+            echo "📊 Scan results uploaded as artifact"
+          else
+            echo "❌ Trivy scan failed"
+          fi
+        } >> "$GITHUB_STEP_SUMMARY"
+
+  # Additional job for Dockerfile security analysis
+  dockerfile-scan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        persist-credentials: false
+
+    - name: Run Trivy config scan on Dockerfile
+      uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
+      with:
+        scan-type: 'config'
+        scan-ref: '.'
+        format: 'sarif'
+        output: 'trivy-config-results.sarif'
+        hide-progress: true
+        version: 'v0.69.2'
+
+    - name: Check if config SARIF file exists
+      id: check-config-sarif
+      if: always()
+      run: |
+        if [ -f trivy-config-results.sarif ]; then
+          echo "exists=true" >> "$GITHUB_OUTPUT"
+        else
+          echo "::error::Trivy did not produce trivy-config-results.sarif — scan needs to be rerun"
+          exit 1
+        fi
+
+    - name: Upload Trivy config scan results
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      if: always() && steps.check-config-sarif.outputs.exists == 'true'
+      with:
+        name: trivy-config-results
+        path: trivy-config-results.sarif
+        retention-days: 7  # Reduced for security
+
+    - name: Display config scan summary
+      if: always()
+      run: |
+        {
+          echo "## Docker Configuration Security Analysis"
+          echo ""
+          if [ -f trivy-config-results.sarif ]; then
+            echo "✅ Docker configuration analysis completed"
+            echo "📋 Check for Docker best practices and security misconfigurations"
+          else
+            echo "❌ Docker configuration scan failed"
+          fi
+        } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/danger-zone-alert.yml

@@ -0,0 +1,224 @@
+name: 🚨 Danger Zone Alert
+
+on:
+  pull_request:
+    types: [opened, synchronize]
+    paths:
+      # Explicit paths only — no keyword globs. A filename containing
+      # "decrypt" or "password" is not enough to imply security impact.
+      #
+      # ⚠️ KEEP IN SYNC with two other lists below:
+      #   1. EXPLICIT_WATCHED array in the "Validate watchlist" step
+      #      (drops glob entries; literal paths only — used for drift detection).
+      #   2. `case` patterns in the "Classify critical changes" step
+      #      (routes each path to a label bucket).
+      # A path here that's missing from (2) silently fails to label;
+      # a path in (2) that's missing here never triggers the workflow.
+      #
+      # Encryption / at-rest crypto / credential storage
+      - 'src/local_deep_research/database/encrypted_db.py'
+      - 'src/local_deep_research/database/sqlcipher_*.py'
+      - 'src/local_deep_research/database/credential_store_base.py'
+      - 'src/local_deep_research/database/backup/**'
+      # Authentication / sessions / passwords / auth DB / tenant isolation
+      - 'src/local_deep_research/web/auth/**'
+      - 'src/local_deep_research/database/auth_db.py'
+      - 'src/local_deep_research/database/session_passwords.py'
+      - 'src/local_deep_research/database/temp_auth.py'
+      - 'src/local_deep_research/database/session_context.py'
+      # Web hardening — CSP, SSRF, CSRF, rate-limit, headers, validators
+      - 'src/local_deep_research/security/**'
+      - 'src/local_deep_research/settings/env_definitions/security.py'
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  danger-alert:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+      checks: write
+      issues: write  # needed for createLabel (auto-create on first run)
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+        fetch-depth: 0
+
+    - name: Validate watchlist against repo tree
+      # Drift-guard: every explicit (non-glob) path below must exist on the
+      # branch head. Fails the workflow if a watched file has been renamed
+      # or deleted without updating this list. Glob patterns are excluded
+      # — they self-heal on rename and would false-positive this check.
+      # (Tradeoff: if all files matching a glob like `sqlcipher_*.py` are
+      # deleted, the trigger path silently becomes a no-op. Acceptable —
+      # disappearing an entire subsystem will be caught in code review.)
+      #
+      # ⚠️ KEEP IN SYNC with `on.pull_request.paths` above and the `case`
+      # patterns in the "Classify critical changes" step below.
+      run: |
+        EXPLICIT_WATCHED=(
+          "src/local_deep_research/database/encrypted_db.py"
+          "src/local_deep_research/database/credential_store_base.py"
+          "src/local_deep_research/database/auth_db.py"
+          "src/local_deep_research/database/session_passwords.py"
+          "src/local_deep_research/database/temp_auth.py"
+          "src/local_deep_research/database/session_context.py"
+          "src/local_deep_research/settings/env_definitions/security.py"
+        )
+        missing=0
+        for p in "${EXPLICIT_WATCHED[@]}"; do
+          if [ ! -e "$p" ]; then
+            echo "::error::danger-zone-alert watchlist drift: $p does not exist"
+            missing=1
+          fi
+        done
+        [ $missing -eq 0 ]
+
+    - name: Classify critical changes
+      id: analyze
+      env:
+        BASE_REF: ${{ github.base_ref }}
+      run: |
+        # Use env var to prevent template injection from malicious branch names
+        CHANGED=$(git diff --name-only "origin/$BASE_REF...HEAD")
+
+        ENCRYPTION_MODIFIED=false
+        AUTH_MODIFIED=false
+        HARDENING_MODIFIED=false
+
+        # Shell `case` `*` matches `/` (unlike file globs), so `web/auth/*`
+        # and `backup/*` cover nested subdirs — equivalent to the `**` in
+        # GitHub Actions path globs above. Do NOT "fix" these to `**` —
+        # bash `case` treats `**` as two literal `*`s with no special
+        # meaning and it would stop matching anything with a slash.
+        #
+        # ⚠️ KEEP IN SYNC with `on.pull_request.paths` above and the
+        # EXPLICIT_WATCHED array in the "Validate watchlist" step above.
+        for file in $CHANGED; do
+          case "$file" in
+            src/local_deep_research/database/encrypted_db.py|\
+            src/local_deep_research/database/sqlcipher_*.py|\
+            src/local_deep_research/database/credential_store_base.py|\
+            src/local_deep_research/database/backup/*)
+              ENCRYPTION_MODIFIED=true ;;
+            src/local_deep_research/web/auth/*|\
+            src/local_deep_research/database/auth_db.py|\
+            src/local_deep_research/database/session_passwords.py|\
+            src/local_deep_research/database/temp_auth.py|\
+            src/local_deep_research/database/session_context.py)
+              AUTH_MODIFIED=true ;;
+            src/local_deep_research/security/*|\
+            src/local_deep_research/settings/env_definitions/security.py)
+              HARDENING_MODIFIED=true ;;
+          esac
+        done
+
+        HAS_CRITICAL=false
+        if [ "$ENCRYPTION_MODIFIED" = "true" ] || [ "$AUTH_MODIFIED" = "true" ] || [ "$HARDENING_MODIFIED" = "true" ]; then
+          HAS_CRITICAL=true
+        fi
+
+        {
+          echo "has_critical=$HAS_CRITICAL"
+          echo "encryption_modified=$ENCRYPTION_MODIFIED"
+          echo "auth_modified=$AUTH_MODIFIED"
+          echo "hardening_modified=$HARDENING_MODIFIED"
+        } >> "$GITHUB_OUTPUT"
+
+    - name: Create status check
+      if: steps.analyze.outputs.has_critical == 'true'
+      uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+      with:
+        script: |
+          // Neutral check — advisory, not blocking. Real enforcement, if
+          // desired, belongs in branch-protection rules, not in a status
+          // title that misleads reviewers.
+          try {
+            await github.rest.checks.create({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              name: 'Security Review Required',
+              head_sha: context.sha,
+              status: 'completed',
+              conclusion: 'neutral',
+              output: {
+                title: 'Security-sensitive paths modified',
+                summary: 'This PR modifies critical security code (encryption, authentication, or web hardening). Please review carefully.'
+              }
+            });
+          } catch (err) {
+            // Fork PRs get a read-only token — checks.create 403s. Log
+            // and stay green (same pattern as pr-triage.yml); reviewers
+            // still get the signal from CODEOWNERS on these paths.
+            if (err.status !== 403) throw err;
+            console.log('checks.create returned 403 (read-only fork token). Skipping.');
+          }
+
+    - name: Add labels
+      if: steps.analyze.outputs.has_critical == 'true'
+      uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+      env:
+        ENCRYPTION_MODIFIED: ${{ steps.analyze.outputs.encryption_modified }}
+        AUTH_MODIFIED: ${{ steps.analyze.outputs.auth_modified }}
+        HARDENING_MODIFIED: ${{ steps.analyze.outputs.hardening_modified }}
+      with:
+        script: |
+          const wanted = [];
+          if (process.env.ENCRYPTION_MODIFIED === 'true') {
+            wanted.push({ name: 'touches-encryption', color: 'b60205', description: 'Modifies at-rest crypto, credential storage, or backup encryption' });
+          }
+          if (process.env.AUTH_MODIFIED === 'true') {
+            wanted.push({ name: 'touches-authentication', color: 'd93f0b', description: 'Modifies login, sessions, passwords, auth DB, or tenant isolation' });
+          }
+          if (process.env.HARDENING_MODIFIED === 'true') {
+            wanted.push({ name: 'touches-web-hardening', color: 'fbca04', description: 'Modifies CSP, SSRF, CSRF, rate-limit, headers, validators, or hardening config' });
+          }
+
+          if (wanted.length === 0) return;
+
+          // Ensure each label exists before applying (addLabels 422s on unknown labels).
+          for (const l of wanted) {
+            try {
+              await github.rest.issues.createLabel({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                name: l.name,
+                color: l.color,
+                description: l.description,
+              });
+            } catch (e) {
+              // 422 = already exists. 403 = read-only fork token (the
+              // addLabels call below degrades the same way).
+              if (e.status !== 422 && e.status !== 403) throw e;
+            }
+          }
+
+          try {
+            await github.rest.issues.addLabels({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+              labels: wanted.map(l => l.name),
+            });
+          } catch (err) {
+            // Fork PRs get a read-only token — the call 403s. Log and
+            // stay green (same pattern as pr-triage.yml); a maintainer
+            // can apply the labels manually on fork PRs.
+            if (err.status !== 403) throw err;
+            console.log('Label call returned 403 (read-only fork token). Skipping.');
+          }

.github/workflows/dependency-review.yml

@@ -0,0 +1,76 @@
+name: Dependency Review
+
+on:
+  # NOTE: dependency-review requires PR context to compare changes.
+  # It remains on PRs only and is NOT included in release gate.
+  # The osv-scanner and npm-audit in release gate cover dependency vulnerabilities.
+  pull_request:
+    branches: [main, develop]
+    paths:
+      - 'pyproject.toml'
+      - 'pdm.lock'
+      - 'requirements*.txt'
+      - 'package.json'
+      - 'package-lock.json'
+      - 'tests/ui_tests/package.json'
+      - 'tests/ui_tests/package-lock.json'
+  workflow_dispatch:
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  dependency-review:
+    name: Dependency Review
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      pull-requests: write
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout Repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Dependency Review
+        uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
+        with:
+          # Fail on high and critical severity vulnerabilities
+          fail-on-severity: high
+          # Allow only known permissive + weak-copyleft licenses (ASF Category A + B)
+          # All licenses below are commercially usable
+          # LGPL is excluded per ASF Category X policy
+          allow-licenses: >-
+            MIT,
+            Apache-2.0,
+            BSD-2-Clause,
+            BSD-3-Clause,
+            ISC,
+            MPL-2.0,
+            Unlicense,
+            CC0-1.0,
+            CC-BY-3.0,
+            CC-BY-4.0,
+            Python-2.0,
+            PSF-2.0,
+            0BSD,
+            OFL-1.1,
+            Zlib,
+            BlueOak-1.0.0
+          # Allow specific packages with complex license expressions
+          # dompurify is dual-licensed (Apache-2.0 OR MPL-2.0) but npm reports complex SPDX
+          allow-dependencies-licenses: >-
+            pkg:npm/dompurify
+          # Comment on PR with review summary
+          comment-summary-in-pr: on-failure

.github/workflows/devskim.yml

@@ -0,0 +1,152 @@
+name: DevSkim Security Linter
+
+on:
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+  schedule:
+    # Run security linter daily at 10 AM UTC
+    - cron: '0 10 * * *'
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  devskim-scan:
+    name: DevSkim Security Linter
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Run DevSkim security linter
+        uses: microsoft/DevSkim-Action@4b5047945a44163b94642a1cecc0d93a3f428cc6 # v1.0.16
+        with:
+          # Scan all source code
+          directory-to-scan: '.'
+
+          # Output SARIF for GitHub Security tab
+          output-filename: 'devskim-results.sarif'
+
+          # Ignore test fixtures and documentation
+          # - tests/: All test code uses mock data and localhost URLs
+          # - examples/: Example code with placeholder values
+          # - docs/: Documentation with example URLs
+          # - node_modules/: Third-party code
+          ignore-globs: 'tests/**,examples/**,docs/**,node_modules/**,**/node_modules/**'
+
+          # Exclude rules that produce false positives in this codebase:
+          #
+          # DS176209 - "Suspicious comment" (TODO/FIXME/HACK)
+          #   These are standard development annotations, not security issues.
+          #   Every codebase has TODO comments for tracking technical debt.
+          #
+          # DS162092 - "Hardcoded URL"
+          #   This research tool integrates with legitimate external APIs
+          #   (ArXiv, PubMed, Semantic Scholar, etc.). All URLs are intentional
+          #   service endpoints, not security vulnerabilities.
+          #
+          # DS137138 - "Hardcoded credentials"
+          #   Analysis showed 100% of alerts are test fixtures (api_key="test_key",
+          #   password="testpass", etc.). Zero real credentials found. Gitleaks
+          #   handles actual secret detection separately.
+          #
+          # DS148264 - "Use cryptographic random"
+          #   Flags ALL Python `random` module usage without context. This codebase
+          #   correctly uses `secrets` for security (session tokens, auth tokens,
+          #   Flask keys) and `random` only for non-security purposes: ML fairness
+          #   shuffling in search strategies, distributed systems jitter in scheduler,
+          #   and session cleanup probability. None involve cryptographic operations.
+          #
+          # DS172411 - "setTimeout code injection"
+          #   Flags ALL JavaScript setTimeout() calls. The dangerous pattern is
+          #   setTimeout("string", delay) which can execute arbitrary code. However,
+          #   100% of usages in this codebase (80+) pass safe function references:
+          #   setTimeout(() => {...}, delay) or setTimeout(functionName, delay).
+          #   No string-based setTimeout usage exists.
+          #
+          # DS126858 - "Weak/Broken Hash Algorithm"
+          #   Flags any occurrence of the literal string "sha1". In this codebase
+          #   the only matches are the JSON key name `"sha1"` inside SLSA provenance
+          #   attestations (.github/workflows/prerelease-docker.yml) — that key name
+          #   is *required* by the SLSA in-toto schema for git-commit digests, and
+          #   Git itself uses SHA-1 for commit identifiers. It is not a cryptographic
+          #   choice we make. Legitimate SHA-1 references elsewhere (PBKDF2_HMAC_SHA1,
+          #   HMAC_SHA1 in db_config.py for SQLCipher backwards compatibility) carry
+          #   their own inline `# DevSkim: ignore DS126858` annotations explaining the
+          #   rationale. See .github/SECURITY_ALERTS.md for the full assessment.
+          #
+          exclude-rules: 'DS176209,DS162092,DS137138,DS148264,DS172411,DS126858'
+
+      - name: Upload DevSkim results to GitHub Security
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        if: always()
+        with:
+          sarif_file: 'devskim-results.sarif'
+          category: devskim
+
+      - name: Upload DevSkim results as artifact
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: devskim-results
+          path: devskim-results.sarif
+          retention-days: 7  # Reduced for security
+
+      - name: Generate summary
+        run: |
+          {
+            echo "## DevSkim Security Linter Summary"
+            echo ""
+            echo "✅ DevSkim security scan completed"
+            echo ""
+            echo "### What was scanned:"
+            echo "- 🔍 All source code in the repository"
+            echo "- 🛡️ Common security issues and anti-patterns"
+            echo "- 📋 Insecure coding practices"
+            echo "- 🔑 Hardcoded secrets and credentials"
+            echo ""
+            echo "### Security checks include:"
+            echo "- **Hardcoded Credentials**: Passwords, API keys, tokens"
+            echo "- **Insecure Functions**: Dangerous API usage"
+            echo "- **Buffer Overflows**: Memory safety issues"
+            echo "- **Crypto Issues**: Weak encryption practices"
+            echo "- **Network Security**: Insecure protocols and configurations"
+            echo "- **File System**: Insecure file operations"
+            echo "- **Input Validation**: Missing input sanitization"
+            echo ""
+            echo "### Language Support:"
+            echo "- 🐍 Python (.py files)"
+            echo "- 📜 JavaScript/TypeScript (.js, .ts files)"
+            echo "- 🖥️ C/C++ (.c, .cpp, .h files)"
+            echo "- 🔷 C# (.cs files)"
+            echo "- 📝 More languages supported"
+            echo ""
+            echo "### Why DevSkim?"
+            echo "- **Microsoft Maintained**: Backed by Microsoft security team"
+            echo "- **Truly Free**: No registration, no API keys, no upsells"
+            echo "- **Fast & Lightweight**: Quick scans with minimal overhead"
+            echo "- **Comprehensive Rules**: 200+ security rules"
+            echo "- **Open Source**: Community-driven rule improvements"
+            echo ""
+            echo "📊 **Results uploaded to:**"
+            echo "- GitHub Security tab (if issues found)"
+            echo "- Workflow artifacts (SARIF files)"
+            echo ""
+            echo "🔗 **Learn more about DevSkim:**"
+            echo "- [DevSkim GitHub](https://github.com/microsoft/DevSkim)"
+            echo "- [DevSkim Documentation](https://microsoft.github.io/DevSkim/)"
+            echo "- [DevSkim Rules](https://github.com/microsoft/DevSkim/tree/main/rules)"
+          } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/docker-multiarch-test.yml

@@ -0,0 +1,156 @@
+name: Test Multi-Architecture Docker Build
+
+on:
+  workflow_call:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test-multiarch-build:
+    name: Test Multi-Arch Docker Build
+    strategy:
+      matrix:
+        include:
+          - platform: linux/amd64
+            runs-on: ubuntu-latest
+            platform_slug: linux-amd64
+          - platform: linux/arm64
+            runs-on: ubuntu-24.04-arm
+            platform_slug: linux-arm64
+
+    runs-on: ${{ matrix.runs-on }}
+    timeout-minutes: 60
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Check out the repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      - name: Build Docker image for ${{ matrix.platform }}
+        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        with:
+          context: .
+          platforms: ${{ matrix.platform }}
+          push: false
+          load: false
+          cache-from: type=gha,scope=${{ matrix.platform_slug }}
+          cache-to: type=gha,mode=max,scope=${{ matrix.platform_slug }}
+          tags: ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
+          outputs: type=docker,dest=/tmp/image-${{ matrix.platform_slug }}.tar
+
+      - name: Inspect built image
+        run: |
+          # Load and inspect the image
+          docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
+          docker image inspect ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
+
+          # Get image size
+          IMAGE_SIZE=$(docker image inspect ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} --format='{{.Size}}' | numfmt --to=iec)
+          echo "Image size for ${{ matrix.platform }}: $IMAGE_SIZE"
+
+      - name: Test image startup
+        run: |
+          docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
+
+          # Test that the container can start and Python works
+          docker run --rm --platform ${{ matrix.platform }} \
+            ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} \
+            python -c "import sys; print(f'Python {sys.version} on {sys.platform}')"
+
+          # Test that the LDR package is installed
+          docker run --rm --platform ${{ matrix.platform }} \
+            ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} \
+            python -c "import local_deep_research; print('LDR package imported successfully')"
+
+      - name: Test web server startup
+        run: |
+          docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
+
+          # Start the container in background
+          docker run -d --name ldr-test \
+            --platform ${{ matrix.platform }} \
+            -p 5000:5000 \
+            -e LDR_DB_CONFIG_KDF_ITERATIONS=1000 \
+            -e LDR_TEST_MODE=1 \
+            -e LDR_NEWS_SCHEDULER_ENABLED=false \
+            ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
+
+          # Wait for server to start
+          TIMEOUT_SECONDS=60
+          echo "Waiting for server to start (timeout: ${TIMEOUT_SECONDS}s)..."
+          for i in $(seq 1 ${TIMEOUT_SECONDS}); do
+            if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
+              echo "✅ Server started successfully after ${i} seconds on ${{ matrix.platform }}!"
+              break
+            fi
+            # Show progress every 10 seconds
+            if [ $((i % 10)) -eq 0 ]; then
+              echo "Still waiting... (${i}s elapsed)"
+            fi
+            if [ "$i" -eq "${TIMEOUT_SECONDS}" ]; then
+              echo "❌ Server failed to start within ${TIMEOUT_SECONDS} seconds"
+              docker logs ldr-test
+              exit 1
+            fi
+            sleep 1
+          done
+
+          # Check server response
+          RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:5000)
+          echo "Server response code: $RESPONSE"
+
+          # Clean up
+          docker stop ldr-test
+          docker rm ldr-test
+
+  summary:
+    name: Build Summary
+    needs: test-multiarch-build
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    if: always()
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Summary
+        env:
+          BUILD_RESULT: ${{ needs.test-multiarch-build.result }}
+        run: |
+          {
+            echo "## Multi-Architecture Build Test Results"
+            echo ""
+          } >> "$GITHUB_STEP_SUMMARY"
+
+          if [ "$BUILD_RESULT" == "success" ]; then
+            {
+              echo "✅ **All platforms built and tested successfully!**"
+              echo ""
+              echo "### Tested Platforms:"
+              echo "- linux/amd64 ✅"
+              echo "- linux/arm64 ✅"
+              echo ""
+              echo "The Docker image is ready for multi-architecture deployment."
+            } >> "$GITHUB_STEP_SUMMARY"
+          else
+            {
+              echo "❌ **Build or tests failed**"
+              echo ""
+              echo "Please check the logs above for details."
+            } >> "$GITHUB_STEP_SUMMARY"
+          fi

.github/workflows/docker-publish.yml

@@ -0,0 +1,334 @@
+name: Publish Docker image
+
+# SECURITY: Only invoked as a reusable workflow (`workflow_call`) from
+# release.yml after the release security gate AND the `release` environment
+# approval pass. We intentionally do NOT support workflow_dispatch — that
+# would bypass gates. We also intentionally do NOT support
+# repository_dispatch any more: with the atomicity refactor, this workflow
+# runs as a job inside release.yml's run so its result is visible to
+# downstream jobs (create-release, cleanup-on-rejection) — a property that
+# repository_dispatch fanout broke.
+#
+# This workflow is a thin RETAG step in the build-once-promote pipeline.
+# The actual build happens in prerelease-docker.yml; the multi-arch manifest
+# is signed and attested there. Here we only:
+#   - Verify the source manifest digest matches what prerelease produced
+#   - Retag the prerelease manifest to release tags (:1.6.9, :1.6, :latest)
+#   - Verify the digest is preserved (defends against imagetools re-encoding)
+#   - Re-run Trivy against the digest (catches CVE-database updates between
+#     prerelease build and release promote)
+#   - Verify cosign signature transitivity from the original digest
+#   - Clean up the prerelease tags
+#
+# To re-publish, trigger a new release through release.yml.
+on:
+  workflow_call:
+    inputs:
+      tag:
+        description: "Release tag, e.g. 'v1.6.9' (with leading 'v')"
+        type: string
+        required: true
+      source_tag:
+        description: "Prerelease manifest tag to retag, e.g. 'prerelease-v1.6.9-abc1234'"
+        type: string
+        required: true
+      expected_digest:
+        description: "sha256:... digest of the prerelease manifest, captured by prerelease-docker.yml. Used to verify retag preserves the digest end-to-end."
+        type: string
+        required: true
+    secrets:
+      DOCKER_USERNAME:
+        required: true
+        description: "Docker Hub username (env-scoped to `release`)"
+      DOCKER_PASSWORD:
+        required: true
+        description: "Docker Hub PAT with Read+Write+Delete scopes (env-scoped to `release`)"
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+# NOTE: no workflow-level `concurrency:` block. As a reusable workflow
+# called from release.yml, this workflow runs as part of the caller's run,
+# and release.yml's caller-level concurrency (keyed on github.workflow +
+# github.ref) already serialises release runs for the same tag. Adding a
+# callee-level block would be a no-op for the documented threat model
+# (two simultaneous releases for the same ref) and would not be reachable
+# anyway because there is no longer any standalone invocation path.
+
+jobs:
+  promote:
+    name: Retag prerelease manifest as release
+    runs-on: ubuntu-latest
+    environment: release
+    permissions:
+      # cosign verify is read-only against public Rekor/Fulcio — does not
+      # mint a GitHub OIDC token, so id-token: write is not required here.
+      # Signing (which does need id-token: write) happens in prerelease-docker.yml.
+      contents: read
+
+    steps:
+      - name: Harden Runner
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Check out the repo at the triggering commit
+        # Pin the checkout to the EXACT commit that the prerelease was
+        # built/scanned from, so .trivyignore (and any other repo-state-
+        # dependent file the promote step reads) matches that commit.
+        # We use github.sha, NOT inputs.tag — the v* git tag is created
+        # by create-release LATER in this run (after publish-docker
+        # completes), so it does not exist yet when this checkout runs
+        # on a push-to-main trigger. github.sha is the triggering commit
+        # for every event type (push to main, tag push, workflow_dispatch)
+        # and is the same SHA the build/prerelease-docker jobs used.
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Install Cosign
+        uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
+        with:
+          # Pin to cosign v2.x to match the version that signed the artifact
+          # in prerelease-docker.yml. Mismatched versions across sign/verify
+          # work today but new-bundle-format (cosign v3 default) would only
+          # produce/consume on v3.
+          cosign-release: 'v2.6.3'
+
+      - name: Determine release tags
+        id: version
+        env:
+          DISPATCH_TAG: ${{ inputs.tag }}
+          SOURCE_TAG: ${{ inputs.source_tag }}
+          EXPECTED_DIGEST: ${{ inputs.expected_digest }}
+        run: |
+          set -euo pipefail
+          if [[ -z "$DISPATCH_TAG" || -z "$SOURCE_TAG" || -z "$EXPECTED_DIGEST" ]]; then
+            echo "::error::Missing required workflow_call input. Got tag='${DISPATCH_TAG}' source_tag='${SOURCE_TAG}' expected_digest='${EXPECTED_DIGEST}'"
+            exit 1
+          fi
+          if [[ "$EXPECTED_DIGEST" != sha256:* ]]; then
+            echo "::error::expected_digest must be of the form sha256:... — got '${EXPECTED_DIGEST}'"
+            exit 1
+          fi
+          VERSION="${DISPATCH_TAG#v}"
+          MAJOR_MINOR=$(echo "$VERSION" | cut -d. -f1,2)
+          # Group writes into one redirect (shellcheck SC2129).
+          {
+            echo "tag=${DISPATCH_TAG}"
+            echo "version=${VERSION}"
+            echo "major_minor=${MAJOR_MINOR}"
+            echo "source_tag=${SOURCE_TAG}"
+            echo "expected_digest=${EXPECTED_DIGEST}"
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Verify source digest matches expected
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          SOURCE_TAG: ${{ steps.version.outputs.source_tag }}
+          EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
+        run: |
+          set -euo pipefail
+          # Defends against the prerelease tag being swapped between
+          # prerelease-docker's signing and this promote step.
+          SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"
+          ACTUAL=$(docker buildx imagetools inspect "$SOURCE" --format '{{json .Manifest.Digest}}' | tr -d '"')
+          if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then
+            echo "::error::Source digest mismatch — possible tag tampering between prerelease and promote"
+            echo "  expected: $EXPECTED_DIGEST"
+            echo "  actual:   $ACTUAL"
+            exit 1
+          fi
+          echo "Source digest verified: $ACTUAL"
+
+      - name: Promote (retag) prerelease manifest to release tags
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          SOURCE_TAG: ${{ steps.version.outputs.source_tag }}
+          VERSION: ${{ steps.version.outputs.version }}
+          MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
+        run: |
+          set -euo pipefail
+          SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"
+          # Single imagetools create with multiple -t — registry-side
+          # metadata-only operation, takes seconds, preserves digest.
+          docker buildx imagetools create \
+            -t "${DOCKER_USERNAME}/local-deep-research:${VERSION}" \
+            -t "${DOCKER_USERNAME}/local-deep-research:${MAJOR_MINOR}" \
+            -t "${DOCKER_USERNAME}/local-deep-research:latest" \
+            "$SOURCE"
+          echo "Promoted ${SOURCE} to :${VERSION}, :${MAJOR_MINOR}, :latest"
+
+      - name: Verify promoted tags share the source digest
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          VERSION: ${{ steps.version.outputs.version }}
+          MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
+          EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
+        run: |
+          set -euo pipefail
+          # Defends against `imagetools create` re-encoding the manifest.
+          # If digests diverge, signatures and attestations (keyed by the
+          # original digest) won't be discoverable from the new tags.
+          for TAG in "${VERSION}" "${MAJOR_MINOR}" "latest"; do
+            REF="${DOCKER_USERNAME}/local-deep-research:${TAG}"
+            ACTUAL=$(docker buildx imagetools inspect "$REF" --format '{{json .Manifest.Digest}}' | tr -d '"')
+            if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then
+              echo "::error::Digest mismatch on ${TAG} — imagetools create may have re-encoded the manifest"
+              echo "  expected: $EXPECTED_DIGEST"
+              echo "  actual:   $ACTUAL"
+              exit 1
+            fi
+            echo "${TAG} -> ${ACTUAL} ✓"
+          done
+
+      # Catches CVE-database updates that landed between the prerelease
+      # build and this promote step. Use the SHA-pinned action wrapper
+      # (same pin as prerelease-docker.yml's security-scan) with an
+      # explicit binary version pin — the prior `apt-get install -y trivy`
+      # approach was unpinned and exposed the release path to the Trivy
+      # apt-repo supply chain. The pinned action downloads the v0.69.2
+      # binary from GitHub releases by exact tag, which is the same
+      # binary the prerelease scan validated, keeping the two scans
+      # consistent.
+      #
+      # Unlike prerelease-docker.yml's security-scan (which scans a
+      # locally-loaded image, no registry pull), this step scans by
+      # registry digest — Trivy must pull the manifest + layers from
+      # Docker Hub. TRIVY_USERNAME/TRIVY_PASSWORD is the action's
+      # documented auth path; the `docker/login-action` above also
+      # writes ~/.docker/config.json which Trivy reads as a fallback,
+      # but the explicit env vars are more reliable (Trivy has
+      # documented docker.io credential-helper quirks — aquasecurity/
+      # trivy#432, aquasecurity/trivy#8385) and the image we scan IS on
+      # Docker Hub so this is the path most likely to keep working.
+      - name: Re-scan release digest with Trivy
+        uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
+        env:
+          TRIVY_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          TRIVY_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+        with:
+          image-ref: ${{ secrets.DOCKER_USERNAME }}/local-deep-research@${{ steps.version.outputs.expected_digest }}
+          severity: 'CRITICAL,HIGH'
+          ignore-unfixed: true
+          trivyignores: '.trivyignore'
+          exit-code: '1'
+          version: 'v0.69.2'
+
+      - name: Verify cosign signature on promoted digest
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
+          REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          # The cert was issued to the prerelease-docker.yml workflow when
+          # signing happened there, so the identity regex must match that
+          # workflow's path. Fulcio's SAN is built from job_workflow_ref,
+          # which for reusable workflows is the CALLEE.
+          #
+          # Verify by IMMUTABLE digest (not the :VERSION tag) so this step
+          # is invariant under any retag race between the verify-promoted-
+          # tags step above and this one. Trivy's re-scan above also uses
+          # @${EXPECTED_DIGEST}; keeping cosign on the same reference is
+          # consistent and avoids a tag-resolution TOCTOU window.
+          IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${EXPECTED_DIGEST}"
+          echo "Verifying signature for: $IMAGE_REF"
+          cosign verify \
+            --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
+            --certificate-identity-regexp "^https://github.com/${REPO}/\.github/workflows/prerelease-docker\.yml@refs/(heads|tags)/" \
+            --certificate-github-workflow-repository "${REPO}" \
+            "$IMAGE_REF"
+          echo "Signature transitivity verified ✓"
+
+      - name: Clean up prerelease tags
+        continue-on-error: true
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+          RELEASE_VERSION: ${{ steps.version.outputs.version }}
+        run: |
+          set +e  # Best-effort: never fail the workflow
+          # Scope deletion to prereleases of THIS version only, so concurrent
+          # prereleases for other versions (and any unrelated prerelease-* tags)
+          # are left untouched.
+          PREFIX="prerelease-v${RELEASE_VERSION}-"
+          echo "Cleaning up ${PREFIX}* tags from Docker Hub..."
+
+          # Authenticate with Docker Hub API (password-based JWT)
+          TOKEN=$(curl -s -X POST "https://hub.docker.com/v2/users/login/" \
+            -H "Content-Type: application/json" \
+            -d "{\"username\": \"${DOCKER_USERNAME}\", \"password\": \"${DOCKER_PASSWORD}\"}" | jq -r '.token')
+
+          if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
+            echo "WARNING: Failed to authenticate with Docker Hub API. Skipping cleanup."
+            exit 0
+          fi
+
+          REPO="${DOCKER_USERNAME}/local-deep-research"
+          PAGE=1
+          DELETED=0
+
+          while true; do
+            RESPONSE=$(curl -s -H "Authorization: JWT ${TOKEN}" \
+              "https://hub.docker.com/v2/repositories/${REPO}/tags/?page=${PAGE}&page_size=100")
+
+            RESULTS=$(echo "$RESPONSE" | jq -r '.results[]?.name // empty')
+            if [ -z "$RESULTS" ]; then
+              break
+            fi
+
+            while IFS= read -r tag; do
+              if [[ "$tag" == "${PREFIX}"* ]]; then
+                echo "Deleting tag: ${tag}"
+                HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X DELETE \
+                  -H "Authorization: JWT ${TOKEN}" \
+                  "https://hub.docker.com/v2/repositories/${REPO}/tags/${tag}/")
+                if [ "$HTTP_CODE" -eq 204 ] || [ "$HTTP_CODE" -eq 200 ]; then
+                  echo "  Deleted: ${tag}"
+                  DELETED=$((DELETED + 1))
+                else
+                  echo "  WARNING: Failed to delete ${tag} (HTTP ${HTTP_CODE})"
+                fi
+              fi
+            done <<< "$RESULTS"
+
+            # Check if there are more pages
+            NEXT=$(echo "$RESPONSE" | jq -r '.next // empty')
+            if [ -z "$NEXT" ]; then
+              break
+            fi
+            PAGE=$((PAGE + 1))
+          done
+
+          echo "Prerelease tag cleanup complete. Deleted ${DELETED} tag(s) matching ${PREFIX}*."
+
+      - name: Summary
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          VERSION: ${{ steps.version.outputs.version }}
+          MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
+          EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
+        run: |
+          {
+            echo "## Docker Release Promoted"
+            echo ""
+            echo "**Digest:** \`${EXPECTED_DIGEST}\`"
+            echo ""
+            echo "**Tags:** \`${VERSION}\`, \`${MAJOR_MINOR}\`, \`latest\` — all share the same digest as the prerelease manifest, so cosign signatures, SBOM, and SLSA provenance from the prerelease step are transitively valid."
+            echo ""
+            echo '```'
+            echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${VERSION}"
+            echo "docker pull ${DOCKER_USERNAME}/local-deep-research:latest"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/dockle.yml

@@ -0,0 +1,211 @@
+name: Dockle Container Security Linting
+
+on:
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+  schedule:
+    # Run weekly on Tuesday at 10 AM UTC (staggered with other container scans)
+    - cron: '0 10 * * 2'
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  dockle:
+    name: Dockle Container Image Security
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout repository
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+    - name: Build Docker image for scanning
+      uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+      with:
+        context: .
+        push: false
+        load: true
+        tags: local-deep-research:dockle-scan
+        cache-from: type=gha,scope=dockle-scan
+        cache-to: type=gha,mode=max,scope=dockle-scan
+
+    - name: Run Dockle security scan
+      id: dockle
+      run: |
+        echo "=== Running Dockle Container Image Security Scan ==="
+        # Run Dockle with SARIF output for GitHub Security tab
+        # Mount current directory to /output so SARIF file is written to host filesystem
+        # Exit code 1 means issues found, but we don't fail the workflow
+        #
+        # Ignored checks (false positives or intentional design):
+        # - CIS-DI-0001: Create a user - image uses setpriv entrypoint pattern for privilege drop
+        #   (see scripts/ldr_entrypoint.sh which uses setpriv to switch to ldruser)
+        # - CIS-DI-0005: Enable Content trust for Docker - runtime env var (DOCKER_CONTENT_TRUST),
+        #   not a Dockerfile concern; enforced at deployment time
+        # - CIS-DI-0008: setuid/setgid files - unix_chkpwd is standard Debian base utility
+        #   required for PAM authentication, inherited from python:3.14-slim base image
+        # - DKL-DI-0005: Clear apt-get caches - already done in Dockerfile (rm -rf /var/lib/apt/lists/*),
+        #   but base image (python:3.14-slim) layers trigger this check
+        #
+        # Accepted files (false positives):
+        # - unix_chkpwd: PAM utility from Debian base image
+        # - settings.py: Any file named settings.py (config files, not credentials)
+        # Without --exit-code flag, Dockle always exits 0 regardless of findings.
+        # Findings are reported via SARIF and enforced by the release gate
+        # (check-code-scanning-alerts job in release-gate.yml).
+        # Any non-zero exit code here means Docker itself failed (image pull error,
+        # socket unavailable, etc.) — a broken scanner that MUST fail the step,
+        # otherwise no SARIF is produced and the release gate has nothing to check.
+        set +e
+        docker run --rm \
+          -v /var/run/docker.sock:/var/run/docker.sock \
+          -v "$(pwd):/output" \
+          goodwithtech/dockle:v0.4.14 \
+          --format sarif \
+          --output /output/dockle-results.sarif \
+          --ignore CIS-DI-0001 \
+          --ignore CIS-DI-0005 \
+          --ignore CIS-DI-0008 \
+          --ignore DKL-DI-0005 \
+          --accept-file "usr/sbin/unix_chkpwd" \
+          --accept-file "settings.py" \
+          local-deep-research:dockle-scan
+        DOCKLE_EXIT_CODE=$?
+        set -e
+
+        if [ "$DOCKLE_EXIT_CODE" -ne 0 ]; then
+          echo "::error::Dockle/Docker failed with exit code $DOCKLE_EXIT_CODE"
+          exit 1
+        fi
+
+        # Check if SARIF file was created (for use in subsequent steps)
+        if [ -f "dockle-results.sarif" ]; then
+          echo "sarif_exists=true" >> "$GITHUB_OUTPUT"
+          # Check if SARIF contains any findings (used by summary step)
+          RESULT_COUNT=$(python3 -c "import json; print(len(json.load(open('dockle-results.sarif')).get('runs',[{}])[0].get('results',[])))" 2>/dev/null || echo "0")
+          if [ "$RESULT_COUNT" -gt 0 ]; then
+            echo "DOCKLE_FOUND_ISSUES=true" >> "$GITHUB_ENV"
+          fi
+        fi
+
+    - name: Run Dockle with human-readable output
+      if: always()
+      run: |
+        echo "=== Dockle Scan Results ==="
+        # Use same ignore flags as SARIF scan for consistent output
+        docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
+          goodwithtech/dockle:v0.4.14 \
+          --format list \
+          --ignore CIS-DI-0001 \
+          --ignore CIS-DI-0005 \
+          --ignore CIS-DI-0008 \
+          --ignore DKL-DI-0005 \
+          --accept-file "usr/sbin/unix_chkpwd" \
+          --accept-file "settings.py" \
+          local-deep-research:dockle-scan || true
+
+    - name: Upload Dockle results to GitHub Security tab
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      # Note: hashFiles() is evaluated at workflow parse time, not runtime.
+      # Use step output instead to check if SARIF file was created during the run.
+      if: always() && steps.dockle.outputs.sarif_exists == 'true'
+      with:
+        sarif_file: dockle-results.sarif
+        category: dockle
+
+    - name: Upload Dockle results as artifact
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      if: always() && steps.dockle.outputs.sarif_exists == 'true'
+      with:
+        name: dockle-results
+        path: dockle-results.sarif
+        retention-days: 7  # Reduced for security
+
+    - name: Display Dockle summary
+      if: always()
+      run: |
+        {
+          echo "## 🐋 Dockle Container Image Security Summary"
+          echo ""
+          echo "### What is Dockle?"
+          echo "Dockle is a container image linting tool that checks for:"
+          echo ""
+          echo "#### 🎯 Security Checks:"
+          echo "- **Sensitive Files**: Keys, certificates, passwords in image"
+          echo "- **Root User**: Warning if running as root"
+          echo "- **Clear-text Passwords**: Detects plaintext secrets"
+          echo "- **CIS Benchmarks**: Container security best practices"
+          echo "- **Outdated Packages**: Vulnerable base images"
+          echo ""
+          echo "#### 📊 Dockle vs Hadolint:"
+          echo "- **Hadolint**: Checks Dockerfile source code (best practices)"
+          echo "- **Dockle**: Scans built container image (runtime security)"
+          echo "- **Both Complement**: Check different stages of container lifecycle"
+          echo ""
+          echo "#### 🔍 Scan Results:"
+        } >> "$GITHUB_STEP_SUMMARY"
+
+        if [ "$DOCKLE_FOUND_ISSUES" = "true" ]; then
+          {
+            echo "⚠️ **Issues Found**"
+            echo ""
+            echo "Dockle detected security or best practice issues:"
+            echo ""
+            echo "**Common Issues:**"
+            echo "- Running as root user (consider USER directive)"
+            echo "- Sensitive files potentially included in image"
+            echo "- Missing security labels or metadata"
+            echo "- Outdated base image packages"
+            echo ""
+            echo "📋 **Detailed Results:**"
+            echo "- GitHub Security tab: SARIF results uploaded"
+            echo "- Artifacts: Full SARIF report available"
+            echo "- Check output above for human-readable format"
+          } >> "$GITHUB_STEP_SUMMARY"
+        else
+          {
+            echo "✅ **No Issues Found!**"
+            echo ""
+            echo "Container image passed all Dockle security checks."
+          } >> "$GITHUB_STEP_SUMMARY"
+        fi
+
+        {
+          echo ""
+          echo "#### 🔗 Resources:"
+          echo "- [Dockle GitHub](https://github.com/goodwithtech/dockle)"
+          echo "- [CIS Docker Benchmark](https://www.cisecurity.org/benchmark/docker)"
+          echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
+          echo ""
+          echo "#### 💡 Best Practices:"
+          echo "- Use multi-stage builds to reduce image size"
+          echo "- Add USER directive to run as non-root"
+          echo "- Don't include secrets or keys in image"
+          echo "- Keep base images updated"
+          echo "- Scan images with both Dockle (runtime) and Hadolint (build)"
+          echo ""
+          echo "#### 🔇 Ignored Checks (false positives or intentional design):"
+          echo "- **CIS-DI-0001** (Create a user): Uses setpriv entrypoint pattern for privilege drop"
+          echo "- **CIS-DI-0005** (Content trust): Runtime env var, not Dockerfile concern"
+          echo "- **CIS-DI-0008** (setuid/setgid): unix_chkpwd from Debian base, required for PAM"
+          echo "- **DKL-DI-0005** (apt-get cache): Dockerfile clears caches, base image triggers this"
+          echo "- **settings.py**: Any file named settings.py accepted (config files, not credentials)"
+        } >> "$GITHUB_STEP_SUMMARY"
+
+    - name: Clean up Docker image
+      if: always()
+      run: |
+        docker rmi local-deep-research:dockle-scan || true

.github/workflows/e2e-research-test.yml

@@ -0,0 +1,201 @@
+name: E2E Research Test
+
+on:
+  pull_request:
+    types: [labeled]
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions: {}  # Minimal top-level for OSSF Scorecard
+
+jobs:
+  build-query:
+    name: Assemble query
+    if: github.event.label.name == 'ldr_research' || github.event.label.name == 'ldr_research_static'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    outputs:
+      query: ${{ steps.assemble.outputs.query }}
+      header: ${{ steps.headers.outputs.header }}
+      subheader: ${{ steps.headers.outputs.subheader }}
+    env:
+      LABEL_NAME: ${{ github.event.label.name }}
+      MAX_DIFF_SIZE: ${{ vars.MAX_DIFF_SIZE || '8000' }}
+      BASE_REF: ${{ github.event.pull_request.base.ref }}
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Build PR-research query
+        id: assemble
+        run: |
+          if [ "$LABEL_NAME" = "ldr_research_static" ]; then
+            cat > query.txt <<'STATIC_EOF'
+          What is Local Deep Research and how does it work?
+          STATIC_EOF
+          else
+            git fetch origin "$BASE_REF"
+            git diff "origin/$BASE_REF...HEAD" --no-color > diff.txt
+            DIFF_BYTES=$(wc -c < diff.txt)
+            echo "Diff size: $DIFF_BYTES bytes (cap: $MAX_DIFF_SIZE)"
+
+            head -c "$MAX_DIFF_SIZE" diff.txt > diff-trunc.txt
+            if [ "$DIFF_BYTES" -gt "$MAX_DIFF_SIZE" ]; then
+              printf '\n... (truncated)\n' >> diff-trunc.txt
+            fi
+
+            {
+              cat <<'PROMPT_HEAD_EOF'
+          Based on these code changes, research relevant documentation,
+          best practices, and potential issues. Focus on any libraries, APIs, or patterns used.
+
+          Code changes:
+          PROMPT_HEAD_EOF
+              cat diff-trunc.txt
+              cat <<'PROMPT_TAIL_EOF'
+
+          Research topics to cover:
+          1. Documentation for any libraries or APIs being used/modified
+          2. Best practices for the patterns shown
+          3. Known issues or gotchas related to these changes
+          4. Security considerations if applicable
+          PROMPT_TAIL_EOF
+            } > query.txt
+          fi
+
+          # Multi-line GH Actions output via heredoc-style delimiter.
+          # Use a randomized delimiter so a query containing the literal
+          # delimiter on its own line cannot prematurely terminate the
+          # heredoc.
+          DELIM="LDR_QUERY_EOF_$$_${RANDOM}_$(date +%N)"
+          {
+            echo "query<<$DELIM"
+            cat query.txt
+            echo "$DELIM"
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Compute headers
+        id: headers
+        run: |
+          if [ "$LABEL_NAME" = "ldr_research_static" ]; then
+            echo "header=## 🧪 LDR Static Query Test Results" >> "$GITHUB_OUTPUT"
+            echo "subheader=**Query:** What is Local Deep Research and how does it work?" >> "$GITHUB_OUTPUT"
+          else
+            echo "header=## 🔬 LDR Research Results" >> "$GITHUB_OUTPUT"
+            echo "subheader=_Analysis of PR code changes_" >> "$GITHUB_OUTPUT"
+          fi
+
+  research:
+    name: Run LDR research
+    needs: build-query
+    # Grant the perms the reusable's job needs:
+    # - contents: read for actions/checkout
+    # - actions: write for actions/upload-artifact@v5+
+    # Without this, the reusable's `permissions: contents: read` exceeds
+    # the inherited empty permissions and GitHub rejects the workflow at
+    # load time (startup_failure with zero jobs).
+    permissions:
+      contents: read
+      actions: write
+    uses: ./.github/workflows/ldr-research-reusable.yml
+    with:
+      query: ${{ needs.build-query.outputs.query }}
+      model: ${{ github.event.label.name == 'ldr_research_static' && (vars.LDR_RESEARCH_CHEAP_MODEL || 'google/gemini-2.0-flash-001') || (vars.LDR_RESEARCH_MODEL || 'google/gemini-2.0-flash-001') }}
+      provider: ${{ vars.LDR_PROVIDER || 'openrouter' }}
+      search-tool: ${{ vars.LDR_SEARCH_TOOL || 'serper' }}
+      strategy: ${{ vars.LDR_STRATEGY || 'langgraph-agent' }}
+      comment-header: ${{ needs.build-query.outputs.header }}
+      comment-subheader: ${{ needs.build-query.outputs.subheader }}
+    secrets:
+      OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+      SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
+
+  post-comment:
+    name: Post PR comment
+    needs: research
+    # always() so the label-removal step runs even if research was skipped
+    # (e.g. build-query failed). The download/post steps are guarded by
+    # needs.research.outputs.success == 'true' so they self-skip in that
+    # case.
+    if: always()
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
+    env:
+      LABEL_NAME: ${{ github.event.label.name }}
+      PR_NUMBER: ${{ github.event.pull_request.number }}
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Download research artifact
+        if: needs.research.outputs.success == 'true'
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: ${{ needs.research.outputs.comment-artifact-name }}
+
+      - name: Post or update comment
+        if: needs.research.outputs.success == 'true'
+        continue-on-error: true   # a GitHub API hiccup shouldn't fail the job
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        with:
+          retries: 3
+          script: |
+            const fs = require('fs');
+            const issue_number = Number(process.env.PR_NUMBER);
+            // Key the marker by label so re-running the same label edits its
+            // comment in place, while the other label keeps a separate comment.
+            const marker = `<!-- ldr-research-results:${process.env.LABEL_NAME} -->`;
+            const body = marker + '\n' + fs.readFileSync('comment.md', 'utf8');
+
+            // Paginate — on a long-lived PR the existing comment may be past the
+            // first page (listComments defaults to 30), which would otherwise
+            // create a new comment every run instead of editing in place.
+            const comments = await github.paginate(github.rest.issues.listComments, {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number,
+              per_page: 100,
+            });
+            const existing = comments.find(
+              (c) => c.user.type === 'Bot' && c.body.includes(marker)
+            );
+
+            if (existing) {
+              await github.rest.issues.updateComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                comment_id: existing.id,
+                body,
+              });
+            } else {
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number,
+                body,
+              });
+            }
+
+      - name: Remove label for re-triggering
+        if: always()
+        run: |
+          gh issue edit "$PR_NUMBER" --remove-label "$LABEL_NAME" 2>/dev/null || true

.github/workflows/file-whitelist-check.yml

@@ -0,0 +1,47 @@
+name: File Whitelist Security Check
+# Enhanced security checks with comprehensive file type detection
+
+on:
+  pull_request:
+    branches: [ main ]
+  workflow_call:  # Called by ci-gate.yml for release pipeline
+    inputs:
+      check-all-files:
+        description: 'Check ALL tracked files (not just changed files)'
+        required: false
+        type: boolean
+        default: true
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions:
+  contents: read
+
+jobs:
+  whitelist-check:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+        fetch-depth: 0
+
+    - name: Security checks for files
+      env:
+        GITHUB_EVENT_NAME: ${{ github.event_name }}
+        GITHUB_BASE_REF: ${{ github.base_ref }}
+        CHECK_ALL_FILES: ${{ inputs.check-all-files }}
+      run: |
+        .github/scripts/file-whitelist-check.sh

.github/workflows/fuzz.yml

@@ -0,0 +1,82 @@
+name: Fuzzing Tests
+
+on:
+  schedule:
+    - cron: '0 0 * * 0'  # Weekly on Sunday at midnight
+  workflow_dispatch:  # Allow manual triggering
+  pull_request:
+    paths:
+      - 'tests/fuzz/**'
+      - 'src/local_deep_research/security/**'
+      - 'src/local_deep_research/utilities/**'
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions: {}
+
+jobs:
+  fuzz-tests:
+    name: Hypothesis Fuzz Tests
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Cache PDM dependencies
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: |
+            ~/.cache/pdm
+            .venv
+          key: pdm-${{ runner.os }}-${{ hashFiles('pdm.lock') }}
+          restore-keys: |
+            pdm-${{ runner.os }}-
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip==25.0
+          pip install pdm==2.26.2
+          # Retry pdm install up to 3 times with backoff
+          for i in 1 2 3; do
+            pdm install --dev --no-editable && break
+            echo "Attempt $i failed, retrying in $((i * 10)) seconds..."
+            sleep $((i * 10))
+          done || exit 1
+
+      - name: Run fuzz tests
+        run: |
+          pdm run pytest tests/fuzz/ -v --tb=short \
+            --hypothesis-show-statistics \
+            --hypothesis-seed=0
+        env:
+          HYPOTHESIS_PROFILE: ci
+
+      - name: Run extended fuzz tests (scheduled only)
+        if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+        run: |
+          # Run with more examples for scheduled/manual runs
+          pdm run pytest tests/fuzz/ -v --tb=short \
+            --hypothesis-show-statistics \
+            -x  # Stop on first failure for investigation
+        env:
+          HYPOTHESIS_PROFILE: extended

.github/workflows/gitleaks-main.yml

@@ -0,0 +1,55 @@
+name: Gitleaks Main Branch Scan
+
+# This workflow is specifically for the security release gate.
+# It only scans the main branch history to avoid false positives
+# from feature branches with intentional test data.
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  gitleaks-scan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+        fetch-depth: 0
+        ref: main  # Always checkout main branch
+
+    - name: Install Gitleaks
+      run: |
+        curl -sSfL https://github.com/gitleaks/gitleaks/releases/download/v8.21.2/gitleaks_8.21.2_linux_x64.tar.gz | tar -xz
+        sudo mv gitleaks /usr/local/bin/
+
+    - name: Run Gitleaks on main branch only
+      run: |
+        gitleaks detect \
+          --config=.gitleaks.toml \
+          --gitleaks-ignore-path=.gitleaksignore \
+          --log-opts="origin/main" \
+          --report-format=sarif \
+          --report-path=gitleaks-results.sarif \
+          --verbose
+      id: gitleaks
+
+    - name: Upload SARIF report
+      if: always()
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      with:
+        sarif_file: gitleaks-results.sarif

.github/workflows/gitleaks.yml

@@ -0,0 +1,46 @@
+name: Gitleaks Secret Detection
+
+on:
+  pull_request:
+    branches: [ main ]
+  workflow_dispatch:
+  schedule:
+    # Run secret scan daily at 3 AM UTC
+    - cron: '0 3 * * *'
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  gitleaks:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+        fetch-depth: 0  # Fetch full history for comprehensive secret scanning
+
+    - name: Run Gitleaks Secret Scanner
+      uses: gitleaks/gitleaks-action@e0c47f4f8be36e29cdc102c57e68cb5cbf0e8d1e # v3.0.0
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        GITHUB_USERNAME: ${{ github.actor }}
+        GITHUB_REPOSITORY: ${{ github.repository }}
+        GITLEAKS_CONFIG: .gitleaks.toml
+        GITLEAKS_BASELINE_PATH: .gitleaksignore

.github/workflows/grype.yml

@@ -0,0 +1,158 @@
+name: Grype Vulnerability Scan
+
+# Second dependency/container vulnerability scanner (complements Trivy).
+# Grype uses Anchore's curated vulnerability DB (NVD, GitHub Advisories,
+# Alpine SecDB, etc.) — a different data source from Trivy's own aggregation.
+# Running both is standard practice: they catch different CVEs.
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  grype-filesystem:
+    name: Grype Filesystem Scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Run Grype filesystem scan
+      uses: anchore/scan-action@e1165082ffb1fe366ebaf02d8526e7c4989ea9d2 # v7.4.0
+      id: grype-fs
+      with:
+        path: '.'
+        # DO NOT change to fail-build: true — findings are enforced via SARIF alerts
+        # in the release gate (check-code-scanning-alerts job in release-gate.yml).
+        # Failing here would break CI without adding security value.
+        fail-build: false
+        output-format: sarif
+        severity-cutoff: medium
+
+    # Fail loudly if Grype produced no SARIF — never fabricate an empty one.
+    # An empty-results SARIF uploaded under the grype-filesystem category would
+    # make GitHub mark every previously-open Grype alert as fixed, silently
+    # clearing real findings. Mirror the Trivy job: error + exit 1, skip upload.
+    - name: Ensure SARIF file exists
+      id: check-fs-sarif
+      if: always()
+      run: |
+        SARIF_FILE="${{ steps.grype-fs.outputs.sarif }}"
+        if [ -f "$SARIF_FILE" ]; then
+          echo "exists=true" >> "$GITHUB_OUTPUT"
+        else
+          echo "::error::Grype filesystem scan did not produce a SARIF file — scan needs to be rerun"
+          exit 1
+        fi
+
+    - name: Upload Grype filesystem SARIF to GitHub Security tab
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      if: always() && steps.check-fs-sarif.outputs.exists == 'true'
+      with:
+        sarif_file: ${{ steps.grype-fs.outputs.sarif }}
+        category: grype-filesystem
+
+  grype-container:
+    name: Grype Container Scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Free up disk space
+      run: |
+        sudo rm -rf /usr/share/dotnet
+        sudo rm -rf /usr/local/lib/android
+        sudo rm -rf /opt/ghc
+        sudo rm -rf /opt/hostedtoolcache/CodeQL
+        sudo docker image prune -af
+        df -h
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+    - name: Build Docker image for scanning
+      uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+      with:
+        context: .
+        push: false
+        load: true
+        tags: local-deep-research:grype-scan
+        cache-from: type=gha,scope=grype-scan
+        cache-to: type=gha,mode=max,scope=grype-scan
+
+    - name: Run Grype container scan
+      uses: anchore/scan-action@e1165082ffb1fe366ebaf02d8526e7c4989ea9d2 # v7.4.0
+      id: grype-container
+      with:
+        image: 'local-deep-research:grype-scan'
+        # DO NOT change to fail-build: true — findings are enforced via SARIF alerts
+        # in the release gate (check-code-scanning-alerts job in release-gate.yml).
+        # Failing here would break CI without adding security value.
+        fail-build: false
+        output-format: sarif
+        severity-cutoff: medium
+
+    # Fail loudly if Grype produced no SARIF — never fabricate an empty one.
+    # An empty-results SARIF uploaded under the grype-container category would
+    # make GitHub mark every previously-open Grype alert as fixed, silently
+    # clearing real findings. Mirror the Trivy job: error + exit 1, skip upload.
+    - name: Ensure SARIF file exists
+      id: check-container-sarif
+      if: always()
+      run: |
+        SARIF_FILE="${{ steps.grype-container.outputs.sarif }}"
+        if [ -f "$SARIF_FILE" ]; then
+          echo "exists=true" >> "$GITHUB_OUTPUT"
+        else
+          echo "::error::Grype container scan did not produce a SARIF file — scan needs to be rerun"
+          exit 1
+        fi
+
+    - name: Upload Grype container SARIF to GitHub Security tab
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      if: always() && steps.check-container-sarif.outputs.exists == 'true'
+      with:
+        sarif_file: ${{ steps.grype-container.outputs.sarif }}
+        category: grype-container
+
+    - name: Display Grype summary
+      if: always()
+      run: |
+        {
+          echo "## Grype Container Vulnerability Scan"
+          echo ""
+          if [ "${{ steps.check-container-sarif.outputs.exists }}" = "true" ]; then
+            echo "✅ Grype scan completed - Results available in Security tab"
+          else
+            echo "❌ Grype scan failed - no SARIF produced (see error above); rerun required"
+          fi
+        } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/hadolint.yml

@@ -0,0 +1,111 @@
+name: Hadolint Dockerfile Linting
+
+on:
+  pull_request:
+    paths:
+      - 'Dockerfile'
+      - '.hadolint.yaml'
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+  schedule:
+    # Run weekly on Tuesday at 9 AM UTC
+    - cron: '0 9 * * 2'
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  hadolint:
+    name: Hadolint Dockerfile Analysis
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      security-events: write
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Run Hadolint on Dockerfile
+        uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
+        with:
+          dockerfile: Dockerfile
+          format: sarif
+          output-file: hadolint.sarif
+          no-fail: false
+          verbose: true
+
+      - name: Upload Hadolint results to GitHub Security
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        if: always()
+        with:
+          sarif_file: hadolint.sarif
+          category: hadolint
+
+      - name: Upload Hadolint results as artifact
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: hadolint-results
+          path: hadolint.sarif
+          retention-days: 7  # Reduced for security
+
+      - name: Run Hadolint with detailed output
+        uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
+        if: always()
+        with:
+          dockerfile: Dockerfile
+          format: tty
+          no-fail: false
+        continue-on-error: false
+
+      - name: Display Hadolint summary
+        if: always()
+        run: |
+          {
+          echo "## Hadolint Dockerfile Analysis Summary"
+          echo ""
+          echo "✅ Hadolint analysis completed"
+          echo ""
+          echo "### What is Hadolint?"
+          echo "Hadolint is a Dockerfile linter that checks for best practices and common mistakes:"
+          echo ""
+          echo "#### Security Checks"
+          echo "- 🔒 Running as root user"
+          echo "- 🔑 Using latest tag"
+          echo "- 📦 Missing version pinning"
+          echo "- ⚠️ Exposed secrets in build args"
+          echo ""
+          echo "#### Best Practices"
+          echo "- 🎯 Layer optimization"
+          echo "- 🗑️ Cleaning package manager cache"
+          echo "- 📝 Using COPY instead of ADD"
+          echo "- 🔧 Proper WORKDIR usage"
+          echo "- 💾 Efficient layer caching"
+          echo ""
+          echo "#### Build Efficiency"
+          echo "- 🚀 Multi-stage build opportunities"
+          echo "- 📦 Unnecessary packages"
+          echo "- 🔄 Combining RUN commands"
+          echo ""
+          echo "📊 **Results:**"
+          echo "- Detailed results uploaded to GitHub Security tab"
+          echo "- SARIF file available in artifacts"
+          echo ""
+          echo "🔗 **Links:**"
+          echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
+          echo "- [Hadolint Rules](https://github.com/hadolint/hadolint#rules)"
+          } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/issue-research.yml

@@ -0,0 +1,139 @@
+name: Issue Research
+
+# When a maintainer applies the `ldr_research` label to an issue,
+# run LDR research on the issue title + body and post a comment
+# with two sections: a brief, cautious response for the reporter,
+# and adjacent context for maintainer triage.
+#
+# The same `ldr_research` label is reused from the PR workflow.
+# GitHub gates by event type (`on: issues:` here vs
+# `on: pull_request:` in e2e-research-test.yml), so they don't
+# conflict.
+
+on:
+  issues:
+    types: [labeled]
+
+permissions: {}  # Minimal top-level for OSSF Scorecard
+
+jobs:
+  build-query:
+    name: Assemble query
+    if: github.event.label.name == 'ldr_research'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    outputs:
+      query: ${{ steps.assemble.outputs.query }}
+    env:
+      ISSUE_TITLE: ${{ github.event.issue.title }}
+      ISSUE_BODY: ${{ github.event.issue.body }}
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Sanitize and assemble query
+        id: assemble
+        run: |
+          # Strip control chars (defence-in-depth against weird issue
+          # bodies) and truncate to 4000 chars before embedding in the
+          # prompt. The issue body is semi-trusted user input — never
+          # eval'd, never shelled out, only passed to the LLM as text.
+          SAFE_TITLE=$(printf '%s' "$ISSUE_TITLE" | tr -d '\000-\010\013-\037\177' | head -c 500)
+          SAFE_BODY=$(printf '%s' "$ISSUE_BODY" | tr -d '\000-\010\013-\037\177' | head -c 4000)
+
+          {
+            cat <<'PROMPT_HEAD_EOF'
+          A user filed a GitHub issue against the Local Deep Research project. Produce a comment with two distinct sections:
+
+          (1) "**For the reporter**" — a brief, cautious, 2-4 sentence summary of likely diagnostic directions, framed as suggestions and not authoritative diagnosis. If you don't have enough context to be useful, say so plainly.
+
+          (2) "**For maintainers**" — adjacent external context: similar reports in other projects, relevant upstream library documentation, known issues with the components mentioned, and related discussions. Treat the maintainer as the primary audience for the substantive research.
+
+          Issue title:
+          PROMPT_HEAD_EOF
+            printf '%s\n' "$SAFE_TITLE"
+            cat <<'PROMPT_MID_EOF'
+
+          Issue body:
+          <<<ISSUE_BODY
+          PROMPT_MID_EOF
+            printf '%s\n' "$SAFE_BODY"
+            cat <<'PROMPT_TAIL_EOF'
+          ISSUE_BODY>>>
+          PROMPT_TAIL_EOF
+          } > query.txt
+
+          # Randomized delimiter so a query containing the literal delimiter
+          # on its own line cannot prematurely terminate the heredoc.
+          DELIM="LDR_QUERY_EOF_$$_${RANDOM}_$(date +%N)"
+          {
+            echo "query<<$DELIM"
+            cat query.txt
+            echo "$DELIM"
+          } >> "$GITHUB_OUTPUT"
+
+  research:
+    name: Run LDR research
+    needs: build-query
+    # Grant the perms the reusable's job needs:
+    # - contents: read for actions/checkout
+    # - actions: write for actions/upload-artifact@v5+
+    # Without this, the reusable's `permissions: contents: read` exceeds
+    # the inherited empty permissions and GitHub rejects the workflow at
+    # load time (startup_failure with zero jobs).
+    permissions:
+      contents: read
+      actions: write
+    uses: ./.github/workflows/ldr-research-reusable.yml
+    with:
+      query: ${{ needs.build-query.outputs.query }}
+      model: ${{ vars.LDR_RESEARCH_MODEL || 'google/gemini-2.0-flash-001' }}
+      provider: ${{ vars.LDR_PROVIDER || 'openrouter' }}
+      search-tool: ${{ vars.LDR_SEARCH_TOOL || 'serper' }}
+      strategy: ${{ vars.LDR_STRATEGY || 'langgraph-agent' }}
+      comment-header: '## 🤖 LDR Research'
+      comment-subheader: '_Auto-generated context for maintainer triage and the reporter — see disclaimer below._'
+    secrets:
+      OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+      SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
+
+  post-comment:
+    name: Post issue comment
+    needs: research
+    # always() so the label-removal step runs even if research was skipped
+    # (e.g. build-query failed). The download/post steps are guarded by
+    # needs.research.outputs.success == 'true' so they self-skip in that
+    # case.
+    if: always()
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      issues: write
+    env:
+      ISSUE_NUMBER: ${{ github.event.issue.number }}
+      LABEL_NAME: ${{ github.event.label.name }}
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Download research artifact
+        if: needs.research.outputs.success == 'true'
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: ${{ needs.research.outputs.comment-artifact-name }}
+
+      - name: Post comment
+        if: needs.research.outputs.success == 'true'
+        run: |
+          gh issue comment "$ISSUE_NUMBER" --repo "${{ github.repository }}" -F comment.md
+
+      - name: Remove label for re-triggering
+        if: always()
+        run: |
+          gh issue edit "$ISSUE_NUMBER" --remove-label "$LABEL_NAME" 2>/dev/null || true

.github/workflows/journal-data-integration.yml

@@ -0,0 +1,79 @@
+name: Journal Quality Data Integration
+
+# Validates that the five external journal-quality data sources can
+# still be downloaded and the read-only reference DB can be built from
+# them end to end. Catches upstream schema breaks (OpenAlex renaming a
+# field, DOAJ changing their dump format, etc.) BEFORE we cut a release.
+#
+# This is intentionally a separate workflow rather than a job inside the
+# main test suite — it pulls ~25 MB from third-party APIs and takes
+# ~30–60 seconds, so it shouldn't run on every PR. It runs:
+#   - Weekly (Mondays 4 AM UTC) — catches drift between releases
+#   - On the release-gate workflow_call — blocks publishing if upstreams
+#     have broken our build
+#   - Manually via workflow_dispatch — for ad-hoc validation
+
+on:
+  workflow_call:  # called by release-gate.yml
+  workflow_dispatch:
+  schedule:
+    # Mondays 4 AM UTC — staggered after release-gate (2 AM) so the two
+    # don't collide if upstream rate-limits are touchy.
+    - cron: '0 4 * * 1'
+
+permissions: {}  # minimal top-level for OSSF Scorecard
+
+jobs:
+  download-and-build:
+    name: Download external sources + build reference DB
+    runs-on: ubuntu-latest
+    # 45 min job timeout: parallel downloads are bounded by the slowest
+    # source (OpenAlex Institutions, ~10 min for ~110K rows via cursor
+    # pagination). Plus build_db (~15s) + setup overhead. Generous
+    # headroom so transient API slowness doesn't false-fail the gate.
+    timeout-minutes: 45
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.11'
+
+      - name: Install system dependencies for SQLCipher
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libsqlcipher-dev
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.11'
+
+      - name: Install dependencies
+        run: pdm install --dev
+
+      - name: Run journal-quality release-gate integration test
+        # The test is marked `integration and slow` so it's skipped by
+        # the regular suite. We opt in here. The session fixture
+        # downloads all 5 sources in PARALLEL via ThreadPoolExecutor;
+        # wall-clock is bounded by the slowest source (institutions,
+        # ~10 min). Per-test pytest-timeout=2400s (40 min) overrides
+        # the global 60s default for these slow tests.
+        run: |
+          pdm run pytest tests/integration/test_journal_quality_release_gate.py \
+            -m "integration and slow" \
+            -v --tb=short \
+            --timeout=2400
+        timeout-minutes: 42

.github/workflows/labels-sync.yml

@@ -0,0 +1,46 @@
+name: Sync repo labels
+
+# Declaratively syncs labels listed in .github/labels.yml.
+# Additive only: labels not listed here are left untouched (delete-other-labels: false).
+# Lifecycle labels (needs-codeowner-review, awaiting-author, awaiting-codeowner) are
+# created by this workflow but toggled per-PR by .github/workflows/pr-triage.yml.
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - '.github/labels.yml'
+      - '.github/workflows/labels-sync.yml'
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted, matching .github/workflows/label-fixed-in-dev.yml.
+# Previous attempts (#3554, reverted #3599) showed that cancel-in-progress on label workflows
+# kills useful in-flight runs. Sync is idempotent so concurrent runs are safe.
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  sync-labels:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          sparse-checkout: .github/labels.yml
+          sparse-checkout-cone-mode: false
+
+      - name: Sync labels
+        uses: EndBug/label-sync@52074158190acb45f3077f9099fea818aa43f97a # v2.3.3
+        with:
+          config-file: .github/labels.yml
+          delete-other-labels: false

.github/workflows/ldr-research-reusable.yml

@@ -0,0 +1,300 @@
+name: LDR Research (reusable)
+
+# Reusable workflow that runs the LDR research script on a caller-supplied
+# query and produces a comment-ready markdown blob (uploaded as an
+# artifact). Callers download the artifact and post it to their target
+# (PR comment, issue comment, Reddit, etc.). This workflow does NOT post
+# anywhere — that's caller-specific.
+
+on:
+  workflow_call:
+    inputs:
+      query:
+        description: 'Fully-assembled research prompt. The caller does all prompt engineering.'
+        type: string
+        required: true
+      model:
+        description: 'OpenRouter model slug. Empty falls back to vars.LDR_RESEARCH_MODEL then a hard default.'
+        type: string
+        default: ''
+      provider:
+        description: 'LLM provider.'
+        type: string
+        default: 'openrouter'
+      search-tool:
+        description: 'Search tool.'
+        type: string
+        default: 'serper'
+      strategy:
+        description: 'LDR search strategy.'
+        type: string
+        default: 'langgraph-agent'
+      iterations:
+        description: 'Override the strategy iteration cap. 0 keeps the strategy default.'
+        type: number
+        default: 0
+      max-query-length:
+        description: 'Backstop truncation on the query string before invoking the script.'
+        type: number
+        default: 12000
+      max-sources:
+        description: 'Cap sources rendered in the markdown.'
+        type: number
+        default: 10
+      comment-header:
+        description: 'Top header line of the formatted markdown.'
+        type: string
+        default: '## LDR Research Results'
+      comment-subheader:
+        description: 'Optional second-line subheader.'
+        type: string
+        default: ''
+      comment-footer:
+        description: 'Footer line. Reddit will swap in a bot disclaimer.'
+        type: string
+        default: '_Generated by [Local Deep Research](https://github.com/LearningCircuit/local-deep-research) E2E test_'
+      include-sources-section:
+        description: 'Render a dedicated sources section. Reddit will set false to fit length caps.'
+        type: boolean
+        default: true
+      output-truncate-chars:
+        description: 'Truncate the rendered markdown to this many chars. 0 disables truncation.'
+        type: number
+        default: 0
+      runner:
+        description: 'Runner label.'
+        type: string
+        default: 'ubuntu-latest'
+      artifact-suffix:
+        description: 'Optional disambiguator appended to the artifact name. Use a unique value per matrix entry when calling this workflow multiple times in one caller run.'
+        type: string
+        default: ''
+    secrets:
+      OPENROUTER_API_KEY:
+        required: true
+      SERPER_API_KEY:
+        required: true
+    outputs:
+      comment-artifact-name:
+        description: 'Name of the artifact containing comment.md and response.json.'
+        value: ${{ jobs.research.outputs.comment-artifact-name }}
+      success:
+        description: "'true' if LDR returned valid non-error JSON."
+        value: ${{ jobs.research.outputs.success }}
+
+permissions: {}
+
+jobs:
+  research:
+    name: Run LDR research
+    runs-on: ${{ inputs.runner }}
+    permissions:
+      contents: read
+    outputs:
+      comment-artifact-name: ${{ steps.artifact-name.outputs.name }}
+      success: ${{ steps.run-script.outputs.success }}
+    env:
+      QUERY: ${{ inputs.query }}
+      MAX_QUERY_LENGTH: ${{ inputs.max-query-length }}
+      MAX_SOURCES: ${{ inputs.max-sources }}
+      COMMENT_HEADER: ${{ inputs.comment-header }}
+      COMMENT_SUBHEADER: ${{ inputs.comment-subheader }}
+      COMMENT_FOOTER: ${{ inputs.comment-footer }}
+      INCLUDE_SOURCES: ${{ inputs.include-sources-section }}
+      OUTPUT_TRUNCATE_CHARS: ${{ inputs.output-truncate-chars }}
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 1
+
+      - name: Install jq
+        run: sudo apt-get update && sudo apt-get install -y jq
+
+      - name: Compute artifact name
+        id: artifact-name
+        env:
+          SUFFIX: ${{ inputs.artifact-suffix }}
+        run: |
+          NAME="ldr-research-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}"
+          if [ -n "$SUFFIX" ]; then
+            # Sanitize: artifact names allow [A-Za-z0-9._-] only.
+            SAFE_SUFFIX=$(printf '%s' "$SUFFIX" | tr -c 'A-Za-z0-9._-' '_')
+            NAME="${NAME}-${SAFE_SUFFIX}"
+          fi
+          echo "name=$NAME" >> "$GITHUB_OUTPUT"
+
+      - name: Resolve model
+        id: resolve-model
+        env:
+          INPUT_MODEL: ${{ inputs.model }}
+          REPO_VAR_MODEL: ${{ vars.LDR_RESEARCH_MODEL }}
+        run: |
+          if [ -n "$INPUT_MODEL" ]; then
+            echo "model=$INPUT_MODEL" >> "$GITHUB_OUTPUT"
+          elif [ -n "$REPO_VAR_MODEL" ]; then
+            echo "model=$REPO_VAR_MODEL" >> "$GITHUB_OUTPUT"
+          else
+            echo "model=google/gemini-2.0-flash-001" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+          cache: true
+
+      - name: Install LDR
+        run: pdm install
+
+      - name: Write query to file (with backstop truncation)
+        run: |
+          # Truncate by total byte count, not per-line. Avoids putting the
+          # entire query on the command line where it could exceed ARG_MAX.
+          QUERY_BYTES=$(printf '%s' "$QUERY" | wc -c)
+          if [ "$QUERY_BYTES" -gt "$MAX_QUERY_LENGTH" ]; then
+            printf '%s' "$QUERY" | head -c "$MAX_QUERY_LENGTH" > query.txt
+            printf '\n... (truncated)\n' >> query.txt
+            echo "Query truncated: $QUERY_BYTES -> $MAX_QUERY_LENGTH bytes"
+          else
+            printf '%s' "$QUERY" > query.txt
+          fi
+          echo "Final query size: $(wc -c < query.txt) bytes"
+
+      - name: Run LDR Research
+        id: run-script
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+          SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
+          LDR_PROVIDER: ${{ inputs.provider }}
+          LDR_SEARCH_TOOL: ${{ inputs.search-tool }}
+          LDR_RESEARCH_MODEL: ${{ steps.resolve-model.outputs.model }}
+          LDR_STRATEGY: ${{ inputs.strategy }}
+          ITERATIONS: ${{ inputs.iterations }}
+        run: |
+          set +e
+          if [ "$ITERATIONS" -gt 0 ] 2>/dev/null; then
+            pdm run python scripts/ldr-research.py --iterations "$ITERATIONS" \
+              < query.txt 2> >(tee stderr.log >&2) > response.json
+          else
+            pdm run python scripts/ldr-research.py \
+              < query.txt 2> >(tee stderr.log >&2) > response.json
+          fi
+          LDR_EXIT_CODE=$?
+          set -e
+
+          echo "=== Response (first 2000 chars): ==="
+          head -c 2000 response.json
+          echo ""
+          echo "=== End response ==="
+
+          # Catches SIGABRT/native crashes where stdout never flushed; JSON check alone can't see this.
+          if [ "$LDR_EXIT_CODE" -ne 0 ]; then
+            LAST_ERR=$(tail -c 500 stderr.log 2>/dev/null || echo "")
+            echo "::error::ldr-research.py exited with code $LDR_EXIT_CODE: $LAST_ERR"
+            echo "success=false" >> "$GITHUB_OUTPUT"
+            exit 1
+          fi
+
+          # `jq .` exits 0 on a zero-byte file, so guard explicitly.
+          if [ ! -s response.json ]; then
+            echo "::error::response.json is empty"
+            echo "success=false" >> "$GITHUB_OUTPUT"
+            exit 1
+          fi
+
+          # Shape validation: must be a JSON object.
+          if ! jq -e 'type == "object"' response.json > /dev/null 2>&1; then
+            echo "::error::Response is not a JSON object"
+            echo "success=false" >> "$GITHUB_OUTPUT"
+            exit 1
+          fi
+
+          ERROR=$(jq -r '.error // empty' response.json)
+          if [ -n "$ERROR" ]; then
+            echo "::error::LDR error: $ERROR"
+            echo "success=false" >> "$GITHUB_OUTPUT"
+            exit 1
+          fi
+
+          # Empty .research would still produce a hollow downstream comment.
+          RESEARCH=$(jq -r '.research // empty' response.json)
+          if [ -z "$RESEARCH" ]; then
+            echo "::error::Response missing or empty .research field"
+            echo "success=false" >> "$GITHUB_OUTPUT"
+            exit 1
+          fi
+
+          echo "success=true" >> "$GITHUB_OUTPUT"
+          echo "✅ Research completed"
+
+      - name: Build comment markdown
+        run: |
+          jq -r '.research' response.json > result.md
+
+          # Optional sources section
+          if [ "$INCLUDE_SOURCES" = "true" ]; then
+            SOURCES=$(jq -r --argjson cap "$MAX_SOURCES" \
+              '.sources[:$cap][] | "- [\(.title // "Source")](\(.link // .url // ""))"' \
+              response.json 2>/dev/null || echo "")
+          else
+            SOURCES=""
+          fi
+
+          {
+            echo "$COMMENT_HEADER"
+            echo ""
+            if [ -n "$COMMENT_SUBHEADER" ]; then
+              echo "$COMMENT_SUBHEADER"
+              echo ""
+            fi
+            cat result.md
+            echo ""
+            if [ -n "$SOURCES" ]; then
+              echo "### 🔍 Search Sources"
+              echo ""
+              echo "$SOURCES"
+              echo ""
+            fi
+            echo "---"
+            echo "$COMMENT_FOOTER"
+          } > comment.md
+
+          # Optional truncation (Reddit will use this)
+          if [ "$OUTPUT_TRUNCATE_CHARS" -gt 0 ] 2>/dev/null; then
+            CURRENT=$(wc -c < comment.md)
+            if [ "$CURRENT" -gt "$OUTPUT_TRUNCATE_CHARS" ]; then
+              # Truncate at last paragraph break before the limit, with a budget
+              # for the trailing marker.
+              MARKER='…[truncated]'
+              BUDGET=$((OUTPUT_TRUNCATE_CHARS - ${#MARKER} - 1))
+              head -c "$BUDGET" comment.md > comment.md.tmp
+              # Trim trailing partial line so we end on a clean paragraph
+              awk 'BEGIN{RS=""} {gsub(/[[:space:]]+$/,""); print}' \
+                comment.md.tmp > comment.md
+              echo "" >> comment.md
+              echo "$MARKER" >> comment.md
+              rm -f comment.md.tmp
+              echo "Truncated comment.md to $OUTPUT_TRUNCATE_CHARS chars."
+            fi
+          fi
+
+          echo "Final comment.md size: $(wc -c < comment.md) bytes"
+
+      - name: Upload research artifact
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ${{ steps.artifact-name.outputs.name }}
+          path: |
+            comment.md
+            response.json
+            stderr.log
+          if-no-files-found: ignore
+          retention-days: 7

.github/workflows/mcp-tests.yml

@@ -0,0 +1,115 @@
+name: MCP Server Tests
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - 'src/local_deep_research/mcp/**'
+      - 'tests/mcp/**'
+      - 'scripts/mcp_smoke_test.sh'
+      - '.github/workflows/mcp-tests.yml'
+  pull_request:
+    branches: [ main ]
+    paths:
+      - 'src/local_deep_research/mcp/**'
+      - 'tests/mcp/**'
+      - 'scripts/mcp_smoke_test.sh'
+      - '.github/workflows/mcp-tests.yml'
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions:
+  contents: read
+
+jobs:
+  mcp-tests:
+    name: MCP Server Tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install pip==25.0
+          pip install pdm==2.26.2
+          pdm install --dev --no-editable -G mcp
+
+      - name: Check if MCP server module exists
+        id: check-mcp
+        run: |
+          if [ -f "src/local_deep_research/mcp/server.py" ]; then
+            echo "mcp_exists=true" >> "$GITHUB_OUTPUT"
+            echo "MCP server module found, will run tests"
+          else
+            echo "mcp_exists=false" >> "$GITHUB_OUTPUT"
+            echo "MCP server module not found (feature not yet merged), skipping tests"
+          fi
+
+      - name: Run MCP smoke tests
+        if: steps.check-mcp.outputs.mcp_exists == 'true'
+        run: pdm run bash scripts/mcp_smoke_test.sh
+
+      - name: Run MCP unit tests
+        if: steps.check-mcp.outputs.mcp_exists == 'true'
+        run: |
+          pdm run pytest tests/mcp/ -v --tb=short -n auto
+        env:
+          LDR_TESTING_WITH_MOCKS: "true"
+
+      - name: Skip notice
+        if: steps.check-mcp.outputs.mcp_exists != 'true'
+        run: |
+          echo "::notice::MCP server module not yet implemented. Tests will run once feature/mcp-server is merged."
+
+      - name: Generate test summary
+        if: always()
+        env:
+          MCP_EXISTS: ${{ steps.check-mcp.outputs.mcp_exists }}
+        run: |
+          {
+            echo "## MCP Server Test Summary"
+            echo ""
+            if [ "$MCP_EXISTS" != "true" ]; then
+              echo "### ⏭️ Tests Skipped"
+              echo ""
+              echo "MCP server module (\`src/local_deep_research/mcp/server.py\`) not yet implemented."
+              echo "Tests will run automatically once the MCP feature branch is merged."
+            else
+              echo "### What was tested:"
+              echo "- 🔌 MCP server module loading"
+              echo "- 🔧 Discovery tools (list_strategies, list_search_engines, get_configuration)"
+              echo "- 🧪 Unit tests for all MCP tools"
+              echo "- 🚀 Server startup verification"
+              echo ""
+              echo "### MCP Tools Tested:"
+              echo "| Tool | Description |"
+              echo "|------|-------------|"
+              echo "| \`quick_research\` | Fast research summary (1-5 min) |"
+              echo "| \`detailed_research\` | Comprehensive analysis (5-15 min) |"
+              echo "| \`generate_report\` | Full markdown report (10-30 min) |"
+              echo "| \`analyze_documents\` | Search local collections |"
+              echo "| \`list_search_engines\` | List available search engines |"
+              echo "| \`list_strategies\` | List research strategies |"
+              echo "| \`get_configuration\` | Get current config |"
+            fi
+          } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/mypy-type-check.yml

@@ -0,0 +1,81 @@
+name: Mypy Type Checking
+
+on:
+  pull_request:
+    branches: [ main ]
+  workflow_call:  # Called by ci-gate.yml for release pipeline
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions:
+  contents: read
+
+jobs:
+  mypy-analysis:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout repository
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        persist-credentials: false
+
+    - name: Set up Python
+      uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+      with:
+        python-version: '3.12'
+
+    - name: Install system dependencies for SQLCipher
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y libsqlcipher-dev
+
+    - name: Install PDM
+      run: |
+        pip install pdm==2.26.2
+
+    - name: Cache PDM dependencies
+      uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      with:
+        path: |
+          ~/.cache/pdm
+          .venv
+        key: pdm-${{ runner.os }}-${{ hashFiles('pdm.lock') }}
+        restore-keys: |
+          pdm-${{ runner.os }}-
+
+    - name: Install project dependencies
+      run: |
+        pdm sync -d
+
+    - name: Run mypy type checking
+      run: |
+        pdm run mypy src/ --config-file=pyproject.toml --show-error-codes --pretty
+
+    - name: Display mypy summary
+      if: always()
+      run: |
+        {
+          echo "## Mypy Type Checking Results"
+          echo ""
+          echo "Type checking analysis completed."
+          echo ""
+          echo "### Benefits"
+          echo "- Catches type errors before runtime"
+          echo "- Improves code quality and maintainability"
+          echo "- Provides better IDE autocomplete"
+          echo "- Detects uninitialized variables and type mismatches"
+          echo ""
+          echo "See full output above for any type errors found."
+        } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/npm-audit.yml

@@ -0,0 +1,81 @@
+name: npm Security Audit
+
+on:
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+
+permissions:
+  contents: read
+
+jobs:
+  npm-audit:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Setup Node.js
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: '24'
+
+      - name: Run npm audit on all package.json locations
+        env:
+          # Advisories with NO upstream fix, accepted ONLY because they live in
+          # dev/test-only tooling that never ships to users. Keep tight + justified:
+          #   GHSA-h67p-54hq-rp68 — js-yaml 3.x DoS, pulled transitively by
+          #   @lhci/utils (accessibility_tests) & @istanbuljs/load-nyc-config
+          #   (infrastructure_tests); both pin js-yaml ^3, no patched 3.x exists.
+          #   Dev-only, parses trusted local config. (dismissed alerts #97/#7893)
+          AUDIT_ALLOWLIST: "GHSA-h67p-54hq-rp68"
+        run: |
+          FAILED=false
+          FILTER="${GITHUB_WORKSPACE}/.github/scripts/npm_audit_allowlist.py"
+          # Audit root and every test directory that contains a package.json
+          DIRS=(
+            .
+            tests
+            tests/ui_tests
+            tests/ui_tests/playwright
+            tests/accessibility_tests
+            tests/api_tests_with_login
+            tests/infrastructure_tests
+            tests/puppeteer
+          )
+          for dir in "${DIRS[@]}"; do
+            echo "=== Running npm audit on ${dir} ==="
+            if [ ! -f "${dir}/package.json" ]; then
+              echo "No package.json found in ${dir}"
+              continue
+            fi
+            # Require committed lockfile for reproducible security audits
+            if [ ! -f "${dir}/package-lock.json" ]; then
+              echo "::error::Missing ${dir}/package-lock.json. Please commit your lockfile for security audits."
+              FAILED=true
+              continue
+            fi
+            # Audit, then filter out allowlisted (no-fix, dev-only) advisories.
+            # `|| true` is on the assignment (not inside $()) so npm's non-zero
+            # exit is swallowed without the SC2015 `A && B || C` idiom; an
+            # empty/error capture still fails safe in the filter.
+            audit_json="$(cd "${dir}" && npm audit --audit-level=moderate --json 2>/dev/null)" || true
+            if ! printf '%s' "${audit_json}" | python3 "${FILTER}"; then
+              FAILED=true
+            fi
+          done
+          if [[ "$FAILED" == "true" ]]; then
+            echo "❌ npm audit found non-allowlisted moderate or higher severity vulnerabilities"
+            echo "Fix: bump the dep / 'npm audit fix'; or if no fix exists and it is dev-only, add the advisory to AUDIT_ALLOWLIST with justification."
+            exit 1
+          else
+            echo "✅ No non-allowlisted moderate or higher severity vulnerabilities found"
+          fi

.github/workflows/nuclei.yml

@@ -0,0 +1,250 @@
+name: Nuclei Vulnerability Scan
+
+# Template-driven DAST scanner — complements ZAP with known-CVE checks,
+# misconfiguration detection, exposed panel discovery, and default credential
+# testing. Uses ProjectDiscovery's 6,500+ community-maintained templates.
+#
+# Authenticated scan: the runner pre-creates the standard CI test_admin user,
+# logs in via the real /auth/login flow to capture a session cookie, and
+# seeds Nuclei with the full Flask url_map so the scanner probes the entire
+# authenticated app surface (settings, research, history, API, …) — not just
+# the unauthenticated landing page.
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+permissions: {}
+
+jobs:
+  nuclei-scan:
+    name: Nuclei DAST Scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    env:
+      CI: true
+      TEST_ENV: true
+      FLASK_ENV: testing
+      LDR_DATA_DIR: ${{ github.workspace }}/data
+      LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
+      # Honour the fast test KDF: without LDR_TEST_MODE the value is clamped
+      # to the production minimum (256000), making every SQLCipher open/backup
+      # ~256x slower — see #4430 and tests/shared/chrome_profile.js context.
+      LDR_TEST_MODE: "1"
+      LDR_DISABLE_RATE_LIMITING: "true"
+      SECRET_KEY: test-secret-key-for-ci
+      TEST_USERNAME: test_admin
+      TEST_PASSWORD: testpass123
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Install system dependencies for SQLCipher
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libsqlcipher-dev
+
+      - name: Install dependencies
+        run: |
+          pdm sync -d
+
+      - name: Setup test data directory
+        run: |
+          mkdir -p "$LDR_DATA_DIR/encrypted_databases"
+
+      - name: Pre-create CI test user
+        # Avoids the slow registration path (KDF + 500 settings rows) and
+        # avoids hitting the registration rate limit during the scan window.
+        run: |
+          export PYTHONPATH=$PWD/src:$PYTHONPATH
+          pdm run python scripts/ci/init_test_database.py
+
+      - name: Start LDR server for testing
+        run: |
+          export PYTHONPATH=$PWD/src:$PYTHONPATH
+          pdm run python -m local_deep_research.web.app > server.log 2>&1 &
+          SERVER_PID=$!
+          echo "$SERVER_PID" > server.pid
+
+          # Wait for server to start.
+          # --connect-timeout/--max-time bound TCP and total request time so a
+          # hung connection fails fast instead of eating the job timeout.
+          for _ in {1..90}; do
+            if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
+              echo "Server started successfully"
+              break
+            fi
+            sleep 1
+          done
+
+          # Verify server is running
+          if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
+            echo "Server failed to start"
+            cat server.log
+            exit 1
+          fi
+
+      - name: Dump Flask url_map for URL seeding
+        # Without -list, Nuclei only probes the single -target URL. Seeding
+        # with the real url_map lets it exercise every blueprint route.
+        run: |
+          export PYTHONPATH=$PWD/src:$PYTHONPATH
+          pdm run python scripts/ci/dump_url_map.py http://127.0.0.1:5000 > urls.txt
+          echo "Seeded $(wc -l < urls.txt) URLs:"
+          head -20 urls.txt
+          echo "..."
+
+      - name: Authenticate and capture session cookie
+        id: login
+        run: |
+          # Step 1: GET /auth/login → establishes a Flask session cookie and
+          # returns the HTML form with the per-session CSRF token.
+          curl -sS -c cookies.txt -o login.html http://127.0.0.1:5000/auth/login
+
+          CSRF=$(grep -oE 'name="csrf_token"[[:space:]]+value="[^"]+"' login.html \
+            | head -n1 \
+            | sed -E 's/.*value="([^"]+)".*/\1/')
+          if [ -z "$CSRF" ]; then
+            echo "Failed to extract CSRF token from login page"
+            head -200 login.html
+            exit 1
+          fi
+
+          # Step 2: POST credentials with the CSRF token and the cookie jar.
+          # -L follows the post-login 302 to /. -o /dev/null discards body,
+          # -w prints the final HTTP code so a non-2xx fails the step.
+          HTTP_CODE=$(curl -sS -L \
+            -b cookies.txt -c cookies.txt \
+            -o /dev/null -w '%{http_code}' \
+            --data-urlencode "username=${TEST_USERNAME}" \
+            --data-urlencode "password=${TEST_PASSWORD}" \
+            --data-urlencode "csrf_token=${CSRF}" \
+            http://127.0.0.1:5000/auth/login)
+          if [ "$HTTP_CODE" != "200" ]; then
+            echo "Login failed with HTTP $HTTP_CODE"
+            cat server.log
+            exit 1
+          fi
+
+          # Verify the session is actually authenticated (not just that
+          # the login page rendered with a 200 from a re-display of errors).
+          AUTHED=$(curl -sS -b cookies.txt http://127.0.0.1:5000/auth/check)
+          echo "auth/check response: $AUTHED"
+          echo "$AUTHED" | grep -q '"authenticated":[[:space:]]*true' || {
+            echo "Authenticated check failed"
+            exit 1
+          }
+
+          # Extract the session cookie value from the Netscape cookie jar.
+          # Format: domain TAB tailmatch TAB path TAB secure TAB expires TAB name TAB value
+          SESSION=$(awk '$6 == "session" { print $7 }' cookies.txt | tail -n1)
+          if [ -z "$SESSION" ]; then
+            # Print every column EXCEPT the value ($7) so we can debug
+            # without leaking the cookie if the awk filter is broken.
+            echo "Could not find 'session' cookie in jar (values redacted):"
+            awk '{ print $1, $2, $3, $4, $5, $6 }' cookies.txt
+            exit 1
+          fi
+          # Mask the cookie in logs — it grants full app access for this run.
+          echo "::add-mask::$SESSION"
+          echo "session=$SESSION" >> "$GITHUB_OUTPUT"
+
+          # Give the post-login background thread a moment to finish
+          # the settings-migration / library-init pass it kicks off
+          # (see _perform_post_login_tasks in web/auth/routes.py).
+          # Otherwise the first authenticated probes can race those
+          # writes and 500 on settings-dependent routes.
+          sleep 2
+
+      - name: Create empty SARIF fallback
+        run: |
+          cat > nuclei.sarif << 'SARIF'
+          {
+            "version": "2.1.0",
+            "$schema": "https://json.schemastore.org/sarif-2.1.0.json",
+            "runs": [{
+              "tool": {
+                "driver": {
+                  "name": "Nuclei",
+                  "informationUri": "https://github.com/projectdiscovery/nuclei",
+                  "rules": []
+                }
+              },
+              "results": []
+            }]
+          }
+          SARIF
+
+      - name: Run Nuclei scan
+        uses: projectdiscovery/nuclei-action@cc153d0541e1adf8a42bbe31c0a4fb2376147538 # v3.1.1
+        with:
+          # -list: probe every route in the Flask url_map (not just /).
+          # -H: attach the authenticated session cookie so probes hit the
+          #     real app surface, not the login redirect.
+          # -severity: drop info-level noise (form-detection, options-method,
+          #     intentional CSP/cookie choices). Real findings are >= low.
+          # -etags intrusive,dos,fuzz: with a live session, default templates
+          #     can mutate state or DoS the runner. Exclude the standard
+          #     destructive tag set for authenticated DAST.
+          # -eid http-missing-security-headers: HSTS is correctly omitted on
+          #     plain HTTP localhost; X-XSS-Protection is intentionally
+          #     omitted (deprecated, replaced by CSP). See
+          #     security/security_headers.py.
+          args: >-
+            -list urls.txt
+            -H "Cookie: session=${{ steps.login.outputs.session }}"
+            -severity low,medium,high,critical
+            -etags intrusive,dos,fuzz
+            -eid http-missing-security-headers
+            -sarif-export nuclei.sarif
+            -output nuclei.log
+
+      - name: Upload Nuclei SARIF to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        if: always()
+        with:
+          sarif_file: nuclei.sarif
+          category: nuclei-dast
+
+      - name: Upload Nuclei scan artifacts
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: nuclei-scan-results
+          path: |
+            nuclei.log
+            nuclei.sarif
+            urls.txt
+            server.log
+          retention-days: 30
+
+      - name: Stop server
+        if: always()
+        run: |
+          if [ -f server.pid ]; then
+            kill "$(cat server.pid)" || true
+            rm server.pid
+          fi

.github/workflows/ossf-scorecard.yml

@@ -0,0 +1,62 @@
+name: OSSF Scorecard
+
+on:
+  # Run on branches (for analysis purposes)
+  branch_protection_rule:
+  # Run on schedule
+  schedule:
+    # Run weekly on Monday at 8 AM UTC
+    - cron: '0 8 * * 1'
+  # Allow manual runs
+  workflow_dispatch:
+  # Run on push to default branch
+  push:
+    branches: [ main ]
+
+# Declare default permissions as read only.
+permissions: read-all
+
+jobs:
+  analysis:
+    name: OSSF Security Scorecard Analysis
+    runs-on: ubuntu-latest
+    permissions:
+      # Needed to upload the results to code-scanning dashboard.
+      security-events: write
+      # Needed to publish results and get a badge
+      id-token: write
+      # Needed for private repositories
+      contents: read
+      actions: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Run OSSF Scorecard analysis
+        uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a # v2.4.3
+        with:
+          results_file: results.sarif
+          results_format: sarif
+          # Publish results to enable scorecard badges
+          publish_results: true
+
+      - name: Upload OSSF Scorecard results to GitHub Security
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        with:
+          sarif_file: results.sarif
+          category: ossf-scorecard
+
+      - name: Upload OSSF Scorecard results as artifact
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ossf-scorecard-results
+          path: results.sarif
+          retention-days: 7  # Reduced for security

.github/workflows/osv-scanner-scheduled.yml

@@ -0,0 +1,35 @@
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+# Scheduled OSV-Scanner scanning for vulnerabilities.
+# Runs on push to main and weekly schedule.
+# Split from osv-scanner.yml to prevent "skipped" noise on PRs.
+#
+# For more examples and options, including how to ignore specific vulnerabilities,
+# see https://google.github.io/osv-scanner/github-action/
+
+name: OSV-Scanner (Scheduled)
+
+on:
+  push:
+    branches: [ "main" ]  # Create baseline for Code Scanning comparison
+  schedule:
+    - cron: '41 21 * * 1'  # Weekly on Monday at 21:41 UTC
+  workflow_dispatch:
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  scan-scheduled:
+    permissions:
+      security-events: write
+      contents: read
+      actions: read
+    uses: "google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2" # v2.3.8
+    with:
+      scan-args: |-
+        -r
+        --skip-git
+        ./

.github/workflows/osv-scanner.yml

@@ -0,0 +1,48 @@
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+# OSV-Scanner for detecting vulnerabilities in dependencies.
+# Runs on PRs to catch vulnerable dependencies before merge (shift-left security).
+#
+# NOTE: This workflow is NOT included in release-gate.yml because
+# GitHub Actions limits reusable workflow nesting to 2 levels. Since this
+# workflow calls google/osv-scanner-reusable.yml, including it in the gate
+# would create 4 levels of nesting and cause startup_failure:
+#   release.yml → release-gate.yml → osv-scanner.yml → google/osv-scanner-reusable.yml
+#
+# For more examples and options, including how to ignore specific vulnerabilities,
+# see https://google.github.io/osv-scanner/github-action/
+
+name: OSV-Scanner
+
+on:
+  pull_request:
+    branches: [ "main" ]
+  merge_group:
+    branches: [ "main" ]
+  schedule:
+    - cron: '39 12 * * 1'  # Weekly scan for newly disclosed CVEs
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results.
+# Future iteration could safely add concurrency for scheduled/push-only
+# triggers (where head_ref is empty and runs get unique groups).
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  scan:
+    permissions:
+      security-events: write
+      contents: read
+      actions: read
+    uses: "google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2" # v2.3.8
+    with:
+      scan-args: |-
+        -r
+        --skip-git
+        ./

.github/workflows/owasp-zap-scan.yml

@@ -0,0 +1,212 @@
+name: OWASP ZAP Security Scan
+
+# Baseline and API scans - called by release-gate.yml
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+# Restrict top-level permissions (jobs define their own)
+permissions: {}
+
+jobs:
+  zap-baseline-scan:
+    name: ZAP Baseline Scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    permissions:
+      contents: read
+      security-events: write
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Install system dependencies for SQLCipher
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libsqlcipher-dev
+
+      - name: Install dependencies
+        run: |
+          pdm sync -d
+
+      - name: Start LDR server for testing
+        run: |
+          export CI=true
+          export LDR_DB_CONFIG_KDF_ITERATIONS=1000
+          # Without LDR_TEST_MODE the 1000 is clamped to the production
+          # minimum (256000), making SQLCipher ~256x slower — see #4430.
+          export LDR_TEST_MODE=1
+          export PYTHONPATH=$PWD/src:$PYTHONPATH
+          pdm run python -m local_deep_research.web.app > server.log 2>&1 &
+          SERVER_PID=$!
+          echo "$SERVER_PID" > server.pid
+
+          # Wait for server to start.
+          # --connect-timeout/--max-time bound TCP and total request time so a
+          # hung connection fails fast instead of eating the job timeout.
+          for _ in {1..90}; do
+            if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
+              echo "Server started successfully"
+              break
+            fi
+            sleep 1
+          done
+
+          # Verify server is running
+          if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
+            echo "Server failed to start"
+            cat server.log
+            exit 1
+          fi
+
+      - name: Run ZAP Baseline Scan
+        uses: zaproxy/action-baseline@de8ad967d3548d44ef623df22cf95c3b0baf8b25 # v0.15.0
+        with:
+          target: 'http://localhost:5000'
+          rules_file_name: '.zap/rules.tsv'
+          cmd_options: '-a -I'  # -I: only fail on FAIL-level alerts per rules.tsv (ignore WARN/INFO)
+          allow_issue_writing: false
+          fail_action: true
+          artifact_name: zapbaseline
+
+      - name: Upload ZAP scan results
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: zap-baseline-scan-results
+          path: |
+            report_html.html
+            report_json.json
+            report_md.md
+          retention-days: 30
+
+      - name: Stop server
+        if: always()
+        run: |
+          if [ -f server.pid ]; then
+            kill "$(cat server.pid)" || true
+            rm server.pid
+          fi
+
+  zap-api-scan:
+    name: ZAP API Scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    permissions:
+      contents: read
+      security-events: write
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Install system dependencies for SQLCipher
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libsqlcipher-dev
+
+      - name: Install dependencies
+        run: |
+          pdm sync -d
+
+      - name: Start LDR server for testing
+        run: |
+          export CI=true
+          export LDR_DB_CONFIG_KDF_ITERATIONS=1000
+          # Without LDR_TEST_MODE the 1000 is clamped to the production
+          # minimum (256000), making SQLCipher ~256x slower — see #4430.
+          export LDR_TEST_MODE=1
+          export PYTHONPATH=$PWD/src:$PYTHONPATH
+          pdm run python -m local_deep_research.web.app > server.log 2>&1 &
+          SERVER_PID=$!
+          echo "$SERVER_PID" > server.pid
+
+          # Wait for server to start.
+          # --connect-timeout/--max-time bound TCP and total request time so a
+          # hung connection fails fast instead of eating the job timeout.
+          for _ in {1..90}; do
+            if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
+              echo "Server started successfully"
+              break
+            fi
+            sleep 1
+          done
+
+          # Verify server is running
+          if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
+            echo "Server failed to start"
+            cat server.log
+            exit 1
+          fi
+
+      - name: Generate OpenAPI specification
+        run: |
+          # If you have an OpenAPI spec, use it. Otherwise, ZAP will discover endpoints
+          # For now, we'll let ZAP discover API endpoints automatically
+          echo "Using automatic API endpoint discovery"
+
+      - name: Run ZAP API Scan
+        uses: zaproxy/action-api-scan@5158fe4d9d8fcc75ea204db81317cce7f9e5453d # v0.10.0
+        with:
+          target: 'http://localhost:5000/api/v1'
+          format: 'openapi'
+          rules_file_name: '.zap/rules.tsv'
+          cmd_options: '-a -I'  # -I: only fail on FAIL-level alerts per rules.tsv (ignore WARN/INFO)
+          allow_issue_writing: false
+          fail_action: true
+          artifact_name: zapapi
+
+      - name: Upload ZAP API scan results
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: zap-api-scan-results
+          path: |
+            report_html.html
+            report_json.json
+            report_md.md
+          retention-days: 30
+
+      - name: Stop server
+        if: always()
+        run: |
+          if [ -f server.pid ]; then
+            kill "$(cat server.pid)" || true
+            rm server.pid
+          fi

.github/workflows/playwright-webkit-tests.yml

@@ -0,0 +1,351 @@
+name: Playwright WebKit Tests
+
+on:
+  workflow_call:      # Called by release pipeline
+  workflow_dispatch:  # Manual trigger
+  schedule:
+    - cron: '0 2 * * *'  # Daily at 2 AM UTC
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  # Shared setup: start server, register user, then run Desktop Safari tests
+  desktop-safari:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Set up PDM
+      uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+      with:
+        python-version: '3.12'
+        version: 2.26.2
+        cache: true
+
+    - name: Install Python dependencies
+      run: |
+        pdm install -d
+
+    - name: Setup Node.js
+      uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+      with:
+        node-version: '24'
+        cache: 'npm'
+        cache-dependency-path: |
+          package-lock.json
+          tests/ui_tests/playwright/package-lock.json
+
+    - name: Build Vite frontend assets
+      run: |
+        npm ci
+        npm run build
+
+    - name: Install Playwright dependencies (npm)
+      working-directory: tests/ui_tests/playwright
+      run: npm ci
+
+    # ``playwright install --with-deps`` downloads ~150 MB of browser
+    # binaries from the playwright CDN per attempt. Single CDN
+    # hiccups have failed the whole release pipeline in the past
+    # (e.g. transient 5xx mid-download); a single retry costs little
+    # and removes the dominant transient-network failure mode for
+    # this workflow. Two attempts only — three attempts of a 150 MB
+    # download starts hiding real outages.
+    - name: Install Playwright browsers
+      uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
+      with:
+        timeout_minutes: 10
+        max_attempts: 2
+        retry_on: error
+        shell: bash
+        # Install chromium for auth setup step + webkit for Safari tests
+        command: |
+          cd tests/ui_tests/playwright
+          npx playwright install chromium webkit --with-deps
+
+    - name: Setup test directories
+      run: |
+        mkdir -p data/encrypted_databases
+        echo "Created data directories for tests"
+
+    - name: Start LDR server
+      env:
+        CI: true
+        TEST_ENV: true
+        FLASK_ENV: testing
+        LDR_DATA_DIR: ${{ github.workspace }}/data
+        # The server MUST open test_admin's encrypted DB with the same KDF it
+        # was created with. init_test_database.py creates it at 1000 iterations
+        # (LDR_TEST_MODE relaxes the floor to allow it); without these two vars
+        # here the requested 1000 is below the production floor, so it falls
+        # back to the 256000 default, the server derives the wrong key, and
+        # every login fails with a 401 — see #4558, which set these on the init
+        # step but missed this server step.
+        LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
+        LDR_TEST_MODE: "1"
+        LDR_DISABLE_RATE_LIMITING: true
+        SECRET_KEY: test-secret-key-for-ci
+      run: |
+        export PYTHONPATH="$PWD/src:$PYTHONPATH"
+        echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
+        pdm run python -m local_deep_research.web.app &
+        SERVER_PID=$!
+        echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
+
+        # Wait for server to be ready.
+        # --connect-timeout/--max-time bound TCP and total request time so a
+        # hung connection fails fast instead of eating the job timeout.
+        for i in {1..30}; do
+          if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
+            echo "Server is ready"
+            break
+          fi
+          echo "Waiting for server... ($i/30)"
+          sleep 2
+        done
+
+        # Verify server is running
+        if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
+          echo "Server failed to start"
+          exit 1
+        fi
+
+    - name: Initialize test database
+      env:
+        TEST_ENV: true
+        LDR_DATA_DIR: ${{ github.workspace }}/data
+        LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
+        # Without LDR_TEST_MODE the 1000 is clamped to the production
+        # minimum (256000), making SQLCipher ~256x slower — see #4430.
+        LDR_TEST_MODE: "1"
+      run: |
+        export PYTHONPATH="$PWD/src:$PYTHONPATH"
+        pdm run python scripts/ci/init_test_database.py
+
+    - name: Register CI test user
+      working-directory: tests/ui_tests
+      run: |
+        npm ci
+        node register_ci_user.js http://127.0.0.1:5000
+
+    - name: Run Desktop Safari Tests
+      working-directory: tests/ui_tests/playwright
+      env:
+        CI: true
+        TEST_BASE_URL: http://127.0.0.1:5000
+        TEST_USERNAME: test_admin
+        TEST_PASSWORD: testpass123
+      # Safari CI runs a critical subset for fast release gate:
+      # - all-pages-mobile: overflow, touch targets, nav clearance across all pages
+      # - auth-pages-mobile: login, register, change-password mobile layouts
+      # - desktop-layout: sidebar positioning and breakpoint testing
+      # - interactive-states: advanced options, search, help panels, mobile menus
+      # - settings-subpages-mobile: settings tab navigation and form layouts
+      # - mobile-ui-audit: comprehensive mobile UI quality checks
+      # - embedding-settings-dropdown: regression for #3863 (model dropdown reset)
+      # - settings-mobile-collapse: sections collapse by default on mobile (#4032),
+      #   stay expanded on desktop, and search force-expands survivors
+      # Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
+      # When adding Safari-critical tests, update this filter pattern.
+      run: |
+        npx playwright test \
+          --project="Desktop Safari" \
+          "all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit|embedding-settings-dropdown|settings-mobile-collapse"
+
+    - name: Upload Playwright Report
+      if: always()
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      with:
+        name: playwright-report-desktop-safari
+        path: |
+          tests/ui_tests/playwright/playwright-report/
+          tests/ui_tests/playwright/test-results/
+
+    - name: Stop server
+      if: always()
+      run: |
+        if [ -n "$SERVER_PID" ]; then
+          kill "$SERVER_PID" || true
+        fi
+        pkill -f "python -m local_deep_research.web.app" || true
+
+  # Mobile Safari tests run in parallel on a separate runner
+  mobile-safari:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Set up PDM
+      uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+      with:
+        python-version: '3.12'
+        version: 2.26.2
+        cache: true
+
+    - name: Install Python dependencies
+      run: |
+        pdm install -d
+
+    - name: Setup Node.js
+      uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+      with:
+        node-version: '24'
+        cache: 'npm'
+        cache-dependency-path: |
+          package-lock.json
+          tests/ui_tests/playwright/package-lock.json
+
+    - name: Build Vite frontend assets
+      run: |
+        npm ci
+        npm run build
+
+    - name: Install Playwright dependencies (npm)
+      working-directory: tests/ui_tests/playwright
+      run: npm ci
+
+    # ``playwright install --with-deps`` downloads ~150 MB of browser
+    # binaries from the playwright CDN per attempt. Single CDN
+    # hiccups have failed the whole release pipeline in the past
+    # (e.g. transient 5xx mid-download); a single retry costs little
+    # and removes the dominant transient-network failure mode for
+    # this workflow. Two attempts only — three attempts of a 150 MB
+    # download starts hiding real outages.
+    - name: Install Playwright browsers
+      uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
+      with:
+        timeout_minutes: 10
+        max_attempts: 2
+        retry_on: error
+        shell: bash
+        # Install chromium for auth setup step + webkit for Safari tests
+        command: |
+          cd tests/ui_tests/playwright
+          npx playwright install chromium webkit --with-deps
+
+    - name: Setup test directories
+      run: |
+        mkdir -p data/encrypted_databases
+        echo "Created data directories for tests"
+
+    - name: Start LDR server
+      env:
+        CI: true
+        TEST_ENV: true
+        FLASK_ENV: testing
+        LDR_DATA_DIR: ${{ github.workspace }}/data
+        # The server MUST open test_admin's encrypted DB with the same KDF it
+        # was created with. init_test_database.py creates it at 1000 iterations
+        # (LDR_TEST_MODE relaxes the floor to allow it); without these two vars
+        # here the requested 1000 is below the production floor, so it falls
+        # back to the 256000 default, the server derives the wrong key, and
+        # every login fails with a 401 — see #4558, which set these on the init
+        # step but missed this server step.
+        LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
+        LDR_TEST_MODE: "1"
+        LDR_DISABLE_RATE_LIMITING: true
+        SECRET_KEY: test-secret-key-for-ci
+      run: |
+        export PYTHONPATH="$PWD/src:$PYTHONPATH"
+        echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
+        pdm run python -m local_deep_research.web.app &
+        SERVER_PID=$!
+        echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
+
+        # Wait for server to be ready.
+        # --connect-timeout/--max-time bound TCP and total request time so a
+        # hung connection fails fast instead of eating the job timeout.
+        for i in {1..30}; do
+          if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
+            echo "Server is ready"
+            break
+          fi
+          echo "Waiting for server... ($i/30)"
+          sleep 2
+        done
+
+        # Verify server is running
+        if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
+          echo "Server failed to start"
+          exit 1
+        fi
+
+    - name: Initialize test database
+      env:
+        TEST_ENV: true
+        LDR_DATA_DIR: ${{ github.workspace }}/data
+        LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
+        # Without LDR_TEST_MODE the 1000 is clamped to the production
+        # minimum (256000), making SQLCipher ~256x slower — see #4430.
+        LDR_TEST_MODE: "1"
+      run: |
+        export PYTHONPATH="$PWD/src:$PYTHONPATH"
+        pdm run python scripts/ci/init_test_database.py
+
+    - name: Register CI test user
+      working-directory: tests/ui_tests
+      run: |
+        npm ci
+        node register_ci_user.js http://127.0.0.1:5000
+
+    - name: Run Mobile Safari Tests
+      working-directory: tests/ui_tests/playwright
+      env:
+        CI: true
+        TEST_BASE_URL: http://127.0.0.1:5000
+        TEST_USERNAME: test_admin
+        TEST_PASSWORD: testpass123
+      # Safari CI runs a critical subset for fast release gate:
+      # - all-pages-mobile: overflow, touch targets, nav clearance across all pages
+      # - auth-pages-mobile: login, register, change-password mobile layouts
+      # - desktop-layout: sidebar positioning and breakpoint testing
+      # - interactive-states: advanced options, search, help panels, mobile menus
+      # - settings-subpages-mobile: settings tab navigation and form layouts
+      # - mobile-ui-audit: comprehensive mobile UI quality checks
+      # - settings-mobile-collapse: sections collapse by default on mobile (#4032);
+      #   this is the viewport where the >16384px overflow regression bites
+      # Note: embedding-settings-dropdown is desktop-only (skips on isMobile),
+      # so it intentionally only appears in the Desktop Safari filter above.
+      # Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
+      # When adding Safari-critical tests, update this filter pattern.
+      run: |
+        npx playwright test \
+          --project="Mobile Safari" \
+          "all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit|settings-mobile-collapse"
+
+    - name: Upload Playwright Report
+      if: always()
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      with:
+        name: playwright-report-mobile-safari
+        path: |
+          tests/ui_tests/playwright/playwright-report/
+          tests/ui_tests/playwright/test-results/
+
+    - name: Stop server
+      if: always()
+      run: |
+        if [ -n "$SERVER_PID" ]; then
+          kill "$SERVER_PID" || true
+        fi
+        pkill -f "python -m local_deep_research.web.app" || true

.github/workflows/pr-triage.yml

@@ -0,0 +1,219 @@
+name: PR triage labels
+
+# Auto-applies author-class labels at PR open and toggles lifecycle labels
+# on review/synchronize events. Labels are defined in .github/labels.yml
+# and synced by .github/workflows/labels-sync.yml.
+#
+# Also applies path-derived topic labels (documentation, tests, docker,
+# github-actions, ...) that feed release-notes categorization. These were
+# historically applied by the AI code reviewer, which became label-only
+# opt-in (#4955), so triage now owns the mechanical subset. Topic labels
+# are applied on opened and ready_for_review only, and are additive-only:
+# a label a maintainer removed is not re-added on every push, and
+# semantic labels (bugfix, feature, security, performance...) stay
+# human/AI-applied.
+#
+# Uses pull_request (not pull_request_target) — fork PRs get a read-only
+# token, so label calls return 403. The script catches 403 and continues
+# so fork PRs don't show a failing check; maintainers can apply labels
+# manually for fork PRs that need them. We accept the no-op behavior on
+# forks vs the security cost of pull_request_target running with secrets
+# on fork code.
+#
+# CODEOWNERS list below is mirrored in .github/CODEOWNERS global owners
+# (line 6) — keep both in sync when the maintainer roster changes.
+
+on:
+  pull_request:
+    types: [opened, synchronize, ready_for_review]
+  pull_request_review:
+    types: [submitted, dismissed]
+
+# No concurrency group — intentionally omitted.
+# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
+# killed in-progress PR runs before they produced useful results. Race
+# between synchronize and pull_request_review can produce transient
+# label flapping in rare cases; maintainers can correct manually.
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  triage:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      issues: write         # PR labels live on the underlying issue resource
+      pull-requests: read   # listFiles for path-derived topic labels
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Apply triage labels
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        with:
+          script: |
+            // Keep CODEOWNERS list in sync with .github/CODEOWNERS line 6.
+            const CODEOWNERS = ['LearningCircuit', 'hashedviking', 'djpetti'];
+            // Known AI-bot accounts that don't carry the [bot] suffix.
+            const KNOWN_BOTS = ['moltenbot000', 'mseep-ai', 'Nexus-Digital-Automations'];
+
+            const pr = context.payload.pull_request;
+            const issueNumber = pr.number;
+            const author = pr.user.login;
+            const association = pr.author_association;
+            const isBot = author.endsWith('[bot]') || KNOWN_BOTS.includes(author);
+            const isInternal = ['OWNER', 'MEMBER', 'COLLABORATOR'].includes(association);
+
+            // 403 on label calls is the expected outcome for fork PRs —
+            // pull_request gives forks a read-only token by design. We
+            // log and continue so the workflow run stays green and a
+            // maintainer can apply labels manually instead of seeing
+            // a red check on every fork contribution.
+            const isReadOnlyTokenError = (err) => {
+              if (err.status !== 403) return false;
+              console.log(`Label call returned 403 (read-only token, likely a fork PR). Skipping.`);
+              return true;
+            };
+
+            const addLabels = async (labels) => {
+              if (!labels.length) return;
+              try {
+                await github.rest.issues.addLabels({
+                  owner: context.repo.owner,
+                  repo: context.repo.repo,
+                  issue_number: issueNumber,
+                  labels,
+                });
+              } catch (err) {
+                if (!isReadOnlyTokenError(err)) throw err;
+              }
+            };
+
+            const removeLabel = async (name) => {
+              try {
+                await github.rest.issues.removeLabel({
+                  owner: context.repo.owner,
+                  repo: context.repo.repo,
+                  issue_number: issueNumber,
+                  name,
+                });
+              } catch (err) {
+                if (err.status === 404) return;  // label wasn't applied
+                if (!isReadOnlyTokenError(err)) throw err;
+              }
+            };
+
+            // Path-derived topic labels for release-notes categorization.
+            // Mechanical mappings only — a rule belongs here iff the label
+            // follows from *which files* changed, never from what the
+            // change means. First-match-per-rule over all changed files.
+            const TOPIC_RULES = [
+              [/^\.github\/workflows\//, 'github-actions'],
+              [/^(\.github\/|\.pre-commit-hooks\/|\.pre-commit-config\.yaml$|scripts\/ci\/)/, 'ci-cd'],
+              [/^(Dockerfile|docker-compose|cookiecutter-docker\/|unraid-templates\/)/, 'docker'],
+              [/^(docs\/|[^/]+\.md$)/, 'documentation'],
+              [/^tests\//, 'tests'],
+              [/^src\/local_deep_research\/database\//, 'database'],
+              [/^src\/local_deep_research\/(defaults|settings)\//, 'configuration'],
+              [/^(pyproject\.toml|pdm\.lock|package\.json|package-lock\.json)$/, 'dependencies'],
+              [/^src\/local_deep_research\/(advanced_search_system|web_search_engines)\//, 'research'],
+              [/^src\/.*\.py$/, 'python'],
+              [/^src\/.*\.(js|mjs)$/, 'javascript'],
+              [/^src\/.*\.css$/, 'css'],
+            ];
+
+            const topicLabels = async () => {
+              const files = await github.paginate(github.rest.pulls.listFiles, {
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                pull_number: issueNumber,
+                per_page: 100,
+              });
+              const labels = new Set();
+              for (const f of files) {
+                for (const [pattern, label] of TOPIC_RULES) {
+                  if (pattern.test(f.filename)) labels.add(label);
+                }
+              }
+              return [...labels];
+            };
+
+            const action = context.payload.action;
+            const eventName = context.eventName;
+
+            if (eventName === 'pull_request' && action === 'opened') {
+              const labels = [];
+              if (isBot) labels.push('bot');
+              if (!isInternal && !isBot) labels.push('external-contributor');
+              if (association === 'FIRST_TIME_CONTRIBUTOR') labels.push('first-time-contributor');
+              if (!isInternal) labels.push('needs-codeowner-review');
+              labels.push(...await topicLabels());
+              await addLabels(labels);
+              return;
+            }
+
+            if (eventName === 'pull_request' && (action === 'synchronize' || action === 'ready_for_review')) {
+              const current = pr.labels.map((l) => l.name);
+              if (current.includes('awaiting-author')) {
+                await removeLabel('awaiting-author');
+                await addLabels(['awaiting-codeowner']);
+              }
+              // Draft → ready is a deliberate "the diff is final" moment, so
+              // recompute topic labels once there. Deliberately NOT done on
+              // synchronize: that would re-add labels a maintainer removed,
+              // on every push.
+              if (action === 'ready_for_review') {
+                await addLabels(await topicLabels());
+              }
+              return;
+            }
+
+            if (eventName === 'pull_request_review' && action === 'submitted') {
+              const review = context.payload.review;
+              const reviewer = review.user.login;
+              // Strict CODEOWNERS-only check. The hardcoded list above must
+              // mirror .github/CODEOWNERS line 6. We deliberately do NOT
+              // accept any OWNER/MEMBER/COLLABORATOR association as a
+              // codeowner: branch protection here may later adopt
+              // require_code_owner_reviews=true, at which point clearing
+              // needs-codeowner-review on a non-codeowner MEMBER review
+              // would be a security-relevant mislabel.
+              if (!CODEOWNERS.includes(reviewer)) return;
+
+              if (review.state === 'approved') {
+                await removeLabel('needs-codeowner-review');
+                await removeLabel('awaiting-codeowner');
+                // Also clear awaiting-author: a codeowner can resolve
+                // their own changes_requested review by approving without
+                // an intervening synchronize (e.g., the author convinced
+                // them via comments).
+                await removeLabel('awaiting-author');
+              } else if (review.state === 'changes_requested') {
+                await removeLabel('needs-codeowner-review');
+                await removeLabel('awaiting-codeowner');
+                await addLabels(['awaiting-author']);
+              }
+              // 'commented' → no-op.
+              return;
+            }
+
+            if (eventName === 'pull_request_review' && action === 'dismissed') {
+              // GitHub sets review.state to "dismissed" on this event — the
+              // original state is not preserved (github/docs#20216). Use the
+              // awaiting-author label as the discriminator: it's only set by
+              // a codeowner's changes_requested review, so its presence is
+              // proof the dismissal is the one we care about. Dismissals of
+              // approval/comment reviews are naturally no-ops because
+              // awaiting-author won't be present.
+              const review = context.payload.review;
+              if (!CODEOWNERS.includes(review.user.login)) return;
+
+              const current = pr.labels.map((l) => l.name);
+              if (current.includes('awaiting-author')) {
+                await removeLabel('awaiting-author');
+                await addLabels(['needs-codeowner-review']);
+              }
+              return;
+            }

.github/workflows/pre-commit.yml

@@ -0,0 +1,121 @@
+name: Pre-commit Checks
+
+on:
+  pull_request:
+    branches: [ main ]
+  workflow_call:  # Called by ci-gate.yml for release pipeline
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions:
+  contents: read
+
+jobs:
+  pre-commit:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        persist-credentials: false
+        # Full history so the author-identity hook can resolve the PR commit
+        # range (merge-base..head) from local objects WITHOUT fetching: a hook
+        # that mutates git state mid-run trips pre-commit's "files were
+        # modified by this hook".
+        fetch-depth: 0
+
+    - name: Set up Python
+      uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+      with:
+        python-version: '3.12'
+        cache: 'pip'
+
+    - name: Set up PDM
+      uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+      with:
+        python-version: '3.12'
+        cache: true
+
+    - name: Set up Node.js
+      uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+      with:
+        node-version: '24'
+
+    - name: Install Node.js dependencies
+      run: npm ci
+
+    - name: Install system dependencies for SQLCipher
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y libsqlcipher-dev
+
+    # Replicate pre-commit/action@v3.0.1 manually so the network-flaky hook
+    # *download* can be retried WITHOUT retrying the actual lint check.
+    #
+    # pre-commit lazily downloads hook environments from external sources
+    # (PyPI for ruff, GitHub release binaries for shellcheck, etc.), and a
+    # single HTTP 5xx from any one of them fails the whole job. See run #2524
+    # where `Building wheel for shellcheck_py` hit HTTP 502 fetching the
+    # shellcheck binary during the wheel build. A second attempt benefits from
+    # the partially-populated cache and almost always succeeds; two attempts is
+    # enough (a hook env that fails to install twice in a row is not a
+    # transient outage).
+    #
+    # That download is done up-front by `pre-commit install-hooks`, so only
+    # THAT step is wrapped in nick-fields/retry. The lint check itself runs
+    # exactly once: an auto-fixing hook (ruff-format, …) that modifies a file
+    # makes `pre-commit run` exit non-zero, and that MUST fail the job.
+    # Retrying the run would mask it — the second pass sees the already-fixed
+    # tree and exits 0, so unformatted code would sail through with a green
+    # check.
+    - name: Compute pre-commit cache key
+      run: echo "PY=$(python -VV | sha256sum | cut -d' ' -f1)" >> "$GITHUB_ENV"
+
+    - name: Cache pre-commit hook environments
+      uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      with:
+        path: ~/.cache/pre-commit
+        key: pre-commit|${{ env.PY }}|${{ hashFiles('.pre-commit-config.yaml') }}
+
+    - name: Install pre-commit
+      # Version-pinned (matches pyproject.toml's `pre-commit~=4.5`, lock at
+      # 4.6.0) for reproducibility, consistent with every other CI tool
+      # install (checkov, semgrep, pdm, towncrier, …) which all pin `==`.
+      # NOTE: this is *not* hash-pinned, so Scorecard's Pinned-Dependencies
+      # check still flags it ("pipCommand not pinned by hash"). That is an
+      # accepted risk — this is a read-only lint job (contents: read,
+      # persist-credentials: false), and --require-hashes would mean pinning
+      # all transitive deps and regenerating on every (automated) bump.
+      # Tracked as dismissed Scorecard alert #7777 ("won't fix").
+      run: python -m pip install pre-commit==4.6.0
+
+    # Retry ONLY the hook-environment download (the network-flaky step).
+    - name: Install pre-commit hook environments
+      uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
+      with:
+        timeout_minutes: 15
+        max_attempts: 2
+        retry_on: error
+        shell: bash
+        command: pre-commit install-hooks
+
+    # Run the checks exactly once — NO retry. An auto-fixing hook that
+    # modifies a file must fail the job, not be masked by a clean re-run.
+    # timeout-minutes restores the 15-minute ceiling the retry wrapper used to
+    # put on this step (a hung hook is otherwise bounded only by GitHub's
+    # 6-hour job default); install-hooks keeps its own retry timeout above.
+    - name: Run pre-commit
+      timeout-minutes: 15
+      run: pre-commit run --show-diff-on-failure --color=always --all-files

.github/workflows/prerelease-docker.yml

@@ -0,0 +1,512 @@
+name: Prerelease Docker Image
+
+# Build the canonical Docker image for a release.  In the build-once-promote
+# pipeline, this workflow IS the build — docker-publish.yml only retags the
+# manifest produced here.  Cosign signing, SBOM attestation, and SLSA
+# provenance are attached here once, keyed by manifest digest, so they're
+# discoverable from any tag (including the release tags later created by
+# imagetools create).
+#
+# Triggered exclusively via workflow_call from release.yml (after security
+# gates pass).  No workflow_dispatch — security and gate semantics are
+# enforced by the caller.  The build runs automatically; the only human
+# approval in the release flow is the `release` env on this workflow's
+# jobs + publish-docker + trigger-pypi + create-release in release.yml
+# (gates the actual publish, not the canonical build).
+
+on:
+  workflow_call:
+    inputs:
+      version:
+        description: "Bare semver, e.g. '1.6.9' (no leading 'v')"
+        type: string
+        required: true
+      short_sha:
+        description: "First 7 chars of commit SHA (used in the prerelease tag)"
+        type: string
+        required: true
+    secrets:
+      # Explicit secrets contract instead of `secrets: inherit` on the
+      # caller side — narrower blast radius if a future caller misuses
+      # this reusable workflow.
+      DOCKER_USERNAME:
+        required: true
+        description: "Docker Hub username for image push"
+      DOCKER_PASSWORD:
+        required: true
+        description: "Docker Hub PAT (Read+Write+Delete scopes)"
+    outputs:
+      manifest_digest:
+        description: "sha256:... digest of the multi-arch prerelease manifest. Used by docker-publish.yml to verify retag preserves the digest."
+        value: ${{ jobs.create-manifest.outputs.digest }}
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+# No approval gate at the build step — the build runs automatically once
+# security gates and CI gates in release.yml pass. The only meaningful
+# human decision in the release flow is "should this signed, attested,
+# tested image become the official release?" — gated by the `release`
+# environment on this workflow's jobs + `publish-docker` + `trigger-pypi`
+# + `create-release` in release.yml. The maintainer can pull
+# `:prerelease-v<ver>-<sha>` and smoke-test between build completion
+# and approving the release env.
+
+jobs:
+  build-amd64:
+    name: Build AMD64 Prerelease Image
+    runs-on: ubuntu-latest
+    environment: release
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden Runner
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Check out the repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Build and push AMD64 image
+        id: build
+        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        with:
+          context: .
+          platforms: linux/amd64
+          push: true
+          tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-amd64
+          cache-from: type=gha,scope=linux-amd64
+          cache-to: type=gha,mode=max,scope=linux-amd64
+
+  build-arm64:
+    name: Build ARM64 Prerelease Image
+    runs-on: ubuntu-24.04-arm
+    environment: release
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden Runner
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Check out the repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Build and push ARM64 image
+        id: build
+        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        with:
+          context: .
+          platforms: linux/arm64
+          push: true
+          tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-arm64
+          cache-from: type=gha,scope=linux-arm64
+          cache-to: type=gha,mode=max,scope=linux-arm64
+
+  security-scan:
+    name: Security Scan
+    runs-on: ubuntu-latest
+    environment: release
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Check out the repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Free disk space
+        run: |
+          sudo rm -rf /usr/share/dotnet || true
+          sudo rm -rf /usr/local/lib/android || true
+          sudo rm -rf /opt/ghc || true
+          sudo rm -rf /opt/hostedtoolcache/CodeQL || true
+          sudo docker image prune --all --force || true
+          df -h
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      - name: Build Docker image for security scan
+        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
+        with:
+          context: .
+          platforms: linux/amd64
+          push: false
+          load: true
+          tags: local-deep-research:security-scan
+          cache-from: type=gha,scope=linux-amd64
+          cache-to: type=gha,mode=max,scope=linux-amd64
+
+      # Generate Trivy SARIF for archival as a workflow artifact (all severities, never fails).
+      # Severity-gating happens in the next step.
+      - name: Generate Trivy SARIF report
+        uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
+        with:
+          image-ref: local-deep-research:security-scan
+          format: 'sarif'
+          output: 'trivy-prerelease-scan.sarif'
+          ignore-unfixed: true
+          exit-code: '0'
+          version: 'v0.69.2'
+
+      # Separate scan that fails build only on fixable HIGH/CRITICAL vulnerabilities
+      - name: Check for fixable HIGH/CRITICAL vulnerabilities
+        uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
+        with:
+          image-ref: local-deep-research:security-scan
+          severity: 'CRITICAL,HIGH'
+          ignore-unfixed: true
+          trivyignores: '.trivyignore'
+          exit-code: '1'
+          version: 'v0.69.2'
+
+      - name: Upload Trivy scan results
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: trivy-prerelease-scan
+          path: trivy-prerelease-scan.sarif
+          retention-days: 7
+
+  create-manifest:
+    name: Create Multi-Platform Prerelease Manifest
+    needs: [build-amd64, build-arm64, security-scan]
+    runs-on: ubuntu-latest
+    environment: release
+    permissions:
+      contents: read
+      id-token: write    # Required for cosign keyless OIDC signing
+      # No `packages: write` — Docker Hub auth uses DOCKER_PASSWORD secret,
+      # not GITHUB_TOKEN. `packages: write` only matters for ghcr.io pushes.
+    outputs:
+      digest: ${{ steps.capture-digest.outputs.digest }}
+
+    steps:
+      - name: Harden Runner
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Check out the repo
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Install Cosign
+        uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
+        with:
+          # Pin to cosign v2.x — see release.yml for rationale (v3 enables
+          # --new-bundle-format by default which changes the on-wire format
+          # and breaks downstream verifiers still on v2).
+          cosign-release: 'v2.6.3'
+
+      - name: Install Syft
+        uses: anchore/sbom-action/download-syft@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
+
+      - name: Create and push multi-platform manifest
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          VERSION: ${{ inputs.version }}
+          SHORT_SHA: ${{ inputs.short_sha }}
+        run: |
+          set -euo pipefail
+          TAG="prerelease-v${VERSION}-${SHORT_SHA}"
+          echo "Creating manifest for: ${DOCKER_USERNAME}/local-deep-research:${TAG}"
+          docker buildx imagetools create -t "${DOCKER_USERNAME}/local-deep-research:${TAG}" \
+            "${DOCKER_USERNAME}/local-deep-research:${TAG}-amd64" \
+            "${DOCKER_USERNAME}/local-deep-research:${TAG}-arm64"
+          echo "Manifest created successfully"
+
+          # Floating tag: re-point :prerelease at the manifest just created so
+          # testers can pin compose to `:prerelease` and pull the latest RC via
+          # `docker compose pull` without editing the tag each cycle. The
+          # versioned tag above remains for reproducibility (and is what
+          # docker-publish.yml retags by digest into :1.6.9 / :1.6 / :latest).
+          echo "Updating floating tag: ${DOCKER_USERNAME}/local-deep-research:prerelease"
+          docker buildx imagetools create -t "${DOCKER_USERNAME}/local-deep-research:prerelease" \
+            "${DOCKER_USERNAME}/local-deep-research:${TAG}"
+          echo "Floating :prerelease tag updated"
+
+      - name: Capture manifest digest
+        id: capture-digest
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          VERSION: ${{ inputs.version }}
+          SHORT_SHA: ${{ inputs.short_sha }}
+        run: |
+          set -euo pipefail
+          IMAGE_REF="${DOCKER_USERNAME}/local-deep-research:prerelease-v${VERSION}-${SHORT_SHA}"
+          # Same form as the existing docker-publish.yml inspector — avoids jq.
+          DIGEST=$(docker buildx imagetools inspect "$IMAGE_REF" --format '{{json .Manifest.Digest}}' | tr -d '"')
+          if [[ -z "$DIGEST" || "$DIGEST" != sha256:* ]]; then
+            echo "::error::Failed to capture manifest digest (got '${DIGEST}')"
+            exit 1
+          fi
+          echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"
+          echo "Manifest digest: ${DIGEST}"
+
+      - name: Sign manifest with Cosign
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DIGEST: ${{ steps.capture-digest.outputs.digest }}
+        run: |
+          set -euo pipefail
+          # Sign by digest — signature artifact lands at sha256-<digest>.sig
+          # in the same repo, discoverable from ANY tag pointing at the same
+          # digest (including release tags created later by docker-publish.yml's
+          # imagetools-create retag).
+          IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
+          echo "Signing image by digest: $IMAGE_REF"
+          cosign sign --yes "$IMAGE_REF"
+          # Brief sleep to allow registry to propagate signature
+          sleep 5
+
+      - name: Generate SLSA provenance attestation
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DIGEST: ${{ steps.capture-digest.outputs.digest }}
+        run: |
+          set -euo pipefail
+          IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
+
+          # entryPoint is the TOP-LEVEL caller (release.yml), not this
+          # reusable workflow. Per SLSA GHA buildtype v1 and the canonical
+          # slsa-github-generator, reusable workflows are explicitly NOT
+          # entryPoints. github.run_id / github.repository / github.sha all
+          # resolve to the caller's run context inside a reusable workflow.
+          # builder.id pins the workflow that actually defines the build
+          # steps — the trust root a verifier policy can pin against. We
+          # compose it from `github.repository` and a hardcoded path to
+          # THIS workflow file, with `github.ref` for the ref portion.
+          # Rationale: inside a workflow_call callee, the `github` context
+          # is scoped to the CALLER, so `github.workflow_ref` would point
+          # at release.yml (the wrong builder). The `job` context has no
+          # `workflow_ref` property either (only check_run_id, container,
+          # services, status — actionlint confirms). For a local-path
+          # reusable workflow (`uses: ./.github/workflows/...`), the
+          # callee's ref equals the caller's `github.ref`, so composing
+          # the path manually gives the correct
+          # `<owner>/<repo>/.github/workflows/prerelease-docker.yml@<ref>`
+          # format that matches the Fulcio cert SAN. Cosign and
+          # slsa-verifier both anchor on the cert anyway, so this fix is
+          # about correctness for raw-JSON policy engines / audit tools
+          # that read builder.id directly.
+          #
+          # completeness.* are FALSE because we don't capture invocation
+          # parameters or environment, and the build does network I/O for
+          # apt/pip/npm. Honest emptiness > false claims of completeness.
+          #
+          # buildInvocationId includes run_attempt so re-runs are
+          # distinguishable in audit logs.
+          #
+          # Uses auto-injected GITHUB_* shell env vars (GITHUB_REPOSITORY,
+          # GITHUB_REF, GITHUB_SHA, GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT) instead
+          # of github-context template expansion to satisfy zizmor's
+          # template-injection check. The values are semantically identical
+          # inside a workflow_call callee (both scope to the caller's context),
+          # but shell-var expansion happens at runtime in a confined string
+          # context, eliminating any theoretical injection surface from a
+          # ref/branch name containing shell metacharacters.
+          cat > provenance.json <<EOF
+          {
+            "buildType": "https://github.com/${GITHUB_REPOSITORY}/docker-build@v1",
+            "builder": {
+              "id": "https://github.com/${GITHUB_REPOSITORY}/.github/workflows/prerelease-docker.yml@${GITHUB_REF}"
+            },
+            "invocation": {
+              "configSource": {
+                "uri": "https://github.com/${GITHUB_REPOSITORY}",
+                "digest": {
+                  "sha1": "${GITHUB_SHA}"
+                },
+                "entryPoint": ".github/workflows/release.yml"
+              }
+            },
+            "metadata": {
+              "buildInvocationId": "${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}",
+              "buildStartedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
+              "completeness": {
+                "parameters": false,
+                "environment": false,
+                "materials": false
+              },
+              "reproducible": false
+            },
+            "materials": [
+              {
+                "uri": "https://github.com/${GITHUB_REPOSITORY}",
+                "digest": {
+                  "sha1": "${GITHUB_SHA}"
+                }
+              }
+            ]
+          }
+          EOF
+
+          # --replace prevents duplicate SLSA attestations on re-run. Cosign's
+          # Replace logic is keyed by predicate-type URI, so it leaves the
+          # SBOM SPDX attestation (different predicateType) untouched.
+          cosign attest --yes --replace --predicate provenance.json --type slsaprovenance "$IMAGE_REF"
+
+      - name: Verify image signature
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DIGEST: ${{ steps.capture-digest.outputs.digest }}
+        run: |
+          set -euo pipefail
+          IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
+          echo "Verifying signature for: $IMAGE_REF"
+          # Retry to handle registry propagation delay after signing
+          MAX_RETRIES=5
+          RETRY_DELAY=10
+          for i in $(seq 1 "$MAX_RETRIES"); do
+            echo "Verification attempt $i of $MAX_RETRIES..."
+            # Use GITHUB_REPOSITORY shell env var (auto-injected) instead of
+            # github.repository template expansion to avoid zizmor's
+            # template-injection finding.
+            if cosign verify \
+              --certificate-identity-regexp="https://github.com/${GITHUB_REPOSITORY}" \
+              --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
+              "$IMAGE_REF"; then
+              echo "Signature verification successful!"
+              exit 0
+            fi
+            if [ "$i" -lt "$MAX_RETRIES" ]; then
+              echo "Verification failed, waiting ${RETRY_DELAY}s before retry..."
+              sleep "$RETRY_DELAY"
+            fi
+          done
+          echo "::error::Signature verification failed after $MAX_RETRIES attempts"
+          exit 1
+
+      - name: Generate per-platform SBOMs and attest each
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DIGEST: ${{ steps.capture-digest.outputs.digest }}
+        run: |
+          set -euo pipefail
+          REPO="${DOCKER_USERNAME}/local-deep-research"
+          MANIFEST_REF="${REPO}@${DIGEST}"
+
+          # Multi-arch SBOM correctness: syft against a manifest list digest
+          # only scans the host platform's layers (per anchore/syft#1708),
+          # which would lie to ARM64 consumers. We attest each per-arch
+          # digest with its OWN SBOM so end-user verification is honest.
+          # We deliberately do NOT also produce a "manifest-level SBOM" —
+          # that would be amd64-only (host arch) and re-introduce the lie
+          # for any arm64 consumer running the README verifier recipe.
+          # The README documents the per-arch verification flow instead.
+          MANIFEST_JSON=$(docker buildx imagetools inspect "${MANIFEST_REF}" --raw)
+
+          # Defense against future buildx output changes: assert at least
+          # one per-arch entry exists. Without this, an empty/malformed
+          # manifest list would silently produce zero SBOMs and pass CI green.
+          PER_ARCH_COUNT=$(echo "${MANIFEST_JSON}" \
+            | jq '[.manifests[] | select(.platform.architecture != "unknown")] | length')
+          if [[ "${PER_ARCH_COUNT}" -lt 1 ]]; then
+            echo "::error::No per-arch manifest entries found in ${MANIFEST_REF} — SBOM generation cannot proceed"
+            echo "Raw manifest: ${MANIFEST_JSON}"
+            exit 1
+          fi
+          echo "Found ${PER_ARCH_COUNT} per-arch manifest(s) to scan"
+
+          echo "$MANIFEST_JSON" \
+            | jq -r '.manifests[] | select(.platform.architecture != "unknown") | "\(.platform.os)/\(.platform.architecture)\t\(.digest)"' \
+            | while IFS=$'\t' read -r PLAT PER_ARCH_DIGEST; do
+                ARCH="${PLAT##*/}"
+                PER_ARCH_REF="${REPO}@${PER_ARCH_DIGEST}"
+                SBOM_FILE="sbom-${ARCH}.spdx.json"
+                echo "=== Scanning ${PLAT} (${PER_ARCH_DIGEST}) ==="
+                # --platform tells syft which arch to scan — matters when
+                # the host runner can't natively execute the image.
+                syft --platform "${PLAT}" "${PER_ARCH_REF}" -o spdx-json > "${SBOM_FILE}"
+                # --replace prevents accumulation when a re-run lands on
+                # the same digest (e.g. "Re-run failed jobs" after a flake).
+                # Per cosign source pkg/cosign/remote/remote.go, --replace
+                # is per-predicate-type, so it doesn't disturb the SLSA
+                # attestation already on the manifest list digest.
+                cosign attest --yes --replace \
+                  --predicate "${SBOM_FILE}" --type spdxjson "${PER_ARCH_REF}"
+              done
+
+      - name: Upload SBOMs artifact
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: sbom
+          # Per-arch SBOMs only — `sbom-amd64.spdx.json`, `sbom-arm64.spdx.json`,
+          # one per platform in the manifest list. No manifest-level SBOM is
+          # produced (would be host-arch-only and misleading for non-amd64
+          # consumers).
+          path: sbom-*.spdx.json
+          retention-days: 90
+
+      - name: Summary
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          VERSION: ${{ inputs.version }}
+          SHORT_SHA: ${{ inputs.short_sha }}
+          DIGEST: ${{ steps.capture-digest.outputs.digest }}
+        run: |
+          TAG="prerelease-v${VERSION}-${SHORT_SHA}"
+          {
+            echo "## Prerelease Docker Image"
+            echo ""
+            echo "**Versioned tag:** \`${TAG}\`"
+            echo "**Floating tag:** \`prerelease\` (now points at this build)"
+            echo ""
+            echo "**Digest:** \`${DIGEST}\`"
+            echo ""
+            echo '```'
+            echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${TAG}"
+            echo "docker pull ${DOCKER_USERNAME}/local-deep-research:prerelease"
+            echo '```'
+            echo ""
+            echo "Signed and attested. After release approval, docker-publish.yml"
+            echo "will retag this exact digest as \`:${VERSION}\`, \`:major.minor\`,"
+            echo "and \`:latest\`."
+          } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/publish.yml

@@ -0,0 +1,625 @@
+name: Publish to PyPI
+
+# SECURITY: Only triggered via repository_dispatch from release.yml
+# (after security gate passes). We intentionally do NOT support
+# workflow_dispatch because that would bypass security checks.
+# To re-publish, trigger a new release through release.yml.
+#
+# REPRODUCIBILITY: repository_dispatch runs on default-branch HEAD at
+# dispatch time — NOT the release commit. The `release` env approval is
+# human-paced, so main can move between the release commit and the
+# dispatch (on v1.9.0 it was 4 PRs ahead, and PyPI shipped code absent
+# from the tag/Docker image). Both build jobs therefore check out
+# `client_payload.sha` — the exact commit every gate ran against — and
+# fail closed if the dispatcher didn't pin it.
+on:
+  repository_dispatch:
+    types: [publish-pypi]
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  # Build frontend if package.json exists (isolated, no secrets)
+  build-frontend:
+    runs-on: ubuntu-latest
+    outputs:
+      has-frontend: ${{ steps.check.outputs.has-frontend }}
+    container:
+      # Node 24 to match `package.json`'s `engines: { node: ">=24.0.0" }`.
+      # Was previously node:20 — npm could resolve dependencies that target
+      # APIs missing on 20 and the wheel-building publish path could ship
+      # frontend assets that break at runtime on the Node-24 Docker image.
+      image: node:24-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f # node:24-alpine
+      # Note: Network is needed for npm ci to work, but no secrets are available
+      options: --user 1001
+    permissions:
+      contents: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Validate pinned release commit
+      # POSIX sh (this job's container is Alpine — no bash). Fail closed:
+      # without a full 40-hex SHA, checkout would silently fall back to
+      # default-branch HEAD and reintroduce the drift this pin prevents.
+      # The length test matters: grep -E matches PER LINE, so a crafted
+      # multiline value ("<40-hex>\n<junk>") would pass the regex alone;
+      # ${#VAR} counts bytes in POSIX sh, so -eq 40 also excludes any
+      # embedded newline. Ancestry (sha actually on main) is enforced in
+      # build-package — this container has no python3/gh; this job holds
+      # no secrets and its output only reaches PyPI through build-package,
+      # which checks out the same validated sha.
+      env:
+        RELEASE_SHA: ${{ github.event.client_payload.sha }}
+      run: |
+        if [ "${#RELEASE_SHA}" -ne 40 ] || ! printf '%s' "$RELEASE_SHA" | grep -qE '^[0-9a-f]{40}$'; then
+          echo "::error::client_payload.sha is missing or not a full commit SHA — refusing to build from default-branch HEAD. The dispatcher (release.yml trigger-pypi) must pin the release commit."
+          exit 1
+        fi
+        echo "Building from pinned release commit $RELEASE_SHA"
+
+    - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        ref: ${{ github.event.client_payload.sha }}
+        persist-credentials: false
+
+    - name: Check for frontend assets
+      id: check
+      run: |
+        if [ -f "package.json" ]; then
+          echo "has-frontend=true" >> "$GITHUB_OUTPUT"
+          echo "Found package.json - will build frontend"
+        else
+          echo "has-frontend=false" >> "$GITHUB_OUTPUT"
+          echo "ERROR: No package.json found - frontend build is required for PyPI releases"
+          exit 1
+        fi
+
+    - name: Build frontend
+      if: steps.check.outputs.has-frontend == 'true'
+      run: |
+        echo "=== Frontend Build Process Starting ==="
+        echo "Current directory: $(pwd)"
+        echo "Node version: $(node --version)"
+        echo "NPM version: $(npm --version)"
+
+        # Install dependencies from root package.json
+        echo ""
+        echo "📦 Installing dependencies..."
+        # Use npm ci for reproducible builds from lockfile
+        npm ci --ignore-scripts --no-audit --no-fund
+        echo "✅ Dependencies installed"
+
+        # Show pre-build directory structure
+        echo ""
+        echo "📁 Pre-build directory structure:"
+        echo "Contents of src/local_deep_research/web/static/:"
+        find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "Directory not found"
+
+        # Build with Vite (outputs to src/local_deep_research/web/static/dist)
+        echo ""
+        echo "🔨 Running Vite build..."
+        npm run build
+        echo "✅ Vite build completed"
+
+        # Show post-build directory structure
+        echo ""
+        echo "📁 Post-build directory structure:"
+        echo "Contents of src/local_deep_research/web/static/:"
+        find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
+        echo ""
+        echo "Contents of src/local_deep_research/web/static/dist/:"
+        find src/local_deep_research/web/static/dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
+        echo ""
+        echo "Looking for manifest.json:"
+        find src/local_deep_research/web/static/dist -name "manifest.json" -type f 2>/dev/null || echo "No manifest.json found"
+
+        # Verify the build completed successfully
+        if [ ! -f "src/local_deep_research/web/static/dist/.vite/manifest.json" ]; then
+          echo "❌ ERROR: Build failed - manifest.json not created at expected location"
+          echo "Expected location: src/local_deep_research/web/static/dist/.vite/manifest.json"
+          echo "Actual dist contents:"
+          find src/local_deep_research/web/static/dist -type f | head -20
+          exit 1
+        fi
+
+        echo "✅ Found manifest.json at: src/local_deep_research/web/static/dist/.vite/manifest.json"
+        echo "Manifest contents (first 10 lines):"
+        head -10 src/local_deep_research/web/static/dist/.vite/manifest.json
+
+        # Create build marker
+        echo ""
+        echo "📝 Creating build marker..."
+        echo "{\"status\":\"complete\",\"built\":\"$(date -Iseconds)\"}" > src/local_deep_research/web/static/.frontend-built
+        echo "✅ Build marker created"
+
+    - name: Upload frontend artifacts
+      if: steps.check.outputs.has-frontend == 'true'
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      with:
+        name: frontend-assets
+        path: src/local_deep_research/web/static/
+        retention-days: 1
+        include-hidden-files: true  # Ensure .vite directory is included
+
+  # Build Python package (isolated, no PyPI access)
+  build-package:
+    needs: build-frontend
+    runs-on: ubuntu-latest
+    container:
+      image: python:3.12-slim@sha256:971f04b358cf483ec445a8d388fb55267451f080d90fb136c8e69684a02a9604 # python:3.12-slim
+    permissions:
+      contents: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Validate pinned release commit
+      # Same fail-closed shape guard as build-frontend (length test: see
+      # comment there), plus the ANCESTRY check that only this job can
+      # run (python3 in the container): the pinned sha must be reachable
+      # from main. Anyone able to send repository_dispatch (repo write /
+      # the PAT holder) chooses the payload — without this check they
+      # could point the build at ANY commit object in the repo network
+      # (scratch branches, fork-PR heads) that never passed review or the
+      # release gates. release.yml's own build job verifies its sha is on
+      # main, but a forged dispatch bypasses release.yml entirely; the
+      # `release` environment approval on the publish job is then the
+      # only barrier, and an approver has no reason to suspect a
+      # legitimate-looking pending deployment. Enforced here (not in
+      # build-frontend) because every byte that reaches PyPI flows
+      # through this job's checkout.
+      env:
+        RELEASE_SHA: ${{ github.event.client_payload.sha }}
+        GH_TOKEN: ${{ github.token }}
+      run: |
+        if [ "${#RELEASE_SHA}" -ne 40 ] || ! printf '%s' "$RELEASE_SHA" | grep -qE '^[0-9a-f]{40}$'; then
+          echo "::error::client_payload.sha is missing or not a full commit SHA — refusing to build from default-branch HEAD. The dispatcher (release.yml trigger-pypi) must pin the release commit."
+          exit 1
+        fi
+        python3 - <<'PYEOF'
+        import json
+        import os
+        import sys
+        import urllib.request
+
+        sha = os.environ["RELEASE_SHA"]
+        repo = os.environ["GITHUB_REPOSITORY"]
+        req = urllib.request.Request(
+            f"https://api.github.com/repos/{repo}/compare/{sha}...heads/main",
+            headers={
+                "Authorization": f"Bearer {os.environ['GH_TOKEN']}",
+                "Accept": "application/vnd.github+json",
+            },
+        )
+        try:
+            with urllib.request.urlopen(req) as resp:
+                status = json.load(resp)["status"]
+        except urllib.error.HTTPError as exc:
+            # 404 = sha unknown to the repo network. Fail closed either way.
+            print(f"::error::compare API rejected sha {sha}: HTTP {exc.code}")
+            sys.exit(1)
+        # base=sha, head=main: main being "ahead" of (or "identical" to)
+        # the sha means the sha is an ancestor of main. "diverged" or
+        # "behind" means it is not.
+        if status not in ("ahead", "identical"):
+            print(
+                f"::error::pinned sha {sha} is not an ancestor of main "
+                f"(compare status: {status}) — refusing to build unreviewed code"
+            )
+            sys.exit(1)
+        print(f"Pinned sha is on main (compare status: {status})")
+        PYEOF
+        echo "Building from pinned release commit $RELEASE_SHA"
+
+    - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        ref: ${{ github.event.client_payload.sha }}
+        persist-credentials: false
+
+    - name: Install build dependencies
+      run: |
+        apt-get update && apt-get install -y libsqlcipher-dev build-essential
+        pip install pdm==2.26.2
+
+    - name: Download frontend artifacts
+      if: needs.build-frontend.outputs.has-frontend == 'true'
+      uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+      continue-on-error: true
+      with:
+        name: frontend-assets
+        path: src/local_deep_research/web/static/
+
+    - name: Verify frontend build artifacts
+      run: |
+        echo "=== Frontend Artifacts Verification ==="
+        echo "Current directory: $(pwd)"
+        echo ""
+        echo "📁 Checking downloaded artifacts structure:"
+        echo "Contents of src/:"
+        find src/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/ directory not found"
+        echo ""
+        echo "Contents of src/local_deep_research/:"
+        find src/local_deep_research/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/ not found"
+        echo ""
+        echo "Contents of src/local_deep_research/web/:"
+        find src/local_deep_research/web/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/web/ not found"
+        echo ""
+        echo "Contents of src/local_deep_research/web/static/:"
+        find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/web/static/ not found"
+
+        # Check if dist directory exists
+        echo ""
+        echo "🔍 Checking for dist directory..."
+        if [ ! -d "src/local_deep_research/web/static/dist" ]; then
+          echo "❌ ERROR: Frontend dist directory not found!"
+          echo "Expected location: src/local_deep_research/web/static/dist"
+          echo "Actual structure of static directory:"
+          find src/local_deep_research/web/static -type d 2>/dev/null | head -20
+          echo "The frontend build artifacts are missing."
+          exit 1
+        fi
+        echo "✅ Found dist directory at: src/local_deep_research/web/static/dist"
+
+        # Show dist directory structure
+        echo ""
+        echo "📁 Dist directory structure:"
+        echo "Contents of dist/:"
+        find src/local_deep_research/web/static/dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
+        echo ""
+        echo "All files in dist (recursive):"
+        find src/local_deep_research/web/static/dist -type f | head -20
+
+        # Check for critical files
+        echo ""
+        echo "🔍 Checking for manifest.json..."
+        # Check both possible locations for manifest.json
+        MANIFEST_PATH=""
+        if [ -f "src/local_deep_research/web/static/dist/.vite/manifest.json" ]; then
+          MANIFEST_PATH="src/local_deep_research/web/static/dist/.vite/manifest.json"
+        elif [ -f "src/local_deep_research/web/static/dist/manifest.json" ]; then
+          MANIFEST_PATH="src/local_deep_research/web/static/dist/manifest.json"
+        fi
+
+        if [ -z "$MANIFEST_PATH" ]; then
+          echo "❌ ERROR: Vite manifest.json not found!"
+          echo "Checked locations:"
+          echo "  - src/local_deep_research/web/static/dist/.vite/manifest.json"
+          echo "  - src/local_deep_research/web/static/dist/manifest.json"
+          echo "Looking for any manifest.json files:"
+          find src/local_deep_research/web/static -name "manifest.json" -type f 2>/dev/null || echo "None found"
+          echo "Checking if .vite directory exists:"
+          find src/local_deep_research/web/static/dist/ -maxdepth 1 -name ".*" -type d || echo "No hidden directories found"
+          echo ""
+          echo "This is likely due to artifact transfer losing the hidden .vite directory."
+          echo "The include-hidden-files option should fix this."
+          echo "The frontend build appears to be incomplete - cannot continue."
+          exit 1
+        fi
+        echo "✅ Found manifest.json at: $MANIFEST_PATH"
+        echo "Manifest size: $(wc -c < "$MANIFEST_PATH") bytes"
+
+        # Check for JS files
+        echo ""
+        echo "🔍 Checking for JavaScript files..."
+        JS_COUNT=$(find src/local_deep_research/web/static/dist -name "*.js" -type f | wc -l)
+        if [ "$JS_COUNT" -eq 0 ]; then
+          echo "❌ ERROR: No JavaScript files found!"
+          echo "Expected JS files in dist/"
+          echo "The frontend build appears to be incomplete."
+          exit 1
+        fi
+        echo "✅ Found $JS_COUNT JavaScript file(s)"
+        echo "JS files:"
+        find src/local_deep_research/web/static/dist -name "*.js" -type f | head -5
+
+        # Check for CSS files
+        echo ""
+        echo "🔍 Checking for CSS files..."
+        CSS_COUNT=$(find src/local_deep_research/web/static/dist -name "*.css" -type f | wc -l)
+        if [ "$CSS_COUNT" -eq 0 ]; then
+          echo "❌ ERROR: No CSS files found!"
+          echo "Expected CSS files in dist/"
+          echo "The frontend build appears to be incomplete."
+          exit 1
+        fi
+        echo "✅ Found $CSS_COUNT CSS file(s)"
+        echo "CSS files:"
+        find src/local_deep_research/web/static/dist -name "*.css" -type f | head -5
+
+        # Check for build marker
+        echo ""
+        echo "🔍 Checking for build marker..."
+        if [ ! -f "src/local_deep_research/web/static/.frontend-built" ]; then
+          echo "⚠️  WARNING: Frontend build marker not found (non-critical)"
+        else
+          echo "✅ Found build marker"
+          echo "Build marker contents:"
+          cat src/local_deep_research/web/static/.frontend-built
+        fi
+
+        echo ""
+        echo "=== ✅ Frontend build artifacts verified successfully ==="
+        echo "Summary:"
+        echo "  - Dist directory: ✓"
+        echo "  - Manifest.json: ✓"
+        echo "  - JS files: $JS_COUNT file(s)"
+        echo "  - CSS files: $CSS_COUNT file(s)"
+
+    - name: Build Python package
+      run: |
+        echo "=== Python Package Build ==="
+        echo "Current directory: $(pwd)"
+        echo ""
+        echo "📦 Building Python package with PDM..."
+        pdm build
+        echo ""
+        echo "✅ Package build completed"
+        echo ""
+        echo "📁 Package contents:"
+        find dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
+        echo ""
+        echo "Package sizes:"
+        du -h dist/*
+
+    - name: Upload package
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      with:
+        name: python-dist
+        path: dist/
+        retention-days: 1
+
+  # Publish (ONLY job with PyPI access)
+  publish:
+    needs: build-package
+    runs-on: ubuntu-latest
+    environment: release
+    permissions:
+      id-token: write
+      attestations: write  # Required for generating attestations
+
+    steps:
+    - name: Harden Runner
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Download package
+      uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+      with:
+        name: python-dist
+        path: dist/
+
+    - name: List packages
+      run: |
+        echo "=== Downloaded Package Artifacts ==="
+        echo "📦 Packages in dist/:"
+        ls -lh dist/
+        echo ""
+        echo "Package details:"
+        for file in dist/*; do
+          echo "  - $(basename "$file"): $(du -h "$file" | cut -f1)"
+        done
+
+    - name: Verify package contents
+      run: |
+        echo "=== Package Content Verification ==="
+        echo "Verifying package contains frontend assets..."
+        echo ""
+
+        # Install wheel to inspect package
+        echo "📦 Installing wheel for package inspection..."
+        pip install wheel==0.46.2
+
+        # Find the wheel file
+        WHEEL_FILE=$(find dist -name "*.whl" -type f 2>/dev/null | head -1)
+        TAR_FILE=$(find dist -name "*.tar.gz" -type f 2>/dev/null | head -1)
+
+        if [ -z "$WHEEL_FILE" ]; then
+          echo "❌ ERROR: No wheel file found in dist/"
+          echo "Found files:"
+          find dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
+          exit 1
+        fi
+
+        echo "✅ Found wheel file: $WHEEL_FILE"
+        echo "Wheel size: $(du -h "$WHEEL_FILE" | cut -f1)"
+        echo ""
+
+        if [ -n "$TAR_FILE" ]; then
+          echo "✅ Found source distribution: $TAR_FILE"
+          echo "Source dist size: $(du -h "$TAR_FILE" | cut -f1)"
+        fi
+        echo ""
+
+        echo "🔍 Inspecting wheel contents..."
+        # Extract and check for frontend files
+        python -m zipfile -l "$WHEEL_FILE" > wheel_contents.txt
+
+        TOTAL_FILES=$(wc -l < wheel_contents.txt)
+        echo "Total files in wheel: $TOTAL_FILES"
+        echo ""
+
+        # Count different file types
+        echo "📊 File type breakdown:"
+        echo "  Python files: $(grep -c '\.py' wheel_contents.txt || echo 0)"
+        echo "  JavaScript files: $(grep -c '\.js' wheel_contents.txt || echo 0)"
+        echo "  CSS files: $(grep -c '\.css' wheel_contents.txt || echo 0)"
+        echo "  JSON files: $(grep -c '\.json' wheel_contents.txt || echo 0)"
+        echo "  HTML files: $(grep -c '\.html' wheel_contents.txt || echo 0)"
+        echo ""
+
+        # Check for critical frontend directories/files
+        echo "🔍 Checking for frontend assets..."
+
+        echo "Looking for dist directory..."
+        if ! grep -q "local_deep_research/web/static/dist/" wheel_contents.txt; then
+          echo "❌ ERROR: Frontend dist directory not found in package!"
+          echo "Package is missing frontend build artifacts."
+          echo "Static files found in package:"
+          grep "static" wheel_contents.txt | head -10 || echo "No static files found"
+          exit 1
+        fi
+        echo "✅ Found dist directory"
+
+        echo ""
+        echo "Looking for JavaScript files..."
+        JS_IN_WHEEL=$(grep -c "local_deep_research/web/static/dist.*\.js" wheel_contents.txt || echo 0)
+        if [ "$JS_IN_WHEEL" -eq 0 ]; then
+          echo "❌ ERROR: Frontend JS files not found in package!"
+          echo "JavaScript files in package:"
+          grep "\.js" wheel_contents.txt | head -10 || echo "No JS files found"
+          exit 1
+        fi
+        echo "✅ Found $JS_IN_WHEEL JavaScript file(s) in dist"
+
+        echo ""
+        echo "Looking for CSS files..."
+        CSS_IN_WHEEL=$(grep -c "local_deep_research/web/static/dist.*\.css" wheel_contents.txt || echo 0)
+        if [ "$CSS_IN_WHEEL" -eq 0 ]; then
+          echo "❌ ERROR: Frontend CSS files not found in package!"
+          echo "CSS files in package:"
+          grep "\.css" wheel_contents.txt | head -10 || echo "No CSS files found"
+          exit 1
+        fi
+        echo "✅ Found $CSS_IN_WHEEL CSS file(s) in dist"
+
+        echo ""
+        echo "Looking for manifest.json..."
+        if ! grep -q "local_deep_research/web/static/dist/.vite/manifest.json" wheel_contents.txt; then
+          echo "⚠️  WARNING: manifest.json not found in package (may be okay)"
+        else
+          echo "✅ Found manifest.json"
+        fi
+
+        echo ""
+        echo "=== ✅ Package verification successful ==="
+        echo "Summary:"
+        echo "  - Wheel file: ✓"
+        echo "  - Frontend dist: ✓"
+        echo "  - JS files: $JS_IN_WHEEL file(s)"
+        echo "  - CSS files: $CSS_IN_WHEEL file(s)"
+        echo ""
+        echo "Sample frontend assets in package:"
+        grep "local_deep_research/web/static/dist/" wheel_contents.txt | head -10
+
+    - name: Generate attestations
+      uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
+      with:
+        subject-path: 'dist/*'
+
+    - name: Pre-publish summary
+      env:
+        EVENT_NAME: ${{ github.event_name }}
+        DISPATCH_TAG: ${{ github.event.client_payload.tag }}
+        DISPATCH_PRERELEASE: ${{ github.event.client_payload.prerelease }}
+        RELEASE_TAG: ${{ github.event.release.tag_name }}
+        RELEASE_PRERELEASE: ${{ github.event.release.prerelease }}
+      run: |
+        echo "=== 🚀 Ready to Publish ==="
+        echo ""
+        echo "📦 Package Information:"
+        # Handle both release event and repository_dispatch
+        if [ "$EVENT_NAME" = "repository_dispatch" ]; then
+          echo "  - Trigger: repository_dispatch"
+          echo "  - Tag: $DISPATCH_TAG"
+          echo "  - Prerelease: $DISPATCH_PRERELEASE"
+        else
+          echo "  - Trigger: release"
+          echo "  - Tag: $RELEASE_TAG"
+          echo "  - Prerelease: $RELEASE_PRERELEASE"
+        fi
+        echo ""
+        echo "📁 Package files to publish:"
+        ls -lh dist/
+        echo ""
+        echo "✅ All checks passed - proceeding with publication"
+
+    - name: Publish to Test PyPI
+      if: (github.event_name == 'release' && github.event.release.prerelease == true) || (github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease == true)
+      uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
+      with:
+        repository-url: https://test.pypi.org/legacy/
+        skip-existing: true  # Don't fail if version already exists
+
+    - name: Publish to PyPI
+      if: (github.event_name == 'release' && github.event.release.prerelease != true) || (github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease != true)
+      uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
+      with:
+        skip-existing: true  # Don't fail if version already exists
+
+  # Verify the published package is installable (skipped for prereleases which go to Test PyPI)
+  verify-publish:
+    needs: publish
+    if: ${{ !(github.event_name == 'release' && github.event.release.prerelease == true) && !(github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease == true) }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    permissions:
+      contents: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Determine published version
+      id: version
+      env:
+        EVENT_NAME: ${{ github.event_name }}
+        PAYLOAD_TAG: ${{ github.event.client_payload.tag }}
+        RELEASE_TAG: ${{ github.event.release.tag_name }}
+      run: |
+        if [ "$EVENT_NAME" = "repository_dispatch" ]; then
+          TAG="$PAYLOAD_TAG"
+        else
+          TAG="$RELEASE_TAG"
+        fi
+        # Strip leading 'v' if present
+        VERSION="${TAG#v}"
+        echo "version=$VERSION" >> "$GITHUB_OUTPUT"
+        echo "Published version: $VERSION"
+
+    - name: Install system dependencies
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y libsqlcipher-dev
+
+    - name: Wait for PyPI index to update
+      run: |
+        echo "Waiting 60 seconds for PyPI index propagation..."
+        sleep 60
+
+    - name: Verify package installation
+      run: |
+        python3 -m venv /tmp/verify-env
+        . /tmp/verify-env/bin/activate
+
+        VERSION="${{ steps.version.outputs.version }}"
+        echo "Installing local-deep-research==${VERSION}..."
+
+        # Retry up to 5 times with increasing delay for PyPI propagation
+        for i in 1 2 3 4 5; do
+          if pip install "local-deep-research==${VERSION}" 2>&1; then
+            echo "Installation successful on attempt $i"
+            break
+          fi
+          if [ "$i" -lt 5 ]; then
+            echo "Attempt $i failed, waiting $((i * 30))s..."
+            sleep $((i * 30))
+          else
+            echo "::error::Failed to install local-deep-research==${VERSION} after 5 attempts"
+            exit 1
+          fi
+        done
+
+        # Verify import works
+        python3 -c "import local_deep_research; print(f'Successfully imported v{local_deep_research.__version__}')"

.github/workflows/puppeteer-e2e-tests.yml

@@ -0,0 +1,495 @@
+name: Puppeteer E2E Tests
+
+on:
+  pull_request:
+    types: [labeled]
+  workflow_call:  # Called by release.yml (e2e-test-gate)
+    secrets:
+      OPENROUTER_API_KEY:
+        required: false
+      SERPER_API_KEY:
+        required: false
+  workflow_dispatch:  # Manual trigger runs all tests
+  schedule:
+    # Run weekly on Sunday at 2 AM UTC
+    - cron: '0 2 * * 0'
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions: {}  # Minimal top-level for OSSF Scorecard
+
+jobs:
+  puppeteer-tests:
+    name: Puppeteer E2E Tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    # For pull_request events, only run when labeled test:puppeteer or test:e2e.
+    # For all other triggers (push via workflow_call from release.yml,
+    # workflow_dispatch, schedule), always run.
+    # NOTE: workflow_call inherits the caller's github.event_name (e.g. "push"),
+    # so checking for 'workflow_call' never matches — use != 'pull_request' instead.
+    if: |
+      github.event_name != 'pull_request' ||
+      github.event.label.name == 'test:puppeteer' ||
+      github.event.label.name == 'test:e2e'
+    environment: ci
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
+    env:
+      LABEL_NAME: ${{ github.event.label.name || 'manual' }}
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Setup PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Setup Node.js
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: '24'
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          # Note: libasound2t64 is the Ubuntu 24.04 name for libasound2
+          sudo apt-get install -y libsqlcipher-dev xvfb \
+            libnss3 libgbm1 libasound2t64 libatk1.0-0 libatk-bridge2.0-0 \
+            libcups2 libdrm2 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
+            fonts-liberation xdg-utils
+
+      - name: Install Python dependencies
+        run: pdm install
+
+      - name: Install root frontend dependencies
+        run: npm ci
+
+      - name: Build Vite frontend bundle
+        # Generates src/local_deep_research/web/static/dist/, which the
+        # Flask app loads via vite_helper. Without this the page renders
+        # without bundled CSS, causing tests to run against a partially
+        # unstyled UI (the responsive baseline tracked in PR #3979 was
+        # captured against a built bundle).
+        run: npm run build
+
+      - name: Install Puppeteer dependencies
+        working-directory: tests/puppeteer
+        run: |
+          npm ci
+          npx puppeteer browsers install chrome
+
+      - name: Setup directories
+        run: |
+          mkdir -p data/encrypted_databases
+          mkdir -p tests/puppeteer/screenshots
+
+      - name: Initialize database
+        env:
+          TEST_ENV: true
+          LDR_DATA_DIR: ${{ github.workspace }}/data
+          LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
+          # Without LDR_TEST_MODE the 1000 is clamped to the production
+          # minimum (256000), making SQLCipher ~256x slower — see #4430.
+          LDR_TEST_MODE: "1"
+        run: |
+          cd src
+          pdm run python ../scripts/ci/init_test_database.py
+
+      - name: Create test configuration
+        run: |
+          # Create config directory if needed
+          mkdir -p ~/.config/local-deep-research
+
+          # Create settings with OpenRouter + Gemini 2.5 Flash Lite
+          # (Gemini 2.0 Flash is heavily rate-limited on the OpenRouter free
+          #  pool; 2.5 Flash Lite is GA, cheaper, and far less contended.)
+          cat > ~/.config/local-deep-research/settings.toml << 'EOF'
+          [llm]
+          provider = "openrouter"
+          model = "google/gemini-2.5-flash-lite"
+
+          [search]
+          tool = "serper"
+          iterations = 1
+          questions_per_iteration = 2
+
+          [general]
+          report_type = "quick"
+          EOF
+
+      - name: Start LDR server
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+          SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
+          # Settings manager env var overrides (format: LDR_<KEY_WITH_UNDERSCORES>)
+          # LLM settings - use OpenRouter with Gemini 2.5 Flash Lite
+          LDR_LLM_PROVIDER: openrouter
+          LDR_LLM_MODEL: google/gemini-2.5-flash-lite
+          LDR_LLM_OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+          # Search settings
+          LDR_SEARCH_TOOL: serper
+          LDR_SEARCH_ENGINE_WEB_SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
+          # Database and server settings
+          LDR_DATA_DIR: ${{ github.workspace }}/data
+          LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
+          # Without LDR_TEST_MODE the 1000 is clamped to the production
+          # minimum (256000), making SQLCipher ~256x slower — see #4430.
+          LDR_TEST_MODE: "1"
+          CI: true
+          LDR_DISABLE_RATE_LIMITING: true
+        run: |
+          # Start server in background
+          cd src
+          nohup pdm run python -m local_deep_research.web.app > /tmp/ldr_server.log 2>&1 &
+
+          echo "Waiting for server to start..."
+          for i in {1..60}; do
+            if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
+              echo "Server is ready after $i seconds!"
+              break
+            fi
+            if ! pgrep -f 'python -m local_deep_research.web.app' > /dev/null; then
+              echo "Server process died!"
+              cat /tmp/ldr_server.log
+              exit 1
+            fi
+            echo "Waiting... ($i/60)"
+            sleep 1
+          done
+
+          # Final check if server is ready
+          if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
+            echo "Server failed to start. Logs:"
+            cat /tmp/ldr_server.log
+            exit 1
+          fi
+
+      - name: Run Puppeteer tests
+        working-directory: tests/puppeteer
+        env:
+          TEST_URL: http://localhost:5000
+          HEADLESS: true
+        run: |
+          set -o pipefail
+          # Run all test suites with xvfb for virtual display
+          xvfb-run -a -s "-screen 0 1920x1080x24" npm run test:ci 2>&1 | tee test-output.log
+          TEST_RESULT="${PIPESTATUS[0]}"
+
+          # Save exit code for downstream steps
+          echo "TEST_EXIT_CODE=${TEST_RESULT}" >> "$GITHUB_ENV"
+
+          # Exit with test result so the step reflects pass/fail
+          exit "${TEST_RESULT}"
+
+      - name: Run egress-policy UI test
+        # Release gate for the Privacy & Egress UI (scope dropdown, the
+        # require-local toggles, the STRICT+meta-picker guard, and the
+        # data-scope propagation across pages). This test lives in
+        # tests/ui_tests (a standalone Puppeteer script, NOT the mocha suite
+        # above) and runs against the SAME booted server. It authenticates as
+        # the CI test user (test_admin) created by init_test_database.py and
+        # matched in tests/ui_tests/auth_helper.js, and skips screenshots when
+        # CI=true. `if: always()` so its result is reported even when the mocha
+        # suite above failed; the script exits non-zero on any failed check, so
+        # a regression fails the release gate.
+        if: always()
+        working-directory: tests/ui_tests
+        env:
+          BASE_URL: http://localhost:5000
+          CI: true
+          HEADLESS: true
+        run: |
+          set -o pipefail
+          npm ci
+          npx puppeteer browsers install chrome
+          xvfb-run -a -s "-screen 0 1920x1080x24" \
+            node test_egress_policy_ui.js 2>&1 | tee egress-ui-output.log
+
+      - name: Extract test summary
+        if: always()
+        working-directory: tests/puppeteer
+        run: |
+          # Extract pass/fail summary from mocha output
+          if [ -f test-output.log ]; then
+            # Get the summary line (e.g., "26 passing (3m)")
+            PASSING=$(grep -oP '\d+ passing' test-output.log | tail -1 || echo "0 passing")
+            FAILING=$(grep -oP '\d+ failing' test-output.log | tail -1 || echo "0 failing")
+
+            echo "TESTS_PASSING=${PASSING}" >> "$GITHUB_ENV"
+            echo "TESTS_FAILING=${FAILING}" >> "$GITHUB_ENV"
+
+            # Extract key test results (collection creation, subscription, research)
+            echo "## Key Test Results" > test-summary.md
+            echo "" >> test-summary.md
+
+            # Check for collection creation
+            if grep -q "Collection found in dropdown: true" test-output.log; then
+              echo "- ✅ Collection creation: **Success**" >> test-summary.md
+            elif grep -q "Collection found in dropdown: false" test-output.log; then
+              echo "- ❌ Collection creation: **Failed**" >> test-summary.md
+            fi
+
+            # Check for subscription creation
+            if grep -q "Subscription name found: true" test-output.log; then
+              echo "- ✅ Subscription creation: **Success**" >> test-summary.md
+            elif grep -q "Subscription name found: false" test-output.log; then
+              echo "- ❌ Subscription creation: **Failed**" >> test-summary.md
+            fi
+
+            # Check for research completion
+            if grep -q "Research completed: true" test-output.log; then
+              RESEARCH_TIME=$(grep -oP "Research completed: true \(took \K\d+(?=s\))" test-output.log || echo "?")
+              echo "- ✅ Research workflow: **Completed** (${RESEARCH_TIME}s)" >> test-summary.md
+            elif grep -q "Research completed: false" test-output.log; then
+              echo "- ⚠️ Research workflow: **Timed out**" >> test-summary.md
+            fi
+
+            # Check for settings persistence
+            if grep -q "After reload value:" test-output.log; then
+              echo "- ✅ Settings persistence: **Working**" >> test-summary.md
+            fi
+
+            # Add screenshot count
+            SCREENSHOT_COUNT=$(find screenshots -name "*.png" 2>/dev/null | wc -l || echo "0")
+            echo "" >> test-summary.md
+            echo "**Screenshots captured:** ${SCREENSHOT_COUNT}" >> test-summary.md
+          fi
+
+      - name: Show test summary in workflow UI
+        if: always()
+        working-directory: tests/puppeteer
+        # zizmor: ignore[template-injection] — env.TEST_EXIT_CODE/TESTS_PASSING/TESTS_FAILING are set by this workflow, not user input
+        run: |
+          # Output to GITHUB_STEP_SUMMARY so results are visible in Actions UI
+          {
+            echo "## 🧪 Puppeteer E2E Test Results"
+            echo ""
+            if [ "$TEST_EXIT_CODE" == "0" ]; then
+              echo "### ✅ Tests Passed!"
+            else
+              echo "### ❌ Tests Failed"
+            fi
+            echo ""
+            echo "**Summary:** $TESTS_PASSING, $TESTS_FAILING"
+            echo ""
+
+            # Include key test results
+            if [ -f test-summary.md ]; then
+              cat test-summary.md
+              echo ""
+            fi
+
+            echo "---"
+            echo ""
+            echo "<details>"
+            echo "<summary>📋 Last 50 lines of test output</summary>"
+            echo ""
+            echo '```'
+            if [ -f test-output.log ]; then
+              tail -50 test-output.log
+            fi
+            echo '```'
+            echo "</details>"
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload server logs
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: server-logs
+          path: /tmp/ldr_server.log
+          retention-days: 7
+
+      - name: Upload test artifacts
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: puppeteer-test-results
+          path: |
+            tests/puppeteer/test-output.log
+            tests/puppeteer/test-summary.md
+            tests/puppeteer/screenshots/
+          retention-days: 7
+
+      - name: Find PR number
+        id: find-pr
+        if: always()
+        # zizmor: ignore[template-injection] — github.ref_name and github.repository are safe GitHub context values
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          EVENT_NAME: ${{ github.event_name }}
+          PR_NUM: ${{ github.event.pull_request.number }}
+          REF_NAME: ${{ github.ref_name }}
+          REPO: ${{ github.repository }}
+        run: |
+          if [ "$EVENT_NAME" == "pull_request" ]; then
+            # Labeled PR run — the primary, correct path. Real branches targeting main.
+            echo "pr_number=$PR_NUM" >> "$GITHUB_OUTPUT"
+          elif [ "$EVENT_NAME" == "workflow_dispatch" ] && [ "$REF_NAME" != "main" ]; then
+            # Manual run on a feature branch. Map branch -> its open PR, but only a
+            # UNIQUE, SAME-REPO match: cross-repo (fork) PRs and ambiguous matches are
+            # skipped so we never post to the wrong PR (e.g. a fork PR'd from `main`).
+            PR_NUMBER=$(gh pr list --head "$REF_NAME" --state open --repo "$REPO" \
+              --json number,isCrossRepository \
+              --jq '[.[] | select(.isCrossRepository == false)]
+                    | if length == 1 then .[0].number | tostring else "" end')
+            echo "pr_number=${PR_NUMBER}" >> "$GITHUB_OUTPUT"
+          else
+            # release.yml workflow_call (event=push) and the weekly schedule both run
+            # on the default branch. There is never a PR from main, so any
+            # `gh pr list --head main` match is a coincidental fork PR — skip
+            # entirely (results still go to the Actions step summary). See #3884.
+            echo "pr_number=" >> "$GITHUB_OUTPUT"
+            echo "event=$EVENT_NAME ref=$REF_NAME has no associated PR — skipping comment"
+          fi
+
+      - name: Build PR comment
+        if: always() && steps.find-pr.outputs.pr_number != ''
+        # zizmor: ignore[template-injection] — env.TEST_EXIT_CODE/TESTS_PASSING/TESTS_FAILING are set by this workflow, not user input
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+        run: |
+          {
+            # Hidden marker — the "Post or update PR comment" step finds this
+            # comment by this marker so re-runs edit it in place (see #3884).
+            echo '<!-- puppeteer-e2e-results -->'
+            echo "## 🧪 Puppeteer E2E Test Results"
+            echo ""
+            if [ "$TEST_EXIT_CODE" == "0" ]; then
+              echo "### ✅ Tests Passed!"
+            else
+              echo "### ❌ Tests Failed"
+            fi
+            echo ""
+            echo "**Summary:** $TESTS_PASSING, $TESTS_FAILING"
+            echo ""
+
+            # Include key test results
+            if [ -f tests/puppeteer/test-summary.md ]; then
+              cat tests/puppeteer/test-summary.md
+              echo ""
+            fi
+
+            # Include research output if available
+            if [ -f tests/puppeteer/research-output/research-result.txt ]; then
+              echo "---"
+              echo ""
+              echo "### 📝 Research Output"
+              echo ""
+              if [ -f tests/puppeteer/research-output/research-metadata.json ]; then
+                QUERY=$(jq -r '.query // "unknown"' tests/puppeteer/research-output/research-metadata.json)
+                echo "**Query:** \`$QUERY\`"
+                echo ""
+              fi
+              echo "<details>"
+              echo "<summary>📄 Research Result (click to expand)</summary>"
+              echo ""
+              echo '```markdown'
+              # Get first 5000 chars of research output
+              head -c 5000 tests/puppeteer/research-output/research-result.txt
+              TOTAL_SIZE=$(wc -c < tests/puppeteer/research-output/research-result.txt)
+              if [ "$TOTAL_SIZE" -gt 5000 ]; then
+                echo ""
+                echo "... [truncated, total $TOTAL_SIZE chars]"
+              fi
+              echo '```'
+              echo "</details>"
+              echo ""
+            fi
+
+            echo "---"
+            echo ""
+            echo "<details>"
+            echo "<summary>📋 Full Test Output (click to expand)</summary>"
+            echo ""
+            echo '```'
+            # Get last 100 lines of test output (most relevant)
+            if [ -f tests/puppeteer/test-output.log ]; then
+              tail -100 tests/puppeteer/test-output.log
+            fi
+            echo '```'
+            echo "</details>"
+            echo ""
+            echo "---"
+            echo ""
+            echo "**Configuration:**"
+            echo "- Model: \`google/gemini-2.5-flash-lite\` (via OpenRouter)"
+            echo "- Search: \`serper\`"
+            echo "- Test suite: \`all\`"
+            echo ""
+            echo "_Triggered by: \`${EVENT_NAME}\` (label: \`${LABEL_NAME}\`)_"
+          } > comment.md
+
+      - name: Post or update PR comment
+        if: always() && steps.find-pr.outputs.pr_number != ''
+        continue-on-error: true   # required release gate must not fail on a GitHub API hiccup
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        env:
+          PR_NUMBER: ${{ steps.find-pr.outputs.pr_number }}
+        with:
+          retries: 3
+          script: |
+            const fs = require('fs');
+            const issue_number = Number(process.env.PR_NUMBER);
+            const marker = '<!-- puppeteer-e2e-results -->';
+            const body = fs.readFileSync('comment.md', 'utf8');
+
+            // Paginate — on a long-lived PR the existing comment may be past the
+            // first page (default 30), which would otherwise create a new comment
+            // every run and reintroduce the spam this fix removes (#3884).
+            const comments = await github.paginate(github.rest.issues.listComments, {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number,
+              per_page: 100,
+            });
+            const existing = comments.find(
+              (c) => c.user.type === 'Bot' && c.body.includes(marker)
+            );
+
+            if (existing) {
+              await github.rest.issues.updateComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                comment_id: existing.id,
+                body,
+              });
+            } else {
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number,
+                body,
+              });
+            }
+
+      - name: Remove label for re-triggering
+        if: always() && github.event_name == 'pull_request'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUM: ${{ github.event.pull_request.number }}
+        run: |
+          gh issue edit "$PR_NUM" --remove-label "$LABEL_NAME" 2>/dev/null || true
+
+      - name: Cleanup
+        if: always()
+        run: |
+          # Kill server
+          pkill -f 'python -m local_deep_research.web.app' || true
+          rm -f comment.md

.github/workflows/responsive-ui-tests-enhanced.yml

@@ -0,0 +1,375 @@
+name: Responsive UI Tests
+
+on:
+  workflow_call:
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow runs only via workflow_call (from release.yml's
+# responsive-test-gate) and workflow_dispatch. The pull_request trigger
+# was deliberately removed in #2248 to reduce PR CI load on what is a
+# heavy ~20-minute matrix build (mobile + desktop). A shared concurrency
+# key here previously caused workflow_call invocations to cancel each
+# other mid-flight; see #3554 (reverted in #3599) for that history.
+
+permissions:
+  contents: read
+
+jobs:
+  ui-tests:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+
+    strategy:
+      fail-fast: false
+      matrix:
+        viewport: [mobile, desktop]  # Test mobile and desktop on each PR
+
+    services:
+      postgres:
+        image: postgres:14@sha256:ca25035f7e6f74552655a1c5e4a9eb21f85e9d316f1f70371f790ef70095dd58 # v14
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: ldr_test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Set up Node.js
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: '24'
+          cache: 'npm'
+          cache-dependency-path: tests/ui_tests/package-lock.json
+
+      - name: Free up disk space
+        run: |
+          # Remove unnecessary large packages to prevent disk space issues during cache save
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /usr/local/lib/android
+          sudo rm -rf /opt/ghc
+          sudo rm -rf /opt/hostedtoolcache/CodeQL
+          df -h
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y \
+            wget \
+            gnupg \
+            ca-certificates \
+            fonts-liberation \
+            libasound2t64 \
+            libatk-bridge2.0-0 \
+            libatk1.0-0 \
+            libcups2 \
+            libdbus-1-3 \
+            libdrm2 \
+            libgbm1 \
+            libgtk-3-0 \
+            libnspr4 \
+            libnss3 \
+            libx11-xcb1 \
+            libxcomposite1 \
+            libxdamage1 \
+            libxrandr2 \
+            xdg-utils \
+            imagemagick
+
+      - name: Install Python dependencies
+        run: pdm install
+
+      - name: Install root frontend dependencies
+        run: npm ci
+
+      - name: Build Vite frontend bundle
+        # Generates src/local_deep_research/web/static/dist/, which the
+        # Flask app loads via vite_helper. Without this the responsive UI
+        # tests run against an unstyled page (no styles.css), which means
+        # any CSS source changes between PRs are invisible to the test
+        # baseline. See follow-up to PR #3985 for context.
+        run: npm run build
+
+      - name: Install Node test dependencies
+        working-directory: tests/ui_tests
+        run: |
+          npm ci
+          # Keep this version in sync with tests/ui_tests/package-lock.json's
+          # puppeteer entry. A mismatch makes npx fetch a second puppeteer
+          # that targets a different Chrome build, corrupting the browser
+          # cache ("folder exists but executable is missing"). Matching the
+          # locked version lets npx reuse the puppeteer just installed above.
+          npx puppeteer@25.1.0 browsers install chrome
+
+      - name: Set up test directories
+        run: |
+          mkdir -p ${{ github.workspace }}/data/encrypted_databases
+          mkdir -p tests/ui_tests/screenshots
+          echo "Created data and screenshots directories for tests"
+
+      - name: Start test server
+        env:
+          CI: true
+          FLASK_ENV: testing
+          TEST_ENV: true
+          SECRET_KEY: test-secret-key-for-ci  # Security: CI test credential, not production secret
+          LDR_DISABLE_RATE_LIMITING: true
+          LDR_DATA_DIR: ${{ github.workspace }}/data
+          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/ldr_test  # Security: CI test database credentials, not production secrets
+        run: |
+          cd src
+          # Start server and get its PID
+          pdm run python -m local_deep_research.web.app > server.log 2>&1 &
+          SERVER_PID=$!
+          echo "Server PID: $SERVER_PID"
+          # Persist the PID for the "Stop application server" step. That step
+          # runs at the workspace root (no working-directory) while we are in
+          # src/ here, so write an absolute path — otherwise the pidfile never
+          # lines up and cleanup is a silent no-op.
+          echo "$SERVER_PID" > "$GITHUB_WORKSPACE/server.pid"
+
+          # Wait for server to start
+          SERVER_READY=false
+          for i in {1..60}; do
+            # Probe the health endpoint, not bare `/`: `/` 302-redirects to
+            # /auth/login and `curl -f` treats a 302 as success the instant
+            # the socket binds — before DB/app init finishes — so it can pass
+            # while the app is still not ready. /api/v1/health is a true
+            # readiness signal and matches the docker-tests.yml gate.
+            if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health 2>/dev/null; then
+              echo "Server is ready after $i seconds"
+              SERVER_READY=true
+              break
+            fi
+            # Check if process is still running
+            if ! kill -0 "$SERVER_PID" 2>/dev/null; then
+              echo "Server process died!"
+              echo "Server log:"
+              cat server.log
+              exit 1
+            fi
+            echo "Waiting for server... ($i/60)"
+            sleep 1
+          done
+
+          # Fail fast if the server is alive but never became ready in time.
+          # Without this guard the loop fell through silently on a slow-but-
+          # not-crashed start, so "Register CI test user" and the UI tests ran
+          # against a server that was not yet listening and failed with a
+          # confusing `net::ERR_CONNECTION_REFUSED` in the *test* step instead
+          # of a clear startup failure here. Window was also bumped 30s -> 60s
+          # to match the proven puppeteer-e2e-tests.yml / docker-tests.yml
+          # startup gates (Postgres-backed boot can exceed 30s on a loaded
+          # runner).
+          if [ "$SERVER_READY" != "true" ]; then
+            echo "❌ Server did not become ready within 60 seconds"
+            echo "Server log:"
+            cat server.log
+            exit 1
+          fi
+
+      - name: Register CI test user
+        working-directory: tests/ui_tests
+        run: node register_ci_user.js http://127.0.0.1:5000
+
+      - name: Run responsive UI tests - ${{ matrix.viewport }}
+        id: run-tests
+        working-directory: tests/ui_tests
+        env:
+          VIEWPORT: ${{ matrix.viewport }}
+        run: |
+          set +e  # Don't exit on test failure
+
+          # Run tests and capture output
+          HEADLESS=true node test_responsive_ui_comprehensive.js "$VIEWPORT" 2>&1 | tee test-output.log
+          TEST_EXIT_CODE="${PIPESTATUS[0]}"
+
+          # Extract summary for PR comment
+          echo "### 📱 $VIEWPORT Test Results" > test-summary.md
+          echo "" >> test-summary.md
+
+          # Extract pass/fail counts
+          PASSED=$(grep -oP '\d+(?= passed)' test-output.log | tail -1 || echo "0")
+          FAILED=$(grep -oP '\d+(?= failed)' test-output.log | tail -1 || echo "0")
+          WARNINGS=$(grep -oP '\d+(?= warnings)' test-output.log | tail -1 || echo "0")
+
+          if [ "$FAILED" -eq "0" ]; then
+            echo "✅ **All tests passed!**" >> test-summary.md
+          else
+            echo "❌ **$FAILED critical issues found**" >> test-summary.md
+          fi
+
+          {
+            echo ""
+            echo "- ✅ Passed: $PASSED"
+            echo "- ❌ Failed: $FAILED"
+            echo "- ⚠️ Warnings: $WARNINGS"
+          } >> test-summary.md
+
+          {
+            echo "test_exit_code=$TEST_EXIT_CODE"
+            echo "test_failed=$FAILED"
+          } >> "$GITHUB_OUTPUT"
+
+          # Only fail if critical failures
+          exit "$TEST_EXIT_CODE"
+
+      - name: Upload viewport test results
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ui-test-results-${{ matrix.viewport }}
+          path: |
+            tests/ui_tests/test-output.log
+            tests/ui_tests/test-summary.md
+            tests/ui_tests/screenshots/
+            tests/ui_tests/responsive/
+            src/server.log
+          retention-days: 7
+          if-no-files-found: ignore
+
+      # NOTE: Earlier screenshot generation/upload steps were removed to fix actionlint warnings
+      # caused by "if: false" disablement. Screenshots are now captured into the artifact above
+      # when they exist. Visual regression workflows can re-enable dedicated screenshot steps later.
+
+      - name: Upload to GitHub Pages (if available)
+        if: always() && github.event_name == 'pull_request'
+        continue-on-error: true
+        env:
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          VIEWPORT: ${{ matrix.viewport }}
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_REPOSITORY: ${{ github.repository }}
+          GH_REPO_OWNER: ${{ github.repository_owner }}
+          GH_REPO_NAME: ${{ github.event.repository.name }}
+        run: |
+          # This requires GitHub Pages to be enabled for the repo
+          # Create a branch for the screenshots
+          BRANCH_NAME="pr-screenshots-$PR_NUMBER-$VIEWPORT"
+
+          # Check if screenshots directory exists
+          if [ ! -d "tests/ui_tests/responsive" ]; then
+            echo "No screenshots directory found, skipping upload"
+            exit 0
+          fi
+
+          cd tests/ui_tests/responsive
+          git init
+          git config user.name "GitHub Actions"
+          git config user.email "actions@github.com"
+
+          # Add all screenshots and gallery
+          git add responsive-ui-tests screenshot-gallery.html
+          git commit -m "Screenshots for PR #$PR_NUMBER"
+
+          # Push to a dedicated branch (requires write permissions)
+          git push --force "https://x-access-token:$GH_TOKEN@github.com/$GH_REPOSITORY.git" "HEAD:$BRANCH_NAME" 2>/dev/null || true
+
+          # The URL would be: https://[owner].github.io/[repo]/pr-screenshots-[number]-[viewport]/screenshot-gallery.html
+          echo "Screenshots available at: https://$GH_REPO_OWNER.github.io/$GH_REPO_NAME/$BRANCH_NAME/screenshot-gallery.html" || true
+
+      - name: Stop application server
+        if: always()
+        run: |
+          if [ -f server.pid ]; then
+            kill "$(cat server.pid)" || true
+            rm server.pid
+          fi
+          # `pdm run` stays alive as a wrapper, so the pidfile holds the
+          # wrapper PID and killing it can orphan the actual python server.
+          # Add the same pkill fallback the webkit workflow uses so the
+          # python process (and port 5000) is actually released.
+          pkill -f "python -m local_deep_research.web.app" || true
+
+  post-results:
+    needs: ui-tests
+    runs-on: ubuntu-latest
+    # Workflow has no pull_request trigger; gating on event_name == 'pull_request' kept this job
+    # from ever running. Always run so the combined report is built for workflow_dispatch / release.
+    if: always()
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Download all artifacts
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          path: test-artifacts/
+
+      - name: Generate combined report
+        run: |
+          # Create a combined markdown report with screenshot links
+          {
+            echo "# 📊 Responsive UI Test Report"
+            echo ""
+            echo "## Test Summary"
+            echo ""
+          } > combined-report.md
+
+          # Process each viewport's results
+          for dir in test-artifacts/ui-test-results-*/; do
+            if [ -d "$dir" ]; then
+              viewport=$(basename "$dir" | sed 's/ui-test-results-//')
+              echo "### $viewport" >> combined-report.md
+
+              if [ -f "$dir/test-summary.md" ]; then
+                cat "$dir/test-summary.md" >> combined-report.md
+              fi
+
+              # Count screenshots
+              screenshot_count=$(find "$dir" -name "*.png" 2>/dev/null | wc -l)
+              if [ "$screenshot_count" -gt "0" ]; then
+                {
+                  echo ""
+                  echo "📸 **$screenshot_count screenshots captured**"
+                } >> combined-report.md
+              fi
+
+              echo "" >> combined-report.md
+            fi
+          done
+
+          {
+            echo "## 📥 Download Options"
+            echo ""
+            echo "- [Download all test artifacts](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})"
+            echo "- View artifacts in the 'Artifacts' section below the workflow summary"
+          } >> combined-report.md
+
+      - name: Upload combined report
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: combined-test-report
+          path: combined-report.md
+          retention-days: 30

.github/workflows/retirejs.yml

@@ -0,0 +1,86 @@
+name: Retire.js Security Scan
+
+on:
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+  schedule:
+    # Run weekly security scan on Mondays at 4 AM UTC
+    - cron: '0 4 * * 1'
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+# Note: No concurrency block - this workflow is called by release-gate.yml
+# and concurrency settings can cause unexpected cancellations for reusable workflows
+
+jobs:
+  retirejs:
+    name: Retire.js Vulnerability Scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    permissions:
+      contents: read
+      security-events: write
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Setup Node.js
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: '24'
+
+      - name: Run Retire.js scan
+        id: retire-scan
+        run: |
+          echo "=== Running Retire.js scan ==="
+          # Use npx with specific version for reproducible builds
+          # Scan JavaScript files and node_modules for known vulnerabilities
+          # Only flag medium severity and above (exit code 13 when issues found)
+          # Output both SARIF (for GitHub Security) and human-readable format
+          npx retire@5.2.4 --js --node --severity medium --outputformat sarif --outputpath retire-results.sarif 2>&1 | tee retire-output.txt
+          RETIRE_EXIT_CODE=$?
+
+          # Check if vulnerabilities were found (exit code 13 indicates issues)
+          if [ $RETIRE_EXIT_CODE -eq 13 ]; then
+            echo "RETIRE_FOUND_ISSUES=true" >> "$GITHUB_ENV"
+          fi
+
+          # Display human-readable summary
+          echo ""
+          echo "=== Scan Summary ==="
+          cat retire-output.txt
+
+          # Check if SARIF file was created (for use in upload step)
+          if [ -f "retire-results.sarif" ]; then
+            echo "sarif_exists=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Upload SARIF results
+        uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+        if: always() && steps.retire-scan.outputs.sarif_exists == 'true'
+        with:
+          sarif_file: retire-results.sarif
+          category: retirejs
+
+      - name: Check scan results
+        run: |
+          if [[ "$RETIRE_FOUND_ISSUES" == "true" ]]; then
+            echo "::error::Retire.js found medium or higher severity vulnerabilities"
+            echo "Review the Security tab for details"
+            echo "Consider updating or replacing affected libraries"
+            # Only fail for PR/push events, not scheduled scans
+            if [[ "${{ github.event_name }}" != "schedule" ]]; then
+              exit 1
+            fi
+            echo "::warning::Scheduled scan - reporting only, not failing the workflow"
+          else
+            echo "✅ No medium or higher severity vulnerabilities detected"
+          fi

.github/workflows/sbom.yml

@@ -0,0 +1,81 @@
+name: SBOM Generation
+
+on:
+  workflow_dispatch:
+  schedule:
+    # Run weekly on Wednesday at 10 AM UTC (staggered with other scans)
+    - cron: '0 10 * * 3'
+  release:
+    types: [published]
+
+permissions: {}
+
+jobs:
+  sbom:
+    name: Generate Software Bill of Materials
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout repository
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Generate SBOM for source code
+      uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
+      with:
+        path: .
+        artifact-name: sbom-source.spdx.json
+        output-file: sbom-source.spdx.json
+        format: spdx-json
+
+    - name: Generate SBOM for Python dependencies
+      uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
+      with:
+        path: .
+        artifact-name: sbom-python.cyclonedx.json
+        output-file: sbom-python.cyclonedx.json
+        format: cyclonedx-json
+
+    - name: Upload SBOMs as artifacts
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      with:
+        name: sbom-reports
+        path: |
+          sbom-source.spdx.json
+          sbom-python.cyclonedx.json
+        retention-days: 90
+
+    - name: Display SBOM summary
+      run: |
+        {
+          echo "## SBOM Generation Summary"
+          echo ""
+          echo "### Generated SBOMs"
+          echo "- **Source Code SBOM**: \`sbom-source.spdx.json\` (SPDX format)"
+          echo "- **Dependencies SBOM**: \`sbom-python.cyclonedx.json\` (CycloneDX format)"
+          echo ""
+          echo "### What is an SBOM?"
+          echo "A Software Bill of Materials (SBOM) is a formal record of all components,"
+          echo "libraries, and dependencies used in building software. It enables:"
+          echo ""
+          echo "- **Vulnerability tracking**: Know exactly what's in your software"
+          echo "- **License compliance**: Verify all dependencies have compatible licenses"
+          echo "- **Supply chain security**: Detect compromised dependencies"
+          echo "- **Regulatory compliance**: Meet requirements (EO 14028, etc.)"
+          echo ""
+          echo "### Artifacts"
+          echo "SBOMs are available as workflow artifacts for 90 days."
+          if [ "${{ github.event_name }}" = "release" ]; then
+            echo ""
+            echo "SBOMs have also been attached to the release assets."
+          fi
+        } >> "$GITHUB_STEP_SUMMARY"

.github/workflows/security-file-write-check.yml

@@ -0,0 +1,34 @@
+name: Security File Write Check
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+  workflow_call:  # Called by ci-gate.yml for release pipeline
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions:
+  contents: read
+
+jobs:
+  check-file-writes:
+    name: Check for Unencrypted File Writes
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Check for potential unencrypted file writes
+        run: ./.github/scripts/check-file-writes.sh

.github/workflows/security-headers-validation.yml

@@ -0,0 +1,194 @@
+name: Security Headers Validation
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  schedule:
+    - cron: '0 3 * * *'  # Daily scan for early detection
+  workflow_dispatch:  # Manual trigger for debugging/verification
+
+permissions:
+  contents: read
+
+# NOTE: No concurrency block here. When called via workflow_call from
+# release-gate.yml, the caller's workflow name is used for the concurrency
+# group, which can cause this job to be cancelled if another release gate
+# run starts on the same ref. The parent workflow manages concurrency.
+
+jobs:
+  validate-headers:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+
+    services:
+      postgres:
+        image: postgres:13@sha256:4689940c683801b4ab839ab3b0a0a3555a5fe425371422310944e89eca7d8068 # v13
+        env:
+          POSTGRES_USER: ldr_test
+          POSTGRES_PASSWORD: ${{ github.run_id }}_test_pwd
+          POSTGRES_DB: test_db
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Cache pip packages
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-headers-${{ hashFiles('**/pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-headers-
+            ${{ runner.os }}-pip-
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libsqlcipher-dev build-essential
+
+      - name: Install dependencies
+        run: |
+          pdm install --dev
+
+      - name: Set up test environment
+        run: |
+          mkdir -p data
+          mkdir -p research_outputs
+          cp -r src/local_deep_research/defaults/.env.template .env.test
+
+      - name: Start Flask application in background
+        run: |
+          # Use FLASK_DEBUG=0 (FLASK_ENV is deprecated)
+          export FLASK_DEBUG=0
+          export DATABASE_URL=postgresql://ldr_test:${{ github.run_id }}_test_pwd@localhost:5432/test_db
+          pdm run python -m local_deep_research.web.app &
+          echo $! > flask.pid
+
+      - name: Wait for application to be ready
+        run: |
+          timeout 60 bash -c 'until curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; do sleep 2; done'
+
+      - name: Validate security headers
+        run: |
+          echo "=== Testing Security Headers ==="
+          echo ""
+
+          # Capture headers once for efficiency (with timeout)
+          HEADERS=$(curl -sI --max-time 10 http://localhost:5000/)
+
+          echo "Full response headers:"
+          echo "$HEADERS"
+          echo ""
+
+          # Check for required security headers and validate values
+          ERRORS=()
+
+          # X-Frame-Options - must be SAMEORIGIN or DENY
+          XFO=$(echo "$HEADERS" | grep -i "^X-Frame-Options:" | cut -d':' -f2- | tr -d '\r' | xargs)
+          if [[ -z "$XFO" ]]; then
+            ERRORS+=("X-Frame-Options: MISSING")
+          elif [[ "$XFO" != "SAMEORIGIN" && "$XFO" != "DENY" ]]; then
+            ERRORS+=("X-Frame-Options: Invalid value '$XFO' (expected SAMEORIGIN or DENY)")
+          else
+            echo "✅ X-Frame-Options: $XFO"
+          fi
+
+          # X-Content-Type-Options - must be nosniff
+          XCTO=$(echo "$HEADERS" | grep -i "^X-Content-Type-Options:" | cut -d':' -f2- | tr -d '\r' | xargs)
+          if [[ -z "$XCTO" ]]; then
+            ERRORS+=("X-Content-Type-Options: MISSING")
+          elif [[ "$XCTO" != "nosniff" ]]; then
+            ERRORS+=("X-Content-Type-Options: Invalid value '$XCTO' (expected nosniff)")
+          else
+            echo "✅ X-Content-Type-Options: $XCTO"
+          fi
+
+          # Referrer-Policy - must not be unsafe
+          RP=$(echo "$HEADERS" | grep -i "^Referrer-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
+          if [[ -z "$RP" ]]; then
+            ERRORS+=("Referrer-Policy: MISSING")
+          elif [[ "$RP" == "unsafe-url" || "$RP" == "no-referrer-when-downgrade" ]]; then
+            ERRORS+=("Referrer-Policy: Insecure value '$RP'")
+          else
+            echo "✅ Referrer-Policy: $RP"
+          fi
+
+          # Permissions-Policy - must restrict dangerous features
+          PP=$(echo "$HEADERS" | grep -i "^Permissions-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
+          if [[ -z "$PP" ]]; then
+            ERRORS+=("Permissions-Policy: MISSING")
+          elif [[ "$PP" != *"geolocation=()"* ]]; then
+            ERRORS+=("Permissions-Policy: Should restrict geolocation")
+          else
+            echo "✅ Permissions-Policy: $PP"
+          fi
+
+          # Content-Security-Policy - must have default-src and not be overly permissive
+          CSP=$(echo "$HEADERS" | grep -i "^Content-Security-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
+          if [[ -z "$CSP" ]]; then
+            ERRORS+=("Content-Security-Policy: MISSING")
+          elif [[ "$CSP" == *"default-src *"* || "$CSP" == *"default-src 'unsafe-inline' 'unsafe-eval'"* ]]; then
+            ERRORS+=("Content-Security-Policy: Overly permissive policy detected")
+          elif [[ "$CSP" != *"default-src"* ]]; then
+            ERRORS+=("Content-Security-Policy: Missing default-src directive")
+          else
+            echo "✅ Content-Security-Policy: Present with default-src"
+          fi
+
+          # Report results
+          echo ""
+          if [ ${#ERRORS[@]} -eq 0 ]; then
+            echo "✅ All required security headers are present and valid"
+          else
+            echo "❌ Security header issues found:"
+            for error in "${ERRORS[@]}"; do
+              echo "  - $error"
+            done
+            exit 1
+          fi
+
+      - name: Test API endpoint headers
+        run: |
+          echo "=== Testing Security Headers on API Endpoints ==="
+          echo ""
+
+          # Test health endpoint (with timeout)
+          API_HEADERS=$(curl -sI --max-time 10 http://localhost:5000/api/health)
+
+          if echo "$API_HEADERS" | grep -iq "^X-Frame-Options:"; then
+            echo "✅ Security headers present on API endpoints"
+          else
+            echo "❌ Security headers missing on API endpoints"
+            echo "$API_HEADERS"
+            exit 1
+          fi
+
+      - name: Cleanup
+        if: always()
+        run: |
+          if [ -f flask.pid ]; then
+            kill "$(cat flask.pid)" 2>/dev/null || true
+            rm -f flask.pid
+          fi

.github/workflows/security-tests.yml

@@ -0,0 +1,137 @@
+name: Security Tests
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  security-events: write  # Required for uploading SARIF results to Code Scanning
+
+jobs:
+  security-tests:
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+
+    services:
+      postgres:
+        image: postgres:13@sha256:1094e2cdc5605e5c7914633bcd93758a9c52ae8c8b2855ddd1c3a8afbe4795d5 # postgres:13
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: test_security_db
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+        ports:
+          - 5432:5432
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      with:
+        persist-credentials: false
+
+    - name: Set up Python 3.12
+      uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+      with:
+        python-version: '3.12'
+
+    - name: Cache pip packages
+      uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-security-${{ hashFiles('**/requirements.txt') }}
+        restore-keys: |
+          ${{ runner.os }}-pip-security-
+          ${{ runner.os }}-pip-
+
+    - name: Set up PDM
+      uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+      with:
+        python-version: '3.12'
+
+    - name: Install system dependencies for SQLCipher
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y libsqlcipher-dev
+
+    - name: Install dependencies
+      run: |
+        pdm sync -d
+        pdm add "bandit[sarif]" safety sqlparse pytest pytest-cov --no-sync
+        pdm sync
+
+    - name: Run Bandit security linter
+      run: |
+        # --exit-zero: report findings without failing (findings go to SARIF/Security tab)
+        # If bandit crashes (bad config, missing files), it exits non-zero WITHOUT --exit-zero
+        # and won't produce output files — the checks below catch that.
+        pdm run bandit -r src/ -f json -o bandit-report.json -lll --exit-zero
+        pdm run bandit -r src/ -f sarif -o bandit-results.sarif -lll --exit-zero
+        # Crash detection: if bandit crashed, no output files were produced
+        if [ ! -f bandit-report.json ] || [ ! -f bandit-results.sarif ]; then
+          echo "::error::Bandit crashed — expected output files not produced"
+          exit 1
+        fi
+        echo "Bandit security scan completed"
+
+    - name: Upload Bandit SARIF to Code Scanning
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      if: always()
+      with:
+        sarif_file: bandit-results.sarif
+        category: bandit
+
+    # Dependency vulnerability scanning is handled by OSV-Scanner
+    # (osv-scanner.yml), which reads pdm.lock directly and doesn't
+    # fight PDM's resolution overrides the way pip-audit did.
+
+    - name: Run security test suite
+      env:
+        DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test_security_db
+      run: |
+        # Run the entire tests/security/ tree as a directory rather than
+        # enumerating individual files. The old per-file list only covered
+        # ~9 of the 87 test files and silently broke whenever a file was
+        # renamed or deleted: a hardcoded reference to the deleted
+        # test_input_validation.py (removed in #4243) made pytest exit with
+        # code 5 ("no tests collected") and failed this gate (#4411).
+        # Running the directory picks up new security tests automatically and
+        # never rots when files move.
+        pdm run python -m pytest tests/security/ -v --tb=short -n auto
+
+    - name: Check for hardcoded secrets
+      run: |
+        # Check for potential secrets in code
+        grep -r -E "(api[_-]?key|secret[_-]?key|password|token)" src/ --include="*.py" | \
+          grep -v -E "(os\.environ|getenv|config\[|placeholder|example|test)" | \
+          grep -E "=\s*['\"]" || echo "No hardcoded secrets found"
+
+    - name: Generate security report
+      if: always()
+      run: |
+        echo "Security Test Report"
+        echo "==================="
+        echo ""
+        if [ -f bandit-report.json ]; then
+          echo "Bandit Security Issues:"
+          python -c "import json; data=json.load(open('bandit-report.json')); print(f'  High: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"HIGH\"])}'); print(f'  Medium: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"MEDIUM\"])}'); print(f'  Low: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"LOW\"])}')" || true
+        fi
+        echo ""
+        echo "Dependency vulnerabilities: covered by OSV-Scanner (osv-scanner.yml)"
+
+    - name: Upload security reports
+      if: always()
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      with:
+        name: security-reports
+        path: |
+          bandit-report.json
+          bandit-results.sarif

.github/workflows/semgrep.yml

@@ -0,0 +1,251 @@
+name: Semgrep Security Scan
+
+on:
+  workflow_call:  # Called by release-gate.yml
+  workflow_dispatch:
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  semgrep-scan:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+    - name: Harden the runner (Audit all outbound calls)
+      uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+      with:
+        egress-policy: audit
+
+    - name: Checkout code
+      uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+      with:
+        persist-credentials: false
+
+    - name: Set up Python
+      uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+      with:
+        python-version: '3.12'
+
+    - name: Install Semgrep
+      run: |
+        # setuptools required: semgrep 1.87.0's opentelemetry dep imports pkg_resources,
+        # which is not bundled with Python 3.12 by default on GitHub runners.
+        # Pin setuptools<82 because 82.0 removed the pkg_resources module.
+        pip install "setuptools<82" semgrep==1.87.0
+
+    - name: Run Semgrep security scan
+      run: |
+        semgrep \
+          --config=p/security-audit \
+          --config=p/secrets \
+          --severity=INFO \
+          --json \
+          --output=semgrep-results.json \
+          src/ || true
+        # Crash detection: if semgrep crashed, no output file was produced
+        if [ ! -f semgrep-results.json ]; then
+          echo "::error::Semgrep security scan crashed — no output produced"
+          exit 1
+        fi
+
+    - name: Run Semgrep custom rules for LDR
+      run: |
+        semgrep \
+          --config=.semgrep/rules/ \
+          --severity=INFO \
+          --json \
+          --output=semgrep-custom-results.json \
+          src/ || true
+        # Crash detection: if semgrep crashed, no output file was produced
+        if [ ! -f semgrep-custom-results.json ]; then
+          echo "::error::Semgrep custom rules scan crashed — no output produced"
+          exit 1
+        fi
+
+    - name: Merge Semgrep results
+      run: |
+        python3 -c "
+        import json
+        import glob
+
+        results = []
+        for file in glob.glob('semgrep-*.json'):
+            try:
+                with open(file) as f:
+                    data = json.load(f)
+                    if isinstance(data, dict) and 'results' in data:
+                        results.extend(data['results'])
+                    elif isinstance(data, list):
+                        results.extend(data)
+            except Exception as e:
+                print(f'Error reading {file}: {e}')
+
+        output = {'results': results, 'version': '1.0.0'}
+        with open('semgrep-combined-results.json', 'w') as f:
+            json.dump(output, f, indent=2)
+
+        print(f'Combined {len(results)} findings from Semgrep scans')
+        "
+
+    - name: Convert to SARIF format
+      run: |
+        python3 -c "
+        import json
+        import uuid
+        from datetime import datetime
+
+        # Load combined results
+        with open('semgrep-combined-results.json') as f:
+            semgrep_data = json.load(f)
+
+        # Convert to SARIF
+        sarif = {
+            '\$schema': 'https://json.schemastore.org/sarif-2.1.0',
+            'version': '2.1.0',
+            'runs': [{
+                'tool': {
+                    'driver': {
+                        'name': 'Semgrep',
+                        'version': '1.87.0',
+                        'informationUri': 'https://semgrep.dev'
+                    }
+                },
+                'results': []
+            }]
+        }
+
+        for result in semgrep_data.get('results', []):
+            sarif_result = {
+                'ruleId': result.get('check_id', 'unknown'),
+                'message': {
+                    'text': result.get('message', 'Security issue detected')
+                },
+                'level': 'warning' if result.get('metadata', {}).get('severity', 'INFO') in ['ERROR', 'WARNING'] else 'note',
+                'locations': [{
+                    'physicalLocation': {
+                        'artifactLocation': {
+                            'uri': result.get('path', 'unknown')
+                        },
+                        'region': {
+                            'startLine': result.get('start', {}).get('line', 1),
+                            'startColumn': result.get('start', {}).get('col', 1),
+                            'endLine': result.get('end', {}).get('line', result.get('start', {}).get('line', 1)),
+                            'endColumn': result.get('end', {}).get('col', result.get('start', {}).get('col', 1) + 1)
+                        }
+                    }
+                }]
+            }
+
+            # Add rule information
+            metadata = result.get('metadata', {})
+            sarif_result['rule'] = {
+                'id': result.get('check_id', 'unknown'),
+                'name': metadata.get('name', 'Security Issue'),
+                'shortDescription': {
+                    'text': metadata.get('name', 'Security Issue')
+                },
+                'fullDescription': {
+                    'text': metadata.get('description', 'Security vulnerability detected')
+                },
+                'help': {
+                    'text': metadata.get('remediation', 'Review and fix the security issue')
+                },
+                'properties': {
+                    'precision': 'medium',
+                    'tags': ['security', 'semgrep']
+                }
+            }
+
+            if 'security-severity' in metadata:
+                sarif_result['rule']['properties']['security-severity'] = metadata['security-severity']
+
+            sarif['runs'][0]['results'].append(sarif_result)
+
+        # Write SARIF file
+        with open('semgrep-results.sarif', 'w') as f:
+            json.dump(sarif, f, indent=2)
+
+        print(f'Converted {len(sarif[\"runs\"][0][\"results\"])} findings to SARIF format')
+        "
+
+    # Fail loudly if the SARIF conversion produced no file — never fabricate an
+    # empty one. An empty-results SARIF uploaded under the semgrep-security
+    # category would make GitHub mark every previously-open Semgrep alert as
+    # fixed, silently clearing real findings. The "Convert to SARIF" step above
+    # always writes the file (results:[] on a clean scan), so a missing file
+    # means a real conversion failure. Mirror the Grype/Trivy jobs.
+    - name: Ensure SARIF file exists
+      id: check-sarif
+      if: always()
+      run: |
+        if [ -f semgrep-results.sarif ]; then
+          echo "exists=true" >> "$GITHUB_OUTPUT"
+        else
+          echo "::error::Semgrep SARIF conversion did not produce a file — scan needs to be rerun"
+          exit 1
+        fi
+
+    - name: Upload Semgrep results to GitHub Security tab
+      uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
+      if: always() && steps.check-sarif.outputs.exists == 'true'
+      with:
+        sarif_file: 'semgrep-results.sarif'
+        category: semgrep-security
+
+    - name: Upload Semgrep results as artifact
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+      if: always()
+      with:
+        name: semgrep-scan-results
+        path: |
+          semgrep-results.json
+          semgrep-custom-results.json
+          semgrep-combined-results.json
+          semgrep-results.sarif
+        retention-days: 7  # Reduced for security
+
+    - name: Display Semgrep summary
+      if: always()
+      run: |
+        {
+          echo "## Semgrep Security Scan Summary"
+          echo ""
+        } >> "$GITHUB_STEP_SUMMARY"
+
+        if [ -f semgrep-combined-results.json ]; then
+          # Count results by severity
+          CRITICAL=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'ERROR']))" 2>/dev/null || echo "0")
+          HIGH=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'WARNING']))" 2>/dev/null || echo "0")
+          MEDIUM=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'INFO']))" 2>/dev/null || echo "0")
+          TOTAL=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len(data['results']))" 2>/dev/null || echo "0")
+
+          {
+            echo "📊 **Scan Results:**"
+            echo "- **Critical:** $CRITICAL"
+            echo "- **High:** $HIGH"
+            echo "- **Medium:** $MEDIUM"
+            echo "- **Total:** $TOTAL"
+            echo ""
+          } >> "$GITHUB_STEP_SUMMARY"
+
+          if [ "$CRITICAL" -gt 0 ] || [ "$HIGH" -gt 0 ]; then
+            echo "⚠️ **Action Required:** Critical or High severity issues found" >> "$GITHUB_STEP_SUMMARY"
+          else
+            echo "✅ **No Critical or High severity issues found**" >> "$GITHUB_STEP_SUMMARY"
+          fi
+
+          {
+            echo ""
+            echo "📋 **Detailed Results:**"
+            echo "- Security tab: Results uploaded to GitHub Security tab"
+            echo "- Artifacts: Full JSON and SARIF reports available"
+          } >> "$GITHUB_STEP_SUMMARY"
+        else
+          echo "❌ Semgrep scan failed or no results generated" >> "$GITHUB_STEP_SUMMARY"
+        fi

.github/workflows/ui-full-shards.yml

@@ -0,0 +1,52 @@
+name: UI Tests (Full 14 Shards via Label)
+
+# Triggered by adding the `test:ui-full-shards` label to a PR.
+# Calls docker-tests.yml with strict-mode=true so the sharded ui-tests
+# matrix runs — it's otherwise gated on strict-mode and only fires at
+# release time.
+#
+# NOTE on duplicated work: docker-tests.yml in strict mode runs its FULL
+# job graph (image build, pytest, infra, etc.), so triggering this label
+# re-runs everything that the normal PR docker-tests already ran. That's
+# acceptable cost for a deliberate opt-in trigger; cheaper than refactoring
+# docker-tests.yml to expose just ui-tests as a standalone reusable workflow.
+
+on:
+  pull_request:
+    types: [labeled]
+  # Manual fallback — useful for one-off maintainer runs and for smoke-testing
+  # the wrapper when label-triggered workflows can't fire (e.g. on the PR that
+  # first introduces this file, before it lands on main).
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted, matching docker-tests.yml.
+# See its top-level comment for the rationale (#3554 / #3599).
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  full-shards:
+    name: Full UI Shards
+    # Fire on the label add, OR on any manual dispatch (maintainer override).
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'pull_request' &&
+       github.event.label.name == 'test:ui-full-shards')
+    uses: ./.github/workflows/docker-tests.yml
+    with:
+      strict-mode: true
+    permissions:
+      # MUST match the scopes docker-tests.yml's jobs DECLARE, not just the
+      # ones whose steps run. Reusable-workflow permission validation is
+      # compile-time: a called job that declares `contents: write` (the
+      # pytest/gh-pages job) fails the whole run at startup with
+      # "requesting 'contents: write', but is only allowed 'contents: none'"
+      # if the caller grants less — producing a 0-job startup_failure with
+      # no logs. #4209 dropped contents:write here for OSSF Scorecard, which
+      # silently broke this wrapper on every run since (the gh-pages STEP is
+      # runtime-gated to push@main, but the JOB still declares the scope).
+      # ci-gate.yml grants the same pair, which is why it works.
+      contents: write       # docker-tests jobs checkout + declare gh-pages deploy scope
+      pull-requests: write  # docker-tests pytest job posts PR comments
+    secrets:
+      OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}

.github/workflows/update-dependencies.yml

@@ -0,0 +1,73 @@
+name: Update PDM dependencies
+on:
+  workflow_call:
+    inputs:
+      pdm_args:
+        description: Arguments to pass to pdm
+        type: string
+        default: ''
+  workflow_dispatch:
+    inputs:
+      pdm_args:
+        description: Arguments to pass to pdm lock
+        type: string
+        default: ''
+  schedule:
+    - cron: '0 8 * * 3'   # every Wednesday at 08:00 UTC
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  update:
+    name: 👛 Update with PDM
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write  # Required for creating PRs
+      pull-requests: write  # Required for creating PRs
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: 📰 Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          ref: main
+
+      - name: 📦 Setup PDM
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          # Pin to the floor of `requires-python` in pyproject.toml.
+          # Resolving on the lowest supported interpreter guarantees the
+          # chosen package versions are installable across the whole
+          # supported range. Resolving on a newer interpreter (e.g. 3.x)
+          # can pick packages that dropped support for the floor, producing
+          # a lockfile that fails `pdm lock --check` downstream (see #3480).
+          python-version: '3.12'
+          cache: true
+
+      - name: 👚 Update to latest compatible versions
+        env:
+          PDM_ARGS: ${{ inputs.pdm_args || '' }}
+        run: |
+          # Intentional word splitting for PDM args
+          # shellcheck disable=SC2086
+          pdm lock $PDM_ARGS
+
+      - name: 📝 Create pull request
+        uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
+        with:
+          branch: update-dependencies-${{ github.run_number }}
+          title: 🤖 Update dependencies
+          body: |
+            This PR updates dependencies to their latest compatible versions.
+            This PR was created by the PDM Update Bot.
+          sign-commits: true
+          add-paths: pdm.lock
+          commit-message: 🤖 Update dependencies
+          labels: maintenance
+          draft: false
+          base: main
+          reviewers: djpetti,HashedViking,LearningCircuit

.github/workflows/update-npm-dependencies.yml

@@ -0,0 +1,158 @@
+name: Update NPM dependencies
+on:
+  workflow_call:
+    inputs:
+      npm_args:
+        description: Arguments to pass to npm update
+        type: string
+        default: --save
+  workflow_dispatch:
+    inputs:
+      npm_args:
+        description: Arguments to pass to npm update
+        type: string
+        default: --save
+  schedule:
+    - cron: '0 8 * * 4'   # every Thursday at 08:00 UTC (day after PDM updates)
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  update:
+    name: 📦 Update with NPM
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write  # Required for creating PRs
+      pull-requests: write  # Required for creating PRs
+    strategy:
+      matrix:
+        npm_directory:
+          - { path: '.', build_cmd: 'npm run build', test_cmd: '', cache_path: 'package-lock.json' }
+          - { path: 'tests', build_cmd: '', test_cmd: 'echo "Skipping API-key tests in CI - requires real API key"', cache_path: 'package-lock.json' }
+          - { path: 'tests/ui_tests', build_cmd: '', test_cmd: 'echo "Skipping UI tests in CI - requires server"', cache_path: 'package-lock.json' }
+          - { path: 'tests/puppeteer', build_cmd: '', test_cmd: 'echo "Skipping Puppeteer tests in CI - requires server"', cache_path: 'package-lock.json' }
+          - { path: 'tests/api_tests_with_login', build_cmd: '', test_cmd: 'echo "Skipping API-with-login tests in CI - requires server + auth"', cache_path: 'package-lock.json' }
+          - { path: 'tests/infrastructure_tests', build_cmd: '', test_cmd: 'echo "Skipping infrastructure tests in CI - requires server"', cache_path: 'package-lock.json' }
+          - { path: 'tests/accessibility_tests', build_cmd: '', test_cmd: 'echo "Skipping accessibility tests in CI - requires server"', cache_path: 'package-lock.json' }
+          - { path: 'tests/ui_tests/playwright', build_cmd: '', test_cmd: 'echo "Skipping playwright UI tests in CI - requires server"', cache_path: 'package-lock.json' }
+    continue-on-error: false
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: 📰 Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          ref: main
+
+      - name: 🟢 Setup Node.js
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: '24'
+          cache: 'npm'
+          cache-dependency-path: ${{ matrix.npm_directory.path }}/${{ matrix.npm_directory.cache_path }}
+
+      - name: 🔍 Security audit
+        env:
+          NPM_DIR: ${{ matrix.npm_directory.path }}
+        run: |
+          cd "$NPM_DIR"
+          # Don't fail the workflow if unfixable vulns remain — the next step
+          # (`npm audit fix --package-lock-only`) will attempt the auto-fix and
+          # report what it couldn't fix. Failing here would skip the dependency
+          # update entirely on the first unpatchable advisory. Use a ::warning
+          # workflow command (not a bare echo) so the soft-fail surfaces as a
+          # visible annotation in the run UI instead of getting lost in the
+          # step log.
+          npm audit --audit-level moderate || echo "::warning::Vulnerabilities detected — proceeding to auto-fix attempt"
+
+      - name: 🔐 Fix security vulnerabilities (lockfile only)
+        env:
+          NPM_DIR: ${{ matrix.npm_directory.path }}
+        run: |
+          cd "$NPM_DIR"
+          # Update lockfile only - actual install happens via npm ci below
+          npm audit fix --package-lock-only || echo "Some vulnerabilities could not be auto-fixed"
+
+      - name: 👚 Update to latest compatible versions (lockfile only)
+        env:
+          NPM_DIR: ${{ matrix.npm_directory.path }}
+          NPM_ARGS: ${{ inputs.npm_args || '--save' }}
+        run: |
+          cd "$NPM_DIR"
+          # Update lockfile only - actual install happens via npm ci below
+          # Intentional word splitting for npm args
+          # shellcheck disable=SC2086
+          npm update --package-lock-only $NPM_ARGS
+
+      - name: 📦 Install from lockfile with integrity verification
+        env:
+          NPM_DIR: ${{ matrix.npm_directory.path }}
+        run: |
+          cd "$NPM_DIR"
+          # npm ci installs from lockfile with integrity hash verification
+          # This satisfies OSSF Scorecard pinned-dependencies requirement
+          npm ci
+
+      - name: 🔨 Build (if applicable)
+        env:
+          NPM_DIR: ${{ matrix.npm_directory.path }}
+          BUILD_CMD: ${{ matrix.npm_directory.build_cmd }}
+        run: |
+          cd "$NPM_DIR"
+          eval "$BUILD_CMD"
+        if: matrix.npm_directory.build_cmd != ''
+
+      - name: 🧪 Test (if applicable)
+        env:
+          NPM_DIR: ${{ matrix.npm_directory.path }}
+          TEST_CMD: ${{ matrix.npm_directory.test_cmd }}
+        run: |
+          cd "$NPM_DIR"
+          eval "$TEST_CMD"
+        if: matrix.npm_directory.test_cmd != ''
+
+      - name: 📝 Check for changes
+        id: verify-changed-files
+        env:
+          NPM_DIR: ${{ matrix.npm_directory.path }}
+        run: |
+          cd "$NPM_DIR"
+
+          # Check if package.json or package-lock.json have changed
+          if git diff --quiet package.json package-lock.json; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+            echo "path=${{ matrix.npm_directory.path }}" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: 📝 Create pull request
+        uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
+        if: steps.verify-changed-files.outputs.changed == 'true'
+        with:
+          branch: update-npm-dependencies-${{ github.run_number }}
+          title: 🤖 Update dependencies
+          body: |
+            This PR updates dependencies to their latest compatible versions.
+
+            **Updated directory:** `${{ steps.verify-changed-files.outputs.path }}`
+
+            **Changes include:**
+            - Security vulnerability fixes (moderate severity and above)
+            - Compatible version updates for dependencies
+            - Updated lock files for reproducible builds
+
+            This PR was created by the Dependency Update Bot.
+          sign-commits: true
+          add-paths: |
+            ${{ steps.verify-changed-files.outputs.path }}/package.json
+            ${{ steps.verify-changed-files.outputs.path }}/package-lock.json
+          commit-message: 🤖 Update dependencies
+          labels: maintenance
+          draft: false
+          base: main
+          reviewers: djpetti,HashedViking,LearningCircuit

.github/workflows/update-precommit-hooks.yml

@@ -0,0 +1,78 @@
+name: Update Pre-commit Hooks
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 8 * * 5'   # every Friday at 08:00 UTC (day after NPM updates)
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  update:
+    name: 🪝 Update Pre-commit Hooks
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write  # Required for creating PRs
+      pull-requests: write  # Required for creating PRs
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: 📰 Checkout
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+          ref: main
+
+      - name: 🐍 Setup Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.x'
+
+      - name: 📦 Install pre-commit-update
+        run: |
+          python -m pip install pip==25.0 \
+            --hash=sha256:b6eb97a803356a52b2dd4bb73ba9e65b2ba16caa6bcb25a7497350a4e5859b65
+          pip install pre-commit-update==0.6.1 \
+            --hash=sha256:db00891b3384776daaaa5721fd54a448ded19daf87635a3c77b7508eaf7d1634
+
+      - name: 🔄 Update pre-commit hooks (stable versions only)
+        run: |
+          # pre-commit-update skips alpha/beta/rc versions by default
+          pre-commit-update
+
+      - name: 📝 Check for changes
+        id: verify-changed-files
+        run: |
+          if git diff --quiet .pre-commit-config.yaml; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+            echo "No changes detected in .pre-commit-config.yaml"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+            echo "Changes detected in .pre-commit-config.yaml"
+            echo "## Changes:" >> "$GITHUB_STEP_SUMMARY"
+            git diff .pre-commit-config.yaml >> "$GITHUB_STEP_SUMMARY"
+          fi
+
+      - name: 📝 Create pull request
+        uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
+        if: steps.verify-changed-files.outputs.changed == 'true'
+        with:
+          branch: update-precommit-hooks-${{ github.run_number }}
+          title: 🤖 Update pre-commit hooks
+          body: |
+            This PR updates pre-commit hooks to their latest versions.
+
+            **Changes include:**
+            - Updated hook versions for better linting and bug fixes
+            - Ensures consistency with latest tool versions
+
+            This PR was created by the Pre-commit Update Bot.
+          sign-commits: true
+          add-paths: .pre-commit-config.yaml
+          commit-message: 🤖 Update pre-commit hooks
+          labels: maintenance
+          draft: false
+          base: main
+          reviewers: djpetti,HashedViking,LearningCircuit

.github/workflows/validate-image-pinning.yml

@@ -0,0 +1,162 @@
+name: Validate Docker Image Pinning
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - '**/Dockerfile*'
+      - '**/docker-compose*.yml'
+      - '**/docker-compose*.yaml'
+      - '.github/workflows/*.yml'
+      - '.github/workflows/*.yaml'
+      - '.github/workflows/validate-image-pinning.yml'
+      - '.github/scripts/validate-docker-compose-images.sh'
+      - '.github/scripts/validate-workflow-images.py'
+  workflow_call:  # Called by ci-gate.yml for release pipeline
+  workflow_dispatch:
+
+# No concurrency group — intentionally omitted.
+# This workflow triggers on both pull_request and workflow_call (from
+# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
+# direct PR runs and workflow_call runs to cancel each other mid-flight.
+# See #3554 (reverted in #3599) for context.
+
+permissions: {}  # Minimal permissions for OSSF Scorecard
+
+jobs:
+  validate-docker-compose:
+    name: Validate docker-compose Images
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Validate docker-compose image pinning
+        run: |
+          chmod +x .github/scripts/validate-docker-compose-images.sh
+          .github/scripts/validate-docker-compose-images.sh
+
+  validate-workflow-images:
+    name: Validate Workflow Service Containers
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Install PyYAML
+        run: pip install pyyaml==6.0.2
+
+      - name: Validate workflow image pinning
+        run: |
+          chmod +x .github/scripts/validate-workflow-images.py
+          python .github/scripts/validate-workflow-images.py
+
+  summary:
+    name: Image Pinning Validation Summary
+    needs: [validate-docker-compose, validate-workflow-images]
+    runs-on: ubuntu-latest
+    if: always()
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      # zizmor: ignore[template-injection] - values passed via env vars, not interpolated in shell
+      - name: Generate summary
+        env:
+          DOCKER_COMPOSE_RESULT: ${{ needs.validate-docker-compose.result }}
+          WORKFLOW_IMAGES_RESULT: ${{ needs.validate-workflow-images.result }}
+        run: |
+          {
+            echo "## 🔒 Docker Image Pinning Validation"
+            echo ""
+            echo "### What is Image Pinning?"
+            echo ""
+            echo "Pinning Docker images with SHA256 digests ensures:"
+            echo ""
+            echo "- 🔒 **Security**: Protection against supply chain attacks"
+            echo "- 🔄 **Reproducibility**: Exact same image bytes every time"
+            echo "- 🛡️ **Immutability**: Tags like \`:latest\` can be changed, but SHA digests cannot"
+            echo ""
+            echo "### Validation Results"
+            echo ""
+
+            if [ "$DOCKER_COMPOSE_RESULT" = "success" ] && \
+               [ "$WORKFLOW_IMAGES_RESULT" = "success" ]; then
+              echo "✅ **All images properly pinned with SHA256 digests**"
+              echo ""
+              echo "All docker-compose and workflow files pass validation."
+            else
+              echo "❌ **Image pinning violations found**"
+              echo ""
+              if [ "$DOCKER_COMPOSE_RESULT" != "success" ]; then
+                echo "- ❌ docker-compose files have unpinned images"
+              fi
+              if [ "$WORKFLOW_IMAGES_RESULT" != "success" ]; then
+                echo "- ❌ Workflow files have unpinned service containers"
+              fi
+              echo ""
+              echo "See job logs above for details and fix instructions."
+            fi
+
+            echo ""
+            echo "### How to Fix Unpinned Images"
+            echo ""
+            echo "\`\`\`bash"
+            echo "# 1. Pull the image"
+            echo "docker pull <image:tag>"
+            echo ""
+            echo "# 2. Get the SHA digest"
+            echo "docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
+            echo ""
+            echo "# 3. Update your file"
+            echo "image: <image:tag>@sha256:..."
+            echo "\`\`\`"
+            echo ""
+            echo "### Resources"
+            echo ""
+            echo "- [OSSF Scorecard - Pinned Dependencies](https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies)"
+            echo "- [Docker Image Digests Documentation](https://docs.docker.com/engine/reference/commandline/pull/#pull-an-image-by-digest-immutable-identifier)"
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Fail if validation found issues
+        env:
+          DOCKER_COMPOSE_RESULT: ${{ needs.validate-docker-compose.result }}
+          WORKFLOW_IMAGES_RESULT: ${{ needs.validate-workflow-images.result }}
+        run: |
+          if [ "$DOCKER_COMPOSE_RESULT" != "success" ] || \
+             [ "$WORKFLOW_IMAGES_RESULT" != "success" ]; then
+            echo "::error::Image pinning validation failed"
+            exit 1
+          fi

.github/workflows/version_check.yml

@@ -0,0 +1,203 @@
+name: Version Auto-Bump
+
+# Run AFTER merge to main, creates a PR for version bump
+# Benefits:
+# - Respects branch protection rules (no direct push to main)
+# - PR checks run once without restarts
+# - No version merge conflicts between parallel PRs
+# - Version bumps are visible and reviewable
+# - Uses fixed branch name to avoid clutter
+#
+# Manual dispatch supports minor/major bumps via the Actions tab dropdown.
+# Auto-triggered runs (push to main) always bump patch.
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'src/**'
+      - 'pdm.lock'
+      - 'pyproject.toml'
+      - 'package.json'
+      - 'package-lock.json'
+      - 'Dockerfile'
+  workflow_dispatch:
+    inputs:
+      release_type:
+        description: 'Version bump type'
+        required: true
+        type: choice
+        options:
+          - patch
+          - minor
+          - major
+        default: 'patch'
+
+permissions: {}
+
+jobs:
+  version-bump:
+    runs-on: ubuntu-latest
+    # Skip if this push is already an auto-bump commit
+    if: "!contains(github.event.head_commit.message, 'chore: auto-bump version')"
+    permissions:
+      contents: write
+      pull-requests: write
+      actions: read  # for generate_workflow_status.py to read run history
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          fetch-depth: 2
+          persist-credentials: false
+
+      - name: Determine release type
+        id: release
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            release_type="${{ inputs.release_type }}"
+          else
+            release_type="patch"
+          fi
+          echo "type=$release_type" >> "$GITHUB_OUTPUT"
+          echo "Release type: $release_type"
+
+      - name: Check if version was bumped in this push
+        id: check
+        if: github.event_name != 'workflow_dispatch'
+        run: |
+          # Check if version changed in the commits being pushed
+          if git diff HEAD~1 -G"__version__" -- src/local_deep_research/__version__.py | grep -E '\+.*__version__.*='; then
+            echo "Version was manually bumped in this push"
+            echo "needs_bump=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "Version not bumped, will auto-bump"
+            echo "needs_bump=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Set up Python
+        if: steps.check.outputs.needs_bump != 'false'
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Set up PDM with pdm-bump plugin
+        if: steps.check.outputs.needs_bump != 'false'
+        uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
+        with:
+          python-version: '3.12'
+
+      - name: Install pdm-bump plugin
+        if: steps.check.outputs.needs_bump != 'false'
+        run: |
+          pdm self add pdm-bump
+
+      - name: Bump version
+        if: steps.check.outputs.needs_bump != 'false'
+        id: bump
+        run: |
+          release_type="${{ steps.release.outputs.type }}"
+
+          # Get current version before bump (prefer pdm, fallback to grep)
+          current_version=$(pdm show --version 2>/dev/null || grep -oP '(?<=__version__ = ")[^"]*' src/local_deep_research/__version__.py)
+          echo "Current version: $current_version"
+          echo "current_version=$current_version" >> "$GITHUB_OUTPUT"
+
+          # Bump version using pdm-bump
+          echo "Bumping $release_type version..."
+          pdm bump "$release_type"
+
+          # Get new version after bump (prefer pdm, fallback to grep)
+          new_version=$(pdm show --version 2>/dev/null || grep -oP '(?<=__version__ = ")[^"]*' src/local_deep_research/__version__.py)
+          echo "New version: $new_version"
+          echo "new_version=$new_version" >> "$GITHUB_OUTPUT"
+
+          # Sync package.json version with __version__.py
+          jq --arg v "$new_version" '.version = $v' package.json > package.json.tmp
+          mv package.json.tmp package.json
+          echo "Updated package.json version to $new_version"
+
+      - name: Regenerate configuration docs
+        if: steps.check.outputs.needs_bump != 'false'
+        run: python scripts/generate_config_docs.py
+
+      # Refreshes docs/ci/workflow-status.md so the version-bump PR
+      # carries the current snapshot. Output uses coarse "last week /
+      # last month" buckets so within-day reruns produce zero diff —
+      # the diff that lands in this PR only shows workflows whose
+      # bucket has actually shifted since the previous release.
+      #
+      # ~340 GitHub API calls per run (well under the GITHUB_TOKEN
+      # 1000/hr workflow-runs limit). Needs `actions: read` on the
+      # job permissions block above.
+      - name: Regenerate workflow status dashboard
+        if: steps.check.outputs.needs_bump != 'false'
+        # Don't block the version bump if the dashboard refresh fails.
+        # The regen calls ~340 GitHub API endpoints; a transient outage
+        # or rate-limit hit would otherwise prevent the version-bump PR
+        # from being created at all. On failure the dashboard just stays
+        # at its previous snapshot until the next successful run.
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          pip install pyyaml==6.0.3
+          python scripts/generate_workflow_status.py
+
+      - name: Generate PR body
+        if: steps.check.outputs.needs_bump != 'false'
+        id: pr_body
+        run: |
+          release_type="${{ steps.release.outputs.type }}"
+          current="${{ steps.bump.outputs.current_version }}"
+          new="${{ steps.bump.outputs.new_version }}"
+
+          body_file=$(mktemp)
+
+          {
+            echo "## Summary"
+            echo "- Bump **${release_type}** version: ${current} → ${new}"
+            echo ""
+
+            if [ "$release_type" = "major" ]; then
+              echo "> [!CAUTION]"
+              echo "> This is a **major** version bump. It signals breaking changes."
+              echo "> Please review carefully before merging."
+            elif [ "$release_type" = "minor" ]; then
+              echo "> [!IMPORTANT]"
+              echo "> This is a **minor** version bump, indicating new features or significant changes."
+            fi
+
+            echo ""
+            if [ "$release_type" = "patch" ]; then
+              echo "This PR was automatically created by the version bump workflow."
+            else
+              echo "This PR was manually triggered via the Actions tab."
+            fi
+            echo "Approve and merge to trigger a new release."
+            echo ""
+            echo "Configuration docs (docs/CONFIGURATION.md) and the workflow status dashboard (docs/ci/workflow-status.md) have been regenerated."
+          } > "$body_file"
+
+          echo "body_file=$body_file" >> "$GITHUB_OUTPUT"
+
+      # Use peter-evans/create-pull-request with GITHUB_TOKEN so the PR is created
+      # by github-actions[bot] - this allows the repo owner to approve the PR
+      - name: Create Pull Request
+        if: steps.check.outputs.needs_bump != 'false'
+        uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          commit-message: "chore: auto-bump version to ${{ steps.bump.outputs.new_version }}"
+          branch: chore/auto-version-bump
+          delete-branch: true
+          title: "chore: bump ${{ steps.release.outputs.type }} version to ${{ steps.bump.outputs.new_version }}"
+          body-path: ${{ steps.pr_body.outputs.body_file }}
+          labels: |
+            automation
+            maintenance

.github/workflows/vulture-dead-code.yml

@@ -0,0 +1,37 @@
+name: Vulture Dead Code Detection
+
+on:
+  workflow_call:  # Allows this workflow to be called by other workflows
+  workflow_dispatch:
+
+# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)
+permissions: {}
+
+jobs:
+  vulture:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    name: Scan for dead code
+    permissions:
+      contents: read
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: '3.12'
+
+      - name: Install vulture
+        run: pip install "vulture~=2.14"
+
+      - name: Run vulture (src/ only)
+        run: vulture src/local_deep_research/ vulture_whitelist.py --min-confidence 80

.github/workflows/welcome-first-time.yml

@@ -0,0 +1,110 @@
+name: Welcome first-time contributors
+
+# Posts a single welcome comment on a contributor's FIRST PR (filtered per
+# user). Uses pull_request_target so forked PRs receive a writable token;
+# the script reads only sender.login (operator-trusted via GitHub) and
+# never executes fork-controlled content — no checkout, no shell.
+#
+# We do NOT use actions/first-interaction: its isFirstPullRequest check
+# has no author filter (lists all repo PRs and matches only on the lowest
+# PR number), so it would never fire on a repo with prior history.
+
+on:
+  # zizmor: ignore[dangerous-triggers] — pull_request_target is required so
+  # fork PRs receive a writable token to post the welcome comment. The job
+  # never checks out PR content, never runs fork-controlled scripts, and
+  # only reads `sender.login` (operator-trusted GitHub event metadata). The
+  # comment body is a static template with no PR-controlled interpolation.
+  pull_request_target:
+    types: [opened]
+
+permissions: {}  # Minimal top-level for OSSF Scorecard Token-Permissions
+
+jobs:
+  welcome:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      # createComment on a PR returns 403 with only `issues: write` —
+      # GitHub requires `pull-requests: write` when the issue resource
+      # is actually a PR (Accepted-Permissions header lists both).
+      issues: write
+      pull-requests: write
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Welcome first-time contributor
+        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        with:
+          script: |
+            // Iterative pattern from actions/github-script@v9.0.0 README:
+            // skip the just-opened PR by number and return early if ANY
+            // other PR by this author exists. Robust to GitHub's eventual
+            // consistency on listForRepo (the just-opened PR may not yet
+            // be indexed). Using sender.login (canonical) — for `opened`
+            // it equals pull_request.user.login.
+            const sender = context.payload.sender;
+
+            // Skip bots. user.type === 'Bot' is the canonical signal;
+            // also belt-and-suspenders the [bot]-suffix check (consistent
+            // with the PR triage workflow's KNOWN_BOTS handling).
+            if (sender.type === 'Bot' || sender.login.endsWith('[bot]')) {
+              console.log(`Sender ${sender.login} is a bot. Skipping.`);
+              return;
+            }
+
+            const opts = github.rest.issues.listForRepo.endpoint.merge({
+              ...context.issue,
+              creator: sender.login,
+              state: 'all',
+            });
+            const items = await github.paginate(opts);
+
+            for (const item of items) {
+              if (item.number === context.issue.number) continue;
+              if (item.pull_request) {
+                console.log(
+                  `Sender ${sender.login} already has PR #${item.number}. Skipping.`,
+                );
+                return;
+              }
+            }
+
+            const body = [
+              `Welcome to local-deep-research — and thank you for your contribution. We genuinely appreciate the time you're putting in.`,
+              ``,
+              `Here's a starter pack to help your PR move smoothly:`,
+              ``,
+              `**Get set up locally**`,
+              `- [Installation guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/installation.md) — running LDR`,
+              `- [Developer guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/developing.md) — setting up a dev environment`,
+              `- Install our pre-commit hooks (one-time, [details in CONTRIBUTING.md](https://github.com/LearningCircuit/local-deep-research/blob/main/CONTRIBUTING.md#-quick-start)):`,
+              `  \`\`\``,
+              `  pre-commit install`,
+              `  pre-commit install-hooks`,
+              `  \`\`\``,
+              ``,
+              `**Understand the codebase**`,
+              `- [Architecture overview](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/architecture.md) — how the pieces fit together`,
+              `- [Tests README](https://github.com/LearningCircuit/local-deep-research/blob/main/tests/README.md) — how to run the test suite locally`,
+              `- [FAQ](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/faq.md) and [Troubleshooting](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/troubleshooting.md) — common setup, config, and operational issues`,
+              `- Found a security issue? See [SECURITY.md](https://github.com/LearningCircuit/local-deep-research/blob/main/SECURITY.md) for the responsible-disclosure process — please don't open a public PR for it.`,
+              ``,
+              `**Before you ask for review**`,
+              `- Open PRs against \`main\` unless a maintainer says otherwise.`,
+              `- Confirm your PR has a single clear purpose ([PR process](https://github.com/LearningCircuit/local-deep-research/blob/main/CONTRIBUTING.md#-pull-request-process)).`,
+              `- Describe in your PR body what you tested by hand — "CI is green" alone isn't enough.`,
+              `- Drop by [Discord](https://discord.gg/ttcqQeFcJ3) if you'd like to chat with maintainers or other contributors.`,
+              ``,
+              `A maintainer will take a look — if you don't hear back within 7 days, feel free to ping.`,
+            ].join('\n');
+
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.payload.pull_request.number,
+              body,
+            });

.github/workflows/zizmor-security.yml

@@ -0,0 +1,65 @@
+name: Zizmor GitHub Actions Security
+
+on:
+  workflow_dispatch:
+  workflow_call:  # Called by release-gate.yml
+  schedule:
+    # Run security audit weekly on Monday at 9 AM UTC
+    - cron: '0 9 * * 1'
+
+permissions:
+  contents: read
+
+jobs:
+  zizmor-scan:
+    name: Zizmor Workflow Security Scan
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      security-events: write
+      actions: read
+
+    steps:
+      - name: Harden Runner
+        uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
+        with:
+          egress-policy: audit
+
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+
+        with:
+          persist-credentials: false
+
+      - name: Run zizmor security scan
+        uses: zizmorcore/zizmor-action@192e21d79ab29983730a13d1382995c2307fbcaa # v0.5.7
+        with:
+          inputs: .github/workflows/
+          min-severity: low
+          advanced-security: true
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Display scan summary
+        if: always()
+        run: |
+          {
+            echo "## Zizmor GitHub Actions Security Scan"
+            echo ""
+            echo "### What is Zizmor?"
+            echo "Zizmor is a security linter for GitHub Actions workflows that detects:"
+            echo ""
+            echo "- **Template Injection**: Attacker-controlled input in expressions"
+            echo "- **ArtiPACKED**: Credentials leaked in workflow artifacts"
+            echo "- **Vulnerable Actions**: Third-party actions with known CVEs"
+            echo "- **Impostor Commits**: Hash-pinned actions pointing to forks"
+            echo "- **Hardcoded Credentials**: Plain-text secrets in workflows"
+            echo "- **Excessive Permissions**: Over-privileged workflow tokens"
+            echo "- **Unpinned Actions**: Mutable action references"
+            echo "- **Cache Poisoning**: Unsafe cache usage patterns"
+            echo ""
+            echo "### Resources"
+            echo "- [Zizmor GitHub](https://github.com/woodruffw/zizmor)"
+            echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
+            echo "- Results uploaded to GitHub Security tab for detailed analysis"
+          } >> "$GITHUB_STEP_SUMMARY"