chore: import upstream snapshot with attribution
Backwards Compatibility / Verify Encryption Constants (push) Waiting to run
Backwards Compatibility / PyPI Version Compatibility (push) Waiting to run
Backwards Compatibility / Database Migration Tests (push) Waiting to run
CodeQL Advanced / Analyze (javascript-typescript) (push) Waiting to run
CodeQL Advanced / Analyze (python) (push) Waiting to run
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Blocked by required conditions
Docker Tests (Consolidated) / detect-changes (push) Waiting to run
Docker Tests (Consolidated) / Build Test Image (push) Waiting to run
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Blocked by required conditions
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Blocked by required conditions
Docker Tests (Consolidated) / Accessibility Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / LLM Unit Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / LLM Example Tests (push) Blocked by required conditions
Docker Tests (Consolidated) / Production Image Smoke Test (push) Blocked by required conditions
Docker Tests (Consolidated) / Infrastructure Tests (push) Blocked by required conditions
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Waiting to run
OSV-Scanner (Scheduled) / scan-scheduled (push) Waiting to run
Create Release / test-gate (push) Blocked by required conditions
Create Release / release-gate (push) Blocked by required conditions
Create Release / ci-gate (push) Blocked by required conditions
Create Release / version-check (push) Waiting to run
Create Release / e2e-test-gate (push) Blocked by required conditions
Create Release / responsive-test-gate (push) Blocked by required conditions
Create Release / compat-test-gate (push) Blocked by required conditions
Create Release / compose-integration-gate (push) Blocked by required conditions
Create Release / vulture-gate (push) Blocked by required conditions
Create Release / build (push) Blocked by required conditions
Create Release / provenance (push) Blocked by required conditions
Create Release / prerelease-docker (push) Blocked by required conditions
Create Release / publish-docker (push) Blocked by required conditions
Create Release / create-release (push) Blocked by required conditions
Create Release / cleanup-changelog (push) Blocked by required conditions
Create Release / trigger-pypi (push) Blocked by required conditions
Create Release / monitor-pypi (push) Blocked by required conditions
Create Release / Clean up orphan prerelease tags and signatures (push) Blocked by required conditions

This commit is contained in:
wehub-resource-sync
2026-07-13 13:08:55 +08:00
commit 7a0da7932b
2985 changed files with 1049377 additions and 0 deletions
+50
View File
@@ -0,0 +1,50 @@
# .dockerignore
#
# Primary purpose: keep the .git directory out of the build context and
# the published image. .git carries the full commit history (author
# emails, and anything ever committed then removed). The Dockerfile
# copies selectively today (COPY src/, COPY pyproject.toml, ...), so .git
# is not shipped currently -- this file makes that guarantee robust: a
# future `COPY . .`, or a local build from a stale clone, would otherwise
# embed the entire history into the image.
#
# Only files the Dockerfile explicitly COPYs are needed in the context;
# everything below is either never copied or is regenerated in-image.
# ── Version control ──────────────────────────────────────────────
.git
.gitignore
.gitattributes
.github
# ── Python bytecode / caches / virtualenvs ───────────────────────
**/__pycache__/
**/*.py[cod]
**/*$py.class
.venv/
venv/
**/.pytest_cache/
**/.ruff_cache/
**/.mypy_cache/
.coverage
htmlcov/
# ── Node (installed fresh in the image; the local copy is ~300 MB) ─
**/node_modules/
# ── Secrets, local databases, logs — never belong in an image ────
**/.env
**/.env.local
**/.env.*.local
**/*.log
**/*.db
**/*.sqlite
**/*.sqlite3
# ...but the checked-in defaults template IS part of the source the
# build copies via `COPY src/` -- keep it.
!**/.env.template
# ── Editor / OS cruft ────────────────────────────────────────────
.idea/
.vscode/
.DS_Store
+108
View File
@@ -0,0 +1,108 @@
# ============================================================================
# Allowed file patterns for git pre-commit and CI whitelist checks.
# One regex pattern per line. Lines starting with # are comments.
# Blank lines are ignored.
#
# PRINCIPLE: text source types are kept broad (low risk, high churn).
# Config, binary, and other types are path-scoped.
#
# Binary files bloat the repo — do NOT add new binary wildcards without
# maintainer approval. See CODEOWNERS.
# ============================================================================
# ---------------------------------------------------------------------------
# Text source code — kept broad (low risk)
# ---------------------------------------------------------------------------
\.py$
\.js$
\.mjs$
\.html$
\.css$
\.md$
# ---------------------------------------------------------------------------
# JSON — path-scoped
# ---------------------------------------------------------------------------
^src/local_deep_research/defaults/.*\.json$
^cookiecutter-docker/.*\.json$
^package\.json$
^package-lock\.json$
^lighthouserc\.json$
^tests/(.*/)?package\.json$
^tests/(.*/)?package-lock\.json$
^tests/settings/golden_master_settings\.json$
# Allowlist of currently-known responsive UI bugs; checked-in baseline
# for test_responsive_ui_comprehensive.js so new regressions can fail.
^tests/ui_tests/responsive_baseline\.json$
# Journal-quality data: all datasets are runtime-downloaded into the user
# data dir, never bundled. Only __init__.py lives in the package data dir.
# ---------------------------------------------------------------------------
# YAML — path-scoped
# ---------------------------------------------------------------------------
^\.github/.*\.(yml|yaml)$
^\.semgrep/.*\.(yml|yaml)$
^bearer\.yml$
^\.grype\.yaml$
^\.hadolint\.yaml$
^\.pre-commit-config\.yaml$
^\.yamllint\.yaml$
^\.safety-policy\.yml$
^docker-compose.*\.(yml|yaml)$
^examples/.*/docker-compose\.yml$
^community_benchmark_results/.*\.yaml$
^cookiecutter-docker/.*\.yml$
# ---------------------------------------------------------------------------
# Shell scripts — path-scoped
# ---------------------------------------------------------------------------
^\.github/scripts/.*\.sh$
^\.pre-commit-hooks/.*\.sh$
^scripts/.*\.sh$
^tests/.*\.sh$
^docs/.*\.sh$
^examples/.*\.sh$
# ---------------------------------------------------------------------------
# Other config — explicit paths only
# ---------------------------------------------------------------------------
^src/local_deep_research/defaults/.*\.template$
^src/local_deep_research/notifications/templates/.*\.jinja2$
^unraid-templates/.*\.xml$
^pyproject\.toml$
^pdm\.lock$
^\.gitleaks\.toml$
^\.zap/rules\.tsv$
# ---------------------------------------------------------------------------
# Specific root/config files
# ---------------------------------------------------------------------------
.*\.gitkeep$
.*\.gitignore$
^\.coveragerc$
^\.gitleaksignore$
^\.semgrepignore$
^\.trivyignore$
^\.nvmrc$
^playwright\.config\.js$
^\.github/CODEOWNERS$
^MANIFEST\.in$
^LICENSE$
^README$
^Dockerfile$
^\.dockerignore$
^\.file-whitelist\.txt$
^\.github/security/.*\.txt$
# ---------------------------------------------------------------------------
# Binary assets — explicit paths only.
# Binary files bloat the repo; do NOT add new ones without maintainer approval.
# ---------------------------------------------------------------------------
^docs/images/Local Search\.png$
^docs/images/local_search_embedding_model_type\.png$
^docs/images/local_search_paths\.png$
^src/local_deep_research/web/static/favicon\.png$
^docs/.*\.ps1$
^src/local_deep_research/web/static/sounds/error\.mp3$
^src/local_deep_research/web/static/sounds/success\.mp3$
^installers/.*\.(bat|ps1|iss)$
+41
View File
@@ -0,0 +1,41 @@
# Branch Protection Configuration
Required settings for OSSF Scorecard compliance on `main` and `dev` branches.
## Required Settings (GitHub UI: Settings -> Branches -> Add rule)
### Protect matching branches
- [x] Require a pull request before merging
- [x] Require at least 1 approving review
- [x] Dismiss stale PR approvals when new commits are pushed
- [x] Require status checks to pass before merging
- [x] Require branches to be up to date before merging
- [x] Do not allow bypassing the above settings
### Status checks to require
- `test` (unit tests)
- `lint` (code quality)
- `typecheck` (mypy)
## Setup Instructions
1. Go to repository **Settings** -> **Branches**
2. Click **Add branch protection rule**
3. Set "Branch name pattern" to `main`
4. Configure the settings listed above
5. Click **Create**
6. Repeat for `dev` branch
## Why This Matters
OSSF Scorecard's Branch-Protection check verifies that:
- Direct pushes to protected branches are prevented
- Code review is required before merging
- Status checks must pass before merging
These settings help prevent accidental or malicious changes to production code.
## References
- [OSSF Scorecard Branch-Protection Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection)
- [GitHub Branch Protection Documentation](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)
+134
View File
@@ -0,0 +1,134 @@
# Code Owners for Local Deep Research
# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners
# Global owners - these people are automatically requested for review on all PRs
# and their approval is required for merging to protected branches.
# NOTE: this list is mirrored in .github/workflows/pr-triage.yml (CODEOWNERS const).
# Keep both in sync when the maintainer roster changes.
* @LearningCircuit @hashedviking @djpetti
# Critical infrastructure and release files require maintainer approval
/Dockerfile @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/__version__.py @LearningCircuit @hashedviking @djpetti
# Core API changes
/src/local_deep_research/api/ @LearningCircuit @hashedviking @djpetti
# Database migrations and metrics (sensitive for data integrity)
/src/local_deep_research/metrics/ @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/migrate*.py @LearningCircuit @hashedviking @djpetti
# Database and encryption - critical for security
/src/local_deep_research/database/ @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/database/ @LearningCircuit @hashedviking @djpetti
# Authentication and security
/src/local_deep_research/web/auth/ @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/security/ @LearningCircuit @hashedviking @djpetti
# Core application
/src/local_deep_research/web/app.py @LearningCircuit @hashedviking @djpetti
# Security configurations
/docker-compose*.yml @LearningCircuit @hashedviking @djpetti
# Pre-commit security hooks (contains security checks - do not tamper)
/.pre-commit-config.yaml @LearningCircuit @hashedviking @djpetti
/.pre-commit-hooks/ @LearningCircuit @hashedviking @djpetti
# Security scan ignore files (lower risk - for managing false positives)
/.trivyignore* @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.gitleaksignore @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.gitleaks.toml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.semgrepignore @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# Notifications
/src/local_deep_research/notifications/ @tombii @LearningCircuit @hashedviking @djpetti
/tests/notifications/ @tombii @LearningCircuit @hashedviking @djpetti
# Scripts
/scripts/ @tombii @LearningCircuit @hashedviking @djpetti
# Tests and CI
/tests/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# Test workflows (excluding security-tests.yml)
/.github/workflows/docker-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/e2e-research-test.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/responsive-ui-tests-enhanced.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/playwright-webkit-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/puppeteer-e2e-tests.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/fuzz.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# Linting and checking workflows
/.github/workflows/mypy-type-check.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/hadolint.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/validate-image-pinning.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# CI utilities
/.github/workflows/docker-multiarch-test.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/.github/workflows/label-fixed-in-dev.yml @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# Documentation and examples
/docs/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/examples/ @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/README.md @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @djpetti @hashedviking @LearningCircuit
# UI templates (cosmetic changes)
/src/local_deep_research/web/templates/pages/benchmark.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/benchmark_results.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/collection_details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/collections.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/cost_analytics.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/document_details.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/download_manager.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/embedding_settings.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/history.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/library.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/metrics.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/news.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/progress.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/research.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/results.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/templates/pages/subscriptions.html @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# CSS styles
/src/local_deep_research/web/static/css/styles.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/collections.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/collection_details.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/document_details.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/mobile-responsive.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/mobile-navigation.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/news.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/settings.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/web/static/css/subscriptions.css @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# LLM provider settings (schema only, no secrets)
/src/local_deep_research/defaults/llm_providers/google_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/defaults/llm_providers/ionos_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/defaults/llm_providers/openrouter_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/defaults/llm_providers/xai_settings.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# Search engine settings (schema only, no secrets)
/src/local_deep_research/defaults/settings/search_engines/elasticsearch.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/defaults/settings/search_engines/paperless.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/defaults/settings/search_engines/scaleserp.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
/src/local_deep_research/defaults/settings/search_engines/serper.json @scottvr @tombii @prashant-sharma-cmd @elpikola @shreydekate @LearningCircuit @hashedviking @djpetti
# ──────────────────────────────────────────────────────────────────────
# Repository integrity guardrails — maintainer-only approval
#
# These files control what is tracked in git and what passes CI/pre-commit
# whitelist checks. Restricted to prevent re-introduction of broad
# file-type exceptions (e.g. binary wildcards) without explicit maintainer
# approval. Binary files bloat the repo and should not be added to main.
#
# IMPORTANT: these rules MUST remain the last entries in this file
# (GitHub CODEOWNERS uses last-match-wins).
# ──────────────────────────────────────────────────────────────────────
/.gitignore @LearningCircuit
/.file-whitelist.txt @LearningCircuit
/.pre-commit-hooks/file-whitelist-check.sh @LearningCircuit
/.github/scripts/file-whitelist-check.sh @LearningCircuit
/.github/CODEOWNERS @LearningCircuit
+13
View File
@@ -0,0 +1,13 @@
# These are supported funding model platforms
# How donations are used:
# - Ongoing development and maintenance
# - Infrastructure costs (hosting, API keys for testing)
# - Feature development and improvements
# - Bug fixes and security updates
# - Supporting contributors who need it most
# - Building project reserves for future needs
# - Exploring options to provide free public hosting of LDR
github: [LearningCircuit, djpetti, hashedviking] # GitHub Sponsors for individual contributors
ko_fi: localdeepresearch # Support the project with one-time donations
+68
View File
@@ -0,0 +1,68 @@
# Fuzzing Strategy
This document explains our fuzzing approach and addresses OSSF Scorecard's
Fuzzing check.
## Current Implementation
This project uses **Hypothesis** for property-based fuzz testing:
- **Workflow**: `.github/workflows/fuzz.yml`
- **Tests**: `tests/fuzz/test_fuzz_security.py`, `tests/fuzz/test_fuzz_utilities.py`
- **Schedule**: Weekly on Sunday + on changes to security/utilities code
### What We Test
Our fuzz tests cover:
- Input validation edge cases
- Security-sensitive string handling
- API boundary testing
- File path validation
- URL parsing and sanitization
### Test Configuration
```yaml
# Regular CI runs
--hypothesis-seed=0 # Reproducible tests
# Extended scheduled runs
HYPOTHESIS_PROFILE=extended # More examples, deeper exploration
```
## Why Not OSS-Fuzz?
OSSF Scorecard's Fuzzing check looks for integration with:
- [OSS-Fuzz](https://github.com/google/oss-fuzz) - Google's continuous fuzzing
- [ClusterFuzzLite](https://google.github.io/clusterfuzzlite/) - Lightweight alternative
- [OneFuzz](https://github.com/microsoft/onefuzz) - Microsoft's fuzzing platform
**These are not appropriate for this project because:**
1. **OSS-Fuzz targets native code** - Designed for C/C++ vulnerabilities using
libFuzzer/AFL++. This project is primarily Python.
2. **Hypothesis is the Python equivalent** - Property-based testing with
automatic example generation provides equivalent security value.
3. **No native code attack surface** - Our security-sensitive code is Python,
where Hypothesis testing is the standard approach.
## OSSF Scorecard Note
Scorecard's "Fuzzing" check does not recognize Hypothesis-based testing.
This is a known limitation of the check. Our fuzzing implementation provides
equivalent security value for Python codebases.
## Future Considerations
If the project adds significant native code dependencies (C extensions, FFI),
we would consider:
- ClusterFuzzLite integration for continuous fuzzing
- Native fuzz targets for critical C code paths
## References
- [Hypothesis Documentation](https://hypothesis.readthedocs.io/)
- [OSSF Scorecard Fuzzing Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#fuzzing)
- [Property-Based Testing in Python](https://hypothesis.works/)
+34
View File
@@ -0,0 +1,34 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: 'bug'
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Run command '...'
2. Enter query '....'
3. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**System Information:**
- OS: [e.g. Ubuntu 22.04]
- Python Version: [e.g. 3.9.0]
- Model Used: [e.g. deepseek-r1:14b]
- Hardware Specs: [e.g. 32GB RAM, NVIDIA RTX 3080]
**Additional context**
Add any other context about the problem here.
**Output/Logs**
```
If applicable, add relevant output or error logs here
```
+79
View File
@@ -0,0 +1,79 @@
# Contributing to Local Deep Research
We love your input! We want to make contributing to Local Deep Research as easy and transparent as possible, whether it's:
- Reporting a bug
- Discussing the current state of the code
- Submitting a fix
- Proposing new features
- Becoming a maintainer
## Development Process
We use GitHub to host code, to track issues and feature requests, as well as accept pull requests.
1. Fork the repo and create your branch from `main`
2. If you've added code that should be tested, add tests
3. If you've changed APIs, update the documentation
4. Ensure the test suite passes
5. Make sure your code follows the existing style
6. Issue that pull request!
## Any Contributions You Make Will Be Under the MIT License
When you submit code changes, your submissions are understood to be under the same [MIT License](../../LICENSE) that covers the project. Feel free to contact the maintainers if that's a concern.
## Report Bugs Using GitHub's [Issue Tracker](https://github.com/LearningCircuit/local-deep-research/issues)
Report a bug by [opening a new issue](https://github.com/LearningCircuit/local-deep-research/issues/new); it's that easy!
**Great Bug Reports** tend to have:
- A quick summary and/or background
- Steps to reproduce
- Be specific!
- Give sample code if you can
- What you expected would happen
- What actually happens
- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
## Development Setup
1. Clone your fork of the repository:
```bash
git clone https://github.com/YOUR_USERNAME/local-deep-research.git
cd local-deep-research
```
2. Create a virtual environment and activate it:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. Install dependencies:
```bash
pip install -r requirements.txt
```
4. Install Ollama and set up your preferred model:
```bash
# Follow instructions at https://ollama.ai for installation
ollama pull deepseek-r1:14b # Or mistral:7b for lighter hardware
```
## Code Style
- Follow PEP 8 guidelines
- Use type hints where possible
- Write descriptive docstrings for functions and classes
- Keep functions focused and modular
- Comment complex logic
## License
By contributing, you agree that your contributions will be licensed under its MIT License.
## References
This document was adapted from the open-source contribution guidelines for [Facebook's Draft](https://github.com/facebook/draft-js/blob/master/CONTRIBUTING.md).
@@ -0,0 +1,21 @@
## Description
Fixes #
## Note on File Types
This project uses a file whitelist for security (configured in `.gitignore`). Currently allowed: `.py`, `.js`, `.html`, `.css`, `.json`, `.md`, `.yml`, `.yaml`, `.sh`, `.cfg`, `.ipynb`, `.template`
Need a different file type? We're happy to discuss it in the comments, and if approved, it can be added to `.gitignore`.
## Pre-commit Hooks
If you haven't already, consider setting up pre-commit hooks for automatic code formatting:
```bash
pre-commit install
pre-commit install-hooks
```
---
Thanks for your first contribution to Local Deep Research! Welcome to the community. We genuinely appreciate the time and effort you've put into this PR 🎉
@@ -0,0 +1,12 @@
## Description
Fixes #
## CI test coverage
By default, this PR runs the unit/lint checks. Heavy E2E suites are label-gated — add the label that matches what you touched so the relevant workflow runs:
- `test:puppeteer` or `test:e2e` — Puppeteer E2E suite (~3060 min; uses paid LLM/search API quotas).
- `ldr_research` or `ldr_research_static` — LDR research integration workflow that posts findings as a PR comment.
WebKit/Mobile Safari tests run on the daily 02:00 UTC schedule and at release; the responsive UI suite runs at release and on manual dispatch.
+218
View File
@@ -0,0 +1,218 @@
# Security Alert Assessment
This document explains the security scanning alerts that have been assessed
and determined to be false positives or intentionally suppressed.
## DS162092 - Hardcoded URLs
**Status:** Excluded in DevSkim workflow via `exclude-rules`
### Explanation
This rule is excluded because this research tool legitimately integrates with
external APIs. All hardcoded URLs are intentional service endpoints:
- **ArXiv** - Academic paper repository (`https://arxiv.org/`)
- **PubMed** - Medical literature database
- **Semantic Scholar** - AI-powered research tool
- **OpenAlex** - Open catalog of scholarly works
- **Archive.org** - Wayback Machine integration
### Why Exclusion Is Safe
1. **Legitimate API endpoints** - URLs are for real research services
2. **SSRF protection** - Production code uses
`src/local_deep_research/security/ssrf_validator.py` to block dangerous URLs
3. **No user-controlled URLs** - All URLs are hardcoded service endpoints
4. **Test coverage** - URL handling is tested in `tests/fuzz/test_security_fuzzing.py`
---
## DS137138 - Hardcoded Credentials
**Status:** Excluded via workflow configuration (no action needed)
### Explanation
All ~100+ hardcoded credential alerts are in the `tests/` directory and are
intentional mock data for testing.
### Why They Are Safe
1. **All instances are test fixtures** - Named clearly as mock data:
- `api_key="test_key"`
- `password="testpass"`
- `sample_data_with_secrets()`
2. **DevSkim already excludes tests** - The `.github/workflows/devskim.yml`
configuration includes:
```yaml
ignore-globs: 'examples/**,tests/**'
```
3. **Never real credentials** - All test values are obviously fake
4. **Gitleaks handles real secrets** - The `.github/workflows/gitleaks.yml`
workflow scans for actual leaked credentials
### Example Test Fixtures
- `tests/fixtures/mock_credentials.py`
- `tests/unit/auth/test_login.py`
- `tests/integration/api/test_authentication.py`
---
## DS172411 - JavaScript DOM (innerHTML)
**Status:** Addressed with XSS protection infrastructure
### Explanation
The codebase has comprehensive XSS protection infrastructure in
`src/local_deep_research/web/static/js/security/xss-protection.js`:
| Protection Function | Purpose |
|---------------------|---------|
| `escapeHtml()` | HTML entity escaping for text content |
| `sanitizeHtml()` | DOMPurify-based HTML sanitization |
| `safeSetInnerHTML()` | Safe innerHTML wrapper |
| `sanitizeUserInput()` | User input validation and sanitization |
### Current Status
| Category | Count | Status |
|----------|-------|--------|
| Using `escapeHtml()` | ~35 | Safe |
| Using `textContent` | ~20 | Safe |
| Static HTML only | ~15 | Safe |
| Using `sanitizeHtml()` | ~5 | Safe |
All innerHTML usages have been reviewed and appropriate sanitization applied.
---
## DS176209 - Suspicious Comments
**Status:** Excluded in DevSkim workflow via `exclude-rules`
### Explanation
DevSkim flags comments containing words like `TODO`, `FIXME`, `HACK`, `BUG`,
`XXX` as "suspicious". These are **standard development annotations** used
to track technical debt and future work.
### Why Exclusion Is Safe
1. **Not a security rule** - This is a code quality check, not security
2. **Standard practice** - TODO/FIXME comments are used in every codebase
3. **No runtime impact** - Comments have no effect on application behavior
4. **IDE support** - Development tools already track these annotations
---
## DS126858 - Weak/Broken Hash Algorithm
**Status:** Excluded in DevSkim workflow via `exclude-rules`
### Explanation
DevSkim flags any literal occurrence of `sha1` as a "weak/broken hash
algorithm". In this codebase the only matches are:
1. **SLSA provenance JSON keys** in `.github/workflows/prerelease-docker.yml` —
the `"sha1"` key inside `digest` objects is part of the
[SLSA in-toto provenance schema](https://slsa.dev/spec/v0.2/provenance) and
identifies the algorithm Git itself uses for commit hashes. We are not
choosing SHA-1 as a cryptographic primitive — Git's commit identifier
format is fixed.
2. **SQLCipher KDF/HMAC algorithm enums** in
`src/local_deep_research/settings/env_definitions/db_config.py`
(`PBKDF2_HMAC_SHA1`, `HMAC_SHA1`). These exist for backwards-compatibility
with existing user databases; the default is SHA-512. Each occurrence
carries an inline `# DevSkim: ignore DS126858` annotation with rationale.
### Why Exclusion Is Safe
1. **Not a cryptographic choice** - The `sha1` strings in SLSA provenance are
*protocol-mandated key names*, not crypto operations we control.
2. **Git's commit hashing is SHA-1 by design** - The Linux kernel and every
git-backed project produces SHA-1 commit IDs; SLSA records them honestly.
3. **Real SHA-1 misuse would be reviewed** - The SQLCipher backwards-compat
uses are documented and reviewed; new uses of SHA-1 as a cryptographic
primitive would be caught in code review and by CodeQL.
4. **No password/signature SHA-1 in this codebase** - Authentication uses
`secrets`/Argon2-class KDFs and SQLCipher's SHA-512 default.
---
## Container Image CVEs
**Status:** Documented, awaiting upstream fixes
### No Fix Available
The following CVEs are in the Debian base image packages with no upstream
fixes currently available:
| CVE | Package | Severity | Notes |
|-----|---------|----------|-------|
| CVE-2025-14104 | util-linux | Medium | No fix version |
| CVE-2022-0563 | util-linux | Low | Debian won't fix |
| CVE-2025-6141 | Various | Low | No fix version |
These are monitored and will be addressed when fixes become available.
---
## DevSkim Rule Exclusions Summary
The following rules are excluded in `.github/workflows/devskim.yml`:
| Rule | Name | Reason |
|------|------|--------|
| DS162092 | Hardcoded URL | Legitimate API endpoints for research services |
| DS176209 | Suspicious Comment | Standard TODO/FIXME annotations |
| DS137138 | Hardcoded Credentials | All matches are test fixtures (mock data) |
| DS148264 | Use cryptographic random | All `random` usages are non-security (ML shuffle, jitter) |
| DS172411 | setTimeout code injection | All setTimeout calls pass function refs, never strings |
| DS126858 | Weak/Broken Hash Algorithm | SLSA-schema-required `sha1` JSON key + SQLCipher backwards-compat enums |
### Review Cadence
These exclusions should be reviewed **quarterly** to ensure:
- No new security-relevant URLs are being masked
- Exclusions remain appropriate as the codebase evolves
- New DevSkim rules are evaluated for applicability
**Last reviewed:** May 2026
---
## GitHub Security Tab Dismissals
Some security alerts can only be dismissed, or are very difficult to
suppress, outside the
[GitHub Security tab](https://docs.github.com/en/code-security/dependabot/dependabot-alerts/viewing-and-updating-dependabot-alerts).
This is a
[GitHub platform limitation](https://github.com/orgs/community/discussions/163277) —
Dependabot alerts, code scanning alerts, and secret scanning alerts are
managed primarily through the repository UI rather than via configuration
files or inline annotations.
Dismissals made through the Security tab include a reason (e.g.,
"tolerable in this context", "no bandwidth to fix", "false positive") and
an optional comment, but these are only visible to users with repository
write access. GitHub provides no export or in-repo tracking mechanism, so
unlike the other suppressions documented in this file, these dismissals
cannot be tracked in version-controlled files.
---
## References
- [DevSkim Configuration](workflows/devskim.yml)
- [Gitleaks Configuration](workflows/gitleaks.yml)
- [XSS Protection Module](../src/local_deep_research/web/static/js/security/xss-protection.js)
- [SSRF Validator](../src/local_deep_research/security/ssrf_validator.py)
+203
View File
@@ -0,0 +1,203 @@
# OSSF Scorecard Compliance Notes
This document explains our compliance status with OSSF Scorecard checks
and documents any accepted risks or false positives.
## Current Score: 8/10 for Pinned-Dependencies
**Summary:**
- ✅ 196/196 GitHub Actions pinned by SHA
- ✅ 4/4 Container images pinned by digest
- ✅ 1/1 Security tools use official SHA-pinned actions (zizmor)
- ⚠️ 2/27 pip commands pinned by hash (remainder version-pinned — accepted risk)
- ⚠️ 21/24 npm commands pinned (3 are operational commands)
- ⚠️ 1 false positive for downloadThenRun
- ⚠️ APT packages intentionally unpinned (base image controls versions)
## Pinned-Dependencies
### GitHub Actions: COMPLIANT ✅
All GitHub Actions use commit SHA pinning (40-character hex):
```yaml
# Examples from our workflows:
actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6
step-security/harden-runner@df199fb7be9f65074067a9eb93f12bb4c5547cf2 # v2.13.3
```
### Docker Images: COMPLIANT ✅
All Docker images use SHA256 digest pinning:
```yaml
# Examples from our workflows and docker-compose:
python:3.13.9-slim@sha256:326df678c20c78d465db501563f3492d17c42a4afe33a1f2bf5406a1d56b0e86
redis:alpine@sha256:8360960f5fb56a282d78686203dd875862cd4b52a4184c17ac753690252d6d31
node:20-alpine@sha256:bcd88137d802e2482c9df3cdec71e0431857ebbbdba6973776b5593214056d86
```
### Official GitHub Actions: COMPLIANT ✅
One security tool uses an official GitHub Action with SHA-pinning,
which provides equivalent security to hash-pinned pip installs:
| Tool | Action | SHA |
|------|--------|-----|
| zizmor | `zizmorcore/zizmor-action` | `135698455da5c3b3e55f73f4419e481ab68cdd95` |
This official action runs the tool in a container with internal integrity verification,
which OSSF Scorecard accepts as equivalent to hash pinning.
> **Note:** pip-audit was removed because it duplicates OSV-Scanner's coverage (both
> query the OSV database) and its internal use of pip's dependency resolver conflicts
> with PDM's `[tool.pdm.resolution.overrides]`, causing persistent CI failures.
> **Note:** Checkov previously used `bridgecrewio/checkov-action` but was reverted to CLI
> installation (`pip install checkov==3.2.499`) due to a known bug with multiple
> consecutive action calls. See [checkov-action#170](https://github.com/bridgecrewio/checkov-action/issues/170)
> and [checkov#5866](https://github.com/bridgecrewio/checkov/issues/5866).
### pip install: VERSION-PINNED (Accepted Risk) ⚠️
Scorecard flags `pip install package==version` because it prefers hash pinning.
The remaining pip commands use exact version pinning.
**Flagged commands and their status:**
| File | Line | Command | Status |
|------|------|---------|--------|
| Dockerfile | 49-50 | `pip3 install pip==26.0` `pip install pdm==2.26.2...` | Version-pinned |
| Dockerfile | 194 | `pip3 install pip==26.0` | Version-pinned |
| check-env-vars.yml | 36 | `pip install loguru==0.7.3 sqlalchemy==2.0.36...` | Version-pinned |
| checkov.yml | 33 | `pip install checkov==3.2.499` | Version-pinned |
| fuzz.yml | 51-52 | `pip install pip==25.0` `pip install pdm==2.26.2` | Version-pinned |
| mypy-type-check.yml | 39,47-48 | `pip install pdm==2.26.2 mypy==1.14.1...` | Version-pinned |
| publish.yml | 139,328 | `pip install pdm==2.26.2` `pip install wheel==0.46.2` | Version-pinned |
| puppeteer-e2e-tests.yml | 65 | `pip install -e .` | Local package |
| responsive-ui-tests-enhanced.yml | 97-98 | `pip install pip==25.0` `pip install -e .` | Version-pinned + local (Alert #5688) |
| semgrep.yml | 36 | `pip install semgrep==1.87.0` | Version-pinned |
| update-precommit-hooks.yml | 35-38 | `pip install pip==25.0` `pip install pre-commit-update==0.6.1` | **Hash-pinned** |
| validate-image-pinning.yml | 67 | `pip install pyyaml==6.0.2` | Version-pinned |
| backwards-compatibility.yml | 66-68 | `pip install --upgrade pip` `pip install pytest` `pip install -e .` | Intentionally unpinned |
| backwards-compatibility.yml | 106-107 | `pip install --upgrade pip` `pip install pytest` | Intentionally unpinned |
| backwards-compatibility.yml | 154-155 | `pip install --upgrade pip` `pip install "local-deep-research==..."` | Intentionally unpinned |
| backwards-compatibility.yml | 169 | `pip install -e .` | Local package |
> **Note:** `backwards-compatibility.yml` intentionally uses unpinned pip commands
> because it tests compatibility with prior PyPI releases of local-deep-research.
> Pinning these commands would defeat the purpose of the compatibility tests.
**Why we don't use hash pinning:**
1. **Platform-specific hashes**: pip package hashes vary by Python version, OS, and architecture.
A single hash won't work across different CI runners.
2. **Maintenance burden**: Every version update requires regenerating hashes for all platforms.
3. **Marginal security benefit**: These are dev/CI tools running in hardened CI environments
(step-security/harden-runner) with egress auditing. Supply chain attacks on PyPI packages
are mitigated by version pinning and short execution windows.
4. **Industry practice**: Version pinning (`==`) is the standard for CI tool installation.
Hash pinning is typically reserved for production dependencies.
5. **Local packages**: `pip install -e .` installs the local source code and cannot be hash-pinned.
6. **Bootstrap commands**: `pip install pip==X.Y.Z` cannot be hash-pinned because pip is the
tool performing the verification — it cannot verify its own integrity during a self-upgrade.
### Specific OSSF Scorecard Alerts
| Alert | File | Lines | Description | Status |
|-------|------|-------|-------------|--------|
| #5688 | responsive-ui-tests-enhanced.yml | 97-98 | `pip install pip==25.0` (bootstrap) and `pip install -e .` (local source) | Accepted limitation |
### npm Commands: MOSTLY COMPLIANT (21/24) ⚠️
The 3 "unpinned" npm commands are operational commands, not package installations:
| File | Line | Command | Reason Not Pinned |
|------|------|---------|-------------------|
| npm-audit.yml | 56 | `npm i --package-lock-only` | Generates lockfile only |
| npm-audit.yml | 72 | `npm i --package-lock-only` | Generates lockfile only |
| update-npm-dependencies.yml | 74 | `npm update` | Intentionally updates to latest |
These commands don't install packages directly - they either generate lockfiles
or intentionally update packages. They cannot and should not be "pinned".
### downloadThenRun: FALSE POSITIVE ⚠️
**Flagged:** `examples/elasticsearch/test_elasticsearch.sh:60`
```bash
curl -s http://localhost:9200 | python3 -m json.tool | head -10
```
**Why it's a false positive:**
- This fetches JSON from localhost:9200 (local Elasticsearch)
- Pipes to `python3 -m json.tool` (stdlib JSON formatter)
- Shows first 10 lines of pretty-printed output
This is NOT downloading and running a remote script. It's formatting local JSON output.
The scorecard pattern-matches `curl | python` as potentially dangerous, but this
is a safe operation on localhost data.
**OSSF Scorecard Alert:** #4411
### APT Packages: INTENTIONALLY UNPINNED ⚠️
**Files affected:** `publish.yml`, `e2e-research-test.yml`, `responsive-ui-tests-enhanced.yml`, `Dockerfile`
| File | Packages | Runner/Base |
|------|----------|-------------|
| publish.yml | libsqlcipher-dev, patchelf | ubuntu-22.04 |
| e2e-research-test.yml | jq | ubuntu-22.04 |
| responsive-ui-tests-enhanced.yml | wget, gnupg, ca-certificates, fonts-liberation, etc. | ubuntu-latest |
| Dockerfile | curl, git, build-essential, etc. | python:3.13.9-slim@sha256:... |
**Rationale for NOT pinning APT packages:**
1. **Version availability**: Old APT package versions are removed from Ubuntu archives after 6-12 months.
Pinning to `package=1.2.3-1ubuntu1` causes builds to fail when that version is removed.
2. **Base image controls versions**: Docker base images are SHA-pinned, which deterministically controls
which APT package versions are available. The combination of `python:3.13.9-slim@sha256:326df678...`
and `apt-get install curl` produces the same result every time that base image is used.
3. **Runner stability**: GitHub workflow runners use pinned Ubuntu versions (e.g., `ubuntu-22.04`)
which provide consistent package versions throughout the runner's lifecycle.
4. **Version variation**: APT package version strings vary between Ubuntu releases and architectures,
making cross-platform pinning impractical.
5. **Industry consensus**: Security experts recommend pinning the base image/runner rather than
individual packages. Base image pinning provides stronger guarantees with lower maintenance burden.
**Mitigations in place:**
- ✅ Docker base images pinned to SHA256 digests (see Docker Images section above)
- ✅ GitHub runner versions pinned where practical (ubuntu-22.04)
- ✅ Dependabot configured to monitor for security updates
- ✅ Step-security/harden-runner audits all egress traffic
- ✅ Minimal package sets installed (only what's needed)
### Enforcement
We have automated verification for our pinning strategy:
- `.github/workflows/validate-image-pinning.yml` - Validates Docker image digests
- Pre-commit hooks verify action SHA pinning
- All pip install commands use explicit version specifiers (except `backwards-compatibility.yml` which intentionally tests with unpinned versions)
### Review Cadence
These decisions are reviewed quarterly to ensure they remain appropriate:
- **Next review:** Q2 2026
- **Owner:** Security team
## References
- [OSSF Scorecard Pinned-Dependencies Check](https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies)
- [StepSecurity Harden Runner](https://github.com/step-security/harden-runner)
- [pip Hash Checking Mode](https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode)
+4
View File
@@ -0,0 +1,4 @@
self-hosted-runner:
# Custom runner labels used in this repository
labels:
- ubuntu-24.04-arm
+8
View File
@@ -0,0 +1,8 @@
name: "CodeQL Config"
# Paths to ignore - exclude test files and security-reviewed intentional features
paths-ignore:
- '**/tests/**'
- '**/test_*.py'
- '**/*_test.py'
- 'src/local_deep_research/security/file_write_verifier.py'
+95
View File
@@ -0,0 +1,95 @@
# Dependabot configuration for automated dependency updates
version: 2
updates:
# Python dependencies
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
time: "04:00"
open-pull-requests-limit: 5
labels:
- "dependencies"
- "python"
commit-message:
prefix: "chore"
include: "scope"
ignore:
# faiss-cpu >=1.14 wheels execute AVX2 instructions unconditionally at
# dlopen and SIGILL on CPUs without AVX2 (#4480). Do not propose bumping
# the constraint past 1.13.x until upstream ships an AVX2-safe wheel
# (facebookresearch/faiss#5296). Remove this ignore together with the
# pyproject pin when lifting it — exit conditions tracked in #4499.
- dependency-name: "faiss-cpu"
versions: [">=1.14"]
# JavaScript dependencies
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
time: "04:00"
open-pull-requests-limit: 5
labels:
- "dependencies"
- "javascript"
commit-message:
prefix: "chore"
include: "scope"
# GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
time: "04:00"
labels:
- "dependencies"
- "github-actions"
commit-message:
prefix: "chore"
include: "scope"
- package-ecosystem: docker
directory: /
schedule:
interval: daily
- package-ecosystem: npm
directory: /tests/api_tests_with_login
schedule:
interval: daily
- package-ecosystem: npm
directory: /tests/infrastructure_tests
schedule:
interval: daily
- package-ecosystem: npm
directory: /tests
schedule:
interval: daily
- package-ecosystem: npm
directory: /tests/puppeteer
schedule:
interval: daily
- package-ecosystem: npm
directory: /tests/ui_tests
schedule:
interval: daily
- package-ecosystem: npm
directory: /tests/ui_tests/playwright
schedule:
interval: daily
- package-ecosystem: npm
directory: /tests/accessibility_tests
schedule:
interval: daily
+58
View File
@@ -0,0 +1,58 @@
# PR triage labels managed declaratively by .github/workflows/labels-sync.yml.
# This file is an additive sync — labels not listed here are left untouched.
# Persistent labels (applied at PR open by .github/workflows/pr-triage.yml):
- name: external-contributor
color: 'cccccc'
description: PR from someone outside the maintainer team
- name: first-time-contributor
color: 'c2e0c6'
description: First contribution to this repo
- name: bot
color: 'ededed'
description: PR opened by an automated account
- name: needs-rework
color: 'e99695'
description: PR needs substantive rework before it can be reviewed in depth
# Lifecycle labels (toggled per-PR by .github/workflows/pr-triage.yml):
- name: needs-codeowner-review
color: 'fbca04'
description: Awaiting first review from a global codeowner
- name: awaiting-author
color: 'd4c5f9'
description: Codeowner requested changes; ball is with author
- name: awaiting-codeowner
color: 'fef2c0'
description: Author has responded; needs codeowner re-review
# LDR research trigger labels (used by .github/workflows/e2e-research-test.yml
# and .github/workflows/issue-research.yml — applied manually by maintainers,
# auto-removed by the workflow after the run completes).
# Descriptions are phrased as direct instructions so the AI code reviewer
# (.github/workflows/ai-code-reviewer.yml) can decide when to apply them.
- name: ldr_research
color: '7057ff'
description: Apply for substantive logic/architecture changes — when in doubt, add it. Runs full LDR research.
- name: ldr_research_static
color: '7057ff'
description: Cheaper variant of ldr_research for routine code changes — favor running this over no analysis.
# E2E test trigger labels (used by .github/workflows/puppeteer-e2e-tests.yml).
# Both labels trigger the same Puppeteer suite; descriptions guide the AI
# reviewer toward running the suite for any PR that touches the web stack.
- name: 'test:puppeteer'
color: '0e8a16'
description: Apply if PR touches UI, web routes, auth, or templates — when in doubt, add it. Runs Puppeteer E2E.
- name: 'test:e2e'
color: '1d76db'
description: Alias for test:puppeteer (same Puppeteer workflow). Apply one; both trigger the same job.
# Full sharded UI test trigger (used by .github/workflows/ui-full-shards.yml).
# Invokes docker-tests.yml with strict-mode=true so the 14-way ui-tests matrix
# runs. Heavy — re-runs the full docker-tests job graph including pytest.
- name: 'test:ui-full-shards'
color: '0e8a16'
description: Apply if PR touches puppeteer test infra or you want all 14 UI shards — costs ~45 min CI time.
# Human-only signal labels — must NOT be set by any workflow, bot, or AI
# reviewer. These exist to communicate a human judgement that automation
# cannot make. If you find a workflow or hook applying one of these,
# that's a bug — remove the automation, don't extend the label list.
- name: code-ready
color: '006b75'
description: Code looks technically ready; still needs CI + approver. Apply manually — never auto-applied.
+84
View File
@@ -0,0 +1,84 @@
changelog:
categories:
- title: 🔒 Security Updates
labels:
- security
- title: 💥 Breaking Changes
labels:
- breaking-change
- critical-changes
- title: ✨ New Features
labels:
- feature
- enhancement
- dev-enhancement
- title: 🐛 Bug Fixes
labels:
- bug
- bugfix
- dev-bugfix
- title: ⚡ Performance Improvements
labels:
- performance
- title: 🗄️ Database Changes
labels:
- database
- title: 📚 Documentation
labels:
- documentation
- title: 🔧 CI/CD & Maintenance
labels:
- ci-cd
- maintenance
- github-actions
- title: ⬆️ Dependencies
labels:
- dependencies
- title: 🧹 Code Quality & Refactoring
labels:
- refactoring
- technical-debt
- developer-experience
- title: 🧪 Tests
labels:
- tests
- title: ⚙️ Configuration
labels:
- configuration
- title: 🐍 Python Changes
labels:
- python
- title: 🎨 Frontend Changes
labels:
- javascript
- visualization
- css
- ui-ux
- accessibility
- snappy
- title: 🐳 Docker & Deployment
labels:
- docker
- title: 🔍 Research Features
labels:
- research
- research-quality
- research-speed
- title: 📊 Metrics & Analytics
labels:
- metrics
- benchmark-results
- title: 🔄 Branch Syncs & Automation
labels:
- sync
- auto-merge
- automation
- ai_code_review
- title: 🚧 Review & Blocked Items
labels:
- merge-blocked
- security-review-needed
- conflicts
- title: Other Changes
labels:
- "*"
+234
View File
@@ -0,0 +1,234 @@
#!/bin/bash
# Security check for potential unencrypted file writes to disk
# This script helps prevent accidentally bypassing encryption at rest
set -e
echo "Checking for potential unencrypted file writes to disk..."
echo "========================================="
# Patterns that might indicate writing sensitive data to disk
# Note: Using basic grep patterns without lookaheads
SUSPICIOUS_PATTERNS=(
# Python patterns
"\.write\("
"\.save\("
"\.dump\("
"open\(.*['\"]w['\"].*\)"
"open\(.*['\"]wb['\"].*\)"
"with.*open\(.*['\"]w['\"]"
"with.*open\(.*['\"]wb['\"]"
"\.to_csv\("
"\.to_json\("
"\.to_excel\("
"\.to_pickle\("
"tempfile\.NamedTemporaryFile.*delete=False"
"Path.*\.write_text\("
"Path.*\.write_bytes\("
"shutil\.copy"
"shutil\.move"
"\.export_to_file\("
"\.save_to_file\("
"\.write_pdf\("
"\.savefig\("
# JavaScript patterns
"fs\.writeFile"
"fs\.writeFileSync"
"fs\.createWriteStream"
"fs\.appendFile"
)
# Directories to exclude from checks
EXCLUDE_DIRS=(
"tests"
"test"
"__pycache__"
".git"
"node_modules"
".venv"
"venv"
"migrations"
"static"
"vendor"
"dist"
"build"
".next"
"coverage"
"examples"
"scripts"
".github"
"cookiecutter-docker"
)
# Files to exclude
EXCLUDE_FILES=(
"*_test.py"
"test_*.py"
"*.test.js"
"*.spec.js"
"*.test.ts"
"*.spec.ts"
"setup.py"
"webpack.config.js"
"**/migrations/*.py"
"*.min.js"
"*.bundle.js"
"*-min.js"
"*.min.css"
)
# Safe keywords that indicate encrypted or safe operations
# These patterns indicate that file writes have been security-verified
SAFE_KEYWORDS=(
"write_file_verified"
"write_json_verified"
)
# Known safe usage patterns (logs, configs, etc.)
SAFE_USAGE_PATTERNS=(
"security/file_write_verifier.py"
"import tempfile"
"tempfile\.mkdtemp"
"tmp_path"
"tmp_file"
)
# Build exclude arguments for grep
EXCLUDE_ARGS=""
for dir in "${EXCLUDE_DIRS[@]}"; do
EXCLUDE_ARGS="$EXCLUDE_ARGS --exclude-dir=$dir"
done
for file in "${EXCLUDE_FILES[@]}"; do
EXCLUDE_ARGS="$EXCLUDE_ARGS --exclude=$file"
done
# Track if we found any issues
FOUND_ISSUES=0
ALL_MATCHES=""
echo "Scanning codebase for suspicious patterns..."
# Search only in src/ directory to avoid .venv and other non-source directories
SEARCH_PATHS="src/"
# Single pass to collect all matches
for pattern in "${SUSPICIOUS_PATTERNS[@]}"; do
# Use grep with binary files excluded and max line length to avoid issues with minified files
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
matches=$(grep -rn -I $EXCLUDE_ARGS -- "$pattern" $SEARCH_PATHS --include="*.py" --include="*.js" --include="*.ts" 2>/dev/null | head -1000 || true)
if [ -n "$matches" ]; then
ALL_MATCHES="$ALL_MATCHES$matches\n"
fi
done
# Also check for specific problematic patterns in one pass
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
temp_matches=$(grep -rn -I $EXCLUDE_ARGS -E "tmp_path|tempfile|/tmp/" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
if [ -n "$temp_matches" ]; then
ALL_MATCHES="$ALL_MATCHES$temp_matches\n"
fi
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
db_matches=$(grep -rn -I $EXCLUDE_ARGS -E "report_content.*open|report_content.*write|markdown_content.*open|markdown_content.*write" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
if [ -n "$db_matches" ]; then
ALL_MATCHES="$ALL_MATCHES$db_matches\n"
fi
# shellcheck disable=SC2086 # Word splitting is intentional for EXCLUDE_ARGS
export_matches=$(grep -rn -I $EXCLUDE_ARGS -E "export.*Path|export.*path\.open|export.*\.write" $SEARCH_PATHS --include="*.py" 2>/dev/null | head -500 || true)
if [ -n "$export_matches" ]; then
ALL_MATCHES="$ALL_MATCHES$export_matches\n"
fi
# Now filter all matches at once
if [ -n "$ALL_MATCHES" ]; then
echo "Filtering results for false positives..."
# Remove duplicates and sort (use tr to handle potential null bytes)
ALL_MATCHES=$(echo -e "$ALL_MATCHES" | tr -d '\0' | sort -u)
filtered_matches=""
while IFS= read -r line; do
[ -z "$line" ] && continue
# Check if line contains safe keywords
skip_line=0
for safe_pattern in "${SAFE_KEYWORDS[@]}"; do
if echo "$line" | grep -qE -- "$safe_pattern"; then
skip_line=1
break
fi
done
# Check if line contains safe usage patterns
if [ "$skip_line" -eq 0 ]; then
for usage_pattern in "${SAFE_USAGE_PATTERNS[@]}"; do
if echo "$line" | grep -qE -- "$usage_pattern"; then
skip_line=1
break
fi
done
fi
# Additional filters for test/mock files that might not be caught by path exclusion
if [ "$skip_line" -eq 0 ]; then
if echo "$line" | grep -qE "test|mock|stub" && ! echo "$line" | grep -q "#"; then
skip_line=1
fi
fi
# Allowlist of files that legitimately write to disk without encryption.
# These must NOT touch user data or secrets — only:
# - web/app_factory.py — Flask/framework config writes
# - document_loaders/bytes_loader.py — in-memory → tmp for parsers
# - journal_quality/downloader.py — public OpenAlex/DOAJ/predatory/
# JabRef/ROR snapshots downloaded to the user data dir (bibliographic
# metadata only — journal names, ISSNs, h-indices; no PII/secrets)
# - journal_quality/data_sources/*.py — same family, per-source adapters
# that write the intermediate JSON manifests under the user data dir
# If you add an entry here, document WHY the file's writes are safe
# (public data, not user-specific, not encrypted at rest by design).
if [ "$skip_line" -eq 0 ]; then
if echo "$line" | grep -qE "web/app_factory\.py|document_loaders/bytes_loader\.py|journal_quality/downloader\.py|journal_quality/data_sources/.+\.py"; then
skip_line=1
fi
fi
# Filter safe temp files with proper cleanup
if [ "$skip_line" -eq 0 ]; then
if echo "$line" | grep -q "database/encrypted_db.py"; then
skip_line=1
fi
fi
if [ "$skip_line" -eq 0 ]; then
filtered_matches="$filtered_matches$line\n"
FOUND_ISSUES=1
fi
done <<< "$ALL_MATCHES"
if [ -n "$filtered_matches" ] && [ "$FOUND_ISSUES" -eq 1 ]; then
echo "⚠️ Found potential unencrypted file writes:"
echo "========================================="
echo -e "$filtered_matches"
fi
fi
echo "========================================="
if [ $FOUND_ISSUES -eq 1 ]; then
echo "❌ Security check failed: Found potential unencrypted file writes"
echo ""
echo "Please review the above findings and ensure:"
echo "1. Sensitive data is not written to disk unencrypted"
echo "2. Temporary files are properly cleaned up"
echo "3. Use in-memory operations where possible"
echo "4. If file writes are necessary, ensure they're encrypted or add '# Safe: <reason>' comment"
echo ""
echo "For exports, use the in-memory pattern like in export_report_to_memory()"
exit 1
else
echo "✅ Security check passed: No suspicious unencrypted file writes detected"
fi
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env bash
#
# Combine multiple per-model AI code reviews into a single sticky PR comment.
#
# This is the pure-assembly half of the AI Code Reviewer workflow
# (.github/workflows/ai-code-reviewer.yml), split out so the non-trivial parts —
# header/footer stripping, "Reviewer N" section assembly, label union, and the
# pass/fail aggregation — can be unit-tested without any network or GitHub API
# access. The workflow runs the models (network) and writes each reviewer's raw
# stdout to resp_<i>.json and its exit code to code_<i>; this script turns those
# into the comment body, label set, decision, and success count.
#
# Usage:
# combine-ai-reviews.sh <work_dir> <model_1> [<model_2> ...]
#
# Reads, for i in 0..N-1 (N = number of model args):
# <work_dir>/resp_<i>.json - raw stdout from ai-reviewer.sh for reviewer i
# <work_dir>/code_<i> - that reviewer's integer exit code
# <work_dir>/err_<i>.log - that reviewer's stderr (only shown in DEBUG)
# The model names are used only for the per-reviewer log-group labels; the
# posted comment stays anonymized ("Reviewer 1", "Reviewer 2", ...).
#
# Environment:
# HEAD_SHA - head commit sha for the "Last reviewed at commit" line (opt)
# DEBUG_MODE - "true" to echo raw responses + stderr to this script's stderr
#
# Writes (into <work_dir>):
# comment_body.md - the combined sticky-comment body
# labels.txt - deduped labels, one per line (may be empty)
# decision.txt - "pass" or "fail"
# success_count.txt - number of reviewers that produced a usable review
#
# Logs ::group:: sections and debug output to stderr (kept off stdout so the
# output files are the only contract). Exit status is 0 even when every
# reviewer failed — the caller inspects success_count.txt and decides whether
# to fail the workflow. A non-zero exit means a usage error only.
set -euo pipefail
WORK_DIR="${1:-}"
if [ -z "$WORK_DIR" ] || [ ! -d "$WORK_DIR" ]; then
echo "usage: $0 <work_dir> <model> [<model> ...]" >&2
exit 2
fi
shift
MODELS=("$@")
if [ "${#MODELS[@]}" -eq 0 ]; then
echo "error: at least one model name is required" >&2
exit 2
fi
DEBUG_MODE="${DEBUG_MODE:-false}"
HEAD_SHA="${HEAD_SHA:-}"
COMBINED_REVIEW=""
ALL_LABELS=""
ANY_FAIL="false"
SUCCESS_COUNT=0
# Assemble the combined comment in model order. Reviewers are anonymized as
# "Reviewer N" — the model -> number mapping appears only in the (debug) logs
# below, so readers judge the feedback, not the model.
for i in "${!MODELS[@]}"; do
REVIEWER_NUM=$((i + 1))
EXIT_CODE="$(cat "$WORK_DIR/code_$i" 2>/dev/null || echo 1)"
# A missing/empty/garbage exit-code file counts as a failed reviewer rather
# than tripping the numeric comparison below (which would print "integer
# expression expected" and then fall through to wrongly treating the reviewer
# as a success).
[[ "$EXIT_CODE" =~ ^[0-9]+$ ]] || EXIT_CODE=1
AI_RESPONSE="$(cat "$WORK_DIR/resp_$i.json" 2>/dev/null || echo "")"
echo "::group::Reviewer #$REVIEWER_NUM (${MODELS[i]})" >&2
if [ "$DEBUG_MODE" = "true" ]; then
{
echo "=== RAW AI RESPONSE (reviewer #$REVIEWER_NUM, exit $EXIT_CODE) ==="
echo "$AI_RESPONSE"
echo "--- stderr ---"
cat "$WORK_DIR/err_$i.log" 2>/dev/null || true
echo "=== END RAW AI RESPONSE ==="
} >&2
fi
# One model failing (hard error, empty, or non-JSON) must not sink the others:
# record a short failure note and keep going. The caller fails the workflow
# only if *every* reviewer failed (success_count == 0). A reviewer that errors
# out never sets a verdict, so it can never flip the aggregate decision to
# "fail" — only a model that returns a valid "fail" verdict does. Sensitive raw
# output is never written to the comment.
# Require a JSON *object*: a bare `jq .` accepts any valid JSON, including a
# top-level string/array/number that a refusing or misbehaving model might
# emit ("I refuse to review this"). That would pass the gate and then crash
# the `.review` access below under `set -e`, sinking *every* reviewer and
# writing no output at all. `type == "object"` rejects non-objects so they
# degrade to a per-reviewer failure note, preserving the others.
if [ "$EXIT_CODE" -ne 0 ] || [ -z "$AI_RESPONSE" ] || ! echo "$AI_RESPONSE" | jq -e 'type == "object"' >/dev/null 2>&1; then
echo "⚠️ Reviewer #$REVIEWER_NUM did not return a usable review (exit $EXIT_CODE)" >&2
REVIEW_BODY="_This reviewer could not complete its review (see workflow logs for details)._"
LABELS=""
else
REVIEW_RAW=$(echo "$AI_RESPONSE" | jq -r '.review // "No review provided"')
# Trim whitespace with tr (the verdict is a single token). The previous
# `| xargs` aborted the entire script on an unbalanced quote in the value,
# which—inside this per-reviewer loop—would sink every reviewer.
DECISION=$(echo "$AI_RESPONSE" | jq -r '.fail_pass_workflow // "uncertain"' | tr -d '[:space:]')
LABELS=$(echo "$AI_RESPONSE" | jq -r '.labels_added[]? // empty')
SUCCESS_COUNT=$((SUCCESS_COUNT + 1))
[ "$DECISION" = "fail" ] && ANY_FAIL="true"
# Each review opens with a "## AI Code Review" H2 and closes with the
# singular Friendly AI Reviewer footer. Under a "### 👤 Reviewer N"
# subheading the H2 is redundant and the footer would repeat once per model,
# so strip both here; a single (plural) footer is added to the combined
# comment below. The footer match keys on the stable "Review by [Friendly AI
# Reviewer]" prefix and the \s*\z tail absorbs trailing whitespace; if the
# upstream markers ever change and aren't found, the text is left intact
# (nothing is dropped).
REVIEW_BODY=$(printf '%s' "$REVIEW_RAW" \
| perl -0pe 's/\A\s*##\s*AI Code Review\s*\n+//' \
| perl -0pe 's/\n*---\s*\n\*Review by \[Friendly AI Reviewer\][^\n]*\n?\s*\z//s')
fi
# Lead each section with a blank line so the "### Reviewer N" heading always
# renders (command substitution strips trailing newlines, so the separator
# must precede the heading, not follow the previous body).
COMBINED_REVIEW=$(printf '%s\n\n### 👤 Reviewer %s\n\n%s' "$COMBINED_REVIEW" "$REVIEWER_NUM" "$REVIEW_BODY")
[ -n "$LABELS" ] && ALL_LABELS=$(printf '%s\n%s' "$ALL_LABELS" "$LABELS")
echo "::endgroup::" >&2
done
# Aggregate labels (union across reviewers) and the pass/fail decision (request
# changes if ANY reviewer did).
LABELS=$(printf '%s\n' "$ALL_LABELS" | sed '/^[[:space:]]*$/d' | sort -u)
if [ "$ANY_FAIL" = "true" ]; then
DECISION="fail"
else
DECISION="pass"
fi
REVIEWER_WORD="reviewers"
[ "${#MODELS[@]}" -eq 1 ] && REVIEWER_WORD="reviewer"
STICKY_MARKER="<!-- ai-code-review:sticky -->"
FOOTER="*Reviews by [Friendly AI Reviewer](https://github.com/LearningCircuit/Friendly-AI-Reviewer) - made with ❤️*"
# COMBINED_REVIEW already starts with "\n\n", so it follows the title directly
# (no extra newline) to yield exactly one blank line before "### Reviewer 1".
# The backticks in `%s` are intentional literal Markdown (inline-code the SHA),
# not a command substitution — single quotes keep them literal on purpose.
# shellcheck disable=SC2016
COMMENT_BODY=$(printf '%s\n\n## 🤖 AI Code Review (%s %s)%s\n\n---\n%s\n\n_Last reviewed at commit `%s`_' \
"$STICKY_MARKER" "${#MODELS[@]}" "$REVIEWER_WORD" "$COMBINED_REVIEW" "$FOOTER" "${HEAD_SHA:0:7}")
printf '%s' "$COMMENT_BODY" > "$WORK_DIR/comment_body.md"
printf '%s' "$LABELS" > "$WORK_DIR/labels.txt"
printf '%s' "$DECISION" > "$WORK_DIR/decision.txt"
printf '%s' "$SUCCESS_COUNT" > "$WORK_DIR/success_count.txt"
+594
View File
@@ -0,0 +1,594 @@
#!/bin/bash
# File Whitelist Security Check Script
# Enhanced security checks with comprehensive file type detection
set -euo pipefail
# Load allowed file patterns from shared whitelist (single source of truth)
REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || echo ".")"
WHITELIST_FILE="$REPO_ROOT/.file-whitelist.txt"
if [ ! -f "$WHITELIST_FILE" ]; then
echo "❌ Missing .file-whitelist.txt — cannot run whitelist check."
exit 1
fi
ALLOWED_PATTERNS=()
while IFS= read -r line; do
[[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
ALLOWED_PATTERNS+=("$line")
done < "$WHITELIST_FILE"
# Load per-check ignore lists (exact paths to skip for specific checks)
IGNORE_ENV_FILES=()
IGNORE_ENV_FILE="$REPO_ROOT/.github/security/ignore-env-files.txt"
if [ -f "$IGNORE_ENV_FILE" ]; then
while IFS= read -r line; do
[[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
IGNORE_ENV_FILES+=("$line")
done < "$IGNORE_ENV_FILE"
fi
IGNORE_SUSPICIOUS_FILETYPES=()
IGNORE_SUSPICIOUS_FILE="$REPO_ROOT/.github/security/ignore-suspicious-filetypes.txt"
if [ -f "$IGNORE_SUSPICIOUS_FILE" ]; then
while IFS= read -r line; do
[[ -z "$line" || "$line" =~ ^[[:space:]]*# ]] && continue
IGNORE_SUSPICIOUS_FILETYPES+=("$line")
done < "$IGNORE_SUSPICIOUS_FILE"
fi
# Get list of files to check
if [ "${CHECK_ALL_FILES:-}" = "true" ]; then
echo "🔍 Checking ALL tracked files (release gate mode)..."
CHANGED_FILES=$(git ls-files)
TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
echo "📋 Found $TOTAL_FILE_COUNT tracked files to check"
elif [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
# For PRs: check all files that would be added/modified in the entire PR
echo "🔍 Checking files in PR from $GITHUB_BASE_REF to HEAD..."
CHANGED_FILES=$(git diff --name-only --diff-filter=AM origin/"$GITHUB_BASE_REF"..HEAD)
FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
echo "📋 Found $FILE_COUNT changed files with git diff"
# Also get newly added files across all commits in the PR
# Use a more robust approach that handles edge cases
ALL_NEW_FILES=$(git log --name-only --pretty=format: --diff-filter=A origin/"$GITHUB_BASE_REF"..HEAD 2>/dev/null | grep -v '^$' | sort | uniq || echo "")
NEW_FILE_COUNT=$(echo "$ALL_NEW_FILES" | wc -w)
echo "📋 Found $NEW_FILE_COUNT newly added files with git log"
# Combine both lists and remove duplicates - handle empty ALL_NEW_FILES
if [ -n "$ALL_NEW_FILES" ]; then
CHANGED_FILES=$(echo -e "$CHANGED_FILES\n$ALL_NEW_FILES" | sort | uniq | grep -v '^$')
fi
TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
echo "📋 Total unique files to check: $TOTAL_FILE_COUNT"
else
# For direct pushes: check files in the current commit
echo "🔍 Checking files in latest commit..."
CHANGED_FILES=$(git diff --name-only --diff-filter=AM HEAD~1..HEAD)
TOTAL_FILE_COUNT=$(echo "$CHANGED_FILES" | wc -l)
echo "📋 Found $TOTAL_FILE_COUNT files in direct push"
fi
echo "🔍 Running comprehensive security checks..."
echo ""
FILES_CHECKED=0
WHITELIST_VIOLATIONS=()
LARGE_FILES=()
BINARY_FILES=()
SUSPICIOUS_FILES=()
RESEARCH_DATA_VIOLATIONS=()
FLASK_SECRET_VIOLATIONS=()
ENV_FILE_VIOLATIONS=()
HIGH_ENTROPY_VIOLATIONS=()
HARDCODED_PATH_VIOLATIONS=()
HARDCODED_IP_VIOLATIONS=()
SUSPICIOUS_FILETYPE_VIOLATIONS=()
# Use improved file processing that handles spaces and special characters
while IFS= read -r file; do
[ -z "$file" ] && continue
# Skip deleted files
if [ ! -f "$file" ]; then
continue
fi
FILES_CHECKED=$((FILES_CHECKED + 1))
if [ $((FILES_CHECKED % 10)) -eq 0 ]; then
printf "."
fi
# 1. Whitelist check
ALLOWED=false
for pattern in "${ALLOWED_PATTERNS[@]}"; do
if echo "$file" | grep -qE "$pattern"; then
ALLOWED=true
break
fi
done
if [ "$ALLOWED" = "false" ]; then
WHITELIST_VIOLATIONS+=("$file")
fi
# 2. Large file check (>1MB)
if [ -f "$file" ]; then
FILE_SIZE=$(stat -c%s "$file" 2>/dev/null || echo 0)
if [ "$FILE_SIZE" -gt 1048576 ]; then
LARGE_FILES+=("$file ($(echo "$FILE_SIZE" | awk '{printf "%.1fMB", $1/1024/1024}'))")
fi
fi
# 3. Binary file check
if file "$file" | grep -q "binary"; then
BINARY_FILES+=("$file")
fi
# 4. Secret pattern check - REMOVED: gitleaks workflow handles this more accurately
# 5. Suspicious filename patterns - whitelist approach
SAFE_FILENAME_PATTERNS=(
".*token_counter.*\.py$"
".*migrate.*token.*\.py$"
".*enhanced.*token.*\.md$"
"docs/.*token.*\.md$"
"tests/.*\.py$"
"docs/decisions/.*\.md$"
".*session_passwords\.py$"
".*change_password\.html$"
"tests/ui_tests/.*password.*\.js$"
".*password_validator\.py$"
".*password_utils\.py$"
)
# Check if filename looks suspicious
if echo "$file" | grep -iE "(secret|password|token|\.key$|\.pem$|\.p12$|\.pfx$|\.env$)" >/dev/null; then
# Check if filename matches whitelist patterns
FILENAME_WHITELISTED=false
for pattern in "${SAFE_FILENAME_PATTERNS[@]}"; do
if echo "$file" | grep -qE "$pattern"; then
FILENAME_WHITELISTED=true
break
fi
done
if [ "$FILENAME_WHITELISTED" = "false" ]; then
SUSPICIOUS_FILES+=("$file")
fi
fi
# 6. LDR-specific security checks
# Check for research data leakage
if [ -f "$file" ] && [ -r "$file" ]; then
# Check for hardcoded research queries in non-test files
if ! echo "$file" | grep -qE "(test|mock|example)"; then
if grep -E "(research_id|session_id|query_id).*=.*[\"'][0-9a-f]{8,}[\"']" "$file" >/dev/null 2>&1; then
RESEARCH_DATA_VIOLATIONS+=("$file")
fi
fi
# Check for Flask secret keys
if grep -E "SECRET_KEY.*=.*[\"'][^\"']{16,}[\"']" "$file" >/dev/null 2>&1; then
if ! grep -iE "(os\.environ|getenv|config\[|example|placeholder)" "$file" >/dev/null 2>&1; then
FLASK_SECRET_VIOLATIONS+=("$file")
fi
fi
# Check for environment files
if echo "$file" | grep -E "\.(env|env\.[a-zA-Z]+)$" >/dev/null; then
ENV_IGNORED=false
for epath in "${IGNORE_ENV_FILES[@]+${IGNORE_ENV_FILES[@]}}"; do
[ "$file" = "$epath" ] && ENV_IGNORED=true && break
done
if [ "$ENV_IGNORED" = "false" ]; then
ENV_FILE_VIOLATIONS+=("$file")
fi
fi
# Check for high-entropy strings (potential keys/secrets)
if [ -f "$file" ] && [ -r "$file" ]; then
# Skip HTML files and other safe file types for entropy checks
if ! echo "$file" | grep -qE "\.(html|css|js|json|yml|yaml|md)$"; then
# Skip news_strategy.py which contains example categories in prompts
if ! echo "$file" | grep -qE "news_strategy\.py$"; then
# Look for base64-like strings or hex strings that are suspiciously long
if grep -E "[a-zA-Z0-9+/]{40,}={0,2}|[a-f0-9]{40,}" "$file" >/dev/null 2>&1; then
# Exclude common false positives
if ! grep -iE "(sha256|md5|hash|test|example|fixture|integrity)" "$file" >/dev/null 2>&1; then
HIGH_ENTROPY_VIOLATIONS+=("$file")
fi
fi
fi
fi
fi
# Check for hardcoded paths (Unix/Windows)
if ! echo "$file" | grep -qE "(test|mock|example|\.md$|docker|Docker|\.yml$|\.yaml$|config/paths\.py$|security/path_validator\.py$)"; then
# Look for absolute paths and user home directories
if grep -E "(/home/[a-zA-Z0-9_-]+|/Users/[a-zA-Z0-9_-]+|C:\\\\Users\\\\[a-zA-Z0-9_-]+|/opt/|/var/|/etc/|/usr/local/)" "$file" >/dev/null 2>&1; then
# Exclude common false positives and Docker volume mounts
if ! grep -iE "(example|sample|placeholder|TODO|FIXME|/usr/local/bin|/etc/hosts|documentation|/etc/searxng|volumes?:|docker)" "$file" >/dev/null 2>&1; then
HARDCODED_PATH_VIOLATIONS+=("$file")
fi
fi
fi
# Check for hardcoded IP addresses
if ! echo "$file" | grep -qE "(test|mock|example|\.md$)"; then
# Look for IPv4 addresses (excluding common safe ones)
if grep -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$file" >/dev/null 2>&1; then
# Exclude localhost, documentation IPs, and common examples
if ! grep -E "\b(127\.0\.0\.1|0\.0\.0\.0|localhost|192\.168\.|10\.|172\.(1[6-9]|2[0-9]|3[0-1])\.|255\.255\.255\.|192\.0\.2\.|198\.51\.100\.|203\.0\.113\.)" "$file" >/dev/null 2>&1; then
# Additional check to exclude obvious non-IPs (version numbers, etc)
if grep -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$file" | grep -vE "(version|v[0-9]+\.|release|tag)" >/dev/null 2>&1; then
HARDCODED_IP_VIOLATIONS+=("$file")
fi
fi
fi
fi
# 7. Suspicious file type check - detect potentially dangerous file types
if [ -f "$file" ]; then
# Check if file is in the suspicious-filetypes ignore list
FILETYPE_IGNORED=false
for fpath in "${IGNORE_SUSPICIOUS_FILETYPES[@]+${IGNORE_SUSPICIOUS_FILETYPES[@]}}"; do
[ "$file" = "$fpath" ] && FILETYPE_IGNORED=true && break
done
if [ "$FILETYPE_IGNORED" = "false" ]; then
# Check for suspicious file extensions
if echo "$file" | grep -iE "\.(exe|dll|so|dylib|bin|deb|rpm|msi|dmg|pkg|app)$" >/dev/null; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (executable/binary)")
elif echo "$file" | grep -iE "\.(zip|tar|gz|rar|7z|tar\.gz|tar\.bz2|tgz)$" >/dev/null; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (compressed archive)")
elif echo "$file" | grep -iE "\.(log|tmp|temp|cache|bak|backup|swp|swo|DS_Store|thumbs\.db|desktop\.ini|~|\.orig|\.rej|\.patch)$" >/dev/null; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (temporary/cache)")
elif echo "$file" | grep -iE "\.(png|jpg|jpeg|gif|bmp|tiff|svg|ico|webp)$" >/dev/null; then
# Images are suspicious unless in specific directories
if ! echo "$file" | grep -qE "(^docs/images/|^src/local_deep_research/web/static/favicon\.png$|^installers/.*\.ico$)"; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (image file)")
fi
elif echo "$file" | grep -iE "\.(mp3|mp4|wav|avi|mov|mkv|flv|wmv|webm|m4a|ogg)$" >/dev/null; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (media file)")
elif echo "$file" | grep -iE "\.(csv|xlsx|xls|doc|docx|pdf|ppt|pptx)$" >/dev/null; then
# Documents are suspicious unless in docs directory
if ! echo "$file" | grep -qE "docs/"; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (document file)")
fi
elif echo "$file" | grep -iE "\.(db|sqlite|sqlite3)$" >/dev/null; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (database file)")
elif echo "$file" | grep -iE "node_modules/|__pycache__/|\.pyc$|\.pyo$|\.egg-info/|dist/|build/|\.cache/" >/dev/null; then
SUSPICIOUS_FILETYPE_VIOLATIONS+=("$file (build artifact/cache)")
fi
fi
fi
fi
done <<< "$CHANGED_FILES"
echo ""
echo "✓ Checked $FILES_CHECKED files"
echo ""
# Report all violations with detailed explanations
echo "📊 Security scan completed. Analyzing results..."
echo "📋 Summary of findings:"
echo " - File type violations: ${#WHITELIST_VIOLATIONS[@]}"
echo " - Large files: ${#LARGE_FILES[@]}"
echo " - Binary files: ${#BINARY_FILES[@]}"
echo " - Suspicious filenames: ${#SUSPICIOUS_FILES[@]}"
echo " - Research data leaks: ${#RESEARCH_DATA_VIOLATIONS[@]}"
echo " - Hardcoded Flask secrets: ${#FLASK_SECRET_VIOLATIONS[@]}"
echo " - Environment files: ${#ENV_FILE_VIOLATIONS[@]}"
echo " - High-entropy strings: ${#HIGH_ENTROPY_VIOLATIONS[@]}"
echo " - Hardcoded paths: ${#HARDCODED_PATH_VIOLATIONS[@]}"
echo " - Hardcoded IPs: ${#HARDCODED_IP_VIOLATIONS[@]}"
echo " - Suspicious file types: ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]}"
TOTAL_VIOLATIONS=0
if [ ${#WHITELIST_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ WHITELIST VIOLATIONS - File types not allowed in repository:"
echo " These files don't match any pattern in .file-whitelist.txt."
echo " Binary files (images, audio, etc.) bloat the repo and should NOT be committed."
echo " Only a small set of explicitly listed binary files is allowed — store others externally."
echo " If this is a legitimate text/config file, add it to .file-whitelist.txt (requires maintainer approval)."
echo ""
for violation in "${WHITELIST_VIOLATIONS[@]}"; do
echo " 🚫 $violation"
# Show file type and extension
FILE_EXT="${violation##*.}"
if [ -f "$violation" ]; then
FILE_TYPE=$(file -b "$violation" 2>/dev/null || echo "unknown")
echo " → File extension: .$FILE_EXT"
echo " → File type: $FILE_TYPE"
echo " → First few lines:"
head -3 "$violation" 2>/dev/null | while read -r line; do
echo " $line"
done
fi
echo " → Issue: File extension/type not in .file-whitelist.txt"
echo " → Fix: For text/config files, add pattern to .file-whitelist.txt"
echo " → Note: Binary files should NOT be added to the repo — store them externally"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#WHITELIST_VIOLATIONS[@]}))
fi
if [ ${#LARGE_FILES[@]} -gt 0 ]; then
echo ""
echo "❌ LARGE FILES (>1MB) - Files too big for repository:"
echo " Large files should typically be stored externally or compressed."
echo ""
for violation in "${LARGE_FILES[@]}"; do
echo " 📏 $violation"
echo " → Issue: File size exceeds 1MB limit"
echo " → Fix: Use Git LFS, external storage, or compress the file"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#LARGE_FILES[@]}))
fi
if [ ${#BINARY_FILES[@]} -gt 0 ]; then
echo ""
echo "⚠️ BINARY FILES DETECTED - Review these carefully:"
echo " Binary files may contain sensitive data and can't be easily reviewed."
echo ""
for violation in "${BINARY_FILES[@]}"; do
echo " 🔒 $violation"
echo " → Issue: Binary file detected (contents not reviewable)"
echo " → Action: Verify this file doesn't contain sensitive data"
echo ""
done
fi
if [ ${#SUSPICIOUS_FILES[@]} -gt 0 ]; then
echo ""
echo "❌ SUSPICIOUS FILENAMES - Files with security-sensitive names:"
echo " These filenames contain words that often indicate sensitive files."
echo ""
for violation in "${SUSPICIOUS_FILES[@]}"; do
echo " 🚨 $violation"
# Show which keyword triggered the detection
if echo "$violation" | grep -qi "secret"; then
echo " → Triggered by: 'secret' in filename"
elif echo "$violation" | grep -qi "password"; then
echo " → Triggered by: 'password' in filename"
elif echo "$violation" | grep -qi "token"; then
echo " → Triggered by: 'token' in filename"
elif echo "$violation" | grep -qi "api"; then
echo " → Triggered by: 'api' in filename"
elif echo "$violation" | grep -qi "key"; then
echo " → Triggered by: 'key' in filename"
fi
# Show file content preview if it exists
if [ -f "$violation" ]; then
FILE_TYPE=$(file -b "$violation" 2>/dev/null || echo "unknown")
FILE_SIZE=$(stat -c%s "$violation" 2>/dev/null || echo "unknown")
echo " → File info: $FILE_TYPE (${FILE_SIZE} bytes)"
echo " → Content preview:"
head -3 "$violation" 2>/dev/null | while read -r line; do
echo " $line"
done
fi
echo " → Issue: Filename contains suspicious keywords (secret/password/token/key)"
echo " → Fix: Rename file or add to SAFE_FILENAME_PATTERNS whitelist"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#SUSPICIOUS_FILES[@]}))
fi
# LDR-specific violation reports
if [ ${#RESEARCH_DATA_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ RESEARCH DATA LEAKAGE - Hardcoded research session data found:"
echo " Research IDs and session data should never be hardcoded in production code."
echo ""
for violation in "${RESEARCH_DATA_VIOLATIONS[@]}"; do
echo " 📊 $violation"
# Show the specific lines with research data
echo " → Found hardcoded research data:"
grep -n -E "(research_id|session_id|query_id).*=.*[\"'][0-9a-f]{8,}[\"']" "$violation" 2>/dev/null | head -3 | while read -r line; do
echo " $line"
done
echo " → Issue: Hardcoded research/session IDs in non-test file"
echo " → Fix: Use environment variables or configuration files"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#RESEARCH_DATA_VIOLATIONS[@]}))
fi
if [ ${#FLASK_SECRET_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ FLASK SECRET KEY - Hardcoded Flask secret keys found:"
echo " Flask secret keys must never be hardcoded for security reasons."
echo ""
for violation in "${FLASK_SECRET_VIOLATIONS[@]}"; do
echo " 🔐 $violation"
# Show the specific lines with secret keys
echo " → Found hardcoded Flask secret key:"
grep -n -E "SECRET_KEY.*=.*[\"'][^\"']{16,}[\"']" "$violation" 2>/dev/null | head -3 | while read -r line; do
echo " $line"
done
echo " → Issue: Hardcoded Flask SECRET_KEY"
echo " → Fix: Use os.environ or load from secure config file"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#FLASK_SECRET_VIOLATIONS[@]}))
fi
if [ ${#ENV_FILE_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ ENVIRONMENT FILES - .env files detected:"
echo " Environment files contain sensitive configuration and should never be committed."
echo ""
for violation in "${ENV_FILE_VIOLATIONS[@]}"; do
echo " 🌍 $violation"
echo " → Issue: Environment file in repository"
echo " → Fix: Add to .gitignore and use .env.example instead"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#ENV_FILE_VIOLATIONS[@]}))
fi
if [ ${#HIGH_ENTROPY_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ HIGH ENTROPY STRINGS - Potential secrets or keys detected:"
echo " Long random strings may be API keys, tokens, or other secrets."
echo ""
for violation in "${HIGH_ENTROPY_VIOLATIONS[@]}"; do
echo " 🎲 $violation"
# Show sample of high entropy strings
echo " → Found high-entropy strings:"
grep -n -E "[a-zA-Z0-9+/]{40,}={0,2}|[a-f0-9]{40,}" "$violation" 2>/dev/null | head -3 | while read -r line; do
# Truncate long lines for readability
echo " ${line:0:120}..."
done
echo " → Issue: High-entropy strings that could be secrets"
echo " → Fix: Review and move to environment variables if sensitive"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HIGH_ENTROPY_VIOLATIONS[@]}))
fi
if [ ${#HARDCODED_PATH_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ HARDCODED PATHS - System-specific paths detected:"
echo " Absolute paths can expose system structure and break portability."
echo ""
for violation in "${HARDCODED_PATH_VIOLATIONS[@]}"; do
echo " 📁 $violation"
# Show the specific hardcoded paths
echo " → Found hardcoded paths:"
grep -n -E "(/home/[a-zA-Z0-9_-]+|/Users/[a-zA-Z0-9_-]+|C:\\\\Users\\\\[a-zA-Z0-9_-]+|/opt/|/var/|/etc/|/usr/local/)" "$violation" 2>/dev/null | head -5 | while read -r line; do
echo " $line"
done
echo " → Issue: Hardcoded absolute paths reduce portability"
echo " → Fix: Use relative paths, environment variables, or config files"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HARDCODED_PATH_VIOLATIONS[@]}))
fi
if [ ${#HARDCODED_IP_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ HARDCODED IP ADDRESSES - External IP addresses detected:"
echo " Hardcoded IPs can expose infrastructure and cause connectivity issues."
echo ""
for violation in "${HARDCODED_IP_VIOLATIONS[@]}"; do
echo " 🌐 $violation"
# Show the specific IP addresses
echo " → Found hardcoded IP addresses:"
grep -n -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$violation" 2>/dev/null | grep -v -E "(127\.0\.0\.1|0\.0\.0\.0|localhost|192\.168\.|10\.|172\.(1[6-9]|2[0-9]|3[0-1])\.|255\.255\.255\.|192\.0\.2\.|198\.51\.100\.|203\.0\.113\.)" | head -5 | while read -r line; do
echo " $line"
done
echo " → Issue: Hardcoded IP addresses (non-private/localhost)"
echo " → Fix: Use DNS names, environment variables, or config files"
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#HARDCODED_IP_VIOLATIONS[@]}))
fi
if [ ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]} -gt 0 ]; then
echo ""
echo "❌ SUSPICIOUS FILE TYPES - Potentially dangerous file types detected:"
echo " These file types are commonly used for malware, data leaks, or bloat the repository."
echo ""
for violation in "${SUSPICIOUS_FILETYPE_VIOLATIONS[@]}"; do
echo " 🚨 $violation"
FILE_PATH="${violation%% (*}"
FILE_CATEGORY="${violation##*\\(}"
FILE_CATEGORY="${FILE_CATEGORY%\\)}"
# Provide specific guidance based on file category
case "$FILE_CATEGORY" in
"executable/binary")
echo " → Issue: Executable/binary files can contain malware"
echo " → Fix: Remove executable files, use package managers instead"
;;
"compressed archive")
echo " → Issue: Compressed archives hide their contents from review"
echo " → Fix: Extract contents and commit individual files instead"
;;
"temporary/cache")
echo " → Issue: Temporary/cache files should not be committed"
echo " → Fix: Add to .gitignore and remove from repository"
;;
"image file")
echo " → Issue: Binary image files bloat the repo and should NOT be committed"
echo " → Fix: Store images externally. Only a few explicitly listed images in docs/images/ are allowed"
;;
"media file")
echo " → Issue: Media files are large and rarely needed in code repos"
echo " → Fix: Use external hosting or remove if unnecessary"
;;
"document file")
echo " → Issue: Office documents should be in docs/ directory if needed"
echo " → Fix: Move to docs/ directory or convert to markdown"
;;
"database file")
echo " → Issue: Database files contain data that shouldn't be in source control"
echo " → Fix: Add to .gitignore and use migrations/seeds instead"
;;
"build artifact/cache")
echo " → Issue: Build artifacts and cache files bloat the repository"
echo " → Fix: Add to .gitignore and remove from repository"
;;
esac
# Show file info if available
if [ -f "$FILE_PATH" ]; then
FILE_SIZE=$(stat -c%s "$FILE_PATH" 2>/dev/null || echo "unknown")
if [ "$FILE_SIZE" != "unknown" ]; then
READABLE_SIZE=$(echo "$FILE_SIZE" | awk '{if($1>=1048576) printf "%.1fMB", $1/1048576; else if($1>=1024) printf "%.1fKB", $1/1024; else printf "%dB", $1}')
echo " → File size: $READABLE_SIZE"
fi
fi
echo ""
done
TOTAL_VIOLATIONS=$((TOTAL_VIOLATIONS + ${#SUSPICIOUS_FILETYPE_VIOLATIONS[@]}))
fi
# Final result
if [ $TOTAL_VIOLATIONS -eq 0 ]; then
echo ""
echo "✅ All security checks passed!"
exit 0
else
echo ""
echo "💡 To fix these issues:"
echo " - For text/config files: add pattern to .file-whitelist.txt (requires maintainer approval)"
echo " - For binary files (images, audio, video, archives): do NOT add to the repo"
echo " Binary files permanently bloat git history. Store them externally instead."
echo " Only a small set of explicitly listed binary files is permitted."
echo " - Use environment variables for secrets"
echo " - Never hardcode research data or session IDs"
echo " - Use .env.example files instead of .env"
echo " - Replace absolute paths with relative paths or configs"
echo " - Use DNS names instead of hardcoded IP addresses"
echo ""
echo "⚠️ SECURITY REMINDER: This is a public repository!"
echo " Never commit sensitive data, API keys, or personal information."
exit 1
fi
+102
View File
@@ -0,0 +1,102 @@
#!/usr/bin/env python3
"""Filter ``npm audit --json`` output against an allowlist of advisories.
Reads ``npm audit --json`` on stdin. Exits non-zero (1) if any
moderate-or-higher vulnerability is NOT fully explained by allowlisted
advisories, otherwise 0.
npm reports a parent package as vulnerable when it (transitively) depends on
a vulnerable child; the parent's ``via`` then contains a *string* reference to
the child rather than an advisory object. We resolve those ``via`` chains so a
package that is vulnerable *only* because of an allowlisted advisory is itself
treated as handled — otherwise allowlisting one leaf (e.g. js-yaml) would still
leave its dozen parents (lhci/*, jest/*) failing the gate.
The allowlist is supplied via the ``AUDIT_ALLOWLIST`` env var as a
space-separated list of GHSA IDs. Keep it tight and justified in the workflow
that sets it — only advisories with **no available fix** in **dev/test-only**
tooling belong here.
"""
import json
import os
import sys
ALLOW = set(os.environ.get("AUDIT_ALLOWLIST", "").split())
SEVERITIES = {"moderate", "high", "critical"}
def main() -> int:
raw = sys.stdin.read().strip()
if not raw:
print(
"::error::empty npm audit output (audit did not run?)",
file=sys.stderr,
)
return 1
try:
data = json.loads(raw)
except json.JSONDecodeError as exc:
print(
f"::error::could not parse npm audit JSON: {exc}", file=sys.stderr
)
return 1
# A genuine npm audit report always carries these keys (an empty/clean
# audit still has "vulnerabilities": {}). Their absence means the audit
# did NOT actually run — e.g. a registry/network error returns valid JSON
# like {"message": ..., "error": ...} and npm exits non-zero, which the
# caller's `|| true` swallows. Fail safe (gate red) rather than pass a
# green gate on an audit that never happened.
if "vulnerabilities" not in data or "auditReportVersion" not in data:
print(
"::error::npm audit did not return a valid report (audit failed "
f"to run?): {data.get('message', 'unknown error')}",
file=sys.stderr,
)
return 1
vulns = data.get("vulnerabilities") or {}
if not isinstance(vulns, dict):
print(
"::error::npm audit 'vulnerabilities' is not an object — "
"treating as a failed audit",
file=sys.stderr,
)
return 1
def reachable_ghsas(name, seen):
"""All advisory GHSAs reachable from a package's via-chain."""
if name in seen:
return set()
seen.add(name)
found = set()
for via in (vulns.get(name) or {}).get("via", []):
if isinstance(via, dict) and "GHSA-" in (via.get("url") or ""):
# Take only the GHSA token, not any trailing path/query the
# URL might carry (e.g. .../GHSA-xxxx/foo?bar).
tail = via["url"].split("GHSA-", 1)[1]
found.add("GHSA-" + tail.split("/")[0].split("?")[0])
elif isinstance(via, str):
found |= reachable_ghsas(via, seen)
return found
unhandled = 0
for name, info in vulns.items():
if info.get("severity") not in SEVERITIES:
continue
ghsas = reachable_ghsas(name, set())
if ghsas and ghsas <= ALLOW:
print(f" allowlisted: {name} ({info['severity']}) {sorted(ghsas)}")
else:
unhandled += 1
detail = sorted(ghsas - ALLOW) if ghsas else "no-GHSA"
print(f" UNHANDLED: {name} ({info['severity']}) {detail}")
suffix = "y" if unhandled == 1 else "ies"
print(f" -> {unhandled} non-allowlisted moderate+ vulnerabilit{suffix}")
return 1 if unhandled else 0
if __name__ == "__main__":
sys.exit(main())
+122
View File
@@ -0,0 +1,122 @@
#!/bin/bash
# Validates that all docker-compose image references use SHA256 digests
# Prevents supply chain attacks by ensuring immutable image references
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Configuration - images that are allowed without SHA digests
ALLOWED_EXCEPTIONS=(
"localdeepresearch/local-deep-research:latest" # Own image, built by CI
)
# Check if an image reference is in the exceptions list
is_exception() {
local image="$1"
for exception in "${ALLOWED_EXCEPTIONS[@]}"; do
if [[ "$image" == "$exception" ]]; then
return 0
fi
done
return 1
}
# Validate a single docker-compose file
validate_compose_file() {
local file="$1"
local violations=0
local line_num=0
while IFS= read -r line; do
line_num=$((line_num + 1))
# Check if this line contains an image reference
if [[ "$line" =~ ^[[:space:]]*image:[[:space:]]*(.+)$ ]]; then
local image="${BASH_REMATCH[1]}"
image=$(echo "$image" | tr -d '"' | xargs) # Remove quotes and whitespace
# Skip if it's an exception
if is_exception "$image"; then
echo -e "${YELLOW} Line $line_num: $image (exception)${NC}"
continue
fi
# Check if image has SHA digest
if [[ ! "$image" =~ @sha256: ]]; then
echo -e "${RED} ❌ Line $line_num: Missing SHA digest${NC}"
echo -e "${RED} Image: $image${NC}"
violations=$((violations + 1))
else
echo -e "${GREEN} ✓ Line $line_num: $image${NC}"
fi
fi
done < "$file"
return $violations
}
# Main validation logic
main() {
local total_violations=0
local files_checked=0
echo "🔍 Validating docker-compose image pinning..."
echo ""
# Find all docker-compose files
while IFS= read -r compose_file; do
# Skip cookiecutter templates and examples (documentation only)
if [[ "$compose_file" =~ cookiecutter-docker/ ]] || [[ "$compose_file" =~ examples/ ]]; then
echo -e "${YELLOW}⏭ Skipping: $compose_file (template/example)${NC}"
continue
fi
echo "📄 Checking: $compose_file"
if validate_compose_file "$compose_file"; then
: # No violations
else
violations=$?
total_violations=$((total_violations + violations))
fi
files_checked=$((files_checked + 1))
echo ""
done < <(find . -name "docker-compose*.yml" -o -name "docker-compose*.yaml")
# Summary
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "📊 Summary:"
echo " Files checked: $files_checked"
echo " Violations: $total_violations"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
if [ $total_violations -gt 0 ]; then
echo ""
echo -e "${RED}❌ Found $total_violations unpinned images in docker-compose files${NC}"
echo ""
echo "Images must use SHA256 digests for security and reproducibility."
echo ""
echo "To fix:"
echo " 1. Pull the image: docker pull <image:tag>"
echo " 2. Get digest: docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
echo " 3. Update file: image: <image:tag>@sha256:..."
echo ""
echo "Example:"
echo " # Bad"
echo " image: ollama/ollama:latest"
echo ""
echo " # Good"
echo " image: ollama/ollama:latest@sha256:8850b8b33936b9fb..."
exit 1
fi
echo ""
echo -e "${GREEN}✅ All docker-compose images properly pinned${NC}"
exit 0
}
main "$@"
+164
View File
@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""
Validates that all GitHub Actions workflow service containers and container images
use SHA256 digests for supply chain security.
Prevents tag tampering attacks by ensuring immutable image references.
"""
import sys
from pathlib import Path
from typing import List, Tuple
try:
import yaml
except ImportError:
print("❌ Error: PyYAML is required. Install with: pip install pyyaml")
sys.exit(1)
# ANSI color codes
class Colors:
RED = "\033[0;31m"
GREEN = "\033[0;32m"
YELLOW = "\033[1;33m"
BLUE = "\033[0;34m"
NC = "\033[0m" # No Color
def has_sha_digest(image_ref: str) -> bool:
"""Check if image reference includes SHA256 digest."""
return "@sha256:" in image_ref
def validate_workflow(workflow_path: Path) -> List[Tuple[str, str, str]]:
"""
Validate a workflow file for unpinned images.
Returns:
List of (job_name, violation_type, image) tuples
"""
violations = []
try:
with open(workflow_path, encoding="utf-8") as f:
data = yaml.safe_load(f)
if not isinstance(data, dict) or "jobs" not in data:
return violations
# Check each job
for job_name, job_def in data["jobs"].items():
if not isinstance(job_def, dict):
continue
# Check container: field
if "container" in job_def:
container = job_def["container"]
# Container can be a string or dict with 'image' key
if isinstance(container, str):
if not has_sha_digest(container):
violations.append((job_name, "container", container))
elif isinstance(container, dict) and "image" in container:
image = container["image"]
if not has_sha_digest(image):
violations.append((job_name, "container", image))
# Check services: field
if "services" in job_def and isinstance(job_def["services"], dict):
for service_name, service_def in job_def["services"].items():
if isinstance(service_def, dict) and "image" in service_def:
image = service_def["image"]
if not has_sha_digest(image):
violations.append(
(job_name, f"service '{service_name}'", image)
)
except yaml.YAMLError as e:
print(f"{Colors.RED}❌ YAML parse error in {workflow_path}:{Colors.NC}")
print(f" {e}")
# Return a violation to fail the check
violations.append(("parse_error", "error", str(e)))
except Exception as e:
print(
f"{Colors.RED}❌ Error processing {workflow_path}: {e}{Colors.NC}"
)
violations.append(("error", "error", str(e)))
return violations
def main():
"""Main validation logic."""
workflows_dir = Path(".github/workflows")
if not workflows_dir.exists():
print(
f"{Colors.RED}❌ .github/workflows directory not found{Colors.NC}"
)
sys.exit(1)
print("🔍 Validating GitHub Actions workflow images...")
print()
total_violations = 0
files_checked = 0
# Process all workflow files
for workflow_file in sorted(workflows_dir.glob("*.yml")) + sorted(
workflows_dir.glob("*.yaml")
):
violations = validate_workflow(workflow_file)
if violations:
print(f"{Colors.RED}📄 {workflow_file.name}:{Colors.NC}")
for job_name, violation_type, image in violations:
print(
f"{Colors.RED} ❌ Job '{job_name}' {violation_type}: {image}{Colors.NC}"
)
print()
total_violations += len(violations)
else:
print(f"{Colors.GREEN}{workflow_file.name}{Colors.NC}")
files_checked += 1
# Summary
print("" * 50)
print("📊 Summary:")
print(f" Files checked: {files_checked}")
print(f" Violations: {total_violations}")
print("" * 50)
if total_violations > 0:
print()
print(
f"{Colors.RED}❌ Found {total_violations} unpinned images in workflow files{Colors.NC}"
)
print()
print("Service container images must use SHA256 digests for security.")
print()
print("To fix:")
print(" 1. Pull the image: docker pull <image:tag>")
print(
" 2. Get digest: docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
)
print(" 3. Update workflow:")
print()
print("Example:")
print(" services:")
print(" redis:")
print(f"{Colors.RED} image: redis:alpine # Bad{Colors.NC}")
print(
f"{Colors.GREEN} image: redis:alpine@sha256:... # Good{Colors.NC}"
)
return 1
print()
print(f"{Colors.GREEN}✅ All workflow images properly pinned{Colors.NC}")
return 0
if __name__ == "__main__":
sys.exit(main())
+4
View File
@@ -0,0 +1,4 @@
# Exact file paths excluded from the environment-file check.
# These are NOT actual .env files — they are templates/documentation.
src/local_deep_research/defaults/.env.template
@@ -0,0 +1,5 @@
# Exact file paths excluded from the suspicious-file-type check.
# These are legitimate assets required by the application.
src/local_deep_research/web/static/sounds/error.mp3
src/local_deep_research/web/static/sounds/success.mp3
+36
View File
@@ -0,0 +1,36 @@
# Configuration for automated main-to-dev sync
sync:
# Enable/disable automatic syncing
enabled: false
# Auto-merge PRs when there are no conflicts
auto_merge_clean: true
# Branches to sync from main to
target_branches:
- dev
# Ignore sync if these strings are in the commit message
ignore_patterns:
- "Merge pull request"
- "from LearningCircuit/local-deep-research:dev"
- "[skip sync]"
- "[no sync]"
# Labels to add to sync PRs
labels:
clean: ["sync", "auto-merge"]
conflicts: ["sync", "conflicts", "manual-review-required"]
# Reviewers to assign for conflict resolution
conflict_reviewers:
- "LearningCircuit"
- "djpetti"
- "HashedViking"
# Additional settings
settings:
# Delete sync branches after successful merge
delete_branch_after_merge: true
# Maximum number of sync attempts per day
max_daily_syncs: 10
+50
View File
@@ -0,0 +1,50 @@
# GitHub Actions Workflows
This directory contains GitHub Actions workflows for automated development tasks.
## Update NPM Dependencies Workflow
**File**: `update-npm-dependencies.yml`
### Purpose
Automatically updates NPM dependencies across all package.json files in the project and fixes security vulnerabilities.
### Triggers
- **Scheduled**: Every Thursday at 08:00 UTC (day after PDM updates)
- **Manual**: Can be triggered manually via GitHub Actions UI
- **Workflow Call**: Can be called by other workflows
### What it does
1. **Security Audit**: Runs `npm audit` to identify security vulnerabilities
2. **Security Fixes**: Automatically fixes moderate+ severity vulnerabilities with `npm audit fix`
3. **Dependency Updates**: Updates all dependencies to latest compatible versions with `npm update`
4. **Testing**: Runs relevant tests to ensure updates don't break functionality
5. **Pull Request**: Creates automated PR with all changes
### Directories Managed
- `/` - Main web dependencies (Vite, Bootstrap, etc.)
- `/tests/ui_tests` - UI test dependencies (Puppeteer)
### Branch Strategy
- Creates branch: `update-npm-dependencies-{run_number}`
- Targets: `main` branch
- Labels: `maintenance`
- Reviewers: `djpetti,HashedViking,LearningCircuit`
### Security Focus
- Only auto-fixes moderate+ severity vulnerabilities
- Preserves compatible version updates (no major version bumps)
- Runs security audit before and after updates
- Requires tests to pass before creating PR
### Manual Usage
You can manually trigger this workflow:
1. Go to Actions tab in GitHub
2. Select "Update NPM dependencies"
3. Click "Run workflow"
4. Optionally specify custom npm arguments
### Troubleshooting
- If tests fail, the PR won't be created
- Check the workflow logs for specific error messages
- Security issues that can't be auto-fixed will need manual intervention
@@ -0,0 +1,58 @@
name: Advanced-search change reminder
on:
# pull_request (not pull_request_target) — fork PRs get a read-only
# token, so the label call 403s; the script catches that and stays
# green (the original comment claimed the 403 was silent, but nothing
# caught it, so fork PRs got a red check — e.g. #4872). Intentional
# no-op on forks: the advisory label is only meaningful for
# internal-branch PRs anyway.
pull_request:
types: [opened, synchronize, reopened]
paths:
- 'src/local_deep_research/advanced_search_system/**'
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
label:
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
pull-requests: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Apply benchmark-needed label
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
// Advisory only — asks reviewers to consider running
// src/local_deep_research/benchmarks/compare_configurations()
// before merging changes that can affect research quality.
// The label's description (set on the label itself in repo
// settings) carries the reminder text so reviewers see it
// on hover without needing an inline comment.
try {
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
labels: ['benchmark-needed']
});
} catch (err) {
// Fork PRs get a read-only token — the call 403s ("Resource
// not accessible by integration"). Advisory label only, so
// log and stay green (same pattern as pr-triage.yml).
if (err.status !== 403) throw err;
console.log('Label call returned 403 (read-only fork token). Skipping.');
}
+291
View File
@@ -0,0 +1,291 @@
name: AI Code Reviewer
# Reviews are opt-in per PR: add the ai_code_review label to request one
# (the label is auto-removed after the run so re-adding it re-triggers).
# The automatic opened/synchronize/ready_for_review triggers were removed —
# reviewing every push burned reviewer-API spend on PRs nobody asked to have
# reviewed, and a billing outage (HTTP 402) then showed a red check on every
# PR in the repo.
#
# Residual risk accepted: the diff text goes into the LLM prompt, so a
# hostile diff can try to steer the posted comment or labels (prompt
# injection). The maintainer-applied label gate plus the advisory-only
# decision (FAIL_ON_REQUESTED_CHANGES defaults to false) bound that
# blast radius, and it exists for internal PRs too.
on:
# zizmor: ignore[dangerous-triggers] — pull_request_target is required so
# the review also works on fork PRs: fork-triggered pull_request runs get
# no secrets (OPENROUTER_API_KEY) and a read-only token, so the reviewer
# could neither call the model nor post its comment. This is safe under
# two invariants — KEEP BOTH when editing this file:
# 1. Only the ai_code_review label triggers a run, and only users with
# triage permission can add labels: every run on fork code is an
# explicit maintainer action, never attacker-initiated.
# 2. Nothing from the PR head is ever executed. The checkout is the
# trusted base branch (pull_request_target default), so
# .github/scripts/combine-ai-reviews.sh runs from base; the PR head
# is fetched as *data only* to compute the diff.
pull_request_target:
types: [labeled]
# Per-PR concurrency with cancel-in-progress: re-adding the label while a
# review is still running cancels the now-stale run. Keyed on the PR number
# so runs for *different* PRs never cancel each other — that repo-wide
# cancellation was the problem in the earlier attempt (#3554, reverted
# #3599), not per-PR cancellation.
#
# The labeled trigger fires for *every* label added to the PR, but the job's
# `if` only proceeds when the label is ai_code_review. Without the run_id
# suffix below, a skip-only labeled event would still enter this group,
# cancel a real in-flight review, and then skip itself — leaving the PR with
# no review at all. Routing those into a unique group keyed on run_id means
# they cancel nothing; the `if` discards them quietly.
concurrency:
group: >-
ai-code-review-${{ github.event.pull_request.number }}${{
(github.event.action == 'labeled' && github.event.label.name != 'ai_code_review')
&& format('-skip-{0}', github.run_id) || ''
}}
cancel-in-progress: true
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
comprehensive-review:
name: AI Code Review
runs-on: ubuntu-latest
timeout-minutes: 30
environment: ci
# Only the ai_code_review label triggers a review (works on drafts too);
# every other labeled event is discarded here.
if: github.event.label.name == 'ai_code_review'
permissions:
contents: read
pull-requests: write
issues: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
# SECURITY: this checks out the trusted BASE branch (the
# pull_request_target default), which is what makes running
# .github/scripts/combine-ai-reviews.sh below safe on fork PRs.
# Never add `ref: ${{ github.event.pull_request.head.sha }}` here —
# that would execute fork-controlled scripts with secrets in env.
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y jq
- name: Get PR diff
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
# The PR head — possibly fork code — is fetched as *data only*
# through the base repo's refs/pull/N/head mirror (present for
# fork and same-repo PRs alike); it is never checked out and
# nothing from it is executed. Env vars prevent template
# injection from malicious branch names.
git fetch origin "$BASE_REF" "+refs/pull/$PR_NUMBER/head:refs/remotes/origin/pr-head"
git diff "origin/$BASE_REF...origin/pr-head" --no-color > diff.txt
- name: Download AI reviewer script
run: |
curl -fsSL https://raw.githubusercontent.com/LearningCircuit/Friendly-AI-Reviewer/main/ai-reviewer.sh -o ai-reviewer.sh
chmod +x ai-reviewer.sh
- name: AI Code Review
id: ai-review
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
# Comma-separated list of models. Each model produces one independent
# review, presented as "Reviewer 1", "Reviewer 2", ... in a single
# combined comment. Falls back to the shared AI_MODEL (also used by
# release.yml, which must stay a single model) when unset, so a
# one-entry list reproduces the original single-review behavior.
AI_REVIEW_MODELS: ${{ vars.AI_REVIEW_MODELS || vars.AI_MODEL || 'moonshotai/kimi-k2-thinking' }}
AI_TEMPERATURE: ${{ vars.AI_TEMPERATURE || '0.1' }}
AI_MAX_TOKENS: ${{ vars.AI_MAX_TOKENS || '64000' }}
MAX_DIFF_SIZE: ${{ vars.MAX_DIFF_SIZE || '800000' }}
EXCLUDE_FILE_PATTERNS: ${{ vars.EXCLUDE_FILE_PATTERNS || '*.lock,*.min.js,*.min.css,package-lock.json,yarn.lock' }}
STRUCTURED_OUTPUT: ${{ vars.STRUCTURED_OUTPUT || 'true' }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO_FULL_NAME: ${{ github.repository }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
FAIL_ON_REQUESTED_CHANGES: ${{ vars.FAIL_ON_REQUESTED_CHANGES || 'false' }}
DEBUG_MODE: ${{ vars.DEBUG_MODE || 'false' }}
run: |
echo "Running AI code review..."
# Parse the comma-separated model list. Each entry becomes one
# independent reviewer ("Reviewer 1", "Reviewer 2", ...). A single
# entry reproduces the original one-review behavior.
# `read` returns non-zero at EOF; the here-string always appends a
# trailing newline so it returns 0 here, but `|| true` keeps a future
# edge case from tripping `set -e`. An empty/whitespace AI_REVIEW_MODELS
# yields only empty entries, which the filter below drops, correctly
# falling through to the "No models configured" error.
IFS=',' read -ra RAW_MODELS <<< "$AI_REVIEW_MODELS" || true
MODELS=()
for m in "${RAW_MODELS[@]}"; do
# Trim surrounding whitespace with parameter expansion. (xargs would
# choke on a model id containing a quote and word-collapse internal
# spaces — model ids shouldn't have either, but this is robust.)
m="${m#"${m%%[![:space:]]*}"}" # strip leading whitespace
m="${m%"${m##*[![:space:]]}"}" # strip trailing whitespace
[ -n "$m" ] && MODELS+=("$m")
done
if [ "${#MODELS[@]}" -eq 0 ]; then
echo "❌ No models configured in AI_REVIEW_MODELS"
exit 1
fi
echo "Configured ${#MODELS[@]} reviewer model(s)"
# Launch every model concurrently so the job's wall-clock is the
# slowest single review, not the sum of all of them. Each reviewer
# writes its JSON response and its diagnostics to its own index-keyed
# files, so the "Reviewer N" numbering stays stable and the parallel
# output never interleaves. AI_MODEL is overridden per call; all other
# settings (temperature, token limits, diff size, context fetching)
# are shared.
pids=()
for i in "${!MODELS[@]}"; do
AI_MODEL="${MODELS[i]}" bash ai-reviewer.sh < diff.txt \
> "resp_$i.json" 2> "err_$i.log" &
pids[i]=$!
done
# Record each reviewer's exit code to code_<i> (a failing model must
# not abort the others, so disable errexit around wait).
for i in "${!MODELS[@]}"; do
set +e
wait "${pids[i]}"
echo $? > "code_$i"
set -e
done
# Assemble the combined comment with the extracted, unit-tested helper
# (.github/scripts/combine-ai-reviews.sh). It reads resp_<i>.json /
# code_<i> from the working dir and writes the comment body, label set,
# decision, and success count back out. Keeping the assembly in a
# script is what lets tests/ci/test_combine_ai_reviews.py cover the
# parsing, footer-stripping, label union, and pass/fail logic without
# any network or GitHub API access.
bash .github/scripts/combine-ai-reviews.sh "$PWD" "${MODELS[@]}"
COMMENT_BODY="$(cat comment_body.md)"
LABELS="$(cat labels.txt)"
DECISION="$(cat decision.txt)"
SUCCESS_COUNT="$(cat success_count.txt)"
# Post (or update) the reviews as a single sticky PR comment. On each
# push we edit the same comment in place instead of stacking a new one
# per push. The hidden marker identifies our comment specifically, so
# we never touch comments from humans or other bots. (Review comments
# made before this change lack the marker and are left as-is.)
# A commit-sha line lets reviewers tell which push the (edited-in-place)
# comment reflects, since GitHub only shows a vague "edited" marker.
# This literal MUST match the marker the helper script prepends to the
# comment body.
STICKY_MARKER="<!-- ai-code-review:sticky -->"
# STICKY_MARKER is interpolated into the jq program below, so it must
# stay free of `"` and `\`. gh api's --jq takes no --arg, so we can't
# pass it as a jq variable; keeping the marker a literal constant is
# what keeps this safe. The filter yields a single id (we maintain
# exactly one sticky comment), so no post-filtering is needed.
EXISTING_COMMENT_ID=$(gh api "repos/$REPO_FULL_NAME/issues/$PR_NUMBER/comments" \
--paginate \
--jq "[.[] | select(.body | contains(\"$STICKY_MARKER\"))] | last | .id // empty")
if [ -n "$EXISTING_COMMENT_ID" ]; then
echo "Updating existing AI review comment ($EXISTING_COMMENT_ID)"
printf '%s' "$COMMENT_BODY" | gh api \
"repos/$REPO_FULL_NAME/issues/comments/$EXISTING_COMMENT_ID" \
--method PATCH -F body=@- >/dev/null
else
echo "Posting new AI review comment"
printf '%s' "$COMMENT_BODY" | gh pr comment "$PR_NUMBER" --repo "$REPO_FULL_NAME" -F -
fi
# Add labels to PR. Iterate line-by-line (labels.txt is one label per
# line) rather than word-splitting, so a multi-word label such as
# "good first issue" is treated as a single label, not three.
echo "Labels to add: $LABELS"
while IFS= read -r label; do
[ -n "$label" ] || continue
echo "Adding label: $label"
if gh label create "$label" --color "0366d6" --description "Auto-created by AI reviewer" 2>/dev/null; then
echo "Created new label: $label"
else
echo "Label already exists: $label"
fi
if gh issue edit "$PR_NUMBER" --add-label "$label" 2>/dev/null; then
echo "Successfully added label: $label"
else
echo "Failed to add label: $label"
fi
done <<< "$LABELS"
# Handle workflow decision (after posting review and labels)
echo "AI decision: $DECISION"
# Store the decision as a step output for the downstream
# "Fail Workflow if Requested" step. $GITHUB_OUTPUT is the modern
# mechanism — values are scoped to steps.<id>.outputs.<name> and
# cannot re-enter the runner's process environment, eliminating the
# $GITHUB_ENV injection class (zizmor github-env audit #8017).
#
# Defense-in-depth: an allowlist before the write blocks newline
# injection. If a manipulated model response put a newline in
# $DECISION, the bare echo above could write an additional output
# key. Outputs can't reach the process environment so impact is
# minimal, but the case statement closes the door cheaply and
# replaces the lost-on-rework intent of #4985.
case "$DECISION" in pass|fail) ;; *) DECISION="fail" ;; esac
echo "DECISION=$DECISION" >> "$GITHUB_OUTPUT"
# Remove the ai_code_review label so re-adding it re-triggers a review.
echo "Removing ai_code_review label to allow easy re-triggering"
if gh issue edit "$PR_NUMBER" --remove-label "ai_code_review" 2>/dev/null; then
echo "Successfully removed ai_code_review label"
else
echo "Failed to remove ai_code_review label (may have been already removed)"
fi
# If every reviewer failed, the posted comment carries only failure
# notes — surface that as a red workflow run rather than a silent pass,
# matching the original single-review behavior on an unusable response.
if [ "$SUCCESS_COUNT" -eq 0 ]; then
echo "❌ All ${#MODELS[@]} reviewer(s) failed to produce a usable review."
exit 1
fi
echo "✅ AI review completed successfully ($SUCCESS_COUNT/${#MODELS[@]} reviewers)"
- name: Fail Workflow if Requested
if: env.FAIL_ON_REQUESTED_CHANGES == 'true' && steps.ai-review.outputs.DECISION == 'fail'
run: |
echo "❌ AI requested changes and FAIL_ON_REQUESTED_CHANGES is enabled. Failing workflow."
echo "The review has been posted above. Please address the requested changes."
exit 1
- name: Cleanup
if: always()
run: |
rm -f diff.txt
# Per-reviewer response/log/exit-code files from the parallel fan-out,
# plus the assembly helper's output files.
rm -f resp_*.json err_*.log code_* comment_body.md labels.txt decision.txt success_count.txt
# Only remove ai_response.txt if it exists (only created in debug mode)
if [ -f "ai_response.txt" ]; then
rm -f ai_response.txt
fi
@@ -0,0 +1,264 @@
name: Backwards Compatibility
on:
workflow_call: # Allows this workflow to be called by other workflows
pull_request:
types: [opened, synchronize, reopened]
branches: [main]
paths:
# Core encryption and database management
- 'src/local_deep_research/database/encrypted_db.py'
- 'src/local_deep_research/database/sqlcipher_*.py'
- 'src/local_deep_research/database/auth_db.py'
- 'src/local_deep_research/database/session_*.py'
# Database initialization and migrations
- 'src/local_deep_research/database/initialize.py'
- 'src/local_deep_research/database/library_init.py'
- 'src/local_deep_research/database/thread_local_session.py'
# Database models (schema changes)
- 'src/local_deep_research/database/models/*.py'
# Encryption settings configuration
- 'src/local_deep_research/settings/env_definitions/db_config.py'
# Dependencies
- 'pyproject.toml'
- 'pdm.lock'
# Alembic migration infrastructure
- 'src/local_deep_research/database/alembic_runner.py'
- 'src/local_deep_research/database/migrations/**'
# Test files (ensure tests themselves are valid)
- 'tests/performance/database/test_backwards_compatibility.py'
- 'tests/database/test_encryption_constants.py'
- 'tests/database/test_alembic_migrations.py'
- 'tests/database/test_migration_0003_indexes.py'
- 'tests/database/test_migration_0004_app_settings.py'
- 'tests/performance/database/scripts/create_compat_db.py'
push:
branches: [main]
release:
types: [created]
schedule:
# Weekly on Sundays at 2am UTC - catch dependency drift
- cron: '0 2 * * 0'
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)
permissions: {}
jobs:
# Fast test - runs on every trigger including PRs.
# Provides quick feedback on encryption constant stability.
encryption-constants:
runs-on: ubuntu-latest
timeout-minutes: 10
name: Verify Encryption Constants
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
sudo docker image prune --all --force
- name: Install dependencies
run: pdm install --dev
- name: Run encryption constants tests
run: |
pdm run pytest tests/database/test_encryption_constants.py -v --tb=short
# Slow test - full PyPI version compatibility.
# Skipped on PRs because it installs previous PyPI versions which is slow
# and the encryption-constants job already provides fast PR feedback.
pypi-compatibility:
runs-on: ubuntu-latest
timeout-minutes: 30
name: PyPI Version Compatibility
permissions:
contents: read
# Run on main push, releases, schedule, manual dispatch, and workflow_call (release gate)
if: |
github.event_name == 'push' ||
github.event_name == 'release' ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch' ||
github.event_name == 'workflow_call'
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
sudo docker image prune --all --force
- name: Install test dependencies
run: |
python -m pip install --upgrade pip
pip install pytest
- name: Get previous PyPI version
id: prev-version
run: |
# Get list of available versions from PyPI
VERSIONS_OUTPUT=$(pip index versions local-deep-research 2>&1) || true
if echo "$VERSIONS_OUTPUT" | grep -q "Available versions:"; then
VERSIONS=$(echo "$VERSIONS_OUTPUT" | grep "Available versions:" | cut -d: -f2 | tr ',' '\n' | tr -d ' ')
else
echo "::warning::Could not fetch versions from PyPI: ${VERSIONS_OUTPUT}"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ -z "$VERSIONS" ]; then
echo "::warning::No versions found on PyPI"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# Get current and previous versions
CURRENT=$(echo "$VERSIONS" | head -1)
PREVIOUS=$(echo "$VERSIONS" | head -2 | tail -1)
if [ "$CURRENT" = "$PREVIOUS" ] || [ -z "$PREVIOUS" ]; then
echo "No previous version available"
echo "skip=true" >> "$GITHUB_OUTPUT"
exit 0
fi
{
echo "current=$CURRENT"
echo "previous=$PREVIOUS"
echo "skip=false"
} >> "$GITHUB_OUTPUT"
echo "Previous version: $PREVIOUS, Current: $CURRENT"
- name: Create database with previous version
id: create-db
if: steps.prev-version.outputs.skip != 'true'
run: |
# Create isolated venv for previous version
python -m venv prev_venv
source prev_venv/bin/activate
# Install previous version — may fail if the published package
# has broken dependency metadata (e.g. requests version conflict).
# Upgrade pip for reliable dependency resolution in fresh venvs.
python -m pip install --upgrade pip
if ! pip install "local-deep-research==${{ steps.prev-version.outputs.previous }}" 2>&1; then
echo "::warning::Previous version ${{ steps.prev-version.outputs.previous }} has unresolvable dependencies — skipping compat test"
echo "install_failed=true" >> "$GITHUB_OUTPUT"
deactivate
exit 0
fi
echo "install_failed=false" >> "$GITHUB_OUTPUT"
# Create test database
mkdir -p test_db
python tests/performance/database/scripts/create_compat_db.py test_db compat_user "TestPass123!"
deactivate
echo "Database created with version ${{ steps.prev-version.outputs.previous }}"
ls -la test_db/
- name: Install current version
if: steps.prev-version.outputs.skip != 'true' && steps.create-db.outputs.install_failed != 'true'
run: pdm install --dev
- name: Test opening database with current version
if: steps.prev-version.outputs.skip != 'true' && steps.create-db.outputs.install_failed != 'true'
run: |
# Run the backwards compatibility test with RUN_SLOW_TESTS enabled
RUN_SLOW_TESTS=true pdm run pytest tests/performance/database/test_backwards_compatibility.py -v --tb=short
- name: Report skipped
if: steps.prev-version.outputs.skip == 'true' || steps.create-db.outputs.install_failed == 'true'
run: |
if [ "${{ steps.create-db.outputs.install_failed }}" = "true" ]; then
echo "::notice::Backwards compatibility test skipped - previous version has unresolvable dependency conflicts"
else
echo "::notice::Backwards compatibility test skipped - no previous PyPI version available"
fi
# Database migration tests - validates Alembic migration infrastructure.
# Runs on any trigger — including PRs that touch migration-relevant paths
# (gated by the top-level `pull_request: paths:` filter).
migration-tests:
runs-on: ubuntu-latest
timeout-minutes: 20
name: Database Migration Tests
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
sudo docker image prune --all --force
- name: Install dependencies
run: pdm install --dev
- name: Run Alembic migration tests
run: |
pdm run pytest tests/database/test_alembic_migrations.py -v --tb=short
- name: Run migration version tests
run: |
pdm run pytest tests/database/test_migration_0003_indexes.py tests/database/test_migration_0004_app_settings.py -v --tb=short
- name: Run database initialization tests
run: |
pdm run pytest tests/database/test_initialize_functions.py tests/test_database_initialization.py -v --tb=short
- name: Run encrypted DB integration tests
run: |
pdm run pytest tests/auth_tests/test_encrypted_db.py -v --tb=short
+117
View File
@@ -0,0 +1,117 @@
name: Bearer Security Scan
on:
workflow_dispatch:
workflow_call: # Called by release-gate.yml
schedule:
# Run security scan daily at 4 AM UTC
- cron: '0 4 * * *'
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
bearer-scan:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Run Bearer SAST Scanner
uses: bearer/bearer-action@828eeb928ce2f4a7ca5ed57fb8b59508cb8c79bc # v2
with:
scanner: sast,secrets
config-file: bearer.yml
format: sarif
output: bearer-results.sarif
# DO NOT change to exit-code: 1 — findings are enforced via SARIF alerts
# in the release gate (check-code-scanning-alerts job in release-gate.yml).
# Failing here would break CI without adding security value.
exit-code: 0
# Fail loudly if Bearer produced no SARIF — never fabricate an empty one.
# An empty-results SARIF uploaded under the bearer-security category would
# make GitHub mark every previously-open Bearer alert as fixed, silently
# clearing real findings. Mirror the Grype/Trivy jobs: error + exit 1, skip
# upload. bearer-action writes a SARIF even with zero findings, so a missing
# file means a real scan failure, not a clean run.
- name: Ensure SARIF file exists
id: check-sarif
if: always()
run: |
if [ -f bearer-results.sarif ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "::error::Bearer did not produce a SARIF file — scan needs to be rerun"
exit 1
fi
- name: Upload Bearer results to GitHub Security tab
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always() && steps.check-sarif.outputs.exists == 'true'
with:
sarif_file: bearer-results.sarif
category: bearer-security
- name: Upload Bearer results as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always() && steps.check-sarif.outputs.exists == 'true'
with:
name: bearer-scan-results
path: bearer-results.sarif
retention-days: 7
- name: Display Bearer summary
if: always()
run: |
{
echo "## Bearer Security Scan Summary"
echo ""
} >> "$GITHUB_STEP_SUMMARY"
if [ -f bearer-results.sarif ]; then
# Count findings by level
ERRORS=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='error']))" 2>/dev/null || echo "0")
WARNINGS=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='warning']))" 2>/dev/null || echo "0")
NOTES=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); results=data.get('runs',[{}])[0].get('results',[]); print(len([r for r in results if r.get('level')=='note']))" 2>/dev/null || echo "0")
TOTAL=$(python3 -c "import json; data=json.load(open('bearer-results.sarif')); print(len(data.get('runs',[{}])[0].get('results',[])))" 2>/dev/null || echo "0")
{
echo "📊 **Scan Results:**"
echo "- **Errors:** $ERRORS"
echo "- **Warnings:** $WARNINGS"
echo "- **Notes:** $NOTES"
echo "- **Total:** $TOTAL"
echo ""
} >> "$GITHUB_STEP_SUMMARY"
if [ "$ERRORS" -gt 0 ]; then
echo "⚠️ **Action Required:** Error-level issues found" >> "$GITHUB_STEP_SUMMARY"
elif [ "$WARNINGS" -gt 0 ]; then
echo "⚠️ **Review Recommended:** Warning-level issues found" >> "$GITHUB_STEP_SUMMARY"
else
echo "✅ **No Error or Warning level issues found**" >> "$GITHUB_STEP_SUMMARY"
fi
{
echo ""
echo "📋 **Details:**"
echo "- Bearer scans for sensitive data flow and secrets exposure"
echo "- Results uploaded to GitHub Security tab"
echo "- SARIF report available in artifacts"
} >> "$GITHUB_STEP_SUMMARY"
else
echo "❌ Bearer scan failed or no results generated" >> "$GITHUB_STEP_SUMMARY"
fi
+32
View File
@@ -0,0 +1,32 @@
name: Check Configuration Docs
# Manual-only check. Config docs are auto-regenerated by the version bump
# workflow (version_check.yml) on every push to main, so a blocking PR
# check is unnecessary and creates manual work for contributors.
on:
workflow_dispatch:
permissions:
contents: read
jobs:
check-config-docs:
runs-on: ubuntu-latest
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Check configuration docs are up to date
run: python scripts/generate_config_docs.py --check
+43
View File
@@ -0,0 +1,43 @@
name: Check Environment Variables
on:
pull_request:
workflow_call: # Called by ci-gate.yml for release pipeline
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions:
contents: read
jobs:
check-env-vars:
runs-on: ubuntu-latest
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install dependencies
run: |
pip install loguru==0.7.3 sqlalchemy==2.0.36 sqlalchemy-utc==0.14.0 platformdirs==4.3.6 pydantic==2.10.4
- name: Run environment variable validation
run: |
python tests/settings/env_vars/test_env_var_usage.py
@@ -0,0 +1,52 @@
name: Check Workflow Status Dashboard
# Fails when a workflow file is added/renamed without a corresponding row
# in docs/ci/workflow-status.md. Pure structural check — no GitHub API
# calls, no live data — so it runs fast and doesn't need any auth.
#
# To fix a failure: regenerate the dashboard with
# `pdm run python scripts/generate_workflow_status.py`
# This requires `gh` authenticated against the repo. If you can't run it
# locally, ping a maintainer to regenerate, or add a temporary placeholder
# `\`<your-new-workflow>.yml\`` mention in the file's manual-edit region
# to unblock the PR.
on:
pull_request:
paths:
- '.github/workflows/**'
- 'docs/ci/workflow-status.md'
- 'scripts/generate_workflow_status.py'
workflow_dispatch:
permissions:
contents: read
jobs:
check-structure:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install PyYAML
# Pinned to match pdm.lock; the rest of the repo uses exact
# `==` pins for ad-hoc workflow installs (see e.g.
# validate-image-pinning.yml). Floating `~=` ranges can pick up
# yanked / replaced patch versions silently.
run: pip install pyyaml==6.0.3
- name: Verify dashboard structure
run: python scripts/generate_workflow_status.py --check-structure
+100
View File
@@ -0,0 +1,100 @@
name: Checkov IaC Security Scan
on:
workflow_dispatch:
workflow_call: # Called by release-gate.yml
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
checkov:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
security-events: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install Checkov
run: pip install checkov==3.2.499
# Reverted to CLI approach due to known bug in bridgecrewio/checkov-action
# when running multiple consecutive action calls (heredoc delimiter overflow)
# See: https://github.com/bridgecrewio/checkov-action/issues/170
# See: https://github.com/bridgecrewio/checkov/issues/5866
- name: Run Checkov on Dockerfile
run: |
checkov -f Dockerfile \
--framework dockerfile \
--skip-check CKV_DOCKER_3 \
-o cli -o sarif \
--output-file-path console,checkov-docker.sarif
- name: Run Checkov on docker-compose files
run: |
checkov \
-f docker-compose.yml \
-f docker-compose.gpu.override.yml \
-f docker-compose.tts.yml \
-f docker-compose.unraid.yml \
--framework yaml \
-o cli -o sarif \
--output-file-path console,checkov-compose.sarif
- name: Run Checkov on GitHub Actions workflows
run: |
checkov -d .github/workflows \
--framework github_actions \
--skip-check CKV_GHA_7 \
-o cli -o sarif \
--output-file-path console,checkov-workflows.sarif
- name: Check SARIF files exist
id: check-sarif
if: always()
run: |
[ -f checkov-docker.sarif ] && echo "docker=true" >> "$GITHUB_OUTPUT"
[ -f checkov-compose.sarif ] && echo "compose=true" >> "$GITHUB_OUTPUT"
[ -f checkov-workflows.sarif ] && echo "workflows=true" >> "$GITHUB_OUTPUT"
- name: Upload Dockerfile SARIF results
if: always() && steps.check-sarif.outputs.docker == 'true'
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
with:
sarif_file: checkov-docker.sarif
category: checkov-dockerfile
- name: Upload docker-compose SARIF results
if: always() && steps.check-sarif.outputs.compose == 'true'
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
with:
sarif_file: checkov-compose.sarif
category: checkov-compose
- name: Upload GitHub Actions SARIF results
if: always() && steps.check-sarif.outputs.workflows == 'true'
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
with:
sarif_file: checkov-workflows.sarif
category: checkov-workflows
- name: Summary
if: always()
run: |
echo "=== Checkov IaC Security Scan Summary ==="
echo "SARIF results uploaded to GitHub Security tab"
echo "Scan completed (strict mode - failures block CI)"
+259
View File
@@ -0,0 +1,259 @@
name: CI Gate
# CI quality gate for the release pipeline.
#
# Ensures all PR-quality checks (linting, type checking, tests, validation)
# run and pass before any release proceeds. Complements the security-focused
# release-gate.yml with code quality and correctness checks.
#
# Architecture:
# release.yml → ci-gate.yml → {pre-commit, mypy, docker-tests, ...}
# Nesting depth: release.yml(1) → ci-gate.yml(2) → workflow(3) — safe limit.
on:
workflow_call: # Called by release.yml
secrets:
OPENROUTER_API_KEY:
required: false
workflow_dispatch: # Manual trigger
permissions: {} # Minimal top-level for OSSF Scorecard
jobs:
# ============================================
# Code Quality
# ============================================
pre-commit:
uses: ./.github/workflows/pre-commit.yml
permissions:
contents: read
mypy-type-check:
uses: ./.github/workflows/mypy-type-check.yml
permissions:
contents: read
# ============================================
# Test Suites
# ============================================
docker-tests:
uses: ./.github/workflows/docker-tests.yml
with:
strict-mode: true
permissions:
contents: write # Needed by pytest-tests for gh-pages deployment
pull-requests: write # Needed by pytest-tests for PR comments (no-ops in release context)
secrets:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
# ============================================
# Validation
# ============================================
validate-image-pinning:
uses: ./.github/workflows/validate-image-pinning.yml
permissions:
contents: read
file-whitelist-check:
uses: ./.github/workflows/file-whitelist-check.yml
with:
check-all-files: true
permissions:
contents: read
check-env-vars:
uses: ./.github/workflows/check-env-vars.yml
permissions:
contents: read
security-file-write-check:
uses: ./.github/workflows/security-file-write-check.yml
permissions:
contents: read
# ============================================
# Summary job that reports overall status
# ============================================
ci-gate-summary:
name: CI Gate Summary
runs-on: ubuntu-latest
needs:
# Code Quality
- pre-commit
- mypy-type-check
# Test Suites
- docker-tests
# Validation
- validate-image-pinning
- file-whitelist-check
- check-env-vars
- security-file-write-check
if: always()
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Check CI scan results
env:
PRE_COMMIT_RESULT: ${{ needs.pre-commit.result }}
MYPY_RESULT: ${{ needs.mypy-type-check.result }}
DOCKER_TESTS_RESULT: ${{ needs.docker-tests.result }}
IMAGE_PINNING_RESULT: ${{ needs.validate-image-pinning.result }}
FILE_WHITELIST_RESULT: ${{ needs.file-whitelist-check.result }}
ENV_VARS_RESULT: ${{ needs.check-env-vars.result }}
FILE_WRITE_RESULT: ${{ needs.security-file-write-check.result }}
run: |
# Redirect all output to GITHUB_STEP_SUMMARY (fixes SC2129)
exec >> "$GITHUB_STEP_SUMMARY"
# Count results first
FAILED=""
PASS_COUNT=0
FAIL_COUNT=0
check_result() {
local result="$1"
if [ "$result" = "success" ]; then
PASS_COUNT=$((PASS_COUNT + 1))
return 0
else
FAIL_COUNT=$((FAIL_COUNT + 1))
FAILED="true"
return 1
fi
}
# Check all results silently first
check_result "$PRE_COMMIT_RESULT" || true
check_result "$MYPY_RESULT" || true
check_result "$DOCKER_TESTS_RESULT" || true
check_result "$IMAGE_PINNING_RESULT" || true
check_result "$FILE_WHITELIST_RESULT" || true
check_result "$ENV_VARS_RESULT" || true
check_result "$FILE_WRITE_RESULT" || true
TOTAL=$((PASS_COUNT + FAIL_COUNT))
# ============================================
# BIG STATUS BANNER
# ============================================
if [ -z "$FAILED" ]; then
echo "# :white_check_mark: CI GATE: PASSED"
echo ""
echo "> **All $TOTAL CI checks passed successfully.**"
echo "> This release is approved from a code quality perspective."
else
echo "# :x: CI GATE: FAILED"
echo ""
echo "> **$FAIL_COUNT of $TOTAL checks failed.** Release is blocked."
echo "> Review the failures below and fix before releasing."
fi
echo ""
echo "---"
echo ""
echo "## Detailed Results"
echo ""
# Reset for detailed output
FAILED=""
# ============================================
# Code Quality
# ============================================
echo "### Code Quality"
if [ "$PRE_COMMIT_RESULT" = "success" ]; then
echo ":white_check_mark: **Pre-commit (linting, formatting)**: Passed"
else
echo ":x: **Pre-commit (linting, formatting)**: $PRE_COMMIT_RESULT"
FAILED="true"
fi
if [ "$MYPY_RESULT" = "success" ]; then
echo ":white_check_mark: **Mypy Type Check**: Passed"
else
echo ":x: **Mypy Type Check**: $MYPY_RESULT"
FAILED="true"
fi
# ============================================
# Test Suites
# ============================================
echo ""
echo "### Test Suites"
if [ "$DOCKER_TESTS_RESULT" = "success" ]; then
echo ":white_check_mark: **Docker Tests (pytest + UI + LLM + infra + smoke)**: Passed"
else
echo ":x: **Docker Tests (pytest + UI + LLM + infra + smoke)**: $DOCKER_TESTS_RESULT"
FAILED="true"
fi
# ============================================
# Validation
# ============================================
echo ""
echo "### Validation"
if [ "$IMAGE_PINNING_RESULT" = "success" ]; then
echo ":white_check_mark: **Docker Image Pinning**: Passed"
else
echo ":x: **Docker Image Pinning**: $IMAGE_PINNING_RESULT"
FAILED="true"
fi
if [ "$FILE_WHITELIST_RESULT" = "success" ]; then
echo ":white_check_mark: **File Whitelist Security**: Passed"
else
echo ":x: **File Whitelist Security**: $FILE_WHITELIST_RESULT"
FAILED="true"
fi
if [ "$ENV_VARS_RESULT" = "success" ]; then
echo ":white_check_mark: **Environment Variables**: Passed"
else
echo ":x: **Environment Variables**: $ENV_VARS_RESULT"
FAILED="true"
fi
if [ "$FILE_WRITE_RESULT" = "success" ]; then
echo ":white_check_mark: **Security File Writes**: Passed"
else
echo ":x: **Security File Writes**: $FILE_WRITE_RESULT"
FAILED="true"
fi
# ============================================
# Final result with prominent summary
# ============================================
echo ""
echo "---"
echo ""
if [ -n "$FAILED" ]; then
echo "## :rotating_light: Action Required"
echo ""
echo "| Status | Result |"
echo "|--------|--------|"
echo "| **Gate** | :x: **BLOCKED** |"
echo "| **Passed** | $PASS_COUNT |"
echo "| **Failed** | $FAIL_COUNT |"
echo ""
echo "_Fix the failing checks above before releasing._"
exit 1
else
echo "## :tada: Ready for Release"
echo ""
echo "| Status | Result |"
echo "|--------|--------|"
echo "| **Gate** | :white_check_mark: **APPROVED** |"
echo "| **Passed** | $PASS_COUNT / $TOTAL |"
echo ""
echo "_All CI checks passed. Security scans run as separate gate in release pipeline._"
fi
+57
View File
@@ -0,0 +1,57 @@
name: Claude Code Review
on:
pull_request:
types: [labeled]
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
claude-review:
# Only run when 'claude-review' label is added (opt-in)
if: github.event.label.name == 'claude-review'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: ci
permissions:
contents: read
pull-requests: write
issues: write
id-token: write # Required for OIDC authentication with claude-code-action
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@558b1d6cab4085c7753fe402c10bef0fbb92ac7a # v1.0.165
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# CLI arguments for Claude (v1.0+ format)
claude_args: |
--model claude-opus-4-5-20251101
# Prompt for automated review (no @claude mention needed)
prompt: |
Please review this pull request and provide feedback on:
- Code quality and best practices
- Potential bugs or issues
- Performance considerations
- Security concerns
- Test coverage
Be constructive and helpful in your feedback.
+114
View File
@@ -0,0 +1,114 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL Advanced"
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '45 5 * * 0'
workflow_call: # Called by release-gate.yml
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
analyze:
name: Analyze (${{ matrix.language }})
# Runner size impacts CodeQL analysis time. To learn more, please see:
# - https://gh.io/recommended-hardware-resources-for-running-codeql
# - https://gh.io/supported-runners-and-hardware-resources
# - https://gh.io/using-larger-runners (GitHub.com only)
# Consider using larger runners or machines with greater resources for possible analysis time improvements.
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
timeout-minutes: 30
permissions:
contents: read
security-events: write
packages: read
actions: read
strategy:
fail-fast: false
matrix:
include:
- language: javascript-typescript
build-mode: none
- language: python
build-mode: none
# CodeQL supports the following values keywords for 'language': 'actions', 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
# Use `c-cpp` to analyze code written in C, C++ or both
# Use 'java-kotlin' to analyze code written in Java, Kotlin or both
# Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
# To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
# see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
# If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
# Add any setup steps before running the `github/codeql-action/init` action.
# This includes steps like installing compilers or runtimes (`actions/setup-node`
# or others). This is typically only required for manual builds.
# - name: Setup runtime (example)
# uses: actions/setup-example@v1
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
config-file: ./.github/codeql/codeql-config.yml
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality
# If the analyze step fails for one of the languages you are analyzing with
# "We were unable to automatically build your code", modify the matrix above
# to set the build mode to "manual" for that language. Then modify this step
# to build your code.
# ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
- if: matrix.build-mode == 'manual'
shell: bash
run: |
echo 'If you are using a "manual" build mode for one or more of the' \
'languages you are analyzing, replace this with the commands to build' \
'your code, for example:'
echo ' make bootstrap'
echo ' make release'
exit 1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4.36.2
with:
category: "/language:${{matrix.language}}"
@@ -0,0 +1,195 @@
name: Compose Integration Test
# Brings up the bundled docker-compose.yml end-to-end in CI and verifies the
# whole stack (searxng, ollama, local-deep-research) reaches a healthy state.
# This is the test that would have caught #3874 (broken cap_drop on searxng)
# before users hit it. See #3886 for the rationale.
#
# Cost: ~3-6 min per run depending on cache state. Runs only via the release
# pipeline (`release.yml` includes it as `compose-integration-gate`) and on
# manual dispatch. NOT inside `release-gate.yml`, because release-gate runs
# daily and the failure modes here (compose/image changes) are tied to
# actual release events, not time. NOT on pull_request — too expensive for
# per-PR feedback latency.
#
# Do not move this into release-gate's daily cron. Ongoing compose drift
# between releases is already covered by `compose-published-smoke.yml`,
# which runs weekly against main's compose.yml + the *published* Docker Hub
# image — the more meaningful drift to catch, since users pull the
# published image, not main's build override. PR #3962 tried adding this
# to the daily cron and was closed for that reason.
#
# Tests the docker-compose.yml as users actually run it — no test-only
# overrides. The model pre-pull was removed from the compose itself in this
# same PR so the stack starts in seconds rather than waiting on a multi-GB
# download that the test doesn't need.
on:
workflow_call:
workflow_dispatch:
permissions:
contents: read
jobs:
compose-up:
name: docker compose up + healthcheck
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
# Build the LDR image from the working tree and tag it as the name the
# bundled compose references. Compose then uses the local image instead
# of pulling the published one — so we test the current code path, not
# whatever's on Docker Hub.
- name: Build LDR image with the published tag
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
# Same target as docker-tests.yml's `ldr-prod` build, so its cache
# layers (scope=ldr-prod, populated by ci-gate during the same
# release pipeline) are reusable here. Explicit target guards
# against future Dockerfile reorderings making the default stage
# something other than `ldr`.
target: ldr
load: true
tags: localdeepresearch/local-deep-research:latest
# Read from both our own scope and the shared `ldr-prod` scope —
# whichever has the layers wins. Falls back to a fresh build if
# neither does (e.g. on a brand-new branch). Only write to our
# own scope so we don't poison the cross-workflow cache.
cache-from: |
type=gha,scope=compose-integration
type=gha,scope=ldr-prod
cache-to: type=gha,mode=max,scope=compose-integration
- name: Bring up the stack
# `docker compose up -d` pulls any image it doesn't have locally
# (default pull_policy: missing), so ollama and searxng are pulled
# inline. The LDR image is the locally-built one from the previous
# step — compose sees it's already present and uses it as-is.
# `--no-build` defends against a future docker-compose.yml change
# adding a `build:` directive — we want the image we tagged above,
# not a fresh build that bypasses the cache strategy.
run: docker compose up -d --no-build
- name: Wait for the stack to be healthy and serving
# Budget: 6 min covers cold daemon startup + LDR migrations + flask
# boot + slow CI runners. Without the model pull this completes in
# ~1-2 min on a warm runner.
#
# Container resolution: always via `docker compose ps -q <service>`
# so we don't couple to compose's `container_name:` values. If those
# drift, this still works.
run: |
set -euo pipefail
deadline=$(( $(date +%s) + 360 ))
cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
status() {
local cid=$1
[ -n "$cid" ] || { echo missing; return; }
docker inspect -f "{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}" "$cid" 2>/dev/null || echo missing
}
while :; do
now=$(date +%s)
if [ "$now" -ge "$deadline" ]; then
echo "::error::Timed out after 6 min waiting for stack to be healthy"
docker compose ps
exit 1
fi
ollama_id=$(cid_for ollama)
searxng_id=$(cid_for searxng)
ldr_id=$(cid_for local-deep-research)
ollama_h=$(status "$ollama_id")
searxng_h=$(status "$searxng_id")
ldr_h=$(status "$ldr_id")
echo "[$(date -u +%H:%M:%S)] ollama=$ollama_h searxng=$searxng_h ldr=$ldr_h"
# All three services have healthchecks: ollama and searxng via
# docker-compose.yml, LDR via the Dockerfile (HEALTHCHECK at
# Dockerfile:306, probing /api/v1/health). status() returns the
# health status when one is defined, so require "healthy" for
# all three — strictly stronger signal than "running".
if [ "$ollama_h" = "healthy" ] && [ "$searxng_h" = "healthy" ] && [ "$ldr_h" = "healthy" ]; then
echo "All services healthy."
break
fi
# Fail fast on any container exiting non-zero rather than burning
# the whole 6 min budget.
for svc in ollama searxng local-deep-research; do
cid=$(cid_for "$svc")
[ -n "$cid" ] || continue
s=$(docker inspect -f '{{.State.Status}}' "$cid" 2>/dev/null || true)
if [ "$s" = "exited" ] || [ "$s" = "dead" ]; then
echo "::error::Container for service $svc has exited"
exit 1
fi
done
sleep 10
done
- name: Probe LDR HTTP endpoint
# Avoids `curl ... | grep` so we don't need pipefail to surface curl
# failures — the HTTP code is captured directly via -w and checked
# with a case statement. `|| echo "000"` is the sentinel for true
# network failures (connection refused, DNS, etc.) so we can log
# on retry.
#
# Deliberately no `-f`: with `-f`, curl exits non-zero on HTTP
# 4xx/5xx AND suppresses -w output — which would collapse "404",
# "503", and network-error all into "000" and erase the most
# interesting failure signal (LDR up but serving an error page).
# Without -f, every HTTP response gives us its real code; only
# true network failures fall through to "000".
run: |
set -euo pipefail
for i in $(seq 1 30); do
code=$(curl -sS -o /dev/null -w "%{http_code}" http://localhost:5000/ 2>/dev/null || echo "000")
case "$code" in
200|301|302|303|307|308)
echo "LDR is serving on :5000 (HTTP $code)"
exit 0
;;
esac
echo "Waiting for LDR HTTP (attempt $i/30, last code: $code)..."
sleep 5
done
echo "::error::LDR HTTP probe failed after ~150s"
exit 1
- name: Dump compose state and logs
if: always()
run: |
echo "::group::docker compose ps"
docker compose ps || true
echo "::endgroup::"
for svc in ollama searxng local-deep-research; do
echo "::group::$svc logs"
docker compose logs --no-color --tail=500 "$svc" || true
echo "::endgroup::"
done
- name: Tear down
if: always()
run: docker compose down -v --remove-orphans
@@ -0,0 +1,257 @@
name: Compose Published-Image Smoke
# Tests "main's docker-compose.yml + the currently-published Docker Hub image"
# end-to-end. Catches drift between compose changes that have landed on main
# and the image artefact that users actually pull when they follow the README
# quickstart:
#
# curl -O .../docker-compose.yml && docker compose up -d
#
# This complements the release-pipeline test in compose-integration-test.yml,
# which builds the image from the working tree (validating "this code's
# compose with this code's image") and runs only at release time + manual
# dispatch. That test cannot catch the drift case — the published image
# always lags main between releases.
#
# Triggers: weekly schedule + manual dispatch only. No pull_request trigger
# because PRs don't change the published image — by definition this test
# can only fail on drift that's already landed on main.
#
# On failure, opens a tracking issue (or comments on the existing one) so a
# scheduled-job failure isn't lost in the noise.
on:
workflow_dispatch:
schedule:
# 05:00 UTC every Monday — offset from release-gate's 02:00 daily cron
# to avoid runner contention. Weekly cadence is enough since the failure
# modes are slow-moving (compose-vs-published-image drift) and the test
# itself burns CI minutes.
- cron: '0 5 * * 1'
permissions:
contents: read
jobs:
compose-up:
name: docker compose up + healthcheck (published image)
runs-on: ubuntu-latest
timeout-minutes: 25
permissions:
contents: read
issues: write # for the failure-issue step
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout (for docker-compose.yml on default branch)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
# Pull all three services from their registries, including the LDR
# image. This is the whole point of this workflow — exercise the
# exact artefact users get from `docker compose up -d`.
- name: Pull all images
run: docker compose pull
- name: Bring up the stack
# `--no-build` defends against a future docker-compose.yml change
# adding a `build:` directive — this workflow specifically tests the
# *published* image artifact, not a fresh build from source.
run: docker compose up -d --no-build
- name: Wait for the stack to be healthy and serving
# Same wait logic as compose-integration-test.yml. 6 min budget
# covers cold daemon startup + LDR migrations + flask boot.
# Container resolution: always via `docker compose ps -q <service>`
# so we don't couple to compose's `container_name:` values.
run: |
set -euo pipefail
deadline=$(( $(date +%s) + 360 ))
cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
status() {
local cid=$1
[ -n "$cid" ] || { echo missing; return; }
docker inspect -f "{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}" "$cid" 2>/dev/null || echo missing
}
while :; do
now=$(date +%s)
if [ "$now" -ge "$deadline" ]; then
echo "::error::Timed out after 6 min waiting for stack to be healthy"
docker compose ps
exit 1
fi
ollama_id=$(cid_for ollama)
searxng_id=$(cid_for searxng)
ldr_id=$(cid_for local-deep-research)
ollama_h=$(status "$ollama_id")
searxng_h=$(status "$searxng_id")
ldr_h=$(status "$ldr_id")
echo "[$(date -u +%H:%M:%S)] ollama=$ollama_h searxng=$searxng_h ldr=$ldr_h"
# All three services have healthchecks: ollama and searxng via
# docker-compose.yml, LDR via the Dockerfile (HEALTHCHECK at
# Dockerfile:306, probing /api/v1/health). status() returns the
# health status when one is defined, so require "healthy" for
# all three.
if [ "$ollama_h" = "healthy" ] && [ "$searxng_h" = "healthy" ] && [ "$ldr_h" = "healthy" ]; then
echo "All services healthy."
break
fi
for svc in ollama searxng local-deep-research; do
cid=$(cid_for "$svc")
[ -n "$cid" ] || continue
s=$(docker inspect -f '{{.State.Status}}' "$cid" 2>/dev/null || true)
if [ "$s" = "exited" ] || [ "$s" = "dead" ]; then
echo "::error::Container for service $svc has exited"
exit 1
fi
done
sleep 10
done
- name: Probe LDR HTTP endpoint
# Avoids `curl ... | grep` so we don't need pipefail to surface curl
# failures — the HTTP code is captured directly via -w and checked
# with a case statement. `|| echo "000"` is the sentinel for true
# network failures (connection refused, DNS, etc.) so we can log
# on retry.
#
# Deliberately no `-f`: with `-f`, curl exits non-zero on HTTP
# 4xx/5xx AND suppresses -w output — which would collapse "404",
# "503", and network-error all into "000" and erase the most
# interesting failure signal (LDR up but serving an error page).
# Without -f, every HTTP response gives us its real code; only
# true network failures fall through to "000".
run: |
set -euo pipefail
for i in $(seq 1 30); do
code=$(curl -sS -o /dev/null -w "%{http_code}" http://localhost:5000/ 2>/dev/null || echo "000")
case "$code" in
200|301|302|303|307|308)
echo "LDR is serving on :5000 (HTTP $code)"
exit 0
;;
esac
echo "Waiting for LDR HTTP (attempt $i/30, last code: $code)..."
sleep 5
done
echo "::error::LDR HTTP probe failed after ~150s"
exit 1
- name: Dump compose state and logs
if: always()
run: |
echo "::group::docker compose ps"
docker compose ps || true
echo "::endgroup::"
for svc in ollama searxng local-deep-research; do
echo "::group::$svc logs"
docker compose logs --no-color --tail=500 "$svc" || true
echo "::endgroup::"
done
# Capture image digests for both the workflow log (every run, via
# the cat below) and the auto-failure-issue body (failure runs
# only, via the next step). Always-logging gives us audit and
# bisection signal — "what was the published image SHA when this
# passed on date X?" — without needing to re-run the workflow.
# Resolve every container by service name through `docker compose
# ps -q` so we don't depend on `container_name:` values.
cid_for() { docker compose ps -q "$1" 2>/dev/null || true; }
{
echo "## Image digests"
for svc in ollama searxng local-deep-research; do
cid=$(cid_for "$svc")
if [ -n "$cid" ]; then
img=$(docker inspect -f '{{.Image}}' "$cid" 2>/dev/null || echo missing)
echo "- $svc: image=$img container=$cid"
else
echo "- $svc: container=missing"
fi
done
} > /tmp/digests.md
echo "::group::Image digests"
cat /tmp/digests.md
echo "::endgroup::"
- name: Open or update failure-tracking issue
if: failure() && github.event_name == 'schedule'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
# Stable title prefix so we dedup across weeks. If an issue with
# this title is already open, comment on it; otherwise create a
# fresh one. We don't auto-close on success — let a human triage.
TITLE="[compose-published-smoke] published-image drift detected"
DATE=$(date -u +%Y-%m-%d)
DIGESTS=$(cat /tmp/digests.md 2>/dev/null || echo "(digests not captured)")
BODY=$(cat <<MARKDOWN
The weekly **published-image smoke test** failed on $DATE.
This usually means \`main\`'s \`docker-compose.yml\` and the currently-published
\`localdeepresearch/local-deep-research:latest\` on Docker Hub have drifted —
either a compose change landed without an image republish, or the
published image (or one of its dependencies) regressed.
**Workflow run:** $RUN_URL
**What to check first:**
- Compose changes since the last release: \`git log --oneline <last-release-tag>..main -- docker-compose.yml\`
- Whether the LDR image needs a republish (cut a release, or the image was repointed)
- Upstream image regressions: ollama, searxng image digests below
$DIGESTS
*This issue is auto-managed by \`.github/workflows/compose-published-smoke.yml\`.
Subsequent failures will add comments here. Close it when the underlying drift is fixed
and a successful run rolls in.*
MARKDOWN
)
# Find existing open issue with the same title
NUM=$(gh issue list -R "$REPO" \
--state open \
--search "in:title \"[compose-published-smoke] published-image drift detected\"" \
--json number,title \
--jq '.[0].number // empty' \
--limit 1)
if [ -n "$NUM" ]; then
echo "Commenting on existing issue #$NUM"
# `--body=` (= form) prevents gh from interpreting body content
# as flags if it ever starts with `-`. Same below.
gh issue comment "$NUM" -R "$REPO" --body="$BODY"
else
echo "Creating new issue"
gh issue create -R "$REPO" \
--title="$TITLE" \
--label="bug,docker,ci-cd" \
--body="$BODY"
fi
- name: Tear down
# `|| true` so a teardown flake (daemon hiccup, hung container) can't
# flip the job to failure() after the smoke test itself passed —
# which would falsely trigger the auto-issue step. CI runners are
# ephemeral, so any leftover containers/volumes vanish with the
# runner regardless.
if: always()
run: docker compose down -v --remove-orphans || true
+166
View File
@@ -0,0 +1,166 @@
name: Container Security
on:
workflow_call: # Called by release-gate.yml
workflow_dispatch:
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
trivy-scan:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Free up disk space
run: |
# Remove unnecessary files to free up disk space for Docker image scanning
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo docker image prune -af
df -h
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Build Docker image for scanning
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
push: false
load: true
tags: local-deep-research:scan
cache-from: type=gha,scope=trivy-scan
cache-to: type=gha,mode=max,scope=trivy-scan
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
with:
image-ref: local-deep-research:scan
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH,MEDIUM'
ignore-unfixed: true
scan-type: 'image'
trivyignores: '.trivyignore'
version: 'v0.69.2'
- name: Check if SARIF file exists
id: check-sarif
if: always()
run: |
if [ -f trivy-results.sarif ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "::error::Trivy did not produce trivy-results.sarif — scan needs to be rerun"
exit 1
fi
- name: Upload Trivy scan results to GitHub Security tab
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always() && steps.check-sarif.outputs.exists == 'true'
with:
sarif_file: 'trivy-results.sarif'
category: container-security
- name: Upload Trivy scan results as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always() && steps.check-sarif.outputs.exists == 'true'
with:
name: trivy-scan-results
path: trivy-results.sarif
retention-days: 7 # Reduced for security
- name: Display Trivy summary
if: always()
run: |
{
echo "## Container Security Scan Summary"
echo ""
if [ -f trivy-results.sarif ]; then
echo "✅ Trivy scan completed - Results available in Security tab"
echo "📊 Scan results uploaded as artifact"
else
echo "❌ Trivy scan failed"
fi
} >> "$GITHUB_STEP_SUMMARY"
# Additional job for Dockerfile security analysis
dockerfile-scan:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Run Trivy config scan on Dockerfile
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
with:
scan-type: 'config'
scan-ref: '.'
format: 'sarif'
output: 'trivy-config-results.sarif'
hide-progress: true
version: 'v0.69.2'
- name: Check if config SARIF file exists
id: check-config-sarif
if: always()
run: |
if [ -f trivy-config-results.sarif ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "::error::Trivy did not produce trivy-config-results.sarif — scan needs to be rerun"
exit 1
fi
- name: Upload Trivy config scan results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always() && steps.check-config-sarif.outputs.exists == 'true'
with:
name: trivy-config-results
path: trivy-config-results.sarif
retention-days: 7 # Reduced for security
- name: Display config scan summary
if: always()
run: |
{
echo "## Docker Configuration Security Analysis"
echo ""
if [ -f trivy-config-results.sarif ]; then
echo "✅ Docker configuration analysis completed"
echo "📋 Check for Docker best practices and security misconfigurations"
else
echo "❌ Docker configuration scan failed"
fi
} >> "$GITHUB_STEP_SUMMARY"
+224
View File
@@ -0,0 +1,224 @@
name: 🚨 Danger Zone Alert
on:
pull_request:
types: [opened, synchronize]
paths:
# Explicit paths only — no keyword globs. A filename containing
# "decrypt" or "password" is not enough to imply security impact.
#
# ⚠️ KEEP IN SYNC with two other lists below:
# 1. EXPLICIT_WATCHED array in the "Validate watchlist" step
# (drops glob entries; literal paths only — used for drift detection).
# 2. `case` patterns in the "Classify critical changes" step
# (routes each path to a label bucket).
# A path here that's missing from (2) silently fails to label;
# a path in (2) that's missing here never triggers the workflow.
#
# Encryption / at-rest crypto / credential storage
- 'src/local_deep_research/database/encrypted_db.py'
- 'src/local_deep_research/database/sqlcipher_*.py'
- 'src/local_deep_research/database/credential_store_base.py'
- 'src/local_deep_research/database/backup/**'
# Authentication / sessions / passwords / auth DB / tenant isolation
- 'src/local_deep_research/web/auth/**'
- 'src/local_deep_research/database/auth_db.py'
- 'src/local_deep_research/database/session_passwords.py'
- 'src/local_deep_research/database/temp_auth.py'
- 'src/local_deep_research/database/session_context.py'
# Web hardening — CSP, SSRF, CSRF, rate-limit, headers, validators
- 'src/local_deep_research/security/**'
- 'src/local_deep_research/settings/env_definitions/security.py'
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
danger-alert:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
checks: write
issues: write # needed for createLabel (auto-create on first run)
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Validate watchlist against repo tree
# Drift-guard: every explicit (non-glob) path below must exist on the
# branch head. Fails the workflow if a watched file has been renamed
# or deleted without updating this list. Glob patterns are excluded
# — they self-heal on rename and would false-positive this check.
# (Tradeoff: if all files matching a glob like `sqlcipher_*.py` are
# deleted, the trigger path silently becomes a no-op. Acceptable —
# disappearing an entire subsystem will be caught in code review.)
#
# ⚠️ KEEP IN SYNC with `on.pull_request.paths` above and the `case`
# patterns in the "Classify critical changes" step below.
run: |
EXPLICIT_WATCHED=(
"src/local_deep_research/database/encrypted_db.py"
"src/local_deep_research/database/credential_store_base.py"
"src/local_deep_research/database/auth_db.py"
"src/local_deep_research/database/session_passwords.py"
"src/local_deep_research/database/temp_auth.py"
"src/local_deep_research/database/session_context.py"
"src/local_deep_research/settings/env_definitions/security.py"
)
missing=0
for p in "${EXPLICIT_WATCHED[@]}"; do
if [ ! -e "$p" ]; then
echo "::error::danger-zone-alert watchlist drift: $p does not exist"
missing=1
fi
done
[ $missing -eq 0 ]
- name: Classify critical changes
id: analyze
env:
BASE_REF: ${{ github.base_ref }}
run: |
# Use env var to prevent template injection from malicious branch names
CHANGED=$(git diff --name-only "origin/$BASE_REF...HEAD")
ENCRYPTION_MODIFIED=false
AUTH_MODIFIED=false
HARDENING_MODIFIED=false
# Shell `case` `*` matches `/` (unlike file globs), so `web/auth/*`
# and `backup/*` cover nested subdirs — equivalent to the `**` in
# GitHub Actions path globs above. Do NOT "fix" these to `**` —
# bash `case` treats `**` as two literal `*`s with no special
# meaning and it would stop matching anything with a slash.
#
# ⚠️ KEEP IN SYNC with `on.pull_request.paths` above and the
# EXPLICIT_WATCHED array in the "Validate watchlist" step above.
for file in $CHANGED; do
case "$file" in
src/local_deep_research/database/encrypted_db.py|\
src/local_deep_research/database/sqlcipher_*.py|\
src/local_deep_research/database/credential_store_base.py|\
src/local_deep_research/database/backup/*)
ENCRYPTION_MODIFIED=true ;;
src/local_deep_research/web/auth/*|\
src/local_deep_research/database/auth_db.py|\
src/local_deep_research/database/session_passwords.py|\
src/local_deep_research/database/temp_auth.py|\
src/local_deep_research/database/session_context.py)
AUTH_MODIFIED=true ;;
src/local_deep_research/security/*|\
src/local_deep_research/settings/env_definitions/security.py)
HARDENING_MODIFIED=true ;;
esac
done
HAS_CRITICAL=false
if [ "$ENCRYPTION_MODIFIED" = "true" ] || [ "$AUTH_MODIFIED" = "true" ] || [ "$HARDENING_MODIFIED" = "true" ]; then
HAS_CRITICAL=true
fi
{
echo "has_critical=$HAS_CRITICAL"
echo "encryption_modified=$ENCRYPTION_MODIFIED"
echo "auth_modified=$AUTH_MODIFIED"
echo "hardening_modified=$HARDENING_MODIFIED"
} >> "$GITHUB_OUTPUT"
- name: Create status check
if: steps.analyze.outputs.has_critical == 'true'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
// Neutral check — advisory, not blocking. Real enforcement, if
// desired, belongs in branch-protection rules, not in a status
// title that misleads reviewers.
try {
await github.rest.checks.create({
owner: context.repo.owner,
repo: context.repo.repo,
name: 'Security Review Required',
head_sha: context.sha,
status: 'completed',
conclusion: 'neutral',
output: {
title: 'Security-sensitive paths modified',
summary: 'This PR modifies critical security code (encryption, authentication, or web hardening). Please review carefully.'
}
});
} catch (err) {
// Fork PRs get a read-only token — checks.create 403s. Log
// and stay green (same pattern as pr-triage.yml); reviewers
// still get the signal from CODEOWNERS on these paths.
if (err.status !== 403) throw err;
console.log('checks.create returned 403 (read-only fork token). Skipping.');
}
- name: Add labels
if: steps.analyze.outputs.has_critical == 'true'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
ENCRYPTION_MODIFIED: ${{ steps.analyze.outputs.encryption_modified }}
AUTH_MODIFIED: ${{ steps.analyze.outputs.auth_modified }}
HARDENING_MODIFIED: ${{ steps.analyze.outputs.hardening_modified }}
with:
script: |
const wanted = [];
if (process.env.ENCRYPTION_MODIFIED === 'true') {
wanted.push({ name: 'touches-encryption', color: 'b60205', description: 'Modifies at-rest crypto, credential storage, or backup encryption' });
}
if (process.env.AUTH_MODIFIED === 'true') {
wanted.push({ name: 'touches-authentication', color: 'd93f0b', description: 'Modifies login, sessions, passwords, auth DB, or tenant isolation' });
}
if (process.env.HARDENING_MODIFIED === 'true') {
wanted.push({ name: 'touches-web-hardening', color: 'fbca04', description: 'Modifies CSP, SSRF, CSRF, rate-limit, headers, validators, or hardening config' });
}
if (wanted.length === 0) return;
// Ensure each label exists before applying (addLabels 422s on unknown labels).
for (const l of wanted) {
try {
await github.rest.issues.createLabel({
owner: context.repo.owner,
repo: context.repo.repo,
name: l.name,
color: l.color,
description: l.description,
});
} catch (e) {
// 422 = already exists. 403 = read-only fork token (the
// addLabels call below degrades the same way).
if (e.status !== 422 && e.status !== 403) throw e;
}
}
try {
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
labels: wanted.map(l => l.name),
});
} catch (err) {
// Fork PRs get a read-only token — the call 403s. Log and
// stay green (same pattern as pr-triage.yml); a maintainer
// can apply the labels manually on fork PRs.
if (err.status !== 403) throw err;
console.log('Label call returned 403 (read-only fork token). Skipping.');
}
+76
View File
@@ -0,0 +1,76 @@
name: Dependency Review
on:
# NOTE: dependency-review requires PR context to compare changes.
# It remains on PRs only and is NOT included in release gate.
# The osv-scanner and npm-audit in release gate cover dependency vulnerabilities.
pull_request:
branches: [main, develop]
paths:
- 'pyproject.toml'
- 'pdm.lock'
- 'requirements*.txt'
- 'package.json'
- 'package-lock.json'
- 'tests/ui_tests/package.json'
- 'tests/ui_tests/package-lock.json'
workflow_dispatch:
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
pull-requests: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout Repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Dependency Review
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
with:
# Fail on high and critical severity vulnerabilities
fail-on-severity: high
# Allow only known permissive + weak-copyleft licenses (ASF Category A + B)
# All licenses below are commercially usable
# LGPL is excluded per ASF Category X policy
allow-licenses: >-
MIT,
Apache-2.0,
BSD-2-Clause,
BSD-3-Clause,
ISC,
MPL-2.0,
Unlicense,
CC0-1.0,
CC-BY-3.0,
CC-BY-4.0,
Python-2.0,
PSF-2.0,
0BSD,
OFL-1.1,
Zlib,
BlueOak-1.0.0
# Allow specific packages with complex license expressions
# dompurify is dual-licensed (Apache-2.0 OR MPL-2.0) but npm reports complex SPDX
allow-dependencies-licenses: >-
pkg:npm/dompurify
# Comment on PR with review summary
comment-summary-in-pr: on-failure
+152
View File
@@ -0,0 +1,152 @@
name: DevSkim Security Linter
on:
workflow_dispatch:
workflow_call: # Called by release-gate.yml
schedule:
# Run security linter daily at 10 AM UTC
- cron: '0 10 * * *'
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
devskim-scan:
name: DevSkim Security Linter
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Run DevSkim security linter
uses: microsoft/DevSkim-Action@4b5047945a44163b94642a1cecc0d93a3f428cc6 # v1.0.16
with:
# Scan all source code
directory-to-scan: '.'
# Output SARIF for GitHub Security tab
output-filename: 'devskim-results.sarif'
# Ignore test fixtures and documentation
# - tests/: All test code uses mock data and localhost URLs
# - examples/: Example code with placeholder values
# - docs/: Documentation with example URLs
# - node_modules/: Third-party code
ignore-globs: 'tests/**,examples/**,docs/**,node_modules/**,**/node_modules/**'
# Exclude rules that produce false positives in this codebase:
#
# DS176209 - "Suspicious comment" (TODO/FIXME/HACK)
# These are standard development annotations, not security issues.
# Every codebase has TODO comments for tracking technical debt.
#
# DS162092 - "Hardcoded URL"
# This research tool integrates with legitimate external APIs
# (ArXiv, PubMed, Semantic Scholar, etc.). All URLs are intentional
# service endpoints, not security vulnerabilities.
#
# DS137138 - "Hardcoded credentials"
# Analysis showed 100% of alerts are test fixtures (api_key="test_key",
# password="testpass", etc.). Zero real credentials found. Gitleaks
# handles actual secret detection separately.
#
# DS148264 - "Use cryptographic random"
# Flags ALL Python `random` module usage without context. This codebase
# correctly uses `secrets` for security (session tokens, auth tokens,
# Flask keys) and `random` only for non-security purposes: ML fairness
# shuffling in search strategies, distributed systems jitter in scheduler,
# and session cleanup probability. None involve cryptographic operations.
#
# DS172411 - "setTimeout code injection"
# Flags ALL JavaScript setTimeout() calls. The dangerous pattern is
# setTimeout("string", delay) which can execute arbitrary code. However,
# 100% of usages in this codebase (80+) pass safe function references:
# setTimeout(() => {...}, delay) or setTimeout(functionName, delay).
# No string-based setTimeout usage exists.
#
# DS126858 - "Weak/Broken Hash Algorithm"
# Flags any occurrence of the literal string "sha1". In this codebase
# the only matches are the JSON key name `"sha1"` inside SLSA provenance
# attestations (.github/workflows/prerelease-docker.yml) — that key name
# is *required* by the SLSA in-toto schema for git-commit digests, and
# Git itself uses SHA-1 for commit identifiers. It is not a cryptographic
# choice we make. Legitimate SHA-1 references elsewhere (PBKDF2_HMAC_SHA1,
# HMAC_SHA1 in db_config.py for SQLCipher backwards compatibility) carry
# their own inline `# DevSkim: ignore DS126858` annotations explaining the
# rationale. See .github/SECURITY_ALERTS.md for the full assessment.
#
exclude-rules: 'DS176209,DS162092,DS137138,DS148264,DS172411,DS126858'
- name: Upload DevSkim results to GitHub Security
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always()
with:
sarif_file: 'devskim-results.sarif'
category: devskim
- name: Upload DevSkim results as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: devskim-results
path: devskim-results.sarif
retention-days: 7 # Reduced for security
- name: Generate summary
run: |
{
echo "## DevSkim Security Linter Summary"
echo ""
echo "✅ DevSkim security scan completed"
echo ""
echo "### What was scanned:"
echo "- 🔍 All source code in the repository"
echo "- 🛡️ Common security issues and anti-patterns"
echo "- 📋 Insecure coding practices"
echo "- 🔑 Hardcoded secrets and credentials"
echo ""
echo "### Security checks include:"
echo "- **Hardcoded Credentials**: Passwords, API keys, tokens"
echo "- **Insecure Functions**: Dangerous API usage"
echo "- **Buffer Overflows**: Memory safety issues"
echo "- **Crypto Issues**: Weak encryption practices"
echo "- **Network Security**: Insecure protocols and configurations"
echo "- **File System**: Insecure file operations"
echo "- **Input Validation**: Missing input sanitization"
echo ""
echo "### Language Support:"
echo "- 🐍 Python (.py files)"
echo "- 📜 JavaScript/TypeScript (.js, .ts files)"
echo "- 🖥️ C/C++ (.c, .cpp, .h files)"
echo "- 🔷 C# (.cs files)"
echo "- 📝 More languages supported"
echo ""
echo "### Why DevSkim?"
echo "- **Microsoft Maintained**: Backed by Microsoft security team"
echo "- **Truly Free**: No registration, no API keys, no upsells"
echo "- **Fast & Lightweight**: Quick scans with minimal overhead"
echo "- **Comprehensive Rules**: 200+ security rules"
echo "- **Open Source**: Community-driven rule improvements"
echo ""
echo "📊 **Results uploaded to:**"
echo "- GitHub Security tab (if issues found)"
echo "- Workflow artifacts (SARIF files)"
echo ""
echo "🔗 **Learn more about DevSkim:**"
echo "- [DevSkim GitHub](https://github.com/microsoft/DevSkim)"
echo "- [DevSkim Documentation](https://microsoft.github.io/DevSkim/)"
echo "- [DevSkim Rules](https://github.com/microsoft/DevSkim/tree/main/rules)"
} >> "$GITHUB_STEP_SUMMARY"
+156
View File
@@ -0,0 +1,156 @@
name: Test Multi-Architecture Docker Build
on:
workflow_call:
workflow_dispatch:
permissions:
contents: read
jobs:
test-multiarch-build:
name: Test Multi-Arch Docker Build
strategy:
matrix:
include:
- platform: linux/amd64
runs-on: ubuntu-latest
platform_slug: linux-amd64
- platform: linux/arm64
runs-on: ubuntu-24.04-arm
platform_slug: linux-arm64
runs-on: ${{ matrix.runs-on }}
timeout-minutes: 60
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Check out the repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Build Docker image for ${{ matrix.platform }}
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
platforms: ${{ matrix.platform }}
push: false
load: false
cache-from: type=gha,scope=${{ matrix.platform_slug }}
cache-to: type=gha,mode=max,scope=${{ matrix.platform_slug }}
tags: ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
outputs: type=docker,dest=/tmp/image-${{ matrix.platform_slug }}.tar
- name: Inspect built image
run: |
# Load and inspect the image
docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
docker image inspect ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
# Get image size
IMAGE_SIZE=$(docker image inspect ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} --format='{{.Size}}' | numfmt --to=iec)
echo "Image size for ${{ matrix.platform }}: $IMAGE_SIZE"
- name: Test image startup
run: |
docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
# Test that the container can start and Python works
docker run --rm --platform ${{ matrix.platform }} \
ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} \
python -c "import sys; print(f'Python {sys.version} on {sys.platform}')"
# Test that the LDR package is installed
docker run --rm --platform ${{ matrix.platform }} \
ldr-test:${{ matrix.platform_slug }}-${{ github.sha }} \
python -c "import local_deep_research; print('LDR package imported successfully')"
- name: Test web server startup
run: |
docker load < "/tmp/image-${{ matrix.platform_slug }}.tar"
# Start the container in background
docker run -d --name ldr-test \
--platform ${{ matrix.platform }} \
-p 5000:5000 \
-e LDR_DB_CONFIG_KDF_ITERATIONS=1000 \
-e LDR_TEST_MODE=1 \
-e LDR_NEWS_SCHEDULER_ENABLED=false \
ldr-test:${{ matrix.platform_slug }}-${{ github.sha }}
# Wait for server to start
TIMEOUT_SECONDS=60
echo "Waiting for server to start (timeout: ${TIMEOUT_SECONDS}s)..."
for i in $(seq 1 ${TIMEOUT_SECONDS}); do
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
echo "✅ Server started successfully after ${i} seconds on ${{ matrix.platform }}!"
break
fi
# Show progress every 10 seconds
if [ $((i % 10)) -eq 0 ]; then
echo "Still waiting... (${i}s elapsed)"
fi
if [ "$i" -eq "${TIMEOUT_SECONDS}" ]; then
echo "❌ Server failed to start within ${TIMEOUT_SECONDS} seconds"
docker logs ldr-test
exit 1
fi
sleep 1
done
# Check server response
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:5000)
echo "Server response code: $RESPONSE"
# Clean up
docker stop ldr-test
docker rm ldr-test
summary:
name: Build Summary
needs: test-multiarch-build
runs-on: ubuntu-latest
timeout-minutes: 5
if: always()
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Summary
env:
BUILD_RESULT: ${{ needs.test-multiarch-build.result }}
run: |
{
echo "## Multi-Architecture Build Test Results"
echo ""
} >> "$GITHUB_STEP_SUMMARY"
if [ "$BUILD_RESULT" == "success" ]; then
{
echo "✅ **All platforms built and tested successfully!**"
echo ""
echo "### Tested Platforms:"
echo "- linux/amd64 ✅"
echo "- linux/arm64 ✅"
echo ""
echo "The Docker image is ready for multi-architecture deployment."
} >> "$GITHUB_STEP_SUMMARY"
else
{
echo "❌ **Build or tests failed**"
echo ""
echo "Please check the logs above for details."
} >> "$GITHUB_STEP_SUMMARY"
fi
+334
View File
@@ -0,0 +1,334 @@
name: Publish Docker image
# SECURITY: Only invoked as a reusable workflow (`workflow_call`) from
# release.yml after the release security gate AND the `release` environment
# approval pass. We intentionally do NOT support workflow_dispatch — that
# would bypass gates. We also intentionally do NOT support
# repository_dispatch any more: with the atomicity refactor, this workflow
# runs as a job inside release.yml's run so its result is visible to
# downstream jobs (create-release, cleanup-on-rejection) — a property that
# repository_dispatch fanout broke.
#
# This workflow is a thin RETAG step in the build-once-promote pipeline.
# The actual build happens in prerelease-docker.yml; the multi-arch manifest
# is signed and attested there. Here we only:
# - Verify the source manifest digest matches what prerelease produced
# - Retag the prerelease manifest to release tags (:1.6.9, :1.6, :latest)
# - Verify the digest is preserved (defends against imagetools re-encoding)
# - Re-run Trivy against the digest (catches CVE-database updates between
# prerelease build and release promote)
# - Verify cosign signature transitivity from the original digest
# - Clean up the prerelease tags
#
# To re-publish, trigger a new release through release.yml.
on:
workflow_call:
inputs:
tag:
description: "Release tag, e.g. 'v1.6.9' (with leading 'v')"
type: string
required: true
source_tag:
description: "Prerelease manifest tag to retag, e.g. 'prerelease-v1.6.9-abc1234'"
type: string
required: true
expected_digest:
description: "sha256:... digest of the prerelease manifest, captured by prerelease-docker.yml. Used to verify retag preserves the digest end-to-end."
type: string
required: true
secrets:
DOCKER_USERNAME:
required: true
description: "Docker Hub username (env-scoped to `release`)"
DOCKER_PASSWORD:
required: true
description: "Docker Hub PAT with Read+Write+Delete scopes (env-scoped to `release`)"
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
# NOTE: no workflow-level `concurrency:` block. As a reusable workflow
# called from release.yml, this workflow runs as part of the caller's run,
# and release.yml's caller-level concurrency (keyed on github.workflow +
# github.ref) already serialises release runs for the same tag. Adding a
# callee-level block would be a no-op for the documented threat model
# (two simultaneous releases for the same ref) and would not be reachable
# anyway because there is no longer any standalone invocation path.
jobs:
promote:
name: Retag prerelease manifest as release
runs-on: ubuntu-latest
environment: release
permissions:
# cosign verify is read-only against public Rekor/Fulcio — does not
# mint a GitHub OIDC token, so id-token: write is not required here.
# Signing (which does need id-token: write) happens in prerelease-docker.yml.
contents: read
steps:
- name: Harden Runner
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Check out the repo at the triggering commit
# Pin the checkout to the EXACT commit that the prerelease was
# built/scanned from, so .trivyignore (and any other repo-state-
# dependent file the promote step reads) matches that commit.
# We use github.sha, NOT inputs.tag — the v* git tag is created
# by create-release LATER in this run (after publish-docker
# completes), so it does not exist yet when this checkout runs
# on a push-to-main trigger. github.sha is the triggering commit
# for every event type (push to main, tag push, workflow_dispatch)
# and is the same SHA the build/prerelease-docker jobs used.
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.sha }}
persist-credentials: false
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Log in to Docker Hub
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Install Cosign
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
with:
# Pin to cosign v2.x to match the version that signed the artifact
# in prerelease-docker.yml. Mismatched versions across sign/verify
# work today but new-bundle-format (cosign v3 default) would only
# produce/consume on v3.
cosign-release: 'v2.6.3'
- name: Determine release tags
id: version
env:
DISPATCH_TAG: ${{ inputs.tag }}
SOURCE_TAG: ${{ inputs.source_tag }}
EXPECTED_DIGEST: ${{ inputs.expected_digest }}
run: |
set -euo pipefail
if [[ -z "$DISPATCH_TAG" || -z "$SOURCE_TAG" || -z "$EXPECTED_DIGEST" ]]; then
echo "::error::Missing required workflow_call input. Got tag='${DISPATCH_TAG}' source_tag='${SOURCE_TAG}' expected_digest='${EXPECTED_DIGEST}'"
exit 1
fi
if [[ "$EXPECTED_DIGEST" != sha256:* ]]; then
echo "::error::expected_digest must be of the form sha256:... — got '${EXPECTED_DIGEST}'"
exit 1
fi
VERSION="${DISPATCH_TAG#v}"
MAJOR_MINOR=$(echo "$VERSION" | cut -d. -f1,2)
# Group writes into one redirect (shellcheck SC2129).
{
echo "tag=${DISPATCH_TAG}"
echo "version=${VERSION}"
echo "major_minor=${MAJOR_MINOR}"
echo "source_tag=${SOURCE_TAG}"
echo "expected_digest=${EXPECTED_DIGEST}"
} >> "$GITHUB_OUTPUT"
- name: Verify source digest matches expected
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
SOURCE_TAG: ${{ steps.version.outputs.source_tag }}
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
run: |
set -euo pipefail
# Defends against the prerelease tag being swapped between
# prerelease-docker's signing and this promote step.
SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"
ACTUAL=$(docker buildx imagetools inspect "$SOURCE" --format '{{json .Manifest.Digest}}' | tr -d '"')
if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then
echo "::error::Source digest mismatch — possible tag tampering between prerelease and promote"
echo " expected: $EXPECTED_DIGEST"
echo " actual: $ACTUAL"
exit 1
fi
echo "Source digest verified: $ACTUAL"
- name: Promote (retag) prerelease manifest to release tags
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
SOURCE_TAG: ${{ steps.version.outputs.source_tag }}
VERSION: ${{ steps.version.outputs.version }}
MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
run: |
set -euo pipefail
SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"
# Single imagetools create with multiple -t — registry-side
# metadata-only operation, takes seconds, preserves digest.
docker buildx imagetools create \
-t "${DOCKER_USERNAME}/local-deep-research:${VERSION}" \
-t "${DOCKER_USERNAME}/local-deep-research:${MAJOR_MINOR}" \
-t "${DOCKER_USERNAME}/local-deep-research:latest" \
"$SOURCE"
echo "Promoted ${SOURCE} to :${VERSION}, :${MAJOR_MINOR}, :latest"
- name: Verify promoted tags share the source digest
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
VERSION: ${{ steps.version.outputs.version }}
MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
run: |
set -euo pipefail
# Defends against `imagetools create` re-encoding the manifest.
# If digests diverge, signatures and attestations (keyed by the
# original digest) won't be discoverable from the new tags.
for TAG in "${VERSION}" "${MAJOR_MINOR}" "latest"; do
REF="${DOCKER_USERNAME}/local-deep-research:${TAG}"
ACTUAL=$(docker buildx imagetools inspect "$REF" --format '{{json .Manifest.Digest}}' | tr -d '"')
if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then
echo "::error::Digest mismatch on ${TAG} — imagetools create may have re-encoded the manifest"
echo " expected: $EXPECTED_DIGEST"
echo " actual: $ACTUAL"
exit 1
fi
echo "${TAG} -> ${ACTUAL} ✓"
done
# Catches CVE-database updates that landed between the prerelease
# build and this promote step. Use the SHA-pinned action wrapper
# (same pin as prerelease-docker.yml's security-scan) with an
# explicit binary version pin — the prior `apt-get install -y trivy`
# approach was unpinned and exposed the release path to the Trivy
# apt-repo supply chain. The pinned action downloads the v0.69.2
# binary from GitHub releases by exact tag, which is the same
# binary the prerelease scan validated, keeping the two scans
# consistent.
#
# Unlike prerelease-docker.yml's security-scan (which scans a
# locally-loaded image, no registry pull), this step scans by
# registry digest — Trivy must pull the manifest + layers from
# Docker Hub. TRIVY_USERNAME/TRIVY_PASSWORD is the action's
# documented auth path; the `docker/login-action` above also
# writes ~/.docker/config.json which Trivy reads as a fallback,
# but the explicit env vars are more reliable (Trivy has
# documented docker.io credential-helper quirks — aquasecurity/
# trivy#432, aquasecurity/trivy#8385) and the image we scan IS on
# Docker Hub so this is the path most likely to keep working.
- name: Re-scan release digest with Trivy
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
env:
TRIVY_USERNAME: ${{ secrets.DOCKER_USERNAME }}
TRIVY_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
with:
image-ref: ${{ secrets.DOCKER_USERNAME }}/local-deep-research@${{ steps.version.outputs.expected_digest }}
severity: 'CRITICAL,HIGH'
ignore-unfixed: true
trivyignores: '.trivyignore'
exit-code: '1'
version: 'v0.69.2'
- name: Verify cosign signature on promoted digest
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
# The cert was issued to the prerelease-docker.yml workflow when
# signing happened there, so the identity regex must match that
# workflow's path. Fulcio's SAN is built from job_workflow_ref,
# which for reusable workflows is the CALLEE.
#
# Verify by IMMUTABLE digest (not the :VERSION tag) so this step
# is invariant under any retag race between the verify-promoted-
# tags step above and this one. Trivy's re-scan above also uses
# @${EXPECTED_DIGEST}; keeping cosign on the same reference is
# consistent and avoids a tag-resolution TOCTOU window.
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${EXPECTED_DIGEST}"
echo "Verifying signature for: $IMAGE_REF"
cosign verify \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
--certificate-identity-regexp "^https://github.com/${REPO}/\.github/workflows/prerelease-docker\.yml@refs/(heads|tags)/" \
--certificate-github-workflow-repository "${REPO}" \
"$IMAGE_REF"
echo "Signature transitivity verified ✓"
- name: Clean up prerelease tags
continue-on-error: true
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
RELEASE_VERSION: ${{ steps.version.outputs.version }}
run: |
set +e # Best-effort: never fail the workflow
# Scope deletion to prereleases of THIS version only, so concurrent
# prereleases for other versions (and any unrelated prerelease-* tags)
# are left untouched.
PREFIX="prerelease-v${RELEASE_VERSION}-"
echo "Cleaning up ${PREFIX}* tags from Docker Hub..."
# Authenticate with Docker Hub API (password-based JWT)
TOKEN=$(curl -s -X POST "https://hub.docker.com/v2/users/login/" \
-H "Content-Type: application/json" \
-d "{\"username\": \"${DOCKER_USERNAME}\", \"password\": \"${DOCKER_PASSWORD}\"}" | jq -r '.token')
if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
echo "WARNING: Failed to authenticate with Docker Hub API. Skipping cleanup."
exit 0
fi
REPO="${DOCKER_USERNAME}/local-deep-research"
PAGE=1
DELETED=0
while true; do
RESPONSE=$(curl -s -H "Authorization: JWT ${TOKEN}" \
"https://hub.docker.com/v2/repositories/${REPO}/tags/?page=${PAGE}&page_size=100")
RESULTS=$(echo "$RESPONSE" | jq -r '.results[]?.name // empty')
if [ -z "$RESULTS" ]; then
break
fi
while IFS= read -r tag; do
if [[ "$tag" == "${PREFIX}"* ]]; then
echo "Deleting tag: ${tag}"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" -X DELETE \
-H "Authorization: JWT ${TOKEN}" \
"https://hub.docker.com/v2/repositories/${REPO}/tags/${tag}/")
if [ "$HTTP_CODE" -eq 204 ] || [ "$HTTP_CODE" -eq 200 ]; then
echo " Deleted: ${tag}"
DELETED=$((DELETED + 1))
else
echo " WARNING: Failed to delete ${tag} (HTTP ${HTTP_CODE})"
fi
fi
done <<< "$RESULTS"
# Check if there are more pages
NEXT=$(echo "$RESPONSE" | jq -r '.next // empty')
if [ -z "$NEXT" ]; then
break
fi
PAGE=$((PAGE + 1))
done
echo "Prerelease tag cleanup complete. Deleted ${DELETED} tag(s) matching ${PREFIX}*."
- name: Summary
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
VERSION: ${{ steps.version.outputs.version }}
MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}
EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}
run: |
{
echo "## Docker Release Promoted"
echo ""
echo "**Digest:** \`${EXPECTED_DIGEST}\`"
echo ""
echo "**Tags:** \`${VERSION}\`, \`${MAJOR_MINOR}\`, \`latest\` — all share the same digest as the prerelease manifest, so cosign signatures, SBOM, and SLSA provenance from the prerelease step are transitively valid."
echo ""
echo '```'
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${VERSION}"
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:latest"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
File diff suppressed because it is too large Load Diff
+211
View File
@@ -0,0 +1,211 @@
name: Dockle Container Security Linting
on:
workflow_dispatch:
workflow_call: # Called by release-gate.yml
schedule:
# Run weekly on Tuesday at 10 AM UTC (staggered with other container scans)
- cron: '0 10 * * 2'
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
dockle:
name: Dockle Container Image Security
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Build Docker image for scanning
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
push: false
load: true
tags: local-deep-research:dockle-scan
cache-from: type=gha,scope=dockle-scan
cache-to: type=gha,mode=max,scope=dockle-scan
- name: Run Dockle security scan
id: dockle
run: |
echo "=== Running Dockle Container Image Security Scan ==="
# Run Dockle with SARIF output for GitHub Security tab
# Mount current directory to /output so SARIF file is written to host filesystem
# Exit code 1 means issues found, but we don't fail the workflow
#
# Ignored checks (false positives or intentional design):
# - CIS-DI-0001: Create a user - image uses setpriv entrypoint pattern for privilege drop
# (see scripts/ldr_entrypoint.sh which uses setpriv to switch to ldruser)
# - CIS-DI-0005: Enable Content trust for Docker - runtime env var (DOCKER_CONTENT_TRUST),
# not a Dockerfile concern; enforced at deployment time
# - CIS-DI-0008: setuid/setgid files - unix_chkpwd is standard Debian base utility
# required for PAM authentication, inherited from python:3.14-slim base image
# - DKL-DI-0005: Clear apt-get caches - already done in Dockerfile (rm -rf /var/lib/apt/lists/*),
# but base image (python:3.14-slim) layers trigger this check
#
# Accepted files (false positives):
# - unix_chkpwd: PAM utility from Debian base image
# - settings.py: Any file named settings.py (config files, not credentials)
# Without --exit-code flag, Dockle always exits 0 regardless of findings.
# Findings are reported via SARIF and enforced by the release gate
# (check-code-scanning-alerts job in release-gate.yml).
# Any non-zero exit code here means Docker itself failed (image pull error,
# socket unavailable, etc.) — a broken scanner that MUST fail the step,
# otherwise no SARIF is produced and the release gate has nothing to check.
set +e
docker run --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v "$(pwd):/output" \
goodwithtech/dockle:v0.4.14 \
--format sarif \
--output /output/dockle-results.sarif \
--ignore CIS-DI-0001 \
--ignore CIS-DI-0005 \
--ignore CIS-DI-0008 \
--ignore DKL-DI-0005 \
--accept-file "usr/sbin/unix_chkpwd" \
--accept-file "settings.py" \
local-deep-research:dockle-scan
DOCKLE_EXIT_CODE=$?
set -e
if [ "$DOCKLE_EXIT_CODE" -ne 0 ]; then
echo "::error::Dockle/Docker failed with exit code $DOCKLE_EXIT_CODE"
exit 1
fi
# Check if SARIF file was created (for use in subsequent steps)
if [ -f "dockle-results.sarif" ]; then
echo "sarif_exists=true" >> "$GITHUB_OUTPUT"
# Check if SARIF contains any findings (used by summary step)
RESULT_COUNT=$(python3 -c "import json; print(len(json.load(open('dockle-results.sarif')).get('runs',[{}])[0].get('results',[])))" 2>/dev/null || echo "0")
if [ "$RESULT_COUNT" -gt 0 ]; then
echo "DOCKLE_FOUND_ISSUES=true" >> "$GITHUB_ENV"
fi
fi
- name: Run Dockle with human-readable output
if: always()
run: |
echo "=== Dockle Scan Results ==="
# Use same ignore flags as SARIF scan for consistent output
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
goodwithtech/dockle:v0.4.14 \
--format list \
--ignore CIS-DI-0001 \
--ignore CIS-DI-0005 \
--ignore CIS-DI-0008 \
--ignore DKL-DI-0005 \
--accept-file "usr/sbin/unix_chkpwd" \
--accept-file "settings.py" \
local-deep-research:dockle-scan || true
- name: Upload Dockle results to GitHub Security tab
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
# Note: hashFiles() is evaluated at workflow parse time, not runtime.
# Use step output instead to check if SARIF file was created during the run.
if: always() && steps.dockle.outputs.sarif_exists == 'true'
with:
sarif_file: dockle-results.sarif
category: dockle
- name: Upload Dockle results as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always() && steps.dockle.outputs.sarif_exists == 'true'
with:
name: dockle-results
path: dockle-results.sarif
retention-days: 7 # Reduced for security
- name: Display Dockle summary
if: always()
run: |
{
echo "## 🐋 Dockle Container Image Security Summary"
echo ""
echo "### What is Dockle?"
echo "Dockle is a container image linting tool that checks for:"
echo ""
echo "#### 🎯 Security Checks:"
echo "- **Sensitive Files**: Keys, certificates, passwords in image"
echo "- **Root User**: Warning if running as root"
echo "- **Clear-text Passwords**: Detects plaintext secrets"
echo "- **CIS Benchmarks**: Container security best practices"
echo "- **Outdated Packages**: Vulnerable base images"
echo ""
echo "#### 📊 Dockle vs Hadolint:"
echo "- **Hadolint**: Checks Dockerfile source code (best practices)"
echo "- **Dockle**: Scans built container image (runtime security)"
echo "- **Both Complement**: Check different stages of container lifecycle"
echo ""
echo "#### 🔍 Scan Results:"
} >> "$GITHUB_STEP_SUMMARY"
if [ "$DOCKLE_FOUND_ISSUES" = "true" ]; then
{
echo "⚠️ **Issues Found**"
echo ""
echo "Dockle detected security or best practice issues:"
echo ""
echo "**Common Issues:**"
echo "- Running as root user (consider USER directive)"
echo "- Sensitive files potentially included in image"
echo "- Missing security labels or metadata"
echo "- Outdated base image packages"
echo ""
echo "📋 **Detailed Results:**"
echo "- GitHub Security tab: SARIF results uploaded"
echo "- Artifacts: Full SARIF report available"
echo "- Check output above for human-readable format"
} >> "$GITHUB_STEP_SUMMARY"
else
{
echo "✅ **No Issues Found!**"
echo ""
echo "Container image passed all Dockle security checks."
} >> "$GITHUB_STEP_SUMMARY"
fi
{
echo ""
echo "#### 🔗 Resources:"
echo "- [Dockle GitHub](https://github.com/goodwithtech/dockle)"
echo "- [CIS Docker Benchmark](https://www.cisecurity.org/benchmark/docker)"
echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
echo ""
echo "#### 💡 Best Practices:"
echo "- Use multi-stage builds to reduce image size"
echo "- Add USER directive to run as non-root"
echo "- Don't include secrets or keys in image"
echo "- Keep base images updated"
echo "- Scan images with both Dockle (runtime) and Hadolint (build)"
echo ""
echo "#### 🔇 Ignored Checks (false positives or intentional design):"
echo "- **CIS-DI-0001** (Create a user): Uses setpriv entrypoint pattern for privilege drop"
echo "- **CIS-DI-0005** (Content trust): Runtime env var, not Dockerfile concern"
echo "- **CIS-DI-0008** (setuid/setgid): unix_chkpwd from Debian base, required for PAM"
echo "- **DKL-DI-0005** (apt-get cache): Dockerfile clears caches, base image triggers this"
echo "- **settings.py**: Any file named settings.py accepted (config files, not credentials)"
} >> "$GITHUB_STEP_SUMMARY"
- name: Clean up Docker image
if: always()
run: |
docker rmi local-deep-research:dockle-scan || true
+201
View File
@@ -0,0 +1,201 @@
name: E2E Research Test
on:
pull_request:
types: [labeled]
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions: {} # Minimal top-level for OSSF Scorecard
jobs:
build-query:
name: Assemble query
if: github.event.label.name == 'ldr_research' || github.event.label.name == 'ldr_research_static'
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
query: ${{ steps.assemble.outputs.query }}
header: ${{ steps.headers.outputs.header }}
subheader: ${{ steps.headers.outputs.subheader }}
env:
LABEL_NAME: ${{ github.event.label.name }}
MAX_DIFF_SIZE: ${{ vars.MAX_DIFF_SIZE || '8000' }}
BASE_REF: ${{ github.event.pull_request.base.ref }}
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Build PR-research query
id: assemble
run: |
if [ "$LABEL_NAME" = "ldr_research_static" ]; then
cat > query.txt <<'STATIC_EOF'
What is Local Deep Research and how does it work?
STATIC_EOF
else
git fetch origin "$BASE_REF"
git diff "origin/$BASE_REF...HEAD" --no-color > diff.txt
DIFF_BYTES=$(wc -c < diff.txt)
echo "Diff size: $DIFF_BYTES bytes (cap: $MAX_DIFF_SIZE)"
head -c "$MAX_DIFF_SIZE" diff.txt > diff-trunc.txt
if [ "$DIFF_BYTES" -gt "$MAX_DIFF_SIZE" ]; then
printf '\n... (truncated)\n' >> diff-trunc.txt
fi
{
cat <<'PROMPT_HEAD_EOF'
Based on these code changes, research relevant documentation,
best practices, and potential issues. Focus on any libraries, APIs, or patterns used.
Code changes:
PROMPT_HEAD_EOF
cat diff-trunc.txt
cat <<'PROMPT_TAIL_EOF'
Research topics to cover:
1. Documentation for any libraries or APIs being used/modified
2. Best practices for the patterns shown
3. Known issues or gotchas related to these changes
4. Security considerations if applicable
PROMPT_TAIL_EOF
} > query.txt
fi
# Multi-line GH Actions output via heredoc-style delimiter.
# Use a randomized delimiter so a query containing the literal
# delimiter on its own line cannot prematurely terminate the
# heredoc.
DELIM="LDR_QUERY_EOF_$$_${RANDOM}_$(date +%N)"
{
echo "query<<$DELIM"
cat query.txt
echo "$DELIM"
} >> "$GITHUB_OUTPUT"
- name: Compute headers
id: headers
run: |
if [ "$LABEL_NAME" = "ldr_research_static" ]; then
echo "header=## 🧪 LDR Static Query Test Results" >> "$GITHUB_OUTPUT"
echo "subheader=**Query:** What is Local Deep Research and how does it work?" >> "$GITHUB_OUTPUT"
else
echo "header=## 🔬 LDR Research Results" >> "$GITHUB_OUTPUT"
echo "subheader=_Analysis of PR code changes_" >> "$GITHUB_OUTPUT"
fi
research:
name: Run LDR research
needs: build-query
# Grant the perms the reusable's job needs:
# - contents: read for actions/checkout
# - actions: write for actions/upload-artifact@v5+
# Without this, the reusable's `permissions: contents: read` exceeds
# the inherited empty permissions and GitHub rejects the workflow at
# load time (startup_failure with zero jobs).
permissions:
contents: read
actions: write
uses: ./.github/workflows/ldr-research-reusable.yml
with:
query: ${{ needs.build-query.outputs.query }}
model: ${{ github.event.label.name == 'ldr_research_static' && (vars.LDR_RESEARCH_CHEAP_MODEL || 'google/gemini-2.0-flash-001') || (vars.LDR_RESEARCH_MODEL || 'google/gemini-2.0-flash-001') }}
provider: ${{ vars.LDR_PROVIDER || 'openrouter' }}
search-tool: ${{ vars.LDR_SEARCH_TOOL || 'serper' }}
strategy: ${{ vars.LDR_STRATEGY || 'langgraph-agent' }}
comment-header: ${{ needs.build-query.outputs.header }}
comment-subheader: ${{ needs.build-query.outputs.subheader }}
secrets:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
post-comment:
name: Post PR comment
needs: research
# always() so the label-removal step runs even if research was skipped
# (e.g. build-query failed). The download/post steps are guarded by
# needs.research.outputs.success == 'true' so they self-skip in that
# case.
if: always()
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
issues: write
env:
LABEL_NAME: ${{ github.event.label.name }}
PR_NUMBER: ${{ github.event.pull_request.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Download research artifact
if: needs.research.outputs.success == 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: ${{ needs.research.outputs.comment-artifact-name }}
- name: Post or update comment
if: needs.research.outputs.success == 'true'
continue-on-error: true # a GitHub API hiccup shouldn't fail the job
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
retries: 3
script: |
const fs = require('fs');
const issue_number = Number(process.env.PR_NUMBER);
// Key the marker by label so re-running the same label edits its
// comment in place, while the other label keeps a separate comment.
const marker = `<!-- ldr-research-results:${process.env.LABEL_NAME} -->`;
const body = marker + '\n' + fs.readFileSync('comment.md', 'utf8');
// Paginate — on a long-lived PR the existing comment may be past the
// first page (listComments defaults to 30), which would otherwise
// create a new comment every run instead of editing in place.
const comments = await github.paginate(github.rest.issues.listComments, {
owner: context.repo.owner,
repo: context.repo.repo,
issue_number,
per_page: 100,
});
const existing = comments.find(
(c) => c.user.type === 'Bot' && c.body.includes(marker)
);
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number,
body,
});
}
- name: Remove label for re-triggering
if: always()
run: |
gh issue edit "$PR_NUMBER" --remove-label "$LABEL_NAME" 2>/dev/null || true
@@ -0,0 +1,47 @@
name: File Whitelist Security Check
# Enhanced security checks with comprehensive file type detection
on:
pull_request:
branches: [ main ]
workflow_call: # Called by ci-gate.yml for release pipeline
inputs:
check-all-files:
description: 'Check ALL tracked files (not just changed files)'
required: false
type: boolean
default: true
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions:
contents: read
jobs:
whitelist-check:
runs-on: ubuntu-latest
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Security checks for files
env:
GITHUB_EVENT_NAME: ${{ github.event_name }}
GITHUB_BASE_REF: ${{ github.base_ref }}
CHECK_ALL_FILES: ${{ inputs.check-all-files }}
run: |
.github/scripts/file-whitelist-check.sh
+82
View File
@@ -0,0 +1,82 @@
name: Fuzzing Tests
on:
schedule:
- cron: '0 0 * * 0' # Weekly on Sunday at midnight
workflow_dispatch: # Allow manual triggering
pull_request:
paths:
- 'tests/fuzz/**'
- 'src/local_deep_research/security/**'
- 'src/local_deep_research/utilities/**'
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions: {}
jobs:
fuzz-tests:
name: Hypothesis Fuzz Tests
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Cache PDM dependencies
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/.cache/pdm
.venv
key: pdm-${{ runner.os }}-${{ hashFiles('pdm.lock') }}
restore-keys: |
pdm-${{ runner.os }}-
- name: Install dependencies
run: |
python -m pip install --upgrade pip==25.0
pip install pdm==2.26.2
# Retry pdm install up to 3 times with backoff
for i in 1 2 3; do
pdm install --dev --no-editable && break
echo "Attempt $i failed, retrying in $((i * 10)) seconds..."
sleep $((i * 10))
done || exit 1
- name: Run fuzz tests
run: |
pdm run pytest tests/fuzz/ -v --tb=short \
--hypothesis-show-statistics \
--hypothesis-seed=0
env:
HYPOTHESIS_PROFILE: ci
- name: Run extended fuzz tests (scheduled only)
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
run: |
# Run with more examples for scheduled/manual runs
pdm run pytest tests/fuzz/ -v --tb=short \
--hypothesis-show-statistics \
-x # Stop on first failure for investigation
env:
HYPOTHESIS_PROFILE: extended
+55
View File
@@ -0,0 +1,55 @@
name: Gitleaks Main Branch Scan
# This workflow is specifically for the security release gate.
# It only scans the main branch history to avoid false positives
# from feature branches with intentional test data.
on:
workflow_call: # Called by release-gate.yml
workflow_dispatch:
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
gitleaks-scan:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
ref: main # Always checkout main branch
- name: Install Gitleaks
run: |
curl -sSfL https://github.com/gitleaks/gitleaks/releases/download/v8.21.2/gitleaks_8.21.2_linux_x64.tar.gz | tar -xz
sudo mv gitleaks /usr/local/bin/
- name: Run Gitleaks on main branch only
run: |
gitleaks detect \
--config=.gitleaks.toml \
--gitleaks-ignore-path=.gitleaksignore \
--log-opts="origin/main" \
--report-format=sarif \
--report-path=gitleaks-results.sarif \
--verbose
id: gitleaks
- name: Upload SARIF report
if: always()
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
with:
sarif_file: gitleaks-results.sarif
+46
View File
@@ -0,0 +1,46 @@
name: Gitleaks Secret Detection
on:
pull_request:
branches: [ main ]
workflow_dispatch:
schedule:
# Run secret scan daily at 3 AM UTC
- cron: '0 3 * * *'
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
gitleaks:
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0 # Fetch full history for comprehensive secret scanning
- name: Run Gitleaks Secret Scanner
uses: gitleaks/gitleaks-action@e0c47f4f8be36e29cdc102c57e68cb5cbf0e8d1e # v3.0.0
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITLEAKS_CONFIG: .gitleaks.toml
GITLEAKS_BASELINE_PATH: .gitleaksignore
+158
View File
@@ -0,0 +1,158 @@
name: Grype Vulnerability Scan
# Second dependency/container vulnerability scanner (complements Trivy).
# Grype uses Anchore's curated vulnerability DB (NVD, GitHub Advisories,
# Alpine SecDB, etc.) — a different data source from Trivy's own aggregation.
# Running both is standard practice: they catch different CVEs.
on:
workflow_call: # Called by release-gate.yml
workflow_dispatch:
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
grype-filesystem:
name: Grype Filesystem Scan
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Run Grype filesystem scan
uses: anchore/scan-action@e1165082ffb1fe366ebaf02d8526e7c4989ea9d2 # v7.4.0
id: grype-fs
with:
path: '.'
# DO NOT change to fail-build: true — findings are enforced via SARIF alerts
# in the release gate (check-code-scanning-alerts job in release-gate.yml).
# Failing here would break CI without adding security value.
fail-build: false
output-format: sarif
severity-cutoff: medium
# Fail loudly if Grype produced no SARIF — never fabricate an empty one.
# An empty-results SARIF uploaded under the grype-filesystem category would
# make GitHub mark every previously-open Grype alert as fixed, silently
# clearing real findings. Mirror the Trivy job: error + exit 1, skip upload.
- name: Ensure SARIF file exists
id: check-fs-sarif
if: always()
run: |
SARIF_FILE="${{ steps.grype-fs.outputs.sarif }}"
if [ -f "$SARIF_FILE" ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "::error::Grype filesystem scan did not produce a SARIF file — scan needs to be rerun"
exit 1
fi
- name: Upload Grype filesystem SARIF to GitHub Security tab
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always() && steps.check-fs-sarif.outputs.exists == 'true'
with:
sarif_file: ${{ steps.grype-fs.outputs.sarif }}
category: grype-filesystem
grype-container:
name: Grype Container Scan
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
sudo docker image prune -af
df -h
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Build Docker image for scanning
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
push: false
load: true
tags: local-deep-research:grype-scan
cache-from: type=gha,scope=grype-scan
cache-to: type=gha,mode=max,scope=grype-scan
- name: Run Grype container scan
uses: anchore/scan-action@e1165082ffb1fe366ebaf02d8526e7c4989ea9d2 # v7.4.0
id: grype-container
with:
image: 'local-deep-research:grype-scan'
# DO NOT change to fail-build: true — findings are enforced via SARIF alerts
# in the release gate (check-code-scanning-alerts job in release-gate.yml).
# Failing here would break CI without adding security value.
fail-build: false
output-format: sarif
severity-cutoff: medium
# Fail loudly if Grype produced no SARIF — never fabricate an empty one.
# An empty-results SARIF uploaded under the grype-container category would
# make GitHub mark every previously-open Grype alert as fixed, silently
# clearing real findings. Mirror the Trivy job: error + exit 1, skip upload.
- name: Ensure SARIF file exists
id: check-container-sarif
if: always()
run: |
SARIF_FILE="${{ steps.grype-container.outputs.sarif }}"
if [ -f "$SARIF_FILE" ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "::error::Grype container scan did not produce a SARIF file — scan needs to be rerun"
exit 1
fi
- name: Upload Grype container SARIF to GitHub Security tab
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always() && steps.check-container-sarif.outputs.exists == 'true'
with:
sarif_file: ${{ steps.grype-container.outputs.sarif }}
category: grype-container
- name: Display Grype summary
if: always()
run: |
{
echo "## Grype Container Vulnerability Scan"
echo ""
if [ "${{ steps.check-container-sarif.outputs.exists }}" = "true" ]; then
echo "✅ Grype scan completed - Results available in Security tab"
else
echo "❌ Grype scan failed - no SARIF produced (see error above); rerun required"
fi
} >> "$GITHUB_STEP_SUMMARY"
+111
View File
@@ -0,0 +1,111 @@
name: Hadolint Dockerfile Linting
on:
pull_request:
paths:
- 'Dockerfile'
- '.hadolint.yaml'
workflow_dispatch:
workflow_call: # Called by release-gate.yml
schedule:
# Run weekly on Tuesday at 9 AM UTC
- cron: '0 9 * * 2'
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
hadolint:
name: Hadolint Dockerfile Analysis
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Run Hadolint on Dockerfile
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
with:
dockerfile: Dockerfile
format: sarif
output-file: hadolint.sarif
no-fail: false
verbose: true
- name: Upload Hadolint results to GitHub Security
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always()
with:
sarif_file: hadolint.sarif
category: hadolint
- name: Upload Hadolint results as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: hadolint-results
path: hadolint.sarif
retention-days: 7 # Reduced for security
- name: Run Hadolint with detailed output
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
if: always()
with:
dockerfile: Dockerfile
format: tty
no-fail: false
continue-on-error: false
- name: Display Hadolint summary
if: always()
run: |
{
echo "## Hadolint Dockerfile Analysis Summary"
echo ""
echo "✅ Hadolint analysis completed"
echo ""
echo "### What is Hadolint?"
echo "Hadolint is a Dockerfile linter that checks for best practices and common mistakes:"
echo ""
echo "#### Security Checks"
echo "- 🔒 Running as root user"
echo "- 🔑 Using latest tag"
echo "- 📦 Missing version pinning"
echo "- ⚠️ Exposed secrets in build args"
echo ""
echo "#### Best Practices"
echo "- 🎯 Layer optimization"
echo "- 🗑️ Cleaning package manager cache"
echo "- 📝 Using COPY instead of ADD"
echo "- 🔧 Proper WORKDIR usage"
echo "- 💾 Efficient layer caching"
echo ""
echo "#### Build Efficiency"
echo "- 🚀 Multi-stage build opportunities"
echo "- 📦 Unnecessary packages"
echo "- 🔄 Combining RUN commands"
echo ""
echo "📊 **Results:**"
echo "- Detailed results uploaded to GitHub Security tab"
echo "- SARIF file available in artifacts"
echo ""
echo "🔗 **Links:**"
echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
echo "- [Hadolint Rules](https://github.com/hadolint/hadolint#rules)"
} >> "$GITHUB_STEP_SUMMARY"
+139
View File
@@ -0,0 +1,139 @@
name: Issue Research
# When a maintainer applies the `ldr_research` label to an issue,
# run LDR research on the issue title + body and post a comment
# with two sections: a brief, cautious response for the reporter,
# and adjacent context for maintainer triage.
#
# The same `ldr_research` label is reused from the PR workflow.
# GitHub gates by event type (`on: issues:` here vs
# `on: pull_request:` in e2e-research-test.yml), so they don't
# conflict.
on:
issues:
types: [labeled]
permissions: {} # Minimal top-level for OSSF Scorecard
jobs:
build-query:
name: Assemble query
if: github.event.label.name == 'ldr_research'
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
query: ${{ steps.assemble.outputs.query }}
env:
ISSUE_TITLE: ${{ github.event.issue.title }}
ISSUE_BODY: ${{ github.event.issue.body }}
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Sanitize and assemble query
id: assemble
run: |
# Strip control chars (defence-in-depth against weird issue
# bodies) and truncate to 4000 chars before embedding in the
# prompt. The issue body is semi-trusted user input — never
# eval'd, never shelled out, only passed to the LLM as text.
SAFE_TITLE=$(printf '%s' "$ISSUE_TITLE" | tr -d '\000-\010\013-\037\177' | head -c 500)
SAFE_BODY=$(printf '%s' "$ISSUE_BODY" | tr -d '\000-\010\013-\037\177' | head -c 4000)
{
cat <<'PROMPT_HEAD_EOF'
A user filed a GitHub issue against the Local Deep Research project. Produce a comment with two distinct sections:
(1) "**For the reporter**" — a brief, cautious, 2-4 sentence summary of likely diagnostic directions, framed as suggestions and not authoritative diagnosis. If you don't have enough context to be useful, say so plainly.
(2) "**For maintainers**" — adjacent external context: similar reports in other projects, relevant upstream library documentation, known issues with the components mentioned, and related discussions. Treat the maintainer as the primary audience for the substantive research.
Issue title:
PROMPT_HEAD_EOF
printf '%s\n' "$SAFE_TITLE"
cat <<'PROMPT_MID_EOF'
Issue body:
<<<ISSUE_BODY
PROMPT_MID_EOF
printf '%s\n' "$SAFE_BODY"
cat <<'PROMPT_TAIL_EOF'
ISSUE_BODY>>>
PROMPT_TAIL_EOF
} > query.txt
# Randomized delimiter so a query containing the literal delimiter
# on its own line cannot prematurely terminate the heredoc.
DELIM="LDR_QUERY_EOF_$$_${RANDOM}_$(date +%N)"
{
echo "query<<$DELIM"
cat query.txt
echo "$DELIM"
} >> "$GITHUB_OUTPUT"
research:
name: Run LDR research
needs: build-query
# Grant the perms the reusable's job needs:
# - contents: read for actions/checkout
# - actions: write for actions/upload-artifact@v5+
# Without this, the reusable's `permissions: contents: read` exceeds
# the inherited empty permissions and GitHub rejects the workflow at
# load time (startup_failure with zero jobs).
permissions:
contents: read
actions: write
uses: ./.github/workflows/ldr-research-reusable.yml
with:
query: ${{ needs.build-query.outputs.query }}
model: ${{ vars.LDR_RESEARCH_MODEL || 'google/gemini-2.0-flash-001' }}
provider: ${{ vars.LDR_PROVIDER || 'openrouter' }}
search-tool: ${{ vars.LDR_SEARCH_TOOL || 'serper' }}
strategy: ${{ vars.LDR_STRATEGY || 'langgraph-agent' }}
comment-header: '## 🤖 LDR Research'
comment-subheader: '_Auto-generated context for maintainer triage and the reporter — see disclaimer below._'
secrets:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
post-comment:
name: Post issue comment
needs: research
# always() so the label-removal step runs even if research was skipped
# (e.g. build-query failed). The download/post steps are guarded by
# needs.research.outputs.success == 'true' so they self-skip in that
# case.
if: always()
runs-on: ubuntu-latest
permissions:
contents: read
issues: write
env:
ISSUE_NUMBER: ${{ github.event.issue.number }}
LABEL_NAME: ${{ github.event.label.name }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Download research artifact
if: needs.research.outputs.success == 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: ${{ needs.research.outputs.comment-artifact-name }}
- name: Post comment
if: needs.research.outputs.success == 'true'
run: |
gh issue comment "$ISSUE_NUMBER" --repo "${{ github.repository }}" -F comment.md
- name: Remove label for re-triggering
if: always()
run: |
gh issue edit "$ISSUE_NUMBER" --remove-label "$LABEL_NAME" 2>/dev/null || true
@@ -0,0 +1,79 @@
name: Journal Quality Data Integration
# Validates that the five external journal-quality data sources can
# still be downloaded and the read-only reference DB can be built from
# them end to end. Catches upstream schema breaks (OpenAlex renaming a
# field, DOAJ changing their dump format, etc.) BEFORE we cut a release.
#
# This is intentionally a separate workflow rather than a job inside the
# main test suite — it pulls ~25 MB from third-party APIs and takes
# ~3060 seconds, so it shouldn't run on every PR. It runs:
# - Weekly (Mondays 4 AM UTC) — catches drift between releases
# - On the release-gate workflow_call — blocks publishing if upstreams
# have broken our build
# - Manually via workflow_dispatch — for ad-hoc validation
on:
workflow_call: # called by release-gate.yml
workflow_dispatch:
schedule:
# Mondays 4 AM UTC — staggered after release-gate (2 AM) so the two
# don't collide if upstream rate-limits are touchy.
- cron: '0 4 * * 1'
permissions: {} # minimal top-level for OSSF Scorecard
jobs:
download-and-build:
name: Download external sources + build reference DB
runs-on: ubuntu-latest
# 45 min job timeout: parallel downloads are bounded by the slowest
# source (OpenAlex Institutions, ~10 min for ~110K rows via cursor
# pagination). Plus build_db (~15s) + setup overhead. Generous
# headroom so transient API slowness doesn't false-fail the gate.
timeout-minutes: 45
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.11'
- name: Install system dependencies for SQLCipher
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.11'
- name: Install dependencies
run: pdm install --dev
- name: Run journal-quality release-gate integration test
# The test is marked `integration and slow` so it's skipped by
# the regular suite. We opt in here. The session fixture
# downloads all 5 sources in PARALLEL via ThreadPoolExecutor;
# wall-clock is bounded by the slowest source (institutions,
# ~10 min). Per-test pytest-timeout=2400s (40 min) overrides
# the global 60s default for these slow tests.
run: |
pdm run pytest tests/integration/test_journal_quality_release_gate.py \
-m "integration and slow" \
-v --tb=short \
--timeout=2400
timeout-minutes: 42
+46
View File
@@ -0,0 +1,46 @@
name: Sync repo labels
# Declaratively syncs labels listed in .github/labels.yml.
# Additive only: labels not listed here are left untouched (delete-other-labels: false).
# Lifecycle labels (needs-codeowner-review, awaiting-author, awaiting-codeowner) are
# created by this workflow but toggled per-PR by .github/workflows/pr-triage.yml.
on:
push:
branches: [main]
paths:
- '.github/labels.yml'
- '.github/workflows/labels-sync.yml'
workflow_dispatch:
# No concurrency group — intentionally omitted, matching .github/workflows/label-fixed-in-dev.yml.
# Previous attempts (#3554, reverted #3599) showed that cancel-in-progress on label workflows
# kills useful in-flight runs. Sync is idempotent so concurrent runs are safe.
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
sync-labels:
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
issues: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
sparse-checkout: .github/labels.yml
sparse-checkout-cone-mode: false
- name: Sync labels
uses: EndBug/label-sync@52074158190acb45f3077f9099fea818aa43f97a # v2.3.3
with:
config-file: .github/labels.yml
delete-other-labels: false
+300
View File
@@ -0,0 +1,300 @@
name: LDR Research (reusable)
# Reusable workflow that runs the LDR research script on a caller-supplied
# query and produces a comment-ready markdown blob (uploaded as an
# artifact). Callers download the artifact and post it to their target
# (PR comment, issue comment, Reddit, etc.). This workflow does NOT post
# anywhere — that's caller-specific.
on:
workflow_call:
inputs:
query:
description: 'Fully-assembled research prompt. The caller does all prompt engineering.'
type: string
required: true
model:
description: 'OpenRouter model slug. Empty falls back to vars.LDR_RESEARCH_MODEL then a hard default.'
type: string
default: ''
provider:
description: 'LLM provider.'
type: string
default: 'openrouter'
search-tool:
description: 'Search tool.'
type: string
default: 'serper'
strategy:
description: 'LDR search strategy.'
type: string
default: 'langgraph-agent'
iterations:
description: 'Override the strategy iteration cap. 0 keeps the strategy default.'
type: number
default: 0
max-query-length:
description: 'Backstop truncation on the query string before invoking the script.'
type: number
default: 12000
max-sources:
description: 'Cap sources rendered in the markdown.'
type: number
default: 10
comment-header:
description: 'Top header line of the formatted markdown.'
type: string
default: '## LDR Research Results'
comment-subheader:
description: 'Optional second-line subheader.'
type: string
default: ''
comment-footer:
description: 'Footer line. Reddit will swap in a bot disclaimer.'
type: string
default: '_Generated by [Local Deep Research](https://github.com/LearningCircuit/local-deep-research) E2E test_'
include-sources-section:
description: 'Render a dedicated sources section. Reddit will set false to fit length caps.'
type: boolean
default: true
output-truncate-chars:
description: 'Truncate the rendered markdown to this many chars. 0 disables truncation.'
type: number
default: 0
runner:
description: 'Runner label.'
type: string
default: 'ubuntu-latest'
artifact-suffix:
description: 'Optional disambiguator appended to the artifact name. Use a unique value per matrix entry when calling this workflow multiple times in one caller run.'
type: string
default: ''
secrets:
OPENROUTER_API_KEY:
required: true
SERPER_API_KEY:
required: true
outputs:
comment-artifact-name:
description: 'Name of the artifact containing comment.md and response.json.'
value: ${{ jobs.research.outputs.comment-artifact-name }}
success:
description: "'true' if LDR returned valid non-error JSON."
value: ${{ jobs.research.outputs.success }}
permissions: {}
jobs:
research:
name: Run LDR research
runs-on: ${{ inputs.runner }}
permissions:
contents: read
outputs:
comment-artifact-name: ${{ steps.artifact-name.outputs.name }}
success: ${{ steps.run-script.outputs.success }}
env:
QUERY: ${{ inputs.query }}
MAX_QUERY_LENGTH: ${{ inputs.max-query-length }}
MAX_SOURCES: ${{ inputs.max-sources }}
COMMENT_HEADER: ${{ inputs.comment-header }}
COMMENT_SUBHEADER: ${{ inputs.comment-subheader }}
COMMENT_FOOTER: ${{ inputs.comment-footer }}
INCLUDE_SOURCES: ${{ inputs.include-sources-section }}
OUTPUT_TRUNCATE_CHARS: ${{ inputs.output-truncate-chars }}
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 1
- name: Install jq
run: sudo apt-get update && sudo apt-get install -y jq
- name: Compute artifact name
id: artifact-name
env:
SUFFIX: ${{ inputs.artifact-suffix }}
run: |
NAME="ldr-research-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}"
if [ -n "$SUFFIX" ]; then
# Sanitize: artifact names allow [A-Za-z0-9._-] only.
SAFE_SUFFIX=$(printf '%s' "$SUFFIX" | tr -c 'A-Za-z0-9._-' '_')
NAME="${NAME}-${SAFE_SUFFIX}"
fi
echo "name=$NAME" >> "$GITHUB_OUTPUT"
- name: Resolve model
id: resolve-model
env:
INPUT_MODEL: ${{ inputs.model }}
REPO_VAR_MODEL: ${{ vars.LDR_RESEARCH_MODEL }}
run: |
if [ -n "$INPUT_MODEL" ]; then
echo "model=$INPUT_MODEL" >> "$GITHUB_OUTPUT"
elif [ -n "$REPO_VAR_MODEL" ]; then
echo "model=$REPO_VAR_MODEL" >> "$GITHUB_OUTPUT"
else
echo "model=google/gemini-2.0-flash-001" >> "$GITHUB_OUTPUT"
fi
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
cache: true
- name: Install LDR
run: pdm install
- name: Write query to file (with backstop truncation)
run: |
# Truncate by total byte count, not per-line. Avoids putting the
# entire query on the command line where it could exceed ARG_MAX.
QUERY_BYTES=$(printf '%s' "$QUERY" | wc -c)
if [ "$QUERY_BYTES" -gt "$MAX_QUERY_LENGTH" ]; then
printf '%s' "$QUERY" | head -c "$MAX_QUERY_LENGTH" > query.txt
printf '\n... (truncated)\n' >> query.txt
echo "Query truncated: $QUERY_BYTES -> $MAX_QUERY_LENGTH bytes"
else
printf '%s' "$QUERY" > query.txt
fi
echo "Final query size: $(wc -c < query.txt) bytes"
- name: Run LDR Research
id: run-script
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
LDR_PROVIDER: ${{ inputs.provider }}
LDR_SEARCH_TOOL: ${{ inputs.search-tool }}
LDR_RESEARCH_MODEL: ${{ steps.resolve-model.outputs.model }}
LDR_STRATEGY: ${{ inputs.strategy }}
ITERATIONS: ${{ inputs.iterations }}
run: |
set +e
if [ "$ITERATIONS" -gt 0 ] 2>/dev/null; then
pdm run python scripts/ldr-research.py --iterations "$ITERATIONS" \
< query.txt 2> >(tee stderr.log >&2) > response.json
else
pdm run python scripts/ldr-research.py \
< query.txt 2> >(tee stderr.log >&2) > response.json
fi
LDR_EXIT_CODE=$?
set -e
echo "=== Response (first 2000 chars): ==="
head -c 2000 response.json
echo ""
echo "=== End response ==="
# Catches SIGABRT/native crashes where stdout never flushed; JSON check alone can't see this.
if [ "$LDR_EXIT_CODE" -ne 0 ]; then
LAST_ERR=$(tail -c 500 stderr.log 2>/dev/null || echo "")
echo "::error::ldr-research.py exited with code $LDR_EXIT_CODE: $LAST_ERR"
echo "success=false" >> "$GITHUB_OUTPUT"
exit 1
fi
# `jq .` exits 0 on a zero-byte file, so guard explicitly.
if [ ! -s response.json ]; then
echo "::error::response.json is empty"
echo "success=false" >> "$GITHUB_OUTPUT"
exit 1
fi
# Shape validation: must be a JSON object.
if ! jq -e 'type == "object"' response.json > /dev/null 2>&1; then
echo "::error::Response is not a JSON object"
echo "success=false" >> "$GITHUB_OUTPUT"
exit 1
fi
ERROR=$(jq -r '.error // empty' response.json)
if [ -n "$ERROR" ]; then
echo "::error::LDR error: $ERROR"
echo "success=false" >> "$GITHUB_OUTPUT"
exit 1
fi
# Empty .research would still produce a hollow downstream comment.
RESEARCH=$(jq -r '.research // empty' response.json)
if [ -z "$RESEARCH" ]; then
echo "::error::Response missing or empty .research field"
echo "success=false" >> "$GITHUB_OUTPUT"
exit 1
fi
echo "success=true" >> "$GITHUB_OUTPUT"
echo "✅ Research completed"
- name: Build comment markdown
run: |
jq -r '.research' response.json > result.md
# Optional sources section
if [ "$INCLUDE_SOURCES" = "true" ]; then
SOURCES=$(jq -r --argjson cap "$MAX_SOURCES" \
'.sources[:$cap][] | "- [\(.title // "Source")](\(.link // .url // ""))"' \
response.json 2>/dev/null || echo "")
else
SOURCES=""
fi
{
echo "$COMMENT_HEADER"
echo ""
if [ -n "$COMMENT_SUBHEADER" ]; then
echo "$COMMENT_SUBHEADER"
echo ""
fi
cat result.md
echo ""
if [ -n "$SOURCES" ]; then
echo "### 🔍 Search Sources"
echo ""
echo "$SOURCES"
echo ""
fi
echo "---"
echo "$COMMENT_FOOTER"
} > comment.md
# Optional truncation (Reddit will use this)
if [ "$OUTPUT_TRUNCATE_CHARS" -gt 0 ] 2>/dev/null; then
CURRENT=$(wc -c < comment.md)
if [ "$CURRENT" -gt "$OUTPUT_TRUNCATE_CHARS" ]; then
# Truncate at last paragraph break before the limit, with a budget
# for the trailing marker.
MARKER='…[truncated]'
BUDGET=$((OUTPUT_TRUNCATE_CHARS - ${#MARKER} - 1))
head -c "$BUDGET" comment.md > comment.md.tmp
# Trim trailing partial line so we end on a clean paragraph
awk 'BEGIN{RS=""} {gsub(/[[:space:]]+$/,""); print}' \
comment.md.tmp > comment.md
echo "" >> comment.md
echo "$MARKER" >> comment.md
rm -f comment.md.tmp
echo "Truncated comment.md to $OUTPUT_TRUNCATE_CHARS chars."
fi
fi
echo "Final comment.md size: $(wc -c < comment.md) bytes"
- name: Upload research artifact
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.artifact-name.outputs.name }}
path: |
comment.md
response.json
stderr.log
if-no-files-found: ignore
retention-days: 7
+115
View File
@@ -0,0 +1,115 @@
name: MCP Server Tests
on:
push:
branches: [ main ]
paths:
- 'src/local_deep_research/mcp/**'
- 'tests/mcp/**'
- 'scripts/mcp_smoke_test.sh'
- '.github/workflows/mcp-tests.yml'
pull_request:
branches: [ main ]
paths:
- 'src/local_deep_research/mcp/**'
- 'tests/mcp/**'
- 'scripts/mcp_smoke_test.sh'
- '.github/workflows/mcp-tests.yml'
workflow_dispatch:
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions:
contents: read
jobs:
mcp-tests:
name: MCP Server Tests
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install pip==25.0
pip install pdm==2.26.2
pdm install --dev --no-editable -G mcp
- name: Check if MCP server module exists
id: check-mcp
run: |
if [ -f "src/local_deep_research/mcp/server.py" ]; then
echo "mcp_exists=true" >> "$GITHUB_OUTPUT"
echo "MCP server module found, will run tests"
else
echo "mcp_exists=false" >> "$GITHUB_OUTPUT"
echo "MCP server module not found (feature not yet merged), skipping tests"
fi
- name: Run MCP smoke tests
if: steps.check-mcp.outputs.mcp_exists == 'true'
run: pdm run bash scripts/mcp_smoke_test.sh
- name: Run MCP unit tests
if: steps.check-mcp.outputs.mcp_exists == 'true'
run: |
pdm run pytest tests/mcp/ -v --tb=short -n auto
env:
LDR_TESTING_WITH_MOCKS: "true"
- name: Skip notice
if: steps.check-mcp.outputs.mcp_exists != 'true'
run: |
echo "::notice::MCP server module not yet implemented. Tests will run once feature/mcp-server is merged."
- name: Generate test summary
if: always()
env:
MCP_EXISTS: ${{ steps.check-mcp.outputs.mcp_exists }}
run: |
{
echo "## MCP Server Test Summary"
echo ""
if [ "$MCP_EXISTS" != "true" ]; then
echo "### ⏭️ Tests Skipped"
echo ""
echo "MCP server module (\`src/local_deep_research/mcp/server.py\`) not yet implemented."
echo "Tests will run automatically once the MCP feature branch is merged."
else
echo "### What was tested:"
echo "- 🔌 MCP server module loading"
echo "- 🔧 Discovery tools (list_strategies, list_search_engines, get_configuration)"
echo "- 🧪 Unit tests for all MCP tools"
echo "- 🚀 Server startup verification"
echo ""
echo "### MCP Tools Tested:"
echo "| Tool | Description |"
echo "|------|-------------|"
echo "| \`quick_research\` | Fast research summary (1-5 min) |"
echo "| \`detailed_research\` | Comprehensive analysis (5-15 min) |"
echo "| \`generate_report\` | Full markdown report (10-30 min) |"
echo "| \`analyze_documents\` | Search local collections |"
echo "| \`list_search_engines\` | List available search engines |"
echo "| \`list_strategies\` | List research strategies |"
echo "| \`get_configuration\` | Get current config |"
fi
} >> "$GITHUB_STEP_SUMMARY"
+81
View File
@@ -0,0 +1,81 @@
name: Mypy Type Checking
on:
pull_request:
branches: [ main ]
workflow_call: # Called by ci-gate.yml for release pipeline
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions:
contents: read
jobs:
mypy-analysis:
runs-on: ubuntu-latest
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install system dependencies for SQLCipher
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
- name: Install PDM
run: |
pip install pdm==2.26.2
- name: Cache PDM dependencies
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/.cache/pdm
.venv
key: pdm-${{ runner.os }}-${{ hashFiles('pdm.lock') }}
restore-keys: |
pdm-${{ runner.os }}-
- name: Install project dependencies
run: |
pdm sync -d
- name: Run mypy type checking
run: |
pdm run mypy src/ --config-file=pyproject.toml --show-error-codes --pretty
- name: Display mypy summary
if: always()
run: |
{
echo "## Mypy Type Checking Results"
echo ""
echo "Type checking analysis completed."
echo ""
echo "### Benefits"
echo "- Catches type errors before runtime"
echo "- Improves code quality and maintainability"
echo "- Provides better IDE autocomplete"
echo "- Detects uninitialized variables and type mismatches"
echo ""
echo "See full output above for any type errors found."
} >> "$GITHUB_STEP_SUMMARY"
+81
View File
@@ -0,0 +1,81 @@
name: npm Security Audit
on:
workflow_dispatch:
workflow_call: # Called by release-gate.yml
permissions:
contents: read
jobs:
npm-audit:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
- name: Run npm audit on all package.json locations
env:
# Advisories with NO upstream fix, accepted ONLY because they live in
# dev/test-only tooling that never ships to users. Keep tight + justified:
# GHSA-h67p-54hq-rp68 — js-yaml 3.x DoS, pulled transitively by
# @lhci/utils (accessibility_tests) & @istanbuljs/load-nyc-config
# (infrastructure_tests); both pin js-yaml ^3, no patched 3.x exists.
# Dev-only, parses trusted local config. (dismissed alerts #97/#7893)
AUDIT_ALLOWLIST: "GHSA-h67p-54hq-rp68"
run: |
FAILED=false
FILTER="${GITHUB_WORKSPACE}/.github/scripts/npm_audit_allowlist.py"
# Audit root and every test directory that contains a package.json
DIRS=(
.
tests
tests/ui_tests
tests/ui_tests/playwright
tests/accessibility_tests
tests/api_tests_with_login
tests/infrastructure_tests
tests/puppeteer
)
for dir in "${DIRS[@]}"; do
echo "=== Running npm audit on ${dir} ==="
if [ ! -f "${dir}/package.json" ]; then
echo "No package.json found in ${dir}"
continue
fi
# Require committed lockfile for reproducible security audits
if [ ! -f "${dir}/package-lock.json" ]; then
echo "::error::Missing ${dir}/package-lock.json. Please commit your lockfile for security audits."
FAILED=true
continue
fi
# Audit, then filter out allowlisted (no-fix, dev-only) advisories.
# `|| true` is on the assignment (not inside $()) so npm's non-zero
# exit is swallowed without the SC2015 `A && B || C` idiom; an
# empty/error capture still fails safe in the filter.
audit_json="$(cd "${dir}" && npm audit --audit-level=moderate --json 2>/dev/null)" || true
if ! printf '%s' "${audit_json}" | python3 "${FILTER}"; then
FAILED=true
fi
done
if [[ "$FAILED" == "true" ]]; then
echo "❌ npm audit found non-allowlisted moderate or higher severity vulnerabilities"
echo "Fix: bump the dep / 'npm audit fix'; or if no fix exists and it is dev-only, add the advisory to AUDIT_ALLOWLIST with justification."
exit 1
else
echo "✅ No non-allowlisted moderate or higher severity vulnerabilities found"
fi
+250
View File
@@ -0,0 +1,250 @@
name: Nuclei Vulnerability Scan
# Template-driven DAST scanner — complements ZAP with known-CVE checks,
# misconfiguration detection, exposed panel discovery, and default credential
# testing. Uses ProjectDiscovery's 6,500+ community-maintained templates.
#
# Authenticated scan: the runner pre-creates the standard CI test_admin user,
# logs in via the real /auth/login flow to capture a session cookie, and
# seeds Nuclei with the full Flask url_map so the scanner probes the entire
# authenticated app surface (settings, research, history, API, …) — not just
# the unauthenticated landing page.
on:
workflow_call: # Called by release-gate.yml
workflow_dispatch:
permissions: {}
jobs:
nuclei-scan:
name: Nuclei DAST Scan
runs-on: ubuntu-latest
timeout-minutes: 40
permissions:
contents: read
security-events: write
actions: read
env:
CI: true
TEST_ENV: true
FLASK_ENV: testing
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
# Honour the fast test KDF: without LDR_TEST_MODE the value is clamped
# to the production minimum (256000), making every SQLCipher open/backup
# ~256x slower — see #4430 and tests/shared/chrome_profile.js context.
LDR_TEST_MODE: "1"
LDR_DISABLE_RATE_LIMITING: "true"
SECRET_KEY: test-secret-key-for-ci
TEST_USERNAME: test_admin
TEST_PASSWORD: testpass123
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Install system dependencies for SQLCipher
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
- name: Install dependencies
run: |
pdm sync -d
- name: Setup test data directory
run: |
mkdir -p "$LDR_DATA_DIR/encrypted_databases"
- name: Pre-create CI test user
# Avoids the slow registration path (KDF + 500 settings rows) and
# avoids hitting the registration rate limit during the scan window.
run: |
export PYTHONPATH=$PWD/src:$PYTHONPATH
pdm run python scripts/ci/init_test_database.py
- name: Start LDR server for testing
run: |
export PYTHONPATH=$PWD/src:$PYTHONPATH
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
SERVER_PID=$!
echo "$SERVER_PID" > server.pid
# Wait for server to start.
# --connect-timeout/--max-time bound TCP and total request time so a
# hung connection fails fast instead of eating the job timeout.
for _ in {1..90}; do
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
echo "Server started successfully"
break
fi
sleep 1
done
# Verify server is running
if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
echo "Server failed to start"
cat server.log
exit 1
fi
- name: Dump Flask url_map for URL seeding
# Without -list, Nuclei only probes the single -target URL. Seeding
# with the real url_map lets it exercise every blueprint route.
run: |
export PYTHONPATH=$PWD/src:$PYTHONPATH
pdm run python scripts/ci/dump_url_map.py http://127.0.0.1:5000 > urls.txt
echo "Seeded $(wc -l < urls.txt) URLs:"
head -20 urls.txt
echo "..."
- name: Authenticate and capture session cookie
id: login
run: |
# Step 1: GET /auth/login → establishes a Flask session cookie and
# returns the HTML form with the per-session CSRF token.
curl -sS -c cookies.txt -o login.html http://127.0.0.1:5000/auth/login
CSRF=$(grep -oE 'name="csrf_token"[[:space:]]+value="[^"]+"' login.html \
| head -n1 \
| sed -E 's/.*value="([^"]+)".*/\1/')
if [ -z "$CSRF" ]; then
echo "Failed to extract CSRF token from login page"
head -200 login.html
exit 1
fi
# Step 2: POST credentials with the CSRF token and the cookie jar.
# -L follows the post-login 302 to /. -o /dev/null discards body,
# -w prints the final HTTP code so a non-2xx fails the step.
HTTP_CODE=$(curl -sS -L \
-b cookies.txt -c cookies.txt \
-o /dev/null -w '%{http_code}' \
--data-urlencode "username=${TEST_USERNAME}" \
--data-urlencode "password=${TEST_PASSWORD}" \
--data-urlencode "csrf_token=${CSRF}" \
http://127.0.0.1:5000/auth/login)
if [ "$HTTP_CODE" != "200" ]; then
echo "Login failed with HTTP $HTTP_CODE"
cat server.log
exit 1
fi
# Verify the session is actually authenticated (not just that
# the login page rendered with a 200 from a re-display of errors).
AUTHED=$(curl -sS -b cookies.txt http://127.0.0.1:5000/auth/check)
echo "auth/check response: $AUTHED"
echo "$AUTHED" | grep -q '"authenticated":[[:space:]]*true' || {
echo "Authenticated check failed"
exit 1
}
# Extract the session cookie value from the Netscape cookie jar.
# Format: domain TAB tailmatch TAB path TAB secure TAB expires TAB name TAB value
SESSION=$(awk '$6 == "session" { print $7 }' cookies.txt | tail -n1)
if [ -z "$SESSION" ]; then
# Print every column EXCEPT the value ($7) so we can debug
# without leaking the cookie if the awk filter is broken.
echo "Could not find 'session' cookie in jar (values redacted):"
awk '{ print $1, $2, $3, $4, $5, $6 }' cookies.txt
exit 1
fi
# Mask the cookie in logs — it grants full app access for this run.
echo "::add-mask::$SESSION"
echo "session=$SESSION" >> "$GITHUB_OUTPUT"
# Give the post-login background thread a moment to finish
# the settings-migration / library-init pass it kicks off
# (see _perform_post_login_tasks in web/auth/routes.py).
# Otherwise the first authenticated probes can race those
# writes and 500 on settings-dependent routes.
sleep 2
- name: Create empty SARIF fallback
run: |
cat > nuclei.sarif << 'SARIF'
{
"version": "2.1.0",
"$schema": "https://json.schemastore.org/sarif-2.1.0.json",
"runs": [{
"tool": {
"driver": {
"name": "Nuclei",
"informationUri": "https://github.com/projectdiscovery/nuclei",
"rules": []
}
},
"results": []
}]
}
SARIF
- name: Run Nuclei scan
uses: projectdiscovery/nuclei-action@cc153d0541e1adf8a42bbe31c0a4fb2376147538 # v3.1.1
with:
# -list: probe every route in the Flask url_map (not just /).
# -H: attach the authenticated session cookie so probes hit the
# real app surface, not the login redirect.
# -severity: drop info-level noise (form-detection, options-method,
# intentional CSP/cookie choices). Real findings are >= low.
# -etags intrusive,dos,fuzz: with a live session, default templates
# can mutate state or DoS the runner. Exclude the standard
# destructive tag set for authenticated DAST.
# -eid http-missing-security-headers: HSTS is correctly omitted on
# plain HTTP localhost; X-XSS-Protection is intentionally
# omitted (deprecated, replaced by CSP). See
# security/security_headers.py.
args: >-
-list urls.txt
-H "Cookie: session=${{ steps.login.outputs.session }}"
-severity low,medium,high,critical
-etags intrusive,dos,fuzz
-eid http-missing-security-headers
-sarif-export nuclei.sarif
-output nuclei.log
- name: Upload Nuclei SARIF to GitHub Security tab
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always()
with:
sarif_file: nuclei.sarif
category: nuclei-dast
- name: Upload Nuclei scan artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: nuclei-scan-results
path: |
nuclei.log
nuclei.sarif
urls.txt
server.log
retention-days: 30
- name: Stop server
if: always()
run: |
if [ -f server.pid ]; then
kill "$(cat server.pid)" || true
rm server.pid
fi
+62
View File
@@ -0,0 +1,62 @@
name: OSSF Scorecard
on:
# Run on branches (for analysis purposes)
branch_protection_rule:
# Run on schedule
schedule:
# Run weekly on Monday at 8 AM UTC
- cron: '0 8 * * 1'
# Allow manual runs
workflow_dispatch:
# Run on push to default branch
push:
branches: [ main ]
# Declare default permissions as read only.
permissions: read-all
jobs:
analysis:
name: OSSF Security Scorecard Analysis
runs-on: ubuntu-latest
permissions:
# Needed to upload the results to code-scanning dashboard.
security-events: write
# Needed to publish results and get a badge
id-token: write
# Needed for private repositories
contents: read
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Run OSSF Scorecard analysis
uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a # v2.4.3
with:
results_file: results.sarif
results_format: sarif
# Publish results to enable scorecard badges
publish_results: true
- name: Upload OSSF Scorecard results to GitHub Security
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
with:
sarif_file: results.sarif
category: ossf-scorecard
- name: Upload OSSF Scorecard results as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ossf-scorecard-results
path: results.sarif
retention-days: 7 # Reduced for security
@@ -0,0 +1,35 @@
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
# Scheduled OSV-Scanner scanning for vulnerabilities.
# Runs on push to main and weekly schedule.
# Split from osv-scanner.yml to prevent "skipped" noise on PRs.
#
# For more examples and options, including how to ignore specific vulnerabilities,
# see https://google.github.io/osv-scanner/github-action/
name: OSV-Scanner (Scheduled)
on:
push:
branches: [ "main" ] # Create baseline for Code Scanning comparison
schedule:
- cron: '41 21 * * 1' # Weekly on Monday at 21:41 UTC
workflow_dispatch:
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
scan-scheduled:
permissions:
security-events: write
contents: read
actions: read
uses: "google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2" # v2.3.8
with:
scan-args: |-
-r
--skip-git
./
+48
View File
@@ -0,0 +1,48 @@
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.
# OSV-Scanner for detecting vulnerabilities in dependencies.
# Runs on PRs to catch vulnerable dependencies before merge (shift-left security).
#
# NOTE: This workflow is NOT included in release-gate.yml because
# GitHub Actions limits reusable workflow nesting to 2 levels. Since this
# workflow calls google/osv-scanner-reusable.yml, including it in the gate
# would create 4 levels of nesting and cause startup_failure:
# release.yml → release-gate.yml → osv-scanner.yml → google/osv-scanner-reusable.yml
#
# For more examples and options, including how to ignore specific vulnerabilities,
# see https://google.github.io/osv-scanner/github-action/
name: OSV-Scanner
on:
pull_request:
branches: [ "main" ]
merge_group:
branches: [ "main" ]
schedule:
- cron: '39 12 * * 1' # Weekly scan for newly disclosed CVEs
workflow_dispatch:
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results.
# Future iteration could safely add concurrency for scheduled/push-only
# triggers (where head_ref is empty and runs get unique groups).
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
scan:
permissions:
security-events: write
contents: read
actions: read
uses: "google/osv-scanner-action/.github/workflows/osv-scanner-reusable.yml@9a498708959aeaef5ef730655706c5a1df1edbc2" # v2.3.8
with:
scan-args: |-
-r
--skip-git
./
+212
View File
@@ -0,0 +1,212 @@
name: OWASP ZAP Security Scan
# Baseline and API scans - called by release-gate.yml
on:
workflow_call: # Called by release-gate.yml
workflow_dispatch:
# Restrict top-level permissions (jobs define their own)
permissions: {}
jobs:
zap-baseline-scan:
name: ZAP Baseline Scan
runs-on: ubuntu-latest
timeout-minutes: 45
permissions:
contents: read
security-events: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Install system dependencies for SQLCipher
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
- name: Install dependencies
run: |
pdm sync -d
- name: Start LDR server for testing
run: |
export CI=true
export LDR_DB_CONFIG_KDF_ITERATIONS=1000
# Without LDR_TEST_MODE the 1000 is clamped to the production
# minimum (256000), making SQLCipher ~256x slower — see #4430.
export LDR_TEST_MODE=1
export PYTHONPATH=$PWD/src:$PYTHONPATH
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
SERVER_PID=$!
echo "$SERVER_PID" > server.pid
# Wait for server to start.
# --connect-timeout/--max-time bound TCP and total request time so a
# hung connection fails fast instead of eating the job timeout.
for _ in {1..90}; do
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
echo "Server started successfully"
break
fi
sleep 1
done
# Verify server is running
if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
echo "Server failed to start"
cat server.log
exit 1
fi
- name: Run ZAP Baseline Scan
uses: zaproxy/action-baseline@de8ad967d3548d44ef623df22cf95c3b0baf8b25 # v0.15.0
with:
target: 'http://localhost:5000'
rules_file_name: '.zap/rules.tsv'
cmd_options: '-a -I' # -I: only fail on FAIL-level alerts per rules.tsv (ignore WARN/INFO)
allow_issue_writing: false
fail_action: true
artifact_name: zapbaseline
- name: Upload ZAP scan results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: zap-baseline-scan-results
path: |
report_html.html
report_json.json
report_md.md
retention-days: 30
- name: Stop server
if: always()
run: |
if [ -f server.pid ]; then
kill "$(cat server.pid)" || true
rm server.pid
fi
zap-api-scan:
name: ZAP API Scan
runs-on: ubuntu-latest
timeout-minutes: 45
permissions:
contents: read
security-events: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Install system dependencies for SQLCipher
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
- name: Install dependencies
run: |
pdm sync -d
- name: Start LDR server for testing
run: |
export CI=true
export LDR_DB_CONFIG_KDF_ITERATIONS=1000
# Without LDR_TEST_MODE the 1000 is clamped to the production
# minimum (256000), making SQLCipher ~256x slower — see #4430.
export LDR_TEST_MODE=1
export PYTHONPATH=$PWD/src:$PYTHONPATH
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
SERVER_PID=$!
echo "$SERVER_PID" > server.pid
# Wait for server to start.
# --connect-timeout/--max-time bound TCP and total request time so a
# hung connection fails fast instead of eating the job timeout.
for _ in {1..90}; do
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
echo "Server started successfully"
break
fi
sleep 1
done
# Verify server is running
if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; then
echo "Server failed to start"
cat server.log
exit 1
fi
- name: Generate OpenAPI specification
run: |
# If you have an OpenAPI spec, use it. Otherwise, ZAP will discover endpoints
# For now, we'll let ZAP discover API endpoints automatically
echo "Using automatic API endpoint discovery"
- name: Run ZAP API Scan
uses: zaproxy/action-api-scan@5158fe4d9d8fcc75ea204db81317cce7f9e5453d # v0.10.0
with:
target: 'http://localhost:5000/api/v1'
format: 'openapi'
rules_file_name: '.zap/rules.tsv'
cmd_options: '-a -I' # -I: only fail on FAIL-level alerts per rules.tsv (ignore WARN/INFO)
allow_issue_writing: false
fail_action: true
artifact_name: zapapi
- name: Upload ZAP API scan results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: zap-api-scan-results
path: |
report_html.html
report_json.json
report_md.md
retention-days: 30
- name: Stop server
if: always()
run: |
if [ -f server.pid ]; then
kill "$(cat server.pid)" || true
rm server.pid
fi
@@ -0,0 +1,351 @@
name: Playwright WebKit Tests
on:
workflow_call: # Called by release pipeline
workflow_dispatch: # Manual trigger
schedule:
- cron: '0 2 * * *' # Daily at 2 AM UTC
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
# Shared setup: start server, register user, then run Desktop Safari tests
desktop-safari:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
version: 2.26.2
cache: true
- name: Install Python dependencies
run: |
pdm install -d
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
cache: 'npm'
cache-dependency-path: |
package-lock.json
tests/ui_tests/playwright/package-lock.json
- name: Build Vite frontend assets
run: |
npm ci
npm run build
- name: Install Playwright dependencies (npm)
working-directory: tests/ui_tests/playwright
run: npm ci
# ``playwright install --with-deps`` downloads ~150 MB of browser
# binaries from the playwright CDN per attempt. Single CDN
# hiccups have failed the whole release pipeline in the past
# (e.g. transient 5xx mid-download); a single retry costs little
# and removes the dominant transient-network failure mode for
# this workflow. Two attempts only — three attempts of a 150 MB
# download starts hiding real outages.
- name: Install Playwright browsers
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
with:
timeout_minutes: 10
max_attempts: 2
retry_on: error
shell: bash
# Install chromium for auth setup step + webkit for Safari tests
command: |
cd tests/ui_tests/playwright
npx playwright install chromium webkit --with-deps
- name: Setup test directories
run: |
mkdir -p data/encrypted_databases
echo "Created data directories for tests"
- name: Start LDR server
env:
CI: true
TEST_ENV: true
FLASK_ENV: testing
LDR_DATA_DIR: ${{ github.workspace }}/data
# The server MUST open test_admin's encrypted DB with the same KDF it
# was created with. init_test_database.py creates it at 1000 iterations
# (LDR_TEST_MODE relaxes the floor to allow it); without these two vars
# here the requested 1000 is below the production floor, so it falls
# back to the 256000 default, the server derives the wrong key, and
# every login fails with a 401 — see #4558, which set these on the init
# step but missed this server step.
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
LDR_TEST_MODE: "1"
LDR_DISABLE_RATE_LIMITING: true
SECRET_KEY: test-secret-key-for-ci
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
pdm run python -m local_deep_research.web.app &
SERVER_PID=$!
echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
# Wait for server to be ready.
# --connect-timeout/--max-time bound TCP and total request time so a
# hung connection fails fast instead of eating the job timeout.
for i in {1..30}; do
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
echo "Server is ready"
break
fi
echo "Waiting for server... ($i/30)"
sleep 2
done
# Verify server is running
if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
echo "Server failed to start"
exit 1
fi
- name: Initialize test database
env:
TEST_ENV: true
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
# Without LDR_TEST_MODE the 1000 is clamped to the production
# minimum (256000), making SQLCipher ~256x slower — see #4430.
LDR_TEST_MODE: "1"
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
pdm run python scripts/ci/init_test_database.py
- name: Register CI test user
working-directory: tests/ui_tests
run: |
npm ci
node register_ci_user.js http://127.0.0.1:5000
- name: Run Desktop Safari Tests
working-directory: tests/ui_tests/playwright
env:
CI: true
TEST_BASE_URL: http://127.0.0.1:5000
TEST_USERNAME: test_admin
TEST_PASSWORD: testpass123
# Safari CI runs a critical subset for fast release gate:
# - all-pages-mobile: overflow, touch targets, nav clearance across all pages
# - auth-pages-mobile: login, register, change-password mobile layouts
# - desktop-layout: sidebar positioning and breakpoint testing
# - interactive-states: advanced options, search, help panels, mobile menus
# - settings-subpages-mobile: settings tab navigation and form layouts
# - mobile-ui-audit: comprehensive mobile UI quality checks
# - embedding-settings-dropdown: regression for #3863 (model dropdown reset)
# - settings-mobile-collapse: sections collapse by default on mobile (#4032),
# stay expanded on desktop, and search force-expands survivors
# Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
# When adding Safari-critical tests, update this filter pattern.
run: |
npx playwright test \
--project="Desktop Safari" \
"all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit|embedding-settings-dropdown|settings-mobile-collapse"
- name: Upload Playwright Report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: playwright-report-desktop-safari
path: |
tests/ui_tests/playwright/playwright-report/
tests/ui_tests/playwright/test-results/
- name: Stop server
if: always()
run: |
if [ -n "$SERVER_PID" ]; then
kill "$SERVER_PID" || true
fi
pkill -f "python -m local_deep_research.web.app" || true
# Mobile Safari tests run in parallel on a separate runner
mobile-safari:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
version: 2.26.2
cache: true
- name: Install Python dependencies
run: |
pdm install -d
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
cache: 'npm'
cache-dependency-path: |
package-lock.json
tests/ui_tests/playwright/package-lock.json
- name: Build Vite frontend assets
run: |
npm ci
npm run build
- name: Install Playwright dependencies (npm)
working-directory: tests/ui_tests/playwright
run: npm ci
# ``playwright install --with-deps`` downloads ~150 MB of browser
# binaries from the playwright CDN per attempt. Single CDN
# hiccups have failed the whole release pipeline in the past
# (e.g. transient 5xx mid-download); a single retry costs little
# and removes the dominant transient-network failure mode for
# this workflow. Two attempts only — three attempts of a 150 MB
# download starts hiding real outages.
- name: Install Playwright browsers
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
with:
timeout_minutes: 10
max_attempts: 2
retry_on: error
shell: bash
# Install chromium for auth setup step + webkit for Safari tests
command: |
cd tests/ui_tests/playwright
npx playwright install chromium webkit --with-deps
- name: Setup test directories
run: |
mkdir -p data/encrypted_databases
echo "Created data directories for tests"
- name: Start LDR server
env:
CI: true
TEST_ENV: true
FLASK_ENV: testing
LDR_DATA_DIR: ${{ github.workspace }}/data
# The server MUST open test_admin's encrypted DB with the same KDF it
# was created with. init_test_database.py creates it at 1000 iterations
# (LDR_TEST_MODE relaxes the floor to allow it); without these two vars
# here the requested 1000 is below the production floor, so it falls
# back to the 256000 default, the server derives the wrong key, and
# every login fails with a 401 — see #4558, which set these on the init
# step but missed this server step.
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
LDR_TEST_MODE: "1"
LDR_DISABLE_RATE_LIMITING: true
SECRET_KEY: test-secret-key-for-ci
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
echo "Starting server with LDR_DATA_DIR=$LDR_DATA_DIR"
pdm run python -m local_deep_research.web.app &
SERVER_PID=$!
echo "SERVER_PID=$SERVER_PID" >> "$GITHUB_ENV"
# Wait for server to be ready.
# --connect-timeout/--max-time bound TCP and total request time so a
# hung connection fails fast instead of eating the job timeout.
for i in {1..30}; do
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
echo "Server is ready"
break
fi
echo "Waiting for server... ($i/30)"
sleep 2
done
# Verify server is running
if ! curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health > /dev/null; then
echo "Server failed to start"
exit 1
fi
- name: Initialize test database
env:
TEST_ENV: true
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
# Without LDR_TEST_MODE the 1000 is clamped to the production
# minimum (256000), making SQLCipher ~256x slower — see #4430.
LDR_TEST_MODE: "1"
run: |
export PYTHONPATH="$PWD/src:$PYTHONPATH"
pdm run python scripts/ci/init_test_database.py
- name: Register CI test user
working-directory: tests/ui_tests
run: |
npm ci
node register_ci_user.js http://127.0.0.1:5000
- name: Run Mobile Safari Tests
working-directory: tests/ui_tests/playwright
env:
CI: true
TEST_BASE_URL: http://127.0.0.1:5000
TEST_USERNAME: test_admin
TEST_PASSWORD: testpass123
# Safari CI runs a critical subset for fast release gate:
# - all-pages-mobile: overflow, touch targets, nav clearance across all pages
# - auth-pages-mobile: login, register, change-password mobile layouts
# - desktop-layout: sidebar positioning and breakpoint testing
# - interactive-states: advanced options, search, help panels, mobile menus
# - settings-subpages-mobile: settings tab navigation and form layouts
# - mobile-ui-audit: comprehensive mobile UI quality checks
# - settings-mobile-collapse: sections collapse by default on mobile (#4032);
# this is the viewport where the >16384px overflow regression bites
# Note: embedding-settings-dropdown is desktop-only (skips on isMobile),
# so it intentionally only appears in the Desktop Safari filter above.
# Full suite: run locally via `cd tests/ui_tests/playwright && npm test`
# When adding Safari-critical tests, update this filter pattern.
run: |
npx playwright test \
--project="Mobile Safari" \
"all-pages-mobile|auth-pages-mobile|desktop-layout|interactive-states|settings-subpages-mobile|mobile-ui-audit|settings-mobile-collapse"
- name: Upload Playwright Report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: playwright-report-mobile-safari
path: |
tests/ui_tests/playwright/playwright-report/
tests/ui_tests/playwright/test-results/
- name: Stop server
if: always()
run: |
if [ -n "$SERVER_PID" ]; then
kill "$SERVER_PID" || true
fi
pkill -f "python -m local_deep_research.web.app" || true
+219
View File
@@ -0,0 +1,219 @@
name: PR triage labels
# Auto-applies author-class labels at PR open and toggles lifecycle labels
# on review/synchronize events. Labels are defined in .github/labels.yml
# and synced by .github/workflows/labels-sync.yml.
#
# Also applies path-derived topic labels (documentation, tests, docker,
# github-actions, ...) that feed release-notes categorization. These were
# historically applied by the AI code reviewer, which became label-only
# opt-in (#4955), so triage now owns the mechanical subset. Topic labels
# are applied on opened and ready_for_review only, and are additive-only:
# a label a maintainer removed is not re-added on every push, and
# semantic labels (bugfix, feature, security, performance...) stay
# human/AI-applied.
#
# Uses pull_request (not pull_request_target) — fork PRs get a read-only
# token, so label calls return 403. The script catches 403 and continues
# so fork PRs don't show a failing check; maintainers can apply labels
# manually for fork PRs that need them. We accept the no-op behavior on
# forks vs the security cost of pull_request_target running with secrets
# on fork code.
#
# CODEOWNERS list below is mirrored in .github/CODEOWNERS global owners
# (line 6) — keep both in sync when the maintainer roster changes.
on:
pull_request:
types: [opened, synchronize, ready_for_review]
pull_request_review:
types: [submitted, dismissed]
# No concurrency group — intentionally omitted.
# Previous attempt (#3554, reverted #3599) used cancel-in-progress which
# killed in-progress PR runs before they produced useful results. Race
# between synchronize and pull_request_review can produce transient
# label flapping in rare cases; maintainers can correct manually.
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
triage:
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
issues: write # PR labels live on the underlying issue resource
pull-requests: read # listFiles for path-derived topic labels
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Apply triage labels
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
// Keep CODEOWNERS list in sync with .github/CODEOWNERS line 6.
const CODEOWNERS = ['LearningCircuit', 'hashedviking', 'djpetti'];
// Known AI-bot accounts that don't carry the [bot] suffix.
const KNOWN_BOTS = ['moltenbot000', 'mseep-ai', 'Nexus-Digital-Automations'];
const pr = context.payload.pull_request;
const issueNumber = pr.number;
const author = pr.user.login;
const association = pr.author_association;
const isBot = author.endsWith('[bot]') || KNOWN_BOTS.includes(author);
const isInternal = ['OWNER', 'MEMBER', 'COLLABORATOR'].includes(association);
// 403 on label calls is the expected outcome for fork PRs —
// pull_request gives forks a read-only token by design. We
// log and continue so the workflow run stays green and a
// maintainer can apply labels manually instead of seeing
// a red check on every fork contribution.
const isReadOnlyTokenError = (err) => {
if (err.status !== 403) return false;
console.log(`Label call returned 403 (read-only token, likely a fork PR). Skipping.`);
return true;
};
const addLabels = async (labels) => {
if (!labels.length) return;
try {
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
labels,
});
} catch (err) {
if (!isReadOnlyTokenError(err)) throw err;
}
};
const removeLabel = async (name) => {
try {
await github.rest.issues.removeLabel({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
name,
});
} catch (err) {
if (err.status === 404) return; // label wasn't applied
if (!isReadOnlyTokenError(err)) throw err;
}
};
// Path-derived topic labels for release-notes categorization.
// Mechanical mappings only — a rule belongs here iff the label
// follows from *which files* changed, never from what the
// change means. First-match-per-rule over all changed files.
const TOPIC_RULES = [
[/^\.github\/workflows\//, 'github-actions'],
[/^(\.github\/|\.pre-commit-hooks\/|\.pre-commit-config\.yaml$|scripts\/ci\/)/, 'ci-cd'],
[/^(Dockerfile|docker-compose|cookiecutter-docker\/|unraid-templates\/)/, 'docker'],
[/^(docs\/|[^/]+\.md$)/, 'documentation'],
[/^tests\//, 'tests'],
[/^src\/local_deep_research\/database\//, 'database'],
[/^src\/local_deep_research\/(defaults|settings)\//, 'configuration'],
[/^(pyproject\.toml|pdm\.lock|package\.json|package-lock\.json)$/, 'dependencies'],
[/^src\/local_deep_research\/(advanced_search_system|web_search_engines)\//, 'research'],
[/^src\/.*\.py$/, 'python'],
[/^src\/.*\.(js|mjs)$/, 'javascript'],
[/^src\/.*\.css$/, 'css'],
];
const topicLabels = async () => {
const files = await github.paginate(github.rest.pulls.listFiles, {
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: issueNumber,
per_page: 100,
});
const labels = new Set();
for (const f of files) {
for (const [pattern, label] of TOPIC_RULES) {
if (pattern.test(f.filename)) labels.add(label);
}
}
return [...labels];
};
const action = context.payload.action;
const eventName = context.eventName;
if (eventName === 'pull_request' && action === 'opened') {
const labels = [];
if (isBot) labels.push('bot');
if (!isInternal && !isBot) labels.push('external-contributor');
if (association === 'FIRST_TIME_CONTRIBUTOR') labels.push('first-time-contributor');
if (!isInternal) labels.push('needs-codeowner-review');
labels.push(...await topicLabels());
await addLabels(labels);
return;
}
if (eventName === 'pull_request' && (action === 'synchronize' || action === 'ready_for_review')) {
const current = pr.labels.map((l) => l.name);
if (current.includes('awaiting-author')) {
await removeLabel('awaiting-author');
await addLabels(['awaiting-codeowner']);
}
// Draft → ready is a deliberate "the diff is final" moment, so
// recompute topic labels once there. Deliberately NOT done on
// synchronize: that would re-add labels a maintainer removed,
// on every push.
if (action === 'ready_for_review') {
await addLabels(await topicLabels());
}
return;
}
if (eventName === 'pull_request_review' && action === 'submitted') {
const review = context.payload.review;
const reviewer = review.user.login;
// Strict CODEOWNERS-only check. The hardcoded list above must
// mirror .github/CODEOWNERS line 6. We deliberately do NOT
// accept any OWNER/MEMBER/COLLABORATOR association as a
// codeowner: branch protection here may later adopt
// require_code_owner_reviews=true, at which point clearing
// needs-codeowner-review on a non-codeowner MEMBER review
// would be a security-relevant mislabel.
if (!CODEOWNERS.includes(reviewer)) return;
if (review.state === 'approved') {
await removeLabel('needs-codeowner-review');
await removeLabel('awaiting-codeowner');
// Also clear awaiting-author: a codeowner can resolve
// their own changes_requested review by approving without
// an intervening synchronize (e.g., the author convinced
// them via comments).
await removeLabel('awaiting-author');
} else if (review.state === 'changes_requested') {
await removeLabel('needs-codeowner-review');
await removeLabel('awaiting-codeowner');
await addLabels(['awaiting-author']);
}
// 'commented' → no-op.
return;
}
if (eventName === 'pull_request_review' && action === 'dismissed') {
// GitHub sets review.state to "dismissed" on this event — the
// original state is not preserved (github/docs#20216). Use the
// awaiting-author label as the discriminator: it's only set by
// a codeowner's changes_requested review, so its presence is
// proof the dismissal is the one we care about. Dismissals of
// approval/comment reviews are naturally no-ops because
// awaiting-author won't be present.
const review = context.payload.review;
if (!CODEOWNERS.includes(review.user.login)) return;
const current = pr.labels.map((l) => l.name);
if (current.includes('awaiting-author')) {
await removeLabel('awaiting-author');
await addLabels(['needs-codeowner-review']);
}
return;
}
+121
View File
@@ -0,0 +1,121 @@
name: Pre-commit Checks
on:
pull_request:
branches: [ main ]
workflow_call: # Called by ci-gate.yml for release pipeline
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions:
contents: read
jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
# Full history so the author-identity hook can resolve the PR commit
# range (merge-base..head) from local objects WITHOUT fetching: a hook
# that mutates git state mid-run trips pre-commit's "files were
# modified by this hook".
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
cache: 'pip'
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
cache: true
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
- name: Install Node.js dependencies
run: npm ci
- name: Install system dependencies for SQLCipher
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
# Replicate pre-commit/action@v3.0.1 manually so the network-flaky hook
# *download* can be retried WITHOUT retrying the actual lint check.
#
# pre-commit lazily downloads hook environments from external sources
# (PyPI for ruff, GitHub release binaries for shellcheck, etc.), and a
# single HTTP 5xx from any one of them fails the whole job. See run #2524
# where `Building wheel for shellcheck_py` hit HTTP 502 fetching the
# shellcheck binary during the wheel build. A second attempt benefits from
# the partially-populated cache and almost always succeeds; two attempts is
# enough (a hook env that fails to install twice in a row is not a
# transient outage).
#
# That download is done up-front by `pre-commit install-hooks`, so only
# THAT step is wrapped in nick-fields/retry. The lint check itself runs
# exactly once: an auto-fixing hook (ruff-format, …) that modifies a file
# makes `pre-commit run` exit non-zero, and that MUST fail the job.
# Retrying the run would mask it — the second pass sees the already-fixed
# tree and exits 0, so unformatted code would sail through with a green
# check.
- name: Compute pre-commit cache key
run: echo "PY=$(python -VV | sha256sum | cut -d' ' -f1)" >> "$GITHUB_ENV"
- name: Cache pre-commit hook environments
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/pre-commit
key: pre-commit|${{ env.PY }}|${{ hashFiles('.pre-commit-config.yaml') }}
- name: Install pre-commit
# Version-pinned (matches pyproject.toml's `pre-commit~=4.5`, lock at
# 4.6.0) for reproducibility, consistent with every other CI tool
# install (checkov, semgrep, pdm, towncrier, …) which all pin `==`.
# NOTE: this is *not* hash-pinned, so Scorecard's Pinned-Dependencies
# check still flags it ("pipCommand not pinned by hash"). That is an
# accepted risk — this is a read-only lint job (contents: read,
# persist-credentials: false), and --require-hashes would mean pinning
# all transitive deps and regenerating on every (automated) bump.
# Tracked as dismissed Scorecard alert #7777 ("won't fix").
run: python -m pip install pre-commit==4.6.0
# Retry ONLY the hook-environment download (the network-flaky step).
- name: Install pre-commit hook environments
uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4.0.0
with:
timeout_minutes: 15
max_attempts: 2
retry_on: error
shell: bash
command: pre-commit install-hooks
# Run the checks exactly once — NO retry. An auto-fixing hook that
# modifies a file must fail the job, not be masked by a clean re-run.
# timeout-minutes restores the 15-minute ceiling the retry wrapper used to
# put on this step (a hung hook is otherwise bounded only by GitHub's
# 6-hour job default); install-hooks keeps its own retry timeout above.
- name: Run pre-commit
timeout-minutes: 15
run: pre-commit run --show-diff-on-failure --color=always --all-files
+512
View File
@@ -0,0 +1,512 @@
name: Prerelease Docker Image
# Build the canonical Docker image for a release. In the build-once-promote
# pipeline, this workflow IS the build — docker-publish.yml only retags the
# manifest produced here. Cosign signing, SBOM attestation, and SLSA
# provenance are attached here once, keyed by manifest digest, so they're
# discoverable from any tag (including the release tags later created by
# imagetools create).
#
# Triggered exclusively via workflow_call from release.yml (after security
# gates pass). No workflow_dispatch — security and gate semantics are
# enforced by the caller. The build runs automatically; the only human
# approval in the release flow is the `release` env on this workflow's
# jobs + publish-docker + trigger-pypi + create-release in release.yml
# (gates the actual publish, not the canonical build).
on:
workflow_call:
inputs:
version:
description: "Bare semver, e.g. '1.6.9' (no leading 'v')"
type: string
required: true
short_sha:
description: "First 7 chars of commit SHA (used in the prerelease tag)"
type: string
required: true
secrets:
# Explicit secrets contract instead of `secrets: inherit` on the
# caller side — narrower blast radius if a future caller misuses
# this reusable workflow.
DOCKER_USERNAME:
required: true
description: "Docker Hub username for image push"
DOCKER_PASSWORD:
required: true
description: "Docker Hub PAT (Read+Write+Delete scopes)"
outputs:
manifest_digest:
description: "sha256:... digest of the multi-arch prerelease manifest. Used by docker-publish.yml to verify retag preserves the digest."
value: ${{ jobs.create-manifest.outputs.digest }}
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
# No approval gate at the build step — the build runs automatically once
# security gates and CI gates in release.yml pass. The only meaningful
# human decision in the release flow is "should this signed, attested,
# tested image become the official release?" — gated by the `release`
# environment on this workflow's jobs + `publish-docker` + `trigger-pypi`
# + `create-release` in release.yml. The maintainer can pull
# `:prerelease-v<ver>-<sha>` and smoke-test between build completion
# and approving the release env.
jobs:
build-amd64:
name: Build AMD64 Prerelease Image
runs-on: ubuntu-latest
environment: release
permissions:
contents: read
steps:
- name: Harden Runner
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Check out the repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Log in to Docker Hub
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and push AMD64 image
id: build
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
platforms: linux/amd64
push: true
tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-amd64
cache-from: type=gha,scope=linux-amd64
cache-to: type=gha,mode=max,scope=linux-amd64
build-arm64:
name: Build ARM64 Prerelease Image
runs-on: ubuntu-24.04-arm
environment: release
permissions:
contents: read
steps:
- name: Harden Runner
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Check out the repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Log in to Docker Hub
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and push ARM64 image
id: build
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
platforms: linux/arm64
push: true
tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-arm64
cache-from: type=gha,scope=linux-arm64
cache-to: type=gha,mode=max,scope=linux-arm64
security-scan:
name: Security Scan
runs-on: ubuntu-latest
environment: release
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Check out the repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet || true
sudo rm -rf /usr/local/lib/android || true
sudo rm -rf /opt/ghc || true
sudo rm -rf /opt/hostedtoolcache/CodeQL || true
sudo docker image prune --all --force || true
df -h
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Build Docker image for security scan
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: .
platforms: linux/amd64
push: false
load: true
tags: local-deep-research:security-scan
cache-from: type=gha,scope=linux-amd64
cache-to: type=gha,mode=max,scope=linux-amd64
# Generate Trivy SARIF for archival as a workflow artifact (all severities, never fails).
# Severity-gating happens in the next step.
- name: Generate Trivy SARIF report
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
with:
image-ref: local-deep-research:security-scan
format: 'sarif'
output: 'trivy-prerelease-scan.sarif'
ignore-unfixed: true
exit-code: '0'
version: 'v0.69.2'
# Separate scan that fails build only on fixable HIGH/CRITICAL vulnerabilities
- name: Check for fixable HIGH/CRITICAL vulnerabilities
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
with:
image-ref: local-deep-research:security-scan
severity: 'CRITICAL,HIGH'
ignore-unfixed: true
trivyignores: '.trivyignore'
exit-code: '1'
version: 'v0.69.2'
- name: Upload Trivy scan results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: trivy-prerelease-scan
path: trivy-prerelease-scan.sarif
retention-days: 7
create-manifest:
name: Create Multi-Platform Prerelease Manifest
needs: [build-amd64, build-arm64, security-scan]
runs-on: ubuntu-latest
environment: release
permissions:
contents: read
id-token: write # Required for cosign keyless OIDC signing
# No `packages: write` — Docker Hub auth uses DOCKER_PASSWORD secret,
# not GITHUB_TOKEN. `packages: write` only matters for ghcr.io pushes.
outputs:
digest: ${{ steps.capture-digest.outputs.digest }}
steps:
- name: Harden Runner
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Check out the repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Log in to Docker Hub
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Install Cosign
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
with:
# Pin to cosign v2.x — see release.yml for rationale (v3 enables
# --new-bundle-format by default which changes the on-wire format
# and breaks downstream verifiers still on v2).
cosign-release: 'v2.6.3'
- name: Install Syft
uses: anchore/sbom-action/download-syft@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
- name: Create and push multi-platform manifest
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
VERSION: ${{ inputs.version }}
SHORT_SHA: ${{ inputs.short_sha }}
run: |
set -euo pipefail
TAG="prerelease-v${VERSION}-${SHORT_SHA}"
echo "Creating manifest for: ${DOCKER_USERNAME}/local-deep-research:${TAG}"
docker buildx imagetools create -t "${DOCKER_USERNAME}/local-deep-research:${TAG}" \
"${DOCKER_USERNAME}/local-deep-research:${TAG}-amd64" \
"${DOCKER_USERNAME}/local-deep-research:${TAG}-arm64"
echo "Manifest created successfully"
# Floating tag: re-point :prerelease at the manifest just created so
# testers can pin compose to `:prerelease` and pull the latest RC via
# `docker compose pull` without editing the tag each cycle. The
# versioned tag above remains for reproducibility (and is what
# docker-publish.yml retags by digest into :1.6.9 / :1.6 / :latest).
echo "Updating floating tag: ${DOCKER_USERNAME}/local-deep-research:prerelease"
docker buildx imagetools create -t "${DOCKER_USERNAME}/local-deep-research:prerelease" \
"${DOCKER_USERNAME}/local-deep-research:${TAG}"
echo "Floating :prerelease tag updated"
- name: Capture manifest digest
id: capture-digest
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
VERSION: ${{ inputs.version }}
SHORT_SHA: ${{ inputs.short_sha }}
run: |
set -euo pipefail
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research:prerelease-v${VERSION}-${SHORT_SHA}"
# Same form as the existing docker-publish.yml inspector — avoids jq.
DIGEST=$(docker buildx imagetools inspect "$IMAGE_REF" --format '{{json .Manifest.Digest}}' | tr -d '"')
if [[ -z "$DIGEST" || "$DIGEST" != sha256:* ]]; then
echo "::error::Failed to capture manifest digest (got '${DIGEST}')"
exit 1
fi
echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"
echo "Manifest digest: ${DIGEST}"
- name: Sign manifest with Cosign
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DIGEST: ${{ steps.capture-digest.outputs.digest }}
run: |
set -euo pipefail
# Sign by digest — signature artifact lands at sha256-<digest>.sig
# in the same repo, discoverable from ANY tag pointing at the same
# digest (including release tags created later by docker-publish.yml's
# imagetools-create retag).
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
echo "Signing image by digest: $IMAGE_REF"
cosign sign --yes "$IMAGE_REF"
# Brief sleep to allow registry to propagate signature
sleep 5
- name: Generate SLSA provenance attestation
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DIGEST: ${{ steps.capture-digest.outputs.digest }}
run: |
set -euo pipefail
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
# entryPoint is the TOP-LEVEL caller (release.yml), not this
# reusable workflow. Per SLSA GHA buildtype v1 and the canonical
# slsa-github-generator, reusable workflows are explicitly NOT
# entryPoints. github.run_id / github.repository / github.sha all
# resolve to the caller's run context inside a reusable workflow.
# builder.id pins the workflow that actually defines the build
# steps — the trust root a verifier policy can pin against. We
# compose it from `github.repository` and a hardcoded path to
# THIS workflow file, with `github.ref` for the ref portion.
# Rationale: inside a workflow_call callee, the `github` context
# is scoped to the CALLER, so `github.workflow_ref` would point
# at release.yml (the wrong builder). The `job` context has no
# `workflow_ref` property either (only check_run_id, container,
# services, status — actionlint confirms). For a local-path
# reusable workflow (`uses: ./.github/workflows/...`), the
# callee's ref equals the caller's `github.ref`, so composing
# the path manually gives the correct
# `<owner>/<repo>/.github/workflows/prerelease-docker.yml@<ref>`
# format that matches the Fulcio cert SAN. Cosign and
# slsa-verifier both anchor on the cert anyway, so this fix is
# about correctness for raw-JSON policy engines / audit tools
# that read builder.id directly.
#
# completeness.* are FALSE because we don't capture invocation
# parameters or environment, and the build does network I/O for
# apt/pip/npm. Honest emptiness > false claims of completeness.
#
# buildInvocationId includes run_attempt so re-runs are
# distinguishable in audit logs.
#
# Uses auto-injected GITHUB_* shell env vars (GITHUB_REPOSITORY,
# GITHUB_REF, GITHUB_SHA, GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT) instead
# of github-context template expansion to satisfy zizmor's
# template-injection check. The values are semantically identical
# inside a workflow_call callee (both scope to the caller's context),
# but shell-var expansion happens at runtime in a confined string
# context, eliminating any theoretical injection surface from a
# ref/branch name containing shell metacharacters.
cat > provenance.json <<EOF
{
"buildType": "https://github.com/${GITHUB_REPOSITORY}/docker-build@v1",
"builder": {
"id": "https://github.com/${GITHUB_REPOSITORY}/.github/workflows/prerelease-docker.yml@${GITHUB_REF}"
},
"invocation": {
"configSource": {
"uri": "https://github.com/${GITHUB_REPOSITORY}",
"digest": {
"sha1": "${GITHUB_SHA}"
},
"entryPoint": ".github/workflows/release.yml"
}
},
"metadata": {
"buildInvocationId": "${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}",
"buildStartedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"completeness": {
"parameters": false,
"environment": false,
"materials": false
},
"reproducible": false
},
"materials": [
{
"uri": "https://github.com/${GITHUB_REPOSITORY}",
"digest": {
"sha1": "${GITHUB_SHA}"
}
}
]
}
EOF
# --replace prevents duplicate SLSA attestations on re-run. Cosign's
# Replace logic is keyed by predicate-type URI, so it leaves the
# SBOM SPDX attestation (different predicateType) untouched.
cosign attest --yes --replace --predicate provenance.json --type slsaprovenance "$IMAGE_REF"
- name: Verify image signature
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DIGEST: ${{ steps.capture-digest.outputs.digest }}
run: |
set -euo pipefail
IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"
echo "Verifying signature for: $IMAGE_REF"
# Retry to handle registry propagation delay after signing
MAX_RETRIES=5
RETRY_DELAY=10
for i in $(seq 1 "$MAX_RETRIES"); do
echo "Verification attempt $i of $MAX_RETRIES..."
# Use GITHUB_REPOSITORY shell env var (auto-injected) instead of
# github.repository template expansion to avoid zizmor's
# template-injection finding.
if cosign verify \
--certificate-identity-regexp="https://github.com/${GITHUB_REPOSITORY}" \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com \
"$IMAGE_REF"; then
echo "Signature verification successful!"
exit 0
fi
if [ "$i" -lt "$MAX_RETRIES" ]; then
echo "Verification failed, waiting ${RETRY_DELAY}s before retry..."
sleep "$RETRY_DELAY"
fi
done
echo "::error::Signature verification failed after $MAX_RETRIES attempts"
exit 1
- name: Generate per-platform SBOMs and attest each
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DIGEST: ${{ steps.capture-digest.outputs.digest }}
run: |
set -euo pipefail
REPO="${DOCKER_USERNAME}/local-deep-research"
MANIFEST_REF="${REPO}@${DIGEST}"
# Multi-arch SBOM correctness: syft against a manifest list digest
# only scans the host platform's layers (per anchore/syft#1708),
# which would lie to ARM64 consumers. We attest each per-arch
# digest with its OWN SBOM so end-user verification is honest.
# We deliberately do NOT also produce a "manifest-level SBOM" —
# that would be amd64-only (host arch) and re-introduce the lie
# for any arm64 consumer running the README verifier recipe.
# The README documents the per-arch verification flow instead.
MANIFEST_JSON=$(docker buildx imagetools inspect "${MANIFEST_REF}" --raw)
# Defense against future buildx output changes: assert at least
# one per-arch entry exists. Without this, an empty/malformed
# manifest list would silently produce zero SBOMs and pass CI green.
PER_ARCH_COUNT=$(echo "${MANIFEST_JSON}" \
| jq '[.manifests[] | select(.platform.architecture != "unknown")] | length')
if [[ "${PER_ARCH_COUNT}" -lt 1 ]]; then
echo "::error::No per-arch manifest entries found in ${MANIFEST_REF} — SBOM generation cannot proceed"
echo "Raw manifest: ${MANIFEST_JSON}"
exit 1
fi
echo "Found ${PER_ARCH_COUNT} per-arch manifest(s) to scan"
echo "$MANIFEST_JSON" \
| jq -r '.manifests[] | select(.platform.architecture != "unknown") | "\(.platform.os)/\(.platform.architecture)\t\(.digest)"' \
| while IFS=$'\t' read -r PLAT PER_ARCH_DIGEST; do
ARCH="${PLAT##*/}"
PER_ARCH_REF="${REPO}@${PER_ARCH_DIGEST}"
SBOM_FILE="sbom-${ARCH}.spdx.json"
echo "=== Scanning ${PLAT} (${PER_ARCH_DIGEST}) ==="
# --platform tells syft which arch to scan — matters when
# the host runner can't natively execute the image.
syft --platform "${PLAT}" "${PER_ARCH_REF}" -o spdx-json > "${SBOM_FILE}"
# --replace prevents accumulation when a re-run lands on
# the same digest (e.g. "Re-run failed jobs" after a flake).
# Per cosign source pkg/cosign/remote/remote.go, --replace
# is per-predicate-type, so it doesn't disturb the SLSA
# attestation already on the manifest list digest.
cosign attest --yes --replace \
--predicate "${SBOM_FILE}" --type spdxjson "${PER_ARCH_REF}"
done
- name: Upload SBOMs artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: sbom
# Per-arch SBOMs only — `sbom-amd64.spdx.json`, `sbom-arm64.spdx.json`,
# one per platform in the manifest list. No manifest-level SBOM is
# produced (would be host-arch-only and misleading for non-amd64
# consumers).
path: sbom-*.spdx.json
retention-days: 90
- name: Summary
env:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
VERSION: ${{ inputs.version }}
SHORT_SHA: ${{ inputs.short_sha }}
DIGEST: ${{ steps.capture-digest.outputs.digest }}
run: |
TAG="prerelease-v${VERSION}-${SHORT_SHA}"
{
echo "## Prerelease Docker Image"
echo ""
echo "**Versioned tag:** \`${TAG}\`"
echo "**Floating tag:** \`prerelease\` (now points at this build)"
echo ""
echo "**Digest:** \`${DIGEST}\`"
echo ""
echo '```'
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${TAG}"
echo "docker pull ${DOCKER_USERNAME}/local-deep-research:prerelease"
echo '```'
echo ""
echo "Signed and attested. After release approval, docker-publish.yml"
echo "will retag this exact digest as \`:${VERSION}\`, \`:major.minor\`,"
echo "and \`:latest\`."
} >> "$GITHUB_STEP_SUMMARY"
+625
View File
@@ -0,0 +1,625 @@
name: Publish to PyPI
# SECURITY: Only triggered via repository_dispatch from release.yml
# (after security gate passes). We intentionally do NOT support
# workflow_dispatch because that would bypass security checks.
# To re-publish, trigger a new release through release.yml.
#
# REPRODUCIBILITY: repository_dispatch runs on default-branch HEAD at
# dispatch time — NOT the release commit. The `release` env approval is
# human-paced, so main can move between the release commit and the
# dispatch (on v1.9.0 it was 4 PRs ahead, and PyPI shipped code absent
# from the tag/Docker image). Both build jobs therefore check out
# `client_payload.sha` — the exact commit every gate ran against — and
# fail closed if the dispatcher didn't pin it.
on:
repository_dispatch:
types: [publish-pypi]
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
# Build frontend if package.json exists (isolated, no secrets)
build-frontend:
runs-on: ubuntu-latest
outputs:
has-frontend: ${{ steps.check.outputs.has-frontend }}
container:
# Node 24 to match `package.json`'s `engines: { node: ">=24.0.0" }`.
# Was previously node:20 — npm could resolve dependencies that target
# APIs missing on 20 and the wheel-building publish path could ship
# frontend assets that break at runtime on the Node-24 Docker image.
image: node:24-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f # node:24-alpine
# Note: Network is needed for npm ci to work, but no secrets are available
options: --user 1001
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Validate pinned release commit
# POSIX sh (this job's container is Alpine — no bash). Fail closed:
# without a full 40-hex SHA, checkout would silently fall back to
# default-branch HEAD and reintroduce the drift this pin prevents.
# The length test matters: grep -E matches PER LINE, so a crafted
# multiline value ("<40-hex>\n<junk>") would pass the regex alone;
# ${#VAR} counts bytes in POSIX sh, so -eq 40 also excludes any
# embedded newline. Ancestry (sha actually on main) is enforced in
# build-package — this container has no python3/gh; this job holds
# no secrets and its output only reaches PyPI through build-package,
# which checks out the same validated sha.
env:
RELEASE_SHA: ${{ github.event.client_payload.sha }}
run: |
if [ "${#RELEASE_SHA}" -ne 40 ] || ! printf '%s' "$RELEASE_SHA" | grep -qE '^[0-9a-f]{40}$'; then
echo "::error::client_payload.sha is missing or not a full commit SHA — refusing to build from default-branch HEAD. The dispatcher (release.yml trigger-pypi) must pin the release commit."
exit 1
fi
echo "Building from pinned release commit $RELEASE_SHA"
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.event.client_payload.sha }}
persist-credentials: false
- name: Check for frontend assets
id: check
run: |
if [ -f "package.json" ]; then
echo "has-frontend=true" >> "$GITHUB_OUTPUT"
echo "Found package.json - will build frontend"
else
echo "has-frontend=false" >> "$GITHUB_OUTPUT"
echo "ERROR: No package.json found - frontend build is required for PyPI releases"
exit 1
fi
- name: Build frontend
if: steps.check.outputs.has-frontend == 'true'
run: |
echo "=== Frontend Build Process Starting ==="
echo "Current directory: $(pwd)"
echo "Node version: $(node --version)"
echo "NPM version: $(npm --version)"
# Install dependencies from root package.json
echo ""
echo "📦 Installing dependencies..."
# Use npm ci for reproducible builds from lockfile
npm ci --ignore-scripts --no-audit --no-fund
echo "✅ Dependencies installed"
# Show pre-build directory structure
echo ""
echo "📁 Pre-build directory structure:"
echo "Contents of src/local_deep_research/web/static/:"
find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "Directory not found"
# Build with Vite (outputs to src/local_deep_research/web/static/dist)
echo ""
echo "🔨 Running Vite build..."
npm run build
echo "✅ Vite build completed"
# Show post-build directory structure
echo ""
echo "📁 Post-build directory structure:"
echo "Contents of src/local_deep_research/web/static/:"
find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
echo ""
echo "Contents of src/local_deep_research/web/static/dist/:"
find src/local_deep_research/web/static/dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
echo ""
echo "Looking for manifest.json:"
find src/local_deep_research/web/static/dist -name "manifest.json" -type f 2>/dev/null || echo "No manifest.json found"
# Verify the build completed successfully
if [ ! -f "src/local_deep_research/web/static/dist/.vite/manifest.json" ]; then
echo "❌ ERROR: Build failed - manifest.json not created at expected location"
echo "Expected location: src/local_deep_research/web/static/dist/.vite/manifest.json"
echo "Actual dist contents:"
find src/local_deep_research/web/static/dist -type f | head -20
exit 1
fi
echo "✅ Found manifest.json at: src/local_deep_research/web/static/dist/.vite/manifest.json"
echo "Manifest contents (first 10 lines):"
head -10 src/local_deep_research/web/static/dist/.vite/manifest.json
# Create build marker
echo ""
echo "📝 Creating build marker..."
echo "{\"status\":\"complete\",\"built\":\"$(date -Iseconds)\"}" > src/local_deep_research/web/static/.frontend-built
echo "✅ Build marker created"
- name: Upload frontend artifacts
if: steps.check.outputs.has-frontend == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: frontend-assets
path: src/local_deep_research/web/static/
retention-days: 1
include-hidden-files: true # Ensure .vite directory is included
# Build Python package (isolated, no PyPI access)
build-package:
needs: build-frontend
runs-on: ubuntu-latest
container:
image: python:3.12-slim@sha256:971f04b358cf483ec445a8d388fb55267451f080d90fb136c8e69684a02a9604 # python:3.12-slim
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Validate pinned release commit
# Same fail-closed shape guard as build-frontend (length test: see
# comment there), plus the ANCESTRY check that only this job can
# run (python3 in the container): the pinned sha must be reachable
# from main. Anyone able to send repository_dispatch (repo write /
# the PAT holder) chooses the payload — without this check they
# could point the build at ANY commit object in the repo network
# (scratch branches, fork-PR heads) that never passed review or the
# release gates. release.yml's own build job verifies its sha is on
# main, but a forged dispatch bypasses release.yml entirely; the
# `release` environment approval on the publish job is then the
# only barrier, and an approver has no reason to suspect a
# legitimate-looking pending deployment. Enforced here (not in
# build-frontend) because every byte that reaches PyPI flows
# through this job's checkout.
env:
RELEASE_SHA: ${{ github.event.client_payload.sha }}
GH_TOKEN: ${{ github.token }}
run: |
if [ "${#RELEASE_SHA}" -ne 40 ] || ! printf '%s' "$RELEASE_SHA" | grep -qE '^[0-9a-f]{40}$'; then
echo "::error::client_payload.sha is missing or not a full commit SHA — refusing to build from default-branch HEAD. The dispatcher (release.yml trigger-pypi) must pin the release commit."
exit 1
fi
python3 - <<'PYEOF'
import json
import os
import sys
import urllib.request
sha = os.environ["RELEASE_SHA"]
repo = os.environ["GITHUB_REPOSITORY"]
req = urllib.request.Request(
f"https://api.github.com/repos/{repo}/compare/{sha}...heads/main",
headers={
"Authorization": f"Bearer {os.environ['GH_TOKEN']}",
"Accept": "application/vnd.github+json",
},
)
try:
with urllib.request.urlopen(req) as resp:
status = json.load(resp)["status"]
except urllib.error.HTTPError as exc:
# 404 = sha unknown to the repo network. Fail closed either way.
print(f"::error::compare API rejected sha {sha}: HTTP {exc.code}")
sys.exit(1)
# base=sha, head=main: main being "ahead" of (or "identical" to)
# the sha means the sha is an ancestor of main. "diverged" or
# "behind" means it is not.
if status not in ("ahead", "identical"):
print(
f"::error::pinned sha {sha} is not an ancestor of main "
f"(compare status: {status}) — refusing to build unreviewed code"
)
sys.exit(1)
print(f"Pinned sha is on main (compare status: {status})")
PYEOF
echo "Building from pinned release commit $RELEASE_SHA"
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.event.client_payload.sha }}
persist-credentials: false
- name: Install build dependencies
run: |
apt-get update && apt-get install -y libsqlcipher-dev build-essential
pip install pdm==2.26.2
- name: Download frontend artifacts
if: needs.build-frontend.outputs.has-frontend == 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
continue-on-error: true
with:
name: frontend-assets
path: src/local_deep_research/web/static/
- name: Verify frontend build artifacts
run: |
echo "=== Frontend Artifacts Verification ==="
echo "Current directory: $(pwd)"
echo ""
echo "📁 Checking downloaded artifacts structure:"
echo "Contents of src/:"
find src/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/ directory not found"
echo ""
echo "Contents of src/local_deep_research/:"
find src/local_deep_research/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/ not found"
echo ""
echo "Contents of src/local_deep_research/web/:"
find src/local_deep_research/web/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/web/ not found"
echo ""
echo "Contents of src/local_deep_research/web/static/:"
find src/local_deep_research/web/static/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null || echo "src/local_deep_research/web/static/ not found"
# Check if dist directory exists
echo ""
echo "🔍 Checking for dist directory..."
if [ ! -d "src/local_deep_research/web/static/dist" ]; then
echo "❌ ERROR: Frontend dist directory not found!"
echo "Expected location: src/local_deep_research/web/static/dist"
echo "Actual structure of static directory:"
find src/local_deep_research/web/static -type d 2>/dev/null | head -20
echo "The frontend build artifacts are missing."
exit 1
fi
echo "✅ Found dist directory at: src/local_deep_research/web/static/dist"
# Show dist directory structure
echo ""
echo "📁 Dist directory structure:"
echo "Contents of dist/:"
find src/local_deep_research/web/static/dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
echo ""
echo "All files in dist (recursive):"
find src/local_deep_research/web/static/dist -type f | head -20
# Check for critical files
echo ""
echo "🔍 Checking for manifest.json..."
# Check both possible locations for manifest.json
MANIFEST_PATH=""
if [ -f "src/local_deep_research/web/static/dist/.vite/manifest.json" ]; then
MANIFEST_PATH="src/local_deep_research/web/static/dist/.vite/manifest.json"
elif [ -f "src/local_deep_research/web/static/dist/manifest.json" ]; then
MANIFEST_PATH="src/local_deep_research/web/static/dist/manifest.json"
fi
if [ -z "$MANIFEST_PATH" ]; then
echo "❌ ERROR: Vite manifest.json not found!"
echo "Checked locations:"
echo " - src/local_deep_research/web/static/dist/.vite/manifest.json"
echo " - src/local_deep_research/web/static/dist/manifest.json"
echo "Looking for any manifest.json files:"
find src/local_deep_research/web/static -name "manifest.json" -type f 2>/dev/null || echo "None found"
echo "Checking if .vite directory exists:"
find src/local_deep_research/web/static/dist/ -maxdepth 1 -name ".*" -type d || echo "No hidden directories found"
echo ""
echo "This is likely due to artifact transfer losing the hidden .vite directory."
echo "The include-hidden-files option should fix this."
echo "The frontend build appears to be incomplete - cannot continue."
exit 1
fi
echo "✅ Found manifest.json at: $MANIFEST_PATH"
echo "Manifest size: $(wc -c < "$MANIFEST_PATH") bytes"
# Check for JS files
echo ""
echo "🔍 Checking for JavaScript files..."
JS_COUNT=$(find src/local_deep_research/web/static/dist -name "*.js" -type f | wc -l)
if [ "$JS_COUNT" -eq 0 ]; then
echo "❌ ERROR: No JavaScript files found!"
echo "Expected JS files in dist/"
echo "The frontend build appears to be incomplete."
exit 1
fi
echo "✅ Found $JS_COUNT JavaScript file(s)"
echo "JS files:"
find src/local_deep_research/web/static/dist -name "*.js" -type f | head -5
# Check for CSS files
echo ""
echo "🔍 Checking for CSS files..."
CSS_COUNT=$(find src/local_deep_research/web/static/dist -name "*.css" -type f | wc -l)
if [ "$CSS_COUNT" -eq 0 ]; then
echo "❌ ERROR: No CSS files found!"
echo "Expected CSS files in dist/"
echo "The frontend build appears to be incomplete."
exit 1
fi
echo "✅ Found $CSS_COUNT CSS file(s)"
echo "CSS files:"
find src/local_deep_research/web/static/dist -name "*.css" -type f | head -5
# Check for build marker
echo ""
echo "🔍 Checking for build marker..."
if [ ! -f "src/local_deep_research/web/static/.frontend-built" ]; then
echo "⚠️ WARNING: Frontend build marker not found (non-critical)"
else
echo "✅ Found build marker"
echo "Build marker contents:"
cat src/local_deep_research/web/static/.frontend-built
fi
echo ""
echo "=== ✅ Frontend build artifacts verified successfully ==="
echo "Summary:"
echo " - Dist directory: ✓"
echo " - Manifest.json: ✓"
echo " - JS files: $JS_COUNT file(s)"
echo " - CSS files: $CSS_COUNT file(s)"
- name: Build Python package
run: |
echo "=== Python Package Build ==="
echo "Current directory: $(pwd)"
echo ""
echo "📦 Building Python package with PDM..."
pdm build
echo ""
echo "✅ Package build completed"
echo ""
echo "📁 Package contents:"
find dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
echo ""
echo "Package sizes:"
du -h dist/*
- name: Upload package
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: python-dist
path: dist/
retention-days: 1
# Publish (ONLY job with PyPI access)
publish:
needs: build-package
runs-on: ubuntu-latest
environment: release
permissions:
id-token: write
attestations: write # Required for generating attestations
steps:
- name: Harden Runner
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Download package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: python-dist
path: dist/
- name: List packages
run: |
echo "=== Downloaded Package Artifacts ==="
echo "📦 Packages in dist/:"
ls -lh dist/
echo ""
echo "Package details:"
for file in dist/*; do
echo " - $(basename "$file"): $(du -h "$file" | cut -f1)"
done
- name: Verify package contents
run: |
echo "=== Package Content Verification ==="
echo "Verifying package contains frontend assets..."
echo ""
# Install wheel to inspect package
echo "📦 Installing wheel for package inspection..."
pip install wheel==0.46.2
# Find the wheel file
WHEEL_FILE=$(find dist -name "*.whl" -type f 2>/dev/null | head -1)
TAR_FILE=$(find dist -name "*.tar.gz" -type f 2>/dev/null | head -1)
if [ -z "$WHEEL_FILE" ]; then
echo "❌ ERROR: No wheel file found in dist/"
echo "Found files:"
find dist/ -maxdepth 1 -type f -exec ls -la {} + 2>/dev/null
exit 1
fi
echo "✅ Found wheel file: $WHEEL_FILE"
echo "Wheel size: $(du -h "$WHEEL_FILE" | cut -f1)"
echo ""
if [ -n "$TAR_FILE" ]; then
echo "✅ Found source distribution: $TAR_FILE"
echo "Source dist size: $(du -h "$TAR_FILE" | cut -f1)"
fi
echo ""
echo "🔍 Inspecting wheel contents..."
# Extract and check for frontend files
python -m zipfile -l "$WHEEL_FILE" > wheel_contents.txt
TOTAL_FILES=$(wc -l < wheel_contents.txt)
echo "Total files in wheel: $TOTAL_FILES"
echo ""
# Count different file types
echo "📊 File type breakdown:"
echo " Python files: $(grep -c '\.py' wheel_contents.txt || echo 0)"
echo " JavaScript files: $(grep -c '\.js' wheel_contents.txt || echo 0)"
echo " CSS files: $(grep -c '\.css' wheel_contents.txt || echo 0)"
echo " JSON files: $(grep -c '\.json' wheel_contents.txt || echo 0)"
echo " HTML files: $(grep -c '\.html' wheel_contents.txt || echo 0)"
echo ""
# Check for critical frontend directories/files
echo "🔍 Checking for frontend assets..."
echo "Looking for dist directory..."
if ! grep -q "local_deep_research/web/static/dist/" wheel_contents.txt; then
echo "❌ ERROR: Frontend dist directory not found in package!"
echo "Package is missing frontend build artifacts."
echo "Static files found in package:"
grep "static" wheel_contents.txt | head -10 || echo "No static files found"
exit 1
fi
echo "✅ Found dist directory"
echo ""
echo "Looking for JavaScript files..."
JS_IN_WHEEL=$(grep -c "local_deep_research/web/static/dist.*\.js" wheel_contents.txt || echo 0)
if [ "$JS_IN_WHEEL" -eq 0 ]; then
echo "❌ ERROR: Frontend JS files not found in package!"
echo "JavaScript files in package:"
grep "\.js" wheel_contents.txt | head -10 || echo "No JS files found"
exit 1
fi
echo "✅ Found $JS_IN_WHEEL JavaScript file(s) in dist"
echo ""
echo "Looking for CSS files..."
CSS_IN_WHEEL=$(grep -c "local_deep_research/web/static/dist.*\.css" wheel_contents.txt || echo 0)
if [ "$CSS_IN_WHEEL" -eq 0 ]; then
echo "❌ ERROR: Frontend CSS files not found in package!"
echo "CSS files in package:"
grep "\.css" wheel_contents.txt | head -10 || echo "No CSS files found"
exit 1
fi
echo "✅ Found $CSS_IN_WHEEL CSS file(s) in dist"
echo ""
echo "Looking for manifest.json..."
if ! grep -q "local_deep_research/web/static/dist/.vite/manifest.json" wheel_contents.txt; then
echo "⚠️ WARNING: manifest.json not found in package (may be okay)"
else
echo "✅ Found manifest.json"
fi
echo ""
echo "=== ✅ Package verification successful ==="
echo "Summary:"
echo " - Wheel file: ✓"
echo " - Frontend dist: ✓"
echo " - JS files: $JS_IN_WHEEL file(s)"
echo " - CSS files: $CSS_IN_WHEEL file(s)"
echo ""
echo "Sample frontend assets in package:"
grep "local_deep_research/web/static/dist/" wheel_contents.txt | head -10
- name: Generate attestations
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
with:
subject-path: 'dist/*'
- name: Pre-publish summary
env:
EVENT_NAME: ${{ github.event_name }}
DISPATCH_TAG: ${{ github.event.client_payload.tag }}
DISPATCH_PRERELEASE: ${{ github.event.client_payload.prerelease }}
RELEASE_TAG: ${{ github.event.release.tag_name }}
RELEASE_PRERELEASE: ${{ github.event.release.prerelease }}
run: |
echo "=== 🚀 Ready to Publish ==="
echo ""
echo "📦 Package Information:"
# Handle both release event and repository_dispatch
if [ "$EVENT_NAME" = "repository_dispatch" ]; then
echo " - Trigger: repository_dispatch"
echo " - Tag: $DISPATCH_TAG"
echo " - Prerelease: $DISPATCH_PRERELEASE"
else
echo " - Trigger: release"
echo " - Tag: $RELEASE_TAG"
echo " - Prerelease: $RELEASE_PRERELEASE"
fi
echo ""
echo "📁 Package files to publish:"
ls -lh dist/
echo ""
echo "✅ All checks passed - proceeding with publication"
- name: Publish to Test PyPI
if: (github.event_name == 'release' && github.event.release.prerelease == true) || (github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease == true)
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
with:
repository-url: https://test.pypi.org/legacy/
skip-existing: true # Don't fail if version already exists
- name: Publish to PyPI
if: (github.event_name == 'release' && github.event.release.prerelease != true) || (github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease != true)
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
with:
skip-existing: true # Don't fail if version already exists
# Verify the published package is installable (skipped for prereleases which go to Test PyPI)
verify-publish:
needs: publish
if: ${{ !(github.event_name == 'release' && github.event.release.prerelease == true) && !(github.event_name == 'repository_dispatch' && github.event.client_payload.prerelease == true) }}
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Determine published version
id: version
env:
EVENT_NAME: ${{ github.event_name }}
PAYLOAD_TAG: ${{ github.event.client_payload.tag }}
RELEASE_TAG: ${{ github.event.release.tag_name }}
run: |
if [ "$EVENT_NAME" = "repository_dispatch" ]; then
TAG="$PAYLOAD_TAG"
else
TAG="$RELEASE_TAG"
fi
# Strip leading 'v' if present
VERSION="${TAG#v}"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Published version: $VERSION"
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
- name: Wait for PyPI index to update
run: |
echo "Waiting 60 seconds for PyPI index propagation..."
sleep 60
- name: Verify package installation
run: |
python3 -m venv /tmp/verify-env
. /tmp/verify-env/bin/activate
VERSION="${{ steps.version.outputs.version }}"
echo "Installing local-deep-research==${VERSION}..."
# Retry up to 5 times with increasing delay for PyPI propagation
for i in 1 2 3 4 5; do
if pip install "local-deep-research==${VERSION}" 2>&1; then
echo "Installation successful on attempt $i"
break
fi
if [ "$i" -lt 5 ]; then
echo "Attempt $i failed, waiting $((i * 30))s..."
sleep $((i * 30))
else
echo "::error::Failed to install local-deep-research==${VERSION} after 5 attempts"
exit 1
fi
done
# Verify import works
python3 -c "import local_deep_research; print(f'Successfully imported v{local_deep_research.__version__}')"
+495
View File
@@ -0,0 +1,495 @@
name: Puppeteer E2E Tests
on:
pull_request:
types: [labeled]
workflow_call: # Called by release.yml (e2e-test-gate)
secrets:
OPENROUTER_API_KEY:
required: false
SERPER_API_KEY:
required: false
workflow_dispatch: # Manual trigger runs all tests
schedule:
# Run weekly on Sunday at 2 AM UTC
- cron: '0 2 * * 0'
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions: {} # Minimal top-level for OSSF Scorecard
jobs:
puppeteer-tests:
name: Puppeteer E2E Tests
runs-on: ubuntu-latest
timeout-minutes: 120
# For pull_request events, only run when labeled test:puppeteer or test:e2e.
# For all other triggers (push via workflow_call from release.yml,
# workflow_dispatch, schedule), always run.
# NOTE: workflow_call inherits the caller's github.event_name (e.g. "push"),
# so checking for 'workflow_call' never matches — use != 'pull_request' instead.
if: |
github.event_name != 'pull_request' ||
github.event.label.name == 'test:puppeteer' ||
github.event.label.name == 'test:e2e'
environment: ci
permissions:
contents: read
pull-requests: write
issues: write
env:
LABEL_NAME: ${{ github.event.label.name || 'manual' }}
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Setup PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
- name: Install system dependencies
run: |
sudo apt-get update
# Note: libasound2t64 is the Ubuntu 24.04 name for libasound2
sudo apt-get install -y libsqlcipher-dev xvfb \
libnss3 libgbm1 libasound2t64 libatk1.0-0 libatk-bridge2.0-0 \
libcups2 libdrm2 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
fonts-liberation xdg-utils
- name: Install Python dependencies
run: pdm install
- name: Install root frontend dependencies
run: npm ci
- name: Build Vite frontend bundle
# Generates src/local_deep_research/web/static/dist/, which the
# Flask app loads via vite_helper. Without this the page renders
# without bundled CSS, causing tests to run against a partially
# unstyled UI (the responsive baseline tracked in PR #3979 was
# captured against a built bundle).
run: npm run build
- name: Install Puppeteer dependencies
working-directory: tests/puppeteer
run: |
npm ci
npx puppeteer browsers install chrome
- name: Setup directories
run: |
mkdir -p data/encrypted_databases
mkdir -p tests/puppeteer/screenshots
- name: Initialize database
env:
TEST_ENV: true
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
# Without LDR_TEST_MODE the 1000 is clamped to the production
# minimum (256000), making SQLCipher ~256x slower — see #4430.
LDR_TEST_MODE: "1"
run: |
cd src
pdm run python ../scripts/ci/init_test_database.py
- name: Create test configuration
run: |
# Create config directory if needed
mkdir -p ~/.config/local-deep-research
# Create settings with OpenRouter + Gemini 2.5 Flash Lite
# (Gemini 2.0 Flash is heavily rate-limited on the OpenRouter free
# pool; 2.5 Flash Lite is GA, cheaper, and far less contended.)
cat > ~/.config/local-deep-research/settings.toml << 'EOF'
[llm]
provider = "openrouter"
model = "google/gemini-2.5-flash-lite"
[search]
tool = "serper"
iterations = 1
questions_per_iteration = 2
[general]
report_type = "quick"
EOF
- name: Start LDR server
env:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
# Settings manager env var overrides (format: LDR_<KEY_WITH_UNDERSCORES>)
# LLM settings - use OpenRouter with Gemini 2.5 Flash Lite
LDR_LLM_PROVIDER: openrouter
LDR_LLM_MODEL: google/gemini-2.5-flash-lite
LDR_LLM_OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
# Search settings
LDR_SEARCH_TOOL: serper
LDR_SEARCH_ENGINE_WEB_SERPER_API_KEY: ${{ secrets.SERPER_API_KEY }}
# Database and server settings
LDR_DATA_DIR: ${{ github.workspace }}/data
LDR_DB_CONFIG_KDF_ITERATIONS: "1000"
# Without LDR_TEST_MODE the 1000 is clamped to the production
# minimum (256000), making SQLCipher ~256x slower — see #4430.
LDR_TEST_MODE: "1"
CI: true
LDR_DISABLE_RATE_LIMITING: true
run: |
# Start server in background
cd src
nohup pdm run python -m local_deep_research.web.app > /tmp/ldr_server.log 2>&1 &
echo "Waiting for server to start..."
for i in {1..60}; do
if curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
echo "Server is ready after $i seconds!"
break
fi
if ! pgrep -f 'python -m local_deep_research.web.app' > /dev/null; then
echo "Server process died!"
cat /tmp/ldr_server.log
exit 1
fi
echo "Waiting... ($i/60)"
sleep 1
done
# Final check if server is ready
if ! curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health 2>/dev/null; then
echo "Server failed to start. Logs:"
cat /tmp/ldr_server.log
exit 1
fi
- name: Run Puppeteer tests
working-directory: tests/puppeteer
env:
TEST_URL: http://localhost:5000
HEADLESS: true
run: |
set -o pipefail
# Run all test suites with xvfb for virtual display
xvfb-run -a -s "-screen 0 1920x1080x24" npm run test:ci 2>&1 | tee test-output.log
TEST_RESULT="${PIPESTATUS[0]}"
# Save exit code for downstream steps
echo "TEST_EXIT_CODE=${TEST_RESULT}" >> "$GITHUB_ENV"
# Exit with test result so the step reflects pass/fail
exit "${TEST_RESULT}"
- name: Run egress-policy UI test
# Release gate for the Privacy & Egress UI (scope dropdown, the
# require-local toggles, the STRICT+meta-picker guard, and the
# data-scope propagation across pages). This test lives in
# tests/ui_tests (a standalone Puppeteer script, NOT the mocha suite
# above) and runs against the SAME booted server. It authenticates as
# the CI test user (test_admin) created by init_test_database.py and
# matched in tests/ui_tests/auth_helper.js, and skips screenshots when
# CI=true. `if: always()` so its result is reported even when the mocha
# suite above failed; the script exits non-zero on any failed check, so
# a regression fails the release gate.
if: always()
working-directory: tests/ui_tests
env:
BASE_URL: http://localhost:5000
CI: true
HEADLESS: true
run: |
set -o pipefail
npm ci
npx puppeteer browsers install chrome
xvfb-run -a -s "-screen 0 1920x1080x24" \
node test_egress_policy_ui.js 2>&1 | tee egress-ui-output.log
- name: Extract test summary
if: always()
working-directory: tests/puppeteer
run: |
# Extract pass/fail summary from mocha output
if [ -f test-output.log ]; then
# Get the summary line (e.g., "26 passing (3m)")
PASSING=$(grep -oP '\d+ passing' test-output.log | tail -1 || echo "0 passing")
FAILING=$(grep -oP '\d+ failing' test-output.log | tail -1 || echo "0 failing")
echo "TESTS_PASSING=${PASSING}" >> "$GITHUB_ENV"
echo "TESTS_FAILING=${FAILING}" >> "$GITHUB_ENV"
# Extract key test results (collection creation, subscription, research)
echo "## Key Test Results" > test-summary.md
echo "" >> test-summary.md
# Check for collection creation
if grep -q "Collection found in dropdown: true" test-output.log; then
echo "- ✅ Collection creation: **Success**" >> test-summary.md
elif grep -q "Collection found in dropdown: false" test-output.log; then
echo "- ❌ Collection creation: **Failed**" >> test-summary.md
fi
# Check for subscription creation
if grep -q "Subscription name found: true" test-output.log; then
echo "- ✅ Subscription creation: **Success**" >> test-summary.md
elif grep -q "Subscription name found: false" test-output.log; then
echo "- ❌ Subscription creation: **Failed**" >> test-summary.md
fi
# Check for research completion
if grep -q "Research completed: true" test-output.log; then
RESEARCH_TIME=$(grep -oP "Research completed: true \(took \K\d+(?=s\))" test-output.log || echo "?")
echo "- ✅ Research workflow: **Completed** (${RESEARCH_TIME}s)" >> test-summary.md
elif grep -q "Research completed: false" test-output.log; then
echo "- ⚠️ Research workflow: **Timed out**" >> test-summary.md
fi
# Check for settings persistence
if grep -q "After reload value:" test-output.log; then
echo "- ✅ Settings persistence: **Working**" >> test-summary.md
fi
# Add screenshot count
SCREENSHOT_COUNT=$(find screenshots -name "*.png" 2>/dev/null | wc -l || echo "0")
echo "" >> test-summary.md
echo "**Screenshots captured:** ${SCREENSHOT_COUNT}" >> test-summary.md
fi
- name: Show test summary in workflow UI
if: always()
working-directory: tests/puppeteer
# zizmor: ignore[template-injection] — env.TEST_EXIT_CODE/TESTS_PASSING/TESTS_FAILING are set by this workflow, not user input
run: |
# Output to GITHUB_STEP_SUMMARY so results are visible in Actions UI
{
echo "## 🧪 Puppeteer E2E Test Results"
echo ""
if [ "$TEST_EXIT_CODE" == "0" ]; then
echo "### ✅ Tests Passed!"
else
echo "### ❌ Tests Failed"
fi
echo ""
echo "**Summary:** $TESTS_PASSING, $TESTS_FAILING"
echo ""
# Include key test results
if [ -f test-summary.md ]; then
cat test-summary.md
echo ""
fi
echo "---"
echo ""
echo "<details>"
echo "<summary>📋 Last 50 lines of test output</summary>"
echo ""
echo '```'
if [ -f test-output.log ]; then
tail -50 test-output.log
fi
echo '```'
echo "</details>"
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload server logs
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: server-logs
path: /tmp/ldr_server.log
retention-days: 7
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: puppeteer-test-results
path: |
tests/puppeteer/test-output.log
tests/puppeteer/test-summary.md
tests/puppeteer/screenshots/
retention-days: 7
- name: Find PR number
id: find-pr
if: always()
# zizmor: ignore[template-injection] — github.ref_name and github.repository are safe GitHub context values
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
EVENT_NAME: ${{ github.event_name }}
PR_NUM: ${{ github.event.pull_request.number }}
REF_NAME: ${{ github.ref_name }}
REPO: ${{ github.repository }}
run: |
if [ "$EVENT_NAME" == "pull_request" ]; then
# Labeled PR run — the primary, correct path. Real branches targeting main.
echo "pr_number=$PR_NUM" >> "$GITHUB_OUTPUT"
elif [ "$EVENT_NAME" == "workflow_dispatch" ] && [ "$REF_NAME" != "main" ]; then
# Manual run on a feature branch. Map branch -> its open PR, but only a
# UNIQUE, SAME-REPO match: cross-repo (fork) PRs and ambiguous matches are
# skipped so we never post to the wrong PR (e.g. a fork PR'd from `main`).
PR_NUMBER=$(gh pr list --head "$REF_NAME" --state open --repo "$REPO" \
--json number,isCrossRepository \
--jq '[.[] | select(.isCrossRepository == false)]
| if length == 1 then .[0].number | tostring else "" end')
echo "pr_number=${PR_NUMBER}" >> "$GITHUB_OUTPUT"
else
# release.yml workflow_call (event=push) and the weekly schedule both run
# on the default branch. There is never a PR from main, so any
# `gh pr list --head main` match is a coincidental fork PR — skip
# entirely (results still go to the Actions step summary). See #3884.
echo "pr_number=" >> "$GITHUB_OUTPUT"
echo "event=$EVENT_NAME ref=$REF_NAME has no associated PR — skipping comment"
fi
- name: Build PR comment
if: always() && steps.find-pr.outputs.pr_number != ''
# zizmor: ignore[template-injection] — env.TEST_EXIT_CODE/TESTS_PASSING/TESTS_FAILING are set by this workflow, not user input
env:
EVENT_NAME: ${{ github.event_name }}
run: |
{
# Hidden marker — the "Post or update PR comment" step finds this
# comment by this marker so re-runs edit it in place (see #3884).
echo '<!-- puppeteer-e2e-results -->'
echo "## 🧪 Puppeteer E2E Test Results"
echo ""
if [ "$TEST_EXIT_CODE" == "0" ]; then
echo "### ✅ Tests Passed!"
else
echo "### ❌ Tests Failed"
fi
echo ""
echo "**Summary:** $TESTS_PASSING, $TESTS_FAILING"
echo ""
# Include key test results
if [ -f tests/puppeteer/test-summary.md ]; then
cat tests/puppeteer/test-summary.md
echo ""
fi
# Include research output if available
if [ -f tests/puppeteer/research-output/research-result.txt ]; then
echo "---"
echo ""
echo "### 📝 Research Output"
echo ""
if [ -f tests/puppeteer/research-output/research-metadata.json ]; then
QUERY=$(jq -r '.query // "unknown"' tests/puppeteer/research-output/research-metadata.json)
echo "**Query:** \`$QUERY\`"
echo ""
fi
echo "<details>"
echo "<summary>📄 Research Result (click to expand)</summary>"
echo ""
echo '```markdown'
# Get first 5000 chars of research output
head -c 5000 tests/puppeteer/research-output/research-result.txt
TOTAL_SIZE=$(wc -c < tests/puppeteer/research-output/research-result.txt)
if [ "$TOTAL_SIZE" -gt 5000 ]; then
echo ""
echo "... [truncated, total $TOTAL_SIZE chars]"
fi
echo '```'
echo "</details>"
echo ""
fi
echo "---"
echo ""
echo "<details>"
echo "<summary>📋 Full Test Output (click to expand)</summary>"
echo ""
echo '```'
# Get last 100 lines of test output (most relevant)
if [ -f tests/puppeteer/test-output.log ]; then
tail -100 tests/puppeteer/test-output.log
fi
echo '```'
echo "</details>"
echo ""
echo "---"
echo ""
echo "**Configuration:**"
echo "- Model: \`google/gemini-2.5-flash-lite\` (via OpenRouter)"
echo "- Search: \`serper\`"
echo "- Test suite: \`all\`"
echo ""
echo "_Triggered by: \`${EVENT_NAME}\` (label: \`${LABEL_NAME}\`)_"
} > comment.md
- name: Post or update PR comment
if: always() && steps.find-pr.outputs.pr_number != ''
continue-on-error: true # required release gate must not fail on a GitHub API hiccup
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
PR_NUMBER: ${{ steps.find-pr.outputs.pr_number }}
with:
retries: 3
script: |
const fs = require('fs');
const issue_number = Number(process.env.PR_NUMBER);
const marker = '<!-- puppeteer-e2e-results -->';
const body = fs.readFileSync('comment.md', 'utf8');
// Paginate — on a long-lived PR the existing comment may be past the
// first page (default 30), which would otherwise create a new comment
// every run and reintroduce the spam this fix removes (#3884).
const comments = await github.paginate(github.rest.issues.listComments, {
owner: context.repo.owner,
repo: context.repo.repo,
issue_number,
per_page: 100,
});
const existing = comments.find(
(c) => c.user.type === 'Bot' && c.body.includes(marker)
);
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number,
body,
});
}
- name: Remove label for re-triggering
if: always() && github.event_name == 'pull_request'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUM: ${{ github.event.pull_request.number }}
run: |
gh issue edit "$PR_NUM" --remove-label "$LABEL_NAME" 2>/dev/null || true
- name: Cleanup
if: always()
run: |
# Kill server
pkill -f 'python -m local_deep_research.web.app' || true
rm -f comment.md
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,375 @@
name: Responsive UI Tests
on:
workflow_call:
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow runs only via workflow_call (from release.yml's
# responsive-test-gate) and workflow_dispatch. The pull_request trigger
# was deliberately removed in #2248 to reduce PR CI load on what is a
# heavy ~20-minute matrix build (mobile + desktop). A shared concurrency
# key here previously caused workflow_call invocations to cancel each
# other mid-flight; see #3554 (reverted in #3599) for that history.
permissions:
contents: read
jobs:
ui-tests:
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
viewport: [mobile, desktop] # Test mobile and desktop on each PR
services:
postgres:
image: postgres:14@sha256:ca25035f7e6f74552655a1c5e4a9eb21f85e9d316f1f70371f790ef70095dd58 # v14
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: ldr_test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
cache: 'npm'
cache-dependency-path: tests/ui_tests/package-lock.json
- name: Free up disk space
run: |
# Remove unnecessary large packages to prevent disk space issues during cache save
sudo rm -rf /usr/share/dotnet
sudo rm -rf /usr/local/lib/android
sudo rm -rf /opt/ghc
sudo rm -rf /opt/hostedtoolcache/CodeQL
df -h
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
wget \
gnupg \
ca-certificates \
fonts-liberation \
libasound2t64 \
libatk-bridge2.0-0 \
libatk1.0-0 \
libcups2 \
libdbus-1-3 \
libdrm2 \
libgbm1 \
libgtk-3-0 \
libnspr4 \
libnss3 \
libx11-xcb1 \
libxcomposite1 \
libxdamage1 \
libxrandr2 \
xdg-utils \
imagemagick
- name: Install Python dependencies
run: pdm install
- name: Install root frontend dependencies
run: npm ci
- name: Build Vite frontend bundle
# Generates src/local_deep_research/web/static/dist/, which the
# Flask app loads via vite_helper. Without this the responsive UI
# tests run against an unstyled page (no styles.css), which means
# any CSS source changes between PRs are invisible to the test
# baseline. See follow-up to PR #3985 for context.
run: npm run build
- name: Install Node test dependencies
working-directory: tests/ui_tests
run: |
npm ci
# Keep this version in sync with tests/ui_tests/package-lock.json's
# puppeteer entry. A mismatch makes npx fetch a second puppeteer
# that targets a different Chrome build, corrupting the browser
# cache ("folder exists but executable is missing"). Matching the
# locked version lets npx reuse the puppeteer just installed above.
npx puppeteer@25.1.0 browsers install chrome
- name: Set up test directories
run: |
mkdir -p ${{ github.workspace }}/data/encrypted_databases
mkdir -p tests/ui_tests/screenshots
echo "Created data and screenshots directories for tests"
- name: Start test server
env:
CI: true
FLASK_ENV: testing
TEST_ENV: true
SECRET_KEY: test-secret-key-for-ci # Security: CI test credential, not production secret
LDR_DISABLE_RATE_LIMITING: true
LDR_DATA_DIR: ${{ github.workspace }}/data
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/ldr_test # Security: CI test database credentials, not production secrets
run: |
cd src
# Start server and get its PID
pdm run python -m local_deep_research.web.app > server.log 2>&1 &
SERVER_PID=$!
echo "Server PID: $SERVER_PID"
# Persist the PID for the "Stop application server" step. That step
# runs at the workspace root (no working-directory) while we are in
# src/ here, so write an absolute path — otherwise the pidfile never
# lines up and cleanup is a silent no-op.
echo "$SERVER_PID" > "$GITHUB_WORKSPACE/server.pid"
# Wait for server to start
SERVER_READY=false
for i in {1..60}; do
# Probe the health endpoint, not bare `/`: `/` 302-redirects to
# /auth/login and `curl -f` treats a 302 as success the instant
# the socket binds — before DB/app init finishes — so it can pass
# while the app is still not ready. /api/v1/health is a true
# readiness signal and matches the docker-tests.yml gate.
if curl -fsS --connect-timeout 2 --max-time 5 http://127.0.0.1:5000/api/v1/health 2>/dev/null; then
echo "Server is ready after $i seconds"
SERVER_READY=true
break
fi
# Check if process is still running
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
echo "Server process died!"
echo "Server log:"
cat server.log
exit 1
fi
echo "Waiting for server... ($i/60)"
sleep 1
done
# Fail fast if the server is alive but never became ready in time.
# Without this guard the loop fell through silently on a slow-but-
# not-crashed start, so "Register CI test user" and the UI tests ran
# against a server that was not yet listening and failed with a
# confusing `net::ERR_CONNECTION_REFUSED` in the *test* step instead
# of a clear startup failure here. Window was also bumped 30s -> 60s
# to match the proven puppeteer-e2e-tests.yml / docker-tests.yml
# startup gates (Postgres-backed boot can exceed 30s on a loaded
# runner).
if [ "$SERVER_READY" != "true" ]; then
echo "❌ Server did not become ready within 60 seconds"
echo "Server log:"
cat server.log
exit 1
fi
- name: Register CI test user
working-directory: tests/ui_tests
run: node register_ci_user.js http://127.0.0.1:5000
- name: Run responsive UI tests - ${{ matrix.viewport }}
id: run-tests
working-directory: tests/ui_tests
env:
VIEWPORT: ${{ matrix.viewport }}
run: |
set +e # Don't exit on test failure
# Run tests and capture output
HEADLESS=true node test_responsive_ui_comprehensive.js "$VIEWPORT" 2>&1 | tee test-output.log
TEST_EXIT_CODE="${PIPESTATUS[0]}"
# Extract summary for PR comment
echo "### 📱 $VIEWPORT Test Results" > test-summary.md
echo "" >> test-summary.md
# Extract pass/fail counts
PASSED=$(grep -oP '\d+(?= passed)' test-output.log | tail -1 || echo "0")
FAILED=$(grep -oP '\d+(?= failed)' test-output.log | tail -1 || echo "0")
WARNINGS=$(grep -oP '\d+(?= warnings)' test-output.log | tail -1 || echo "0")
if [ "$FAILED" -eq "0" ]; then
echo "✅ **All tests passed!**" >> test-summary.md
else
echo "❌ **$FAILED critical issues found**" >> test-summary.md
fi
{
echo ""
echo "- ✅ Passed: $PASSED"
echo "- ❌ Failed: $FAILED"
echo "- ⚠️ Warnings: $WARNINGS"
} >> test-summary.md
{
echo "test_exit_code=$TEST_EXIT_CODE"
echo "test_failed=$FAILED"
} >> "$GITHUB_OUTPUT"
# Only fail if critical failures
exit "$TEST_EXIT_CODE"
- name: Upload viewport test results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ui-test-results-${{ matrix.viewport }}
path: |
tests/ui_tests/test-output.log
tests/ui_tests/test-summary.md
tests/ui_tests/screenshots/
tests/ui_tests/responsive/
src/server.log
retention-days: 7
if-no-files-found: ignore
# NOTE: Earlier screenshot generation/upload steps were removed to fix actionlint warnings
# caused by "if: false" disablement. Screenshots are now captured into the artifact above
# when they exist. Visual regression workflows can re-enable dedicated screenshot steps later.
- name: Upload to GitHub Pages (if available)
if: always() && github.event_name == 'pull_request'
continue-on-error: true
env:
PR_NUMBER: ${{ github.event.pull_request.number }}
VIEWPORT: ${{ matrix.viewport }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPOSITORY: ${{ github.repository }}
GH_REPO_OWNER: ${{ github.repository_owner }}
GH_REPO_NAME: ${{ github.event.repository.name }}
run: |
# This requires GitHub Pages to be enabled for the repo
# Create a branch for the screenshots
BRANCH_NAME="pr-screenshots-$PR_NUMBER-$VIEWPORT"
# Check if screenshots directory exists
if [ ! -d "tests/ui_tests/responsive" ]; then
echo "No screenshots directory found, skipping upload"
exit 0
fi
cd tests/ui_tests/responsive
git init
git config user.name "GitHub Actions"
git config user.email "actions@github.com"
# Add all screenshots and gallery
git add responsive-ui-tests screenshot-gallery.html
git commit -m "Screenshots for PR #$PR_NUMBER"
# Push to a dedicated branch (requires write permissions)
git push --force "https://x-access-token:$GH_TOKEN@github.com/$GH_REPOSITORY.git" "HEAD:$BRANCH_NAME" 2>/dev/null || true
# The URL would be: https://[owner].github.io/[repo]/pr-screenshots-[number]-[viewport]/screenshot-gallery.html
echo "Screenshots available at: https://$GH_REPO_OWNER.github.io/$GH_REPO_NAME/$BRANCH_NAME/screenshot-gallery.html" || true
- name: Stop application server
if: always()
run: |
if [ -f server.pid ]; then
kill "$(cat server.pid)" || true
rm server.pid
fi
# `pdm run` stays alive as a wrapper, so the pidfile holds the
# wrapper PID and killing it can orphan the actual python server.
# Add the same pkill fallback the webkit workflow uses so the
# python process (and port 5000) is actually released.
pkill -f "python -m local_deep_research.web.app" || true
post-results:
needs: ui-tests
runs-on: ubuntu-latest
# Workflow has no pull_request trigger; gating on event_name == 'pull_request' kept this job
# from ever running. Always run so the combined report is built for workflow_dispatch / release.
if: always()
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Download all artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: test-artifacts/
- name: Generate combined report
run: |
# Create a combined markdown report with screenshot links
{
echo "# 📊 Responsive UI Test Report"
echo ""
echo "## Test Summary"
echo ""
} > combined-report.md
# Process each viewport's results
for dir in test-artifacts/ui-test-results-*/; do
if [ -d "$dir" ]; then
viewport=$(basename "$dir" | sed 's/ui-test-results-//')
echo "### $viewport" >> combined-report.md
if [ -f "$dir/test-summary.md" ]; then
cat "$dir/test-summary.md" >> combined-report.md
fi
# Count screenshots
screenshot_count=$(find "$dir" -name "*.png" 2>/dev/null | wc -l)
if [ "$screenshot_count" -gt "0" ]; then
{
echo ""
echo "📸 **$screenshot_count screenshots captured**"
} >> combined-report.md
fi
echo "" >> combined-report.md
fi
done
{
echo "## 📥 Download Options"
echo ""
echo "- [Download all test artifacts](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})"
echo "- View artifacts in the 'Artifacts' section below the workflow summary"
} >> combined-report.md
- name: Upload combined report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: combined-test-report
path: combined-report.md
retention-days: 30
+86
View File
@@ -0,0 +1,86 @@
name: Retire.js Security Scan
on:
workflow_dispatch:
workflow_call: # Called by release-gate.yml
schedule:
# Run weekly security scan on Mondays at 4 AM UTC
- cron: '0 4 * * 1'
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
# Note: No concurrency block - this workflow is called by release-gate.yml
# and concurrency settings can cause unexpected cancellations for reusable workflows
jobs:
retirejs:
name: Retire.js Vulnerability Scan
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
security-events: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
- name: Run Retire.js scan
id: retire-scan
run: |
echo "=== Running Retire.js scan ==="
# Use npx with specific version for reproducible builds
# Scan JavaScript files and node_modules for known vulnerabilities
# Only flag medium severity and above (exit code 13 when issues found)
# Output both SARIF (for GitHub Security) and human-readable format
npx retire@5.2.4 --js --node --severity medium --outputformat sarif --outputpath retire-results.sarif 2>&1 | tee retire-output.txt
RETIRE_EXIT_CODE=$?
# Check if vulnerabilities were found (exit code 13 indicates issues)
if [ $RETIRE_EXIT_CODE -eq 13 ]; then
echo "RETIRE_FOUND_ISSUES=true" >> "$GITHUB_ENV"
fi
# Display human-readable summary
echo ""
echo "=== Scan Summary ==="
cat retire-output.txt
# Check if SARIF file was created (for use in upload step)
if [ -f "retire-results.sarif" ]; then
echo "sarif_exists=true" >> "$GITHUB_OUTPUT"
fi
- name: Upload SARIF results
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always() && steps.retire-scan.outputs.sarif_exists == 'true'
with:
sarif_file: retire-results.sarif
category: retirejs
- name: Check scan results
run: |
if [[ "$RETIRE_FOUND_ISSUES" == "true" ]]; then
echo "::error::Retire.js found medium or higher severity vulnerabilities"
echo "Review the Security tab for details"
echo "Consider updating or replacing affected libraries"
# Only fail for PR/push events, not scheduled scans
if [[ "${{ github.event_name }}" != "schedule" ]]; then
exit 1
fi
echo "::warning::Scheduled scan - reporting only, not failing the workflow"
else
echo "✅ No medium or higher severity vulnerabilities detected"
fi
+81
View File
@@ -0,0 +1,81 @@
name: SBOM Generation
on:
workflow_dispatch:
schedule:
# Run weekly on Wednesday at 10 AM UTC (staggered with other scans)
- cron: '0 10 * * 3'
release:
types: [published]
permissions: {}
jobs:
sbom:
name: Generate Software Bill of Materials
runs-on: ubuntu-latest
permissions:
contents: read
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Generate SBOM for source code
uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
with:
path: .
artifact-name: sbom-source.spdx.json
output-file: sbom-source.spdx.json
format: spdx-json
- name: Generate SBOM for Python dependencies
uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0
with:
path: .
artifact-name: sbom-python.cyclonedx.json
output-file: sbom-python.cyclonedx.json
format: cyclonedx-json
- name: Upload SBOMs as artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: sbom-reports
path: |
sbom-source.spdx.json
sbom-python.cyclonedx.json
retention-days: 90
- name: Display SBOM summary
run: |
{
echo "## SBOM Generation Summary"
echo ""
echo "### Generated SBOMs"
echo "- **Source Code SBOM**: \`sbom-source.spdx.json\` (SPDX format)"
echo "- **Dependencies SBOM**: \`sbom-python.cyclonedx.json\` (CycloneDX format)"
echo ""
echo "### What is an SBOM?"
echo "A Software Bill of Materials (SBOM) is a formal record of all components,"
echo "libraries, and dependencies used in building software. It enables:"
echo ""
echo "- **Vulnerability tracking**: Know exactly what's in your software"
echo "- **License compliance**: Verify all dependencies have compatible licenses"
echo "- **Supply chain security**: Detect compromised dependencies"
echo "- **Regulatory compliance**: Meet requirements (EO 14028, etc.)"
echo ""
echo "### Artifacts"
echo "SBOMs are available as workflow artifacts for 90 days."
if [ "${{ github.event_name }}" = "release" ]; then
echo ""
echo "SBOMs have also been attached to the release assets."
fi
} >> "$GITHUB_STEP_SUMMARY"
@@ -0,0 +1,34 @@
name: Security File Write Check
on:
pull_request:
types: [opened, synchronize, reopened]
workflow_call: # Called by ci-gate.yml for release pipeline
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions:
contents: read
jobs:
check-file-writes:
name: Check for Unencrypted File Writes
runs-on: ubuntu-latest
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Check for potential unencrypted file writes
run: ./.github/scripts/check-file-writes.sh
@@ -0,0 +1,194 @@
name: Security Headers Validation
on:
workflow_call: # Called by release-gate.yml
schedule:
- cron: '0 3 * * *' # Daily scan for early detection
workflow_dispatch: # Manual trigger for debugging/verification
permissions:
contents: read
# NOTE: No concurrency block here. When called via workflow_call from
# release-gate.yml, the caller's workflow name is used for the concurrency
# group, which can cause this job to be cancelled if another release gate
# run starts on the same ref. The parent workflow manages concurrency.
jobs:
validate-headers:
runs-on: ubuntu-latest
timeout-minutes: 15
services:
postgres:
image: postgres:13@sha256:4689940c683801b4ab839ab3b0a0a3555a5fe425371422310944e89eca7d8068 # v13
env:
POSTGRES_USER: ldr_test
POSTGRES_PASSWORD: ${{ github.run_id }}_test_pwd
POSTGRES_DB: test_db
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python 3.12
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Cache pip packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-headers-${{ hashFiles('**/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-pip-headers-
${{ runner.os }}-pip-
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev build-essential
- name: Install dependencies
run: |
pdm install --dev
- name: Set up test environment
run: |
mkdir -p data
mkdir -p research_outputs
cp -r src/local_deep_research/defaults/.env.template .env.test
- name: Start Flask application in background
run: |
# Use FLASK_DEBUG=0 (FLASK_ENV is deprecated)
export FLASK_DEBUG=0
export DATABASE_URL=postgresql://ldr_test:${{ github.run_id }}_test_pwd@localhost:5432/test_db
pdm run python -m local_deep_research.web.app &
echo $! > flask.pid
- name: Wait for application to be ready
run: |
timeout 60 bash -c 'until curl -fsS --connect-timeout 2 --max-time 5 http://localhost:5000/api/v1/health > /dev/null; do sleep 2; done'
- name: Validate security headers
run: |
echo "=== Testing Security Headers ==="
echo ""
# Capture headers once for efficiency (with timeout)
HEADERS=$(curl -sI --max-time 10 http://localhost:5000/)
echo "Full response headers:"
echo "$HEADERS"
echo ""
# Check for required security headers and validate values
ERRORS=()
# X-Frame-Options - must be SAMEORIGIN or DENY
XFO=$(echo "$HEADERS" | grep -i "^X-Frame-Options:" | cut -d':' -f2- | tr -d '\r' | xargs)
if [[ -z "$XFO" ]]; then
ERRORS+=("X-Frame-Options: MISSING")
elif [[ "$XFO" != "SAMEORIGIN" && "$XFO" != "DENY" ]]; then
ERRORS+=("X-Frame-Options: Invalid value '$XFO' (expected SAMEORIGIN or DENY)")
else
echo "✅ X-Frame-Options: $XFO"
fi
# X-Content-Type-Options - must be nosniff
XCTO=$(echo "$HEADERS" | grep -i "^X-Content-Type-Options:" | cut -d':' -f2- | tr -d '\r' | xargs)
if [[ -z "$XCTO" ]]; then
ERRORS+=("X-Content-Type-Options: MISSING")
elif [[ "$XCTO" != "nosniff" ]]; then
ERRORS+=("X-Content-Type-Options: Invalid value '$XCTO' (expected nosniff)")
else
echo "✅ X-Content-Type-Options: $XCTO"
fi
# Referrer-Policy - must not be unsafe
RP=$(echo "$HEADERS" | grep -i "^Referrer-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
if [[ -z "$RP" ]]; then
ERRORS+=("Referrer-Policy: MISSING")
elif [[ "$RP" == "unsafe-url" || "$RP" == "no-referrer-when-downgrade" ]]; then
ERRORS+=("Referrer-Policy: Insecure value '$RP'")
else
echo "✅ Referrer-Policy: $RP"
fi
# Permissions-Policy - must restrict dangerous features
PP=$(echo "$HEADERS" | grep -i "^Permissions-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
if [[ -z "$PP" ]]; then
ERRORS+=("Permissions-Policy: MISSING")
elif [[ "$PP" != *"geolocation=()"* ]]; then
ERRORS+=("Permissions-Policy: Should restrict geolocation")
else
echo "✅ Permissions-Policy: $PP"
fi
# Content-Security-Policy - must have default-src and not be overly permissive
CSP=$(echo "$HEADERS" | grep -i "^Content-Security-Policy:" | cut -d':' -f2- | tr -d '\r' | xargs)
if [[ -z "$CSP" ]]; then
ERRORS+=("Content-Security-Policy: MISSING")
elif [[ "$CSP" == *"default-src *"* || "$CSP" == *"default-src 'unsafe-inline' 'unsafe-eval'"* ]]; then
ERRORS+=("Content-Security-Policy: Overly permissive policy detected")
elif [[ "$CSP" != *"default-src"* ]]; then
ERRORS+=("Content-Security-Policy: Missing default-src directive")
else
echo "✅ Content-Security-Policy: Present with default-src"
fi
# Report results
echo ""
if [ ${#ERRORS[@]} -eq 0 ]; then
echo "✅ All required security headers are present and valid"
else
echo "❌ Security header issues found:"
for error in "${ERRORS[@]}"; do
echo " - $error"
done
exit 1
fi
- name: Test API endpoint headers
run: |
echo "=== Testing Security Headers on API Endpoints ==="
echo ""
# Test health endpoint (with timeout)
API_HEADERS=$(curl -sI --max-time 10 http://localhost:5000/api/health)
if echo "$API_HEADERS" | grep -iq "^X-Frame-Options:"; then
echo "✅ Security headers present on API endpoints"
else
echo "❌ Security headers missing on API endpoints"
echo "$API_HEADERS"
exit 1
fi
- name: Cleanup
if: always()
run: |
if [ -f flask.pid ]; then
kill "$(cat flask.pid)" 2>/dev/null || true
rm -f flask.pid
fi
+137
View File
@@ -0,0 +1,137 @@
name: Security Tests
on:
workflow_call: # Called by release-gate.yml
workflow_dispatch:
permissions:
contents: read
security-events: write # Required for uploading SARIF results to Code Scanning
jobs:
security-tests:
runs-on: ubuntu-latest
timeout-minutes: 25
services:
postgres:
image: postgres:13@sha256:1094e2cdc5605e5c7914633bcd93758a9c52ae8c8b2855ddd1c3a8afbe4795d5 # postgres:13
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_security_db
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python 3.12
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Cache pip packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-security-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-security-
${{ runner.os }}-pip-
- name: Set up PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Install system dependencies for SQLCipher
run: |
sudo apt-get update
sudo apt-get install -y libsqlcipher-dev
- name: Install dependencies
run: |
pdm sync -d
pdm add "bandit[sarif]" safety sqlparse pytest pytest-cov --no-sync
pdm sync
- name: Run Bandit security linter
run: |
# --exit-zero: report findings without failing (findings go to SARIF/Security tab)
# If bandit crashes (bad config, missing files), it exits non-zero WITHOUT --exit-zero
# and won't produce output files — the checks below catch that.
pdm run bandit -r src/ -f json -o bandit-report.json -lll --exit-zero
pdm run bandit -r src/ -f sarif -o bandit-results.sarif -lll --exit-zero
# Crash detection: if bandit crashed, no output files were produced
if [ ! -f bandit-report.json ] || [ ! -f bandit-results.sarif ]; then
echo "::error::Bandit crashed — expected output files not produced"
exit 1
fi
echo "Bandit security scan completed"
- name: Upload Bandit SARIF to Code Scanning
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always()
with:
sarif_file: bandit-results.sarif
category: bandit
# Dependency vulnerability scanning is handled by OSV-Scanner
# (osv-scanner.yml), which reads pdm.lock directly and doesn't
# fight PDM's resolution overrides the way pip-audit did.
- name: Run security test suite
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test_security_db
run: |
# Run the entire tests/security/ tree as a directory rather than
# enumerating individual files. The old per-file list only covered
# ~9 of the 87 test files and silently broke whenever a file was
# renamed or deleted: a hardcoded reference to the deleted
# test_input_validation.py (removed in #4243) made pytest exit with
# code 5 ("no tests collected") and failed this gate (#4411).
# Running the directory picks up new security tests automatically and
# never rots when files move.
pdm run python -m pytest tests/security/ -v --tb=short -n auto
- name: Check for hardcoded secrets
run: |
# Check for potential secrets in code
grep -r -E "(api[_-]?key|secret[_-]?key|password|token)" src/ --include="*.py" | \
grep -v -E "(os\.environ|getenv|config\[|placeholder|example|test)" | \
grep -E "=\s*['\"]" || echo "No hardcoded secrets found"
- name: Generate security report
if: always()
run: |
echo "Security Test Report"
echo "==================="
echo ""
if [ -f bandit-report.json ]; then
echo "Bandit Security Issues:"
python -c "import json; data=json.load(open('bandit-report.json')); print(f' High: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"HIGH\"])}'); print(f' Medium: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"MEDIUM\"])}'); print(f' Low: {len([i for i in data.get(\"results\", []) if i[\"issue_severity\"] == \"LOW\"])}')" || true
fi
echo ""
echo "Dependency vulnerabilities: covered by OSV-Scanner (osv-scanner.yml)"
- name: Upload security reports
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: security-reports
path: |
bandit-report.json
bandit-results.sarif
+251
View File
@@ -0,0 +1,251 @@
name: Semgrep Security Scan
on:
workflow_call: # Called by release-gate.yml
workflow_dispatch:
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
semgrep-scan:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install Semgrep
run: |
# setuptools required: semgrep 1.87.0's opentelemetry dep imports pkg_resources,
# which is not bundled with Python 3.12 by default on GitHub runners.
# Pin setuptools<82 because 82.0 removed the pkg_resources module.
pip install "setuptools<82" semgrep==1.87.0
- name: Run Semgrep security scan
run: |
semgrep \
--config=p/security-audit \
--config=p/secrets \
--severity=INFO \
--json \
--output=semgrep-results.json \
src/ || true
# Crash detection: if semgrep crashed, no output file was produced
if [ ! -f semgrep-results.json ]; then
echo "::error::Semgrep security scan crashed — no output produced"
exit 1
fi
- name: Run Semgrep custom rules for LDR
run: |
semgrep \
--config=.semgrep/rules/ \
--severity=INFO \
--json \
--output=semgrep-custom-results.json \
src/ || true
# Crash detection: if semgrep crashed, no output file was produced
if [ ! -f semgrep-custom-results.json ]; then
echo "::error::Semgrep custom rules scan crashed — no output produced"
exit 1
fi
- name: Merge Semgrep results
run: |
python3 -c "
import json
import glob
results = []
for file in glob.glob('semgrep-*.json'):
try:
with open(file) as f:
data = json.load(f)
if isinstance(data, dict) and 'results' in data:
results.extend(data['results'])
elif isinstance(data, list):
results.extend(data)
except Exception as e:
print(f'Error reading {file}: {e}')
output = {'results': results, 'version': '1.0.0'}
with open('semgrep-combined-results.json', 'w') as f:
json.dump(output, f, indent=2)
print(f'Combined {len(results)} findings from Semgrep scans')
"
- name: Convert to SARIF format
run: |
python3 -c "
import json
import uuid
from datetime import datetime
# Load combined results
with open('semgrep-combined-results.json') as f:
semgrep_data = json.load(f)
# Convert to SARIF
sarif = {
'\$schema': 'https://json.schemastore.org/sarif-2.1.0',
'version': '2.1.0',
'runs': [{
'tool': {
'driver': {
'name': 'Semgrep',
'version': '1.87.0',
'informationUri': 'https://semgrep.dev'
}
},
'results': []
}]
}
for result in semgrep_data.get('results', []):
sarif_result = {
'ruleId': result.get('check_id', 'unknown'),
'message': {
'text': result.get('message', 'Security issue detected')
},
'level': 'warning' if result.get('metadata', {}).get('severity', 'INFO') in ['ERROR', 'WARNING'] else 'note',
'locations': [{
'physicalLocation': {
'artifactLocation': {
'uri': result.get('path', 'unknown')
},
'region': {
'startLine': result.get('start', {}).get('line', 1),
'startColumn': result.get('start', {}).get('col', 1),
'endLine': result.get('end', {}).get('line', result.get('start', {}).get('line', 1)),
'endColumn': result.get('end', {}).get('col', result.get('start', {}).get('col', 1) + 1)
}
}
}]
}
# Add rule information
metadata = result.get('metadata', {})
sarif_result['rule'] = {
'id': result.get('check_id', 'unknown'),
'name': metadata.get('name', 'Security Issue'),
'shortDescription': {
'text': metadata.get('name', 'Security Issue')
},
'fullDescription': {
'text': metadata.get('description', 'Security vulnerability detected')
},
'help': {
'text': metadata.get('remediation', 'Review and fix the security issue')
},
'properties': {
'precision': 'medium',
'tags': ['security', 'semgrep']
}
}
if 'security-severity' in metadata:
sarif_result['rule']['properties']['security-severity'] = metadata['security-severity']
sarif['runs'][0]['results'].append(sarif_result)
# Write SARIF file
with open('semgrep-results.sarif', 'w') as f:
json.dump(sarif, f, indent=2)
print(f'Converted {len(sarif[\"runs\"][0][\"results\"])} findings to SARIF format')
"
# Fail loudly if the SARIF conversion produced no file — never fabricate an
# empty one. An empty-results SARIF uploaded under the semgrep-security
# category would make GitHub mark every previously-open Semgrep alert as
# fixed, silently clearing real findings. The "Convert to SARIF" step above
# always writes the file (results:[] on a clean scan), so a missing file
# means a real conversion failure. Mirror the Grype/Trivy jobs.
- name: Ensure SARIF file exists
id: check-sarif
if: always()
run: |
if [ -f semgrep-results.sarif ]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "::error::Semgrep SARIF conversion did not produce a file — scan needs to be rerun"
exit 1
fi
- name: Upload Semgrep results to GitHub Security tab
uses: github/codeql-action/upload-sarif@54f647b7e1bb85c95cddabcd46b0c578ec92bc1a # v4.36.3
if: always() && steps.check-sarif.outputs.exists == 'true'
with:
sarif_file: 'semgrep-results.sarif'
category: semgrep-security
- name: Upload Semgrep results as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: always()
with:
name: semgrep-scan-results
path: |
semgrep-results.json
semgrep-custom-results.json
semgrep-combined-results.json
semgrep-results.sarif
retention-days: 7 # Reduced for security
- name: Display Semgrep summary
if: always()
run: |
{
echo "## Semgrep Security Scan Summary"
echo ""
} >> "$GITHUB_STEP_SUMMARY"
if [ -f semgrep-combined-results.json ]; then
# Count results by severity
CRITICAL=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'ERROR']))" 2>/dev/null || echo "0")
HIGH=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'WARNING']))" 2>/dev/null || echo "0")
MEDIUM=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len([r for r in data['results'] if r.get('metadata', {}).get('severity') == 'INFO']))" 2>/dev/null || echo "0")
TOTAL=$(python3 -c "import json; data=json.load(open('semgrep-combined-results.json')); print(len(data['results']))" 2>/dev/null || echo "0")
{
echo "📊 **Scan Results:**"
echo "- **Critical:** $CRITICAL"
echo "- **High:** $HIGH"
echo "- **Medium:** $MEDIUM"
echo "- **Total:** $TOTAL"
echo ""
} >> "$GITHUB_STEP_SUMMARY"
if [ "$CRITICAL" -gt 0 ] || [ "$HIGH" -gt 0 ]; then
echo "⚠️ **Action Required:** Critical or High severity issues found" >> "$GITHUB_STEP_SUMMARY"
else
echo "✅ **No Critical or High severity issues found**" >> "$GITHUB_STEP_SUMMARY"
fi
{
echo ""
echo "📋 **Detailed Results:**"
echo "- Security tab: Results uploaded to GitHub Security tab"
echo "- Artifacts: Full JSON and SARIF reports available"
} >> "$GITHUB_STEP_SUMMARY"
else
echo "❌ Semgrep scan failed or no results generated" >> "$GITHUB_STEP_SUMMARY"
fi
+52
View File
@@ -0,0 +1,52 @@
name: UI Tests (Full 14 Shards via Label)
# Triggered by adding the `test:ui-full-shards` label to a PR.
# Calls docker-tests.yml with strict-mode=true so the sharded ui-tests
# matrix runs — it's otherwise gated on strict-mode and only fires at
# release time.
#
# NOTE on duplicated work: docker-tests.yml in strict mode runs its FULL
# job graph (image build, pytest, infra, etc.), so triggering this label
# re-runs everything that the normal PR docker-tests already ran. That's
# acceptable cost for a deliberate opt-in trigger; cheaper than refactoring
# docker-tests.yml to expose just ui-tests as a standalone reusable workflow.
on:
pull_request:
types: [labeled]
# Manual fallback — useful for one-off maintainer runs and for smoke-testing
# the wrapper when label-triggered workflows can't fire (e.g. on the PR that
# first introduces this file, before it lands on main).
workflow_dispatch:
# No concurrency group — intentionally omitted, matching docker-tests.yml.
# See its top-level comment for the rationale (#3554 / #3599).
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
full-shards:
name: Full UI Shards
# Fire on the label add, OR on any manual dispatch (maintainer override).
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
github.event.label.name == 'test:ui-full-shards')
uses: ./.github/workflows/docker-tests.yml
with:
strict-mode: true
permissions:
# MUST match the scopes docker-tests.yml's jobs DECLARE, not just the
# ones whose steps run. Reusable-workflow permission validation is
# compile-time: a called job that declares `contents: write` (the
# pytest/gh-pages job) fails the whole run at startup with
# "requesting 'contents: write', but is only allowed 'contents: none'"
# if the caller grants less — producing a 0-job startup_failure with
# no logs. #4209 dropped contents:write here for OSSF Scorecard, which
# silently broke this wrapper on every run since (the gh-pages STEP is
# runtime-gated to push@main, but the JOB still declares the scope).
# ci-gate.yml grants the same pair, which is why it works.
contents: write # docker-tests jobs checkout + declare gh-pages deploy scope
pull-requests: write # docker-tests pytest job posts PR comments
secrets:
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+73
View File
@@ -0,0 +1,73 @@
name: Update PDM dependencies
on:
workflow_call:
inputs:
pdm_args:
description: Arguments to pass to pdm
type: string
default: ''
workflow_dispatch:
inputs:
pdm_args:
description: Arguments to pass to pdm lock
type: string
default: ''
schedule:
- cron: '0 8 * * 3' # every Wednesday at 08:00 UTC
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
update:
name: 👛 Update with PDM
runs-on: ubuntu-latest
permissions:
contents: write # Required for creating PRs
pull-requests: write # Required for creating PRs
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: 📰 Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
ref: main
- name: 📦 Setup PDM
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
# Pin to the floor of `requires-python` in pyproject.toml.
# Resolving on the lowest supported interpreter guarantees the
# chosen package versions are installable across the whole
# supported range. Resolving on a newer interpreter (e.g. 3.x)
# can pick packages that dropped support for the floor, producing
# a lockfile that fails `pdm lock --check` downstream (see #3480).
python-version: '3.12'
cache: true
- name: 👚 Update to latest compatible versions
env:
PDM_ARGS: ${{ inputs.pdm_args || '' }}
run: |
# Intentional word splitting for PDM args
# shellcheck disable=SC2086
pdm lock $PDM_ARGS
- name: 📝 Create pull request
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
branch: update-dependencies-${{ github.run_number }}
title: 🤖 Update dependencies
body: |
This PR updates dependencies to their latest compatible versions.
This PR was created by the PDM Update Bot.
sign-commits: true
add-paths: pdm.lock
commit-message: 🤖 Update dependencies
labels: maintenance
draft: false
base: main
reviewers: djpetti,HashedViking,LearningCircuit
@@ -0,0 +1,158 @@
name: Update NPM dependencies
on:
workflow_call:
inputs:
npm_args:
description: Arguments to pass to npm update
type: string
default: --save
workflow_dispatch:
inputs:
npm_args:
description: Arguments to pass to npm update
type: string
default: --save
schedule:
- cron: '0 8 * * 4' # every Thursday at 08:00 UTC (day after PDM updates)
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
update:
name: 📦 Update with NPM
runs-on: ubuntu-latest
permissions:
contents: write # Required for creating PRs
pull-requests: write # Required for creating PRs
strategy:
matrix:
npm_directory:
- { path: '.', build_cmd: 'npm run build', test_cmd: '', cache_path: 'package-lock.json' }
- { path: 'tests', build_cmd: '', test_cmd: 'echo "Skipping API-key tests in CI - requires real API key"', cache_path: 'package-lock.json' }
- { path: 'tests/ui_tests', build_cmd: '', test_cmd: 'echo "Skipping UI tests in CI - requires server"', cache_path: 'package-lock.json' }
- { path: 'tests/puppeteer', build_cmd: '', test_cmd: 'echo "Skipping Puppeteer tests in CI - requires server"', cache_path: 'package-lock.json' }
- { path: 'tests/api_tests_with_login', build_cmd: '', test_cmd: 'echo "Skipping API-with-login tests in CI - requires server + auth"', cache_path: 'package-lock.json' }
- { path: 'tests/infrastructure_tests', build_cmd: '', test_cmd: 'echo "Skipping infrastructure tests in CI - requires server"', cache_path: 'package-lock.json' }
- { path: 'tests/accessibility_tests', build_cmd: '', test_cmd: 'echo "Skipping accessibility tests in CI - requires server"', cache_path: 'package-lock.json' }
- { path: 'tests/ui_tests/playwright', build_cmd: '', test_cmd: 'echo "Skipping playwright UI tests in CI - requires server"', cache_path: 'package-lock.json' }
continue-on-error: false
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: 📰 Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
ref: main
- name: 🟢 Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '24'
cache: 'npm'
cache-dependency-path: ${{ matrix.npm_directory.path }}/${{ matrix.npm_directory.cache_path }}
- name: 🔍 Security audit
env:
NPM_DIR: ${{ matrix.npm_directory.path }}
run: |
cd "$NPM_DIR"
# Don't fail the workflow if unfixable vulns remain — the next step
# (`npm audit fix --package-lock-only`) will attempt the auto-fix and
# report what it couldn't fix. Failing here would skip the dependency
# update entirely on the first unpatchable advisory. Use a ::warning
# workflow command (not a bare echo) so the soft-fail surfaces as a
# visible annotation in the run UI instead of getting lost in the
# step log.
npm audit --audit-level moderate || echo "::warning::Vulnerabilities detected — proceeding to auto-fix attempt"
- name: 🔐 Fix security vulnerabilities (lockfile only)
env:
NPM_DIR: ${{ matrix.npm_directory.path }}
run: |
cd "$NPM_DIR"
# Update lockfile only - actual install happens via npm ci below
npm audit fix --package-lock-only || echo "Some vulnerabilities could not be auto-fixed"
- name: 👚 Update to latest compatible versions (lockfile only)
env:
NPM_DIR: ${{ matrix.npm_directory.path }}
NPM_ARGS: ${{ inputs.npm_args || '--save' }}
run: |
cd "$NPM_DIR"
# Update lockfile only - actual install happens via npm ci below
# Intentional word splitting for npm args
# shellcheck disable=SC2086
npm update --package-lock-only $NPM_ARGS
- name: 📦 Install from lockfile with integrity verification
env:
NPM_DIR: ${{ matrix.npm_directory.path }}
run: |
cd "$NPM_DIR"
# npm ci installs from lockfile with integrity hash verification
# This satisfies OSSF Scorecard pinned-dependencies requirement
npm ci
- name: 🔨 Build (if applicable)
env:
NPM_DIR: ${{ matrix.npm_directory.path }}
BUILD_CMD: ${{ matrix.npm_directory.build_cmd }}
run: |
cd "$NPM_DIR"
eval "$BUILD_CMD"
if: matrix.npm_directory.build_cmd != ''
- name: 🧪 Test (if applicable)
env:
NPM_DIR: ${{ matrix.npm_directory.path }}
TEST_CMD: ${{ matrix.npm_directory.test_cmd }}
run: |
cd "$NPM_DIR"
eval "$TEST_CMD"
if: matrix.npm_directory.test_cmd != ''
- name: 📝 Check for changes
id: verify-changed-files
env:
NPM_DIR: ${{ matrix.npm_directory.path }}
run: |
cd "$NPM_DIR"
# Check if package.json or package-lock.json have changed
if git diff --quiet package.json package-lock.json; then
echo "changed=false" >> "$GITHUB_OUTPUT"
else
echo "changed=true" >> "$GITHUB_OUTPUT"
echo "path=${{ matrix.npm_directory.path }}" >> "$GITHUB_OUTPUT"
fi
- name: 📝 Create pull request
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
if: steps.verify-changed-files.outputs.changed == 'true'
with:
branch: update-npm-dependencies-${{ github.run_number }}
title: 🤖 Update dependencies
body: |
This PR updates dependencies to their latest compatible versions.
**Updated directory:** `${{ steps.verify-changed-files.outputs.path }}`
**Changes include:**
- Security vulnerability fixes (moderate severity and above)
- Compatible version updates for dependencies
- Updated lock files for reproducible builds
This PR was created by the Dependency Update Bot.
sign-commits: true
add-paths: |
${{ steps.verify-changed-files.outputs.path }}/package.json
${{ steps.verify-changed-files.outputs.path }}/package-lock.json
commit-message: 🤖 Update dependencies
labels: maintenance
draft: false
base: main
reviewers: djpetti,HashedViking,LearningCircuit
@@ -0,0 +1,78 @@
name: Update Pre-commit Hooks
on:
workflow_dispatch:
schedule:
- cron: '0 8 * * 5' # every Friday at 08:00 UTC (day after NPM updates)
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
update:
name: 🪝 Update Pre-commit Hooks
runs-on: ubuntu-latest
permissions:
contents: write # Required for creating PRs
pull-requests: write # Required for creating PRs
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: 📰 Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
ref: main
- name: 🐍 Setup Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.x'
- name: 📦 Install pre-commit-update
run: |
python -m pip install pip==25.0 \
--hash=sha256:b6eb97a803356a52b2dd4bb73ba9e65b2ba16caa6bcb25a7497350a4e5859b65
pip install pre-commit-update==0.6.1 \
--hash=sha256:db00891b3384776daaaa5721fd54a448ded19daf87635a3c77b7508eaf7d1634
- name: 🔄 Update pre-commit hooks (stable versions only)
run: |
# pre-commit-update skips alpha/beta/rc versions by default
pre-commit-update
- name: 📝 Check for changes
id: verify-changed-files
run: |
if git diff --quiet .pre-commit-config.yaml; then
echo "changed=false" >> "$GITHUB_OUTPUT"
echo "No changes detected in .pre-commit-config.yaml"
else
echo "changed=true" >> "$GITHUB_OUTPUT"
echo "Changes detected in .pre-commit-config.yaml"
echo "## Changes:" >> "$GITHUB_STEP_SUMMARY"
git diff .pre-commit-config.yaml >> "$GITHUB_STEP_SUMMARY"
fi
- name: 📝 Create pull request
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
if: steps.verify-changed-files.outputs.changed == 'true'
with:
branch: update-precommit-hooks-${{ github.run_number }}
title: 🤖 Update pre-commit hooks
body: |
This PR updates pre-commit hooks to their latest versions.
**Changes include:**
- Updated hook versions for better linting and bug fixes
- Ensures consistency with latest tool versions
This PR was created by the Pre-commit Update Bot.
sign-commits: true
add-paths: .pre-commit-config.yaml
commit-message: 🤖 Update pre-commit hooks
labels: maintenance
draft: false
base: main
reviewers: djpetti,HashedViking,LearningCircuit
@@ -0,0 +1,162 @@
name: Validate Docker Image Pinning
on:
pull_request:
branches: [main]
paths:
- '**/Dockerfile*'
- '**/docker-compose*.yml'
- '**/docker-compose*.yaml'
- '.github/workflows/*.yml'
- '.github/workflows/*.yaml'
- '.github/workflows/validate-image-pinning.yml'
- '.github/scripts/validate-docker-compose-images.sh'
- '.github/scripts/validate-workflow-images.py'
workflow_call: # Called by ci-gate.yml for release pipeline
workflow_dispatch:
# No concurrency group — intentionally omitted.
# This workflow triggers on both pull_request and workflow_call (from
# ci-gate.yml / release-gate.yml). A shared concurrency key would cause
# direct PR runs and workflow_call runs to cancel each other mid-flight.
# See #3554 (reverted in #3599) for context.
permissions: {} # Minimal permissions for OSSF Scorecard
jobs:
validate-docker-compose:
name: Validate docker-compose Images
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Validate docker-compose image pinning
run: |
chmod +x .github/scripts/validate-docker-compose-images.sh
.github/scripts/validate-docker-compose-images.sh
validate-workflow-images:
name: Validate Workflow Service Containers
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install PyYAML
run: pip install pyyaml==6.0.2
- name: Validate workflow image pinning
run: |
chmod +x .github/scripts/validate-workflow-images.py
python .github/scripts/validate-workflow-images.py
summary:
name: Image Pinning Validation Summary
needs: [validate-docker-compose, validate-workflow-images]
runs-on: ubuntu-latest
if: always()
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
# zizmor: ignore[template-injection] - values passed via env vars, not interpolated in shell
- name: Generate summary
env:
DOCKER_COMPOSE_RESULT: ${{ needs.validate-docker-compose.result }}
WORKFLOW_IMAGES_RESULT: ${{ needs.validate-workflow-images.result }}
run: |
{
echo "## 🔒 Docker Image Pinning Validation"
echo ""
echo "### What is Image Pinning?"
echo ""
echo "Pinning Docker images with SHA256 digests ensures:"
echo ""
echo "- 🔒 **Security**: Protection against supply chain attacks"
echo "- 🔄 **Reproducibility**: Exact same image bytes every time"
echo "- 🛡️ **Immutability**: Tags like \`:latest\` can be changed, but SHA digests cannot"
echo ""
echo "### Validation Results"
echo ""
if [ "$DOCKER_COMPOSE_RESULT" = "success" ] && \
[ "$WORKFLOW_IMAGES_RESULT" = "success" ]; then
echo "✅ **All images properly pinned with SHA256 digests**"
echo ""
echo "All docker-compose and workflow files pass validation."
else
echo "❌ **Image pinning violations found**"
echo ""
if [ "$DOCKER_COMPOSE_RESULT" != "success" ]; then
echo "- ❌ docker-compose files have unpinned images"
fi
if [ "$WORKFLOW_IMAGES_RESULT" != "success" ]; then
echo "- ❌ Workflow files have unpinned service containers"
fi
echo ""
echo "See job logs above for details and fix instructions."
fi
echo ""
echo "### How to Fix Unpinned Images"
echo ""
echo "\`\`\`bash"
echo "# 1. Pull the image"
echo "docker pull <image:tag>"
echo ""
echo "# 2. Get the SHA digest"
echo "docker inspect <image:tag> | jq -r '.[0].RepoDigests[0]'"
echo ""
echo "# 3. Update your file"
echo "image: <image:tag>@sha256:..."
echo "\`\`\`"
echo ""
echo "### Resources"
echo ""
echo "- [OSSF Scorecard - Pinned Dependencies](https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies)"
echo "- [Docker Image Digests Documentation](https://docs.docker.com/engine/reference/commandline/pull/#pull-an-image-by-digest-immutable-identifier)"
} >> "$GITHUB_STEP_SUMMARY"
- name: Fail if validation found issues
env:
DOCKER_COMPOSE_RESULT: ${{ needs.validate-docker-compose.result }}
WORKFLOW_IMAGES_RESULT: ${{ needs.validate-workflow-images.result }}
run: |
if [ "$DOCKER_COMPOSE_RESULT" != "success" ] || \
[ "$WORKFLOW_IMAGES_RESULT" != "success" ]; then
echo "::error::Image pinning validation failed"
exit 1
fi
+203
View File
@@ -0,0 +1,203 @@
name: Version Auto-Bump
# Run AFTER merge to main, creates a PR for version bump
# Benefits:
# - Respects branch protection rules (no direct push to main)
# - PR checks run once without restarts
# - No version merge conflicts between parallel PRs
# - Version bumps are visible and reviewable
# - Uses fixed branch name to avoid clutter
#
# Manual dispatch supports minor/major bumps via the Actions tab dropdown.
# Auto-triggered runs (push to main) always bump patch.
on:
push:
branches:
- main
paths:
- 'src/**'
- 'pdm.lock'
- 'pyproject.toml'
- 'package.json'
- 'package-lock.json'
- 'Dockerfile'
workflow_dispatch:
inputs:
release_type:
description: 'Version bump type'
required: true
type: choice
options:
- patch
- minor
- major
default: 'patch'
permissions: {}
jobs:
version-bump:
runs-on: ubuntu-latest
# Skip if this push is already an auto-bump commit
if: "!contains(github.event.head_commit.message, 'chore: auto-bump version')"
permissions:
contents: write
pull-requests: write
actions: read # for generate_workflow_status.py to read run history
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 2
persist-credentials: false
- name: Determine release type
id: release
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
release_type="${{ inputs.release_type }}"
else
release_type="patch"
fi
echo "type=$release_type" >> "$GITHUB_OUTPUT"
echo "Release type: $release_type"
- name: Check if version was bumped in this push
id: check
if: github.event_name != 'workflow_dispatch'
run: |
# Check if version changed in the commits being pushed
if git diff HEAD~1 -G"__version__" -- src/local_deep_research/__version__.py | grep -E '\+.*__version__.*='; then
echo "Version was manually bumped in this push"
echo "needs_bump=false" >> "$GITHUB_OUTPUT"
else
echo "Version not bumped, will auto-bump"
echo "needs_bump=true" >> "$GITHUB_OUTPUT"
fi
- name: Set up Python
if: steps.check.outputs.needs_bump != 'false'
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Set up PDM with pdm-bump plugin
if: steps.check.outputs.needs_bump != 'false'
uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5
with:
python-version: '3.12'
- name: Install pdm-bump plugin
if: steps.check.outputs.needs_bump != 'false'
run: |
pdm self add pdm-bump
- name: Bump version
if: steps.check.outputs.needs_bump != 'false'
id: bump
run: |
release_type="${{ steps.release.outputs.type }}"
# Get current version before bump (prefer pdm, fallback to grep)
current_version=$(pdm show --version 2>/dev/null || grep -oP '(?<=__version__ = ")[^"]*' src/local_deep_research/__version__.py)
echo "Current version: $current_version"
echo "current_version=$current_version" >> "$GITHUB_OUTPUT"
# Bump version using pdm-bump
echo "Bumping $release_type version..."
pdm bump "$release_type"
# Get new version after bump (prefer pdm, fallback to grep)
new_version=$(pdm show --version 2>/dev/null || grep -oP '(?<=__version__ = ")[^"]*' src/local_deep_research/__version__.py)
echo "New version: $new_version"
echo "new_version=$new_version" >> "$GITHUB_OUTPUT"
# Sync package.json version with __version__.py
jq --arg v "$new_version" '.version = $v' package.json > package.json.tmp
mv package.json.tmp package.json
echo "Updated package.json version to $new_version"
- name: Regenerate configuration docs
if: steps.check.outputs.needs_bump != 'false'
run: python scripts/generate_config_docs.py
# Refreshes docs/ci/workflow-status.md so the version-bump PR
# carries the current snapshot. Output uses coarse "last week /
# last month" buckets so within-day reruns produce zero diff —
# the diff that lands in this PR only shows workflows whose
# bucket has actually shifted since the previous release.
#
# ~340 GitHub API calls per run (well under the GITHUB_TOKEN
# 1000/hr workflow-runs limit). Needs `actions: read` on the
# job permissions block above.
- name: Regenerate workflow status dashboard
if: steps.check.outputs.needs_bump != 'false'
# Don't block the version bump if the dashboard refresh fails.
# The regen calls ~340 GitHub API endpoints; a transient outage
# or rate-limit hit would otherwise prevent the version-bump PR
# from being created at all. On failure the dashboard just stays
# at its previous snapshot until the next successful run.
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
pip install pyyaml==6.0.3
python scripts/generate_workflow_status.py
- name: Generate PR body
if: steps.check.outputs.needs_bump != 'false'
id: pr_body
run: |
release_type="${{ steps.release.outputs.type }}"
current="${{ steps.bump.outputs.current_version }}"
new="${{ steps.bump.outputs.new_version }}"
body_file=$(mktemp)
{
echo "## Summary"
echo "- Bump **${release_type}** version: ${current} → ${new}"
echo ""
if [ "$release_type" = "major" ]; then
echo "> [!CAUTION]"
echo "> This is a **major** version bump. It signals breaking changes."
echo "> Please review carefully before merging."
elif [ "$release_type" = "minor" ]; then
echo "> [!IMPORTANT]"
echo "> This is a **minor** version bump, indicating new features or significant changes."
fi
echo ""
if [ "$release_type" = "patch" ]; then
echo "This PR was automatically created by the version bump workflow."
else
echo "This PR was manually triggered via the Actions tab."
fi
echo "Approve and merge to trigger a new release."
echo ""
echo "Configuration docs (docs/CONFIGURATION.md) and the workflow status dashboard (docs/ci/workflow-status.md) have been regenerated."
} > "$body_file"
echo "body_file=$body_file" >> "$GITHUB_OUTPUT"
# Use peter-evans/create-pull-request with GITHUB_TOKEN so the PR is created
# by github-actions[bot] - this allows the repo owner to approve the PR
- name: Create Pull Request
if: steps.check.outputs.needs_bump != 'false'
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
token: ${{ secrets.GITHUB_TOKEN }}
commit-message: "chore: auto-bump version to ${{ steps.bump.outputs.new_version }}"
branch: chore/auto-version-bump
delete-branch: true
title: "chore: bump ${{ steps.release.outputs.type }} version to ${{ steps.bump.outputs.new_version }}"
body-path: ${{ steps.pr_body.outputs.body_file }}
labels: |
automation
maintenance
+37
View File
@@ -0,0 +1,37 @@
name: Vulture Dead Code Detection
on:
workflow_call: # Allows this workflow to be called by other workflows
workflow_dispatch:
# Top-level permissions set to minimum (OSSF Scorecard Token-Permissions)
permissions: {}
jobs:
vulture:
runs-on: ubuntu-latest
timeout-minutes: 5
name: Scan for dead code
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install vulture
run: pip install "vulture~=2.14"
- name: Run vulture (src/ only)
run: vulture src/local_deep_research/ vulture_whitelist.py --min-confidence 80
+110
View File
@@ -0,0 +1,110 @@
name: Welcome first-time contributors
# Posts a single welcome comment on a contributor's FIRST PR (filtered per
# user). Uses pull_request_target so forked PRs receive a writable token;
# the script reads only sender.login (operator-trusted via GitHub) and
# never executes fork-controlled content — no checkout, no shell.
#
# We do NOT use actions/first-interaction: its isFirstPullRequest check
# has no author filter (lists all repo PRs and matches only on the lowest
# PR number), so it would never fire on a repo with prior history.
on:
# zizmor: ignore[dangerous-triggers] — pull_request_target is required so
# fork PRs receive a writable token to post the welcome comment. The job
# never checks out PR content, never runs fork-controlled scripts, and
# only reads `sender.login` (operator-trusted GitHub event metadata). The
# comment body is a static template with no PR-controlled interpolation.
pull_request_target:
types: [opened]
permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions
jobs:
welcome:
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
# createComment on a PR returns 403 with only `issues: write` —
# GitHub requires `pull-requests: write` when the issue resource
# is actually a PR (Accepted-Permissions header lists both).
issues: write
pull-requests: write
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Welcome first-time contributor
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
// Iterative pattern from actions/github-script@v9.0.0 README:
// skip the just-opened PR by number and return early if ANY
// other PR by this author exists. Robust to GitHub's eventual
// consistency on listForRepo (the just-opened PR may not yet
// be indexed). Using sender.login (canonical) — for `opened`
// it equals pull_request.user.login.
const sender = context.payload.sender;
// Skip bots. user.type === 'Bot' is the canonical signal;
// also belt-and-suspenders the [bot]-suffix check (consistent
// with the PR triage workflow's KNOWN_BOTS handling).
if (sender.type === 'Bot' || sender.login.endsWith('[bot]')) {
console.log(`Sender ${sender.login} is a bot. Skipping.`);
return;
}
const opts = github.rest.issues.listForRepo.endpoint.merge({
...context.issue,
creator: sender.login,
state: 'all',
});
const items = await github.paginate(opts);
for (const item of items) {
if (item.number === context.issue.number) continue;
if (item.pull_request) {
console.log(
`Sender ${sender.login} already has PR #${item.number}. Skipping.`,
);
return;
}
}
const body = [
`Welcome to local-deep-research — and thank you for your contribution. We genuinely appreciate the time you're putting in.`,
``,
`Here's a starter pack to help your PR move smoothly:`,
``,
`**Get set up locally**`,
`- [Installation guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/installation.md) — running LDR`,
`- [Developer guide](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/developing.md) — setting up a dev environment`,
`- Install our pre-commit hooks (one-time, [details in CONTRIBUTING.md](https://github.com/LearningCircuit/local-deep-research/blob/main/CONTRIBUTING.md#-quick-start)):`,
` \`\`\``,
` pre-commit install`,
` pre-commit install-hooks`,
` \`\`\``,
``,
`**Understand the codebase**`,
`- [Architecture overview](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/architecture.md) — how the pieces fit together`,
`- [Tests README](https://github.com/LearningCircuit/local-deep-research/blob/main/tests/README.md) — how to run the test suite locally`,
`- [FAQ](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/faq.md) and [Troubleshooting](https://github.com/LearningCircuit/local-deep-research/blob/main/docs/troubleshooting.md) — common setup, config, and operational issues`,
`- Found a security issue? See [SECURITY.md](https://github.com/LearningCircuit/local-deep-research/blob/main/SECURITY.md) for the responsible-disclosure process — please don't open a public PR for it.`,
``,
`**Before you ask for review**`,
`- Open PRs against \`main\` unless a maintainer says otherwise.`,
`- Confirm your PR has a single clear purpose ([PR process](https://github.com/LearningCircuit/local-deep-research/blob/main/CONTRIBUTING.md#-pull-request-process)).`,
`- Describe in your PR body what you tested by hand — "CI is green" alone isn't enough.`,
`- Drop by [Discord](https://discord.gg/ttcqQeFcJ3) if you'd like to chat with maintainers or other contributors.`,
``,
`A maintainer will take a look — if you don't hear back within 7 days, feel free to ping.`,
].join('\n');
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number,
body,
});
+65
View File
@@ -0,0 +1,65 @@
name: Zizmor GitHub Actions Security
on:
workflow_dispatch:
workflow_call: # Called by release-gate.yml
schedule:
# Run security audit weekly on Monday at 9 AM UTC
- cron: '0 9 * * 1'
permissions:
contents: read
jobs:
zizmor-scan:
name: Zizmor Workflow Security Scan
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
steps:
- name: Harden Runner
uses: step-security/harden-runner@9af89fc71515a100421586dfdb3dc9c984fbf411 # v2.19.4
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Run zizmor security scan
uses: zizmorcore/zizmor-action@192e21d79ab29983730a13d1382995c2307fbcaa # v0.5.7
with:
inputs: .github/workflows/
min-severity: low
advanced-security: true
token: ${{ secrets.GITHUB_TOKEN }}
- name: Display scan summary
if: always()
run: |
{
echo "## Zizmor GitHub Actions Security Scan"
echo ""
echo "### What is Zizmor?"
echo "Zizmor is a security linter for GitHub Actions workflows that detects:"
echo ""
echo "- **Template Injection**: Attacker-controlled input in expressions"
echo "- **ArtiPACKED**: Credentials leaked in workflow artifacts"
echo "- **Vulnerable Actions**: Third-party actions with known CVEs"
echo "- **Impostor Commits**: Hash-pinned actions pointing to forks"
echo "- **Hardcoded Credentials**: Plain-text secrets in workflows"
echo "- **Excessive Permissions**: Over-privileged workflow tokens"
echo "- **Unpinned Actions**: Mutable action references"
echo "- **Cache Poisoning**: Unsafe cache usage patterns"
echo ""
echo "### Resources"
echo "- [Zizmor GitHub](https://github.com/woodruffw/zizmor)"
echo "- [Security Tab](https://github.com/${{ github.repository }}/security/code-scanning)"
echo "- Results uploaded to GitHub Security tab for detailed analysis"
} >> "$GITHUB_STEP_SUMMARY"
+395
View File
@@ -0,0 +1,395 @@
# WHITELIST APPROACH - Only allow specific file types
# Deny everything by default
*
# Allow directories (needed for git to traverse)
!*/
# Allow specific source code files (but not .py in root - see below)
!*.js
!*.html
!*.css
# Auto-generated theme CSS (regenerated on every server start from themes/*/*.css)
src/local_deep_research/web/static/css/themes.css
# JSON — path-scoped (root package.json whitelisted below)
!src/local_deep_research/defaults/**/*.json
!cookiecutter-docker/**/*.json
# YAML — non-root, path-scoped (.github/ and .semgrep/ whitelisted separately)
!community_benchmark_results/**/*.yaml
!cookiecutter-docker/**/*.yml
!examples/elasticsearch/docker-compose.yml
# Shell scripts — path-scoped (.github/ and .pre-commit-hooks/ whitelisted separately)
!scripts/**/*.sh
!tests/**/*.sh
!docs/**/*.sh
!examples/**/*.sh
# Templates, Jinja2, XML — explicit paths
# NOTE: .env.template has a leading-dot basename, so the `.*` catch-all
# below would re-ignore it -- it is re-allowed in the post-catch-all block
# instead (a negation here would be a dead rule).
!src/local_deep_research/notifications/templates/**/*.jinja2
!unraid-templates/**/*.xml
# Allow Python files everywhere except root
!**/*.py
# Block ALL files in root directory - adding files to root should be intentional
/*.*
# Re-allow root-level directories whose name contains a dot.
#
# Why: the `/*.*` pattern above is a glob that matches anything at the
# repo root containing a `.` — it correctly catches files like
# `something.txt`, but it ALSO matches DIRECTORIES whose name happens to
# include a dot (e.g. `changelog.d/`). gitignore's pattern engine does
# not distinguish files from directories at the pattern level unless the
# pattern ends in `/`.
#
# Per git's own gitignore rules:
# "It is not possible to re-include a file if a parent directory of
# that file is excluded."
# So later negations like `!changelog.d/**/*.md` below cannot
# rescue files inside an excluded parent. They silently fail, and new
# files in `changelog.d/` get ignored even though contributors expect
# them to be tracked. This surfaced when adding a new towncrier
# fragment.
#
# Fix: explicitly re-include the directory ITSELF so the parent isn't
# excluded, leaving the per-file negation below to do its job. If a new
# dotted directory is added at root in the future, it needs the same
# treatment here.
!changelog.d/
# Explicitly allow specific Python files in root that are needed
# (Comment out any you don't need)
# Allow specific project files
!LICENSE
!README
!README.md
!CHANGELOG.md
!CONTRIBUTING.md
!SECURITY.md
!LICENSE.md
!Dockerfile
!pyproject.toml
!pdm.lock
!package.json
!MANIFEST.in
!eslint.config.mjs
!eslint.config.js
!playwright.config.js
!vite.config.js
!vulture_whitelist.py
!docker-compose.yml
# Block JSON files in root directory (except package.json which is explicitly allowed above)
/*.json
!package.json
!lighthouserc.json
# Block all other dot files/folders
.*
.*/
!.github/
!.github/**/*.yml
!.github/**/*.yaml
!.github/**/*.md
!.github/**/*.sh
!.github/**/*.py
!.github/CODEOWNERS
!.github/security/*.txt
# Config dotfiles re-allowed AFTER the `.*`/`.*/` catch-all above. A `!`
# negation placed BEFORE the catch-all is a dead rule -- the catch-all
# re-ignores the file, which then stays tracked only if it was committed
# before the catch-all existed (git never ignores an already-tracked path,
# which masks the bug). Put every new dot-config negation in THIS block --
# otherwise e.g. a nested `.gitignore` or a `newdir/.gitkeep` is silently
# dropped by `git add`.
!.gitignore
!.gitkeep
!.nvmrc
!.pre-commit-config.yaml
# Linter, whitelist, and security-scan config dotfiles -- masked by the
# catch-all above like the rest of this block; tracked today only because
# they predate it. .zap/ mirrors the .github/ / .semgrep/ pattern: re-allow
# the dir, then its file TYPE (.tsv) -- deliberately NOT a blanket `**`, so a
# stray key/db/pem dropped into .zap/ stays ignored (deny-by-default).
!.file-whitelist.txt
!.hadolint.yaml
!.yamllint.yaml
!.zap/
!.zap/**/*.tsv
# Shipped source template whose basename starts with a dot -- caught by the
# `.*` catch-all above, so it must be re-allowed here (not in the templates
# block near its non-dotfile siblings).
!src/local_deep_research/defaults/.env.template
!.gitleaksignore
!.grype.yaml
!.dockerignore
# Security-tool configs. Directory rules mirror the .github/ block above:
# re-allow the dir first, then its file types, so newly added
# .semgrep/rules/* files actually get tracked. (bearer.yml is not a dotfile
# -- the catch-all never affected it -- grouped here by topic.)
!.gitleaks.toml
!.semgrepignore
!bearer.yml
!.trivyignore
!.safety-policy.yml
!.semgrep/
!.semgrep/**/*.yml
!.semgrep/**/*.yaml
!.semgrep/**/*.md
# Pre-commit hook scripts (dir + file types), re-allowed after the catch-all.
!.pre-commit-hooks/
!.pre-commit-hooks/*.py
!.pre-commit-hooks/*.sh
# Allow installer files only in installers directory (text scripts only)
!installers/**/*.bat
!installers/**/*.ps1
!installers/**/*.iss
# Block specific problematic directories even if they contain allowed files
src/data/
src/research_outputs/
research_outputs/
data/
# Journal quality data: nothing is bundled. All sources are runtime-downloaded
# into the user data directory by the data_sources package. There is no
# package-level data directory any more.
logs/
cache/
temp/
tmp/
# Block result directories from examples
examples/optimization/examples/optimization/results/
examples/benchmarks/examples/benchmarks/results/
examples/*/results/
**/results/*/
# Block test output and JSON files
tests/**/results/
tests/**/*.json
tests/ui_tests/*.json
tests/ui_tests/results/
# …but the responsive baseline is checked in on purpose: it's the
# allowlist of currently-known UI bugs that test_responsive_ui_comprehensive.js
# compares against. New regressions vs this list fail the test.
!tests/ui_tests/responsive_baseline.json
# Still block Python cache and build artifacts even if they match patterns above
__pycache__/
**/__pycache__/
__pypackages__/
**/__pypackages__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Block virtual environments
.venv/
venv/
ENV/
env/
.env
.pdm*
# Block IDE files
.idea/
.vscode/
*.swp
*.swo
.DS_Store
.qodo/
.cursorrules
# Project specific
temp_output.txt
temp_output_findings.txt
formatted_output.txt
Output.txt
research_outputs
report.md
src/data/*
# Logs
*.log
logs/
# Database
research_history.db
data/ldr.db
*.db
# Data
*.pdf
.cache
*.jsonl
data/cache/pricing/pricing_cache.json
# Tools
python-db/
js-db/
*.sarif
codeql_analysis_results.txt
.coverage
coverage/
# benchmark results for examples
examples/benchmark_results/
# benchmark and optimization results
src/benchmark_results/
data/benchmark_results/
data/optimization_results/
examples/*/benchmark_results/
examples/*/optimization_*/
examples/benchmarks/results/
examples/optimization/results/
tests/import_tests/
benchmark_results/
strategy_benchmark_results/
strategy_sim_results_*/
*_results_*/
optimization_results_*/
optimization_demo_*/
# Utility scripts for benchmarks
update_api_key.py
update_max_tokens.py
verify_grader_config.py
/run_*_benchmark.sh
/run_fixed_benchmark.sh
/run_verification.sh
/run_update_*.sh
/run_*.sh
/*_benchmark.py
benchmark_cleanup.sh
benchmark_*summary.md
simulation_*summary.md
# Exceptions for benchmark example scripts
!examples/benchmarks/**/run_*.sh
!examples/benchmarks/**/*.py
**/.claude/settings.local.json
# Folders created by Docker.
local_collections/
# Node.js dependencies and artifacts
node_modules/
package-lock.json
!/package-lock.json
!tests/ui_tests/package-lock.json
!tests/ui_tests/playwright/package-lock.json
!tests/api_tests_with_login/package-lock.json
!tests/infrastructure_tests/package-lock.json
!tests/accessibility_tests/package.json
!tests/accessibility_tests/package-lock.json
!tests/puppeteer/package-lock.json
!tests/package-lock.json
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# Temporary debug files
debug_*.js
test_*.html
# Test screenshots and artifacts
*.png
charts_*.png
research_*.png
star_reviews_*.png
web*.png
tests/screenshots/
screenshots/
# Explicitly allowed binary files — keep this list small.
# Binary files bloat the repo; do NOT add new ones without maintainer approval.
!docs/images/Local Search.png
!docs/images/local_search_embedding_model_type.png
!docs/images/local_search_paths.png
!src/local_deep_research/web/static/favicon.png
!src/local_deep_research/web/static/sounds/error.mp3
!src/local_deep_research/web/static/sounds/success.mp3
# Block mobile UI test artifacts (html/json reports, screenshot directories)
tests/ui_tests/mobile/**/*.html
tests/ui_tests/mobile/**/*.json
tests/ui_tests/mobile/*-screenshots/
# Ignore cookiecutter-generated docker-compose files (root only).
/docker-compose.*.yml
!docker-compose.gpu.override.yml
!docker-compose.unraid.yml
# Ignore backup data directories (but not backup code modules)
/backup/
!src/**/backup/
!tests/**/backup/
# Security - ignore generated secret keys
.secret_key
.cache_key_secret
# Allow MD files only in specific directories
!docs/**/*.md
!examples/**/*.md
!cookiecutter-docker/**/*.md
# towncrier news fragments — directory itself is re-allowed near the
# `/*.*` rule above (consolidated so the re-include lives in one place).
!changelog.d/**/*.md
# Block MD files in tests directories by default
tests/**/*.md
# To whitelist specific test MD files, add them explicitly like:
# !tests/specific-file-to-include.md
# Whitelist Playwright test infrastructure
!tests/ui_tests/playwright/
!tests/ui_tests/playwright/package.json
!tests/ui_tests/playwright/README.md
!tests/ui_tests/playwright/playwright.config.js
!tests/ui_tests/playwright/tests/
!tests/ui_tests/playwright/tests/*.js
!tests/ui_tests/playwright/tests/helpers/
!tests/ui_tests/playwright/tests/helpers/*.js
# NOTE: Playwright snapshot PNGs are NOT tracked in git.
# Visual regression tests use --update-snapshots to generate baselines locally.
# See: tests/ui_tests/playwright/README.md
# Block Playwright authentication state (contains session cookies/tokens)
tests/ui_tests/playwright/.auth/
# Block Playwright generated reports and test artifacts
tests/ui_tests/playwright/playwright-report/
tests/ui_tests/playwright/test-results/
# Block accessibility test auth state and generated reports
tests/accessibility_tests/.auth/
tests/accessibility_tests/playwright-report/
tests/accessibility_tests/test-results/
tests/accessibility_tests/.lighthouseci/
+326
View File
@@ -0,0 +1,326 @@
title = "Gitleaks Configuration for Local Deep Research"
# NOTE: gitleaks is the sole secret scanner for this repository.
# detect-secrets was removed (2026-02) because its .secrets.baseline file
# (line-number-based tracking) caused constant merge conflicts across branches.
# gitleaks uses path-based and regex-based allowlists that are stable across
# line changes. CI also runs Semgrep (p/secrets) and Bearer (secrets) scanners.
# Do not re-add detect-secrets — use .gitleaks.toml allowlists instead.
# Extend default rules with custom ones
[extend]
useDefault = true
# Custom rules for this repository
[[rules]]
description = "GitHub personal access token"
id = "github-pat"
regex = '''ghp_[a-zA-Z0-9]{36}'''
keywords = ["ghp_"]
[[rules]]
description = "GitHub app token"
id = "github-app-token"
regex = '''(ghu|ghs|ghr)_[a-zA-Z0-9]{36}'''
keywords = ["ghu_", "ghs_", "ghr_"]
[[rules]]
description = "Google API key"
id = "google-api-key"
regex = '''AIza[0-9A-Za-z_-]{35}'''
keywords = ["AIza"]
[[rules]]
description = "Brave Search API key"
id = "brave-api-key"
regex = '''BS[a-zA-Z0-9_-]{32}'''
keywords = ["BS"]
[[rules]]
description = "SERPAPI key"
id = "serpapi-key"
regex = '''[a-zA-Z0-9]{64}'''
keywords = ["serpapi"]
[[rules]]
description = "Tavily API key"
id = "tavily-api-key"
regex = '''tvly-[a-zA-Z0-9_-]{40}'''
keywords = ["tvly-"]
[[rules]]
description = "SQLCipher database key"
id = "sqlcipher-key"
regex = '''PRAGMA key = ['"][a-zA-Z0-9/+]{44}['"]'''
keywords = ["PRAGMA key"]
[[rules]]
description = "Generic private key"
id = "private-key"
regex = '''-----BEGIN (RSA |OPENSSH |DSA |EC |PGP )?PRIVATE KEY-----'''
keywords = ["BEGIN PRIVATE KEY", "BEGIN RSA PRIVATE KEY", "BEGIN OPENSSH PRIVATE KEY"]
[[rules]]
description = "Generic secret/password pattern"
id = "generic-secret"
regex = '''(?i)(password|secret|token|key)\s*[:=]\s*['"]?([a-zA-Z0-9_\-@#$%^&*]{8,})['"]?'''
secretGroup = 2
keywords = ["password", "secret", "token", "key"]
# Allowlist - exclude common false positives
[allowlist]
description = "Global allowlist"
paths = [
'''\.git''',
'''\.venv''',
# detect-secrets baseline file (removed in PR #2476) contains SHA-1 hashes
# of known false positives — not actual secrets. Still appears in git history.
'''\.secrets\.baseline$''',
# Generated HTML documentation files (contain api_key parameter documentation, not secrets)
'''z_[a-f0-9]+_.*\.html$''',
'''node_modules''',
'''\.pytest_cache''',
'''__pycache__''',
'''coverage''',
'''\.coverage''',
'''test_results''',
'''screenshots''',
'''build''',
'''dist''',
'''\.mypy_cache''',
'''\.tox''',
'''tests/.*\.py$''',
'''tests/.*\.js$''',
'''examples/.*\.py$''',
'''.*test.*\.py$''',
'''.*test.*\.js$''',
'''.*example.*\.py$''',
'''mock.*\.py$''',
'''fixture.*\.py$''',
# Note: Workflow files are scanned - use specific commit allowlists for test credentials
'''src/local_deep_research/security/file_integrity/.*\.py$''',
'''src/local_deep_research/library/download_management/.*\.py$''',
'''src/local_deep_research/research_library/services/download_service\.py$''',
'''src/local_deep_research/research_library/routes/library_routes\.py$''',
'''src/local_deep_research/research_library/downloaders/.*\.py$''', # Contains API key parameters (not actual secrets)
'''src/local_deep_research/news/subscription_manager/scheduler\.py$''',
'''pdm\.lock$''', # Contains SHA256 package hashes (not secrets)
# LLM and embedding provider implementations use api_key/provider_key variable names (not actual secrets)
'''src/local_deep_research/llm/providers/.*\.py$''',
'''src/local_deep_research/embeddings/providers/.*\.py$''',
# LLM config dispatcher constructs ChatOpenAI/ChatOllama with api_key= kwargs (not actual secrets)
'''src/local_deep_research/config/llm_config\.py$''',
# Web search engines use api_key variable names (not actual secrets)
'''src/local_deep_research/web_search_engines/engines/.*\.py$''',
'''src/local_deep_research/web_search_engines/search_engine_factory\.py$''',
# search_engines_config resolves api_key from settings_snapshot (not actual secrets)
'''src/local_deep_research/web_search_engines/search_engines_config\.py$''',
# Benchmark service uses user_password parameter names (not actual secrets)
'''src/local_deep_research/benchmarks/web_api/benchmark_service\.py$''',
'''src/local_deep_research/web_search_engines/search_engine_base\.py$''',
# HTML templates use csrfToken variable names (not actual secrets)
'''src/local_deep_research/web/templates/.*\.html$''',
# Notification flow docs contain example code with parameter names
'''docs/NOTIFICATION_FLOW\.md$''',
# UI test files contain test credentials (intentional for testing)
'''tests/ui_tests/.*\.js$''',
# Puppeteer E2E test files contain test credentials (intentional for testing)
'''tests/puppeteer/.*\.js$''',
# CI scripts contain test credentials (intentional for testing)
'''scripts/ci/.*\.sh$''',
# Web routes use api_key/password parameter names (not actual secrets)
'''src/local_deep_research/web/routes/.*\.py$''',
'''src/local_deep_research/web/services/.*\.py$''',
# JavaScript files use csrfToken variable names (not actual secrets)
'''src/local_deep_research/web/static/js/.*\.js$''',
# Database session context uses password variable names (not actual secrets)
'''src/local_deep_research/database/session_context\.py$''',
# Database middleware passes password variables between Flask g context and session stores (not actual secrets)
'''src/local_deep_research/web/auth/database_middleware\.py$''',
# Password utility helper resolves session password (variable names, not actual secrets)
'''src/local_deep_research/web/auth/password_utils\.py$''',
# SQLCipher utils has key/password variable names inherent to encryption operations (not actual secrets)
'''src/local_deep_research/database/sqlcipher_utils\.py$''',
# Environment setting definitions use key= parameter names (not actual secrets)
'''src/local_deep_research/settings/env_definitions/.*\.py$''',
# RAG routes and services use db_password variable names (not actual secrets)
'''src/local_deep_research/research_library/routes/rag_routes\.py$''',
'''src/local_deep_research/research_library/services/library_rag_service\.py$''',
# RAG service factory passes db_password variable names (not actual secrets)
'''src/local_deep_research/research_library/services/rag_service_factory\.py$''',
# Background scheduler passes db_password variable names to DB sessions and
# the RAG factory (not actual secrets)
'''src/local_deep_research/scheduler/background\.py$''',
# Encrypted DB manager uses key derivation and password parameters (not actual secrets)
'''src/local_deep_research/database/encrypted_db\.py$''',
# Metrics search tracker uses password variable names for DB credentials (not actual secrets)
'''src/local_deep_research/metrics/search_tracker\.py$''',
# Auto-generated configuration docs reference env var names like ENCRYPTION_KEY (not actual secrets)
'''docs/CONFIGURATION\.md$''',
# Chat routes use password retrieval helper (same pattern as web routes, not actual secrets)
'''src/local_deep_research/chat/routes\.py$''',
'''src/local_deep_research/research_library/services/research_history_indexer\.py$''',
'''src/local_deep_research/research_library/search/services/research_history_indexer\.py$''',
'''src/local_deep_research/web/queue/processor_v2\.py$''',
# Auth routes pass password variables for DB operations (not actual secrets)
'''src/local_deep_research/web/auth/routes\.py$''',
# Backup service uses password variable for encryption key derivation (not actual secrets)
'''src/local_deep_research/database/backup/.*\.py$'''
]
files = [
"\\.env\\.example$",
"\\.env\\.template$",
"config\\.example\\..*$",
"docker-compose\\.example\\.yml$"
]
commits = [
# AI reviewer documentation commit with API key explanation
"b31ff6fec15c9fe6f523d1f93f23e321b8803799",
# PDF upload feature with CSRF token DOM reading
"e0660dd0da5efed3ddbfcf5cfa1985b39f33df16",
# Download system integration with encrypted database - legitimate password parameters
"65ebe6bd522152ebb00a1fce38b8f6acb8fdcc7d",
# OpenRouter documentation commits with placeholder API keys (fixed in later commits)
"1395290fa5e8a955d5e01ed056e3ee7bad1b888c",
"00f48008cfedcc01adedc2b3f84246e0e3d883ef",
"17e4404af5b891b78e7eda99c196f03710e7107b",
# Notification system commit with user_password parameter names (false positives)
"17762f9e1b48db91092a867195aca81cfce6ab9c",
# PDM lock file update with SHA256 content hashes (false positives)
"3dd78e593550623af172c4d9f857197fdb1dfc99",
# Old .env.template commits with placeholder values (not actual secrets)
"40ca8fff073c20291758841c334081831f75a224",
"02042634506c377e86161cc2ce038eb8c19f10f3",
"03eba268505e3f239643041ce9c3d367d9b42700",
"08eaba75f4e3fb46f72668d3cc1f1d5eb4254f6b",
# README.md commits with placeholder API key examples (not actual secrets)
"27539371e7d2b2b6edb63bc238a6c0347fafef0c",
"792da694efe22304469ec22ff46201080791e3a6",
# Generated HTML documentation files with api_key parameter docs
"9acf5c42fb7ef721499e6d3a637f4faac69bc96b",
# Coverage HTML report files with api_key parameter names in docstrings (false positives)
"bdf812d1dd0efa81971d7500593e5c25339c676c",
# full_serp_search_results.py example code with placeholder API key comments
"f4a447e3cf4bb53ee5a88384dd8fd4c1ce2a9b4c",
"a831a13c5a43cf34eaf3c52495db2e4c2a0005b7",
# openai_endpoint provider with api_key = "not-needed" placeholder (not a real secret)
"e1b7322c73165be9ac651804b7babecc85c0d7df"
]
regexes = [
# Allow placeholder values that are clearly not real secrets
'''your-.*-key-here''',
'''your-api-key-here''',
'''your-openai-key-here''',
'''your-google-key-here''',
# Allow comments that explain API key usage but don't contain actual keys
'''# API_KEY: OpenRouter API key''',
'''OPENROUTER_API_KEY''',
# Allow JavaScript variable declarations that happen to use "token" as a variable name
'''const token = document\.querySelector''',
'''let token = document\.querySelector''',
'''var token = document\.querySelector''',
# Allow CSRF token reading from DOM meta tag (standard web CSRF protection pattern)
'''const csrfToken = document\.querySelector\(['"](meta\[name=)''',
# Allow test credentials used in CI/CD workflows
'''testpass123''',
'''perf-test-secret-key''',
# Password parameters and variable assignments (not actual secrets)
'''self\.password = password''',
'''self\.username = username''',
# Flask g context password variable assignment (not actual secrets)
'''g\.user_password\s*=\s*password''',
'''password: Optional\[str\]''',
'''username: str, password: Optional''',
# Database session context - parameter passing
'''get_user_db_session\(self\.username, self\.password\)''',
'''get_user_db_session\(username, password\)''',
'''username=username, password=password''',
# Session password storage (encrypted in session)
'''user_password = session_password_store''',
'''get_session_password''',
'''user_password = getattr''',
# Allow specific password variable patterns with safe context (not arbitrary assignments)
'''user_password\s*=\s*getattr\(g''',
'''user_password\s*=\s*session_password_store\.get_session_password''',
'''user_password\s*=\s*user_password''',
'''user_password=user_password''',
'''session_password_store\.get_session_password''',
'''password\)\s*:''',
# Allow type hints only (not assignments)
'''\bpassword:\s*Optional\[str\]\s*[=)]''',
'''\bpassword:\s*str\s*[=)]''',
'''password\s*:\s*Optional\[str\]\s*=\s*None''',
# Allow getattr patterns for password retrieval from Flask g context
'''getattr\(g,\s*["']user_password["']''',
# Allow SHA256 hashes in pdm.lock (package integrity hashes)
'''hash\s*=\s*"sha256:[a-fA-F0-9]{64}"''',
'''content_hash\s*=\s*"sha256:[a-fA-F0-9]{64}"''',
# Allow CSRF token meta tag selectors (JavaScript, not actual secrets)
'''csrf_token\s*=\s*document\.querySelector\(['"](meta\[name=)''',
# Allow csrfToken variable assignment in JavaScript/HTML
'''const csrfToken = document\.querySelector''',
'''let csrfToken = document\.querySelector''',
# Allow api_key variable assignment from settings (not actual secrets)
'''api_key\s*=\s*get_setting_from_snapshot''',
'''self\.api_key\s*=\s*settings_snapshot\.get''',
'''self\.api_key\s*=\s*get_setting_from_snapshot''',
# Allow provider_key constant definitions
'''provider_key\s*=\s*"[A-Z_]+"''',
# Allow test credentials in test files (pragma comments indicate intentional)
'''Test@Pass123!''',
'''TestPass123!''',
# Allow api_key type hints and docstrings
'''api_key:\s*Optional''',
'''api_key\s*=\s*None''',
# Allow placeholder API keys for local servers without auth.
# "not-required" is the unified placeholder (providers/base.py
# OPTIONAL_API_KEY_PLACEHOLDER). The other strings are kept for
# backward compat with any third-party provider classes that haven't
# migrated to the constant yet.
'''api_key\s*=\s*"not-required"''',
'''api_key\s*=\s*"not-needed"''',
'''api_key\s*=\s*"lm-studio"''',
# Allow research_id, user_password in notification flow docs (example code)
'''research_id,\s*user_password''',
# Allow CSRF token patterns in HTML templates
'''csrf[Tt]oken\s*=\s*document\.querySelector''',
# Thread context password retrieval (session_context.py)
'''thread_context\.get\("user_password"\)''',
'''thread_context\["user_password"\]''',
'''if not password:''',
# Database password variable patterns (not actual secrets)
'''db_password\s*=\s*session_password_store\.get_session_password''',
'''db_password\s*=\s*db_password''',
'''db_password=db_password''',
'''db_password=user_password''',
'''db_password:\s*Optional\[str\]''',
'''self\._db_password\s*=\s*db_password''',
'''embedding_manager\.db_password\s*=\s*db_password''',
'''rag_service\.db_password\s*=\s*db_password''',
# SQLCipher key derivation function calls (not actual secrets)
'''key\s*=\s*_get_key_from_password''',
'''key\s*=\s*_get_key_from_password_cached''',
# Chat routes / auth helpers (not actual secrets)
'''auth_data\s*=\s*temp_auth_store\.peek_auth''',
'''auth_token\s*=\s*session\.get\("temp_auth_token"\)''',
'''token\s*=\s*session\.get\("temp_auth_token"\)'''
]
# Target specific files for scanning
[target]
paths = [
'''src/''',
'''scripts/''',
'''docs/''',
'''\.github/''',
'''\.py$''',
'''\.js$''',
'''\.ts$''',
'''\.json$''',
'''\.yml$''',
'''\.yaml$''',
'''\.env$''',
'''\.sh$''',
'''Dockerfile.*''',
'''docker-compose.*''',
]
+581
View File
@@ -0,0 +1,581 @@
# Gitleaks Baseline - Known False Positives
# Generated: 2025-01-25
#
# This file contains fingerprints of known false positives from historical commits.
# Format: commit:file:rule:line
#
# These findings are NOT actual secrets - they are:
# - Python variable names containing "key", "password", "secret", "token"
# - Documentation examples with placeholder API keys
# - Test JWT tokens in fuzz testing files
# - Configuration parameter definitions
#
# To regenerate this baseline:
# gitleaks detect --config .gitleaks.toml --report-path findings.json --report-format json
# jq -r '.[] | "\(.Commit):\(.File):\(.RuleID):\(.StartLine)"' findings.json | sort -u
039aa8b4fe244843736e0cfc6495752948191380:src/local_deep_research/news/recommender/base_recommender.py:generic-secret:193
05446cc600a4feb1b1a2585d5f51782fe478d29f:README.md:generic-secret:130
05ef88f0ad1a63739ed3e41810821887b293b588:src/local_deep_research/benchmarks/datasets.py:generic-secret:36
061517d13b8b45b5f82ce0b94d0c7ed4e0d2b2e7:src/local_deep_research/database/backup/backup_service.py:generic-secret:532
0863f927099cf64cec23a35420c0a0a3fbd4bba9:web_search_engines/engines/search_engine_pubmed.py:generic-secret:25
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:150
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:249
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:260
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:271
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:73
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:87
0d6ee63740f36fcf3ab6961096af43892714b1f9:src/local_deep_research/defaults/llm_config.py:generic-secret:97
0ef8b9cf22a728404381de04968182749dff8926:src/local_deep_research/config/tts_config.py:generic-secret:47
0ef8b9cf22a728404381de04968182749dff8926:src/local_deep_research/config/tts_config.py:generic-secret:78
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:203
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:261
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:121
1110352b5c1b9f57afb2dd4a3fd2ccf86cd2bb54:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:122
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:112
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:121
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:130
1350c7022e4457a58f818dd36594b7bef7e1750a:docs/search-engines.md:generic-secret:140
1810651e5f0a29d0aafdcbb25049a99b452d97bd:src/local_deep_research/database/backup/backup_service.py:generic-secret:129
191913aedf4161d1ceafc92f9b0928650efcfa77:web_search_engines/engines/search_engine_google_pse.py:generic-secret:18
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:20
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:25
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:30
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:34
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:40
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:45
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:50
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/bootstrap.py:generic-secret:56
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:19
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:26
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:40
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:48
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:56
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:63
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/env_definitions/db_config.py:generic-secret:68
1a7fe64ee10cd6f9c8e7a14897102bd4865a221f:src/local_deep_research/settings/README.md:generic-secret:30
1cb459d08f9cb6e8cfd66107a533107aafb847d2:src/local_deep_research/security/rate_limiter.py:generic-secret:112
1e976da092355c985bb769b50a6e81a84ef6ac8c:src/local_deep_research/brains/services/brain_rag_service.py:generic-secret:45
1e976da092355c985bb769b50a6e81a84ef6ac8c:src/local_deep_research/brains/services/brain_rag_service.py:generic-secret:55
1e976da092355c985bb769b50a6e81a84ef6ac8c:src/local_deep_research/brains/services/brain_rag_service.py:generic-secret:59
23848d5f9294ea6ba6e62032d4817de50eceee34:src/local_deep_research/benchmarks/graders.py:generic-secret:74
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/benchmarks/graders.py:generic-secret:91
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:203
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:612
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:420
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:441
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:460
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:64
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:75
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/config/llm_config.py:generic-secret:86
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/database/encrypted_db.py:generic-secret:170
23ce85008bd579b1fa106c0ffbafeda9758fb017:src/local_deep_research/database/encrypted_db.py:generic-secret:311
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/database/encrypted_db.py:generic-secret:51
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/database/temp_auth.py:generic-secret:38
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/web/auth/database_middleware.py:generic-secret:27
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/web/auth/routes.py:generic-secret:189
27e2d7092690bc9f2f0b70563ada1927448c08db:src/local_deep_research/web/auth/routes.py:generic-secret:99
297d2fcd3019ec01e7946ad9fb7d4c8b01f83c36:src/local_deep_research/config/llm_config.py:generic-secret:66
297d2fcd3019ec01e7946ad9fb7d4c8b01f83c36:src/local_deep_research/config/llm_config.py:generic-secret:84
297d2fcd3019ec01e7946ad9fb7d4c8b01f83c36:src/local_deep_research/config/llm_config.py:generic-secret:98
2c0cb0ca2c6c46a2a81705c0016a9a7396f541ce:web_search_engines/engines/meta_search_engine.py:generic-secret:64
2c0cb0ca2c6c46a2a81705c0016a9a7396f541ce:web_search_engines/engines/search_engine_guardian.py:generic-secret:14
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:240
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:245
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:257
2c2a5708ad67fa71815bffa323e5365e62570412:src/local_deep_research/followup_research/routes.py:generic-secret:286
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/config/llm_config.py:generic-secret:529
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/database/encryption_check.py:generic-secret:88
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/database/encryption_check.py:generic-secret:94
2eaaf121090b28f150587a928b78d1cffb0c5701:src/local_deep_research/database/models/active_research.py:generic-secret:19
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1324
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1342
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1722
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1738
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-api-key:1917
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-secret:111
2eaaf121090b28f150587a928b78d1cffb0c5701:tests/ui_tests/diagnosis_network_log.json:generic-secret:143
2f745ee032bada469ce76513a7a0d812f8fbc8d1:src/local_deep_research/web/auth/routes.py:generic-secret:41
2f86494aa903f9abbb595d5d5a77009c518ec6c2:web_search_engines/engines/search_engine_github.py:generic-secret:24
32f667285276eeb0d509da1d11e37d22e9e9cd2f:src/local_deep_research/config/llm_config.py:generic-secret:30
32f667285276eeb0d509da1d11e37d22e9e9cd2f:src/local_deep_research/config/llm_config.py:generic-secret:39
32f667285276eeb0d509da1d11e37d22e9e9cd2f:src/local_deep_research/config/llm_config.py:generic-secret:48
338d3107ce4e3f46368835a633c00b14d47f6672:web_search_engines/engines/search_engine_google_pse.py:generic-secret:18
3be40474d246c71e704a9d233071b218c2ac34e0:web_search_engines/engines/search_engine_searxng.py:generic-secret:36
3be40474d246c71e704a9d233071b218c2ac34e0:web_search_engines/engines/search_engine_searxng.py:generic-secret:52
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:235
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:243
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:251
4068be5eadac7790c7bbc977ac11f64d9be006b4:src/local_deep_research/defaults/llm_config.py:generic-secret:77
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:scripts/generate_config_docs.py:generic-secret:75
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:65
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/metrics/search_tracker.py:generic-secret:448
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/metrics/search_tracker.py:generic-secret:455
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/settings/manager.py:generic-secret:390
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/settings/manager.py:generic-secret:496
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_cleanup.py:generic-secret:33
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_manager.py:generic-secret:48
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_manager.py:generic-secret:79
418ed1e41b6d91cf07a94fb4102a9bc10deb9a11:src/local_deep_research/web/auth/session_manager.py:generic-secret:94
421a69f5bfbe42957711121d3f4ebc68f246105a:src/local_deep_research/api/research_functions.py:generic-secret:294
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/auth_fuzzer.py:jwt:92
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:182
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:195
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:208
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:320
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:343
4356c96cea5d2d6233e66d796c25749f32dd53f7:.clusterfuzzlite/fuzz_targets/settings_parser_fuzzer.py:generic-secret:390
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:142
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:77
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:158
440e64b3279093331a49cacd39159a50d9adfe64:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:83
457d630d21b4eb0478b696ca2f36f42c5373d1d3:src/local_deep_research/web/database/migrations.py:generic-secret:277
478bdf14731f46983d3d4d363aa48a89f4fec73d:web_search_engines/engines/search_engine_github.py:generic-secret:24
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:103
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:326
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:335
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:344
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:81
488b3b49694b80472139b0ac494baf607256a05d:src/local_deep_research/config/llm_config.py:generic-secret:94
49bd120055e531e5002d23c584d9179ef96b17fa:README.md:generic-secret:98
4b72eedfad7569743c0ba33f4102117c9f37fb29:src/local_deep_research/config_manager.py:generic-secret:341
4b9beb813c629322538641213acda35b7b02a36d:src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.py:generic-secret:173
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:105
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:196
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:204
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:215
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:87
4efb4296e0fe477597ada0bc1bf5eaf9766bae08:src/local_deep_research/defaults/llm_config.py:generic-secret:97
525aa2619ff5fba4fa305bd4070108a5dd84d775:src/local_deep_research/defaults/llm_config.py:generic-secret:124
5407f220afe58ffcc6e6f2e4976720687691c3af:web_search_engines/engines/search_engine_searxng.py:generic-secret:36
5407f220afe58ffcc6e6f2e4976720687691c3af:web_search_engines/engines/search_engine_searxng.py:generic-secret:52
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:114
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:115
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:118
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:119
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:37
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:38
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:49
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:50
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:96
543383694cce67b511b53c94fc84d5fd819d9155:docs/env_configuration.md:generic-secret:97
55b6414e900e4b2fabf16e29ea260375c7552e5d:.clusterfuzzlite/fuzz_targets/socket_message_fuzzer.py:generic-secret:162
5707371296f47e4ee1ff55f418737cfb9b552124:web_search_engines/search_engine_serpapi.py:generic-secret:16
5707371296f47e4ee1ff55f418737cfb9b552124:web_search_engines/search_engine_serpapi.py:generic-secret:39
5781f183ffa68e2b732f5677c9590e52efff66b3:src/local_deep_research/defaults/llm_config.py:generic-secret:157
57e01f49a2ee51a66b213a6600d16a590799e4c3:web_search_engines/engines/search_engine_brave.py:generic-secret:19
57e01f49a2ee51a66b213a6600d16a590799e4c3:web_search_engines/engines/search_engine_brave.py:generic-secret:77
57e01f49a2ee51a66b213a6600d16a590799e4c3:web_search_engines/search_engine_factory.py:generic-secret:136
5b6ab31410a8ff25edfb3168cde2d1df71d95b28:src/local_deep_research/utilities/threading_utils.py:generic-secret:28
5dc1b0131747e42cd36a912545cac624ad432a93:src/local_deep_research/defaults/llm_config.py:generic-secret:63
5dc1b0131747e42cd36a912545cac624ad432a93:src/local_deep_research/defaults/llm_config.py:generic-secret:79
5dc1b0131747e42cd36a912545cac624ad432a93:src/local_deep_research/defaults/llm_config.py:generic-secret:93
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:271
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:289
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:303
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:357
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:481
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:492
5ed8d8d441c337a77e08ad44f2d447df84730dfe:src/local_deep_research/config.py:generic-secret:503
62928db77729953ce3b1da0e94634cd74362f4d0:docs/api-quickstart.md:generic-secret:52
62928db77729953ce3b1da0e94634cd74362f4d0:docs/api-quickstart.md:generic-secret:77
62928db77729953ce3b1da0e94634cd74362f4d0:docs/MIGRATION_GUIDE_v1.md:generic-secret:206
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:curl-auth-header:72
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:125
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:18
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:19
62928db77729953ce3b1da0e94634cd74362f4d0:docs/troubleshooting-openai-api-key.md:generic-secret:42
62928db77729953ce3b1da0e94634cd74362f4d0:examples/api_usage/README.md:generic-secret:33
62928db77729953ce3b1da0e94634cd74362f4d0:examples/api_usage/UPGRADE_NOTICE.md:generic-secret:52
62928db77729953ce3b1da0e94634cd74362f4d0:README.md:generic-secret:182
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/benchmarks/graders.py:generic-secret:91
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/config/llm_config.py:generic-secret:396
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/config/llm_config.py:generic-secret:417
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/config/llm_config.py:generic-secret:436
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:22
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:34
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/database/thread_local_session.py:generic-secret:168
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:15
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:19
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:23
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/database.py:generic-secret:29
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/search_tracker.py:generic-secret:132
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/search_tracker.py:generic-secret:136
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/token_counter.py:generic-secret:935
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/metrics/token_counter.py:generic-secret:939
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/settings/manager.py:generic-secret:207
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/settings/manager.py:generic-secret:441
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/database_middleware.py:generic-secret:54
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/database_middleware.py:generic-secret:59
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/database_middleware.py:generic-secret:74
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/auth/session_cleanup.py:generic-secret:34
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:111
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:225
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:325
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:497
62928db77729953ce3b1da0e94634cd74362f4d0:src/local_deep_research/web/queue/processor_v2.py:generic-secret:529
62928db77729953ce3b1da0e94634cd74362f4d0:tests/langchain_integration/README.md:generic-secret:142
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:103
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:326
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:335
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:344
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:81
6310d18ce51adf0e8ccaf7c7341525c62270d0cc:src/local_deep_research/config/llm_config.py:generic-secret:94
63cf66a086113d43e3023998883177a39063e6a6:src/local_deep_research/embeddings/embeddings_config.py:generic-secret:81
6784ea9747b62cd511927615da9cbab0c86d1aeb:src/local_deep_research/news/api.py:generic-secret:824
6784ea9747b62cd511927615da9cbab0c86d1aeb:src/local_deep_research/news/api.py:generic-secret:950
69a49966d458741fedb7433012349d056eaeec07:README.md:generic-secret:115
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/multitenant/session_context.py:generic-secret:18
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/multitenant/session_context.py:generic-secret:34
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/multitenant/session_context.py:generic-secret:75
6af754eb3dac2067406e1a0b1e680dc447dae21d:src/local_deep_research/security/safe_requests.py:generic-secret:197
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:405
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:444
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:493
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:496
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:500
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:502
6beef4cd3ab8787746c1b754f6a89ac73de2a722:src/local_deep_research/chat/routes.py:generic-secret:505
6c5d1c58787d26213ff91f94727c3539cfb860e4:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:74
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:103
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:326
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:335
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:344
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:81
6cd6ce3090a58831b922a7094abe91f1609d8ab4:src/local_deep_research/config/llm_config.py:generic-secret:94
72978a7befe34118be5e77377441bf733424bf3e:web_search_engines/engines/search_engine_brave.py:generic-secret:19
72978a7befe34118be5e77377441bf733424bf3e:web_search_engines/engines/search_engine_brave.py:generic-secret:77
72978a7befe34118be5e77377441bf733424bf3e:web_search_engines/search_engine_factory.py:generic-secret:136
78ce195b1a7334967f0ca9802ef9696989c56a59:src/local_deep_research/benchmarks/graders.py:generic-secret:74
7b472e788a0f2268db15c0a0b5e632dde6d1770d:src/benchmark_browser_comp.py:generic-secret:181
7b472e788a0f2268db15c0a0b5e632dde6d1770d:src/benchmark_browser_comp.py:generic-secret:67
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:142
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/cohere.py:generic-secret:77
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:158
7b5978fdfa55311f9aabdcef02c3d080e8be4f21:src/local_deep_research/reranker/providers/implementations/jina.py:generic-secret:83
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/multitenant/session_context.py:generic-secret:18
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/multitenant/session_context.py:generic-secret:34
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/multitenant/session_context.py:generic-secret:75
7be1fe6bf33b19ea19acea5478ad3defe6fbd8d8:src/local_deep_research/security/safe_requests.py:generic-secret:197
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:235
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:243
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:251
7c5f9753aff6b828f10277072f0a0ba12887730e:src/local_deep_research/defaults/llm_config.py:generic-secret:77
7ef658073e4c9ad5cb9cc1926f1fc4733cd54792:src/local_deep_research/web/database/migrations.py:generic-secret:277
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:40
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:56
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:58
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:70
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:73
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:85
7f8cab31448fe25ed5eab71859970cce29c2abc5:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:87
8085f9bd3da3e9376d3a7f25329d4264a303eb05:src/local_deep_research/config/llm_config.py:generic-secret:66
8085f9bd3da3e9376d3a7f25329d4264a303eb05:src/local_deep_research/config/llm_config.py:generic-secret:84
8085f9bd3da3e9376d3a7f25329d4264a303eb05:src/local_deep_research/config/llm_config.py:generic-secret:98
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:235
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:243
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:251
835b022d5b5d459d80462278d1b8185e08505bd6:src/local_deep_research/defaults/llm_config.py:generic-secret:77
8bc38da5bb99dc9272273a6418877739ff94478e:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:160
8bc38da5bb99dc9272273a6418877739ff94478e:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:183
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:40
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:56
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:58
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:70
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:73
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:85
8c327a408d7a627627ccab9366ce818f06c657a9:src/local_deep_research/benchmarks/datasets/utils.py:generic-secret:87
8db7b4c603e88db26fc2767e6d7cdd915c545447:src/local_deep_research/defaults/llm_config.py:generic-secret:157
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:181
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:234
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:411
8e4b85fdd0c949907f0fc6752ee315ebb9267b73:src/local_deep_research/memory_cache/config.py:generic-secret:57
955ef9142ad6a4af1fd63a5045476ceadeeffbd8:tests/test_programatic_access.ipynb:generic-secret:22
965dcec84f5e0a3a4545a61b9cc3422333d0b7e2:src/local_deep_research/config_manager.py:generic-secret:341
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/api.py:generic-secret:43
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:168
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:217
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:244
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:273
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:56
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_auth.py:generic-secret:75
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:109
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:112
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:115
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:32
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:48
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:56
9781ede32b376cbd60e188c6984eb5f936244971:src/local_deep_research/web/auth/api_decorators.py:generic-secret:97
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:28
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:29
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:38
9785428ce9295277cf849608a4bca082c9a38714:src/local_deep_research/utilities/es_utils.py:generic-secret:39
99d2ae3b954224477081d47f5e4d4cf64060d65b:README.md:generic-secret:98
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:197
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:684
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:779
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:82
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:755
9a8b36c19b86471909b3815e1217c70ca4231057:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:835
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:105
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:196
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:204
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:215
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:87
9b207f2e9fbde184263da4165699f8aa591373d6:src/local_deep_research/defaults/llm_config.py:generic-secret:97
9c40d8fb90281c140a2e44016eddedfe0518dee5:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:32
9c40d8fb90281c140a2e44016eddedfe0518dee5:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:51
9e60040f9b721da81910ab97a3f4f43d0276a07c:src/local_deep_research/metrics/search_tracker.py:generic-secret:416
9e60040f9b721da81910ab97a3f4f43d0276a07c:src/local_deep_research/metrics/search_tracker.py:generic-secret:424
a1bb4046c0c03ca984e23d57dbf71604cd013b12:README.md:generic-secret:115
a1cfdb198a37e753b3c777cf8cc30aa6841c2364:src/local_deep_research/defaults/llm_config.py:generic-secret:124
a291a593b6c6d15ca4cd9bcce46df4f90761f9e1:README.md:generic-secret:173
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:28
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:29
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:38
a3a545cc109b82039d5283e773f840ae86b2cfb1:src/local_deep_research/utilities/es_utils.py:generic-secret:39
a416def962c35e49e2673f6253d399042a879652:src/local_deep_research/web/database/migrations.py:generic-secret:277
a58b50fdc33eeaff156fe07230603f6a5c0f33b3:web_search_engines/engines/search_engine_pubmed.py:generic-secret:25
a8163c82d0920aa397bd3ee8d98b73f5efda7484:src/local_deep_research/web/auth/routes.py:generic-secret:41
a94af439f7a4795c595bc484d6bf2f12de77c6db:src/local_deep_research/benchmarks/graders.py:generic-secret:71
a94af439f7a4795c595bc484d6bf2f12de77c6db:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:599
a94af439f7a4795c595bc484d6bf2f12de77c6db:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:93
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:106
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:117
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:346
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:357
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:368
ad2488bb2a1e0ac579cef15596c925502a2b2189:src/local_deep_research/config/llm_config.py:generic-secret:91
b1ab5a5017a2710dee447b2dcaf65a618f3aa195:scripts/run_ollama_benchmark.py:generic-secret:72
b1ab5a5017a2710dee447b2dcaf65a618f3aa195:src/local_deep_research/benchmarks/runners.py:generic-secret:71
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:28
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:29
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:38
b2840cc65ea8581e76bfb53fa5c6a2cd23386c21:src/local_deep_research/utilities/es_utils.py:generic-secret:39
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:114
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:115
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:118
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:119
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:37
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:38
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:49
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:50
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:96
b5e3d8277eb61352cb9e075c09365bdb6afd4cce:docs/env_configuration.md:generic-secret:97
b8cf24fdac30a0221459d9c9c24da9ca04b34f52:src/local_deep_research/database/backup/backup_scheduler.py:generic-secret:115
b8cf24fdac30a0221459d9c9c24da9ca04b34f52:src/local_deep_research/database/backup/backup_service.py:generic-secret:63
bc74842ccdb7a4e7783ea18364e5a71c6b43849a:web_search_engines/engines/meta_search_engine.py:generic-secret:64
bc74842ccdb7a4e7783ea18364e5a71c6b43849a:web_search_engines/engines/search_engine_guardian.py:generic-secret:14
bccbb6e3c668aec2fa7a933b35dba3bbc7eb7434:web_search_engines/search_engine_factory.py:generic-secret:123
be13e7eb657a9ab2eb2f59c943a5baea2a86e870:README.md:generic-secret:115
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:106
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:117
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:346
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:357
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:368
c212d1cab13c014db97431d9da29759e2d233917:src/local_deep_research/config/llm_config.py:generic-secret:91
c842f99f7b3db3d6fca98e22224ae0041a58a5b2:src/local_deep_research/advanced_search_system/candidate_exploration/base_explorer.py:generic-secret:291
c842f99f7b3db3d6fca98e22224ae0041a58a5b2:src/local_deep_research/advanced_search_system/strategies/constrained_search_strategy.py:generic-secret:808
c842f99f7b3db3d6fca98e22224ae0041a58a5b2:src/local_deep_research/advanced_search_system/strategies/modular_strategy.py:generic-secret:792
c8b26efc72e5fc442a942bfb4af3f074b4b24756:scripts/benchmark_all_strategies.py:generic-secret:563
c8dc94feb183e9a9a7b7919cb5ecef63badd8a13:src/local_deep_research/utilities/llm_utils.py:generic-secret:104
c8dc94feb183e9a9a7b7919cb5ecef63badd8a13:src/local_deep_research/utilities/llm_utils.py:generic-secret:76
c8dc94feb183e9a9a7b7919cb5ecef63badd8a13:src/local_deep_research/utilities/llm_utils.py:generic-secret:90
cd4696285e9932236a834d0abe542c6a85601298:src/local_deep_research/web/auth/database_middleware.py:generic-secret:71
cd4696285e9932236a834d0abe542c6a85601298:src/local_deep_research/web/auth/database_middleware.py:generic-secret:73
cd5275b6fdab297a2d762d4e3b9562ed8756248d:web_search_engines/search_engine_serpapi.py:generic-secret:16
cd5275b6fdab297a2d762d4e3b9562ed8756248d:web_search_engines/search_engine_serpapi.py:generic-secret:39
d02c2a76ba6007fad22401be5bae7d225e49e2ea:src/local_deep_research/advanced_search_system/filters/journal_reputation_filter.py:generic-secret:173
d55c1e1efb135eaf5d8ea5d8c8b36896f931aa8b:tests/test_programatic_access.ipynb:generic-secret:22
d5cb26286ab041766db18e57d183de3d6de7121f:src/local_deep_research/settings/manager.py:generic-secret:400
d66706a39fe4005280d437bd5ef21e3dac0d97a4:README.md:generic-secret:172
d66706a39fe4005280d437bd5ef21e3dac0d97a4:README.md:generic-secret:173
d66706a39fe4005280d437bd5ef21e3dac0d97a4:README.md:generic-secret:174
dbb09847bbe20a7b96d0a20f10ad1ce27df81a44:src/local_deep_research/defaults/llm_config.py:generic-secret:63
dbb09847bbe20a7b96d0a20f10ad1ce27df81a44:src/local_deep_research/defaults/llm_config.py:generic-secret:79
dbb09847bbe20a7b96d0a20f10ad1ce27df81a44:src/local_deep_research/defaults/llm_config.py:generic-secret:93
dc1bed6878c867cc32fc0c0544deaf3a01703de4:README.md:generic-secret:130
dd9e741e2fea700cd7dfe19db5fd0ccfc8b013de:web_search_engines/search_engine_factory.py:generic-secret:123
de91a597f2851a551ec95f969e64d20910db8869:src/local_deep_research/news/api.py:generic-secret:56
e15d05700d76d08f24e4f3442598805b655983de:src/local_deep_research/research_library/services/research_history_indexer.py:generic-secret:43
e15d05700d76d08f24e4f3442598805b655983de:src/local_deep_research/research_library/services/research_history_indexer.py:generic-secret:49
e15d05700d76d08f24e4f3442598805b655983de:src/local_deep_research/research_library/services/research_history_indexer.py:generic-secret:52
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:247
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:252
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:264
e2f9318a4e0bc43c2b103299da2519b8c366bd81:src/local_deep_research/followup_research/routes.py:generic-secret:293
e33dbd827ea898df645ba27580c4852b98f9c430:README.md:generic-secret:172
e33dbd827ea898df645ba27580c4852b98f9c430:README.md:generic-secret:173
e33dbd827ea898df645ba27580c4852b98f9c430:README.md:generic-secret:174
e3425ba1a5fe884c09e40e33a789daf26f968a03:src/local_deep_research/benchmarks/datasets.py:generic-secret:36
e41ee71a032162bcc9024d1a8eac634395c10496:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:73
e527d8b7e272fb9ca80011959980cfade7f06550:src/local_deep_research/config/llm_config.py:generic-secret:63
e527d8b7e272fb9ca80011959980cfade7f06550:src/local_deep_research/config/llm_config.py:generic-secret:74
e527d8b7e272fb9ca80011959980cfade7f06550:src/local_deep_research/config/llm_config.py:generic-secret:85
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:150
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:249
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:260
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:271
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:73
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:87
e563b70fbb7360ccd4f0f3172af6d60abb8d4b3e:src/local_deep_research/defaults/llm_config.py:generic-secret:97
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:66
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:15
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:21
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:27
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:33
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:38
e668ad2764d5989b020eeff138050fe927f8dd26:src/local_deep_research/settings/env_definitions/security.py:generic-secret:43
e6ef280aba6d4387fbfa5c0fe2b492ec6ef42c04:web_search_engines/search_engine_serpapi.py:generic-secret:16
e6ef280aba6d4387fbfa5c0fe2b492ec6ef42c04:web_search_engines/search_engine_serpapi.py:generic-secret:39
e7584cd4f68ac6fd4e663f50c6fb74507b84f388:README.md:generic-secret:173
ea1ef53fc4e952d4f980354c0b726bfbb1a47d92:src/local_deep_research/defaults/llm_config.py:generic-secret:124
ef32e88aefc77c6114760aaf64339ef2763b516c:src/local_deep_research/api/client.py:generic-secret:114
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:271
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:289
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:303
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:357
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:481
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:492
f36c295fc291f1aa0db5ac08f0f6da9bcee96717:src/local_deep_research/config.py:generic-secret:503
f47617192518ba781055eee3b24d619282f8e884:src/local_deep_research/web_search_engines/rate_limiting/llm/wrapper.py:generic-secret:115
f4e3c25122dc1f76252e6167422cbc430d359739:README.md:generic-secret:205
f4e3c25122dc1f76252e6167422cbc430d359739:README.md:generic-secret:206
f4e3c25122dc1f76252e6167422cbc430d359739:README.md:generic-secret:207
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:271
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:289
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:303
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:357
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:481
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:492
f6aa37f8f75bdc53df0a8a271a75ad1dcd63f793:src/local_deep_research/config.py:generic-secret:503
fa084c1ea11ce9f44268b5fa63986139dfaeb41c:src/local_deep_research/config/llm_config.py:generic-secret:66
fa084c1ea11ce9f44268b5fa63986139dfaeb41c:src/local_deep_research/config/llm_config.py:generic-secret:84
fa084c1ea11ce9f44268b5fa63986139dfaeb41c:src/local_deep_research/config/llm_config.py:generic-secret:98
fc48711a3b21779934bd5be60ac29112e338ec49:src/local_deep_research/api/research_functions.py:generic-secret:139
fcc47fc370185d328eddb5b20f2cac5d8e49d9e3:README.md:generic-secret:205
fcc47fc370185d328eddb5b20f2cac5d8e49d9e3:README.md:generic-secret:206
fcc47fc370185d328eddb5b20f2cac5d8e49d9e3:README.md:generic-secret:207
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:664
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/benchmarks/web_api/benchmark_routes.py:generic-secret:80
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/benchmarks/web_api/benchmark_service.py:generic-secret:305
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/base.py:generic-secret:53
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/base.py:generic-secret:68
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/__init__.py:generic-secret:14
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/__init__.py:generic-secret:17
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:101
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:136
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:139
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:14
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:188
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/manager.py:generic-secret:81
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/credentials/stores/temp_store.py:generic-secret:37
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/followup_research/routes.py:generic-secret:237
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/metrics/search_tracker.py:generic-secret:425
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/news/api.py:generic-secret:64
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/database_middleware.py:generic-secret:52
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/routes.py:generic-secret:146
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/routes.py:generic-secret:340
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/auth/session_cleanup.py:generic-secret:34
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/queue/processor_v2.py:generic-secret:117
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/queue/processor_v2.py:generic-secret:436
fe354d87ad4535f38e725cafcda38c1236493afd:src/local_deep_research/web/queue/processor_v2.py:generic-secret:658
fe9cec036dc06d6116d4074703a28b7c57637af4:src/benchmark_browser_comp.py:generic-secret:181
fe9cec036dc06d6116d4074703a28b7c57637af4:src/benchmark_browser_comp.py:generic-secret:67
# Fuzz testing files - test JWTs and mock tokens (2026-02-01)
9f288fa71069adfd6493a91612c449fcf6f56d63:.clusterfuzzlite/fuzz_targets/socket_message_fuzzer.py:generic-secret:162
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/auth_fuzzer.py:jwt:92
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/settings_parser_fuzzer.py:generic-secret:390
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:182
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:195
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:208
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:320
f1bc01ebf81069e77799fee2bfece1045d310f35:.clusterfuzzlite/fuzz_targets/html_parser_fuzzer.py:generic-secret:343
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:121
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:jwt:122
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:203
77011460083d8ab90765e39e8ffcf54255d00595:.clusterfuzzlite/fuzz_targets/authorization_fuzzer.py:generic-secret:261
# SQLCipher utils - runtime password parameters (2026-02-01)
d72b7ae6680970282316be0b4377ccfdff674a78:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:57
d72b7ae6680970282316be0b4377ccfdff674a78:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:69
ba12412dc5ba19db6930a31428676d7156dc9e91:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:74
f18883b8403e6e5429e1a64ac067727dc0458fcb:src/local_deep_research/database/sqlcipher_utils.py:generic-secret:83
# SQLCipher utils - refactored set_sqlcipher_rekey (key derivation from password)
src/local_deep_research/database/sqlcipher_utils.py:generic-secret:85
# Python type hints and function calls - false positives (2026-02-02)
# es_utils.py: Optional[str] type hints for password/api_key parameters
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:28
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:29
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:38
b3e9cb9df8080d101deee6ae26fad054ed7a4a77:src/local_deep_research/utilities/es_utils.py:generic-secret:39
# llm_config.py: get_db_setting() function calls reading config values
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:81
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:94
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:103
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:326
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:335
3c09850df77c25ff718809ef8b65bced8689a163:src/local_deep_research/config/llm_config.py:generic-secret:344
# defaults/llm_config.py: Configuration parameter definitions
acc9005801d4361cf3f91ef8a4ab45c42765372b:src/local_deep_research/defaults/llm_config.py:generic-secret:124
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:77
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:235
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:243
dc366dab85c1a02f8ffadc38c8db210cf2a61b95:src/local_deep_research/defaults/llm_config.py:generic-secret:251
# search_engine_serpapi.py: Environment variable reads
7838cd1fd1d1afbc92a2976495d6c06342bca737:web_search_engines/search_engine_serpapi.py:generic-secret:16
7838cd1fd1d1afbc92a2976495d6c06342bca737:web_search_engines/search_engine_serpapi.py:generic-secret:39
# README.md: Documentation example showing settings.get() usage
fed86a7c9481006cb963b1039bf9dc818a5af063:README.md:generic-secret:115
# generate_config_docs.py: AST node value extraction (key = key_node.value), not a secret
scripts/generate_config_docs.py:generic-secret:92
# report_generator.py: variable name original_iterations in settings restoration code
src/local_deep_research/report_generator.py:generic-secret:486
# followup routes.py: password retrieval variable names (user_password, auth_data)
src/local_deep_research/followup_research/routes.py:generic-secret:208
src/local_deep_research/followup_research/routes.py:generic-secret:212
src/local_deep_research/followup_research/routes.py:generic-secret:219
src/local_deep_research/followup_research/routes.py:generic-secret:222
src/local_deep_research/followup_research/routes.py:generic-secret:206
# Semantic search routes — db_password variable names (false positives)
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:49
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:94
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:141
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:200
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:217
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:295
src/local_deep_research/research_library/search/services/research_history_indexer.py:generic-secret:46
src/local_deep_research/research_library/search/services/research_history_indexer.py:generic-secret:52
src/local_deep_research/research_library/search/services/research_history_indexer.py:generic-secret:55
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:293
# Auto-conversion hook — db_password variable name (false positive)
src/local_deep_research/web/queue/processor_v2.py:generic-secret:357
src/local_deep_research/web/queue/processor_v2.py:generic-secret:358
# search_routes.py — db_password variable (shifted line numbers, false positive)
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:51
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:146
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:189
src/local_deep_research/research_library/search/routes/search_routes.py:generic-secret:273
# Placeholder API key for local servers without auth (e.g. llama.cpp)
7c84e267390264c58c3857881c70f9b80677d76b:src/local_deep_research/config/llm_config.py:generic-secret:516
# data_sanitizer.py — `is_secret = DataSanitizer.is_sensitive_setting(...)` reads
# as `secret = <token>` to the generic-secret rule. It is a boolean variable
# assignment in redact_settings_snapshot, not a credential (false positive).
src/local_deep_research/security/data_sanitizer.py:generic-secret:212
282bb0c3d9e9d65ae67fa0acc18cd92e7ad447bc:src/local_deep_research/security/data_sanitizer.py:generic-secret:212
+500
View File
@@ -0,0 +1,500 @@
# Grype vulnerability ignore file
# Companion to .trivyignore — same suppression policy, different scanner.
# See: https://github.com/anchore/grype#specifying-matches-to-ignore
#
# Review Policy: All suppressions should be reviewed periodically.
# Last full review: 2026-04-17
ignore:
# ==========================================================================
# UNFIXABLE VULNERABILITIES — auto-ignored by fix state (no per-CVE entry).
# ==========================================================================
# Suppress only CVEs with NO available fix: Grype fix-state `not-fixed`
# (affected, no fix yet) and `wont-fix` (vendor declines to fix). This is the
# "no fix in Debian Trixie / won't-fix upstream" class we can't act on and
# were hand-listing CVE-by-CVE — it ends that treadmill.
#
# Vulnerabilities that CAN be fixed by a version upgrade (fix-state `fixed`)
# are deliberately NOT matched here, so they still surface and reach the gate
# — those are the actionable ones we want to see. `unknown` fix-state is also
# left visible. Severity cutoff is unchanged (still medium); this filters by
# fixability only, not by severity.
#
# Debian caveat: Grype maps Debian `<no-dsa>` to fix-state `wont-fix`
# (anchore/grype#386). Such CVEs can still get a later point-release fix —
# and when they do, Grype flips them to `fixed`, so they resurface here as
# upgradeable, exactly when we want them. Nothing permanently upgradeable is
# hidden; only the current no-fix window is muted.
# Ref: https://oss.anchore.com/docs/reference/grype/configuration/
- fix-state: not-fixed
- fix-state: wont-fix
# ==========================================================================
# DISPUTED CVEs (upstream and all major distros agree: not a security issue)
# ==========================================================================
# --- libc6/libc-bin (glibc) ---
- vulnerability: CVE-2019-1010022
reason: "Disputed by upstream glibc and all major distros. Mitigation bypass, not standalone vuln."
- vulnerability: CVE-2019-1010023
reason: "Library remapping via ldd. Upstream: not a security issue. Debian: unimportant."
- vulnerability: CVE-2019-1010024
reason: "ASLR bypass via thread stack cache. Upstream: not a security issue. Debian: unimportant."
- vulnerability: CVE-2019-1010025
reason: "ASLR bypass via heap addresses. Upstream: not a vulnerability. Debian: unimportant."
- vulnerability: CVE-2010-4756
reason: "POSIX glob resource limit. Debian: unimportant. Apps must impose own limits (by design)."
- vulnerability: CVE-2019-9192
reason: "Regex recursion with crafted pattern. Upstream disputes. Debian: unimportant."
- vulnerability: CVE-2018-20796
reason: "Regex recursion with crafted pattern. Upstream: not a vulnerability. Debian: unimportant."
# --- tar ---
- vulnerability: CVE-2005-2541
reason: "Disputed. Expected tar behavior (setuid/setgid preserved), not a bug. All distros: won't-fix."
# --- coreutils ---
- vulnerability: CVE-2017-18018
reason: "Upstream chose documentation-only fix. Container doesn't use chown -R -L."
# --- libglib2.0 ---
- vulnerability: CVE-2012-0039
reason: "Hash collision DoS in g_str_hash. Disputed by vendor. Debian: unimportant."
# --- systemd (libsystemd0/libudev1) ---
- vulnerability: CVE-2013-4392
reason: "Symlink attack in tmpfiles. Only relevant with SELinux. Debian: unimportant. No SELinux in container."
- vulnerability: CVE-2023-31437
reason: "Sealed journal log hiding. Disputed by upstream. Debian: unimportant."
- vulnerability: CVE-2023-31438
reason: "Sealed journal truncation bypass. Disputed by upstream. Debian: unimportant."
- vulnerability: CVE-2023-31439
reason: "Sealed journal event modification. Disputed by upstream. Debian: unimportant."
# --- shadow (passwd/login.defs) ---
- vulnerability: CVE-2007-5686
reason: "btmp file permissions (rPath Linux). Debian: unimportant. LOG_UNKFAIL_ENAB=no mitigates."
- vulnerability: CVE-2024-56433
reason: "Default subuid range overlap with network UIDs. Disputed minor issue. No NFS in container."
# --- perl ---
- vulnerability: CVE-2011-4116
reason: "File::Temp symlink handling. Debian: unimportant. No untrusted temp file ops."
# --- apt ---
- vulnerability: CVE-2011-3374
reason: "apt-key GPG validation. Not exploitable in Debian (no keyring URI defined). Debian: unimportant."
# ==========================================================================
# CHROMIUM (Playwright-bundled Chromium 145.0.7632.6)
# Fixes require 145.0.7632.45+ (DSA-6135-1) or 145.0.7632.109+ (DSA-6146-1).
# Will auto-resolve when Playwright 1.59 ships with Chromium 146.0.7680.0.
# Container runs headless Chrome for scraping only.
# ==========================================================================
- vulnerability: CVE-2026-3061
package: { name: chrome, type: binary }
reason: "OOB read in Media. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-3062
package: { name: chrome, type: binary }
reason: "OOB read/write in Tint (WebGPU). Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-3063
reason: "DevTools issue via malicious extension. Container runs headless for scraping."
- vulnerability: CVE-2026-2313
reason: "UAF in CSS. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2314
reason: "Heap overflow in Codecs. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2315
package: { name: chrome, type: binary }
reason: "Inappropriate impl in WebGPU. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2316
package: { name: chrome, type: binary }
reason: "Insufficient policy in Frames. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2317
package: { name: chrome, type: binary }
reason: "Inappropriate impl in Animation. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2318
package: { name: chrome, type: binary }
reason: "Inappropriate impl in PictureInPicture. Playwright 1.58 bundles pre-fix Chromium. Headless only."
- vulnerability: CVE-2026-2319
package: { name: chrome, type: binary }
reason: "Race condition in DevTools. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2320
package: { name: chrome, type: binary }
reason: "Inappropriate impl in File input. Playwright 1.58 bundles pre-fix Chromium. Headless only."
- vulnerability: CVE-2026-2321
reason: "UAF in Ozone. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2322
package: { name: chrome, type: binary }
reason: "Heap overflow in Codecs. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2323
package: { name: chrome, type: binary }
reason: "Inappropriate impl in Downloads. Playwright 1.58 bundles pre-fix Chromium. Headless only."
- vulnerability: CVE-2026-2441
reason: "UAF in CSS. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2648
reason: "Heap overflow in PDFium. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2649
package: { name: chrome, type: binary }
reason: "Integer overflow in V8. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
- vulnerability: CVE-2026-2650
package: { name: chrome, type: binary }
reason: "Heap overflow in Media. Playwright 1.58 bundles pre-fix Chromium. Headless scraping only."
# ==========================================================================
# NODE.JS (patchright-bundled Node 24.16.0 at patchright/driver/node)
# All fixed in Node 24.17.0 (2026-06-18 security release). patchright 1.61.2
# (latest on PyPI as of 2026-07-09) predates the fix and bundles Node 24.16.0.
# patchright is a transitive dep of crawl4ai; the driver/node binary is not
# invoked in the default configuration (web.enable_javascript_rendering
# defaults to false), and the production image does not install
# Playwright/patchright browsers (see Dockerfile note at the ldr stage), so
# the bundled driver/node is dead weight. Auto-resolves when patchright ships
# a rebuild with Node 24.17.0+.
# Severities in the reasons below are the Node.js/GHSA vendor CVSS ratings.
# GitHub's Grype alerts show NVD's context-free auto-scores for some of
# them (CVE-2026-48930 "Critical" 9.8, CVE-2026-48615 / -48619 "High" 7.5);
# the vendor ratings are the authoritative ones — the suppressions are not
# downplaying the alerts.
# Entries are scoped to the bundled binary (package: node, type: binary) —
# matching the Chromium block above — so the same CVE ID on an unrelated
# future node package would not be silently suppressed.
# (The earlier Node 24.13.0 entries — CVE-2026-21710 / -21712..-21717 — were
# fixed in Node 24.14.1, which the bundled 24.15.0/24.16.0 supersedes, so
# they no longer match and were removed.)
# ==========================================================================
- vulnerability: CVE-2026-48933
package: { name: node, type: binary }
reason: "High WebCrypto AES DoS (subtle.encrypt() input multiple of 2 GiB crashes the process) in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48618
package: { name: node, type: binary }
reason: "High TLS wildcard-depth authentication bypass via unicode dot-separator normalisation mismatch in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48615
package: { name: node, type: binary }
reason: "Medium proxy-credentials leak in ERR_PROXY_TUNNEL error messages in Node <24.17.0. Browser-driver node is never launched with proxy URLs. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48619
package: { name: node, type: binary }
reason: "Medium HTTP/2 client OOM via attacker-controlled ORIGIN frames in Node <24.17.0. Exploitable only when the node process is an HTTP/2 client against an attacker-positioned server; browser-driver node never acts as one. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48928
package: { name: node, type: binary }
reason: "Medium mTLS authorisation bypass via case-sensitive SNI context matching in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48930
package: { name: node, type: binary }
reason: "Medium TLS hostname rebinding via embedded-NUL C-string truncation in resolver bindings in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48934
package: { name: node, type: binary }
reason: "Medium TLS host-identity verification bypass via session reuse with different servername in Node <24.17.0. Browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48935
package: { name: node, type: binary }
reason: "Low Permission Model bypass via FileHandle.utimes() promises API in Node <24.17.0. Requires --permission opt-in; browser-driver node is never launched with it. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
- vulnerability: CVE-2026-48937
package: { name: node, type: binary }
reason: "Medium HTTP/2 server DoS (server keeps accepting data after GOAWAY/RST) in Node <24.17.0. Exploitable only against a running Node.js HTTP/2 server; patchright's browser-driver node never starts one. Patchright-bundled driver/node 24.16.0; not invoked in default config."
- vulnerability: CVE-2026-48617
package: { name: node, type: binary }
reason: "Low Permission Model bypass via process.report.writeReport() in Node <24.17.0. Requires the process to opt into Node's experimental --permission model; the browser-driver node is never launched with it. Patchright-bundled driver/node 24.16.0; not invoked in default config."
- vulnerability: CVE-2026-48931
package: { name: node, type: binary }
reason: "Low (CVSS 3.7) TOCTOU race in Node's http.Agent (HTTP response queue poisoning) in Node <24.17.0. Exploitable only when the node process acts as an HTTP client against an attacker-positioned server; patchright's browser-driver node is not invoked in default config. Patchright-bundled driver/node 24.16.0; fixed in Node 24.17.0."
# ==========================================================================
# CPYTHON (base-image binary /usr/local/bin/python3.14)
# The binary-cataloged CPython is matched via NVD CPE data, which carries
# no fix information, so Grype reports its fix-state as "unknown" — the
# fix-state: not-fixed auto-rule at the top of this file deliberately does
# NOT hide unknown, so an explicit entry is needed (empirically confirmed:
# the 2026-07-11 release-gate scan still flagged it with the auto-rule
# active, contrary to the assumption in PR #5032).
# ==========================================================================
- vulnerability: CVE-2026-4360
package: { name: python, type: binary }
reason: "Low tarfile uid/gid extraction-filter gap (GHSA-gf2w-jqmq-fcm8). No fixed release exists yet: Python 3.14.7 is scheduled 2026-08-04 (PEP 745). NVD-CPE binary match carries fix-state unknown, so the not-fixed auto-rule misses it. Remove when the base image ships Python 3.14.7+."
- vulnerability: CVE-2026-15308
package: { name: python, type: binary }
reason: "High (PSF CVSS4 8.7 / NVD 7.5) html.parser quadratic CPU DoS needing unterminated constructs across many incremental feed() calls (PSF-2026-33). Not reachable: all html.parser use is single-shot BeautifulSoup feed(markup); lxml uses libxml2. No fixed release yet: 3.14.7 due 2026-08-04 (PEP 745). NVD-CPE binary match has fix-state unknown. Remove when the base image ships 3.14.7+."
# ==========================================================================
# NO FIX AVAILABLE IN DEBIAN TRIXIE
# ==========================================================================
# --- libc6/libc-bin (needs glibc >=2.43, some >=2.44; Trixie has 2.41) ---
- vulnerability: CVE-2026-0861
reason: "Heap overflow in memalign. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa."
- vulnerability: CVE-2026-0915
reason: "NSS DNS info disclosure. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa."
- vulnerability: CVE-2025-15281
reason: "wordexp uninitialized memory. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa."
- vulnerability: CVE-2026-4046
reason: "High CVE in libc6/libc-bin glibc 2.41. Debian trixie postponed (minor; revisit when upstream fixes). No fix in trixie or sid yet."
- vulnerability: CVE-2026-5358
reason: "Buffer overflow in obsolete nis_local_principal. NIS deprecated since glibc 2.26. Needs glibc 2.44, Trixie has 2.41. Container does not use NIS."
- vulnerability: CVE-2026-5450
reason: "One-byte heap overflow in scanf %mc with width>1024. Needs glibc 2.44, Trixie has 2.41. Python never calls scanf with such format strings. Debian bug #1134543."
- vulnerability: CVE-2026-5928
reason: "Buffer under-read in ungetwc on wrong buffer pointer. Needs glibc 2.44, Trixie has 2.41. Info disclosure requires non-Unicode encodings; Python does not call ungetwc directly. Debian bug #1134544."
- vulnerability: CVE-2026-5435
reason: "Buffer overflow in deprecated ns_sprintrrf TSIG handling. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa. Container never calls deprecated DNS print functions."
- vulnerability: CVE-2026-6238
reason: "Buffer overread in deprecated ns_printrrf RDATA validation. Needs glibc 2.43, Trixie has 2.41. Debian no-dsa. Container never calls deprecated DNS print functions."
# --- libtiff6 (transitive dep via libgdk-pixbuf for WeasyPrint) ---
# Fix in 4.7.1-1 (sid only), Trixie has 4.7.0-3+deb13u1
- vulnerability: CVE-2025-61144
reason: "Stack overflow in readSeparateStripsIntoBuffer. Debian: no security impact. Sid only."
- vulnerability: CVE-2025-8176
reason: "Crash in tiffmedian CLI tool, no security impact. Not backported to Trixie."
- vulnerability: CVE-2025-8177
reason: "Crash in thumbnail CLI tool, no security impact. Not backported to Trixie."
- vulnerability: CVE-2025-61145
reason: "Double free in tiffcrop CLI tool, no security impact. Not backported to Trixie."
- vulnerability: CVE-2025-61143
reason: "NULL deref in tif_open.c CLI tool, no security impact. Not backported to Trixie."
- vulnerability: CVE-2017-16232
reason: "Memory leak DoS in libtiff. Upstream won't fully fix. Debian: unimportant."
- vulnerability: CVE-2018-10126
reason: "NULL deref in tiff2pdf via libjpeg. Debian: unimportant. CLI tool crash only."
- vulnerability: CVE-2022-1210
reason: "DoS in tiff2ps CLI tool. Debian: no-dsa. CLI tool crash only."
- vulnerability: CVE-2025-8534
reason: "NULL deref in tiff2ps. Debian: unimportant. CLI tool crash, fixed in sid 4.7.1-1."
- vulnerability: CVE-2026-4775
reason: "High vuln in libtiff6. Transitive dep via libgdk-pixbuf for WeasyPrint. No fix in Trixie yet."
# --- libjbig0 (jbigkit) ---
- vulnerability: CVE-2017-9937
reason: "JBIG memory alloc failure. Debian: unimportant. Actually a jbigkit bug, not libtiff."
# --- libcairo2 ---
- vulnerability: CVE-2017-7475
reason: "NULL deref in FT_Load_Glyph. Debian: minor/ignored. Crash only, no security impact."
- vulnerability: CVE-2018-18064
reason: "OOB stack write via WebKitGTK+. Upstream: negligible. Debian: unimportant."
- vulnerability: CVE-2025-50422
reason: "Assertion failure in font handling. Debian trixie no-dsa. Crash, no security impact."
# --- libexpat1 ---
- vulnerability: CVE-2025-59375
reason: "Memory amplification via small XML. Needs expat 2.7.2, Trixie has 2.7.1. Debian no-dsa."
- vulnerability: CVE-2025-66382
reason: "2 MiB XML causes DoS via slow parsing. Debian: minor, postponed. Needs expat >2.7.3."
- vulnerability: CVE-2026-24515
reason: "External entity parser missing encoding handler copy. Debian trixie no-dsa. Sid 2.7.4-1."
- vulnerability: CVE-2026-25210
reason: "Integer overflow in doContent tag buffer realloc. Debian trixie no-dsa. Sid 2.7.4-1."
- vulnerability: CVE-2026-32776
reason: "NULL deref in empty external parameter entity. Needs expat 2.7.5, not in Trixie or sid. DoS only (CVSS 4.0). API XML uses defusedxml; XML upload uses lxml/libxml2 (not expat)."
- vulnerability: CVE-2026-32777
reason: "Infinite loop in DTD parsing. Needs expat 2.7.5, not in Trixie or sid. Local attack vector, DoS only (CVSS 4.0). API XML uses defusedxml; XML upload uses lxml/libxml2 (not expat)."
- vulnerability: CVE-2026-32778
reason: "NULL deref in setContext after OOM. Needs expat 2.7.5, not in Trixie or sid. CNA CVSS 2.9 Low / NIST CVSS 5.5 Medium. Requires OOM precondition."
- vulnerability: CVE-2026-50219
reason: "UAF via reentrant parser calls from handlers in expat <2.8.2. Unfixed upstream and in sid (Debian bug 1138862). API XML uses defusedxml; XML upload uses lxml/libxml2 (not expat)."
# libexpat <2.8.2 integer-overflow batch. Fixed upstream in expat 2.8.2
# (released 2026-06-25) but not yet packaged in Debian trixie (2.7.1-2) or
# sid (2.8.1-1) — both still vulnerable, nothing to apt-upgrade to. These
# overflows fire during normal attribute/namespace/CDATA parsing, so
# defusedxml's entity/DTD hardening does NOT mitigate them; the real
# mitigation is that expat only parses TLS-fetched XML from hardcoded trusted
# upstreams — arXiv (export.arxiv.org, via the `arxiv` lib's feedparser ->
# xml.sax -> expat) and PubMed/NCBI (eutils.ncbi.nlm.nih.gov, via
# defusedxml). No attacker-controlled XML reaches expat; document uploads
# use lxml/libxml2, not expat. NVD: AV:L, medium.
- vulnerability: CVE-2026-56403
reason: "Integer overflow in storeAtts (attribute parsing), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
- vulnerability: CVE-2026-56404
reason: "Integer overflow in addBinding (namespace binding), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
- vulnerability: CVE-2026-56405
reason: "Integer overflow in getAttributeId, expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
- vulnerability: CVE-2026-56406
reason: "Integer overflow in XML_ParseBuffer (missing length check), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
- vulnerability: CVE-2026-56407
reason: "Integer overflow in doProlog (storeEntityValue), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
- vulnerability: CVE-2026-56408
reason: "Integer overflow in copyString, expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
- vulnerability: CVE-2026-56412
reason: "UAF in doCdataSection (missing XML_TOK_DATA_CHARS check; incomplete fix for CVE-2026-50219), expat <2.8.2. Trixie 2.7.1-2 / sid 2.8.1-1 vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian. Expat parses only TLS-fetched trusted arXiv/PubMed XML; uploads use lxml not expat. AV:L medium."
# CVE-2026-56409/-56410/-56411 are in the xmlwf CLI tool, which ships in the
# `expat` package (not the installed libexpat1 runtime) and is never invoked
# in this container — no runtime attack surface.
- vulnerability: CVE-2026-56409
reason: "Integer overflow in the xmlwf CLI tool (-d outputDir output filename), expat <2.8.2. Bug is in the xmlwf binary, not the libexpat1 runtime, and xmlwf is never invoked in this container. Trixie/sid vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian."
- vulnerability: CVE-2026-56410
reason: "Integer overflow in the xmlwf CLI tool (resolveSystemId), expat <2.8.2. Bug is in the xmlwf binary, not the libexpat1 runtime, and xmlwf is never invoked in this container. Trixie/sid vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian."
- vulnerability: CVE-2026-56411
reason: "Integer overflow in the xmlwf CLI tool (endDoctypeDecl via NOTATION declarations), expat <2.8.2. Bug is in the xmlwf binary, not the libexpat1 runtime, and xmlwf is never invoked in this container. Trixie/sid vulnerable; expat 2.8.2 (2026-06-25) not yet in Debian."
# --- libgraphite2-3 (font shaping, transitive via harfbuzz for WeasyPrint) ---
- vulnerability: CVE-2026-50593
reason: "Integer underflow OOB write via crafted Graphite font actions. Fixed in graphite2 1.3.15 (forky/sid only); Trixie 1.3.14 no-dsa (point-release candidate). WeasyPrint renders app-generated HTML with trusted fonts only."
# --- perl-base ---
- vulnerability: CVE-2026-7010
reason: "CRLF header injection in HTTP::Tiny <0.093. Fixed in perl 5.40.1-8 (sid only); Trixie 5.40.1-6 no-dsa. perl-base is a dpkg essential package; container runs no perl HTTP clients."
# --- libxml2 ---
# CVE-2026-6653 (high): UAF in xmlParseInternalSubset (XML internal-DTD
# parameter entities), affects 2.9.11-2.11.0; Trixie's 2.9.14 base is
# vulnerable with no backport (fixed only in sid/forky 2.15.x). NOT REACHABLE
# in this image: the flagged system libxml2 (2.9.14) is present but UNUSED.
# All XML/HTML parsing goes through lxml, which statically bundles its own
# libxml2 2.14.x — past the 2.11.0 fix (verified: etree.LIBXML_VERSION is
# (2,14,6) and lxml's etree.so has no libxml2.so in `ldd`; pdm.lock pins
# manylinux lxml wheels). The Python stdlib XML uses expat, not libxml2.
# A build-time assertion in the Dockerfile (after `pdm install`) fails the
# build if lxml ever ends up linked against a pre-2.11 libxml2 (e.g. an sdist
# fallback against the system lib), so a vulnerable parser can never ship.
- vulnerability: CVE-2026-6653
reason: "High UAF in xmlParseInternalSubset (libxml2 2.9.11-2.11.0); Trixie 2.9.14 vulnerable, no backport (fixed in sid/forky 2.15.x). NOT reachable: system libxml2 present but unused — all XML parsing uses lxml's bundled libxml2 2.14.x (verified; manylinux wheels), stdlib uses expat. A Dockerfile assertion blocks any pre-2.11 lxml. DoS-primary, CVSS 4.0 7.0."
- vulnerability: CVE-2026-1757
reason: "Memory leak in xmllint CLI shell. Negligible impact. Debian: unimportant. No xmllint usage."
- vulnerability: CVE-2025-8732
reason: "Recursion in xmlParseSGMLCatalog. Debian: unimportant. SGML catalogs obsolete, no usage."
- vulnerability: CVE-2026-0989
reason: "RelaxNG nested include stack exhaustion. Debian trixie no-dsa. No RelaxNG usage."
- vulnerability: CVE-2026-0990
reason: "Self-referencing XML catalog recursion. Debian trixie no-dsa. No untrusted catalogs."
- vulnerability: CVE-2026-0992
reason: "Repeated nextCatalog CPU exhaustion. Debian trixie no-dsa. No untrusted XML catalogs."
# --- libglib2.0 ---
- vulnerability: CVE-2026-0988
reason: "Integer overflow in g_buffered_input_stream_peek. Debian trixie no-dsa. Sid 2.87.2-3."
- vulnerability: CVE-2026-1484
reason: "Base64 encoding integer overflow on large inputs. Debian trixie no-dsa. Sid 2.86.3-5."
- vulnerability: CVE-2026-1485
reason: "Buffer underflow in content-type parsing. Debian trixie no-dsa. Sid 2.86.3-5."
- vulnerability: CVE-2026-1489
reason: "Integer overflow in Unicode case conversion. Debian trixie no-dsa. Sid 2.86.3-5."
# --- systemd (libsystemd0/libudev1) ---
- vulnerability: CVE-2026-4105
reason: "systemd-machined D-Bus privilege escalation. Not exploitable: container has no systemd/D-Bus. Needs 260~rc3-1, Trixie has 257.9-1~deb13u1."
- vulnerability: CVE-2026-29111
reason: "Vuln in libsystemd0/libudev1. No fix in Trixie. Container has no systemd runtime."
- vulnerability: CVE-2026-40225
reason: "udev local root exec via malicious hardware with unsanitized kernel output. Trixie 257.9-1~deb13u1 vulnerable; fix in 257.12/257.13 not backported. Container has no systemd/udev runtime, no hardware access."
- vulnerability: CVE-2026-40226
reason: "systemd-nspawn escape-to-host via crafted optional config file. Trixie 257.9-1~deb13u1 vulnerable; fix not yet in Trixie. Container does not run systemd-nspawn."
- vulnerability: CVE-2026-40228
reason: "systemd-journald unintended output to user terminals via logger. Low severity (CVSS 2.9). Trixie 257.9-1~deb13u1 vulnerable; no fix available. Container has no journald/interactive terminals."
# --- util-linux (mount, login, libuuid1, etc.) ---
- vulnerability: CVE-2022-0563
reason: "chfn/chsh INPUTRC leak. Debian disables chfn-chsh in util-linux. Not exploitable."
- vulnerability: CVE-2025-14104
reason: "Heap overread in setpwnam with 256-byte usernames. Debian trixie no-dsa. Sid 2.41.3-4."
- vulnerability: CVE-2026-3184
reason: "Hostname canonicalization access-control bypass. Debian trixie no-dsa."
- vulnerability: CVE-2026-27456
reason: "util-linux low-severity CVE. Debian trixie no-dsa. No fix available."
# --- libpng ---
- vulnerability: CVE-2021-4214
reason: "Heap overflow in pngimage CLI tool. Debian: unimportant. CLI tool only, not in library."
- vulnerability: CVE-2026-34757
reason: "UAF in png_set_PLTE/tRNS/hIST leading to heap disclosure. Trixie 1.6.48-1+deb13u4 vulnerable; fix in libpng 1.6.57-1 (sid only). Used transitively via libgdk-pixbuf for WeasyPrint PDF rendering on trusted PNGs."
# --- libcap2 ---
- vulnerability: CVE-2026-4878
reason: "TOCTOU race in cap_set_file. Trixie 1:2.75-10+b8 vulnerable; fix in libcap 1:2.78-1 (sid/forky only). Container never calls cap_set_file; capabilities are set once via setpriv in entrypoint."
# --- tar ---
- vulnerability: CVE-2026-5704
reason: "Path-traversal hidden-file injection via crafted archive. Trixie 1.35+dfsg-3.1 vulnerable; Debian marks <no-dsa>, no fix yet. Container does not extract untrusted tar archives at runtime."
# --- sqlite ---
- vulnerability: CVE-2021-45346
reason: "Memory leak via crafted SQL on corrupt DB. Debian: unimportant. Negligible impact."
- vulnerability: CVE-2025-7709
reason: "FTS5 tombstone integer overflow. Debian trixie no-dsa. Sid 3.46.1-9."
- vulnerability: CVE-2025-70873
reason: "Info disclosure in zipfileInflate. Python sqlite3 module does not load zipfile extension. Not exploitable."
# --- ncurses ---
- vulnerability: CVE-2025-6141
reason: "Stack overflow in termcap postprocessing. Debian trixie no-dsa. Sid 6.6+20251231-1."
- vulnerability: CVE-2025-69720
reason: "Low vuln in ncurses 6.5+20250216-2 (ncurses-bin, ncurses-base, libncursesw6, libtinfo6). No fix in Trixie."
# --- pixman ---
- vulnerability: CVE-2023-37769
reason: "FPE in stress-test tool. Debian: unimportant. Crash in test tool, no security impact."
# --- python3.14 (base image python:3.14.6-slim) ---
# 3.14.5 fixed CVE-2026-6100, -4786, -1502, -6019; 3.14.6 (2026-06-11)
# fixed CVE-2026-8328, -9669, -7774, -3276 — those entries were removed.
# The entries below remain unfixed upstream (no CPython release has them).
#
# NOTE on Grype DB lag: Grype matches the python *binary* against NVD/CPE
# ranges, which can trail an upstream patch release by days to weeks. A
# CVE that the Debian tracker shows as fixed in the installed CPython can
# therefore still be flagged right after a base-image bump. Dismiss those
# alerts via the code-scanning API as false positives (they self-heal
# when the DB updates) instead of adding entries here — a suppression in
# this file would silently outlive the DB lag it papered over.
- vulnerability: CVE-2025-12781
reason: "base64 altchars accepts +/ regardless. Debian: minor. No alt-alphabet base64 usage."
- vulnerability: CVE-2025-15366
reason: "imaplib newline injection. Debian trixie no-dsa. No IMAP usage in container."
- vulnerability: CVE-2025-15367
reason: "poplib newline injection. Debian trixie no-dsa. No POP3 usage in container."
- vulnerability: CVE-2025-13462
reason: "tarfile AREGTYPE/DIRTYPE normalization in multi-block members. Low (CVSS 2.0). No untrusted tar extraction."
# --- coreutils ---
- vulnerability: CVE-2025-5278
reason: "Heap underread in sort CLI tool. Debian: unimportant. Crash only, no security impact."
# --- zlib ---
- vulnerability: CVE-2026-27171
reason: "CPU loop in crc32_combine. Needs zlib 1.3.2, Trixie has 1.3.1. Debian trixie no-dsa."
# --- liblzma5 (xz-utils) ---
- vulnerability: CVE-2026-34743
reason: "Buffer overflow in lzma_index_decoder when decoding Index with no Records. Trixie 5.8.1-1 vulnerable; fix in xz-utils 5.8.3-1 (sid only). Debian <no-dsa>, minor. Container decompresses .xz streams from trusted sources only (pypandoc/unstructured)."
# --- libfreetype6 ---
- vulnerability: CVE-2026-23865
reason: "Integer overflow OOB read in variable font parsing. Needs freetype 2.14.2, not in sid. Debian trixie no-dsa."
# --- openssl (libssl3t64, openssl-provider-legacy) ---
- vulnerability: CVE-2026-2673
reason: "Algorithm downgrade in OpenSSL 3.5/3.6. Low severity. Trixie no-dsa. Awaiting OpenSSL 3.5.6."
# --- harfbuzz ---
- vulnerability: CVE-2026-22693
reason: "NULL deref on malloc failure in SubtableUnicodesCache. Debian trixie no-dsa. Sid 12.3.2."
# --- nltk (transitive dep, never imported — no fix available, 3.9.3 is latest) ---
- vulnerability: GHSA-jm6w-m3j8-898g
package: { name: nltk }
reason: "High vuln in nltk 3.9.3 (Zip Slip in downloader). Transitive dep; project never calls nltk.download(). No fix released."
- vulnerability: GHSA-rf74-v2fm-23pw
package: { name: nltk }
reason: "Medium vuln in nltk 3.9.3. Transitive dep; nltk never imported or used directly. No fix released."
- vulnerability: GHSA-gfwx-w7gr-fvh7
package: { name: nltk }
reason: "Medium vuln in nltk 3.9.3. Transitive dep; nltk never imported or used directly. No fix released."
# --- torch (transitive dep for local embeddings — no fix available) ---
- vulnerability: GHSA-rrmf-rvhw-rf47
package: { name: torch }
reason: "Low-severity memory corruption via torch.jit.script. No patched release (all <=2.12.0 affected). The codebase never calls torch.jit.script and does not load untrusted TorchScript."
# --- libc6/libc-bin (glibc DNS spec violations, no fix anywhere incl. sid) ---
- vulnerability: CVE-2026-4437
reason: "gethostbyaddr DNS response parsing treats non-answer section as answer. Needs glibc >2.43, Trixie has 2.41. Debian no-dsa."
- vulnerability: CVE-2026-4438
reason: "gethostbyaddr returns invalid DNS hostname. Needs glibc >2.43, Trixie has 2.41. Debian no-dsa."
# --- pygments ---
- vulnerability: GHSA-5239-wwwm-4pmq
package: { name: pygments }
reason: "Low vuln in pygments 2.19.2. No fix released yet."
# --- dpkg ---
- vulnerability: CVE-2026-2219
reason: "High vuln in dpkg 1.22.22. No fix in Trixie yet. Base image python:3.14-slim ships this version."
# --- sed (GNU sed from Debian base image python:3.14.6-slim) ---
- vulnerability: CVE-2026-5958
reason: "Low CVE in GNU sed from base image. No public info — may be very new, disputed, or not yet indexed. sed not explicitly installed in Dockerfile; comes from the python slim base image. Container does not process untrusted input with sed. No fix in Trixie."
+24
View File
@@ -0,0 +1,24 @@
# Hadolint configuration file
# See https://github.com/hadolint/hadolint#configure
# Ignore rules that are impractical for our use case
ignored:
# DL3008: Pin versions in apt-get install
# Rationale: Pinning apt package versions is impractical because:
# - Debian package versions change frequently between releases
# - We use digest-pinned base images for reproducibility
# - Pinned versions would require constant maintenance
- DL3008
# DL3013: Pin versions in pip
# Rationale: We install build tools (pdm, playwright) where we want
# the latest compatible version. The actual application dependencies
# are pinned in pdm.lock which provides reproducibility.
- DL3013
# DL3059: Multiple consecutive RUN instructions
# Rationale: Separate RUN instructions are intentional for Docker layer
# caching. npm ci, npm run build, and pdm install have different change
# frequencies — keeping them separate avoids expensive re-downloads when
# only one step's inputs change.
- DL3059
+1
View File
@@ -0,0 +1 @@
24
+466
View File
@@ -0,0 +1,466 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: 3e8a8703264a2f4a69428a0aa4dcb512790b2c8c # v6.0.0 (frozen)
hooks:
- id: check-yaml
exclude: cookiecutter-docker/.*
- id: end-of-file-fixer
- id: trailing-whitespace
- id: check-added-large-files
args: ['--maxkb=1000']
- id: check-case-conflict
- id: forbid-new-submodules
# Secret scanning — gitleaks is the sole pre-commit secret scanner.
# detect-secrets was removed because its line-number-based .secrets.baseline
# caused constant merge conflicts and 173 KB of churn. gitleaks uses
# path/regex-based allowlists (.gitleaks.toml) that are stable across
# line changes. Additional CI coverage: Semgrep (p/secrets), Bearer (secrets).
- repo: https://github.com/adrienverge/yamllint
rev: v1.37.0
hooks:
- id: yamllint
args: ['-c', '.yamllint.yaml']
exclude: cookiecutter-docker/.*
- repo: https://github.com/gitleaks/gitleaks
rev: 2ca41cc1372d1e939a6a879f18cdc19fc1cac1ce # v8.30.0 (frozen)
hooks:
- id: gitleaks
name: Detect secrets with gitleaks
description: "Detect hardcoded secrets like passwords, API keys, and tokens"
args: ["--config", ".gitleaks.toml"]
- repo: https://github.com/shellcheck-py/shellcheck-py
rev: 745eface02aef23e168a8afb6b5737818efbea95 # v0.11.0.1 (frozen)
hooks:
- id: shellcheck
name: Shellcheck - lint shell scripts
description: "Shell script static analysis tool"
# ESLint runs as a local hook so it resolves plugins from the project's
# node_modules (mirrors-eslint installs into its own cache, which flat
# config's `import` cannot reach). Requires `npm install` first.
- repo: local
hooks:
- id: eslint
name: ESLint - lint JavaScript
entry: npx eslint --config eslint.config.js
language: system
files: \.(js|mjs)$
exclude: (node_modules|dist|build|\.min\.js|static/dist)
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: aca6d4c8045a504e2812ea4bedff1d0a09e437bc # v0.15.8 (frozen)
hooks:
# Run the linter.
- id: ruff
args: [--fix]
# Run the formatter.
- id: ruff-format
- repo: https://github.com/rhysd/actionlint
rev: a443f344ff32813837fa49f7aa6cbc478d770e62 # v1.7.9 (frozen)
hooks:
- id: actionlint
name: Lint GitHub Actions workflows
description: "Static checker for GitHub Actions workflow files"
files: ^\.github/workflows/.*\.ya?ml$
- repo: local
hooks:
- id: check-workflow-test-paths
name: Check workflow test paths exist
entry: .pre-commit-hooks/check-workflow-test-paths.py
language: script
files: ^\.github/workflows/.*\.ya?ml$
description: "Ensure pytest paths referenced in GitHub workflows exist
on disk, so a renamed/deleted test file can't silently break a CI
gate with exit code 5 (no tests collected)"
- id: check-release-gate-needs
name: Check release-gate SARIF needs lists are in sync
entry: .pre-commit-hooks/check-release-gate-needs.py
language: script
files: ^\.github/workflows/.*\.ya?ml$
pass_filenames: false
description: "Every scan that uploads SARIF to code scanning must be in
both release-gate.yml needs lists (check-code-scanning-alerts and
release-gate-summary); a missing one can silently bypass the gate (#4817)"
- id: custom-code-checks
name: Custom Code Quality Checks
entry: .pre-commit-hooks/custom-checks.py
language: script
files: \.py$
description: "Check for loguru usage, logger.exception vs logger.error, raw
SQL, redundant {e} in logger.exception, and non-UTC datetime usage"
- id: check-env-vars
name: Environment Variable Access Check
entry: .pre-commit-hooks/check-env-vars.py
language: script
files: \.py$
description: "Ensure environment variables are accessed through SettingsManager"
- id: check-author-identity
name: Author identity consistency
entry: .pre-commit-hooks/check-author-identity.py
language: script
always_run: true
pass_filenames: false
description: "Ensure commits by a declared project author use that author's declared identity"
- id: file-whitelist-check
name: File Whitelist Security Check
entry: .pre-commit-hooks/file-whitelist-check.sh
language: script
types: [file]
description: "Check for allowed file types and file sizes"
- id: check-deprecated-db-connection
name: Check for deprecated get_db_connection usage
entry: .pre-commit-hooks/check-deprecated-db.py
language: script
files: \.py$
description: "Ensure code uses per-user database connections instead of deprecated
shared database"
- id: check-ldr-db-usage
name: Check for ldr.db usage
entry: .pre-commit-hooks/check-ldr-db.py
language: script
files: \.py$
description: "Prevent usage of shared ldr.db database - all data must use
per-user encrypted databases"
- id: check-journal-quality-readonly
name: Check journal_quality.db opens are read-only
entry: .pre-commit-hooks/check-journal-quality-readonly.py
language: script
files: \.py$
description: "The compiled journal-quality DB has exactly one writer
(build_db). All other consumers must open it with SQLite mode=ro."
- id: check-research-id-type
name: Check for incorrect research_id type hints
entry: .pre-commit-hooks/check-research-id-type.py
language: script
files: \.py$
description: "Ensure research_id is always treated as string/UUID, never as
int"
- id: check-deprecated-settings-wrapper
name: Check for deprecated get_setting_from_db_main_thread usage
entry: .pre-commit-hooks/check-deprecated-settings-wrapper.py
language: script
files: \.py$
description: "Prevent usage of redundant get_setting_from_db_main_thread wrapper
- use SettingsManager directly"
- id: check-datetime-timezone
name: Check DateTime columns use UtcDateTime
entry: scripts/pre_commit/check_datetime_timezone.py
language: script
files: \.py$
description: "All DateTime columns (models and migrations) must use UtcDateTime from sqlalchemy_utc"
- id: check-session-context-manager
name: Check for try/finally session patterns
entry: .pre-commit-hooks/check-session-context-manager.py
language: script
files: \.py$
description: "Ensure SQLAlchemy sessions use context managers instead of try/finally
blocks"
- id: check-settings-manager-thread-safety
name: Check get_settings_manager() thread-safety
entry: .pre-commit-hooks/check-settings-manager-thread-safety.py
language: script
files: \.py$
exclude: ^tests/
description: "Ensure get_settings_manager() passes db_session in background-thread
code"
- id: check-settings-key-namespace
name: Check settings key namespace
entry: .pre-commit-hooks/check-settings-key-namespace.py
language: script
files: \.(py|js|mjs)$
exclude: ^(tests/|\.pre-commit-hooks/)
description: "Ensure hardcoded settings keys match allowed prefixes from
settings_routes.py"
- id: check-pathlib-usage
name: Check for os.path usage
entry: .pre-commit-hooks/check-pathlib-usage.py
language: script
files: \.py$
description: "Enforce using pathlib.Path instead of os.path for better cross-platform
compatibility"
- id: check-no-external-resources
name: Check for external CDN/resource references
entry: .pre-commit-hooks/check-external-resources.py
language: script
files: \.(html|js|css|py)$
exclude: ^tests/
description: "Ensure no external CDN URLs or external resources are referenced
in the codebase"
- id: check-target-blank-rel
name: Check external target=_blank links have rel=noopener noreferrer
entry: .pre-commit-hooks/check-target-blank-rel.py
language: script
files: \.(html|js|jsx|ts|tsx|mjs|jinja2|jinja)$
exclude: ^(tests?/|vendor/|.*/vendor/)
description: "Prevent tabnabbing and Referer leakage on external
target=_blank links"
- id: check-css-class-prefix
name: Check CSS class names have ldr- prefix
entry: .pre-commit-hooks/check-css-class-prefix.py
language: script
files: \.(css|html|js|jsx|ts|tsx|mjs)$
exclude: (tests?/|spec\.|test\.|\.test\.|\.spec\.)
description: "Ensure all LDR-specific CSS classes are prefixed with 'ldr-'
to prevent conflicts"
- id: check-sensitive-logging
name: Check for sensitive data in logging
entry: .pre-commit-hooks/check-sensitive-logging.py
language: script
files: \.py$
exclude: ^(tests/|examples/)
description: "Prevent logging of sensitive data"
- id: check-url-security
name: Check for unsafe URL handling in JavaScript
entry: .pre-commit-hooks/check-url-security.py
language: script
files: \.js$
description: "Ensure JavaScript code validates URLs to prevent XSS attacks"
- id: check-double-escaping
name: Check for double HTML escaping in JavaScript
entry: .pre-commit-hooks/check-double-escaping.py
language: script
files: \.js$
exclude: (tests?/|spec\.|test\.|\.test\.|\.spec\.)
description: "Detect escapeHtml() inside functions that already escape internally
(showError, showSuccess, showInfo, showAlert)"
- id: check-raw-console
name: Check for raw console.* calls in JavaScript
entry: .pre-commit-hooks/check-raw-console.py
language: script
files: \.js$
exclude: (tests?/|spec\.|test\.|\.test\.|\.spec\.)
description: "Ensure JS files use SafeLogger instead of raw console.log/error/warn"
- id: check-bearer-disable
name: Check bearer:disable directives are well-formed
entry: .pre-commit-hooks/check-bearer-disable.py
language: script
files: ^src/local_deep_research/.*\.(py|js)$
description: "Bearer ignores a bearer:disable directive unless it is the bare
rule id on its own line above the statement; reject same-line or
trailing-prose directives that would silently suppress nothing"
- id: check-loguru-formatting
name: Check loguru formatting style
entry: .pre-commit-hooks/check-loguru-formatting.py
language: script
files: \.py$
description: "Prevent printf-style %s/%d placeholders in loguru logger calls"
- id: check-open-encoding
name: Check open() calls have explicit encoding
entry: .pre-commit-hooks/check-open-encoding.py
language: script
files: \.py$
exclude: ^tests/
description: "Enforce explicit encoding= on text-mode open() calls (issue #3743)"
- id: check-image-pinning
name: Check Docker Image Pinning
entry: .pre-commit-hooks/check-image-pinning.sh
language: script
files:
(Dockerfile|docker-compose.*\.ya?ml|\.github/workflows/.*\.ya?ml)$
description: "Ensure all Docker images use SHA256 digests for security"
- id: check-safe-requests
name: Check for unsafe requests usage (SSRF protection)
entry: .pre-commit-hooks/check-safe-requests.py
language: script
files: \.py$
description: "Prevent direct requests.get/post/Session - use safe_get/safe_post/SafeSession"
- id: check-silent-cleanup
name: Check for silent exceptions in close() methods
entry: .pre-commit-hooks/check-silent-cleanup.py
language: script
files: \.py$
exclude: ^tests/
description: "Ensure close() methods log cleanup failures instead of silently passing"
- id: check-silent-exceptions
name: Check for silent exception swallowing
entry: .pre-commit-hooks/check-silent-exceptions.py
language: script
files: \.py$
exclude: ^(tests/|examples/)
description: "Flag except Exception: pass without logging — use logger.debug()
or # noqa: silent-exception to suppress"
- id: check-service-context-managers
name: Check services use context managers
entry: .pre-commit-hooks/check-service-context-managers.py
language: script
files: \.py$
description: "Ensure DownloadService, LibraryRAGService, LocalEmbeddingManager use context managers"
- id: check-absolute-module-paths
name: Check for absolute module paths
entry: .pre-commit-hooks/check-absolute-module-paths.py
language: script
files: \.(py|json)$
description: "Ensure module_path values use relative imports, not absolute paths"
- id: check-utcnow-parens
name: Check utcnow() has parentheses in Column defaults
entry: .pre-commit-hooks/check-utcnow-parens.py
language: script
files: \.py$
exclude: ^(\.pre-commit-hooks/|tests/)
description: "Prevent default=utcnow (missing parens) — utcnow is a FunctionElement
class, parens are required to create the SQL expression object"
- id: check-pdm-lock
name: Check pdm.lock sync
entry: .pre-commit-hooks/check-pdm-lock.sh
language: script
files: ^pyproject\.toml$
description: "Ensure pdm.lock is up-to-date with pyproject.toml"
- id: check-version-sync
name: Check Version Sync
entry: .pre-commit-hooks/check-version-sync.py
language: script
files: ^(package\.json|src/local_deep_research/__version__\.py)$
pass_filenames: false
description: "Ensure package.json version matches src/local_deep_research/__version__.py"
- id: check-codeowners-sync
name: Check CODEOWNERS Sync
entry: .pre-commit-hooks/check-codeowners-sync.py
language: script
files: ^\.github/(CODEOWNERS|workflows/pr-triage\.yml)$
pass_filenames: false
description: "Ensure pr-triage.yml CODEOWNERS const matches .github/CODEOWNERS global owners"
- id: check-golden-master-settings
name: Check Golden Master Settings Sync
entry: .pre-commit-hooks/check-golden-master-settings.py
language: script
files: ^src/local_deep_research/defaults/.*\.json$
pass_filenames: false
description: "Ensure golden master settings are updated when defaults change"
- id: check-adr-numbers
name: Check ADR numbering
entry: .pre-commit-hooks/check-adr-numbers.py
language: script
files: ^docs/decisions/.*\.md$
pass_filenames: false
description: "Ensure docs/decisions ADR numbers are unique (no two files share a NNNN prefix)"
- id: check-readme-links
name: Check README/docs links and examples
entry: .pre-commit-hooks/check-readme-links.py
language: script
pass_filenames: false
files: ^(README\.md|SECURITY\.md|CONTRIBUTING\.md|docs/.*\.md|src/local_deep_research/web_search_engines/engine_registry\.py)$
description: "Validate relative links, heading anchors, python -m
examples, and engine=... names in README.md, SECURITY.md,
CONTRIBUTING.md, and docs/**.md against the repo, so renamed
files/headings or removed modules/engines can't silently break
the documentation"
- id: check-layer-imports
name: Check infrastructure packages don't import from web/
entry: .pre-commit-hooks/check-layer-imports.py
language: script
files: \.py$
exclude: ^tests/
description: "Prevent settings/, utilities/, security/, config/ from importing
web/ — breaks CLI/MCP/API usage"
- id: recommend-docs
name: Documentation Reminder
entry: .pre-commit-hooks/recommend-docs.py
language: script
pass_filenames: false
always_run: true
verbose: true
description: "Gentle reminder to update docs when source files change"
- id: recommend-tests
name: Test Coverage Recommendation
entry: .pre-commit-hooks/recommend-tests.py
language: script
pass_filenames: false
always_run: true
verbose: true
description: "Direct recommendation to include tests when source files change"
- id: recommend-vitest
name: Vitest Coverage Nudge
entry: .pre-commit-hooks/recommend-vitest.py
language: script
pass_filenames: false
always_run: true
verbose: true
description: "Nudge to add vitest tests when frontend JS source changes lack them"
- id: recommend-performance-tests
name: Performance Test Reminder
entry: .pre-commit-hooks/recommend-performance-tests.py
language: script
pass_filenames: false
always_run: true
verbose: true
description: "Remind contributors about subsystem fast tests and opt-in live tests"
- id: check-changelog-fragments
name: Check changelog fragment names
entry: .pre-commit-hooks/check-changelog-fragments.py
language: script
files: ^changelog\.d/
pass_filenames: false
description: "Fail on changelog.d/ fragments with undeclared towncrier categories -
towncrier silently drops them from the rendered release notes"
- id: recommend-release-notes
name: Release Notes Reminder
entry: .pre-commit-hooks/recommend-release-notes.py
language: script
pass_filenames: false
always_run: true
verbose: true
description: "Remind contributors to add a changelog.d/<id>.<category>.md fragment for substantial changes (rendered by towncrier at release prep)"
- id: recommend-pr-description-update
name: PR Description Freshness Reminder
entry: .pre-commit-hooks/recommend-pr-description-update.py
language: script
pass_filenames: false
always_run: true
verbose: true
description: "Nudge to update PR description and pending changelog fragments on PR branches"
- id: require-tests
name: Require Tests for Substantial Changes
entry: .pre-commit-hooks/require-tests.py
language: script
pass_filenames: false
always_run: true
description: "Block commits with substantial new code but no tests"
- id: require-strategy-deletion-docs
name: Require docs/strategies/deleted/ note on strategy deletion
entry: .pre-commit-hooks/require-strategy-deletion-docs.py
language: script
pass_filenames: false
always_run: true
description: "Block commits that delete a search strategy file without adding/updating a notes file in docs/strategies/deleted/"
- id: check-unmarked-sleep
name: Check time.sleep() in tests has @pytest.mark.slow
entry: .pre-commit-hooks/check-unmarked-sleep.py
language: script
files: ^tests/.*\.py$
description: "Ensure test functions with time.sleep(>0.1s) are marked
@pytest.mark.slow so CI can skip them with -m 'not slow'"
- id: check-fixture-duplication
name: Check for duplicated test fixtures
entry: .pre-commit-hooks/check-fixture-duplication.py
language: script
files: ^tests/.*\.py$
description: "Warn when test files redefine app/client/authenticated_client
fixtures from root conftest.py"
- id: check-shadow-tests
name: Check for shadow tests (no SUT import)
entry: .pre-commit-hooks/check-shadow-tests.py
language: script
files: ^tests/.*\.py$
# Black-box (HTTP-client) and meta (hook/guardian) test trees
# legitimately never import local_deep_research; scattered
# survivors use a '# allow: no-sut-import' marker instead.
exclude: ^tests/(api_tests|api_tests_with_login|ui_tests|hooks|ci|health_check|accessibility_tests|performance/api_auth)/|_hook\.py$
description: "Block test modules that import nothing from local_deep_research
(shadow tests). Opt out a legitimate no-import test with
'# allow: no-sut-import — <reason>'. See CONTRIBUTING.md."
- id: check-weak-tests
name: Check for tautological / no-op tests
entry: .pre-commit-hooks/check-weak-tests.py
language: script
files: ^tests/.*\.py$
description: "Block tests that pass regardless of the SUT: assert-True-only,
import-existence tautologies, `... or True`, and
`x is None or x is not None`. Opt out per-test with `# allow: weak-test`."
# Hooks intentionally NOT added
# See docs/decisions/0002-pre-commit-hook-reviews.md for full rationale.
#
# codespell (PR #3222): CamelCase false positives on code identifiers
# mypy (PR #3231): ignore_errors covers most modules; already in CI
# vulture (PR #3230): Advisory-only in CI; use workflow instead
# innerHTML regex (PR #3227): Nested-brace regex fails on real JS files
# utcnow callable (PR #3218): Conflicts with existing check-utcnow-parens hook
+56
View File
@@ -0,0 +1,56 @@
"""Shared helpers for changelog.d/ fragment-name validation.
Used by the blocking ``check-changelog-fragments`` hook and the advisory
``recommend-release-notes`` hook so the fragment grammar and category
list can't drift between them.
"""
import re
import tomllib
from pathlib import Path
# changelog.d/<id>.<category>[.<n>].md or changelog.d/+<slug>.<category>[.<n>].md
# - <id>: integer PR/issue number
# - +<slug>: orphan fragment with no PR/issue, slug is [A-Za-z0-9_-]+
# - .<n>: optional integer counter suffix for multiple fragments of the
# same (id, category) — towncrier renders each as a separate bullet,
# all linked back to the same PR/issue.
FRAGMENT_RE = re.compile(
r"^(?:\d+|\+[A-Za-z0-9_-]+)\.(?P<category>[a-z]+)(?:\.\d+)?\.md$"
)
def load_categories(pyproject_path):
"""Read ``[[tool.towncrier.type]].directory`` entries from pyproject.toml.
Raises on an unreadable file or a missing/empty type table. The
advisory hook catches and falls back to a default list; the blocking
hook lets the error propagate so a broken towncrier config fails
loud instead of validating against a guessed category list.
"""
with Path(pyproject_path).open("rb") as fh:
cfg = tomllib.load(fh)
types = cfg["tool"]["towncrier"]["type"]
cats = tuple(t["directory"] for t in types if "directory" in t)
if not cats:
raise ValueError(
f"no [[tool.towncrier.type]] entries in {pyproject_path}"
)
return cats
def classify_fragment(name, categories):
"""Classify a fragment filename against the grammar and *categories*.
Returns ``("ok", category)`` for a valid fragment, ``("bad-category",
category)`` for a well-formed name whose category isn't declared, or
``("bad-name", None)`` for a filename that doesn't match the fragment
pattern at all.
"""
m = FRAGMENT_RE.match(name)
if not m:
return "bad-name", None
category = m.group("category")
if category not in categories:
return "bad-category", category
return "ok", category
+232
View File
@@ -0,0 +1,232 @@
"""Shared utility module for pre-commit hooks analyzing staged files."""
import subprocess
from dataclasses import dataclass, field
from pathlib import PurePosixPath
@dataclass
class StagedFileInfo:
path: str
added: int
removed: int
is_new: bool
category: str
@dataclass
class CommitAnalysis:
source_files: list[StagedFileInfo] = field(default_factory=list)
test_files: list[StagedFileInfo] = field(default_factory=list)
doc_files: list[StagedFileInfo] = field(default_factory=list)
@property
def new_source_files(self):
return [f for f in self.source_files if f.is_new]
@property
def total_source_added(self):
return sum(f.added for f in self.source_files)
@property
def has_tests(self):
return len(self.test_files) > 0
@property
def has_docs(self):
return len(self.doc_files) > 0
# Directories/patterns excluded from "source" classification
_EXCLUDED_DIRS = {
"migrations",
"alembic",
"config",
"settings",
"scripts",
".pre-commit-hooks",
"examples",
"docs",
}
_EXCLUDED_BASENAMES = {"__init__.py", "conftest.py"}
def _run_git(args):
"""Run a git command and return stdout lines.
Returns empty list if git command fails (e.g., not in a git repo).
"""
result = subprocess.run(
["git"] + args,
capture_output=True,
text=True,
)
if result.returncode != 0:
return []
return result.stdout.strip().splitlines() if result.stdout.strip() else []
def _parse_numstat(lines):
"""Parse git diff --numstat output into {filepath: (added, removed)}."""
stats = {}
for line in lines:
parts = line.split("\t")
if len(parts) != 3:
continue
added_str, removed_str, filepath = parts
# Binary files show '-' for counts
added = int(added_str) if added_str != "-" else 0
removed = int(removed_str) if removed_str != "-" else 0
stats[filepath] = (added, removed)
return stats
def classify_file(path):
"""Classify a file path into a category string.
Returns one of: source, test, doc, init, conftest, migration, config,
hook, script, other
"""
p = PurePosixPath(path)
basename = p.name
parts_set = set(p.parts)
# Doc files
if p.suffix == ".md":
return "doc"
# JavaScript files
if p.suffix == ".js":
# Vitest test files: tests/js/**/*.test.js
if "tests" in parts_set and basename.endswith(".test.js"):
return "test"
# Source: under src/.../static/js/
if "static" in parts_set and "js" in parts_set and parts_set & {"src"}:
# Skip vendored / third-party
if "vendor" in parts_set or "lib" in parts_set:
return "other"
return "source"
return "other"
# Non-Python files are "other"
if p.suffix != ".py":
return "other"
# Specific basename exclusions
if basename == "__init__.py":
return "init"
if basename == "conftest.py":
return "conftest"
# Test files: under tests/ or matching test_*.py / *_test.py
if (
"tests" in parts_set
or basename.startswith("test_")
or basename.endswith("_test.py")
):
return "test"
# Check excluded directory patterns
for part in p.parts:
if part.lower() in _EXCLUDED_DIRS:
return _dir_to_category(part.lower())
# Source files: under src/
if len(p.parts) > 0 and p.parts[0] == "src":
return "source"
return "other"
def _dir_to_category(dirname):
"""Map excluded directory names to categories."""
mapping = {
"migrations": "migration",
"alembic": "migration",
"config": "config",
"settings": "config",
"scripts": "script",
".pre-commit-hooks": "hook",
"examples": "other",
"docs": "doc",
}
return mapping.get(dirname, "other")
def suggest_test_path(source_path):
"""Suggest a test file path for a given source file.
Python: src/local_deep_research/web/api.py -> tests/web/test_api.py
JS: src/local_deep_research/web/static/js/components/foo.js
-> tests/js/components/foo.test.js
"""
p = PurePosixPath(source_path)
# JavaScript: mirror tests/js/<subpath under static/js/>
if p.suffix == ".js":
parts = list(p.parts)
if "static" in parts and "js" in parts:
js_idx = parts.index("js", parts.index("static"))
sub_parts = parts[js_idx + 1 :]
if sub_parts:
stem = PurePosixPath(sub_parts[-1]).stem
# Convert snake_case → kebab-case to match existing test naming
stem = stem.replace("_", "-")
test_name = f"{stem}.test.js"
test_parts = ["tests", "js"] + sub_parts[:-1] + [test_name]
return str(PurePosixPath(*test_parts))
return f"tests/js/{p.stem.replace('_', '-')}.test.js"
parts = list(p.parts)
# Strip leading src/ and package name (e.g., src/local_deep_research/)
if parts and parts[0] == "src":
parts = parts[1:]
if parts and not parts[0].startswith("test"):
parts = parts[1:] # Remove package name like local_deep_research
# Build test path
if parts:
test_name = f"test_{parts[-1]}"
test_parts = ["tests"] + parts[:-1] + [test_name]
else:
test_parts = ["tests", f"test_{p.name}"]
return str(PurePosixPath(*test_parts))
def analyze_commit():
"""Analyze staged files and return a CommitAnalysis."""
# Get list of staged files with their status
status_lines = _run_git(["diff", "--cached", "--name-status"])
new_files = set()
for line in status_lines:
parts = line.split("\t")
if len(parts) >= 2 and parts[0].startswith("A"):
new_files.add(parts[-1])
# Get line counts
numstat_lines = _run_git(["diff", "--cached", "--numstat"])
stats = _parse_numstat(numstat_lines)
analysis = CommitAnalysis()
for filepath, (added, removed) in stats.items():
category = classify_file(filepath)
info = StagedFileInfo(
path=filepath,
added=added,
removed=removed,
is_new=filepath in new_files,
category=category,
)
if category == "source":
analysis.source_files.append(info)
elif category == "test":
analysis.test_files.append(info)
elif category == "doc":
analysis.doc_files.append(info)
return analysis

Some files were not shown because too many files have changed in this diff Show More